Predicting Risk of Serious Bacterial Infections in Febrile Children in the Emergency Department
BACKGROUND: Improving the diagnosis of serious bacterial infections (SBIs) in the children’s emergency department is a clinical priority. Early recognition reduces morbidity and mortality, and supporting clinicians in ruling out SBIs may limit unnecessary admissions and antibiotic use.
METHODS: A prospective, diagnostic accuracy study of clinical and biomarker variables in the diagnosis of SBIs (pneumonia or other SBI) in febrile children <16 years old. A diagnostic model was derived by using multinomial logistic regression and internally validated. External validation of a published model was undertaken, followed by model updating and extension by the inclusion of procalcitonin and resistin.
RESULTS: There were 1101 children studied, of whom 264 had an SBI. A diagnostic model discriminated well between pneumonia and no SBI (concordance statistic 0.84, 95% confidence interval 0.78–0.90) and between other SBIs and no SBI (0.77, 95% confidence interval 0.71–0.83) on internal validation. A published model discriminated well on external validation. Model updating yielded good calibration with good performance at both high-risk (positive likelihood ratios: 6.46 and 5.13 for pneumonia and other SBI, respectively) and low-risk (negative likelihood ratios: 0.16 and 0.13, respectively) thresholds. Extending the model with procalcitonin and resistin yielded improvements in discrimination.
CONCLUSIONS: Diagnostic models discriminated well between pneumonia, other SBIs, and no SBI in febrile children in the emergency department. Improvements in the classification of nonevents have the potential to reduce unnecessary hospital admissions and improve antibiotic prescribing. The benefits of this improved risk prediction should be further evaluated in robust impact studies.
- c statistic —
- concordance statistic
- CI —
- confidence interval
- CRP —
- C-reactive protein
- CSF —
- cerebrospinal fluid
- ED —
- emergency department
- NGAL —
- neutrophil gelatinase-associated lipocalin
- OR —
- odds ratio
- PCR —
- polymerase chain reaction
- PCT —
- SBI —
- serious bacterial infection
What’s Known on This Subject:
Failure to identify serious infections in children results in adverse outcomes and a failure to rule out serious infections results in unnecessary antibiotic use and hospital admissions. Multivariable clinical-risk prediction models appear to discriminate well between serious and self-limiting infections.
What This Study Adds:
In a study of 1101 children of all ages, risk prediction models discriminated well between pneumonia, other serious bacterial infections, and none. A published model performed well on external validation, and model extension with procalcitonin and resistin improved discrimination.
Acute febrile illness is among the most common of all presentations to the children’s emergency department (ED).1 In this context, the probability of serious bacterial infections (SBIs) is ∼7%, predominantly lower than respiratory or urinary tract infection.2
The prompt recognition of SBI is fundamental to effective management. Children with meningococcal disease are frequently missed at initial presentation,3 and delayed recognition increases mortality.4,5 Although rates of invasive infection have declined with the introduction of conjugate vaccines,6–8 SBIs remain an important contributor to childhood morbidity and mortality.9
In the United Kingdom, as rates of invasive infections have declined, the number of children who are admitted to the hospital has increased.10 The greatest increase is in young children with uncomplicated admissions for acute infections.11 Supporting clinicians in ruling out SBIs may reduce unnecessary hospital admissions in children.12
A number of studies have reported the diagnostic accuracy of clinical13 and laboratory14 variables in febrile children. More recently, risk prediction models that combine clinical variables have been evaluated,2,15 and in one, the addition of the C-reactive protein (CRP) improved diagnostic accuracy.16 We have previously reported the combined performance of procalcitonin (PCT), resistin, and neutrophil gelatinase-associated lipocalin (NGAL) in Malawian children.17
Diagnostic accuracy studies of febrile children have so far failed to impact clinical practice. Restrictive inclusion criteria (such as age, temperature, or clinical syndrome18) have limited their external validity, and few have progressed to validation in external populations. We therefore set out to derive and internally validate a multivariable risk prediction model and to externally validate a previously published model16 for the diagnosis of SBIs in febrile children of all ages.
This was a prospective, diagnostic accuracy study of clinical and biomarker variables in the diagnosis of SBIs in children who presented to the Alder Hey Children’s Hospital ED. This is the busiest children’s ED in the United Kingdom, and it manages 60 000 attendances each year. Recruitment was undertaken between November 2010 and April 2012. The study is reported in line with the Standards for Reporting of Diagnostic Accuracy and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guidelines.19,20
Children <16 years of age with fever (>38°C) or a history of fever were eligible if they required blood tests as part of their clinical management. Children with primary immunodeficiency were excluded. By using previous estimates of sensitivity (65%) and specificity (90%) and a rate of SBI of 15%, a sample size of 2300 was proposed. For skin and soft tissue infections, the reference standard for an SBI was that children were deemed by the clinical team to require intravenous antibiotics. Because the outcome diagnosis was solely based on a clinical decision (and because this was true of all such cases), these children (n = 82) were excluded (Fig 1).
The Generation R Young Persons’ Advisory Group (www.generationr.org.uk), initiated by the National Institute for Health Research, helped design patient information leaflets for young people and families. In the course of the study, the group explored improvements in the recognition of serious infections and they discussed diagnostic tests by using various samples (such as saliva or blood). This involvement has informed the design of subsequent studies.
Relevant clinical and biomarker variables were identified from the literature, including 2 large systematic reviews.13,14 Clinical data were entered onto a proforma at the time of the clinical assessment. When possible, this was done by the attending clinician. When the proforma was incomplete, missing clinical information was retrieved from the clinical notes when explicitly referenced. Paper proformas were collected daily by the study team. All proformas were cross-checked against the clinical notes, which were electronically scanned and stored. Missing or ambiguous data were recorded as missing. Data collection and entry into the database was blinded to the final outcomes.
The tests performed on study subjects are recorded in Supplemental Table 4. All samples were processed in Clinical Pathology Accredited laboratories. Blood (0.5–1 mL) inoculated into culture bottles was monitored by using the BacT/ALERT 3D system (BioMerieux, Marcy l'Etoile, France) . Positive cultures were processed in line with UK standards for microbiology investigations developed by Public Health England.21 Specific Streptococcus pneumoniae and Neisseria meningitidis polymerase chain reaction (PCR) assays were performed at the Meningococcal Reference Unit in Manchester, United Kingdom.22,23 Urine and cerebrospinal fluid (CSF) underwent microscopy and culture on agar gel plates and were processed in line with UK standards. Multiplex PCR was performed on respiratory (respiratory syncytial virus, influenza A and B, parainfluenza 1-3, adenovirus, rhinovirus, and human metapneumovirus) and CSF (herpes simplex virus 1 and 2, varicella-zoster, and enterovirus) samples at the regional laboratory in Manchester. Starting in April 2011, respiratory PCRs were performed by using the FilmArray respiratory viral panel (BioMerieux; Marcy l'Etoile, France) and additionally identified parainfluenza 4, rhinovirus, enterovirus, and coronavirus 1-4.
Blood (0.5–1 mL) was collected into lithium heparin and plasma stored in Sarstedt microtubes (Sarstedt AG & Co; Nümbrecht, Germany) at −80°C within 1 hour. Before analysis samples were thawed, vortex was mixed and centrifuged to remove bubbles and particulate matter. PCT analysis was undertaken on the B.R.A.H.M.S. Kryptor (Thermo Fisher Scientific Inc; Raleigh, NC) according to the manufacturer’s instructions. Quality-control samples were analyzed with each run. NGAL and resistin were analyzed by using validated commercial enzyme-linked immunosorbent assay.
In common with other published studies, outcome diagnoses were determined by a composite reference standard that incorporated clinical, microbiological, and radiologic features (Supplemental Table 5).14,15,24,25 By using these predefined criteria, a pediatric research fellow and a pediatric infectious-disease consultant independently attributed the outcome diagnosis. In the case of disagreement, a second pediatric infectious-disease consultant determined the final outcome. Children who failed to meet the predefined criteria for SBIs were considered to have no SBI. Subjects were followed up for 28 days to reduce misclassification.
Analysis was undertaken in R, version 3.0.1 (R Foundation for Statistical Computing, Vienna, Austria).26 Missing data were handled by 10-fold multiple imputation by using fully conditional specification implemented by the MICE package.27 In this method, missing values are replaced by values that are drawn from a conditional distribution that is specific to each predictor variable and is defined by its own imputation model. Data were assumed to be missing at random. The proportion of missing data relating to each clinical variable is recorded in Supplemental Table 6.
Model Derivation, Validation, and Updating
The data set was randomized into a split-sample derivation and validation set. Univariate analysis of clinical and biomarker variables was undertaken by using logistic regression for the outcomes of SBIs. Explanatory variables were examined for evidence of collinearity. Scatter plots and generalized additive model plots28 fitted by using the gam() function in the mgcv package were examined for evidence of nonlinearity on the log-odds scale.29 Piecewise and polynomial transformations were undertaken when appropriate. Plausible interaction terms were explored, including interactions between age, heart rate, and respiratory rate. A multivariable model was derived by using a forward stepwise method. Improvements in model fit were tested by means of a likelihood ratio test (α = .05), and variables associated with a significant improvement were retained. Having identified a parsimonious model for SBIs, these variables were then included in a multinomial regression model for the categorical outcomes of pneumonia, other SBI, and no SBI.
External validation of the model published by Nijman et al16 was undertaken by using the published coefficients. A comparison of study participants is given in Supplemental Table 7. The model was updated by refitting variables and estimating the individual coefficients then extended by the inclusion of PCT and resistin. This strategy preserved the original model structure and avoided deriving an entirely new model. The biomarkers were chosen by having observed their values in our earlier model derivation. Additional clinical variables were not investigated because they appeared less predictive in our model derivation, and plausible clinical variables were adequately represented by the published model.
Performance characteristics of the fitted models at various risk thresholds were estimated by using the epiR package.30 Discrimination was measured by using the concordance statistic (c statistic) and illustrated by receiver operating characteristic curves using the pROC package.31 The c statistic estimates the probability that a randomly selected subject with the outcome of interest has a higher predicted probability than a randomly selected subject without it. Comparison of the c statistic was undertaken by using the DeLong method.32 For the multinomial regression model, the c statistic estimated discrimination between pairs of patients (a patient with pneumonia and a patient with no SBI, or a patient with other SBI and a patient with no SBI). The 95% confidence intervals (CIs) were estimated with a bootstrapping process by using 2000 bootstrap replicates. Calibration of the models (how closely risk predictions fit observed cases) was illustrated by using multinomial calibration plots.33
In the absence of established methods to report classification in multinomial risk-prediction models, we compared crude classification (that is, the most likely diagnosis predicted by the multinomial models) in the updated model with the extended model. To investigate potential clinical utility, we estimated the ability of the models to rule out (predictions for both categories of SBIs <5%) or rule in (prediction of either category >20%) SBIs. These thresholds represent approximately half and double, respectively, the observed event rate in the study population.
Approval for the study was granted by the Greater Manchester West Research Ethics Committee (10/H1014/53) and by the Alder Hey Children’s Hospital R&D department.
Between November 1, 2010 and April 3, 2012, 7949 children presented to the Alder Hey Children’s ED with fever. Of these, 1872 were eligible for inclusion, and 1101 were recruited to the study (Fig 1). The median age was 2.4 years (interquartile range 0.9–5.7 years), and 55% were boys. Approximately one-third of the children had significant comorbidities (Table 1). Two hundred and sixty-four children (24.0%) were diagnosed with SBIs (Supplemental Fig 5).
The probability of pneumonia and other SBIs increased linearly with heart rate, respiratory rate, and temperature. Consistent with other studies, increased work of breathing (odds ratio [OR] 10.4, 95% CI 6.69 to 16.2), hypoxia (9.29, 95% CI 5.35 to 16.1), and other respiratory variables were significantly associated with pneumonia. These features reduced the probability of other SBIs. Neck stiffness, a bulging fontanelle, irritability, and dysuria were associated with other SBIs. Prolonged capillary refill time was associated with other SBIs (1.43, 95% CI 1.05 to 1.97) but not pneumonia, whereas the presence of a rash reduced the probability of both pneumonia and other SBIs. Univariate ORs are presented in Supplemental Fig 6. CRP, PCT, NGAL, and resistin were all associated with SBIs (Supplemental Table 8).
Model Derivation and Internal Validation
The derived model included the variables respiratory rate and normal air entry alongside CRP, PCT, and resistin (Supplemental Table 9). Fitting CRP as a piecewise term improved the model fit. The model discriminated well on internal validation (c statistic 0.84, 95% CI 0.78 to 0.90 for pneumonia; and 0.77, 95% CI 0.71 to 0.83 for other SBIs). Calibration plots suggested that the model overestimated the risk of pneumonia (Fig 2).
External Validation and Updating of the Nijman Model
The published model of Nijman et al16,34 was validated in the complete data set (n = 1101). By using the published coefficients, the model discriminated well between pneumonia and no SBI although not as well between other SBIs and no SBI (c statistic 0.85 and 0.76, respectively, Supplemental Fig 7). Model calibration was poor, although calibration plots indicated that predicted risks and observed outcomes were highly correlated (Fig 3).
By observing the correlation between predicted probabilities and observed outcomes in the poorly calibrated model, we updated the model by re-estimating the individual coefficients. No attempt was made to adjust the functional form of the predictor variables. The refitted model discriminated well (c statistic 0.88 and 0.82 for pneumonia and other SBIs, respectively) and was well calibrated (Fig 4). The model was then extended by the inclusion of PCT and resistin. This improved discrimination of pneumonia (c statistic increased from 0.88 to 0.90, P = .03) and other SBI models (from 0.82 to 0.84, P = .03), and calibration remained good (Supplemental Figure 8).
The performance characteristics of the updated and extended models are summarized in Table 2. At a low-risk threshold of 5%, the extended pneumonia model had a sensitivity of 92% (95% CI 85% to 96%) and a negative likelihood ratio of 0.12 (0.06 to 0.23). For other SBIs, model sensitivity was 92% (86% to 95%), and negative likelihood ratio was 0.21 (0.12 to 0.35). At a high-risk threshold (>20%), specificity was 89% (95% CI 87% to 91%) for pneumonia, with a positive likelihood ratio of 6.69 (5.30 to 8.44) and 86% (83% to 88%) and a positive likelihood ratio of 4.96 (4.07 to 6.03) for other SBIs.
Classification (determined by likeliest outcome category) was similar between the updated and extended models (893 of 1101 vs 917 of 1101, 2.2% improvement, 95% CI −1.1% to 5.4%, Supplemental Table 10). By using the extended model, SBI was correctly ruled out in 31 additional children (3.7%, 95% CI −1.0% to 8.4%), and there were 5 fewer potentially missed SBI diagnoses (14 of 264 vs 19 of 264, 1.8% reduction, 95% CI −2.6% to 6.4%, Table 3).
In this large, prospective study of febrile children of all ages who presented to the ED, multinomial risk-prediction models discriminated well between pneumonia, other SBIs, and no SBIs. A newly derived model performed well on internal validation and identified PCT, resistin, and CRP as biomarkers of potential value. A published model performed well on external validation, and the addition of PCT and resistin improved discrimination. At a low-risk threshold (<5%), an NLR of 0.12 (pneumonia) or 0.21 (other SBIs) may help to rule out SBIs, whereas at a high-risk threshold (>20%), PLRs of 6.69 and 4.96, respectively, may expedite treatment.
We present data on multiple biomarkers of SBIs in >1000 children. We have evaluated children irrespective of age, past medical history, and clinical syndrome and obtained comparable discrimination to other studies with more restrictive inclusion criteria. In common with other recent data,2,16 we have demonstrated the value of combining clinical and biomarker variables.
This is the first broad, external validation of the published multivariable model by Nijman et al.16 The model discriminated well but was poorly calibrated. Specifically, there was a problem with calibration overall: the model predicted too few cases in our population. However, correlation between model predictions and observed cases suggested the overall structure of the model was appropriate to our data set, and our approach of re-estimating the model coefficients resulted in a well-calibrated model.
This is a single-center study, and although we have performed internal validation of our derived model, external validity would require demonstration in an alternative setting. We have grouped other SBI into a single outcome category. It would be preferable to model outcomes such as septicemia and meningitis separately, but the infrequency of these outcomes makes this challenging. A pragmatic response is to advocate additional diagnostic testing (including urgent urine or CSF microscopy) of children who are considered at high risk for other SBIs.
Diagnostic studies with imperfect reference standards require a pragmatic approach to determine outcomes. An established approach to this is to use predefined, composite reference standards, as we have done. The universal application of respiratory viral assays may have yielded additional evidence on which to base classification, but such testing was undertaken at the discretion of the clinical team and not applied systematically. Our use of a radiologic diagnosis of pneumonia despite its limitations is common in this setting.15,34 We included a category of probable SBI to account for the lack of sensitivity of conventional diagnostic testing in children. This category accounted for only a small number of cases (8) and was defined in advance. By establishing clear criteria for each outcome diagnosis, we have sought to minimize verification bias.
We studied children who were already considered at risk for SBIs and in whom the clinical team had initiated additional investigation. This unmeasured risk evaluation limits the external validity of our findings. The proportion of SBIs (24%) is significantly higher than that which is observed in all febrile children in the ED, and we agree with other authors who have stressed the importance of diagnostics research in low-risk populations (such as all children attending the ED or primary care).18 Of our sample, ∼80% were admitted to the hospital and received antibiotics, including 60% of those who did not have SBIs. Decision-making on the basis of a low-risk threshold of 5% may reduce admissions and antibiotic use, but it does not (by definition) eliminate risk. Clinicians would need to combine risk evaluation with appropriate safety-netting.
Comparison With Published Studies
Our finding that clinical variables such as hypoxia, abnormal respiratory findings, irritability, and dehydration increase the probability of an SBI is consistent with similar studies.2,13,16 We failed to demonstrate the value of more subjective assessments such as ill appearance and parental concern, although there was a significant problem of missing data for each.
Our results support a growing body of research suggesting that risk prediction models improve the identification of SBIs in the children’s ED. Such models have yet to translate into improved clinical decision-making. Two recent impact studies challenged the assumption that accurate risk prediction will necessarily improve decision-making. In the first, the use of the laboratory score (a decision rule that combines CRP, PCT, and urinalysis) failed to reduce antibiotic prescriptions in children in the ED.35 A second evaluated the use of the Nijman risk prediction model to guide decisions, and no impact on antibiotic prescribing or hospital admission was observed.36
Future impact studies need to evaluate the behaviors associated with decision-making. This has been of considerable importance in evaluating interventions to rationalize antibiotic prescribing.37 To translate estimates of risk into safe clinical decisions and improve the management of children in the ED, it will be necessary to involve clinicians and families. The risk thresholds we have proposed are not yet established in the context of SBIs in the children’s ED, and more work is necessary to determine if they (and the clinical decisions they guide) are appropriate.
A diagnostic model that combined clinical and biomarker variables discriminated well between pneumonia, other SBIs, and no SBI in febrile children of all ages in the ED. External validation of a previously derived risk model yielded encouraging diagnostic accuracy and was improved by the addition of PCT and resistin. Future work should establish the value of decision rules based on risk prediction models in robust impact studies. Such studies must address the complex behaviors that are associated with clinical decisions to yield clinical benefit.
We thank all the children and their parents for agreeing to participate, and we thank the significant number of ED and pediatric clinical and nursing staff who work in the Alder Hey Children’s ED for their substantial contributions to the study. In particular, we thank the children of the Medicines for Children Research Network Young Persons’ Advisory Group. We thank Prof Matthew Peak and Ms Dot Lambert for their logistical support from the Alder Hey Research Department. We thank Ms Elaine Hanmer, Ms Laura Medway, and Ms Gemma Boydell for their informatics support and Mrs Sarah Olsen for her administrative support in the running of the study.
The University of Liverpool acknowledges the support of the National Institute for Health Research through the Comprehensive Clinical Research Network.
- Accepted May 3, 2017.
- Address correspondence to Adam D. Irwin, PhD, MRCPCH, Department of Infectious Disease, Great Ormond Street Hospital for Children, London WC1N 3JH, United Kingdom. E-mail:
FINANCIAL DISCLOSURE: Prof Carrol serves on a childhood systemic infections panel for BioFire Ltd; the other authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by the National Institute for Health Research’s Research for Innovation, Speculation, and Creativity Programme grant RC-PG-0309-10053 to Prof Carrol and the Alder Hey Charity funding grant 8037 to Dr Irwin and Prof Carrol. The funding organizations had no role in the design or undertaking of this study. This article presents independent research that is funded by the National Institute for Health Research’s Research for Innovation, Speculation, and Creativity Programme. The views expressed are those of the author(s) and are not necessarily those of the NHS Foundation Trust, the National Institute for Health Research, or the Department of Health.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Sands R,
- Shanmugavadivel D,
- Stephenson T,
- Wood D
- Craig JC,
- Williams GJ,
- Jones M, et al
- Inwald DP,
- Tasker RC,
- Peters MJ,
- Nadel S; Paediatric Intensive Care Society Study Group (PICS-SG)
- Koshy E,
- Murray J,
- Bottle A,
- Sharland M,
- Saxena S
- Whitney CG,
- Farley MM,
- Hadler J, et al; Active Bacterial Core Surveillance of the Emerging Infections Program Network
- Gill PJ,
- Goldacre MJ,
- Mant D, et al
- Irwin AD,
- Wickenden J,
- Le Doare K,
- Ladhani S,
- Sharland M
- Van den Bruel A,
- Haj-Hassan T,
- Thompson M,
- Buntinx F,
- Mant D; European Research Network on Recognising Serious Infection investigators
- Van den Bruel A,
- Thompson MJ,
- Haj-Hassan T, et al
- Brent AJ,
- Lakhanpaul M,
- Thompson M, et al
- Nijman RG,
- Vergouwe Y,
- Thompson M, et al
- UK Standards for Microbiology Investigations
- Corless CE,
- Guiver M,
- Borrow R,
- Edwards-Jones V,
- Fox AJ,
- Kaczmarski EB
- Gray SJ,
- Trotter CL,
- Ramsay ME, et al; Meningococcal Reference Unit
- Galetto-Lacour A,
- Zamora SA,
- Andreola B, et al
- R Core Team
- Stevenson M
- Nijman RG,
- Zwinkels RL,
- van Veen M, et al
- Little P,
- Stuart B,
- Francis N, et al; GRACE consortium
- Copyright © 2017 by the American Academy of Pediatrics