Predicting Severe Pneumonia Outcomes in Children
- Derek J. Williams, MD, MPHa,
- Yuwei Zhu, MD, MSb,
- Carlos G. Grijalva, MD, MPHc,
- Wesley H. Self, MD, MPHd,
- Frank E. Harrell Jr, PhDb,
- Carrie Reed, DSce,
- Chris Stockmann, PhD, MScf,
- Sandra R. Arnold, MD, MScg,
- Krow K. Ampofo, MDf,
- Evan J. Anderson, MDh,
- Anna M. Bramley, MPHe,
- Richard G. Wunderink, MDi,
- Jonathan A. McCullers, MDg,
- Andrew T. Pavia, MDf,
- Seema Jain, MDe, and
- Kathryn M. Edwards, MDa
- aDepartments of Pediatrics,
- cHealth Policy, and
- dEmergency Medicine, Vanderbilt University Medical Center, Nashville, Tennessee;
- eCenters for Disease Control and Prevention, Atlanta, Georgia;
- fDepartment of Pediatrics, University of Utah Health Sciences Center, Salt Lake City, Utah;
- gDepartment of Pediatrics, University of Tennessee Health Science Center, Memphis, Tennessee;
- hDepartments of Pediatrics and Medicine, Emory University School of Medicine, Atlanta, Georgia; and
- iDepartment of Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois
Dr Williams conceptualized and designed the study, carried out the initial analyses, and drafted and revised the initial manuscript; Drs Zhu and Harrell conceptualized and designed the study, oversaw and carried out analyses, and reviewed and revised the initial manuscript; Drs Grijalva, Self, Reed, Stockmann, Arnold, Ampofo, Anderson, Bramley, Wunderink, McCullers, Pavia, Jain, and Edwards conceptualized and designed the study, assisted with interpretation of the data, and reviewed and revised the initial manuscript; and all authors approved the final manuscript as submitted.
BACKGROUND: Substantial morbidity and excessive care variation are seen with pediatric pneumonia. Accurate risk-stratification tools to guide clinical decision-making are needed.
METHODS: We developed risk models to predict severe pneumonia outcomes in children (<18 years) by using data from the Etiology of Pneumonia in the Community Study, a prospective study of community-acquired pneumonia hospitalizations conducted in 3 US cities from January 2010 to June 2012. In-hospital outcomes were organized into an ordinal severity scale encompassing severe (mechanical ventilation, shock, or death), moderate (intensive care admission only), and mild (non–intensive care hospitalization) outcomes. Twenty predictors, including patient, laboratory, and radiographic characteristics at presentation, were evaluated in 3 models: a full model included all 20 predictors, a reduced model included 10 predictors based on expert consensus, and an electronic health record (EHR) model included 9 predictors typically available as structured data within comprehensive EHRs. Ordinal regression was used for model development. Predictive accuracy was estimated by using discrimination (concordance index).
RESULTS: Among the 2319 included children, 21% had a moderate or severe outcome (14% moderate, 7% severe). Each of the models accurately identified risk for moderate or severe pneumonia (concordance index across models 0.78–0.81). Age, vital signs, chest indrawing, and radiologic infiltrate pattern were the strongest predictors of severity. The reduced and EHR models retained most of the strongest predictors and performed as well as the full model.
CONCLUSIONS: We created 3 risk models that accurately estimate risk for severe pneumonia in children. Their use holds the potential to improve care and outcomes.
- aOR —
- adjusted odds ratio
- c-index —
- concordance index
- ED —
- emergency department
- EHR —
- Electronic Health Record
- EPIC Study —
- Etiology of Pneumonia in the Community Study
- IQR —
- interquartile range
- PF ratio —
- PaO2/FiO2 ratio
What’s Known on This Subject:
Pneumonia is the most common serious infection in childhood. Accurately identifying risk for severe outcomes is critical to optimizing care and outcomes; however, risk-stratification tools to guide disposition and management decisions in children are lacking.
What This Study Adds:
We created 3 risk models that accurately estimate risk for severe pneumonia in US children. The application of risk-stratification tools using these models holds the potential to improve clinical care and outcomes.
Pneumonia is the most common serious infection in children, accounting for 1% to 4% of all pediatric emergency department (ED) visits in the United States.1 Although most children are treated as outpatients, 20% to 25% of pneumonia ED visits result in hospitalization.1,2 Consequently, pneumonia ranks among the top 3 reasons for pediatric hospitalization.3,4 However, the proportion of children presenting to the ED with pneumonia who are hospitalized varies widely,5,6 ranging from 19% to 69% among 35 US children’s hospitals.6 This variation is largely independent of population differences or illness severity,5–7 suggesting a need for standardized methods to improve identification of children at risk for severe outcomes.
Predictive analytics uses statistical modeling to generate reliable and objective risk estimates that facilitate individual patient management decisions. The high incidence, morbidity, and substantial variation in care make pediatric pneumonia an excellent target for such studies. Several prognostic models are available for adults with pneumonia,8–10 and studies of these models indicate that they may reduce broad-spectrum antibiotic use and decrease hospitalization among low-risk individuals.11–14 However, no validated models exist to predict clinical outcomes among children with pneumonia in the developed world, a key research gap highlighted in national childhood pneumonia management guidelines.15 Using data from the Centers for Disease Control and Prevention Etiology of Pneumonia in the Community (EPIC) Study,16 we developed 3 prognostic models to accurately estimate risk of severe pneumonia outcomes in children.
The EPIC study was a prospective, population-based surveillance study of community-acquired pneumonia hospitalizations among children <18 years of age conducted between January 1, 2010, and June 30, 2012, at 3 children’s hospitals in Memphis, TN, Nashville, TN, and Salt Lake City, UT.16 Children were enrolled if they were hospitalized with signs or symptoms of acute infection (eg, fever) and acute respiratory illness (eg, cough), and radiographic evidence of pneumonia. Standardized radiographic interpretation was completed by a board-certified pediatric study radiologist at each study hospital. Children with recent hospitalization, severe immunosuppression, cystic fibrosis, tracheostomy, or a clear alternative diagnosis were excluded. The study was approved by the institutional review board at each participating institution and the Centers for Disease Control and Prevention. Additional details regarding the EPIC study population were published previously,16 and also are presented in the supplementary materials.
Our prognostic models focused on acute in-hospital outcomes, organized into a single ordinal severity scale with 3 levels: severe, moderate, and mild. Severe included children who died, required invasive mechanical ventilation, or developed shock requiring vasoactive medications (eg, dopamine, norepinephrine, vasopressin). Moderate included children admitted to the ICU who did not meet criteria for severe. The remaining children were classified as mild. Children were classified according to the most severe outcome that occurred during the hospitalization.
Predictors were identified a priori and informed by clinical expertise and literature review. We focused on the selection of objective variables with strong hypothesized associations with the outcome that also could be rapidly ascertained in the clinical setting. This resulted in a pool of 20 candidate predictors, encompassing demographics and comorbidities, as well as clinical, laboratory, and radiology variables (Table 1). Comorbidities were evaluated as a single composite variable (0, 1, 2, or 3+). Oxygenation was expressed as the PaO2/FiO2 (PF) ratio, derived from the SpO2/FiO2 ratio.17 All predictors were assessed at the time of admission.
We evaluated 3 primary models. The first model included 20 predictor variables (full model). Next, we created a reduced model limited to the most clinically important predictors as judged by expert consensus. For this process, we surveyed 14 attending physicians from 10 US institutions with expertise in childhood pneumonia (including 5 EPIC study investigators), representing critical care, emergency medicine, hospital medicine, and infectious diseases. Respondents assessed the importance of each of the predictor variables on a 5-point Likert scale (1 = not important to 5 = very important). Variables with an average ranking of important or very important (median score of ≥4) were retained. This resulted in a reduced set of 10 predictors (reduced model), including age, comorbidities, chest indrawing, altered mental status, heart and respiratory rates, systolic blood pressure, PF ratio, infiltrate pattern, and pleural effusion. The third model included 9 predictors that are routinely available within comprehensive electronic health records (EHR model): age, sex, race, temperature, heart and respiratory rates, systolic blood pressure, PF ratio, and white blood cell count. A term for enrollment site was also included in each model.
Contribution of Microbiologic Etiology
To assess the contribution of microbiologic data to prognostic performance, we also evaluated each of the primary models by using etiology data from the EPIC study. Etiologic assessments included blood for bacterial culture, serology for 8 respiratory viruses, and pneumococcal and group A streptococcal polymerase chain reaction; and naso/oropharyngeal swabs for polymerase chain reaction for 13 respiratory viruses, Mycoplasma pneumoniae, and Chlamydophila pneumoniae.16 For this analysis, microbiologic data were summarized by using binary categorical variables for viruses, atypical bacteria (M pneumoniae or C pneumoniae), and other bacteria.
Our severity prediction models were created by using ordinal logistic regression.18,19 To allow for nonlinear associations, continuous predictors were modeled by using restricted cubic spline transformations. To account for known age-based differences in normal heart rate, respiratory rate, and blood pressure, interaction terms between each of these variables and age were included. The effect of individual predictors on severity outcomes were reported by using adjusted odds ratios (aORs). Partial effects plots for continuous predictors also were generated. To illustrate the importance of each predictor, we calculated, and graphically displayed, the fraction of explainable outcome variation contributed by each predictor based on their partial χ2 values.18
Between-model comparisons included assessments of fit (likelihood ratio χ2 statistic), quality (Akaike Information Criterion), and predictive accuracy (discrimination and calibration).18 We assessed the ability of our models to differentiate between children with and without each outcome (discrimination) using the concordance index (c-index), which is analogous to the commonly reported area under the receiver operating characteristic curve. A c-index of 1.0 indicates perfect concordance. Agreement between the predicted and observed outcome frequencies (calibration) was assessed graphically. First, predicted risk for moderate or severe pneumonia was transformed into 100 bins of equal size ranging from 0 to 1. Lowess fit curves were then generated corresponding to the observed proportion of children who experienced a severe, moderate, or mild outcome within each bin of predicted risk. Finally, we explored the potential performance of our models in future populations by using an internal bootstrap validation (500 replications with replacement).
We conducted secondary analyses to assess key assumptions. First, we examined whether our estimated risk for moderate or severe outcomes correlated with hospital length of stay, a commonly used surrogate for disease severity, by using Spearman’s rank correlation. Second, because our predictive models may perform less well in children without evident severe disease at presentation, models were reassessed after excluding children triaged to the ICU on presentation. Analyses were conducted by using Stata 13.1 (StataCorp, College Station, TX) and R 3.2.3 (R Foundation for Statistical Computing, Vienna, Austria). Additional methodologic details are provided in the supplementary materials.
Of the 2358 children enrolled in the EPIC study with radiographically confirmed pneumonia, 39 were excluded due to ≥1 immunocompromising comorbidity (Supplemental Fig 3). Median age of the 2319 included children was 28 months (interquartile range [IQR], 12–71); 55% were boys; 40% were non-Hispanic white, 33% were non-Hispanic black, and 19% were Hispanic; 49% of children had ≥1 comorbidity (Table 1). Seven percent of children had a severe outcome (range across sites 6%–10%); 14% had a moderate outcome (range across sites 9%–16%). One-third of children were hospitalized for <48 hours and almost 10% for <24 hours. Not all children had complete data for all considered predictors (Supplemental Fig 3); peripheral white blood cell count, in particular, was missing in 17% of children (n = 397); for all other variables, data were missing in <2% of observations.
Predictors of Pneumonia Severity
The aORs for each predictor are presented in Table 2; partial effects plots for continuous predictors are shown in Supplemental Fig 4. The presence of altered mental status, chest indrawing, and multilobar or nonlobar (eg, interstitial) infiltrates each predicted a more severe outcome (eg, in the full model, chest indrawing increased the odds of moderate or severe pneumonia 1.77 times). Similarly, among the continuous predictors, decreasing PF ratio, systolic blood pressure, and temperature; extremes of age; and increasing heart and respiratory rates were all associated with more severe pneumonia outcomes (eg, in the full model, the odds of a moderate or severe outcome was increased 1.22 times for a 1-year-old child compared with a 2-year-old). Combined, these 9 variables ranked as the strongest predictors of pneumonia severity, contributing 96% of the explainable outcome variation in the full model (Fig 1). The reduced model retained all but 1 of these predictors (temperature), whereas the EHR model retained 6 (omitting altered mental status, chest indrawing, and infiltrate pattern) (Table 2).
Prognostic Model Performance
Each of the models accurately identified risk for moderate or severe pneumonia (c-index range across models 0.78–0.81, Table 3). Differences in model fit and quality were considered negligible, and results from the internal bootstrap validation were comparable to the primary models (c-index range across models 0.76–0.79). Given similar model performance, the simplified EHR model was selected for illustrative purposes. For this model, the median predicted risk for moderate or severe pneumonia was 18% (IQR 9%–32%) and for severe pneumonia 5% (3%–11%). As demonstrated in Fig 2, among those with a 10% predicted risk of a moderate or severe outcome, 12% actually experienced these outcomes, indicating good agreement between observed and expected outcomes.
In secondary analyses, a positive correlation was noted between hospital length of stay and the predicted risk for moderate or severe pneumonia estimated from the EHR model (ρ = 0.33, P < .01). Twenty-four percent of children with hospital length of stay <24 hours had a <5% predicted risk of moderate or severe pneumonia, compared with 12% of those hospitalized for 24 to 47 hours and 7% of those hospitalized for ≥48 hours (P < .001). Results for the full and reduced models were similar (data not shown). Further, after excluding those initially triaged to the ICU (n = 336), the discriminatory power of the 3 models (c-index range across models 0.76–0.79) was similar to the primary models. Importantly, those with delayed ICU transfer had significantly higher median predicted risk for moderate or severe (25% in EHR model) and severe pneumonia (8%) compared with those not admitted to the ICU (14%, 4%; P < .001 for both).
Contribution of Microbiologic Etiology
Viruses were detected in 72% of children and atypical bacteria in 9%; detection of these pathogens was less common in children with moderate or severe pneumonia (viruses detected in 21% of children with moderate or severe pneumonia [aOR 0.74 in EHR model] and atypical bacteria detected in 11% [aOR 0.35], Supplemental Table 4). Typical bacteria were detected in 6% of children, including Streptococcus pneumoniae in 4%, and both Staphylococcus aureus and Streptococcus pyogenes in 1%.16 Detection of these bacterial pathogens was strongly associated with pneumonia severity (aOR 3.76); 30% of children with S pneumoniae detected had moderate or severe pneumonia, whereas 83% of those with S aureus and 75% with S pyogenes had moderate or severe pneumonia. Nonetheless, the addition of etiology data had a negligible impact on each of the 3 primary models’ discriminatory power (c-index range across models 0.79–0.83, Supplemental Table 5).
Using prospective data from >2300 children hospitalized with pneumonia, we developed and internally validated 3 novel prognostic models to estimate risk for severe in-hospital outcomes. Each model accurately identified risk for moderate (ICU admission) and severe (invasive mechanical ventilation, shock, or death) outcomes. Risk-stratification tools based on these models could be used in emergency care settings to help identify children at low risk for severe outcomes who could be safely discharged, as well as those who may benefit from more intensive management. A simple EHR-based model that included 9 predictors demonstrated similar predictive accuracy to the 2 more-complex models and offers the greatest practical utility. Although predictive models must be externally validated, the use of risk-stratification tools based on these models holds the potential to standardize care and improve outcomes for children with pneumonia.
Variation in disposition decisions is evident among children with pneumonia,5–7 with potential for avoidable harm related to unnecessary hospitalizations as well as delayed triage to higher levels of care. To inform care decisions and optimize outcomes, the 2011 Pediatric Infectious Diseases Society/Infectious Diseases Society of America childhood pneumonia management guidelines emphasized the need for objective risk-stratification tools.15 Reed et al19 previously developed a simple tool to predict in-hospital pneumonia mortality in young South African children, although its utility in settings in which pediatric pneumonia mortality is rare has not been evaluated. Our prognostic models focused on in-hospital outcomes and evaluated an expanded set of predictors routinely collected for clinical care. Each of our models demonstrated potential to improve risk stratification for US children with pneumonia.
Given similar predictive accuracy of the 3 primary models, we anticipate the EHR model could be most readily adopted into clinical settings. Each of the predictors included in this model is routinely available within comprehensive EHRs and easily queried in real time for children presenting for emergency care. Most of these predictors represent physiologic parameters and many are included in existing severity scores for children admitted to intensive care and adult pneumonia severity scores.8,10,22–24 The full model also included several weak or nonsignificant predictors. Comorbidities failed to demonstrate strong associations with pneumonia severity, although we postulate this is due to the wide heterogeneity of chronic comorbidities in children. Of note, functional status and medical complexity were not assessed, a limitation that requires further study.
In our models, peripheral white blood cell count was not associated with severe outcomes. These data were missing for nearly 20% of children. Our observations are consistent with national guidelines that indicate that routine measurement of complete blood counts may be unnecessary except in those with critical illness (eg, sepsis).15 Interestingly, the inclusion of etiologic data also had a negligible effect on our models. However, the detection of viruses and atypical bacteria was associated with a reduced odds of severe outcomes, whereas detection of typical bacteria was associated with an increased odds of severe outcomes. Results of bacterial studies are not available at the initial triage assessment and their sensitivity remains questionable, limiting the utility of such testing to inform initial management decisions. It will be important to reevaluate our findings as new strategies to rapidly diagnose bacteria emerge.
A third of the children in our study were hospitalized for <48 hours and nearly 10% for <24 hours.16 A quarter of children with length of stay <24 hours had a <5% estimated risk of moderate or severe outcome. Hospitalization may be unnecessary in some of these children. In others, factors unrelated to pneumonia severity may influence the hospitalization decision. However, studies in adults with pneumonia indicate that site of care decisions vary considerably by provider and that risk for severe outcomes is often overestimated.25–27 Appropriate use of prognostic models in adults is associated with reduced hospitalizations for low-risk patients and improved guideline-concordant management.12,13,28–30 Thus, although prognostic models do not replace clinician judgment, these studies demonstrate that accurate risk stratification can improve decision-making. Risk-stratification tools based on our models could be used in emergency care settings to help identify children at low risk for severe outcomes and potentially facilitate rapid triage and safe discharge with minimal testing or intervention.15 Conversely, the rapid identification of children at high risk for deterioration could signal the need for closer monitoring and/or more intensive therapy. In either scenario, decision-making informed by an accurate understanding of risk is likely to lead to better care and outcomes.
An important challenge for prognostic models is clinical implementation. Despite the availability of risk-stratification tools with documented clinical utility for adults with pneumonia,11–14 studies indicate these tools are infrequently used or sometimes used inappropriately.26,31–33 Complex score calculations can deter adoption, but the availability of Web-based or mobile device applications may improve accessibility. Clinical decision support using prognostic models embedded within the EHR offer distinct advantages over other electronic applications, chief among these is computational efficiency and seamless workflow integration.34 With EHR-based tools, risk predictions can be obtained in real time at the point of care without requiring providers to input additional data. Risk estimates also can be linked to tailored care recommendations and order sets to ensure that each patient receives the most appropriate level of care.11,35 Although several published studies demonstrate the usefulness of well-designed EHR-based decision support applications,11,35,36 additional study is needed.
As with all prognostic studies, our models may perform less well among other populations; however, this study was nested within a very large, prospective, population-based cohort of children hospitalized with clinical and radiographic pneumonia in 3 US cities. Importantly, although the populations served by these 3 hospitals may not be representative of the US population, characteristics of the study population, including age, sex, race/ethnicity, and frequency of comorbidities, were similar to a previous study of pneumonia hospitalizations conducted within a national sample of 43 US children’s hospitals.37 Although results from our internal validation are very encouraging, external validation in expanded populations is essential before widespread adoption. In particular, inclusion of children discharged from the ED without hospitalization will be important. Prospective validation also allows for model updating and the evaluation of novel predictors. New approaches to predict pneumonia etiology and outcomes, including point-of-care testing by using biomarkers (eg, procalcitonin),38 may further our ability to precisely estimate risk for severe outcomes. Continued evaluations are needed to rigorously evaluate implementation, as well as the clinical utility, of these promising models.
We developed 3 prognostic models that accurately identified risk for severe outcomes in children hospitalized with pneumonia. A simplified model using inputs derived exclusively from data available in the EHR performed similarly to more-complex models. Use of this simpler model to implement risk-stratification tools within comprehensive EHR systems could lead to safer and more effective care that is tailored to each patient’s risk for severe pneumonia outcomes.
- Accepted July 20, 2016.
- Address correspondence to Derek J. Williams, MD, MPH, CCC-5324 Medical Center North,Vanderbilt University Medical Center, 1161 21st Ave S., Nashville, TN 37212. E-mail:
FINANCIAL DISCLOSURE: Dr Self has received payment for consulting and advisory board services from Abbott Point of Care, BioFire Diagnostics, and Venaxis. Dr Anderson has received payment for consulting from AbbVie and research funding from MedImmune/AstraZeneca. Dr Ampofo has received research funding from GlaxoSmithKline, Cubist Pharmaceuticals, and Janssen Pharmaceuticals. The other authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by the National Institutes of Health under award number K23AI104779 to Dr Williams and award number K23GM110469 to Dr Self; by the Agency for Healthcare Research and Quality under award number 1R03HS022342 to Dr Grijalva; and by CTSA award numbers UL1TR000445 and KL2TR000446 from the National Center for Advancing Translational Sciences. The Etiology of Pneumonia in the Community Study was supported by the Influenza Division in the National Center for Immunizations and Respiratory Diseases at the Centers for Disease Control and Prevention through cooperative agreements with each study site and was based on a competitive research funding opportunity. The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the National Institutes of Health, the Agency for Healthcare Research and Quality, or the Centers for Disease Control and Prevention. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Copyright © 2016 by the American Academy of Pediatrics