Derivation of Candidate Clinical Decision Rules to Identify Infants at Risk for Central Apnea
BACKGROUND AND OBJECTIVES: Central apnea complicates, and may be the presenting complaint in, bronchiolitis. Our objective was to prospectively derive candidate clinical decision rules (CDRs) to identify infants in the emergency department (ED) who are at risk for central apnea.
METHODS: We conducted a prospective observational study over 8 years. The primary outcome was central apnea subsequent to the initial ED visit. Infants were enrolled if they presented with central apnea or bronchiolitis. We excluded infants with obstructive apnea, neonatal jaundice, trauma, or suspected sepsis. We developed 3 candidate CDRs by using 3 techniques: (1) Poisson regression clustered on the individual, (2) classification and regression tree analysis (CART), and (3) a random forest (RF).
RESULTS: We analyzed 990 ED visits for 892 infants. Central apnea subsequently occurred in the hospital in 41 (5%) patients. Parental report of apnea, previous history of apnea, congenital heart disease, birth weight ≤2.5 kg, lower weight, and age ≤6 weeks all identified a group at high risk for subsequent central apnea. All CDRs and RFs were 100% sensitive (95% confidence interval [CI] 91%–100%) and had a negative predictive value of 100% (95% CI 99%–100%) for the subsequent apnea. Specificity ranged from 61% to 65% (95% CI 58%–68%) for CDRs based on Poisson models; 65% to 77% (95% CI 62%–90%) for CART; and 81% to 91% (95% CI 78%–92%) for RF models.
CONCLUSIONS: All candidate CDRs had a negative predictive value of 100% for subsequent central apnea.
- ALTE —
- apparent life threatening event
- CART —
- classification and regression tree
- CDR —
- clinical decision rule
- CI —
- confidence interval
- ED —
- emergency department
- NPV —
- negative predictive value
- PGE2 —
- prostaglandin E2
- RF —
- random forest
- RSV —
- respiratory syncytial virus
- URI —
- upper respiratory tract infection
What’s Known on This Subject:
Central apnea sometimes complicates bronchiolitis. Because apnea tends to occur early in the course of bronchiolitis, there is a danger that infants may be discharged from the emergency department only to subsequently develop apnea at home.
What This Study Adds:
This study prospectively derived clinical decision rules to help emergency physicians admit infants at risk for apnea while discharging those not at risk.
Parents of infants brought in dead frequently describe previous mild respiratory tract infection symptoms; postmortem frequently reveals respiratory syncytial virus (RSV), other respiratory viruses, or bronchiolitis.1–3 The mildness of antecedent respiratory symptoms suggests a central apneic terminal event rather than respiratory exhaustion from increased work of breathing.4,5 Infants who present to the emergency department (ED) with witnessed central apnea (as distinct from apparent life-threatening events [ALTE]) have an increased risk of subsequent unexpected death.6,7 Infants who present with central apnea also frequently have RSV, other respiratory viruses, or bronchiolitis.8 Apparently isolated apnea is followed by bronchiolitis 33% of the time; this percentage rises if even early signs of upper respiratory tract infection (URI) are present.9 Apnea complicates bronchiolitis 5% of the time, but emerges after the clinical appearance of bronchiolitis in just 1%.8–10
The respiratory pauses commonly seen in small infants are terminated by hypercarbic or hypoxic autoresuscitation. Failure of this autoresuscitation leads to central apnea. Increasing evidence suggests that viral respiratory tract infections trigger this central apnea by increasing prostaglandin E2 (PGE2) binding of EP3 receptors in the medulla.10–13 In immature or otherwise vulnerable infants PGE2 binding of EP3 receptors appears to induce failure of autoresuscitation; the mature response is fever.10,14–17 PGE2 levels rise within 24 hours of RSV infection.18,19 Although PGE2 can rise and central apnea occur in response to any infection, viral respiratory infections dominate and the seasonality of central apnea mirrors that of RSV bronchiolitis.20,21
The relevance of this pathophysiological model to clinicians is first, central apnea can be expected to occur early in the course of illness when other signs are mild and the infant would be discharged from the ED, and second, the usual clinical sign of elevated PGE2 (fever) cannot be relied on.10,16,22,23 PGE2 levels cannot be measured in the ED so emergency physicians must find other ways to avoid discharging infants who will subsequently have central apnea at home.
Although risk factors for central apnea in infants have been described (previous apnea, bronchiolitis in younger, premature, or comorbid infants), integrating these into a binary admit/discharge decision is challenging.8,16,24–26A clinical decision rule (CDR) would be useful to clinicians and researchers. Our objective was to prospectively derive candidate CDRs to identify infants with central apnea and bronchiolitis at ED presentation who are at high risk for subsequent central apnea.
We conducted a prospective observational study in a county hospital ED (mean census 42 000 with ∼250 children ≤18 months old with bronchiolitis annually) between November 2005 and May 2012. Although we recruited year around, few were enrolled outside bronchiolitis season. Kern Medical Center institutional review board approved this study.
Selection of Participants
Potential participants were consecutively identified by reviewing their presenting complaint and walking the waiting room. Written informed consent was obtained by research personnel or clinician investigators.
We defined central apnea per the American Academy of Pediatrics guidelines with a 15 seconds threshold for apnea in the absence of bradycardia or cyanosis. Children meeting the definition of ALTE who did not also meet the definition for central apnea were not included.27 We defined bronchiolitis as clinical evidence of lower airway obstruction (eg, wheezing, rhonchi, diffuse crackles, chest wall retractions) following URI symptoms.28
Infants were included if they presented with bronchiolitis with or without a history of central apnea, or with central apnea alone. Apnea as a presenting complaint relied on history and is termed “parent-reported apnea.” Isolated pneumonia was not an inclusion criterion.
Infants were excluded if the apnea was obstructive, parents declined consent, or the parent or infant was under the care of law enforcement. We excluded infants being managed for sepsis, trauma, or neonatal jaundice but not those with mild incidental jaundice. During analysis, we excluded infants older than 6 months. We had enrolled patients up to 12 months of age, as this defines infancy, but central apnea does not seem to occur in infants older than 6 months. Including such infants could not add useful information to the analysis and would artificially inflate the negative predictive values (NPVs) of resulting CDRs. Excluding nonevents in rare events modeling improves predictive performance.29–31
Data Collection Procedures
Birth history, prematurity, previous apneic episodes, weight and physical findings, and disposition were recorded by the treating physician on a study template that was embedded in the medical record. These data were transferred to a customized database (Filemaker, Inc, Santa Clara, CA) by trained research assistants. All data entry was checked by a second research assistant or investigator. Differences were resolved by consensus. We used the Centers for Disease Control and Prevention 2000 growth charts to calculate weight-for-age centile.32
All staff who screened patients were trained in study procedures. We repeated training each bronchiolitis season and provided ongoing reinforcement and education.
Outcomes and Follow-up
Our study outcome was defined as either central apnea witnessed by clinical staff in the hospital, or a confirmed apneic event on repeat presentation to the ED if the infant was discharged. The distinction between central and obstructive episodes was based on the observations of treating clinicians. Our outcome was determined by explicit chart review after admission. For patients discharged from the ED, we performed telephone follow-up. When telephone follow-up was unsuccessful, we reviewed the county coroner’s records.
Data Analysis and Rule Derivation
We derived decision rules by using 3 different techniques:
(1) Regression models in which any 1 of several criteria classifies the infant as at risk for apnea,
(2) Classification and regression tree (CART) analysis, which creates a decision tree to be followed algorithmically to determine if the infant is at risk for apnea,
(3) Random forest (RF) modeling. A random forest is an ensemble of many different decision trees derived in CART analyses. The classification of at risk is based on majority voting of the decision trees in the forest.
We included variables if they were clinically important, had an area under the receiver operating curve of >0.65, or a P < .2 in bivariate analysis. Based on their previously described interrater reliability, retraction severity, nasal flaring, increased work of breathing, and dehydration were included and auscultatory findings excluded.33 The variables selected were included in robust Poisson regression with a cluster term to adjust for multiple visits.34 These models were compared with zero-inflated Poisson and negative binomial regression. We used Stata 13.1 (Stata Corp, College Station, TX) for regression.
We built candidate decision trees by using CARTs using the J48 algorithm as implemented in WEKA data mining software (University of Waikato, Hamilton, New Zealand) and by using the ctree program in R statistical software (R Foundation for Statistical Computing, Vienna, Austria).35,36 We weighted apnea in the hospital from 20:1 to 50:1 because of its importance. We retained those trees with 100% sensitivity of 10-fold cross validation. We manually extended decision boundaries to improve usability and counter overfitting, by for example, rounding age or weight to 1 decimal place.
Our RF is an ensemble of 1000 different CART decision trees that “vote” on whether an infant will develop apnea. Unlike a single CART, RFs are not amenable to graphical depiction, nor can one derive a rule-based explanation of its classification.37 RFs are considered “black box methods” that produce accurate but opaque predictions.
We performed 10-fold cross validation on the RFs. We weighted “apnea in the hospital” 2:1; higher weightings did not improve accuracy. RFs were constructed using the cforest program in R statistical software.36
We compared the characteristics of those who presented with bronchiolitis alone, apnea alone, bronchiolitis and apnea, and apnea with URI symptoms, who subsequently developed apnea while in the hospital. Apart from encoding the importance of parent-reported apnea, these groups should be broadly similar if our sampling methodology was valid.
We calculated sensitivity, specificity, positive predictive values, and NPVs for each candidate CDR using the diagt command in Stata.38 The CDRs recommend hospital admission for any infant at risk for apnea; therefore, we also calculated the ratio of patients admitted to those who developed apnea.
We analyzed 990 visits in 892 patients. The median age was 2.26 months (interquartile range 1.32–4.00) and 579 (58%) were boys. Most (528, 53%) were hospitalized locally, 30 (3%) required intubation and were transferred to a PICU, and 432 (44%) were discharged from the ED. Follow-up by telephone or documentation of a follow-up visit was completed in 370 (85%) of 433 discharged patients. One died during his hospital admission. One other death (sudden infant death syndrome) was discovered during coroner record review on a 5-week-old 3.9-kg patient who died after his postdischarge follow-up visit. Two infants discharged from the ED were hospitalized 2 weeks after their initial ED discharge and developed apnea during their second hospital stays. One infant became apneic before leaving the ED after being written for discharge; he was then admitted but is counted as a discharge. No infant died within 72 hours of discharge. Figure 1 shows patient flow through the study.
Parents reported apnea as the presenting complaint in 96 (10%). emergency physicians found this history to be convincing for central apnea in 62 (6%). Apnea subsequently occurred in the hospital in 41 (4%). Baseline characteristics by outcome are shown in Tables 1 and 2.
In bivariate analysis, younger age, prematurity, infants who weighed less, lower birth weights, lower centile weights for age, shorter duration of preceding symptoms, and parent-reported apnea were associated with apnea while in the hospital. A history of fever was protective (Table 2). Figure 2 shows receiver operating curves for infant age, weight, and birth weight. Commodities are listed in Appendix 1 in supplemental material
Among those with a history of apnea as their primary presenting complaint, 71 (75%) received a discharge diagnosis of some combination of bronchiolitis, pneumonia, or RSV (Table 3).
The candidate rules are presented in Fig 3. The regression-based CDR (Rule A) predicts increased risk of apnea (thereby recommending admission) if any predictors is present. The CART CDR (Rule B) requires the user to follow the decision tree until risk (and therefore disposition) is determined. The RF provides a binary outcome: admit (at risk) or discharge (not at risk). The 10-fold validation performance characteristics of the CDRs on the derivation dataset are compared in Table 4. Alternative models are in the online supplemental materials.
Table 5 shows that regardless of the presenting combination of bronchiolitis, URI, and apnea, infants who developed apnea in the hospital were similar.
Infants who present with a (1) parent report of central apnea, or (2) are ≤6 weeks of age, or (3) weighed <2.5 kg at birth are at high risk of central apnea if they develop bronchiolitis, and particularly early in the course of illness, they should be admitted to the hospital. Other risk factors included weight <5.1 kg, previous history of central apnea, congenital heart disease, and hypoxia.
The first tool is straightforward enough to commit to memory. Our second tool is a decision tree that can be followed to determine risk. The third CDR, an RF, is a computationally intensive black box system that substantially outperforms the first 2 in predictive accuracy but would need to be embedded in an electronic medical record. These CDRs provide increasing specificity at the price of increased complexity. All CDRs had an NPV of 100% (95% confidence interval [CI] 99%–100%) in cross validation but should be prospectively validated in a multicenter study before general use.
The CDR that is ultimately selected for use will depend on validation studies; it is likely that more than one rule will be safe for clinical use. Candidate CDRs that can be validated as safe and accurate will represent different trade-offs between ease of use and specificity. We anticipate that electronic medical record–based CDRs to predict central apnea could be used in some clinical settings and we have derived our candidate CDRs accordingly.
We used a 2-pronged approach to enrollment because our study was prospective and apnea typically occurs early, sometimes as the presenting symptom of RSV and bronchiolitis.9 Even when infants presenting with apnea have clinically evident bronchiolitis, it is the apnea that catches the attention of parents and clinicians. Moreover, the auscultatory signs used to diagnose bronchiolitis are often difficult to elicit in infants younger than 16 weeks, and suffer poor interrater reliability.33,39 Consequently, coexisting bronchiolitis may be minimized or missed; without the “Presents with apnea” pathway for enrollment, such infants might have been missed in our study. By the time they were discharged from the hospital, 75% of those who presented with primary complaint of apnea were diagnosed bronchiolitis, pneumonitis, pneumonia, RSV, or a combination of these. Because of the limitations of RSV immunochromatographic testing, even this estimate may be conservative.40 One of the concerns with our 2-pronged enrollment is that risk factors not related to bronchiolitis would be incorporated into the CDRs; however, sensitivity testing showed this was not the case.
From a clinical perspective, predicting apnea may be more important than knowing its cause; using this philosophy, our methods should be judged by the acceptability of oversampling the minority class in a rare outcomes study. In rare events, prediction standard methodologies that avoid oversampling will lead to potentially catastrophic underestimation of the rare but important outcomes. Therefore, oversampling (or statistical adjustment) of the minority class becomes not only acceptable but necessary even though it would ordinarily be considered a form of recruitment bias.29–31
Nonetheless, we believe there is merit to incorporating an underlying pathophysiological model of central apnea. Our 2-pronged enrollment strategy increased the immunologic homogeneity of our sample as would be classified by increased PGE2 levels. Failing to incorporate an underlying pathophysiological model into predictive modeling could increase the risk of deriving CDRs that reflect poorly generalizable local and behavioral phenomena.
Despite our attention to modeling concepts and methodologies, our results are remarkably similar to the guideline described in Willwerth et al.41 Willwerth et al41 created a prediction rule for apnea in bronchiolitis based on expert opinion and then retrospectively validated the rule on an inpatient cohort at its authors’ site. This approach sidesteps the issue of infants who have apnea as their initial presentation of RSV or bronchiolitis. The CDR described in Willwerth et al41 also found that apnea at presentation predicts subsequent apnea. There have been mixed results when trying to validate it.8,24,42
Schroeder et al8 examined an inpatient cohort with planned ICU oversampling and had mostly similar results to ours. Schroeder et al8 did not attempt to distinguish central apnea and apnea from respiratory exhaustion. This likely explains their interesting finding of U-shaped association between apnea and respiratory rate, low consistent with early or terminal disease, an high suggesting respiratory failure from exhaustion. Exhaustion, or obstructive processes, may also explain why it observed apnea in infants older than 6 months. Our results were more similar to Ricart et al,9 that found apnea alone was the first presenting symptom of bronchiolitis in 33% of infants with apnea and bronchiolitis. Similarly, younger age and prematurity were important risk factors.9
This was a single-site study, which decreases external generalizability. It was not possible to perform both prospective derivation and validation given our anticipated enrollment. Our institutional review board did not permit us to collect any information, including the fact of ED attendance, from nonconsenting patients. The study site did not have a PICU; infants who were intubated could not be followed as closely by the study investigators who had to rely on discharge summaries. We took care to distinguish the reason for intubation. When intubation was performed for recurrent central apnea in the ED, we classified the infant as having had apnea witnessed in the hospital.
Distinguishing central from obstructive apnea can be difficult. Whenever possible an investigator (PW) personally attended each infant with apnea to verify the history and observe the episodes. When this was not possible, a study physician reviewed each case of apneic event in detail. In the ideal experiment, central apnea would be determined by continuous video, plethymosgraphic, and EEG monitoring. We argue that such a design is unnecessary; experienced physicians can make reasonable judgments as to whether an apneic episode has a central component.
We accepted parent reports of apnea at presentation because, although the risk is much greater for episodes the physician regarded as convincing, some infants in whom the physician did not find this report convincing went on to have apnea in the hospital. A “convincing history” is also vulnerable to poor interrater reliability. Every time parents were the first to note an apneic episode in the hospital they summoned nursing staff who saw and documented the event in detail. We had incomplete follow-up in 14%; these could have had nonfatal apnea. However, at that time these patients would only rarely have re-presented to another facility, and we would have been contacted by that ED.
Our patients represent only a subset of infants with ALTE, and our candidate CDRs do not apply to most ALTEs. A CDR has been derived for use with ALTEs.43 We had too few patients with congenital heart disease to meaningfully compare types.
CART can produce deceptively simple-appearing models that encode complex rules; all but the simplest decision trees need to be followed precisely rather than approximated from memory. Our RF models offer the greatest specificity, but would require either integration into an electronic medical record or Web page to facilitate widespread use. This complexity limits the use of RF models to the most technologically enabled settings. We have addressed this limitation to some extent by using open source software and publishing the code. These tools predict apnea; infants not at risk for apnea may require admission for their bronchiolitis, and this decreases the number of admissions the CDRs would prevent.
Most important, this is a derivation study, and our results need to be prospectively validated at other sites. RFs are at particular risk of overfitting and their performance will likely degrade more than that of our simpler models during prospective validation.
We prospectively derived CDRs with an NPV of 100% on 10-fold validation for subsequent central apnea in infants who present to the ED with bronchiolitis or apnea.
We gratefully acknowledge the input of Nathan Kuppermann, MD, and David Andersen, MBA.
- Accepted August 18, 2015.
- Address correspondence to Paul Walsh, MD, Sutter Medical Center Sacramento, Pediatric Emergency Medicine, 2825 Capitol Avenue, Sacramento, CA 95816. E-mail:
Dr Walsh conceived, designed, and implemented the experiment, performed the data analysis, and wrote the manuscript; Dr Cunningham designed the random forest portion of the experiment, performed and replicated the classification and regression tree analyses, and contributed to the final draft of the manuscript; Dr Merchant, Mr Walker, Ms Heffner, and Dr Shanholtzer coordinated, performed, supervised, and implemented the experiment and reviewed and approved the final draft of the manuscript; and Dr Rothenberg, assisted in experiment design, provided statistical oversight, and contributed to the final draft of the manuscript.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported in part by The Pediatric Emergency Medicine Research Foundation, Long Beach, CA, and by Award 5K12HL108964-02 from the National Heart, Lung, and Blood Institute at the National Institutes of Health (NIH), which provided partial salary support for PW during part of the analysis and writing phases of this study. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Heart, Lung, and Blood Institute or NIH or The Pediatric Emergency Medicine Research Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Wilson CE
- ↵Al-Kindy HA, Gelinas JF, Hatzakis G, Cote A. Risk factors for extreme events in infants hospitalized for apparent life-threatening events. J Pediatr. 2009;154(3):332–337, 337.e1–e2
- Oren J,
- Kelly D,
- Shannon DC
- ↵Ricart S, Rovira N, Garcia-Garcia JJ, et al. Frequency of apnea and respiratory viruses in infants with bronchiolitis. Pediatr Infect Dis J. 2015;33(9):988–990
- Tai TC,
- Adamson SL
- Hofstetter AO,
- Saha S,
- Siljehav V,
- Jakobsson P-J,
- Herlenius E
- Siljehav V,
- Shvarev Y,
- Herlenius E
- ↵Hofstetter AO. Apnea and infection in neonates: mediatory role of interleukin-1β and prostaglandin E 2. Stockholm: Karolinksa Institutet and Astrid Lindgren's Children’s Hospital; Available at: https://openarchive.ki.se/xmlui/bitstream/handle/10616/38074/thesis.pdf?sequence=1. Accessed August 2, 2015
- Guerra FA,
- Savich RD,
- Wallen LD,
- et al
- Walsh P,
- Shanholtzer L,
- Loewen M,
- Trinh K,
- McEnulty B,
- Rothenberg SJ
- ↵Capote A, Singh M, Trinh K, et al. Seasonal nature of central apnea matches that of respiratory viruses rather than births. Paper presented at: Proceedings of Western Regional Research Forum; March 2010; Sonoma, CA.
- Arms JL,
- Ortega H,
- Reid S
- Task Force on Prolonged Infantile Apnea
- ↵Walsh P, Caldwell J, McQuillan KK, Friese S, Robbins D, Rothenberg SJ. Comparison of nebulized epinephrine to albuterol in bronchiolitis. Acad Emerg Med. 2008;15(4):305–313
- Tomz M,
- King G,
- Zeng L
- Zou G
- ↵Therneau T, Atkinson B, Ripley B. Recursive Partitioning and Regression Trees. 4.1-3 Computer program and documentation. Available at https://cran.r-project.org/web/packages/rpart/index.html. Accessed September 12, 2015
- Hothorn T,
- Hornik K,
- Strobl C,
- Zeileis A
- Breiman L
- Seed P
- McLellan KE,
- Arora M,
- Schwarze J,
- Beattie TF
- Walsh P,
- Overmyer C,
- Hancock C,
- et al
- Copyright © 2015 by the American Academy of Pediatrics