Use of Traumatic Brain Injury Prediction Rules With Clinical Decision Support
OBJECTIVES: We determined whether implementing the Pediatric Emergency Care Applied Research Network (PECARN) traumatic brain injury (TBI) prediction rules and providing risks of clinically important TBIs (ciTBIs) with computerized clinical decision support (CDS) reduces computed tomography (CT) use for children with minor head trauma.
METHODS: Nonrandomized trial with concurrent controls at 5 pediatric emergency departments (PEDs) and 8 general EDs (GEDs) between November 2011 and June 2014. Patients were <18 years old with minor blunt head trauma. Intervention sites received CDS with CT recommendations and risks of ciTBI, both for patients at very low risk of ciTBI (no Pediatric Emergency Care Applied Research Network rule factors) and those not at very low risk. The primary outcome was the rate of CT, analyzed by site, controlling for time trend.
RESULTS: We analyzed 16 635 intervention and 2394 control patients. Adjusted for time trends, CT rates decreased significantly (P < .05) but modestly (2.3%–3.7%) at 2 of 4 intervention PEDs for children at very low risk. The other 2 PEDs had small (0.8%–1.5%) nonsignificant decreases. CT rates did not decrease consistently at the intervention GEDs, with low baseline CT rates (2.1%–4.0%) in those at very low risk. The control PED had little change in CT use in similar children (from 1.6% to 2.9%); the control GED showed a decrease in the CT rate (from 7.1% to 2.6%). For all children with minor head trauma, intervention sites had small decreases in CT rates (1.7%–6.2%).
CONCLUSIONS: The implementation of TBI prediction rules and provision of risks of ciTBIs by using CDS was associated with modest, safe, but variable decreases in CT use. However, some secular trends were also noted.
- CDS —
- clinical decision support
- CI —
- confidence interval
- ciTBI —
- clinically important traumatic brain injury
- CREST —
- Clinical Research on Emergency Services and Treatment
- CT —
- computed tomography
- ED —
- emergency department
- EHR —
- electronic health record
- GCS —
- Glasgow Coma Scale
- GED —
- general emergency department
- LOC —
- loss of consciousness
- LOS —
- length of stay
- OR —
- odds ratio
- PECARN —
- Pediatric Emergency Care Applied Research Network
- PED —
- pediatric emergency department
- TBI —
- traumatic brain injury
What’s Known on This Subject:
Few previous studies have been conducted in the emergency department to evaluate the implementation of prediction rules and guidelines for patients with minor blunt head trauma. The results have been mixed with regards to changing practice.
What This Study Adds:
Implementation of the Pediatric Emergency Care Applied Research Network traumatic brain injury prediction rules and provision of risks of brain injuries via clinical decision support was associated with modest, safe, but variable reduction in computed tomography use in children with minor head trauma. Secular trends were also noted.
Blunt head trauma in children accounts for >450 000 emergency department (ED) visits annually in the United States; most is minor head trauma.1 Population-based estimates of computed tomography (CT) use in children with minor head trauma are lacking, although recent data suggest decreasing overuse of CT.2,3 Several clinical prediction rules and guidelines have been developed to optimize the use of CT in head-injured children, aiming to minimize unnecessary ionizing radiation exposure.3–8 In a cohort of 42 412 children with minor blunt head trauma (CTs obtained for 35%), our investigative team derived and validated prediction rules for CT use in the Pediatric Emergency Care Applied Research Network (PECARN).6
The importance of prediction rules and guidelines lies in their ability to safely impact care.9 Few implementation and impact studies have been conducted in the ED for patients with blunt head trauma; the results have been mixed with regards to changing practice.10–13 No previous published study has implemented the PECARN traumatic brain injury (TBI) prediction rules or provided risk estimates for clinically important TBI (ciTBI) across multiple EDs by using real-time computerized clinical decision support (CDS).
Our primary aim was to determine whether implementing the 2 age-based PECARN TBI clinical prediction rules by using a multifaceted intervention, centered around computerized CDS, would decrease the rate of cranial CT use for children with minor blunt head trauma at very low risk of ciTBIs, without increasing the rate of missed ciTBIs. Our secondary aim was to determine whether CDS that provided risk data for ciTBI for all children with minor blunt head trauma would decrease CT use.
Setting and Population
We included 5 sites in the PECARN and 8 sites in a northern California Kaiser Permanente ED research network (Clinical Research on Emergency Services and Treatment [CREST] network). The PECARN sites included 4 freestanding children’s hospital EDs and 1 pediatric ED (PED) within a general hospital. All sites (other than the PECARN control PED) used Epic (Verona, WI) as their ED electronic health record (EHR). We purposely chose 2 of the PEDs and all of the general EDs (GEDs) because they had not been part of the derivation and validation of the prediction rules. The other 2 PED intervention sites, although part of the rule derivation/validation, used Epic as their EHR and had the informatics capacity to participate.
There were 5 “analytic units” in CREST, consisting of 3 pairs of GEDs (same physicians staffing paired sites) and 2 unpaired EDs. Two CREST GEDs (1 of the paired analytic units) and 1 PECARN site served as controls to track secular trends. Each site’s institutional review board approved the study with waiver or exemption from written informed consent.
Patients (unit of analysis) were eligible if they were <18 years of age and experienced minor blunt head trauma, defined by Glasgow Coma Scale (GCS) scores of 14 to 15, within 24 hours of ED presentation. In order to conduct a more pragmatic trial and because head trauma is an isolated sporadic event, patients could be enrolled more than once (ie, “patient” in the article connotes “patient encounters”). We have previously described the EHR-based method to identify potentially eligible patients based on ED visit chief complaints.14
The patients of interest were those with minor, nontrivial blunt head trauma, similar to the original PECARN TBI prediction rule study.6 We excluded patients with: penetrating trauma, brain tumors, known coagulopathies or ventricular shunts, preexisting neurologic disorders complicating assessments, or previous neuroimaging obtained at an outside hospital.14 We retrospectively distinguished patients with minor head trauma from those with trivial trauma (as defined in the PECARN prediction rule manuscript) based on clinical data entered in the EHR. For analytic purposes, we a priori excluded patients with trivial trauma, as these patients were not included in the original PECARN prediction rule population.6
ED Patient Assessment and Data Completion
For each patient, clinicians completed a blunt head trauma data collection template specifically designed for the study. This template was needed because the prediction rule data points (Supplemental Table 7) were inconsistently documented in the medical record. All types of clinicians (attending physicians, fellow and resident physicians, nurses, nurse practitioners, and physician assistants) could complete the template. Lead site investigators at each site trained the clinical staff on template completion. Attending faculty and fellows were expected to view already completed head trauma template data and amend it if they disagreed with particular findings.15 To assess the frequency and characteristics of eligible patients who were missed from enrollment, research coordinators manually reviewed the EHR for a randomly sampled day for every 2 weeks of enrollment.
Based on preceding research, we developed a multifaceted intervention to deliver computer-based CDS to provide directive recommendations on CT use and assistive information (eg, risk estimates of ciTBI) to the bedside clinicians.15–20 The intervention included (1) real-time EHR-based CDS within site-specific workflows, (2) specific designation of and facilitation by local opinion leaders (physicians and informatics specialists) to train and encourage staff, and (3) focused education at each site (grand rounds and small group sessions).15,20 The components of the CDS included: (1) automatic determination of whether the patient met the age-specific PECARN prediction rule very low risk criteria; (2) a recommendation that CT was not indicated if the child met the very low risk criteria; (3) risk estimates for ciTBI; and (4) links to the prediction rule criteria and publication (Supplemental Fig 3).15,21 The 1-time education consisted of a standard presentation focused on the CDS components and how to navigate to its different components.
Based on the data from the original PECARN prediction rule study, we also provided clinicians with risks of ciTBI if the patients had only 1 prediction rule risk factor and the upper boundary of the 95% confidence interval (CI) was <2% from the point estimate (Supplemental Fig 4).22–27 For all patients, we provided a statement regarding clinical predictors that placed patients at higher risk for ciTBI (eg, altered mental status). The CDS and recommendations provided were based on the most recent data entered into the head trauma template, which was shared among providers.
Assignment of Sites to Intervention or Control Groups
We selected sites to receive the CDS in a nonrandom fashion. We selected the single PECARN control site based on its use of a different EHR; data collection at this site occurred on paper. Clinicians at all participating sites were unaware of whether CDS would be implemented at their site. The control sites did not systematically implement any interventions to assist in CT decision-making for pediatric head trauma patients (monitored by the lead investigator). At the intervention sites, the time of study before (9.6–15.7 months) and after the intervention (10.1–15.7 months) varied based on the site’s readiness to implement the CDS.
The primary study outcome was the CT rate (primary analysis in those at very low risk of ciTBI and secondary analysis in all with minor head trauma). Secondary outcomes included: (1) the number/percentage of patients with ciTBIs not identified on the initial ED visit, and (2) the length of stay (LOS) in the ED for discharged patients. As in the original PECARN study, we defined ciTBI as death from the TBI, neurosurgical procedure for TBI, intubation for at least 24 hours for TBI, or hospitalization for ≥2 nights due to the head trauma in association with TBI on CT.6 Lead site investigators assessed for ciTBI blinded to prediction rule data. We defined TBI on CT as any acute traumatic intracranial finding or a skull fracture depressed by at least the width of the skull.6
For patients discharged from the ED, we defined ED LOS as the interval in minutes between the time-stamped arrival and the time of discharge. Finally, we assessed the sensitivity of the PECARN prediction rules to identify children with ciTBI. We included in this analysis only those patients for whom all age-specific predictors were completed.
For enrolled patients who did not undergo CT (or MRI) and were discharged from the hospital, research coordinators performed medical record reviews to assess for missed ciTBIs and subsequent TBIs on CT (or MRI). For practical reasons, we did not conduct telephone follow-up for those discharged from the ED. In our previous study, only 1 out of 38 591 patients discharged from the ED was subsequently diagnosed with a ciTBI (hospitalized for 2 nights for a cerebral contusion; no neurosurgery).6
We aimed to enroll at least 746 very low risk patients per site to provide 80% power to detect a 7% absolute difference in the rate of CT use, with a baseline rate of 17%.28 At higher volume sites, we targeted an enrollment of 1178 very low risk patients to reach 90% power to detect a 7% difference from a worst-case baseline CT rate of 20%. Based on site pediatric volumes, we expected 15 to 24 months of enrollment per site.
We conducted the primary analysis on children with minor head trauma at very low risk for ciTBI using segmented logistic regression, separately by site. We fit a logistic regression model that included both intercept and slope terms to account for secular trends, as well as patient age group (< 2 years and 2–18 years of age). If the slope was significant (at a conservative P = .1 level), pre- and postintervention slope terms were included in the final model. A postintervention intercept, representing the effect of CDS implementation, was the primary predictive term of interest. We chose the by-site analysis to be primary due to an a priori hypothesis that the CT rates and intervention effect would vary by site.
We conducted secondary analyses that evaluated: (1) all patients with minor trauma (combining those who did and did not meet very low risk criteria) and (2) only those patients who were not at very low risk of ciTBI. We also examined differential intervention effects between patient age groups. Finally, we conducted multivariable logistic regression analyses to determine the potential effect of clinician, site, and ED acuity factors on the odds of CT for all enrolled patients. All analyses were performed by using SAS version 9.4 (SAS Institute, Inc, Cary, NC).
Clinicians entered data into the EHR template for 28 669 patients (Fig 1). Of the 26 280 patients who were not excluded, 19 029 (72.4%) had minor, nontrivial head trauma, including 16 635 and 2394 at intervention and control sites, respectively. Complete data to determine the proportion of children at very low risk of ciTBI varied among sites between 48% (PED site 2) and 93%. The completeness of data varied minimally within sites throughout the study (and mainly at study outset; data available on request).
Supplemental Table 8 displays the characteristics of providers who completed data for the 19 029 patients with minor head trauma. There was variation among PEDs, with some having substantially higher rates of data completion by nurses. The characteristics of the patients for whom data were entered in the EHR were similar to those patients for whom no data were entered (ie the missed eligible population) (Supplemental Table 9).
Table 1 demonstrates the characteristics of the 19 029 enrolled patients with minor blunt head trauma. The control EDs had a higher proportion of patients <2 years and patients with fewer signs and symptoms of TBI.
The results for our primary aim are displayed in Table 2 and Fig 2. Adjusted for age group and time trends, the CT rates for children at very low risk of ciTBIs after CDS decreased significantly but modestly at 2 PEDs, but did not change significantly at the other 2 PEDs. There was little change in CT rates after CDS at the GEDs. However, the GEDs had low baseline rates of CT use before CDS. These results did not change substantially when all the patients with “unknown” status were considered to be at very low risk. The modest decrease in CT rates after CDS at the PEDs were noted mainly in children <2 years (Table 3). Over the study period, the control PED site had little change in CT use in those at very low risk of ciTBI (from 1.6% to 2.9%; assuming a similar midpoint as intervention EDs); the control GED had a significant decrease (from 7.1% to 2.6%).
Table 4 displays CT rates in all patients with minor blunt head trauma. All sites had small absolute decreases in overall CT use, with 2 PEDs showing statistically significant decreases after controlling for age group and secular trends. The decreased odds of CT use for all patients with minor trauma did not change when controlling for potential confounding factors (odds ratio [OR], 0.91; 95% CI, 0.86–0.97). The control PED site also showed a decrease in CT use in all children with minor blunt head trauma (from 14.1% to 12.5%); the GED control site CT rate decreased as well (from 15.5% to11.5%). Table 5 displays CT rates in those not at very low risk. For these patients, there was a decrease in CT rate at all but 1 intervention site; the control PED also showed a decrease in CT use (from 36.5% to 31.4%), but the GED control site demonstrated an increase (from 36.9% to 44.4%).
ciTBI was identified on the day of ED enrollment in 58/58 (100%) patients before and 55/56 patients (98.2%) after CDS implementation. The 1 patient not identified at the index ED visit after CDS implementation did not meet the PECARN very low risk criteria (the patient had a history of loss of consciousness [LOC]). The patient did not have neurosurgery. The median LOS after CDS implementation increased at 7 of the 8 intervention EDs by 7 to 15 minutes (1 reaching statistical significance, P < .001).
Finally, 3 of the 114 patients who had ciTBIs met the very low risk criteria based on the PECARN prediction rule findings documented in the EHR head trauma template (sensitivity, 97.4%; 95% CI, 92.5%–99.5%); none underwent neurosurgery (Table 6). All 3 patients were from 1 site and either (1) did not have the head trauma template completed by an attending physician, (2) had documented PECARN age-specific risk factors noted on review of the medical record text, or (3) there was concern for nonaccidental trauma.
In this clinical trial of children with minor blunt head trauma, implementation of EHR CDS that provided risk estimates of ciTBI and CT recommendations based on the PECARN TBI prediction rules was associated with small but inconsistent decreases in CT rates, with secular trends also noted. Interestingly, the decreased CT rates in those at very low risk were particularly noted in children <2 years of age. Overall, all PED and GED sites had reduced CT rates in all children with blunt minor head trauma, regardless of risk of ciTBI. Importantly, although a potential concern, there was no increase in CT rates among those who were not at very low risk of ciTBI.29 Finally, we noted a low “miss” rate for ciTBI and no missed neurosurgeries with implementation of the rules.
The few previous studies to implement prediction rules or guidelines aimed to decrease CT use for patients with head trauma have shown no change or no differences between implementation and control groups.10,11 However, a recent quality improvement study noted a decrease in CT use (from 21% to 15%) with implementation of a guideline using an EHR order set, with an additional decrease (from 15% to 9%) with practitioner feedback.13
Our data suggest that a benchmark of <5% CT use can be consistently achieved for children at very low risk of ciTBI. Our data, along with previous studies, also suggest that a CT rate of <15% is achievable for all children with minor blunt head trauma.12,13 The low baseline CT use (except at 1 site) suggest passive diffusion of the PECARN rules before our implementation, increased use of ED observation in lieu of CT, a Hawthorne effect because clinicians were aware of the study, or greater than anticipated secular trends toward decreased CT use before study implementation.3 The 1 PED site in which baseline rates were higher is part of a high-volume GED level 1 trauma center for which surgical staff is frequently involved in the management of children with head trauma, which may have impacted the CT rate.
Compared with previous studies, we developed our intervention to provide CDS early in the ED workflow, before clinician decision-making; most previous efforts have initiated and provided CDS at the time of computer order entry.10,11 Some sites had more participation in template documentation by nurses (who often document earlier in the workflow). The strategy of using nurses to collect these data must be weighed against the modest interobserver reliability between nurses and physicians for some clinical findings.30 Experts continue to call for additional study to test the benefits of different workflows and features of EHR-based CDS (eg, requiring clinicians to justify overriding advice).18,19,31,32
As in previous studies, we found that the PECARN TBI prediction rules accurately identified patients with ciTBI.12,13,33 However, our study and previous implementation studies have noted rare patients “missed” by the rules either (1) were truly misidentified by the rule, (2) had findings that were misinterpreted by the clinicians, or (3) had findings inaccurately documented (eg, within the EHR head trauma template).10 Although we cannot be certain, the 3 patients with ciTBI in our study who were missed by the rule either had PECARN TBI rule factors that were inaccurately documented in the head trauma template or had histories that were concerning for child abuse (for whom the PECARN rules were not intended).
The study has several limitations. Due to the complexities of implementing an EHR intervention, the study was not randomized and included only 1 EHR type. We purposefully allowed variation regarding which type of providers could complete the head trauma template. This flexibility facilitated integration within each site’s workflow, although it may have led to some errors in clinical documentation of PECARN risk factors and TBI risk assignment. This flexibility also resulted in 1 site having a higher proportion of patients with incomplete data due to lack of documentation of skull fracture findings by nurses (who, at this site only, felt this was beyond their expertise).15 To capture structured data in the EHR, we required the creation of a blunt head trauma data template, which could be viewed as an intervention itself. Additionally, we did not collect data to assess whether those who completed the template and viewed the CDS were also those who ordered CTs. At all sites, however, if residents evaluated patients before the attending/fellow, they were expected to discuss the case before ordering neuroimaging. The design also allowed patients to be enrolled more than once, although it is unclear how this would meaningfully change CT use given the large overall sample size and the sporadic nature of head trauma.
Our analytical approach adjusted for secular trends. However, it is impossible to completely exclude the influence of secular trends on the assessment of the intervention. The baseline CT rates for those at very low risk of ciTBI were substantially lower than expected, making it difficult to consistently achieve significant reductions or note secular trends (particularly at the control PED). The low baseline CT rates in children at very low risk of ciTBI were potentially due to passive diffusion of the PECARN rules, increased awareness of the risks of CT resulting in more judicial use of CT, and, at the community GEDs, a reflection of an integrated health system focused on highly efficient care.2,3 Additional study should determine whether greater CT reductions occur in EDs with higher baseline CT use. The variability in baseline rates could provide a real-world view of how the CDS might work at institutions with differing characteristics. Additionally, for practical reasons, we used a control site with a different EHR than the intervention sites. A better control site would have been 1 that used the same EHR to limit any variability in CT use due to differences in EHRs. Finally, we were unable to assess for other potential confounders that might impact CT use, such as personal perspectives, organizational initiatives to minimize unnecessary imaging, use of alternative neuroimaging when appropriate (eg MRI), and family, consultant, or other specialist requests.34–36
The implementation of TBI prediction rules and provision of risks of ciTBI by using computerized CDS was associated with modest but variable decreases in rates of CT use for children at very low risk of ciTBI and for all children with minor blunt head trauma, without increasing the rate of missed injuries. However, decreased CT rates were inconsistent across study sites and secular trends were noted.
We thank the research coordinators, informatics specialists, and clinicians at all participating sites, without whose support this study could not have been completed.
- Accepted January 24, 2017.
- Address correspondence to Peter Dayan, MD, MSc, Division of Emergency Medicine, Department of Pediatrics, Columbia University College of Physicians and Surgeons, 3959 Broadway, CHN-1-116, New York, NY 10032. E-mail:
FINANCIAL DISCLOSURE: Dr Goldberg provides consultation to the following companies: Intelligent Medical Objects (Chicago, IL) and Practical Solutions in Medical Informatics (Pt Richmond, CA). Additionally, Dr Goldberg is in on the Advisory Committee of Clearsense (Jacksonville, FL). The other authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This work was supported by American Recovery and Reinvestment Act, Office of the Secretary grant S02MC19289-01-00. The Pediatric Emergency Care Applied Research Network is supported by the Health Resources and Services Administration, Maternal and Child Health Bureau, Emergency Medical Services for Children Program through the following cooperative agreements: U03MC00001, U03MC00003, U03MC00006, U03MC00007, U03MC00008, U03MC22684, and U03MC22685.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Faul M,
- Xu L,
- Wald MM,
- Coronado VG
- Kuppermann N,
- Holmes JF,
- Dayan PS, et al; Pediatric Emergency Care Applied Research Network (PECARN)
- Moons KG,
- Kengne AP,
- Grobbee DE, et al
- Stiell IG,
- Clement CM,
- Grimshaw JM, et al
- Mooney JS,
- Yates A,
- Sellar L, et al
- Deakyne SJ,
- Bajaj L,
- Hoffman J, et al; Pediatric Emergency Care Applied Research Network (PECARN)
- Sheehan B,
- Nigrovic LE,
- Dayan PS, et al; Pediatric Emergency Care Applied Research Network (PECARN)
- Sittig DF,
- Singh H
- Bates DW,
- Kuperman GJ,
- Wang S, et al
- Kawamoto K,
- Houlihan CA,
- Balas EA,
- Lobach DF
- Goldberg HS,
- Paterno MD,
- Grundmeier RW; Pediatric Emergency Care Applied Research Network (PECARN), et al
- Dayan PS,
- Holmes JF,
- Schutzman S, et al; Traumatic Brain Injury Study Group of the Pediatric Emergency Care Applied Research Network (PECARN)
- Nigrovic LE,
- Lee LK,
- Hoyle J, et al; Traumatic Brain Injury (TBI) Working Group of Pediatric Emergency Care Applied Research Network (PECARN)
- Dayan PS,
- Holmes JF,
- Hoyle J Jr, et al; Pediatric Emergency Care Applied Research Network (PECARN)
- Roshanov PS,
- Fernandes N,
- Wilczynski JM, et al
- Tavender EJ,
- Bosch M,
- Gruen RL, et al
- Frush DP,
- Frush KS
- ↵Physician groups join forces to improve use of head CT scans in children “Think a-head campaign” to empower more informed patient-provider imaging communications. Available at: http://www.imagewisely.org/∼/media/ImageWisely-Files/News/IG-Think-Ahead-Launch-Press-Release---Final.pdf. Accessed January 10, 2017
- Copyright © 2017 by the American Academy of Pediatrics