Abstract
BACKGROUND AND OBJECTIVE: To reduce missed cases of pediatric abusive head trauma (AHT), Pediatric Brain Injury Research Network investigators derived a 4-variable AHT clinical prediction rule (CPR) with sensitivity of .96. Our objective was to validate the screening performance of this AHT CPR in a new, equivalent patient population.
METHODS: We conducted a prospective, multicenter, observational, cross-sectional study. Applying the same inclusion criteria, definitional criteria for AHT, and methods used in the completed derivation study, Pediatric Brain Injury Research Network investigators captured complete clinical, historical, and radiologic data on 291 acutely head-injured children <3 years of age admitted to PICUs at 14 participating sites, sorted them into comparison groups of abusive and nonabusive head trauma, and measured the screening performance of the AHT CPR.
RESULTS: In this new patient population, the 4-variable AHT CPR demonstrated sensitivity of .96, specificity of .46, positive predictive value of .55, negative predictive value of .93, positive likelihood ratio of 1.67, and negative likelihood ratio of 0.09. Secondary analysis revealed that the AHT CPR identified 98% of study patients who were ultimately diagnosed with AHT.
CONCLUSIONS: Four readily available variables (acute respiratory compromise before admission; bruising of the torso, ears, or neck; bilateral or interhemispheric subdural hemorrhages or collections; and any skull fractures other than an isolated, unilateral, nondiastatic, linear, parietal fracture) identify AHT with high sensitivity in young, acutely head-injured children admitted to the PICU.
- abusive head trauma
- child abuse
- decision rule
- nonaccidental trauma
- prediction rule
- predictors
- screening tools
- AHT —
- abusive head trauma
- CI —
- confidence interval
- CPR —
- clinical prediction rule
- CT —
- computed tomography
- MRI —
- magnetic resonance imaging
- PediBIRN —
- Pediatric Brain Injury Research Network
What’s Known on This Subject:
Pediatric Brain Injury Research Network investigators recently derived a highly sensitive clinical prediction rule for pediatric abusive head trauma (AHT).
What This Study Adds:
The performance of this AHT screening tool has been validated. Four clinical variables, readily available at the time of admission, detect pediatric AHT with high sensitivity in intensive care settings.
PICU providers are required to make important decisions to launch or forgo abuse evaluations in their young, acutely head-injured patients. These decisions can be quite difficult. Presenting signs and injuries are often nonspecific; the history of head trauma can be absent, minimized, changing, or fabricated; external injuries are not universal; most patients are unable to describe their trauma; and perceptions of psychosocial risk can influence objectivity.
The stakes are high. A flawed decision to forgo an abuse evaluation creates substantial risk of additional injury for an abused, head-injured child returned to his or her abusive caregivers.1 A flawed decision to launch an abuse evaluation can also have adverse consequences, including exacerbation of parental stress, strain of the doctor–patient relationship, false-positive results, exposure of the child to additional risks (sedation, radiation), prolongation of hospital stays, and increased health care costs. Unfortunately, physicians have demonstrated significant bias and disparity in their evaluation of child maltreatment.1–12
To minimize missed cases of abusive head trauma (AHT), some clinicians elect to complete abuse evaluations on every hospitalized, acutely head-injured infant or young child. Although this approach has merit, we suspect that many clinicians defer abuse evaluations in some of their young, acutely head-injured patients when they deem it safe to do so. These clinicians might welcome access to an evidence-based AHT screening tool that can inform their decisions to launch (or to forgo) an abuse evaluation.
A clinical prediction rule (CPR) is an evidence-based tool that measures and then combines the predictive qualities of multiple clinical findings or tests to predict the probability of a diagnosis, prognosis, or response to therapy for an individual patient. Clinical prediction rules are most likely to be useful when decision-making is complex, when clinical stakes are high, or when cost savings can be achieved without compromising quality or safety. By convention, development of any CPR proceeds in 3 sequential stages: derivation, validation, and implementation.13,14
Pediatric Brain Injury Research Network (PediBIRN) investigators have derived a CPR that is simple and reliable and detects AHT with high sensitivity.15 Focusing exclusively on the PICU setting, we captured complete historical, clinical, and radiologic data on 209 acutely head-injured infants and young children admitted to PICUs at 14 sites; applied a priori definitional criteria to sort patients into comparison groups of abusive and nonabusive head trauma; and identified multiple clusters of predictor variables that, alone or in combination, identified AHT with high sensitivity. A 4-variable AHT CPR demonstrated the optimal combination of simplicity, reliability, and performance, with sensitivity of .96. Achieving maximal sensitivity came at a cost; specificity was only .36.
This AHT CPR can be applied at or near the time of PICU admission, categorizes patients presenting with ≥1 of its 4 predictor variables as higher risk, categorizes all remaining patients as lower risk, recommends thorough abuse evaluations for all higher-risk patients, and makes no specific recommendations regarding abuse evaluations for lower-risk patients (Table 1).
The 4-Variable AHT CPR
The primary objective of this validation study was to verify the screening performance of this 4-variable AHT CPR in a new, equivalent patient population. We hypothesized that the CPR would detect AHT with sensitivity ≥.96. This study was not designed or executed to assess physicians’ acceptance or use of the AHT CPR in actual clinical practice or to measure its impact on relevant clinical outcomes.
We also completed secondary analyses designed to compare the predictive accuracy of various sum numbers of the CPR’s 4 predictor variables, estimate the CPR’s power to identify (categorize as higher risk) patients ultimately diagnosed with AHT by their treating or consulting physicians, verify the isolated predictive qualities of each of the CPR’s 4 predictor variables, and compare the CPR’s performance in different geographic regions, in large PICUs versus small PICUs, and in the same PICUs over time.
Methods
Patients and Settings
We conducted a prospective study of acutely head-injured infants and young children admitted to PICUs within the continental United States between March 2012 and July 2013. Ten of the 14 PICUs had participated previously in the CPR derivation study.
Study Design
This multicenter, cross-sectional study was strictly observational. Every participating site received approval from its human subjects research committee with a waiver of informed consent. As required by convention for CPR development, we applied the same inclusion and exclusion criteria, definitional criteria for AHT, and research methods used in the completed CPR derivation study.15
Inclusion and Exclusion Criteria
Eligible patients were children <3 years of age admitted to the PICU for management of symptomatic, acute, closed, traumatic, cranial, or intracranial injuries confirmed by computed tomography (CT) or magnetic resonance imaging (MRI). Treating clinicians made local, independent determinations as to whether their patients’ acute clinical presentations and initial neuroimaging findings resulted specifically from trauma. Patients were excluded if their head injuries resulted from a collision involving a motor vehicle or if initial neuroimaging revealed clear evidence of preexisting brain malformation, disease, infection, or hypoxia-ischemia. Eligible patients who died were not excluded.
Data Collection and Management
As in the completed CPR derivation study, participating PICUs were required to enroll >90% of eligible patients, standard-of-care treatment yielded all required data, and local investigators captured data on electronic data forms within an access-controlled, online data registry. To minimize observational bias, data on the 4 predictor variables and data informing the outcome variable (abusive vs nonabusive head trauma) were captured independently. Concerns about data integrity or internal consistency were directed to local investigators via e-mail and tracked until resolution. Analyses were undertaken using Excel (Microsoft, Redmond, WA).
Outcome Measure
Gold standard criteria for the diagnosis of AHT do not exist. In their absence, we applied the same a priori, definitional criteria for AHT used in the completed derivation study to sort study patients into comparison groups of abusive and nonabusive head trauma (Table 2). Formulated by the lead author, these criteria mirror or replicate the criteria used in other peer-reviewed studies.16–18 To minimize circular reasoning, they contain no references to specific intracranial injuries, to injury severity, or to any of the 4 predictor variables included in the AHT CPR. Patients meeting ≥1 of these criteria were sorted as having AHT. All remaining patients were sorted as having nonabusive head trauma. Study patients’ final diagnoses of abusive versus nonabusive head trauma were not considered in the primary analyses but were used to complete relevant secondary analyses.
A Priori Definitional Criteria for AHT
Primary Analyses
To measure the screening performance of the 4-variable AHT CPR in the new patient population, we applied the definitional criteria to sort study patients into comparison groups of abusive versus nonabusive head trauma, applied the AHT CPR to every study patient, categorized patients who manifested ≥1 of the CPR’s 4 predictor variables as higher risk, categorized all remaining patients as lower risk, compared patient categorizations (higher vs lower risk) with patient sorting (abusive vs nonabusive head trauma), and calculated the AHT CPR’s screening performance (sensitivity, specificity, predictive values, likelihood ratios).
Secondary Analyses
To compare the predictive accuracy of various sum numbers of the 4 predictor variables included in the AHT CPR, we plotted an empirical receiver operator curve (ROC) and calculated its area under the curve.
To estimate the AHT CPR’s power to identify (categorize as higher risk) study patients ultimately diagnosed with AHT by their treating or consulting physicians, we compared patient categorizations (higher vs lower risk) with patients’ final diagnoses (abusive vs nonabusive head trauma) and recalculated the AHT CPR’s screening performance (sensitivity, specificity, predictive values, likelihood ratios). For this secondary analysis, treating and consulting physicians’ final, consensus diagnoses of definitive or probable AHT were collapsed into a single AHT category.
To verify or validate the predictive qualities of the CPR’s 4-predictor variable in isolation, we calculated each variable’s sensitivity, specificity, predictive values, and likelihood ratios and compared these results to equivalent results from the completed derivation study.
To estimate the screening performance of the AHT CPR in different parts of the country (geographic validation), in both large and small PICUs (domain validation), and in the same PICUs over time (temporal validation), we analyzed the data from novel patient cohorts selected specifically to assess geographic, domain and temporal validation.
Results
Primary Analysis
Between March 2012 and June 2013, PediBIRN investigators at the 14 participating sites captured complete clinical, historical, and radiologic data on 326 of 327 (99.7%) acutely head-injured infants and young children admitted to their PICUs. Thirty-five of these patients met exclusion criteria and were excluded from analysis, including 31 injured in motor vehicle collisions and 4 with neuroimaging evidence of preexisting brain malformation, disease, infection, or hypoxia-ischemia. The remaining 291 patients made up the final validation study patient population. Their demographic characteristics mirrored those of the derivation study patient population (Table 3). All questions about data integrity or internal consistency were resolved at the discretion of local investigators.
Patient Demographics
One hundred twenty-four (43%) of 291 study patients met ≥1 definitional criteria for AHT, including 4 whose AHT was witnessed independently, 18 whose perpetrators admitted abusive acts, and 72 who manifested ≥2 categories of extracranial injuries considered moderately or highly suspicious for abuse. These results also mirrored those obtained in the completed derivation study (Table 2). The remaining 167 (57%) of 291 children were sorted as having nonabusive head trauma.
Applied to this new patient population (N = 291), the 4-variable AHT CPR demonstrated sensitivity = .96 (95% confidence interval [CI], .90–.99). That is, of the 124 study patients who met criteria for AHT, 119 (96%) presented with ≥1 of the 4 predictor variables included in the 4-variable AHT CPR. Specificity was .43 (95% CI, .35–.50). Positive and negative predictive values were .55 (95% CI, .48–.62) and .93 (95% CI, .85–.98), respectively. Positive and negative likelihood ratios were 1.67 (95% CI, 1.46–1.9) and 0.09 (95% CI, 0.04–0.23), respectively. All measures of AHT screening performance matched or exceeded equivalent results from the completed CPR derivation study (Table 4).
4-Variable AHT CPR Screening Performance Defining AHT Using Definitional Criteria
Secondary Analyses
The empirical ROC curve in Fig 1 demonstrates the tradeoff between sensitivity and specificity across various cutoff points (ie, across various sum numbers of the 4 predictor variables, from 0 to 4 for this 4-variable CPR). To maximize sensitivity, we selected 1 as the optimal cutoff point. The area under the curve was .78.
Empirical receiver operator curve for 4-variable AHT CPR.
We found significant differences (McNemar’s χ2 test with continuity correction, P ≤ .001) between patient sorting based on the definitional criteria and study patients’ final diagnoses of abusive or nonabusive head trauma (Table 5). More specifically, the definitional criteria sorted fewer study patients as having AHT than physicians themselves. Nevertheless, the 4-variable AHT CPR identified (categorized as higher risk) 98% of study patients who were ultimately diagnosed with AHT (Table 6).
Patient Sorting Using Definitional Criteria Versus Patients’ Final Diagnoses
4-Variable AHT CPR Screening Performance Defining AHT Using Patients’ Final Diagnoses
Considered in isolation, each of the AHT CPR’s 4 predictor variables demonstrated predictive qualities (sensitivity, specificity, predictive values, and likelihood ratios) in this validation study that were very similar to those demonstrated in the completed derivation study (Supplemental Table 8).
Retrospective analyses of AHT CPR performance in novel patient cohorts revealed that the 4-variable AHT CPR demonstrated high sensitivity in different geographic regions, in large PICUs and in small PICUs, and in the same PICUs over time (Supplemental Table 9).
Discussion
An effective screening test is simple, a reliable, performs with high sensitivity, has few false negatives, and is therefore useful to exclude a diagnosis, when negative. An effective diagnostic test is simple, reliable, performs with high specificity, has few false positives, and is therefore useful to confirm a diagnosis, when positive. PediBIRN investigators’ long-term aim is the development and dissemination of an effective AHT screening test, not a diagnostic test. Stated in another way, PediBIRN investigators are working to develop a simple, reliable, and highly sensitive AHT screening test that can be applied at or near the time of PICU admission, helps exclude AHT when negative, helps reduce missed cases of AHT, and can inform PICU providers’ early decisions to launch (or to forgo) abuse evaluations in their young, acutely head-injured patients.
The screening performance of the 4-variable AHT CPR has now been validated in a new, equivalent patient population. All measures of performance matched or exceeded equivalent results from the CPR derivation study (Table 4). In the absence of a gold standard, we derived and validated the CPR’s performance using a priori definitional criteria to define AHT (Table 2). Applied consistently, the CPR would have detected AHT, as defined by these criteria, with sensitivity of .96, and would have detected study patients who were ultimately diagnosed with AHT with sensitivity of .98. This latter result was somewhat surprising, in that the definitional criteria sorted fewer patients as AHT than were ultimately diagnosed with AHT.
To achieve maximum sensitivity, some screening tests demonstrate only marginal specificity.19,20 This CPR was no exception. Specificity was .43. Applied consistently, the AHT CPR would have recommended abuse evaluations for 57% of study patients with nonabusive head trauma.
The 4 predictor variables included in the AHT CPR (acute respiratory compromise before admission; bruising of the torso, ears, or neck; interhemispheric or bilateral subdural hemorrhages or collections; and any skull fractures other than an isolated, unilateral, nondiastatic, linear, parietal fracture) are all readily available at or near the time of PICU admission. Their isolated predictive qualities have been verified or validated (Supplemental Table 8), and each demonstrated high interrater reliability (κ, κ ≥0.80) in the completed CPR derivation study.15 Interestingly, in isolation, the fourth predictor variable (any skull fractures other than an isolated, unilateral, nondiastatic, linear, parietal fracture) is predictive of nonabusive head trauma.
The AHT CPR’s sensitivity is not 100%. It “missed” (categorized as lower risk) 5 (4%) of 124 validation study patients who met criteria for AHT, 3 of whom were also diagnosed with AHT by their treating or consulting physicians. These 5 “missed” cases (Table 7) include a newborn with probable birth injury, and 4 children between the ages of 3 and 8 months with sentinel cranial injuries that have low specificity for AHT (unilateral, nondiastatic, linear, parietal skull fracture, n = 3; epidural hematoma, n = 2; and unilateral or focal subdural hemorrhage, n = 1). Their intensive care providers elected to launch abuse evaluations in these 4 children because their caregivers specifically denied any accidental or inflicted head trauma, or they provided an account of the child’s head injury event interpreted to be inconsistent with the child’s gross motor skills or historically inconsistent with repetition over time. In 2 of these 4 patients, skeletal survey revealed fractures considered moderately or highly specific for abuse. Considered in their entirety, these 5 “missed” cases illustrate the AHT CPR’s potential limitations.
Five “Missed” AHT Patients Who Met Criteria for AHT but Were Categorized as Lower Riska
Study Strengths and Limitations
This study’s primary strengths include its prospective, multicenter, observational design and its method designed and successfully executed to capture complete data and to minimize sampling and observational bias. The study had limited generalizability in that application of the AHT screening tool in non-PICU sites will require separate validation of screening performance in those environments. Also, this study had definitional limitations: There is no gold standard for the diagnosis of AHT, and the definitional criteria used to sort patients into comparison groups of abusive or nonabusive head trauma are probably imperfect. Finally, the study had design limitations: Because our study was strictly observational, variations in the frequency, timing, or modalities of cranial imaging could have affected the validity of conclusions about specific injuries, and local clinicians’ independent determinations that their patients’ head injuries resulted specifically from trauma might have been inaccurate.
Conclusions
The screening performance of a 4-variable AHT CPR has been validated in a new patient population. Applied consistently, it will detect AHT with high sensitivity in young, acutely head-injured patients admitted to the PICU. Additional research is needed to measure and optimize the AHT CPR’s use and screening performance in actual clinical practice and to verify its effect on relevant clinical outcomes.
Acknowledgments
Additional PediBIRN investigators who participated actively in the validation study include the following:
Bruce E. Herman, MD, Department of Pediatrics, University of Utah School of Medicine, Primary Children’s Medical Center, Salt Lake City, Utah; Sandeep K. Narang, MD, JD, Department of Pediatrics, University of Texas Health Science Center, Houston, Texas; Jeanine M. Graf, MD, Department of Pediatrics, Baylor College of Medicine, Texas Children’s Hospital, Houston, Texas; Mark Dias, MD, Departments of Neurosurgery and Pediatrics, Penn State College of Medicine, Hershey, Pennsylvania; Deborah A. Pullen, BSN, MPH, Department of Pediatrics, Dartmouth–Hitchcock Medical Center, Lebanon, New Hampshire; Stephen C. Boos, MD, Department of Pediatrics, Baystate Children’s Hospital, Springfield, Massachusetts.
Footnotes
- Accepted October 3, 2014.
- Address correspondence to Kent P. Hymel, MD, The Penn State Milton S. Hershey Medical Center, Department of Pediatrics, Division of Child Abuse Pediatrics, 500 University Drive, MC H085, Hershey, PA 17033. E-mail: kphymel{at}gmail.com
Dr Hymel conceptualized and designed the study, designed the data collection instruments, supervised data collection at all participating sites, reviewed all data for integrity and internal consistency, carried out the initial analyses, drafted the initial manuscript, and revised the manuscript; Drs Armijo-Garcia, Foster, Frazier, Stoiko, Harper, Weeks, Carroll, Hyden, Sirotnak, Truemper, and Ornstein and Ms Christie participated in the initial conceptual design of the study, supervised data collection at 1 participating site, and revised the manuscript; Dr Wang participated in the later design of the study, verified the appropriateness and accuracy of all preliminary statistical analyses, directed and completed all secondary statistical analyses, and revised the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Dr Hymel’s research activities were supported by The Gerber Foundation.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
References
- Copyright © 2014 by the American Academy of Pediatrics
In this issue
