OBJECTIVE: The Pediatric Emergency Care Applied Research Network (PECARN) traumatic brain injury (TBI) clinical prediction rules identify children with minor blunt head trauma who are at low risk for clinically important traumatic brain injuries. We measured the agreement between the registered nurse (RN) and physician (MD) assessments.
METHODS: We performed a cross-sectional study of all children <18 years of age with minor blunt head trauma who presented to a single emergency department. RNs and MDs independently assessed each child and recorded age-based PECARN predictors. As symptoms can change over time, we included cases only when both evaluations were completed within 60 minutes. We used the κ statistic to measure RN-MD agreement, with the main analysis focusing on the overall PECARN rule agreement.
RESULTS: Of the 1624 eligible children, 1191 (73%) had evaluations completed by both RN and ED providers, of which 437 (37%) were in children <2 years of age. The median time between completions of the provider forms was 12 minutes (interquartile range 4–25 minutes). The overall agreement between the RN and MD was higher for the older children (κ 0.55, 95% confidence interval 0.49–0.61 for children 2–18 years versus κ 0.32, 95% confidence interval 0.23–0.41 for children <2 years).
CONCLUSIONS: The overall agreement between RN and MD for the PECARN TBI prediction rules was moderate for older children and fair for younger children. Initial RN assessments should be verified by the MD before clinical application, especially for the youngest children.
- CI —
- confidence interval
- CT —
- computed tomography
- ED —
- emergency department
- IQR —
- interquartile range
- MD —
- medical doctor
- PECARN —
- Pediatric Emergency Care Applied Research Network
- RN —
- registered nurse
- TBI —
- traumatic brain injury
What’s Known on This Subject:
Effective implementation of Pediatric Emergency Care Applied Research Network head trauma rules depends on their early application. As the registered nurse (RN) is often the first to evaluate children with blunt head trauma, initial RN assessments will be an important component of this strategy.
What This Study Adds:
We demonstrated fair to moderate agreement between RN and physician providers in the application of the Pediatric Emergency Care Applied Research Network head trauma rules. Effective implementation strategies may require physician verification of RN predictor assessments before computed tomography decision-making.
Clinical prediction rules assist with clinical decision-making by providing an estimate of risk or a probability of an outcome, and suggesting a course of action.1 For example, the Ottawa ankle rules identify which adults presenting to the emergency department (ED) with an ankle injury require radiography to evaluate for potential fracture.2,3 In many busy EDs, a registered nurse (RN) routinely applies the Ottawa ankle rules to determine whether to obtain ankle radiography before initial physician (MD) evaluation.4–7
The Pediatric Emergency Care Applied Research Network (PECARN), a federally funded research network, has derived and validated 2 clinical prediction rules to accurately identify children at low risk for clinically important traumatic brain injury (TBI) after minor blunt head trauma, who may safely avoid cranial computed tomography (CT).8 The predictors included these rules were assessed by the attending MD and selected from variables that had at least moderate MD-MD interrater reliability.9 However, the ED RN is typically the first health care provider to evaluate a child with minor blunt head trauma. Application of the PECARN TBI prediction models soon after patient arrival may increase the clinical impact of these prediction rules by improving initial resource allocation (eg, children with minor blunt head trauma at low risk for clinically important TBI might be triaged to a fast-track area, whereas children at higher risk could be more rapidly assessed in an acute-care area of the ED). However, the agreement between the RN and MD interpretation and application of the PECARN TBI prediction rules has not been investigated.
If future implementation strategies for the PECARN TBI clinical prediction rules were to rely on RN assessments, then the level of agreement between the RN and MD must be high. To this end, we measured the agreement between RN and MD assessments of the PECARN TBI predictors for children presenting to the ED for evaluation of head trauma.
We performed a prospective observational study of all children who presented to a single tertiary care pediatric ED (a free-standing children’s hospital) for evaluation of minor blunt head trauma over a 20-month period (between April 27, 2011, and December 16, 2012). The institutional review board of the study institution approved the study protocol.
Study Setting and Population
We enrolled staff RNs and MDs (emergency medicine, pediatric emergency medicine, and general pediatrics) working in the study institution ED. We classified all participating MDs in one of the following categories: emergency medicine attending physicians (pediatric emergency medicine or emergency medicine), pediatric emergency medicine fellows, or general pediatricians. The single nurse practitioner working in the ED during the study period was considered a general pediatrician for the purposes of this study. We obtained written informed consent from all participating clinicians. We collected training level (RN nursing level and MD board certification/eligibility) as well as clinical experience (years working in a pediatric ED at the time of study initiation). The study institution defined the professional nursing model as a clinical ladder with 3 levels. A level 1 RN provides direct patient care. In addition, a level 2 or 3 RN assumes greater leadership responsibilities (eg, charge RN role, peer mentor, and committee work). A level 3 RN also evaluates other nursing staff.
We included patient encounters if the injured child had a Glasgow Coma Scale score of 14 or 15 and the injury occurred <24 hours before ED presentation. We did not include encounters for children who had trivial head trauma (defined as ground-level falls or colliding with stationary objects with no signs of head trauma other than an abrasion or laceration), penetrating trauma, known brain tumors or other significant neurologic disorders, ventricular shunts, bleeding disorders, or neuroimaging performed before ED arrival.
After patient assessment, both treating RN and MD participants completed a paper study form that captured the appropriate 6 PECARN age-based (<2 years versus 2–18 years of age) TBI predictors with study definitions (Table 1). Both RN and MD staff were asked to indicate the time of study form completion. As otoscopy is not part of an ED RN assessment, we defined signs of basilar skull fracture (one of the predictors for children 2–18 years) as retro-auricular bruising (Battle sign) or periorbital bruising (raccoon eyes). In addition, MDs were asked to indicate the presence or absence of a hemotympanum for children in their assessment of signs of basilar skull fracture. Study form completion was facilitated by study investigators as well as research coordinators. Both RN and MD participants were blinded to the content of the other’s study forms. We collected and managed data using REDCap electronic data capture tools hosted by the study institution.10
Measurements: PECARN TBI Prediction Rules
The 2 PECARN TBI prediction rules identify children at low risk of clinically important TBI defined as a TBI resulting in death, neurosurgical intervention, intubation for >24 hours, or 2 or more nights in the hospital for the management of the TBI in association with a positive CT scan.8 Patients with none of the TBI predictors were classified as low risk of clinically important TBI and those with 1 or more TBI predictors were considered not at low risk. For study purposes, we excluded children missing any PECARN TBI predictors from data analysis.
For the primary analysis, we included ED head trauma encounters when both the RN and MD completed the study forms. As clinical symptoms and signs can change over time, we limited our analysis to assessments that were completed within 60 minutes of each other. For children <2 years and those 2 to 18 years of age, we compared the RN and MD assessment for the PECARN TBI rules overall (very low risk versus not very low risk),8 in addition to each TBI predictor individually. If an MD indicated the presence of signs of basilar skull fracture based on hemotympanum alone, we considered the MD assessment as negative for the RN-MD comparison.
First, we tested for differences between the paired RN and MD assessments by using the McNemar test.11 To further quantify the level of RN-MD agreement, we next calculated the percent absolute agreement as well as the Cohen unweighted κ statistic to assess for agreement beyond chance alone.12 We examined overall agreement of the PECARN TBI rules (ie, low risk versus not low risk), in addition to the individual aged-based TBI rule predictors. We calculated the 95% confidence intervals (CIs) for the point estimates using normal approximation methods. We classified the level of agreement based on published standards for the κ point estimates (Table 2).12,13
Last, we investigated whether the overall agreement between RNs and MDs in assessment of the PECARN TBI rules differed by patient age or RN or MD experience. We used a generalized estimating equation to measure the association between RN and MD assessment for the overall PECARN TBI prediction rules and then adjusted for RN staff level (1–3), MD staff type (emergency medicine attending, pediatric emergency medicine attending, pediatric emergency medicine fellow, general pediatrician), years of experience in a pediatric ED, and patient age (<2 years versus 2–18 years of age), as well as clustering by individual provider.
We estimated the required sample size needed to demonstrate a lower end of the 95% CI of the κ statistic of >0.4 (indicating at least moderate agreement),12 assuming a point estimate of 0.5 for the overall agreement between RNs and MDs for assessment of the PECARN prediction rule being positive or negative (with the prevalence of low risk for clinically important TBI estimated to be 50%). This yielded a sample size of ∼400 children in each age group (<2 years versus 2–18 years). As approximately one-third of the children presenting to the study ED with minor blunt head trauma are younger than 2 years, we planned to enroll ∼1200 children overall.
We used the Statistical Program for the Social Sciences, version 22.214.171.124, for all analyses (IBM SPSS Statistics, IBM Corporation, Chicago, IL).
We enrolled 81 RNs, of whom 54 (67%) were level 1, 19 (23%) were level 2, and 8 (10%) were level 3. We enrolled 98 MDs, of whom 55 (56%) were emergency medicine attending physicians (53 pediatric emergency medicine and 2 emergency medicine), 22 (22%) were pediatric emergency fellows, and 21 (21%) were general pediatricians. No staff member approached for this study refused to participate. Each participating RN staff member completed a median of 10 study forms (interquartile range [IQR] 3–18 forms) and each participating MD staff member completed a median of 5 forms (IQR 1–14). Level 1 staff RNs and pediatric emergency medicine staff and fellows completed most of the study forms (Table 3).
During the study period, 1624 eligible children presented to the ED and study forms were completed for 1375 (85% of eligible patients). Of these, 174 (13%) had MD forms only and 1201 (87%) had both RN and MD forms completed. Of the encounters that had both RN and MD forms completed, 1191 (73% of all eligible patients) had both forms completed within 60 minutes of each other and were included in the analysis. Ten RN-MD paired assessments were not included, as the study forms were completed more than 60 minutes apart. The median time between RN and MD form completion was 12 minutes (IQR 4–25 minutes).
Of the 1191 RN-MD paired assessments of children with minor blunt head trauma included in this study, 437 (37%) were for children <2 years and 754 (63%) were for children 2–18 years old. Sixty percent of study patients were boys. As a group, the RNs assessed 61% of children <2 years old, as well as 44% of children 2 to 18 years to be at low risk for a clinically important TBI. Similarly, as a group, the MDs assessed 55% of children <2 years old and 50% of children 2 to 18 years to be at very low risk. Of the 598 children classified as very low risk based on RN assessments, 145 (24%) were classified as not very low risk by MD staff. Of these, 36 children had a CT scan performed, but none had a clinically important TBI. Because of missing predictor data, we were unable to evaluate the PECARN risk group for 28 children (2% of study patients). Of these, 8 children had an incomplete RN assessment, 11 had an incomplete MD assessment, and 1 had both incomplete RN and MD assessments.
The overall agreement for the paired RN-MD assessments (ie, very low risk or not) disagreed more than expected by chance alone in both patient age groups (P = .03 for age <2 years and P = .002 for age 2–18 years). The overall PECARN TBI prediction rule agreement was higher for the older age group (κ 0.55, 95% CI 0.49–0.61 for children 2–18 years of age versus κ 0.32, 95% CI 0.23–0.41 for children <2 years of age). For children 2 to 18 years old, there was almost perfect agreement for the presence of vomiting, and there was substantial agreement regarding a history of any loss of consciousness. There was moderate agreement for altered mental status, whereas there was only fair agreement for severe injury mechanism, signs of basilar skull fracture, and severe headache (Table 4). For children <2 years old, there was moderate agreement for nonfrontal hematoma and loss of consciousness for more than 5 seconds, and there was fair agreement for altered mental status, severe mechanism of injury, and not acting normally per parents. There was only slight agreement over the predictor palpable skull fracture (Table 5). Although the overall RN-MD agreement was fair to moderate, all of the children with a clinically important TBI were classified as not very low risk by both provider groups, minimizing the clinical significance of this disagreement.
Finally, we investigated whether the agreement between the paired RN and MD assessments of the PECARN TBI rules differed by individual provider experience level. After adjustment for clinician experience and patient age, as well as clustering by individual provider, the overall agreement for the PECARN TBI rules between RN and MD providers was similar (unadjusted odds ratio 5.6, 95% CI 2.4–7.7 versus adjusted odds ratio 5.8, 95% CI 3.6–8.1).
The PECARN TBI prediction rules accurately identify children at very low risk for clinically important TBIs after minor blunt head trauma.8 Effective prediction rule implementation strategies for these decision-enhancing tools will depend on application early in the ED visit. In our study conducted at a single tertiary-care pediatric ED, we compared RNs with MDs in patient assessment of the PECARN age-based TBI clinical predictors. The overall RN-MD agreement was moderate for children 2 to 18 years and fair for children <2 years of age. The overall level of agreement did not differ after adjusting for clinician and patient factors. Given the measured agreement in the application of the PECARN TBI rules as well as the potential for harm from an unnecessary CT scan (ie, later lethal malignancy from ionizing radiation), we caution against CT decision-making before MD evaluation.
Patient history and physical examination factors selected for inclusion in clinical prediction rules must be both reproducible and reliable.1 Recent investigators in the pediatric ED have measured MD-MD interrater agreement for potential predictors in a variety of conditions.9,14,15 Across these studies, the agreement beyond chance is often only fair to moderate, particularly for physical examination findings. For clinical prediction rule development, only candidate predictors with high MD-MD reliability should be evaluated for inclusion into the rules.9,14,15 In our study, we measured the agreement between the ED RN and MD in the assessment of the predictors included in the published PECARN TBI prediction rules. Similar to previous work measuring the RN-MD agreement in the evaluation of the Ottawa ankle rules in adult ED patients,5–7 we found fair to moderate agreement between RN and MD evaluation of the validated prediction rule. Although we did not provide decision-support to the treating clinicians in this study, none of the children with discordant RN and MD assessments (very low risk by the RN but not very low risk by the MD or not very low risk by the RN but very low risk by the MD) had a clinically important TBI.
In our study, percent agreement was high for all PECARN TBI predictors across both age groups. However, the κ statistic, which quantifies the agreement above chance alone, varied considerably between predictors. Similar to the previous study of MD-MD κ statistics for the TBI predictors, objective predictors (eg, vomiting) had the highest κ statistic. Rare predictors (eg, palpable skull fractures or basilar skull fracture) as well as composite predictors (eg, altered mental status or severe injury mechanism) had lower κ statistics. Although the study forms included predictor definitions, application of uncommon or composite variables can be more challenging for the clinician. In the creation of future clinical prediction rules, one should try to avoid the inclusion of composite or rare variables, which may have lower interrater agreement.
Our study has several limitations. First, the study was conducted at a single institution, which reduces the generalizability of our findings. As the study staff have extensive pediatric emergency medicine training and experience, the agreement might be lower in settings with RNs and MDs with less expertise (eg, at general EDs without dedicated pediatric providers). Additionally, even experienced pediatric providers had better agreement in older children (>2 years of age) than the youngest children (those <2 years of age). Second, clinical signs and symptoms in children with minor blunt head trauma may at times have changed between the RN and MD patient assessments. However, the short time between assessments (median 12 minutes) minimized the risk of this phenomenon. Last, our sample size limited our ability to accurately and precisely measure RN-MD agreement for rare predictors (eg, palpable skull fracture in children <2 years old and signs of basilar skull fracture in children 2 to 18 years old). However, we adequately powered our study for our primary goal, to measure overall RN-MD agreement in the assessment of the PECARN TBI prediction rules.
Effective implementation of the PECARN TBI rules may depend on early assessment and application. As the ED RN is the first to evaluate children with minor blunt head trauma, initial RN assessments will likely be an important component of this strategy. In our study conducted at a children’s hospital, we demonstrated fair to moderate agreement between RN and MD providers in the evaluation of the PECARN TBI clinical prediction rules. At this time, effective implementation strategies for these rules require MD verification of RN predictor assessments before CT decision-making. Alternately, targeted educational interventions for providers using the PECARN prediction rules may improve the agreement between RN and MD assessments. Future clinical prediction rules should be developed using variables with good interrater reliability between all types of clinicians who are likely to be applying the resulting rule.
We recognize the research coordinators who helped with study enrollment: John Andrea, BA (Boston, MA), Mira Henien, BA (Boston, MA), Brittany Kronick, BA (New York, NY), Kaitlin Morris, BA (Boston, MA), Jill Winnett, BA (Boston, MA), and Sandy Wong, BS (Boston, MA). We also thank the RN and MD clinicians who completed study forms.
- Accepted June 24, 2013.
- Address correspondence to Lise E. Nigrovic, MD, MPH, Boston Children’s Hospital, 300 Longwood Ave, Boston, MA 02115. E-mail:
Dr Nigrovic conceived and designed the study, interpreted the data, and drafted the manuscript; Dr Schonfeld acquired the data, interpreted the data, and critically reviewed the manuscript; Dr Dayan interpreted the data and critically reviewed the manuscript; Ms Fitz acquired the data and critically reviewed the manuscript; and Dr Kuppermann conceived and designed the study, interpreted the data, and critically reviewed the manuscript.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Funded by a Hood Foundation, Child Health Grant to Dr Nigrovic.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Kuppermann N,
- Holmes JF,
- Dayan PS,
- et al.,
- Pediatric Emergency Care Applied Research Network (PECARN)
- Dayan PS,
- Lillis K,
- Bennett J,
- et al.,
- Pediatric Emergency Department Northeast Team of the Pediatric Emergency Care Applied Research Network (PECARN)
- Kharbanda AB,
- Stevenson MD,
- Macias CG,
- et al.,
- Pediatric Emergency Medicine Collaborative Research Committee of the American Academy of Pediatrics
- Copyright © 2013 by the American Academy of Pediatrics