Validation of a Score to Determine Time to Postconcussive Recovery
BACKGROUND: A reliable, developmentally appropriate and standardized method for assessing postconcussive symptoms (PCS) is essential to accurately determine recovery postconcussion and to effectively manage return to normal activities. The aim of this study was to develop an evidence-based, psychometrically validated approach to determining clinically useful cutoff scores by using a commonly administered PCS measure.
METHODS: The current study was a prospective, longitudinal observational study conducted between July 2013 and November of 2015 at a statewide tertiary pediatric hospital. Participants were 120 children (5–18 years of age) presenting to the emergency department with a concussion within 48 hours of injury. PCS were assessed by using the Postconcussion Symptom Inventory (PCSI), acutely, 1 to 4 days postinjury and 2 weeks postinjury. Using comprehensive clinical assessment as gold standard, we assessed the clinical cutoff discrimination ability of PCSI at 2 weeks postinjury by using published approaches, and then varying each approach to optimize their discrimination ability.
RESULTS: Existing and potential clinical cutoff scores were explored in predicting delayed recovery. Receiver operating characteristic curve results returned acceptable discrimination and sensitivity when PCSI items increased in severity from preinjury by 1 or more. Compared with a published cutoff score being 3+ items with increased severity, the current study suggests a more stringent cutoff requirement of 2+ is better able to accurately classify symptomatic children.
CONCLUSIONS: This study provides the first validated index (2+ items, 1+ severity) of concussion recovery for children and youth. Further studies in more varied samples are needed to establish the effectiveness of this method.
- AUC —
- area under the curve
- CT —
- computed tomography
- ED —
- emergency department
- ICD-10 —
- International Classification of Diseases, 10th Revision
- OR —
- odds ratio
- PCS —
- postconcussive symptoms
- PCSI —
- Postconcussion Symptom Inventory
- RA —
- research assistant
- RCH —
- Royal Children’s Hospital
- ROC —
- receiver operating characteristic
What’s Known on This Subject:
To accurately determine recovery postconcussion, and to effectively manage return to normal activities, a standardized method for assessing postconcussive symptoms is essential. This includes application of validated, developmentally appropriate clinical assessment tools; reliable across multiple time points.
What This Study Adds:
This study provides the first validated index of concussion recovery for children and youth, which affords (1) a consistent, quantitative approach to determining recovery from concussion and (2) more accurate clinical decisions regarding symptom resolution and return to normal activities.
Concussion, or mild traumatic brain injury, accounts for up to 90% of all traumatic brain injuries.1 Although the majority of children recover quickly and spontaneously after concussion,2 some studies reveal a significant percentage experience persistent postconcussive symptoms (PCSs) beyond 2 weeks.3 PCSs are somatic, cognitive, emotional, and fatigue complaints that in children include headache, dizziness, insomnia, irritability, difficulty concentrating, mood disturbance, and emotional lability. Recently, studies have attempted to identify clinical predictors of persisting PCSs (eg, previous head injuries).2,4–10 Findings are inconsistent, likely reflecting differences in definitions of concussion and lack of clarity regarding what constitutes persisting symptoms, as well as use of heterogeneous and insensitive outcome measures, variations in follow-up time points, and inconsistencies in determination of symptom resolution. A recent study by Zemek et al11 examined predictors of PCSs in the largest pediatric concussion cohort to date. The authors developed and validated a clinical risk score (derived from demographic and clinical measures) to predict PCSs at 28 days postinjury for children and adolescents presenting to the emergency department (ED) within 48 hours of a concussion. Although the clinical risk score was significantly better than clinician judgment at predicting PCSs at 28 days postinjury, the overall discrimination ability of the risk score was modest.11 The findings from this study highlight the multitude of noninjury-related factors that can contribute to the development of PCSs (eg, psychosocial factors) and the need for evidence-based and psychometrically validated measures to assess PCSs.
To accurately determine recovery postconcussion, and to effectively manage return to normal daily activities, a standardized method for assessing PCSs is essential. In a pediatric setting, this includes application of validated, developmentally appropriate clinical assessment tools, reliable across multiple time points. A review of the literature finds that the current clinical guidelines, mostly adult based, recommend that return to daily activities should be determined by a clinician, and not occur before symptom resolution, or within 48 hours of the injury.12,13 In practice, decision-making varies widely, depending on the practitioners’ experience and knowledge of concussion.14 Further, PCS data are usually determined from self-report (or parent report for young children) via questionnaires and rating scales such as the Postconcussion Symptom Inventory (PCSI)15,16 or the Health and Behavior Inventory.17 Although many of these scales have strong psychometric properties,6,17,18 few provide clinical cutoff scores (binary classifiers), or guidance regarding how ratings might be used in concussion management.
To address this issue, 3 recent studies using the PCSI have included clinical cutoff points—albeit consistent with International Classification of Diseases, 10th Revision (ICD-10) definitions—not yet validated by data. Zemek et al19 classified clinically significant PCSs as 3 or more individual items on the PCSI increasing by 1 point or more from preinjury to time 2 (1 month). In contrast, Smyth et al20 defined clinical significance as an increase of 1 or more individual PCSI items by 2 points or more from preinjury to time 2 (7–10 days), whereas Barlow et al,21 using the same database as Smyth et al,20 included the requirement of an increase of at least 1 in the singular “acting differently” item (1 month post injury). Although all 3 approaches are informative, they lack psychometric rigor and exclude clinically important information (ie, comprehensive clinician assessment) that is the routine clinical indicator for determining timing of symptom resolution and return to normal activities.
The aim of this study was to develop a robust, evidence-based, psychometrically validated approach to determining clinically useful cutoff scores by using the commonly administered PCSI. This will inform clinical management and return to school and play decisions for children and adolescents. After comprehensive clinical assessments by clinicians experienced in concussion management in an independent sample of children with concussion,22 we assessed binary classifier discrimination ability of PCSI first by using published approaches, and then varying each approach in an attempt to optimize their discrimination ability.
This was a single site, prospective, longitudinal observational study conducted at a statewide tertiary pediatric hospital. Full study details are provided in Bressan et al.22 This study was approved by the Royal Children’s Hospital (RCH) Human Research Ethics Committee, study ID number 33122.
Participants were children aged 5 to 18 years presenting to the ED at the RCH (Melbourne) having sustained a concussion within 48 hours. Concussion was defined according to the Zurich Consensus Statement on Concussion in Sport.12
Injury-related exclusion criteria included the following: Glasgow Coma Scale <13, abnormal computed tomography (CT) scan, multiple injuries, neurosurgical intervention, general anesthesia, intentional injury, and/or no clear history of trauma as the primary event (eg, seizure).
We employed a 2-week time frame to classify delayed recovery instead of the 1-month (ICD-10) or 3-month (Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition) cutoffs frequently used in the adult literature. The ICD-10 and Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition time points were developed for adults and thus not necessarily applicable to a pediatric sample. Further, considering the mean age of our sample, (∼11 years old) and the important cognitive, physical, emotional, and educational development occurring at this age, we determined that utilizing an earlier time point for delayed recovery would enable earlier clinical intervention, which is important to mitigate the harms of ongoing symptoms. With this 2-week time point, 19% of our sample was classified as “delayed in recovery” on the basis of comprehensive clinical assessment.
Children’s PCSs were rated by parents using the parent report form of the PCSI.16 The parent form of the PCSI consists of 20 items within 4 subscales for physical, cognitive, emotional, and fatigue PCSs. Symptoms are rated for preinjury and current symptoms (yesterday and today) on a 7-point Guttman scale (0 = not a problem, 3 = moderate problem, 6 = severe problem). The parent report also includes a final, stand-alone item (not included in scale derivation): “In general, to what degree is your child acting ‘differently’ than before the injury (not acting like himself or herself)?”, rated on a 5-point Likert scale (from 1 “no difference” to 5 “major difference”). The PCSI has previously shown good internal consistency (α = 0.8–0.9), acceptable to good test–retest reliability (intraclass correlation coefficient = 0.65–0.89) and good convergent validity (r = 0.8).18
RCH ED presentations were monitored by research staff (including evenings and weekends). Eligible families were identified and approached in the ED by trained research assistants (RAs) and informed consent obtained from parents and children. Treating doctors provided injury-related information in the ED. Participating children and parents were assessed in the Concussion Clinic 1 to 4 days, 2 weeks, and 1 month postinjury by an RA and clinician. Demographic information and baseline PCSs were reported by parents at the first clinic appointment, 1 to 4 days postinjury. Current PCS was reported by parents at the 2-week follow-up appointment.
At each clinic appointment, a clinician experienced in pediatric concussion diagnosis and management (eg, rehabilitation pediatrician, clinical neuropsychologist) assessed the child and provided standard psychoeducation related to concussion management. During the clinical interview at 2-week follow-up, a clinician assessed the participant by using standardized clinical tools and a structured interview addressing significant medical history, symptoms (eg, headaches, balance difficulties), and patient/parent reports of school performance, family functioning, physical activities/sport, and general well being. For a full description of the concussion clinic assessment, see Bressan et al.22 In brief, the assessments involved structured clinical measures of concussion symptoms, parent and child mental health, quality of life, fatigue, posttraumatic stress, family burden, a computerized neurocognitive assessment, and the Sports Concussion Assessment Tool, Third Edition (SCAT3 or Child SCAT3). A participant was categorized as symptomatic if they were experiencing continuing difficulties (eg, headaches, fatigue) that were interfering with everyday activities (eg, school performance). A participant was defined as functionally recovered if they were clinically assessed as having returned back to normal or near normal (eg, occasional headaches that did not interfere with normal activities) at the 2-week time point. For reliability purposes, these assessments were later independently reassessed by a second study clinician, blind to the original assessment.
Demographic data, injury data, and PCSI outcomes were summarized by using descriptive data, and compared between symptomatic and recovered groups, and between the analytic sample and those lost to follow-up. Independent samples t tests were conducted to compare variables and, due to right skew, Wilcoxon rank sum tests were used for PCSI items. Fisher’s exact test for categorical variables was used to account for low cell counts. To establish the reliability of the clinician symptomatic/recovered diagnosis, interrater reliability was calculated between the independent clinician assessments (Cohen’s κ) on a random subgroup of the sample (20 from the initial symptomatic and recovered groups). Diagnostic cutoffs for symptomatic/recovered include 3 published methods (see Table 1):
Zemek et al19: An increase in severity in 3 or more individual items by 1 point or more from preinjury to time 2,
Smyth et al20: An increase in severity in 1 or more individual items by 2 points or more from preinjury to time 2, and
Barlow et al21: An increase in severity in 1 or more individual items by 2 points or more from preinjury to time 2 and an increase of 1 or more in the singular acting differently item.
Zemek et al,19 Smyth et al,20 and Barlow et al21 binary classifiers were calculated and compared. To explore identification performance of the classifiers, summary diagnostic statistics were explored. Sensitivity, specificity, positive and negative predictive value, and odds ratios (ORs) were reported to determine the predictive value of each cutoff in relation to the clinical diagnosis. Receiver operating characteristic (ROC) curves were employed to illustrate classifier performance, and area under the curve (AUC) was used to determine overall cutoff discrimination ability, using the cutoff of 0.7 for acceptable discrimination.23 ROC and AUC also explored continuous PCSI measures, including total score, as well as change score and number of increased items (similar to the Zemek et al19 cutoff method), both from preinjury to time 2. Finally, the Zemek et al,19 Smyth et al,20 and Barlow et al21 classifiers were tested by using cutoff scores of 1+ through 5+ increased items (where Zemek et al19 require any score increase, and Smyth et al20 require an increase of 2+, and Barlow et al21 require an increase in 2+ as well as an increase in 1+ in the singular acting differently item). It should be noted that due to the nature of the injury, more emphasis will be placed on sensitivity (true positives) than specificity (true negatives).
Of those 1205 children eligible to participate, 203 children were enrolled in the study. Recruitment and attrition are presented in Fig 1. The final sample consisted of 120 cases, recruited to the Take Care study, who were clinically diagnosed symptomatic or recovered 2 weeks postconcussion, with 23 (19.2%) participants classified as symptomatic by ED or rehabilitation clinicians. Analysis of demographic and injury-related information revealed no significant difference between symptomatic/recovered groups (see Table 2), nor between the 120 analytic sample and the 83 lost to follow-up. Significant differences were identified, however, for total, change, and number of increased items scores, as well as the singular acting differently item in the PCSI (all P < .001), with the symptomatic group reporting more adverse outcomes.
An analysis of demographic variables was also conducted between the analytic sample and those lost following enrollment (data not presented). No significant differences were found.
Interrater reliability of clinician assessment of symptomatic/asymptomatic at time 2 indicated almost perfect agreement, with a high value κ = 0.90. Thirty-eight of the random 40 (95%) assessed for reliability were consistent between the initial and later assessments, and the remaining 2 inconsistent cases were initially rated as symptomatic.
Published Cutoff Methods
Diagnostic summary statistics of the PCSI binary classifiers and continuous scales can be seen in Table 2. Proportions of participants rated as symptomatic based solely on previously published PCSI cutoffs were similar but much higher than that of the clinical diagnosis (19%). The Zemek et al19 classifier was the highest with 47%, and returned the best true positive rate (sensitivity = 78.3%). All 3 cutoffs were significantly better than chance, with the AUC being 0.69 for the Smyth et al,20 0.70 for the Zemek et al,19 and 0.72 for the Barlow et al21 classifiers, the latter 2 meeting the statistical AUC = 0.7 cutoff. The Barlow et al21 classifier also outperformed the other 2 methods in OR, with a higher diagnostic OR (OR = 7.0) than the Zemek et al19 and Smyth et al20 (OR = 5.6 and 5.3, respectively) methods. Compared with binary classifiers, continuous PCSI measures returned slightly higher AUC values; the best of which was the “number of increased items” scale (AUC = 0.77).
Proposed Cutoff methods
The Zemek et al,19 Smyth et al,20 and Barlow et al21 clinical classifiers were extended to examine the cutoff being determined over a range of increased items (see Figs 2 and 3). That is, ROC analyses were carried out by using a cutoff of 1+ through 5+ severity increased items from preinjury to time 2 (with the Zemek et al19 classifier being 3+ with any increase, Smyth et al20 being 1+ of an increase of 2+, and Barlow et al21 being 1+ of an increase of 2+ as well as an increase of 1+ in the singular acting differently item). Table 3 shows the diagnostic summary statistics for these 15 cutoffs. All Smyth et al20 cutoff AUC values were below 0.7. The Barlow et al21 cutoffs of 1+ and 2+ returned favorable AUC values (0.72 and 0.71), but were led by specificity rather than sensitivity. As such, Smyth et al20 and Barlow et al21 cutoffs were not further considered.
In comparison with the published 3+ cutoff, the Zemek et al19 2+ binary classifier returned a far superior sensitivity, correctly classifying 22 of the 23 symptomatic participants (sensitivity = 95.7). The same sensitivity was found in the 1+ classifier, but with reduced specificity (39.2 vs 54.6). The 2+ classifier also returned the highest positive OR (26.5).
The current study aimed to develop a psychometrically validated approach to determining clinically useful cutoff scores for the parent-reported PCSI, to (1) provide a validated approach to determining recovery from concussion and (2) inform clinical decisions regarding symptom resolution and return to school and play for concussed children and adolescents. Based on the Take Care sample, a modified version of the Zemek et al19 PCSI binary classifier (ie, 2 or more individual items increased in severity by 1 point or more from preinjury to time 2) returned the most psychometrically robust results to distinguish between symptomatic and recovered participants postconcussion. Specifically, the Zemek et al19 method was found to better discriminate symptomatic participants than the Smyth et al20 or Barlow et al21 classifiers, and within the Zemek et al19 classifier a more stringent cutoff of 2 more severe items (as opposed to 3) returned greater accuracy in predicting symptomatic participants (sensitivity). To our knowledge, this is the first study to use a data validated approach to determine a robust clinical cutoff score for the PCSI.
There is considerable clinical utility for a data validated clinical classifier for the PCSI for concussion management. Concussion is a clinical diagnosis, but symptom assessment postinjury is subjective and can be challenging as many of the most common symptoms are nonspecific and frequently endorsed by noninjured children and adolescents (eg, headaches, fatigue). A psychometrically validated cutoff score for the PCSI that accounts for baseline symptoms may assist pediatric clinicians, particularly those with limited concussion experience, with ongoing concussion management and provide a method for assessing recovery.
Establishing a data validated clinical cutoff score for the PCSI also has utility for advancing pediatric concussion research. Currently, substantial heterogeneity in methodology, measures, and definitions of clinically significant symptoms make drawing conclusions about recovery difficult and has precluded metaanalyses.24 The PCSI is a widely used, developmentally appropriate measure of PCS, and as such, developing an accurate standardized classifier for clinically significant symptoms on the PCSI facilitates meaningful comparisons across studies. This study represents the first step toward this goal.
Although this study is the first to take a data validated approach to determining clinical classifiers for the PCSI, there are limitations. The “gold standard” to which the PCSI was compared was clinical judgment, which is open to human error. However, interrater reliability between multiple pediatric clinicians experienced in the diagnosis and management of concussion, and across disciplines, was κ = 0.9, indicating high reliability. Also, the samples described were all recruited at a single center through an ED, thus findings may not generalize to primary care or acute outpatient presentations. Finally, it is important to highlight that of the 790 eligible participants, our analysis included 120, raising the possibility of bias and potentially limiting the generalizability of our conclusions. Although we acknowledge that this is less than optimal, it is typical of ED recruitment rates. Obtaining a truly representative sample in pediatric concussion is challenging because a substantial portion of children do not seek medical care (likely on the milder end of severity) and much of the published research includes samples attending follow-up clinics; likely to be those with ongoing symptoms (ie, more severe concussions). An advantage of the current study is the presentation of detailed recruitment data (Fig 1); the reader is informed and can draw their own conclusions.
There are a number of important clinical implications emerging from these findings. In the acute stages of recovery, concussion management typically involves physical and cognitive rest until the acute symptoms resolve and then a graded return to school and sport activities.12 However, in the pediatric literature, there is increasing evidence that extended periods of rest can exacerbate and prolong postconcussion symptoms25 and growing consensus that a return to normal activities postacute symptom resolution can improve outcomes.26 Thus accurate decisions regarding symptom resolution are of critical importance. The absence of evidence-driven gold standards for assessing recovery, either qualitatively, or quantitatively, on a case-by-case basis has limited the ability to make informed recommendations for rest duration, or for return to daily activities. This study provides an example of a validated approach to determining clinical cutoff scores for the parent-reported PCSI, and thus informing clinical practice. For example, a validated clinical cutoff score for the PCSI can be used by clinicians to more effectively manage a child’s return to normal activities (eg, school, sport) by accurately assessing their current symptom burden. Further studies in more varied samples are needed to establish the effectiveness of this and other clinical binary classifier methods.
This study provides the first validated index of concussion recovery for children and youth. Compared with other published binary classifiers, which require 3 or more items with increased severity, the current study suggests a more stringent cutoff requirement of 2 or more is better able to accurately classify symptomatic children. The availability of a psychometrically validated approach to determining clinically useful cutoff scores provides the opportunity for (1) a consistent, quantitative approach to determining recovery from concussion and (2) more accurate clinical decisions regarding symptom resolution and return to school and play for concussed children and adolescents.
- Accepted November 11, 2016.
- Address correspondence to Franz E. Babl, MD, Emergency Medicine Research, Level 4 West, Murdoch Childrens Research Institute, 50 Flemington Rd, Parkville, Victoria, 3012 Australia. E-mail:
This trial has been registered with the Australian New Zealand Clinical Trials Registry (identifier ACTRN12615000316505).
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This study was funded by the Royal Children’s Hospital Research Foundation and the Victorian Government Operational Infrastructure Scheme. Mr Hearps was funded by an Australian National Health and Medical Research Council (NHMRC) Development grant, Dr Babl was funded by the Royal Children’s Hospital Research Foundation, and Dr Anderson by an NHMRC Senior Practitioner Fellowship. The funding organizations did not have a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Eisenberg MA,
- Andrea J,
- Meehan W,
- Mannix R
- Eisenberg MA,
- Meehan WP III,
- Mannix R
- Castile L,
- Collins CL,
- McIlvain NM,
- Comstock RD
- McCrory P,
- Meeuwisse WH,
- Aubry M, et al
- Giza CC,
- Kutcher JS,
- Ashwal S, et al
- Gioia GA,
- Janusz J,
- Isquith P,
- Vincent D
- Gioia GA,
- Schneider JC,
- Vaughan CG,
- Isquith PK
- Sady MD,
- Vaughan CG,
- Gioia GA
- Zemek R,
- Osmond MH,
- Barrowman N; Pediatric Emergency Research Canada (PERC) Concussion Team
- Hosmer D,
- Lemeshow S
- Thomas DG,
- Apps JN,
- Hoffmann RG,
- McCrea M,
- Hammeke T
- Gioia GA
- Copyright © 2017 by the American Academy of Pediatrics