OBJECTIVES: The goals were to develop and to validate the Internet-based, Behavioral Health Screen (BHS) for adolescents and young adults in primary care.
METHODS: Items assessing risk behaviors and psychiatric symptoms were built into a Internet-based platform with broad functionality. Practicality and acceptability were examined with 24 patients. For psychometric validation, 415 adolescents completed the BHS and well-established rating scales. Participants recruited from primary care waiting rooms were 12 to 21 years of age (mean: 15.8 years); 66.5% were female and 77.5% black.
RESULTS: The BHS screens in 13 domains by using 54 required items and 39 follow-up items. The administration time was 8 to 15 minutes (mean: 12.4 minutes). The scales are unidimensional, are internally consistent (Cronbach's α = 0.75–0.87), and discriminate among adolescents with a range of diagnostic syndromes. Sensitivity and specificity were high, with overall accuracy ranging from 78% to 85%. Patients with scores above scale cutoff values for depression, suicide risk, anxiety, and posttraumatic stress disorder symptoms were ≥4 times more likely to endorse other risk behaviors or stressors.
CONCLUSIONS: The BHS addresses practical and clinical barriers to behavioral health screening in primary care. It is a brief but comprehensive, self-report, biopsychosocial assessment. The psychiatric scales are valid and predictive of risk behaviors, which facilitates exclusion of false-positive results, as well as assessment and triage.
WHAT'S KNOWN ON THIS SUBJECT:
The need for screening in primary care is a national priority, but few validated screening tools are available for this setting.
WHAT THIS STUDY ADDS:
The BHS is a validated screening tool designed specifically for primary care. It goes beyond most depression screening tools by offering a full psychosocial assessment on a self-report, Internet-based system that scores the data for busy clinicians.
A call for behavioral health screening in primary care has been renewed.1,2 In particular, the American Academy of Child and Adolescent Psychiatry and the American Academy of Pediatrics,3 as well as the US Preventive Services Task Force,4 have recommended universal screening for depression. In addition, the Joint Commission now requires suicide screening for youths with psychiatric diagnoses in all inpatient and hospital-based, ambulatory services and recommends assessment for other behavioral health problems (eg, child abuse and substance use).5 Despite decades of interest,6 the integration of behavioral and medical services has been limited,7,–,10 particularly in pediatric and adolescent medicine.11,–,14 Barriers to integration include lack of physician training, inadequate screening tools, minimal reimbursement for behavioral health assessment and treatment, and difficulty accessing behavioral health services.3,4
Comprehensive models to address these deficits have been proposed,15,16 but progress has been stalled in part by the lack of practical, valid, behavioral health screening tools. Self-reported indicated or universal screening increases case identification rates, standardizes assessment questions, and increases staff efficiency.17,–,19 Furthermore, most studies indicate that adolescents prefer revealing personal information on computers, rather than in face-to-face interviews.20
Unfortunately, most screening tools do not respond to the full breadth of behavioral health screening needs in primary care settings. First, many tools assess only 1 domain (eg, depression), but assessments of multiple domains (eg, depression, suicide risk, and child abuse) are recommended and needed to put depression in the context of other risk and protective factors.21 Single-domain screeners also fail to provide information for best-practice recommendations regarding comprehensive psychosocial assessments at well-child visits.21 Second, most rating scales ask about current or past-week symptoms, but patient contact often is yearly (eg, well-child visits) and therefore a broader assessment time frame is required. Third, most screening tools lack psychometric validation. In a review of depression screening studies, only 10 of 25 studies provided at least some psychometric data on the screening tool implemented.17 Fourth, most screening tools fail to use advanced technology to reduce patient and provider burdens.18 For patients, interactive computer administration tailors the task to individuals (eg, allows for skip patterns), reduces literacy obstacles (eg, through audio presentation of items), and is a preferred modality for a young, computer-literate population.20,22 For providers, computer screening reduces the burden of administration, scoring, and report-writing. Furthermore, Internet-based systems can interface with electronic medical records, track patient progress over time, and aggregate data at patient, practice, or broader system levels, providing the foundation for an evolving “learning health care system.”23
The Behavioral Health Screen (BHS) was designed to address many of the limitations outlined above. The BHS is an Internet-based, comprehensive (although brief) screening tool for youths.12,–,24 Here we describe its development, practicality, acceptability, psychometric properties, and clinical utility.
Development of BHS
Initial Item Development
The BHS was drafted by a team of psychologists, pediatricians, and adolescent medicine physicians, through a comprehensive review of best-practice guidelines, existing screening tools, behavioral health and risk behavior measures, and psychiatric diagnostic criteria. Behavioral health items were designed on the basis of Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, criteria for major depression, generalized anxiety disorder, posttraumatic stress disorder (PTSD), anorexia nervosa, bulimia nervosa, schizophrenia, and substance use disorders.24 Risk-assessment modules were developed to address school, family, safety, sexuality, and suicide risks. Items were reviewed by 20 national experts in domain-specific areas. Focus groups with pediatricians helped tailor the BHS specifically to primary care (eg, asking about current and past-year symptoms).
Study 1: Practicality and Acceptability
Twenty-four participants were recruited, provided consent, and underwent administration of the BHS before their medical, well-child visit appointments, and results were provided to their health care providers for use during the appointment. A brief satisfaction questionnaire was given to patients and providers after the appointment. Patients were paid $30 for their participation. All procedures were approved by the Children's Hospital of Philadelphia institutional review board.
Study 2: Validation
The study subjects were recruited from primary care offices in the Philadelphia, Pennsylvania, area over 18 months. Parents' written, informed consent and adolescents' assent were obtained for participants 12 to 13 years of age. In accordance with Pennsylvania law and institutional review board approval, adolescents ≥14 years of age consented for themselves. Subjects were assigned randomly to completion of the BHS (8–15 minutes) and then the validation battery (30–45 minutes) or vice versa. All measures were administered on a computer. Subjects in need of behavioral health services were referred appropriately. Subjects were paid $20 for their participation. All procedures were approved by the Children's Hospital of Philadelphia institutional review board and were conducted by research staff members not associated with development of the BHS.
The BHS is composed of 13 modules, namely, demographic, medical, school, family, safety, substance use, sexuality, nutrition and eating, anxiety, depression, suicide risk, psychosis, and trauma and abuse (Table 1). There are 54 core items and 39 additional items to probe more deeply when relevant core items are endorsed. After completion, the data are scored and a report is printed immediately at a location of the provider's choice (eg, nursing station). A summary page displays scaled scores for depression, anxiety, suicide risk, and PTSD symptoms, as well as urgent risk items (eg, suicidal ideation/attempts or sexual abuse), nonurgent risk items (eg, substance use or self-induced vomiting), and patient strengths (eg, supportive parents or school success).
Beck Depression Inventory II
The Beck Depression Inventory II (BDI-II) is a widely used, 21-item, self-report instrument designed to assess the severity of depressive symptoms.25 Results are strongly correlated with other measures of depression26,27 and have high internal consistency. In this sample, Cronbach's α (internal consistency) was 0.91.
Scale for Suicidal Ideation
The Scale for Suicidal Ideation (SSI) is a 19-item, interviewer-administered, rating scale that measures patients' suicidal ideation in the past week and during their most-severe suicidal episode.28 Results have high internal consistency (SSI for past week, α = 0.84; SSI for worst-ever episode, α = 0.89) and are strongly correlated with suicide items on other measures. In this sample, Cronbach's α for the past week was 0.89 and that for the most-severe suicidal episode was 0.92.
Trauma Symptom Checklist for Children
The Trauma Symptom Checklist for Children (TSCC) is a 54-item, self-report measure that contains 6 clinical subscales,29 although only the posttraumatic stress and anxiety subscales were used in this analysis. TSCC results are strongly correlated with several well-established measures and aspects of child abuse. Cronbach's α values in this sample were 0.83 for PTSD and 0.77 for anxiety.
Statistical and Psychometric Analyses
Responses to all BHS items were coded on a scale of 0, 1, or 2, with 0 indicating no symptoms, 1 moderate symptoms, and 2 severe symptoms. Analyses were conducted to evaluate item and scale properties by using both traditional (ie, classical) and modern (ie, item response theory) psychometric methods. We evaluated the unidimensionality of scales through 1-factor confirmatory factor analyses with multiple fit statistics that provide distinct, complementary information about model fit (criteria were a comparative fit index of ≥0.95, a Tucker-Lewis index of ≥0.95, and a root mean square error of approximation of ≤0.08).30 Local independence was evaluated by examining residual correlations among items in the 1-factor models, and the internal consistency of scales was determined by using Cronbach's α. After item response theory assumptions were confirmed, Rasch-Masters partial credit models were fit to the data, and model and item fits were determined by using Winsteps (Available at: www.winsteps.com/index.htm.). Item properties were evaluated through inspection of infit and outfit statistics, item discrimination parameters, item difficulty, and category threshold estimates. Results of the psychometric analyses were used to revise the BHS scales, and scale scores were calculated by averaging responses for the constituent items (Table 2).
The instrument's convergent validity and divergent validity were evaluated through inspection of bivariate correlations among the BHS and validation scale scores (Table 3). Finally, sensitivity and specificity analyses were used to establish cutoff scores on the BHS scales that maximized both true-positive and true-negative classifications of clinically significant behavioral health problems, as indicated by scores on the validation measures. The sensitivity and specificity of the BHS scales were evaluated by using receiver operating characteristic analyses at different cutoff values. Cutoff scores that maximized both true-positive and true-negative classifications of clinically significant behavioral health problems, as indicated by scores for the validation measures, were identified for each BHS scale.
Finally, for evaluation of the clinical utility of the BHS, adolescents were classified as having results above or below clinically significant cutoff points on the BHS scales. Odds ratios were calculated to compare adolescents with clinically significant behavioral health symptoms and those with values below clinical cutoff values, with respect to a number of social and behavioral risk indicators (eg, substance abuse, risky sexual behavior, violent victimization, disordered eating, school failure, and family disengagement) (Table 4).
To power exploratory factor analysis with an 83-item measure, we needed 415 subjects. This sample size is adequate to power item response theory analyses with unidimensional scales. For concurrent and discriminant validity, 314 subjects were needed to detect r = 0.20 (a small effect size), given α = β = .05.
Study 1: Practicality and Acceptability
Practicality and acceptability were evaluated with 24 adolescents (mean age: 14.9 years; female: 66%; black: 83.3%). On average, patients completed the instrument in 12.4 minutes (SD: 5.04 minutes). Patients and health care providers responded positively to the screening tool. When asked simple questions about acceptability and satisfaction, adolescents reported that (1) they liked the software (75%), (2) they understood the questions (96%), (3) they reported honestly (92% mostly or completely honestly), (4) they thought that the instrument should be used for future appointments (92%), and, most importantly, (5) they found the instrument helpful during the appointment (94% of patients whose doctors used the printout during the appointment). Participating providers stated that the instrument was helpful for facilitation and planning of the visit, as well as guiding of further questioning. Particularly for residents training in the adolescent medicine clinic, it was noted that the breadth of information was helpful.
Study 2: Validation
During the recruitment phase, 1836 eligible adolescents visited the practices, of whom 1038 were approached about participation and 839 expressed interest and provided contact information. Of those adolescents, 770 were called to schedule assessments, but 54 (7%) could not be reached, 100 (13%) were no longer interested, and 190 (24.8%) did not attend their scheduled appointments. In the final sample, 426 adolescents (55%) completed the assessment and 415 (53.9%) had usable data.
Adolescents were 12 to 21 years of age (mean: 15.8 years; SD: 2.2 years). The sample was 66.5% female, 77.5% black, 10.7% white, 9.7% Hispanic, and 2.1% of another race.
Single-factor, confirmatory factor model fit statistics supported the unidimensionality of the depression, anxiety, suicide risk, and PTSD symptom scales (Table 2). Consistent with confirmatory factor analysis results, all scales had adequate internal consistency (range: 0.75–0.87). One-parameter Rasch models fit the data for each subscale adequately, as indicated by infit/outfit statistics (between 0.70 and 1.40) and posthoc estimated discrimination parameters (a ≥ 0.75). Item location (b) parameters indicated that BHS items assessed a full range of depression, suicide risk, anxiety, and PTSD symptoms, with minimal floor/ceiling effects and redundancy among items.
Convergent and Divergent Validity
Bivariate correlations among the BHS and validation scale scores are presented in Table 2. The convergent validity of the psychiatric symptom scales is supported by positive, significant correlations between the BHS depression scale and the BDI-II total (r = 0.66; P < .0001), between the BHS anxiety scale and the anxiety subscale of the TSCC (r = 0.68; P < .0001), between the BHS suicide risk scale and the SSI total (r = 0.72; P < .0001), and between the BHS trauma scale and the TSCC (r = 0.48; P < .0001). Notably, for all subscales, relationships between the BHS scales and the associated validation measures were greater than relationships with other validation measures (Table 3).
Sensitivity and Specificity
Receiver operating characteristic curve characteristics were generally satisfactory for all BHS scales. For each subscale, a cutoff score that was most accurate in distinguishing adolescents who had clinically significant, behavioral health symptoms from those who did not was selected (Table 4). The greatest risk of misclassification was falsely identifying adolescents with subclinical anxiety symptoms as having anxiety (specificity: 67%).
Compared with adolescents who scored below the receiver operating characteristic cutoff values, youths who scored above the cutoff values for all syndromes were 3 to 7 times more likely to report arguing in the home and were 3 times more likely to endorse a nontraditional sexual orientation (except for PTSD); depressed girls were nearly 3 times more likely to have been pregnant (Table 5). In general, youths in the risk range for all scales were 4 to 7 times more likely to report physical or sexual victimization. Youths who scored high for PTSD were 14 times more likely to report assault by a romantic partner, whereas youths with depression or suicide risk were 14 times more likely to report assault by someone in the home. Youths at risk for all 4 syndromes were 2 to 6 times more likely to be using marijuana or alcohol, to meet criteria for substance use disorder, and to be involved in drug use while driving. Interestingly, youths with elevated PTSD or anxiety symptoms were 10 to 14 times more likely to report drug use to help them relax.
The national push for behavioral health screening in primary care is evident, but screening is underused.3,4 Consequently, detection rates are low.3,31 Unfortunately, multiple barriers discourage the adoption of this practice; the BHS aims to overcome some of these barriers. The BHS standardizes screening for a broad spectrum of psychiatric symptoms, urgent and nonurgent risk behaviors, and patient strengths. Many scales assess current and past-year functioning to facilitate evaluation at yearly well-child visits. The tool assesses 4 psychiatric syndromes, drug and alcohol disorder criteria, and several risk behaviors, which allows physicians to put reported problems into a broader behavioral and environmental context and helps rule out false-positive and false-negative results.
Data presented here suggest that patients and providers find the BHS acceptable, practical, and helpful. Psychometric analyses suggest that the 4 BHS scales (depression, suicide risk, anxiety, and PTSD symptoms) have strong internal consistency and the convergent validity and divergent validity are impressive. Scales discriminate from each other, and specificity and sensitivity are more than adequate. The greatest risk of misclassification is falsely identifying adolescents with subclinical anxiety symptoms as having anxiety (specificity: 67%). However, the BHS is a tool for screening, which should not be confused with diagnosis. Screening tools augment and streamline risk assessment, but clinical judgment is required to assess youths accurately.13,32
Limitations of the current study include the convenience sample and the validation measures. This was not a prevalence study, however, but was an examination of the psychometric properties of the BHS. Additional research using diagnostic tools should further validate the BHS. However, the BHS was designed not to be a diagnostic tool but to focus clinical conversations about risk.
Beyond its brevity, breadth, and valid scales, the innovation of the BHS lies in the Internet-based delivery platform. This format solves common problems with administration, interpretation, and data integration.17,33 The program automatically scores the data, generates a report, and populates forms for appropriate documentation, and it can integrate results with electronic medical records. Repeat administration would allow a practice or hospital to track patient progress. Aggregate reports at the patient, practice, or system level can support quality assurance efforts. Site-specific items also can be added. Most importantly, administering the BHS takes less effort than distributing, scoring, interpreting, and filing a single-domain, paper-and-pencil depression screen. These features reduce barriers that contribute to low rates of use of existing screening tools.
Although technology can resolve some problems, it is not a panacea. In a survey of medical providers in Pennsylvania, 80% said they would use this tool but 40% (mostly rural) lacked Internet access. Integration into the workflow also presents challenges. In an emergency department study of the BHS, use rates peaked at 33% of eligible patients.34 In 3 partially rural counties in Pennsylvania, we encountered problems with firewalls, outdated computers, and resistance to incorporating the BHS into the workflow.15 Although the Internet-based format adds unique challenges, paper-and-pencil questionnaires generate similar problems (eg, who distributes, scores, and stores the questionnaires). For example, in a study of PRIME-MD,35 a well-validated, paper-and-pencil tool, clinicians frequently made scoring mistakes because of the complex algorithms. Both paper-based and Internet-based screeners present surmountable implementation challenges; the greater challenge rests with the paradigm shift that would elevate behavioral health assessment to the same level of importance as medical assessment.
The BHS clearly is promising but, without more system-level changes, barriers to use will remain.3,4 Reimbursement for screening and treatment for mental health problems would increase physicians' motivation to add yet another responsibility to their demanding schedules.1,2 Appropriate education about mental health risk assessment could increase expertise and thus reduce avoidance of this important but often-stigmatized medical domain.36 Reducing system-level barriers to collocating medical and behavioral health services could increase collaboration and coordination between these treatment systems.37 In an era of aiming to reduce health care costs, these kinds of changes could increase early identification and utilization of outpatient services and reduce costly emergency department and psychiatric hospital visits.15
This project was funded in part by the Pennsylvania Department of Health (SAP 4100027295), the Centers for Disease Control and Prevention (grant R49 CE000428-03), the Substance Abuse and Mental Health and Services Administration (Garrett Lee Smith Suicide Prevention Initiative), the American Foundation for Suicide Prevention and the Health Resources and Services Administration (grant U49 CE001093-01).
Administrative, technical, or material support was provided by Jennifer Barnett, Jacqueline Reyner, Allison Reamy, Anna Westin, Jacqueline Spitzer, Erin Lucey, Matthew Haines, Stephen DeLuca, Colleen Walsh, Ruth Craig, Jessica Hamilton, and Alana O'Malley. We thank all of the referring medical staff members, social workers, participating family members, and students and volunteers who gave invaluable time and effort.
- Accepted March 10, 2010.
- Address correspondence to Guy S. Diamond, PhD, Center for Family Intervention Science, Children's Hospital of Philadelphia, 34th and Civic Center Boulevard, Philadelphia, PA 19104. E-mail:
The Pennsylvania Department of Health specifically disclaims responsibility for analyses, interpretations, and conclusions.
FINANCIAL DISCLOSURE: Drs Diamond, Levy, Fein, Wintersteen, and Tien are exploring a licensing agreement that would make the BHS available to the public.
- BDI-II =
- Beck Depression Inventory II •
- BHS =
- Behavioral Health Screen •
- CRMS =
- Clinical Research Management System •
- PTSD =
- posttraumatic stress disorder •
- SSI =
- Scale for Suicidal Ideation •
- TSCC =
- Trauma Symptom Checklist for Children
- 1.↵New Freedom Commission on Mental Health. Achieving the Promise: Transforming Mental Health Care in America: Final Report. Rockville, MD: US Department of Health and Human Services; 2003
- 2.↵US Department of Health and Human Services. Mental Health: A Report of the Surgeon General: Executive Summary. Rockville, MD: US Department of Health and Human Services; 1999
- 3.↵American Academy of Child and Adolescent Psychiatry; American Academy of Pediatrics. Improving mental health services in primary care: reducing administrative and financial barriers to access and collaboration. Pediatrics. 2009;123(4):1248–1251
- 4.↵US Preventive Services Task Force. Screening and treatment for major depressive disorder in children and adolescents: US Preventive Services Task Force recommendation statement. Pediatrics. 2009;123(4):1223–1228
- 5.↵The Joint Commission. Suicide risk reduction. Available at: www.jointcommission.org/AccreditationPrograms/Hospitals/Standards/09_FAQs/NPSG/Focused_risk_assessment/NPSG.15.01.01/Suicide+risk+reduction.htm. Accessed November 1, 2009
- 7.↵American Academy of Pediatrics, Committee on Adolescents. Suicide and suicide attempts in adolescents. Pediatrics. 2000;105(4):871–874
- Frank RG,
- McDaniel SH,
- Bray JH,
- Heldring M
- 9.↵Institute of Medicine. Reducing Suicide: A National Imperative. Washington, DC: National Academies Press; 2002
- 11.↵Annenberg Adolescent Mental Health Project. Preliminary Report. Princeton, NJ: Princeton Survey Research Associates International; 2003
- Peters S
- Showstack J,
- Rothman AA,
- Hassmiler SB
- Pincus HA
- 21.↵American Medical Association. Guidelines for Adolescent Preventive Services (GAPS). Chicago, IL: American Medical Association; 1997. Available at: www.ama-assn.org/ama/upload/mm/39/gapsmono.pdf. Accessed July 2, 2009
- 23.↵Institutes of Medicine. Learning Healthcare System Concepts v. 2008. Washington, DC: National Academy Press; 2009
- 24.↵American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th ed. Washington, DC: American Psychiatric Association; 1994
- Beck AT,
- Steer RA,
- Brown GK
- Achenbach TM
- Briere J
- Hu L,
- Bentler PM
- Zuckerbrot RZ,
- Cheung AH,
- Jensen PS,
- Stein REK,
- Laraque D
- Fleegler EW,
- Lieu TA,
- Wise PH,
- Muret-Wagstaff S
- Fein JA,
- Pailler M,
- Diamond G,
- Wintersteen M,
- Tien A,
- Hayes K,
- Barg F
- Stroul BA
- Copyright © 2010 by the American Academy of Pediatrics