OBJECTIVE. Injury risk, depressive symptoms, and substance use are the leading causes of adolescent morbidity and death. The goal of this randomized, controlled trial was to determine whether computerized screening with real-time printing of results for pediatricians increased the identification of these adolescent behavioral concerns.
METHODS. A total of 878 primary care patients 11 to 20 years of age participated in computerized behavioral screening (the Health eTouch system) in waiting rooms of 9 urban clinics. These clinics all served predominantly low-income patients. The clinics were randomly assigned to have pediatricians receive screening results either just before face-to-face encounters with patients (immediate-results condition) or 2 to 3 business days later (delayed-results condition).
RESULTS. Fifty-nine percent of Health eTouch respondents had positive results for ≥1 of the following behavioral concerns: injury risk behaviors, significant depressive symptoms, or substance use. Sixty-eight percent of youths in the immediate-results condition who screened positive were identified as having a problem by their pediatrician. This was significantly higher than the recognition rate of 52% for youths in the delayed-results condition.
CONCLUSION. Immediate provision of an adolescent's self-report of behavioral concerns to a pediatrician increased recognition of those problems, compared with the delayed provision of results.
Three behavioral concerns, namely, injury risk, depressive symptoms, and substance use, are among the most important adolescent health problems. The American Academy of Pediatrics recommends routine screening for these behavioral concerns in primary care, because of their high prevalence among teenagers.1 However, screening often does not occur in pediatric settings, in large part because of logistic constraints.2 The staff time required to distribute, to score, and to file traditional paper-and-pencil questionnaires and the competing demands of primary care physicians during brief face-to-face visits with patients are substantial barriers to such screening. Not surprisingly, primary care providers typically recognize just one half of youths with significant parent-reported symptoms.3
Recent advances in information technology (eg, touchpad computers with wireless Internet connections) may help overcome barriers to behavioral screening. Direct data entry by youths in waiting rooms and automated scoring and printing programs minimize the staff time necessary for selecting a tool, performing screening and scoring, and reporting and filing results. Computer algorithms can select particular questions that are indicated according to the patient's age and previous responses.4 In addition, adults and adolescents are more willing to disclose sensitive information to a computer than to a person, especially a clinician with whom they have a long-standing relationship.5,6 In summary, computerized data collection directly from patients has many advantages for accomplishing behavioral screening in primary care.
Although a few studies have suggested that computerized screening is a feasible way of increasing identification of pediatric behavioral concerns, those investigations did not use a randomized design to test changes in identification rates.7–9 The goal of the present study was to conduct a randomized, controlled trial to test whether computerized screening with real-time printing of results for pediatricians increases the identification of common adolescent behavioral concerns.
The Health eTouch system, an Internet-based application that collects self-report data from patients in clinical settings, was implemented in 9 urban primary care clinics operated by Nationwide Children's Hospital. Eight available sites were matched with respect to volume and patient insurance status and then block-randomized (through coin toss) to 1 of 2 feedback conditions, that is, immediate results or delayed results. A ninth site, which became available midway through the study, was assigned randomly to the first feedback condition. At 5 sites, the patients' screening results were printed, and this 1-page summary was given to the primary care provider just before the face-to-face encounter between the provider and the patient (immediate-results condition). In the remaining 4 sites, the primary care provider did not have access to the screening results during the visit. Instead, the 1-page summary was mailed to the primary care provider 2 to 3 business days later (delayed-results condition). We considered it unethical to withhold screening results for more than a few days. The 1-page summaries were identical for the 2 conditions and highlighted positive screening results and referral telephone numbers where appropriate (see Fig 1 for a sample summary). Because of the nature of this research, neither the youths nor the providers were blinded to group assignment.
If a youth reported thoughts about suicide, then all results from the Health eTouch screening were immediately shared with the primary care provider, regardless of whether the youth was seen at an immediate-results site or a delayed-results site. We thought that primary care providers should have immediate access to all screening results for these youths, in case urgent intervention was required.
Recruitment and Consent Procedures
Adolescents who were 11 to 20 years of age and accompanied by a parent or guardian (if <18 years of age) were approached by either clinic registration staff members or research staff members in the primary care waiting room and were invited to participate in the study. Recruitment rates for the clinic registration staff members were not known, because clinic workflow did not allow staff members to log unsuccessful approaches to possible recruits. Recruitment rates for the study's 3 research assistants ranged from 60% to 95%.
For youths <18 years of age, consent was obtained from the parent or guardian accompanying the patient and assent was obtained from the youth. Youths >18 years of age provided consent. We informed patients that all information entered on the tablet would be available to both clinical and research staff members. The Health eTouch tablets presented online versions of the consent and assent forms to the parent and the child and captured their electronic signatures if the parent and the child agreed to participate. Paper copies of the consent and assent forms were also provided both before and after participants agreed to be in the study. Once consent and assent were obtained, the adolescents completed the Health eTouch screener in the waiting room or in other areas of the clinic, if they were called back to the examination rooms before completion. All aspects of the study were approved by Nationwide Children's Hospital's institutional review board.
One question at a time from the Health eTouch risk assessment was presented to patients on secure wireless-Internet tablets with 10-inch touchscreen displays. Because the items presented varied on the basis of the user's age and reported behaviors, the actual number of items completed ranged from 45 to 101. The median time to complete the Health eTouch risk assessment was 12.5 minutes.
Items for the youth risk assessment were drawn from existing and publicly available validated measures in 4 domains, namely, injury risk, depressive symptoms, suicidality, and substance use. Injury risk was measured by using age group-appropriate items from the Youth Risk Behavior Survey.10 Depressive symptoms were assessed by using the Center for Epidemiological Studies Depression Scale for Children (CES-DC), a 20-item depression screening tool.11,12 The CES-DC has acceptable internal reliability, reasonable test-retest reliability, and moderate concurrent validity for adolescents.13 Suicidal ideation was assessed with a single question from the Patient Health Questionnaire for Adolescents.14 Specifically, youths were asked, “Has there been a time in the past month when you have had serious thoughts about ending your life?” Substance use was measured by using items from the Comprehensive Addiction Severity Index for Adolescents (CASI-A).15 The CASI-A has reasonable concurrent validity with clinical records of adolescent substance abuse.15 All youths completing the Health eTouch assessment were presented with CASI-A items measuring use of tobacco, alcohol, marijuana, and inhalants within the past month. If the adolescent acknowledged use, then he or she was asked about the frequency of use during the last month. All youths who reported use of tobacco, alcohol, marijuana, or inhalants were asked questions about use of other illicit substances (eg, hallucinogens or cocaine). Reported usage rates for these other illicit substances were extremely low (<1%) in the present sample. Youths who reported alcohol or marijuana use were asked questions about the consequences of their substance use, based on impairment items from the CASI-A.
Clinician Visit Questionnaire
Pediatricians at both the immediate-results and delayed-results sites were asked to complete a brief clinician visit questionnaire for each patient on the day the youth was seen. On this questionnaire, adapted from the Child Behavior Study,16 providers reported their clinical perceptions regarding whether each youth had a behavioral concern. We defined recognition of a behavioral concern as the provider endorsing ≥1 of 3 separate items reflecting perceptions that the youth (1) had a mental health or psychosocial problem, (2) had moderate or high risk of injury, or (3) was using alcohol, tobacco, and/or other drugs.
Analyses were conducted by using SPSS 14.0 for Windows (SPSS, Chicago, IL). We conducted linear regression analyses (for continuous variables) and logistic regression analyses (for dichotomous variables) in which we used a dummy variable representing the experimental condition. Clinic variation within each experimental condition (immediate-results sites versus delayed-results sites) likely could result in dependence among observations within clinics.17 Therefore, we also included dummy variables in the regression analyses that captured the fixed effects of clinic variation within experimental condition. We did not account for clustering associated with clinician, however, because treatment providers often did not identify themselves on the clinician visit questionnaire. The rotating nature of the attending physicians and residents in these training clinics did not permit us to replace these missing values.
The primary question addressed by this study was as follows: “Did providing clinicians with screening results immediately before their face-to-face encounter with the family increase their recognition of behavioral concerns?” To test this outcome, we restricted our sample to youths who endorsed a behavioral concern through Health eTouch, because our study focused on primary care providers' recognition of these particular youths. Youths with behavioral concerns met ≥1 of the following criteria: (1) >1 injury risk behavior, (2) clinically significant level of depressive symptoms, and (3) substance use within the past 30 days. On the basis of these 3 criteria, 520 (59%) of the 878 Health eTouch respondents screened positive for a behavioral concern.
Data from 1 site were dropped from the inferential analyses because the site's very small sample size (n = 29) led to unstable parameter estimates. At the 8 sites included in these inferential analyses, 509 youths screened positive for a behavioral concern. On the clinician visit questionnaire, physicians completed items regarding whether they perceived particular youths as having a behavioral concern for 473 (93%) of these 509 youths.
A total of 878 unique patients completed at least part of our Health eTouch randomized trial, 491 at the immediate-results sites and 387 at the delayed-results sites. This randomized trial took place between June 1, 2005, and February 20, 2006. The average age of the participants was 13.9 years (SD: 2.2 years), which indicated a predominantly young adolescent sample. A total of 473 participants (54%) were female; 499 participants (57%) were black, 312 (36%) white, 33 (4%) Hispanic, and 34 (4%) other or unknown race/ethnicity. Six hundred seventy-two participants (77%) had Medicaid as their insurance provider, whereas 133 (15%) had commercial insurance and 59 (7%) had no insurance.
Injury Risk Characteristics
A total of 644 youths (73%) endorsed engaging in ≥1 risky behavior that could result in serious injury, and 361 youths (41%) met the positive screening threshold of >1 injury risk behavior. Responses to individual injury risk questions are presented in Table 1. Risky responses included never or rarely wearing a seat belt or a helmet when bicycle riding, rollerblading, or skateboarding. Risky responses also included riding in a car with someone who had drunk alcohol, carrying a weapon, and being in a physical fight necessitating medical attention.
A total of 842 youths completed all CES-DC items, 33 youths completed some but not all CES-DC items, and 3 youths completed none of the CES-DC items. CES-DC scores were computed by summing the responses to the completed items, as long as ≥1 item was completed. The average CES-DC score was 17.1 (SD: 11.3), indicating that a large proportion of our sample had at least mild levels of depressive symptoms, according to the recommended clinical cutoff score of 16.18 One hundred thirty-five (52%) of the first 261 youths who responded to the CES-DC had results in the clinically significant range. We found that our primary care and behavioral health care systems could not manage the volume of cases generated with our original cutoff score of 16. Therefore, we began using a more-stringent cutoff score of 35 for our provider reports of positive depression screens, which reflected only youths with moderate/severe levels of depressive symptoms. Sixty-four (10%) of the remaining 617 youths who responded to the CES-DC had results in the clinically significant range when this more-stringent cutoff score was used. Across the entire study period, 199 youths (23%) endorsed clinically significant levels of depressive symptoms.
A total of 854 of the 878 youths responded to the question regarding suicidal ideation within the past month. Of those who responded to the question, 127 (15%) endorsed having serious thoughts about ending their lives within the past month.
Substance Use Characteristics
A total of 138 (16%) of 827 youths reported using tobacco, alcohol, marijuana, and/or inhalants in the past month. Table 2 presents information regarding the prevalence, frequency, and quantity of use for each substance individually. Youths who reported alcohol and/or marijuana use were asked to complete the impairment items from the CASI-A. On average, these youths endorsed 3.3 (SD: 3.4) of the 20 items.
Preliminary Inferential Analyses
We assessed whether our randomization procedures were successful in achieving 2 groups of patients with comparable demographic and clinical characteristics. Patients at the immediate-results and delayed-results sites did not differ with respect to race, insurance status, or likelihood of endorsing a behavioral concern (all P > .10). However, we did note differences between the immediate-results and delayed-results sites with respect to age and gender (P < .05). Therefore, age and gender were included as independent variables in the regression analyses described below.
Testing the Impact of Experimental Condition on Overall Identification
We found that 65% of youths (170 of 262 youths) with behavioral concerns at the immediate-results sites were recognized by their primary care provider, compared with 60% of youths (126 of 211 youths) with behavioral concerns at the delayed-results sites. In this intent-to-treat analysis, this difference approached but did not reach statistical significance (adjusted odds ratio [aOR]: 1.573; 95% confidence interval [CI]: 0.99–2.51; P = .058). However, a sizeable proportion (29%) of youths with behavioral concerns at the delayed-results sites endorsed suicidal ideation. When suicidal ideation was endorsed, physicians were provided with immediate results for these patients, regardless of whether the youth was treated at an immediate-results or delayed-results site.
Therefore, we conducted a separate analysis in which all youths who endorsed suicidal ideation, regardless of original condition assignment, were included in the immediate-results condition. This “as-treated” analysis revealed that 68% of youths (211 of 310 youths) with behavioral concerns in the immediate-results condition were recognized by their primary care provider, compared with 52% of youths (85 of 163 youths) with behavioral concerns in the delayed-results condition. In a logistic regression analysis, this difference reached statistical significance (aOR: 2.94; 95% CI: 1.81–4.76; P < .001).
Finally, we restricted our sample to youths with a behavioral concern who did not endorse suicidal ideation. This analysis showed that 63% of such youths (130 of 208 youths) with behavioral concerns at the immediate-results sites were recognized by their primary care provider, whereas 53% of such youths (83 of 156 youths) with behavioral concerns at the delayed-results sites were recognized by their primary care provider. This difference between experimental conditions was significant (aOR: 2.12; 95% CI: 1.24–3.63; P < .01).
Testing the Impact of Experimental Condition on Identification Within Specific Domains
We used the 3 aforementioned approaches (ie, intent to treat, as treated, and suicidal youths excluded) to examine the impact of immediate provision of results on identification of specific behavioral concerns. Three logistic regression analyses tested whether experimental condition predicted physician identification of moderate/severe injury risk for youths who endorsed >1 injury risk behavior through Health eTouch. Experimental condition was significant only in the as-treated analysis (aOR: 2.22; 95% CI: 1.23–4.02; P < .01). Forty-nine percent of youths (105 of 216 youths) with >1 injury risk behavior were recognized by their pediatrician in the immediate-results condition, compared with 44% of youths (51 of 117 youths) with >1 injury risk behavior in the delayed-results condition.
Three additional logistic regression analyses tested whether experimental condition predicted physician identification of substance use for youths who endorsed substance use within the past 30 days. Experimental condition was significant only in the intent-to-treat analysis (aOR: 2.95; 95% CI: 1.11–7.87; P < .05). Seventy-three percent of substance-using youths (61 of 84 youths) were recognized by their pediatrician at immediate-results sites, compared with 42% of substance-using youths (19 of 45 youths) at delayed-results sites. No adverse events as a result of completing the screening were reported.
This study demonstrated that the immediate provision of an adolescent's self-report about behavioral concerns to a primary care provider increased recognition of these concerns, compared with the delayed provision of results. When we analyzed the data according to the initial randomization status of youths (intent-to-treat analysis), clinicians who received immediate feedback recognized more youths. However, the difference did not reach significance, most likely because the treatment effect was diluted through the breaking of the study randomization. A large proportion of youths reported suicidal thoughts on their screens, and clinicians were given those results immediately, regardless of the youths' initial randomization status. However, when we compared recognition rates with immediate feedback and rates with delayed feedback regardless of initial randomization status, the differences were large and statistically significant. The difference was also significant when we excluded suicidal youths from the analysis. Finally, we found that immediate provision of results increased identification of specific behavioral concerns in some instances.
Because all participants completed the study measures in waiting rooms and only the immediacy of the reporting of results varied, we consider our design to be a conservative test of the benefits of computerized behavioral screening with real-time results. The comparison condition was computerized screening with delayed results, but usual care in most practices involves no routine or standardized screening, computerized or otherwise. The computerized screening might have prompted youths from the delayed-results sites to initiate conversations with their primary care providers on topics that otherwise would not have been discussed. This may explain why our recognition rates at the delayed-results sites were higher than recognition rates detected in large, usual-care samples.3 In addition, our hospital's standard-of-care, well-child paperwork reminded providers to inquire about several behavioral issues (eg, seat belt use, tobacco cessation, and bicycle helmet use) during face-to-face encounters. Therefore, we speculate that the improvement in recognition rates that would be found in a comparison between computerized screening with immediate results and a usual-care condition with no routine screening would be substantially larger than the improvement we observed in this study.
In addition to documenting an increase in recognition rates, this study suggests that standardized behavioral screening is feasible in pediatric primary care clinics, through computerized technology. Nearly 900 youths were screened in 9 urban primary care clinics that serve a primarily Medicaid-insured population. Because the median time for protocol completion was 12.5 minutes, the great majority of screenings occurred before the patient saw the primary care provider. Primary care sites are ideal for such screening, because the vast majority of both insured and uninsured youths are seen at least annually at these clinics19 and because families often report feeling more comfortable discussing such concerns with primary care providers than with other types of professionals.20 Primary care sites that serve older adolescents and young adults, such as family medicine clinics and university health centers, might be especially desirable for substance use screening, because they likely would have an older clientele, relative to the present study.
The study's most unexpected finding was that ∼1 in 3 youths at immediate-results sites with positive screening results was not reported as having positive screening results by the primary care provider. It is possible that physicians often overlooked the 1-page summary of the screening results while attending to competing responsibilities. If the physician had seen the screening results, he or she might have disagreed with the results after completion of a face-to-face assessment of the youth. For example, the clinician might not have viewed the youth's problems as being sufficiently severe to merit attention. Alternatively, the physician might have disregarded the screening results if the patient retracted the behavioral concerns endorsed on the computer once he or she met with the primary care provider. Because we did not observe the physician-patient interaction and we did not interview physicians about how they interpreted the screening results, we cannot be sure why so many clinicians did not recognize the behavioral concerns or disagreed with the screening results. Future research should investigate how primary care providers incorporate screening summaries into their overall behavioral assessment of adolescents.
A somewhat unexpected finding was the high rate of depressive symptoms in our urban pediatric sample. On the basis of the report by Yates et al21 that their adolescent primary care sample had a 33% rate of high depressive symptom scores, we anticipated that many of our youths would screen positive for depression. After we found a substantially higher rate (∼50%), however, we decided to increase our cutoff score dramatically, to prevent our primary care and behavioral health care systems from being overwhelmed with positive depression screens. Our experiences underscore the substantial demand for adolescent depression services. Health care systems should consider increasing the number of mental health providers located in pediatric primary care clinics,22 training primary care providers to manage the psychopharmacologic treatment of uncomplicated depression,23 and expanding school-based depression prevention programs.24
Two limitations of the study should be highlighted. First, we did not include a paper-and-pencil questionnaire control condition to determine whether computer administration improved recognition rates. Because paper-and-pencil questionnaires have been available for decades but rarely are used in primary care offices, we thought there was little purpose in testing this comparison.
Second, our recruitment rates were suboptimal because the office receptionists were often too busy to distribute the Health eTouch tablet. Instead, most youths were enrolled only when research assistants had time available to sit in the clinics. Does this mean that computerized screening is unfeasible in primary care offices? We do not think so. The greatest time demand for receptionists or research staff members was in obtaining informed research consent. If such computerized screening becomes part of routine practice (and thus informed research consent is not required), then this time barrier will not be an issue. Furthermore, computerized screening was a “research task” and not a required component of the receptionists' workflow. Receptionist cooperation might have been greater if managers had measured and reinforced rates of screening, as was the case for other routine workflow tasks in those clinics. In addition, expansion of the use of computerized devices for provision of health-promoting information,25 identification of caregivers with health concerns,26 and other purposes (eg, patient registration and collection of copayments), as well as linking of touchscreen responses to electronic medical records, should increase the utility of such information technology in the future. Therefore, providers would have a greater incentive to incorporate this technology into routine practice.
This study has shown the benefits of computerized screening for identifying behavioral concerns in pediatric primary care. We think that screening may be an important first step in improving care for youths with these issues. Clinicians, administrators, and researchers should focus future efforts on developing more-comprehensive systems of care that not only identify these youths but also maximize the likelihood that they will receive and complete empirically supported treatments. Evaluation of the clinical benefits and financial costs of such systems of care should inform decisions regarding which particular adolescent behavioral concerns are most worthy of primary care screening efforts.
This study was funded by the National Institute on Drug Abuse (grant R01DA018943-04).
- Accepted October 4, 2007.
- Address correspondence to Jack Stevens, PhD, Research Institute at Nationwide Children's Hospital, 899 East Broad, 3rd Floor, Columbus, OH 43205. E-mail:
The authors have indicated they have no financial relationships relevant to this article to disclose.
What's Known on This Subject
Computerized screening for adolescent behavioral concerns seems feasible, according to previous research.
What This Study Adds
Immediate provision of an adolescent self-report of behavioral concerns to a pediatrician increases recognition of these problems, compared with the delayed provision of results.
- ↵American Academy of Pediatrics. Insurance coverage of mental health and substance abuse services for children and adolescents: a consensus statement. Pediatrics.2000;106 (4):860– 862
- ↵Kelleher KJ, Childs GE, Wasserman RC, McInerny TK, Nutting PA, Gardner WP. Insurance status and recognition of psychosocial problems: a report from the Pediatric Research in Office Settings and the Ambulatory Sentinel Practice Networks. Arch Pediatr Adolesc Med.1997;151 (11):1109– 1115
- ↵Lucas RW, Mullin PJ, Luna CB, McInroy DC. Psychiatrists and a computer as interrogators of patients with alcohol related illnesses: a comparison. Br J Psychiatry.1977;131 (8):160– 167
- ↵Zuckerbrot RA, Maxon L, Pagar D, Davies M, Fisher PW, Shaffer D. Adolescent depression screening in primary care: feasibility and acceptability. Pediatrics.2007;119 (1):101– 108
- Adams WG, Mann AM, Bauchner H. Use of an electronic medical record improves the quality of urban pediatric primary care. Pediatrics.2003;111 (3):626– 632
- ↵Paperny DM. Computerized health assessment and education for adolescent HIV and STD prevention in health care settings and schools. Health Educ Behav.1997;24 (1):54– 70
- ↵Brener ND, Collins JL, Kann L, Warren CW, Williams BI. Reliability of the Youth Risk Behavior Survey questionnaire. Am J Epidemiol.1995;141 (6):575– 580
- ↵Fendrich M, Weissman MM, Warner V. Screening for depressive disorder in children and adolescents: validating the Center for Epidemiological Studies Depression Scale for Children. Am J Epidemiol.1990;131 (3):538– 551
- ↵Kelleher KJ, McInerny TK, Gardner W, Childs GE, Wasserman R. Increasing identification of psychosocial problems: 1979–1996. Pediatrics.2000;105 (6):1313– 1321
- ↵Newacheck PW, Brindis CD, Cart CU, Marchi K, Irwin CE. Adolescent health insurance coverage: recent changes and access to care. Pediatrics.1999;104 (2):195– 202
- ↵Young PC, Shyr Y, Schork MA. The role of the primary care physician in the care of children with serious heart disease. Pediatrics.1994;94 (3):284– 290
- ↵Williams J, Shore SE, Foy JM. Co-location of mental health professionals in primary care settings: three North Carolina models. Clin Pediatr (Phila).2006;45 (6):537– 543
- ↵Cardemil EV, Reivich KJ, Seligman MEP. The prevention of depressive symptoms in low-income minority middle school students. Prev Treat.2002;5 :article 8
- ↵Thompson DA, Lozano P, Christakis DA. Parent use of touchscreen computer kiosks for child health promotion in community settings. Pediatrics.2007;119 (3):427– 434
- ↵Dubowitz H, Feigelman S, Lane W, et al. Screening for depression in an urban pediatric primary care clinic. Pediatrics.2007;119 (3):435– 443
- Copyright © 2008 by the American Academy of Pediatrics