Objective. To compare providers' impressions of adolescents' level of substance use with diagnostic classifications from a structured diagnostic interview.
Methods. Secondary analysis of data was conducted from a validation study of the CRAFFT substance abuse screening test of 14- to 18-year-old medical clinic patients (n = 533) and their corresponding medical care providers (n = 109) at an adolescent clinic affiliated with a large tertiary care pediatric hospital. Medical care providers completed a form that recorded their clinical impressions of patients' level of alcohol and drug involvement (none, minimal, problem, abuse, dependence) and demographic characteristics. The form included brief diagnostic descriptions for each level of use. After the medical visit, patients completed the Adolescent Diagnostic Interview (ADI), a structured diagnostic interview that yields diagnoses of abuse and dependence according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV). On the basis of their past 12 months of alcohol and drug use on the ADI interview, adolescents were classified into 5 mutually exclusive diagnostic groups. “None” was defined by no reported use of alcohol or drugs during the past year. “Minimal use” was defined as use of alcohol or drugs but no report of any substance-related problems. “Problem use” was defined as reporting 1 or more substance-related problems but no diagnosis of abuse or dependence. “Abuse” was defined by meeting any 1 of 4 DSM-IV diagnostic criteria for either alcohol or drug abuse but no diagnosis of dependence. “Dependence” was defined by meeting any 3 of 7 diagnostic criteria for either alcohol or drug dependence, with or without a diagnosis of abuse. Proportions were compared using Fisher exact test. Agreement was assessed with the weighted κ, and these analyses were stratified by substance used (ie, alcohol vs drug) and demographic characteristics. Sensitivity, specificity, and positive and negative predictive values were calculated from 2 × 2 tables.
Results. Compared with the criterion standard interview, providers identified significantly fewer patients with problem use and abuse and no patients with dependence. Of >100 patients whom the ADI classified with substance problem use, providers correctly identified 18. Of 50 patients who were classified with a diagnosis of alcohol or drug abuse, providers correctly identified 10. Of 36 patients who were classified with a diagnosis of alcohol or drug dependence, providers correctly identified none. For the 86 adolescents who were classified with a substance-related disorder (ie, abuse or dependence), providers' impressions were “none” (24.4%), “minimal use” (50%), “problem use” (15.1%), “abuse” (10.5%), and “dependence” (0%). There was only marginal agreement between providers' impressions and diagnoses related to alcohol use (κ = .29), drug use (κ = .31), and any substance use (κ = .30). Kappa was not significantly affected by the patient's age, but it was by gender. Among boys, κ was significantly higher for impressions of drug use versus alcohol use (κ = .48 vs κ = .27); and, among drug users, κ was significantly higher among boys compared with girls (κ = .48 vs κ = .24). Kappa did not differ significantly across race/ethnicity subgroups, although there is a suggestive trend toward higher agreement for black non-Hispanic compared with white non-Hispanic adolescents (κ = .35 vs κ = .21). Kappa did not differ significantly on the basis of the visit type, but the size of this difference (κ = .36 vs κ = .24) suggests that the longer well-child visit yielded greater identification of substance-related pathology. Providers' impressions had a sensitivity of .63 for identifying use of alcohol or drugs. However, sensitivity was poor for identification of problem use (.14), abuse (.10), and dependence (0), whereas specificity and positive predictive values were high. Of the 86 adolescents with a diagnosis of abuse or dependence, 75.6% were correctly identified by providers as using substances; however, the level of use in 50% of these adolescents was reported by providers as minimal.
Conclusions. In this study, clinical impressions of adolescents' alcohol/drug involvement underestimated substance-related pathology. When providers thought that use was present, there was a very high likelihood that a problem or disorder existed. The use of structured screening devices would likely improve identification of adolescents with substance-related pathology in primary care settings and should be considered for use with all adolescent patients, rather than only those who are perceived to be at higher risk.
Adolescent substance abuse is a major public health problem. The majority of adolescents have used alcohol or another drug by the time they have reached 12th grade. Among adolescents aged 12 to 17 years, it is estimated that 11.6% are current illicit drug users.1 Fifty-one percent of high school students have tried an illicit drug, and ∼16% of 8th graders have experimented with a volatile inhalant.2 More than 25% of adolescents reported having used an illicit substance other than marijuana by the end of 12th grade.2
Despite a minimum legal age requirement to purchase alcohol, ∼10.7 million Americans (28.8%) between the ages of 12 and 20 years reported drinking alcohol in the past month.1 Seventy-seven percent of students have tried drinking by 12th grade, and 46% have done so by 8th grade; 58% and 20%, respectively, have been intoxicated at least once in their life.2
The societal implications of adolescent substance abuse are well known. Teenage pregnancy, date rape, acquisition of sexually transmitted diseases, and overall involvement in high-risk sexual behaviors are all too often associated with adolescent substance use and subsequent hazardous behaviors.3 In 1994, 29% of crash-related deaths among 15- to 17-year-olds were alcohol related.4 Homicides and suicides have also been closely linked to adolescent substance use.5,6
Medical offices provide a unique opportunity for early identification and intervention. The American Medical Association's Guidelines for Adolescent Preventive Services recommend that health care providers screen all adolescent patients annually for use of alcohol and drugs as part of routine care,7 and the American Academy of Pediatrics recognizes the important role of the pediatrician in addressing the issue of adolescent substance use.8,9
A recent study showed that most pediatricians report screening “almost all” of their adolescent patients for substance use.10 However, the majority of pediatricians (84%) do not use a structured screening tool11 and may rely on clinical impressions alone. How accurate these impressions are is currently unknown. The purpose of this study was to compare medical care providers' impressions of adolescents' level of substance use with diagnostic classifications determined by a criterion standard interview.
This was a secondary analysis of data collected during the CRAFFT validation study, and the detailed methods were published previously.12 Briefly stated, data were collected between March 1999 and September 2000 at a large, urban, hospital-based adolescent clinic. Participants were 14- to 18-year-old patients who presented to an ambulatory clinic for routine or urgent care and their corresponding medical care providers.
Providers invited age-eligible patients to participate in the study at the conclusion of the medical visit. A research assistant met with those who agreed, explained the study procedures, obtained informed assent, and administered the assessment battery. The Children's Hospital Boston Committee on Clinical Investigation (institutional review board) waived the requirement for parental consent in accordance with current guidelines for adolescent health research.13,14
For assessing actual substance-related diagnoses of abuse or dependence, adolescents were administered the Adolescent Diagnostic Interview (ADI)15 by a trained research assistant. The ADI is a 30- to 90-minute structured interview that provides alcohol- and drug-related diagnoses according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV).16 The ADI has been shown to have good reliability and validity among adolescents.17,18 On the basis of their past 12 months of alcohol and drug use, adolescents were classified into 5 mutually exclusive diagnostic groups. “None” was defined by no reported use of alcohol or drugs during the past year. “Minimal use” was defined as use of alcohol or drugs but no report of any substance-related problems. “Problem use” was defined as reporting 1 or more substance-related problems but no diagnosis of abuse or dependence. “Abuse” was defined by meeting any 1 of 4 DSM-IV diagnostic criteria for either alcohol or drug abuse but no diagnosis of dependence. “Dependence” was defined by meeting any 3 of 7 diagnostic criteria for either alcohol or drug dependence, with or without a diagnosis of abuse.
Medical care providers completed a 7-item questionnaire that recorded their clinical impressions of the adolescent's alcohol and drug involvement on a forced-choice response scale that included the same 5 diagnostic categories. The questionnaire also included brief diagnostic descriptions: none (n/a), minimal use (“experimentation, occasional use, no significant associated problems”), problem use (“one or more problems associated with use; no established pattern”), abuse (“pattern of recurrent use, recurrent problems, continued use despite harm”), or dependence (“loss of control over use, preoccupation with use, tolerance or withdrawal”). Providers also recorded the adolescent's age, gender, and race/ethnicity. The providers' questionnaire was developed for the primary study as a way of assessing possible differences between study participants and study refusers.
All data were independently entered twice into a data management program based on Access 97 (Microsoft, Redmond, WA). The dual-entry files were compared and discrepancies reconciled by checking the original data source. The cleaned data set was then imported into SPSS version 10.0 (SPSS Inc, Chicago, IL) for analysis. We compared provider impressions with ADI-determined diagnostic classifications using a variety of statistical procedures. Proportions were compared using Fisher exact test. Agreement was assessed with the weighted κ, and we stratified these analyses by substance used (ie, alcohol vs drug) and demographic characteristics. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated from 2 × 2 tables, using the bootstrap technique to compute 95% confidence intervals.19–21 Differences in estimates of substance involvement were assessed using the Wilcoxon signed-rank test. Age was dichotomized at the median to preserve adequate cell size in these analyses.
A total of 533 (80%) of 670 eligible patients participated in the study. Refusers did not differ significantly from participants in age, gender, race/ethnicity, or clinician impressions of substance use. Compared with the primary care clinic demographics, our study sample had an overrepresentation of girls (68.4% vs 59.4%; P < .001) and youths of color (50.6% vs 29% black non-Hispanic; 24.2% vs 36% white non-Hispanic; 18.8% vs 12% Hispanic; 6.5% vs 2% Asian/other race/ethnicity; no data for remaining percentage).
A total of 109 providers participated in the study. Sixty-three percent (n = 68) were residents, 17% (n = 19) were medical students, 13% (n = 14) were staff clinicians (faculty physicians and nurse practitioners), and 7% (n = 8) were fellows. Medical students were the providers for 14% of the adolescent visits, residents for 34%, fellows for 15%, and staff clinicians for 36%.
Of the 533 participants, 37% (n = 197) came to the clinic for a well visit, and 40% (n = 215) came for urgent care. Twenty-three percent (n = 121) arrived for other reasons, including shots (19%; n = 23) and counseling (2%; n = 3); data for visit type were classified as missing for 79% (n = 95) of these participants.
Table 1 presents a comparison of the distribution of substance use classifications for both provider impressions and diagnoses on the basis of the ADI interview. Providers significantly underestimated the prevalence of substance-related problems, as well as disorders of abuse and dependence. Whereas the ADI identified >100 patients with problem substance use, providers identified 18. Whereas 50 patients had an abuse diagnosis, providers identified 10. Most striking, 36 adolescents had a diagnosis of dependence, and providers identified none. For the 86 adolescents with an ADI diagnosis of abuse or dependence, providers' impressions were “none” (24.4%), “minimal use” (50%), “problem use” (15.1%), “abuse” (10.5%), and “dependence” (0%).
Table 2 presents the weighted κ with 95% confidence interval for provider impressions and ADI diagnoses stratified by alcohol use, drug use, and any substance use; patient age, gender, and race/ethnicity; provider level of training; and type of visit (30-minute well-care vs 15- to 30-minute urgent care appointment). In general, weighted κ values >.75 are considered excellent agreement, .4 to .75 good agreement, and <.4 marginal agreement beyond chance.22
There was only marginal agreement between medical care providers' impressions and ADI diagnoses regardless of substance used, and κ did not differ significantly across alcohol use, drug use, or any substance use. Kappa was not significantly affected by the patient's age, but it was by gender. Among boys, κ was significantly higher for impressions of drug use versus alcohol use; and among drug users, κ was significantly higher among boys compared with girls. Kappa did not differ significantly across race/ethnicity subgroups, although there is a suggestive trend toward higher agreement for black non-Hispanic compared with white non-Hispanic adolescents.
Stratification by providers' level of training showed no significant differences; however, our design did not have adequate power to fully examine these differences. Similarly, κ did not differ significantly on the basis of the visit type, but the size of this difference (.36 vs .24) suggests that the longer well-child visit yielded greater identification of substance-related pathology.
Table 3 presents sensitivity, specificity, PPV, and NPV of providers' impressions for 4 dichotomized substance use conditions. “Any use” includes minimal use, problem use, abuse, or dependence; “any problem” includes problem use, abuse, or dependence; “any disorder” includes abuse or dependence; and “dependence” includes participants with an ADI diagnosis of alcohol or drug dependence only.
Providers correctly identified 63% of adolescents who were using alcohol or drugs. However, providers' impressions had a very low sensitivity for the “any problem,” “any disorder,” and “dependence” categories. That is, providers correctly identified only 14% of adolescents with serious problems, only 10% of those with disorders, and none of those with alcohol or drug dependence, the highest level of severity. However, specificity and PPVs were high. That is, when providers thought that a problem was present, there was a very high likelihood that a problem or disorder truly existed. Of the 86 adolescents with a diagnosis of abuse or dependence, 75.6% were correctly identified by providers as using substances; however, the level of use in 50% of these adolescents was reported by providers as minimal.
This study shows that medical care providers underestimate the severity of substance-related pathology in adolescents. Although they are usually aware of patients' use of alcohol or drugs, they seldom identify patients with problematic use, abuse, or dependence. This is unfortunate because it is these teens who might benefit most from early intervention services.
There may be several explanations for this finding. Providers may not have directly asked adolescents about substance-related problems or other signs and symptoms of abuse and dependence. It is also possible that adolescents were reluctant to disclose fully to their providers details of the negative consequences of their substance use. They may have felt more comfortable disclosing to the research assistant with whom they had no therapeutic alliance and who, in turn, had no direct link to their parents. Adolescents may have perceived the research assistants in our study as less of a threat to confidentiality. In addition, providers in our study may have spent less time than usual obtaining substance use histories because they knew that the adolescent would receive a complete substance use assessment from the research assistant as part of the study.
We found that providers were better able to identify male adolescents who were using drugs, as compared with alcohol, and better able to identify drug use among boys, as compared with girls. Research shows that boys are more likely to have higher rates of both drug and alcohol use than girls.23 Thus, it is possible that providers have a higher index of suspicion of drug use when seeing boys. However, girls account for a majority of visits in many clinics, and the opportunity for early identification of drug use should not be lost.
Although not reaching the level of statistical significance, there was a suggestion that providers were better able to identify adolescent substance use during the well-child visit as compared with the urgent-care visit. This likely reflects the well-care visits' greater time, emphasis on preventive care and screening, and lower level of conflicting medical acuity.
Study strengths include the large sample size, high participation rate, and the use of the ADI as the criterion standard. However, the study also had limitations. It was a secondary analysis and relied on adolescents' self-report. The primary purpose of the questionnaire completed by providers was to assess for possible self-selection bias in the parent study, not to measure the accuracy of their clinical impressions. In addition, the study sample was not selected randomly. There was an overrepresentation of girls and youths of color compared with the overall clinic population, and the percentage of visits with trainees (medical students, residents, and fellows) in the study sample was twice the rate of what actually occurs in the clinic, thereby yielding a sample that typically has less continuity of care experience. Also, the provider who completed the impressions form for a given adolescent was not always the adolescent's primary care provider. Thus, a difference in familiarity with the adolescent's history could contribute to inconsistencies seen among provider impressions.
Although providers were given orientation to the purpose of the primary study, they were not given training in forming an impression (eg, diagnostic criteria for abuse or dependence) of the adolescent's substance use, and some providers mentioned that they were unclear of the adolescent's level of use. Also, there may have been some ambiguity with regard to the “none” category on the providers' impressions form. Selecting this category could have been interpreted by some providers as having no impression of the adolescent's substance use, rather than the adolescent's having no use.
Nonetheless, accurately identifying adolescents with substance-related problems and disorders remains a challenge. Providers need enough time for the discussion of substance use during adolescent well visits and for those urgent-care visits that raise a concern about any risky behavior. Doing so may be particularly challenging, as other critical health issues also need to be addressed in the time-limited encounter. We recommend using a structured screening test to assess substance use. A number are available for this purpose, including both questionnaires and brief orally administered screens, such as the CRAFFT, which has been shown to have good sensitivity and specificity for identifying substance-related problems and disorders.12 If the adolescent has a positive screen, then providers should follow-up with appropriate questions aimed at obtaining a more comprehensive assessment of the level of substance-related pathology. In many office settings, it may be impractical to obtain all of this information during the course of a single visit. Therefore, we suggest that providers briefly express their concerns regarding the positive screen and ask their patient to return for a follow-up visit, during which the issue can be addressed more fully. During the follow-up visit, providers should inquire about age of first use, pattern of use, and specific problems associated with each substance used. The information gathered will be helpful in establishing a management plan, which may include an agreement with the patient to reduce his or her substance use or referral of the adolescent for substance abuse counseling or treatment.
Last, we recommend that providers become familiar with the substance abuse prevention and treatment programs in their own communities. The knowledge of resources and referral options should help to clarify the necessary next step when an adolescent patient screens positive for a substance-related problem or disorder.
Clinical impressions of adolescents' alcohol and drug involvement are not always accurate. The use of structured screening devices would likely improve identification of adolescents with substance-related pathology in primary care settings and should be considered for use with all adolescent patients, rather than only those who are perceived to be at higher risk.
This study was supported by grant R01 AA12165 from The National Institute on Alcohol Abuse and Alcoholism and The Substance Abuse and Mental Health Services Administration and grant 036126 from The Robert Wood Johnson Foundation. Other support was provided by grant 5T20MC000-11-06 (Dr Knight) and grant 5T71MC00009-12-0 from the Maternal and Child Health Bureau.
We thank Betsy Gates, BA, Sarah Rosenberg, BA, Katherine O'Connor, BA, and Allison Arneill, MA for assistance in study implementation; Lydia Shrier, MD, MPH, and the clinicians and staff of the Adolescent/Young Adult Medical Practice at Children's Hospital Boston for assistance in recruitment; Ken C. Winters, PhD, for consultation on the study measurement battery; and S. Jean Emans, MD, and Elizabeth R. Woods, MD, MPH, for review of the manuscript.
- ↵SAMHSA (Substance Abuse and Mental Health Services Administration). Overview of Findings From the 2002 National Survey on Drug Use and Health. Rockville, MD: Office of Applied Studies; 2003 (NHSDA Series H-21, DHHS Publication SMA 03-3774)
- ↵Johnston LD, O'Malley PM, Bachman JG. Ecstasy use falls for second year in a row, overall teen drug use drops. Ann Arbor, MI: University of Michigan News and Information Services. Available at: www.monitoringthefuture.org. Accessed January 12, 2004
- ↵American Academy of Pediatrics, Committee on Substance Abuse. Role of the pediatrician in prevention and management of substance abuse. Pediatrics.1993;91 :1010– 1013
- ↵American Academy of Pediatrics: Committee on Substance Abuse. Alcohol use and abuse: a pediatric concern. Pediatrics.2001;108 :185– 189
- ↵American Academy of Pediatrics. Barriers to providing health care for adolescents: the pediatrician's view (Periodic Survey of Fellows #42). Presented at the Association for Health Services Research Annual Meeting; Los Angeles, CA; June 2000
- ↵American Academy of Pediatrics. 45% of Fellows routinely screen for alcohol use (Periodic Survey of Fellows #31). AAP News.1998;14 :1
- ↵Department of Health and Human Services (Title 45 CFR, Part 46). U.S. Department of Health and Human Services, National Institutes of Health, Office for Protection From Research Risks. Available at: http://ohrp.osophs.dhhs.gov/humansubjects/guidance/45cfr46.htm. Accessed May 29, 2003
- ↵Winters K, Henly G. Adolescent Diagnostic Interview (ADI) Manual. Los Angeles, CA: Western Psychological Services; 1993
- ↵American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th ed. Washington, DC: American Psychiatric Association; 1994
- ↵Johnston L, O'Malley P, Bachman J. Monitoring the future: national results on adolescent drug use: overview of key findings, 2002. Bethesda, MD: National Institute on Drug Abuse; 2003 (NIH Publication 03-5374)
- Copyright © 2004 by the American Academy of Pediatrics