BACKGROUND: It is important to improve primary care providers’ capability to identify youth at risk for alcohol and other drug use. To our knowledge, this is the first study to use Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria to compare screeners for youth for both alcohol and marijuana, given that these are the most frequently used substances by this age group.
METHODS: We compared the psychometric performance of 4 screeners: the National Institute on Alcohol Abuse and Alcoholism Screening Guide (NIAAA SG), the Alcohol Use Disorders Identification Test, the Car–Relax–Alone–Forget–Family and Friends–Trouble (CRAFFT) screener, and the Personal Experience Screening Questionnaire Problem Severity Scale (PESQ-PS) in identifying alcohol and marijuana use outcomes. Youth age 12 through 18 (N = 1573; 27% black, 51% Hispanic) were screened with the NIAAA SG, followed by a Web survey that included the other screeners and outcomes.
RESULTS: Sensitivity for alcohol outcomes indicated that the NIAAA SG (0.87) did not perform as well as the CRAFFT (0.97) or PESQ-PS (0.97) screeners but performed better than the Alcohol Use Disorders Identification Test (0.70). The pattern for sensitivity across screeners for marijuana outcomes was similar.
CONCLUSIONS: An important tradeoff in primary care settings is precision versus practicality. Because of brevity and focus on frequency of drinking, the NIAAA SG offers ease of administration and is good at identifying youth with probably problematic drinking levels. The PESQ-PS and the CRAFFT correctly identify more at-risk youth for alcohol and marijuana than the NIAAA SG. Future work is needed to elucidate how to efficiently and accurately identify at-risk youth in the primary care setting, including determining the best cutoff points to use to increase sensitivity.
- AOD —
- alcohol and other drug
- AUD —
- alcohol use disorder
- AUDIT —
- Alcohol Use Disorders Identification Test
- CRAFFT —
- Car–Relax–Alone–Forget–Family and Friends–Trouble
- CUD —
- cannabis use disorder
- DISC-IV —
- Diagnostic Interview Schedule for Children Version IV
- DSM-5 —
- Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition
- NIAAA SG —
- National Institute on Alcohol Abuse and Alcoholism Screening Guide
- NPV —
- negative predictive value
- PESQ-PS —
- Personal Experience Screening Questionnaire Problem Severity Scale
- PPV —
- positive predictive value
What’s Known on This Subject:
Most adolescents are not screened for substance use in primary care because of time constraints or insufficient training. Providers need a screener that can be easily incorporated into an appointment; however, there is a tradeoff between precision and practicality.
What This Study Adds:
We examined sensitivity, specificity, and positive and negative predictive value for the new National Institute on Alcohol Abuse and Alcoholism screener and 3 widely used adolescent screeners for various levels of alcohol and marijuana use and impairment in a large, racially and ethnically diverse sample of adolescents.
It is important to improve the capability of primary care providers and associated health care staff to identify youth at risk for alcohol and other drug (AOD) use because use during adolescence is associated with academic problems, poorer mental health, future use of other illicit drugs (including heroin and cocaine), and a higher likelihood of abuse or dependence in adulthood.1–3 The primary care setting provides a unique opportunity to screen significant numbers of adolescents4 and identify those at risk for problematic use. Guidelines propose that providers screen for AOD use and provide brief counseling and referrals where appropriate5 and that doctors screen all patients for alcohol use starting in middle school.6
Unfortunately, most adolescents are not screened for AOD use in primary care settings,7–11 and significant numbers of at-risk youth remain unidentified and never receive appropriate preventive or treatment services.12–14 Lack of screening and preventive services is even more profound among younger adolescents age 11 to 14 and socioeconomically disadvantaged youth.15,16 Lack of primary care screening typically is caused by provider time constraints, discomfort discussing AOD use, insufficient training, or lack of referral options.9,10,17,18
Given these concerns, providers need an easy-to-administer screener that takes little time or training, can be incorporated into a primary care appointment to determine an adolescent’s risk level, and will facilitate appropriate referral or treatment. There is a difficult tradeoff between precision and practicality in the primary care setting. To increase screening in pediatric settings, the National Institute on Alcohol Abuse and Alcoholism (NIAAA) developed a brief 2-item screening guide (SG) with 2 age-specific screening questions about friend and self-drinking.19 Age-sensitive cutoff points can assist providers in determining whether brief advice, counseling, or referral is appropriate. For example, any report of drinking if the youth is ≤15 years old warrants brief advice and counseling. For youth ≥16 years old, the threshold is a bit higher, and drinking ≥6 days in the past year is considered moderate or high risk.
In 2014, Kelly et al11 evaluated the NIAAA SG in a sample of 525 youth age 12 to 17 years (54% female) who were mainly black (92.8%). Their focus was on expanding the NIAAA SG to include drug and tobacco questions for a total of 10 questions and determining cutoff points for their Brief Screener for Tobacco, Alcohol and Other Drugs. They established cutoff points for this new screener for each substance and compared the Brief Screener for Tobacco, Alcohol and Other Drugs cutoff points with the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) criteria.11 In 2016, Clark et al20 examined cutoff points for the NIAAA SG and found that ≤3 days of alcohol use in the past year yielded optimal psychometric performance.
Several studies21–25 have reviewed adolescent screeners to determine performance against DSM-4 and DSM-5 criteria. Some screeners focus only on alcohol, such as the NIAAA SG, whereas others address AOD more broadly. A comparison of the Alcohol Use Disorders Identification Test (AUDIT; 10 items; alcohol only), Car–Relax–Alone–Forget–Family and Friends–Trouble (CRAFFT; 6 items; AOD), Cut Down, Annoyed, Guilty, Eye-Opener (CAGE; 4 items; alcohol only), and Tolerance, Worry, Eye-Opener, Cutoff (TWEAK; 5 items; alcohol only) in a sample of youth recruited in an emergency department found that Cut Down, Annoyed, Guilty, Eye-Opener items were reported infrequently by adolescents and that the AUDIT performed the best across the range of cutoff scores.21 The Problem Severity Scale on the Personal Experience Screening Questionnaire (PESQ-PS; 18 items; AOD)23,26,27 had the highest reliability estimates when compared with several other screening measures that address behavior and consequences for youth, such as the Rutgers Alcohol Problem Index.28 Furthermore, the 2011 American Academy of Pediatrics guidelines recommended routine use of the CRAFFT, which measures AOD use,29 has been validated in diverse populations,30,31 and has good sensitivity and specificity with new DSM-5 criteria.32
Thus, across diverse studies, the AUDIT, CRAFFT, and PESQ-PS appear to be the most sensitive and reliable screeners with adolescents. They are short, taking ∼5 to 10 minutes to complete and score. The NIAAA SG19 could also easily be integrated into primary care practice given its brevity,11 and studies have shown that asking youth about alcohol use frequency is often the best predictor of their alcohol use over time.33,34 We therefore compared these 4 screeners in identifying both alcohol and marijuana use among adolescents because these substances are the most frequently reported substances for this age group.35
This study moves the field forward by comparing 4 screeners in a large racially and ethnically diverse sample of adolescents, ages 12 through 18, recruited across 4 primary care clinics in Los Angeles, California and Pittsburgh, Pennsylvania. The study’s purpose was to inform primary care providers seeking to screen adolescents for alcohol and marijuana use about the 4 screeners’ strengths and weaknesses; we compared screeners with respect to sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for various levels of alcohol and marijuana use and impairment.
This study involved 4 clinics (1 in Los Angeles, 3 in Pittsburgh). We obtained a certificate of confidentiality; procedures were approved by both the RAND institutional review board and the 4 clinics. Every youth age 12 through 18 who came for an appointment during the 2.5-year study period (April 2013–November 2015) was asked to be in the project. We obtained parental consent and youth assent (<18 years old) or consent (18 years old). Youth were screened with the NIAAA SG, completed a survey via the Web, and paid $25.
Setting, Participants, and Surveys
The Pittsburgh and Los Angeles area clinics are family-based community health clinics that provide care for ethnically and racially diverse and underserved populations of youth. Sites offer both longitudinal, continuity-based care and episode-based urgent care to their patients. Clinics in both cities have a large percentage of minority patients and serve a high proportion of low-income patients. Approximately 3309 youth were approached to be in the project. Of these youth, 27% (n = 892) were ineligible because of age, lack of English proficiency, being present for an appointment other than their own, or disability status; 18.5% (n = 614) declined to participate, mostly because of time constraints or youth being at the clinic for family planning and not wanting their parents to know they were there. This process yielded a total sample of 1803 youth who enrolled or provided consent to contact. Of the 1803 youth, 230 did not complete the baseline within the field period or had unreliable contact information. The final enrolled sample included 1573 youth. Screening and surveys were completed in a private clinic space without a parent present. RAND staff first screened youth in person by using the NIAAA SG; youth then completed a Web-based survey comprising other screeners and outcome variables on a laptop immediately after completion of the NIAAA SG (see Supplemental Information for screener questions).
Two screening questions were asked in a different order depending on age19: “In the past year, on how many days have you had more than a few sips of beer, wine, or any drink containing alcohol?” and “Do any of your friends drink alcohol?” Youth age 12 to 14 years were first asked about friend drinking and then self-drinking as a less threatening way to gauge use, whereas youth age ≥15 years (and 14-year-olds in high school) were first asked about self-drinking and then friend drinking. Adolescents were categorized based solely on the days of use question according to the published NIAAA risk assessment guide, which results in 4 risk categories: no risk, lower risk, moderate risk, and highest risk. For example, youth age 12 to 15 years were categorized as moderate risk if they reported 1 to 5 days of use; adolescents 16 years old were categorized as moderate risk if they reported 6 to 11 days of use.
The 6-item CRAFFT29 addresses both alcohol and other drugs (eg, “Do you ever use AOD to Relax, feel better about yourself, or fit in?”). Response options are “yes” or “no,” and a “yes” response to ≥2 questions indicates risk.
The 18-question PESQ-PS26 assesses AOD use rated on a 4-point response scale (never, once or twice, sometimes, often). The summed score categorizes adolescents into 3 groups, no AOD problem (“Green Flag”), mild or moderate AOD problem (“Yellow Flag”), and severe AOD problem (“Red Flag”), by using established thresholds. Thresholds vary depending on sex and age.
The AUDIT36 focuses on frequency and consequences of drinking. It was modified slightly to be developmentally appropriate for youth.21 Youth are categorized into Zone I (alcohol education), Zone II (simple advice), Zone III (simple advice plus brief intervention and follow-up), or Zone IV (referral to specialist).
Youth with an alcohol use disorder (AUD) or cannabis use disorder (CUD) were identified via the Diagnostic Interview Schedule for Children Version IV (DISC-IV)37 computerized version, valid and reliable in adolescent populations.38–40 We used DSM-5 criteria available in this version (eg, the craving item was included; see Supplemental Figure) to identify subjects with AUD or CUD. Adolescents were classified as having an AUD or CUD if they reported ≥2 of the 11 criteria for AUD or CUD. Past-year alcohol and marijuana use was assessed via well-established measures with adolescents.41 We asked, “During the past year, how many times did you [drink at least one full drink of alcohol] [use marijuana]?” Responses ranged from 1 = “0 times” to 6 = “11–20 times” and were dichotomized (1 = “any use” versus 0 = “no use”). For past-year heavy alcohol use, respondents were asked, “During the past year, how many times have you tried five or more drinks of alcohol in a row, that is, within a couple of hours?” with the same response options and dichotomization. For past-year heavy marijuana use, respondents were asked, “On days you use marijuana, how often do you use it?” Responses were “Once,” “Twice,” and “3 or more times,” and responses of “twice” and “3 or more” were considered heavy marijuana use.
We examined the proportion of youth identified as at risk for each screener. We estimated sensitivity, specificity, PPV, and NPV against DSM-5 diagnoses of AUD and CUD, past-year any use, and past-year heavy alcohol and marijuana use. Sensitivity is the probability that the screener correctly identifies at-risk youth (as at risk), whereas specificity is the probability that the screener correctly identifies no-risk youth (as low or no risk). PPV is the probability of a case screened as positive actually being positive; this depends on specificity of the test and prevalence of the condition. NPV is the probability of a negative diagnosis indicating a true negative. For screeners with multiple risk categories, we used the lowest established risk threshold to dichotomize adolescents into low- and high-risk categories. Specifically, for the AUDIT, youth categorized in Zone II or above were considered at risk, for the PESQ-PS, youth categorized in the yellow or red flag categories were considered at risk, and for the NIAAA SG screener, youth categorized as moderate or high risk were considered at risk. We had <2.5% of data missing for any 1 variable; pairwise deletion was used to handle missingness. We performed analyses via R version 3.2.4 (R Foundation for Statistical Computing, Vienna, Austria).
More than half the sample was female; youth were racially and ethnically diverse, with a mean age of 15.5 years (Table 1). More than 40% of adolescents reported drinking >1 time in the past year, with almost a quarter reporting heavy use. About 37% reported using marijuana >1 time in the past year, with 1 in 5 reporting heavy marijuana use. According to DSM-5 criteria, 3.6% of youth were identified with an AUD; 13.6% were identified with a CUD.
Table 2 shows number and proportion of adolescents in each risk category by screener, stratified by age. Older youth were more likely to be identified as at risk. Overall, the NIAAA SG, CRAFFT, PESQ-PS, and AUDIT identified 19%, 30%, 31%, and 8% of adolescents as at risk, respectively.
Table 3 presents sensitivity, specificity, PPV, and NPV and associated 95% confidence intervals for screeners by AUD, past-year alcohol use, and past-year heavy alcohol use. For sensitivity, for all alcohol outcomes, the NIAAA SG did not perform as well as the CRAFFT or PESQ-PS screeners but performed better than the AUDIT. For AUD, the sensitivity was 0.87 for the NIAAA SG, 0.98 for the CRAFFT, 0.97 for the PESQ-PS, and 0.70 for the AUDIT. Specificity tended to be high, but it was lowest for the CRAFFT and PESQ-PS when AUD was used as the outcome. For PPV, among adolescents identified as at risk on the NIAAA SG, 19% had an AUD, 90% reported past-year alcohol use, and 66% reported heavy past-year alcohol use.
Table 4 parallels Table 3 for marijuana outcomes. Similar to alcohol, the NIAAA SG performed better than the AUDIT but did not perform as well as the CRAFFT or PESQ-PS in terms of sensitivity. For example, among adolescents with CUD, the CRAFFT, PESQ-PS and NIAAA SG correctly identified 88%, 91%, and 54% as at risk, whereas the AUDIT correctly identified 32%. For PPV, among adolescents identified as at risk on the NIAAA SG, 40% had a CUD, 77% reported past-year marijuana use, and 50% reported past-year heavy marijuana use. For both PPV and NPV, the NIAAA SG performed similarly to the CRAFFT and PESQ-PS when CUD and heavy marijuana use were examined.
To our knowledge, this is the first study to use the DSM-5 AUD and CUD criteria to compare several different screeners for youth. We compared 4 adolescent screeners for various levels of alcohol and marijuana use and impairment with a large, racially and ethnically diverse multisite primary care population. The CRAFFT and PESQ-PS, which address AOD, identified about one-third of youth as at risk, the NIAAA SG identified ∼19%, and the AUDIT identified ∼8%. The CRAFFT and PESQ-PS had excellent sensitivity for detecting an AUD and also did well for CUD. The NIAAA SG, briefer and focused exclusively on alcohol, was better at identifying youth with an AUD versus a CUD, and the AUDIT had lower sensitivity for both disorders. Screeners had higher sensitivity for identifying youth who reported past-year heavy alcohol use compared with any past-year drinking. This result is not surprising given that one would generally expect youth with heavy drinking to be easier to identify as high risk compared with youth who have had any past drinking in the last year (which may include very light drinkers). In addition, screeners that addressed AOD versus only alcohol did better at identifying youth who reported heavy past-year marijuana use and any past-year marijuana use.
Overall, specificity was lower than sensitivity for all screeners for AUD and CUD, although it was still good. Given that the potential harm of a false positive is low in this setting, whereas risks associated with a false negative are high, providers might be more willing to accept lower specificity for high sensitivity to ensure that at-risk youth are identified. As expected, the PPV was better for all screeners in identifying past-year use for alcohol and marijuana use versus AUD and CUD because the PPV depends on both specificity of the test and prevalence of the condition. The NPV was also high for all screeners.
An important tradeoff in primary care settings is precision versus practicality. Incorporating screening into everyday practice can be difficult if the screener is long or not intuitive. Because of its brevity and focus on frequency of drinking, the NIAAA SG offers ease of administration, and results show that it is good at identifying youth with problematic drinking levels. However, the PESQ-PS and the CRAFFT correctly identify more at-risk youth for alcohol than the NIAAA SG. One recent study found that computer self-entry for the CRAFFT was valid and time-efficient.43 Other work has also shown that brief screens can be completed electronically as part of routine care.44 Future work is needed to elucidate how to most efficiently and accurately identify more at-risk youth in the primary care setting, including determining the best cutoff points to use to increase sensitivity.
Although the NIAAA SG performed well for alcohol outcomes, it did not do as well identifying youth who report marijuana use. This result is not surprising because the NIAAA SG questions focus on alcohol. However, identifying youth at risk for marijuana use is also important, particularly because it affects more domains of functioning in adolescence than alcohol.45 Furthermore, in this sample, 72% of youth who reported past-year alcohol use also reported past-year marijuana use, and CUD in this sample was >3 times as common as AUD, emphasizing the importance of asking about marijuana use in primary care settings. If a positive screen occurs, providers need to discuss potential harms of marijuana so youth better understand how the drug can affect functioning in both adolescence and adulthood. This discussion is particularly important because many youth view marijuana use as less harmful than alcohol use,46 perhaps because of continuing changes in state laws regarding medical and recreational use of marijuana. Interventions as brief as 15 minutes can lead to subsequent reductions in youth AOD use,47,48 but time constraints, issues of confidentiality, and knowledge of what to do after a positive a screen remain common barriers to screening in the primary care setting.49–51 Training designed to increase provider confidence in screening and discussing AOD use with at-risk youth52,53 and addressing confidentiality concerns51 is 1 approach to decreasing barriers, as is incorporating technology and providing computerized brief interventions in primary care settings for at-risk youth.54
Limitations of the current study include self-report, although self-report limitations are often exaggerated,55 and rates of AOD use in our sample were similar to national rates.41 The sample may also not be generalizable to all adolescents in primary care clinics; however, clinics did cross 2 states, and the sample was racially and ethnically diverse, with a wide range of ages. We also always administered the NIAAA SG to youth first, which could be a reason for lower sensitivity.43 Finally, we cannot speak to performance of these screeners as clinical tools. For example, knowing their doctor will see their answers might change an adolescent’s likelihood of disclosure. Future studies should examine screener performance under more realistic clinical conditions.
In sum, many youth were identified as at risk by these 4 screeners; however, some screeners performed better than others, depending on the substance and severity (eg, frequency of use versus a disorder). Future work could focus on quantifying tradeoffs in precision and practicality of these screeners in primary care.
The authors thank the clinics that participated and supported this project. We also thank Kirsten Becker and Jennifer Parker for overseeing the survey administrations and Michael Woodward for help with manuscript formatting.
- Accepted September 27, 2016.
- Address correspondence to Elizabeth J. D’Amico, PhD, RAND Corporation, 1776 Main St, Santa Monica, CA 90401. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by a grant from the National Institute of Alcohol Abuse and Alcoholism (R01AA021786) to Elizabeth D’Amico. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Berg N,
- Kiviruusu O,
- Karvonen S, et al
- Elster AB,
- Kuznets NK
- Kokotailo PK; Committee on Substance Abuse
- Kelly SM,
- Gryczynski J,
- Mitchell SG,
- Kirk A,
- O’Grady KE,
- Schwartz RP
- Klein JD,
- Allan MJ,
- Elster AB, et al
- Stevens GD,
- Shi L
- Millstein SG,
- Marcell AV
- Clark DB,
- Martin CS,
- Chung T, et al.
- Shields AL,
- Campfield DC,
- Miller CS,
- Howell RT,
- Wallace K,
- Weiss RD
- Orenstein A,
- Davis RB,
- Wolfe H; CASPAR Alcohol and Drug Education Program
- Miech RA,
- Johnston LD,
- O’Malley PM,
- Bachman JG,
- Schulenberg JE
- Babor TF,
- de la Fuente JR,
- Saunders J,
- Grant MA
- Shaffer D,
- Fisher P,
- Lucas C,
- Comer J
- Shaffer D,
- Lucas CP,
- Richters JE
- Johnston LD,
- O’Malley PM,
- Bachman JG,
- Schulenberg JE
- Collett D
- Helitzer DL,
- Sussman AL,
- de Hernandez BU,
- Kong AS
- Committee on Adolescence American Academy of Pediatrics
- Lustig JL,
- Ozer EM,
- Adams SH, et al
- Chan D
- Copyright © 2016 by the American Academy of Pediatrics