Children of substance-abusing parents, including children of alcoholics (COAs), are one of the highest risk groups of youth for substance-abuse problems. For both genetic and family environmental reasons, COAs and children of drug abusers are very vulnerable to becoming alcohol or other drug abusers.1,2 With drug use rates increasing in the past 7 years,3 prevention practitioners must work harder to identify and evaluate effective ways to prevent future substance abuse in these at-risk children. Most prevention programs designed specifically for COAs or children of drug abusers have struggled with identifying, attracting, maintaining, and measuring outcomes.
This article focuses on general and unique measurement methods and instrument problems in prevention interventions for children of substance-abusing parents. Part I covers the need for improved measurement in research and practice with children of substance-abusing parents and recommended measures for different hypothesized outcome variables. Part II covers considerations in selecting measures, and Part III covers how to select measures. This article concludes with recommendations to improve measurement in research and practice.
OVERVIEW OF NEED AND TYPES OF MEASUREMENT
Need for Intervention Outcome Research With Children of Substance Abusers
Importance of Valid Measurement of Interventions With Children of Substance Abusers
Children of substance abusers are the highest risk group of children for becoming alcohol and drug abusers for both genetic and family environment reasons.1 With drug use increasing worldwide and in this nation in the past 8 years,3 we must improve our research knowledge and prevention services for COSAs. Scientific progress in improving our understanding of precursors of problem behaviors or health problems in children of substance abusers (COSAs), as well as designing and testing more effective prevention interventions for COSAs, is closely tied to our ability to identify and use valid measurement strategies. According to Johnson and associates, “Your answers to clinical, applied, or basic scientific questions about COAs depend on the quality of the data obtained from our assessments.”4 Creation of a common language to communicate results of COSA research would help researchers and practitioners to be able to “compare findings and discuss, with confidence, reasons for commonalities and differences.”4 Unfortunately, there has been little agreement about common measurement instruments to use in COSA research.
Within the Institute of Medicine5 system of classification of prevention approaches into universal, selective, and indicated approaches, prevention approaches for COSAs generally can be classified as selective prevention approaches because they are designed specifically for a known, at-risk group. If youth are recruited to the program only because they are identified or self-identify as children of alcohol or drug abusers, the type of prevention programs designed specifically for them is classified as a selective prevention program. However, if the program is designed for COSAs, who are known to be hyperactive, depressed, aggressive, or thrill-seekers, then the prevention program is classified as an indicated prevention program. Most prevention programs for COSAs are family-focused or school-based programs.
According to Adger, “The first step in intervention and treatment is identification.”6 Identification of these children, however, is difficult for prevention programs in schools and communities. If the parents are in alcohol or drug treatment programs or self-help groups, it is easier to locate and recruit these children. However, only a small percentage of drug-abusing parents are in treatment programs. Because of the stigma of being a substance abuser, the parents and the children are less likely to allow themselves to be identified. Sometime the health, behavioral, or academic problems of COSAs bring them to the attention of professionals before the parental substance abuse is diagnosed.6
Goals and Objectives of the Article
To improve research and measurement in interventions for the prevention and treatment of substance abuse with COSAs.
To increase the sophistication in the development of the measurement model to match hypothesized precursors and ultimate alcohol and drug use.
To increase awareness of barriers to measurement that are unique to COSAs (lack of trust, denial, fear of reprisals) and propose possible solutions.
To increase sensitivity to developmental, cultural, and gender issues in measurements, and to increase the use of more valid and reliable measures with ethnic populations of COSAs.
To increase sharing of knowledge of the best measures by domains and by the most common outcome change variables.
To encourage the use of some common measures to increase the generalizability of results across studies and to make meta-analyses more feasible.
History and Current General Practice of Intervention Outcome Measurement
Reviews of measurements currently being used in interventions with COSAs4,,7 reveal little standardization and great variation in quality. Interventions that have been implemented recently in National Institutes of Health clinical trials are using multiinformant, multivariable measurement models, sometime called a multitrait–multimethod (MTMM) measurement strategy.8Self-report measures are collected from multiple sources—the child, parents, teachers, program facilitators, and other adults—to improve triangulation of the data. Other objective data sources, such as archival school and police records, are sought, and videotaped and coded observations of family interactions are conducted. Dishion and associates8 have found considerable variance in data collected from these different data sources (parent, child, staff report), suggesting the need for caution in studies relying on single-source reports. By combining data from different sources on the same construct or variable, measurement error and monomethod bias can be reduced.9,,10 Confirmatory factor analysis and structural equation modeling are used to validate constructs. Direct observations correlated with self-report data can be used to measure criterion validity, and objective school and police records can be used to measure predictive validity. The construct validation process is important and will demonstrate that not all sources are equally valid for different measures. For instance, a child's self-report of parental support and caring may be more predictive of later substance-abuse problems than objective coded observational data or the parent's perceptions or a teacher's observations. According to Fiske,11 this construct validation process cannot be dealt with by a quick pilot study of only internal consistency, but requires careful MTMM analysis using correlational matrices.
In contrast to these high-quality research measurement protocols, evaluations of COSA programs that are practitioner/clinician-developed are using much less effective measurement methods including single-measure, self-developed tests. The worst possible example is when self-report, client-satisfaction measures are used as the only outcome effectiveness measures. It should be stressed that although client satisfaction is important, this is not an outcome measure. The ultimate outcome measures are drug-use measures or risk precursors, such as behavioral and emotional changes in the child. More distal outcomes (derived from locally derived, etiologic models) such as changes in family dynamics, parent drug use, parenting, and peer and community environment also are acceptable change variables that should be measured. Direct observations of parent/child interactions are underutilized and could be used more frequently by researchers and clinicians to study microsocial transactional processes and for diagnostic purposes.
Even when a wide variety of precursor risk or protective factors are measured, there has been little use of common measures. In an extensive cataloging of measurement instruments used in 36 COA research studies using COAs behavioral assessment since the early 1960s, Johnson and associates4 identified over 70 different instruments used to assess COA risk and protective factors. Unfortunately, 80% of these instruments were used only once and only in one study, which hinders comparison or meta-analysis of results. The tests used most frequently include those presented in Table 1.4
Unfortunately, most of these measures are of internal COA behavioral, academic, and psychological variables, rather than external family, school, or community environmental variables, that precede symptoms of negative developmental trajectories in children. Johnson and associates conclude that most behavioral assessment strategies currently being used are not sufficient to “explore the predictive aspects of the developmental process in COAs.”4 They advocate for a developmental framework for assessing COAs' risk and protective precursors that would include multivariable assessment capable of determining subtle, yet important, divergences in normal developmental trajectories.
Dependent Measures for Intervention Research
Hence, it appears that COSA intervention research is deficient in measures of environmental context. In developing effective measurement models for COSA prevention programs, evaluators must measure valid precursors of alcohol and drug use. According to Dishion and associates,8 the field would profit from the development of a measurement model delineating developmental processes leading to adolescent problem behavior and serving as an intervention target. The selection of precursor variables to measure should be based on empirically tested models, not just theoretic assumptions. If prevention programs target the wrong precursors or the least salient precursors, the intervention will fail. Etiologic models suggesting the pathways to drug use12,,13 help COSA prevention program designers to target and measure the most important risk and protective factors.
Reviews of risk factors for alcohol and drug abuse can be found in reports by Hawkins and colleagues14,,15, and by Wright and Wright.16 Specific risk factors for COSAs can be found in Kumpfer,1 Johnson and Leff,17 and Tarter and Mezzich.2
Because currently we can do little to change genetic risk for alcoholism or drug abuse in COSAs, changing the family or peer social environmental risk is the most feasible approach to reducing overall risk. If this is so, what are the primary environmental risks to target and measure in a COA or COSA prevention program?
Measurement of Parent and Family Precursor Variables
The Social Ecology Model of Adolescent Substance Use structural equation model (SEM) data suggest that parents have an early influence on the developmental pathways toward alcohol and other drug use.18 Hence, living with an alcohol- or drug-abusing parent puts these children at great risk. However, what is it about families with an alcohol-abusing parent that puts these children at increased risk? Listed below and in Table 2 are some of the major family variables that could be measured and how they can be measured, and the etiologic findings suggesting their link with later alcohol and drug use.
Family Norms About Use
Although many empirically tested etiologic models13,,19,20 find that peer cluster influence is the final precursor of initiation of alcohol and other drug use, parental disapproval of alcohol use is a major reason not to use.21Because of parental role modeling and family norms, which show that drug use is useful socially and to reduce stress, COSAs frequently are missing this critical protective factor in their homes.
Recent research suggests that beliefs about drinking or alcohol schemas are impacted significantly by exposure to alcoholic parents even as early as preschool.22 Gaines23 reports that parents pass on to their children meta-cognitions about drinking goals, beliefs, expectancies, and rule-bound social competencies. Additionally, the less mature reasoning of COAs appears to be related to increased television watching including more modeling of drinking on TV and less parental interpretation of why people drink. Increasing parental supervision of TV, and parental communication of the negative consequences of alcohol use through family-focused interventions, should help reduce these cognitive risk factors in COAs.
All the researchers noted have developed research measures that can be used to measure these constructs. Dunn and Goldman have developed a positive expectancies measure that could be a way to monitor the changes in the impact of family and peer norms on the COSA. The Monitoring the Future test3 also includes a 3-item Expected Negative Consequences scale. The Oetting and associates24 American Alcohol and Drug Use instrument is a well-used measure for the youth's actual substance use, perceived harm, peer encouragement, and actual consequences. It also contains an optional insert that includes psychosocial and cultural correlates of substance use.
Parental Alcoholism or Drug Dependency
A number of researchers2 have reported a relationship between recent parental alcoholism and increased COA psychopathology and alcohol use. Although current parental alcoholism and alcohol or drug use should be measured, a number of researchers believe that parental alcohol and drug use status or diagnosis of alcohol dependency does not elucidate sufficiently family dynamics leading to drug use or positive targets for interventions in prevention programs. Measures for parental alcohol and drug use include the 30-item Children of Alcoholics Screening Test (Jones), which includes the self-report of the child. The parent also could be asked to complete National Institute on Drug Abuse (NIDA) 30-day quantity and frequency measure or a Short Michigan Alcohol Screening Test.26
Disturbed Family Relationships
Johnson and associates27 suggest the relationship between the parent's alcoholism and the COSA's later use patterns may be more a function of disturbed family relations than of the parent's alcohol status. A Resiliency Model of Family Stress, Adjustment and Adaptation, proposed by McCubbin and associates28 also stresses the importance of family functioning and categorizes families into topologies based on their family functioning. McCubbin and associates have developed measures for each of these family variables using FACES, Family Coping Strategies, and Family Inventory of Life Events and Changes instruments.
Parent/Child Relationship or Affective Quality
The quality of the parent/child relationship has been cited frequently as a critical variable in the intergenerational transmission of alcohol and drug use.29–31 Parental influence on children is frequently dichotomized into relationship quality or discipline/monitoring.8 Reanalysis of the Cambridge–Sommerville data found these two dimensions of parenting most predictive of delinquency.32 The etiologic research of Gerrard33 partially supports this, because she reports that parent/child relationship has a greater SEM β weight with alcohol peer group influence than the parental drinking status (β = .33 vs .28). Family relationships also can be measured using a 15-item family scale on Tarter's Drug Use Student Inventory.2Relationship affective quality is also measured by structured coding of videotapes by raters of the family's behaviors in structured family tasks and the overall impressions of these raters of the family's mutual acceptance and lack rejection. The Family Activities Checklist34 containing 28 activities that parents and children enjoy doing together also can be used.8
Family Conflict and Cohesion
Although Hughes and Gutkin35 have questioned their passive consent methodology, the Havey and Dodd's36 study also supports this conclusion by finding that family conflict, lack of family cohesion, and stressful life events, but not COA status per se, were the best predictors of early experimentation with alcohol, drugs, and tobacco. Hops37 reported that low family cohesion, low playground reciprocity, and number of surrounding adults using alcohol in elementary school children lead to increased alcohol use at age 14. The Moos Family Environment Scale (FES) is one of the most frequently used self-report measures of family environment. The FES includes 10-item scales for family conflict, family cohesion, family communication, and family organization, which the author finds sensitive to positive change in COSA family interventions.7 Tolan and associates38 use their Family Assessment Measure, which reduced the FES to 47-items.
Parental support has been found to be one of the most powerful predictors of reduced alcohol and other drug use in minority youth.39
Family Communication Style
In a very large sample (N = 4100) of Hispanic and white adolescent school youth, Baer40 reported significant relationships between maternal communication style; other family variables (family stress, family conflict, family status, and maternal monitoring); and proximal variables of alcohol use and peer use. Although these distal family environmental variables mediated significantly whether a COA would choose to associate with alcohol-using peers, there was also a small (β =.13) direct pathway from mother's alcohol use status to adolescent's alcohol use. Earlier, the author found this result in a Hispanic youth sample, but not in a white youth sample, in which all family variables were indirect paths mediated by self-esteem, school bonding, and peer use.41 The FES or direct codings or videotapes or observations of structured family situations or problem-solving situations have been used primarily to measure this variable.42
Parental Monitoring and Supervision
Recently, there has been increasing evidence that a major mechanism mediating between family variables and peer use (the final pathway to drug use) is parental supervision.40 Chassin and associates43 report significant SEM pathways from parental alcoholism to increased stress, depression, and decreased parental monitoring leading to increased substance use in adolescent COAs. Also, Dishion44 and Hansen and associates45 have found decreased parental supervision to be a major mediator of increased negative peer influence. Using path analysis (SEM), Ary and colleagues46 found direct paths not just from the frequently found “peer deviance” latent variable cluster to problem behaviors (ie, alcohol and other drug abuse, antisocial behavior, high risk sex, and academic failure), but also directly from “parental supervision” to problem behaviors including alcohol and drug use.
Measures of parental supervision and monitoring are discussed by Dishion and associates8 including self-report and observational coding measures, including the Family Process Code47 and the Pencil and Paper Code48 used at the Oregon Social Learning Center. Personal interviews and telephone self-report interviews ask parents and teens about parental monitoring. Staff members using the Family Process Code and Pencil and Paper Code are asked to rate how well the parent seems to monitor the child or to know the child's whereabouts.
A number of COSA interventions, such as my Strengthening Families Program (SFP),49 target improved parenting skills as a way of improving parental supervision and effective discipline. Useful measures of parenting skills include the SFP Parent Interview Questionnaire,50 the 8-item Family Management Scale,51 and the Alabama Parenting Questionnaire (child, parent, and teacher versions) as modified by Frick and Tolan. The Alabama includes scales for parent involvement, positive parenting, monitoring and supervision, inconsistent discipline, and corporal punishment.
Differential Family Acculturation
In immigrant families, another major risk factor is differential family acculturation.52 If the children become more westernized and reject the traditional ways of their parents, family conflict increases, leading to increased drug use in the children. Szapocznik and associates have developed a measure for differential family acculturation.
These results suggest that not all alcohol and drug misuse family risk processes are mediated by deviant peer involvement, because some family risk process variables have a direct impact. Hops37concludes based on existing etiologic research that to have impact on high-risk youth, we must focus on distal variables (like family dynamics) because proximal variables affect only low risk youth.
Summary of Family Risk and Protective Factors to Measure
The primary family risk factors include parent and sibling alcohol and other drug use, poor socialization, ineffective supervision and discipline, negative parent/child relationships, family conflict, family stress, poor parental mental health and differential family acculturation, and poverty.52
Family protective factors53,,54 include one caring adult,29,,55 emotional support, appropriate developmental expectations, opportunities for meaningful family involvement, support for dreams and goals, setting rules and norms, maintaining strong extended family support networks, and other protective processes.
Social Environment (Community, School, Media, Peer Group)
Other critical variables that preceded drug use according to the Social Ecology Model of Adolescent Substance Use18 include school climate, school bonding or attachment, and association with alcohol- or drug-using friends.
Most etiologic models13,18–20,56 find peer influence is the final pathway to initiation of drug use. Because many drug prevention programs seek to modify youth's susceptibility to peer influence, measures should be taken of changes in peer influence, such as the 7-item Susceptibility to Peer Pressure scale used by Dielman and associates in their school intervention research. Because peer support for nondrug use also is important, the 8-item Social Support for Non-drug Use57 is recommended. Peer use of drugs or alcohol also should be measured using self-report of the COSA, because perceptions of friends' drug use appears to be more critical than the actual drug use.
School Bonding or Attachment and School Climate (BASC)
Bonding to school has been found to be an important protective factor in preventing drug use.15,,18 This is an even more important variable to measure in COSAs because of their increased need for positive socialization. The Effective Schools Battery58 measures not only attachment to school, but also educational expectations, school effort, school involvement, positive peer associations, and clarity and fairness of school rules. The Behavioral Assessment Scale for Children (BASC) (Reynolds and Kamphaus) also has been used as a self-report measure for school bonding, using its “Attitude Toward Teachers” and “Attitude Toward School” scales. Other direct indicators of school attachment could include attendance, times tardy, and negative incident reports. However, with COSAs, the first three measures might not reflect the child's attachment and liking of school because the child may miss school or be late because of the parent's drinking or drug use.
Another indicator of school involvement and liking school would be grades; however, these can be biased because of reduced verbal or overall academic competency because of fetal alcohol or drug syndrome or milder developmental effects. According to Johnson and associates, a number of studies with COSAs measure achievement using the Peabody Individual Achievement Test and Wide Range Achievement Test. The Wechsler Intelligence Scale for Children–Revised or WAIS–R has been used in some studies to measure intellectual capacity.
Negative Developmental Trajectories
Youth who begin using and abusing substances generally have earlier behavioral and emotional signs of negative developmental trajectories. For this reason, interventions for COSAs should monitor:
Youth Social Skills Development using the Social Skills Rating System59 or BASC-Self-report (Reynolds and Kamphaus).
Youth Depression, Self-esteem, or Self-concept using the Achenbach and Edelbrock60 Child Behavior Checklist60 (CBCL) or Youth Self Report depression scale and Piers-Harris Children's Self-concept Inventory or Coopersmith Self-esteem Inventory for older youth.
Conduct Disorder/Self Regulation addresses problems in self-regulation and conduct disorders, which appear in children prone to substance abuse. The CBCL scales on conduct disorder and aggression are useful to measure these constructs.
Research suggests the probability of a child's developing problems increases rapidly as the number of risk factors increase61,,62 compared with the number of protective factors.62,,63 When children and youth are continually bombarded by family problems, their probability of becoming substance users increases.64–66
These etiologic models support the need for COSA prevention programs targeting improvements in distal environments that have a pervasive and long-term influence on COSAs, such as family and social environments (peer, community, cultural, media, and school). Intervention models are needed that can demonstrate improvements in many of these precursors for later drug use. Research is needed that will determine the risk or protective factors that are most amenable to change and that can produce the largest reductions in later drug use.
Importance of Improved Intervention Research on COSA Parents
Both NIAAA and NIDA are desirous of funding more intervention research for the prevention of alcohol and drug abuse in COSAs. They want to move beyond preintervention research to intervention research that will help the profession separate more effectively the influence of genetic and environmental research factors and to prevent future abuse in these high-risk children. Rose67 has hypothesized based on his genetic research that, “Genetic effects are more powerful once one begins drinking, but environmental effects are more influential in predicting abstinence. The choice to use and begin use is not genetic, but more influenced by their family and school environment.”67 A major challenge then is to support high-risk COAs or COSAs to not begin use by changing their family and social environment.
The ability of school-based and community-based selective prevention programs to impact COAs has been equivocal, partially because of problems in screening and identifying COSAs as discussed by Werner and colleagues68 and Emshoff and Price.69Unfortunately, problems exist with children of nonsubstance-abusing parents self-identifying as COSAs so they can join the group, thus increasing the percent of false-positives. Also, the real COSAs do not want to self-identify, thus increasing the percent of false-negatives. One way to avoid this problem is to work with identified substance-abusing parents in treatment programs or self-help groups using family-focused programs that also involve children-only groups. Another solution is to provide universal school-based programs that do not rely on pull-out programs for self-identified COSAs, but include content useful for both COSAs and non-COSAs. Also, with supportive health care professionals or teachers, COSAs can be encouraged to self-identify as COSAs.
Fortunately, there are powerful family interventions that appear capable of changing family dynamics and parenting enough to modify onset risk factors in COAs.52,,70 According to Robert Zucker, a noted researcher in family interventions, the best place to spend limited research funds is in reducing aggression, out-of-control behavior, and inappropriate parental role modeling through a Patterson-type behavioral parenting program. Zucker and associates71,,72 have had good success in reducing risks for alcohol use in preschool COAs through a 12- to 16-session behavioral parenting intervention combined with marital problem-solving. Kumpfer and associates have developed a comprehensive family intervention for COSAs, the Strengthening Families Program,49 that has proven effective in reducing risk factors; increasing resilience; and decreasing actual alcohol, tobacco, and drug use among elementary school COSAs across many different cultural groups (for review, see Kumpfer et al, 1996). This family intervention combines a 16-week Patterson-type behavioral parent training program with a children's social skills training program, and a family relationship-enhancement program. The program is generally conducted at churches, schools, or community centers in weekly sessions that take 2 to 3 hours each.
Measurement of Different Types of Intervention Research
Valid outcome measurement of interventions for COSA parents depends partially on the types of dependent variables measured in the research. There are at least four distinctly different types of outcome research, and despite some overlapping outcome measures, each tends to focus on its own types of dependent measures. These four types of measures in research include 1) measurement in etiologic theory testing research, 2) outcomes of interventions, 3) covariate outcome research, and 4) health services research. Measurement for each of these different types of research is presented below.
In addition to measurement of effectiveness in true randomized clinical trials of prevention or treatment interventions for COSAs, preclinical (NCI phases I and II) and postclinical research (NCI phases IV and V) are needed to bring the most effective interventions to clinical practitioners (Table 2). Measurement is important for both etiologic research on precursors of substance use and outcomes of interventions, because causal research is needed to determine the primary risk and protective factors to target in treatment and prevention-intervention programs. Additionally, considerable internal analysis is needed for health services research of interventions to help interpret the outcomes of randomized clinical trials or demonstration/evaluation research. Hence, quality measures are needed for both etiologic and outcome research analyses.
Etiologic Theory Testing Measurement Issues
Measurement issues in etiologic research include valid measures of primary precursors. Etiologic research involves intake data and longitudinal annual follow-ups to be used for etiologic theory testing of the mediating pathways between major hypothesized domains of risk or protective factors/mechanisms. The Social Ecology Model of Adolescent Substance Abuse organized these major precursor domains by family; community; culture; school climate; internal characteristics of school bonding/achievement; and self-efficacy, peer group, and substance use/abuse in youth.18 Generally, SEM testing is conducted on the waves of longitudinal data to test competing models of precursors of drug use in COSAs.
Currently, exemplary etiologic research on precursors of substance use is being conducted by NIAAA- and NIDA-funded researchers such as Baer,40 Gerrard,33 and Hops and associates.37,,46 Chassin and associates also are conducting excellent research on precursors of drug use in COAs.43
A major measurement problem in etiologic research is locating valid and reliable measures of constructs hypothesized to be primary precursors of alcohol and drug use in COSAs. Valid measures are those that accurately measure the variables or constructs they are intended to measure in the specific target population. There are three types of validity: content or face validity, criterion-related or predictive validity, and construct validity. Reliable measures are those that consistently measure these variables or constructs. There are two different types of reliability, ie, stability (true measurement with low measurement errors) and internal consistency of all items in a scale to measure the same construct. Very few standardized measures check the validity of their measures, particularly in ethnic populations. On the other hand, most measures check the internal consistency of their scales using Chronbach's internal consistency α statistic,73 which is easily available in SPSS computer software. Very few test developers go one step further to check for stability over time using test–retest reliability.
Although there are reasonably good measurement instruments for family and individual psychological and emotional characteristics, it is very difficult to find good measures for cultural constructs (cultural pride, cultural competency); social competencies (problem-solving, decision-making, social skills); and community context and bonding. Additional issues in etiologic measurement include age-appropriate measures because the youth are maturing during the study and a measure valid for one age will not be valid for another age. Sometime there are measures that have different versions for different developmental stages.
Because genetic and biologic differences are hypothesized to differentiate many COSA, particularly children of Type 2 alcoholics, valid measures of physiologic and genetic constructs must be located or created. Assessment of family history of prior alcohol and drug abuse or dependency is critical for etiologic research. Physiologic laboratory measures can be used for autonomic nervous system functioning (ie, heart rate, galvanic skin response, respiration) and central nervous system including brain waves and evoked potentials. Other important etiologic measurement constructs are temperament traits,74 thought to be closely related to inherited biologic status, such as thrill-seeking, hyperactivity, and rapid tempo. Each of these constructs require measurement and each of these types of measures have their own set of measurement issues, which are too complex to cover here.
COSA Intervention Research Measurement Issues
The primary risk vulnerability mediators frequently measured in studies of outcomes of treatment or prevention interventions should reflect the major deficits found in the etiologic research and subject change variables hypothesized. The goal is to improve these risk precursors and strengthen protective factors and processes. Frequent constructs or variables measured include negative peer involvement; school failure; school bonding and attachment; low self-esteem; conduct problems and lack of behavior control; poor social skills; inconsistent and ineffective parenting (monitoring, supervision, discipline, positive rewards); parental and sibling role modeling of substance use or abuse; poor parental mental health; family environment (eg, conflict, communication, cohesion, organization, stress, poverty); and community and cultural environment.
Valid and reliable measures generally are available. Kumpfer and associates75 provide an inventory of measurement instruments by dependent variable useful in prevention-intervention research in their CSAP monograph, Measurements in Prevention: A Manual on Selecting and Using Instruments To Evaluate Prevention Programs. A NIDA research symposium on measurements for family interventions the author organized at Snowbird, UT, in October 1996, identified the most effective parenting measures76 and family measures.42 A discussion also was held of the cultural issues in measurement, which are substantial and are discussed in more detail below.
Alcohol and drug use constructs can be measured in a number of ways: quantity and frequency of use (lifetime, annual, 30-day, and daily); consequences of use; and dependency for different types of alcohol and drugs. Age of first use and regular use are also a useful indicator of risk status since youth who begin use earlier appear to be at higher risk for later serious abuse problems. In addition, expectations to use, positive meta-cognition,23 and alcohol schemas or expectancies are becoming popular precursors of substance use to be measured and precursors to modify in intervention research with COSAs.
These basic risk factors should be reduced significantly by completion of a prevention or treatment intervention. The proposed outcomes should last at least a few years, which means measurement in a longitudinal design. Booster sessions are currently becoming popular as effective ways to extend the effectiveness of prevention or treatment interventions as well as to allow for less costly and more efficient ways to collect longitudinal data on the long-term outcomes of interventions for COSAs.
Covariate Intervention Outcome Measures
In addition to addressing the overall effectiveness of programs specifically for COSAs, research should be directed toward better understanding of which types of clients benefit most from different interventions. It is possible that some prevention or treatment interventions will be differentially effective with different types of parents or youth. Therefore, outcome subanalyses should be conducted by participant covariates to determine whether the COSA interventions are more or less effective for different types of participants or families using post hoc, statistical quasiexperimental analyses as recommended by Cook and Campbell.10 Covariates investigated can include parent and child gender, ethnic status or group, level of parent alcohol or drug use, parental depression, educational status, single versus two-parent families, parent criminal status, the child's baseline level of dysfunction, and program site. Variables found predictive of better preschool COA behavior change because of participation in behavioral parent training include participation by both parents71 and maternal treatment investment.72
Issues in measuring covariates of outcomes for interventions with COAs include locating the best measurement instruments for these individual and program characteristics. Because many of these are family and child demographic measures, it generally is not as difficult to locate valid and reliable measures. The major issue in measuring these demographic variables is sensitivity of the respondent to disclosing this personal information. Hence, missing data from nonresponding is common for measures of variables, such as income, criminal status, parental drug use, and sometime religion, on the baseline needs assessment or pretest.
Health Services Research Intervention Outcome Measures
A strong process evaluation should be designed to examine critical intervention implementation processes to help for new knowledge generation concerning the links between intervention implementation variables and outcomes. These analyses can be accomplished by comparisons of process data with outcome data. The objectives of health services research can include examination of 1) differential recruitment and attrition rates for prevention and treatment interventions across treatment agencies and client characteristics (eg, ethnicity, level of alcohol abuse, child dysfunction level, etc); 2) variables leading to increased program involvement; 3) differential consumer satisfaction and participation rates compared with outcomes; 4) factors related to fidelity of the program implementation between treatment agencies; 5) the impact of trainer variables (eg, years of experience, delivery competence, perceived warmth and supportiveness by clients and evaluators) on program process and outcome variables; and 6) other agency and staff variables by means of force field analyses impacting implementation quality.
Measurement issues in health services research of internal program implementation generally revolve around the development of a management information system (MIS) for documenting program services, client involvement, staff involvement, and units and types of services provided. Model MIS systems can be located by contacting other alcohol and drug treatment agencies who are operating computerized systems on research or Center for Substance Abuse Prevention or Center for Substance Abuse Treatment (CSAT) demonstration/evaluation grants.
One major type of health services research is cost-effectiveness or cost–benefit analysis of the intervention. Administrators of funding sources are increasing their interest in knowing whether the benefits of the intervention outweigh the costs. Despite recommendations made early in the field to conduct cost-effectiveness analyses,77 few quality cost–benefit analyses have been conducted of treatment or prevention interventions.78According to Kim and associates,79 after conducting a retrospective cost–benefit analysis of many substance-abuse prevention programs including those for COSAs using a macrolevel prospective, they concluded that the benefits outweigh the costs of prevention of drug abuse 15:1. However, no prospective, rather than just retrospective, cost–benefit prevention outcome study has yet been conducted. Hence, the substance-abuse prevention and treatment field currently is lacking in prospective cost–benefit studies. To measure these constructs accurately requires developing an MIS to track costs and potential economic cost savings from the beginning of the outcome research.
Results of Outcome Intervention Studies for COSAs
Outcomes of intervention research with COSAs suggest that by changing the children's early family and school environments, risk factors for use can be significantly reduced. One promising area is changing the parent's ability to monitor, supervise, discipline, and reduce negative role modeling of alcohol and drug use. I have found that parent and family skills training can produce immediate positive impacts on these mediating risk factors for alcohol and drug use in elementary school COSA parents.7 Social learning theory80 suggests that youth need exposure to positive adult role models, such as parents, teachers, and COA group leaders, who can provide them with opportunities to learn behavior skills, social competencies, and higher levels of moral thinking.81 Research82 suggests that COAs whose fathers have stopped using alcohol or have no continuing alcohol-related consequences manifest the strongest relationships between self-control reasons for abstaining or limiting drinking and substance use. These COAs perceived more negative effects of alcohol and greater risk for future drug problems if they used.
Results of promising interventions for COSAs are reviewed by Emshoff and Price.69 To measure outcomes in these promising intervention models, the new intervention outcome research is becoming considerably more complex. One of the reasons for this increasing methodologic sophistication is the need to address more risk and protective factors identified in the etiologic research through comprehensive interventions. In addition to more complex measurement models, new intervention research involves more advanced instruments; data collection from multiple informants (children, parents, teachers, and therapists); more advanced process and fidelity measures; and newly emerging data analysis methodology, such as SEM, latent growth modeling, hierarchical linear modeling, and other newly emerging statistical methods not used in previous research.
To be competitive, new outcome studies must propose to have follow-up data for up to 3 years within their 5-year grant period. If intervention research is targeted to young children between 7 and 11 years of age, by the end of the longitudinal study the children would be 12 to 16 years of age, which is old enough to expect the use of tobacco, alcohol, and other drugs in such high-risk COAs.27 Chassin and Barrera82 have found COAs to have the steepest escalation in their drug use in a 3-year longitudinal study of 246 adolescent COAs and 208 control subjects within the same age group proposed in this study (10.5 to 15.5 years of age). Of course, the older COAs showed the steepest escalation in alcohol and other drug use. Such longitudinal studies require measures that are valid and reliable for children spanning a wide age range within a single longitudinal study.
Some etiologic research suggests parenting and family interventions that improve family conflict resolution, family involvement, and parental monitoring should reduce problem behaviors, including alcohol and other drug abuse.83 Parenting skills training programs are effective in reducing coercive family dynamics84–86and improving parental monitoring.87 Dies and Burghardt,88 in a review of COA prevention programs, report that the majority of school-based COA programs are too short term to address the core issues that trouble COAs. Therefore, to have lasting impact, parents' behaviors toward their children must be modified. Many researchers89 believe improving parenting practices is the most effective strategy for reducing adolescent substance-abuse and associated problem behaviors.
As mentioned above, Zucker90 believes there is no better place to invest in prevention than with parent training programs for high-risk children, such as COAs. Hops37 defined parenting skills programs as those that change a parent's behaviors in three critical areas: 1) modeling of negative behaviors, such as alcohol misuse; 2) failure to reinforce or reward positive behaviors in children; and 3) failure to organize children's lives to provide opportunities for them to learn prosocial skills and competencies. SFP supports improvements in all three of these areas.
Need for Culturally Tailored Family Intervention Programs
Research suggests that culturally appropriate prevention interventions are more effective.91 Moran92reports that we need specific prevention approaches, not just generic approaches that currently dominate the prevention profession. The Strengthening Families Program for COSAs has been culturally modified and evaluated in separate CSAP demonstration/evaluation projects for rural African-American COSAs (Alabama), urban African-American COSAs (Detroit),93,,94 Hawaiian COSAs,95Hispanic COSAs (Denver),96 and rural preteens in Iowa.7,,70 Results of the comparison of the generic SFP with minor cultural modifications compared with a substantially revised SFP showed that the first version with minor cultural modifications was more effective. The possible reason for this counterintuitive result is that the dosage of the behavioral parenting, family, and children skills training component was reduced from 14 to 10 sessions to add 10 sessions on family values. Possibly, by reducing the behavior change sessions, the revised program becomes less effective in behavioral change. Hence, core content of model programs shown to be effective should not be removed when making cultural revisions.
CONSIDERATIONS IN SELECTING OUTCOME MEASURES
What Should Be Measured?
The variables hypothesized to change because of the intervention (the independent variable) are the most important dependent variables to measure. Those hypotheses should match the evaluation questions to be answered by the program evaluation or intervention research. Hence, to determine the impact of the program on actual tobacco, alcohol, and drug use, these ultimate outcomes should be measured. Whether to measure them extensively or only briefly depends on the age and expected use levels. For example, with young children, it might not be best to request information on a full range of possible drugs that could be used, but only the gateway drugs or drugs primarily used by the parents. Also, lifetime use or annual use rates may be more useful than daily use rates that are likely to be zero.
If the hypothesis is that the program content should change other intermediate risk or protective variables, these should be measured as well. For instance, if the hypothesis is that the content or curriculum of the intervention changes discipline and parent/child relationships, then these also should be measured. Be sure that the program has sufficient dosage (ie, total number of contact hours including practice and required homework) to clinically change these hypothesized variables. In general, hours of child or parent education are less likely to result in measurable behavioral changes than hours spent in behavioral skills training. It is also wise to measure unintended outcomes as well that could occur—both positive and negative effects based on suggestions in the research literature or anecdotal evidence.
Other data to be collected in the testing battery should include demographic data and covariates that could affect who benefits most from the intervention. Some prevention interventions that appear to have no overall effectiveness on COAs are indeed effective for a subset of children, but this is only discovered through analysis by these separate groups. For instance, Dielman and associates97found that their alcohol prevention program was most effective for a subgroup of students who were allowed to drink at home. These youth had the steepest rise in alcohol use rates and benefited the most if they participated in the school-based program.
In an attempt to improve comparability of results in etiologic and intervention studies, Johnson and associates4 recommended six areas or causal links of risk for alcoholism in COAs that should be measured. These risk factors were derived from the research of Zucker and Fitzgerald98 and include:
Antisocial behavior or aggression;
Poor school achievement and performance;
Lack of family, school, and peer bonding and affiliation;
Family and marital conflict;
Dysfunctional parent-child interactions (eg, inadequate or lax parental supervision, poor parent-child relations, inadequate contact); and
Inadequate role models.
In addition to risk factors, protective factors and resilience should be measured because increasing research suggests that the primary determiners of developmental outcomes are these positive environmental buffering or moderating influences.53,,99Important environmental protective factors to measure as recommended by Kumpfer and Bluth53 and Cowen and associates100 include:
Parental love, care, and supportiveness;
Extra-familial support (eg, teachers, clergy);
Appropriate developmental expectations;
Opportunities for meaningful family, school, and community involvement and rewards;
Support for dreams and goals; and
Setting nonuse rules and norms.
The important factors to measure in resilience can be derived from resilience research with COAs 29,,101 or adult COAs.99 Measures for resilience can be found in Wolin and Wolin,101 Dunn,102 and Walker.103
Special Issues in Measurement with COSAs
Measurement of outcomes of interventions for COSAs is much the same as that for other children. Hence, measurement reference books41 can be used to determine appropriate measures for COSAs. There are a few specific issues that need to be addressed with COAs and COSAs, however.
Lack of Trust in Confidentiality of the Data
The major measurement issues that are specific for COSAs include validity of the data concerning sensitive issues because of fear of disclosure of negative family dynamics or parental drug use. COSAs live with societal shame of their parent's drug use. If they have any mistrust of the confidentiality procedures used in the data collection that is supposed to ensure that the data are protected, they will not disclose fully any negative behaviors or family issues.
Denial of Family Drug Use or Failure to Know
Some COSAs may not self-disclose that they are COSAs, because they genuinely do not know their parents are drug users or abusers of alcohol. Many parents who abuse drugs try to make sure their children do not know they are drug users. If the children are young and the parents are generally functional, there is almost no way for the children to know unless some adult tells them. In older children, they may have some idea that something is wrong, but deny to themselves and others that their parents are drug users. If children do not know or deny they are children of drug users, there is little hope of attracting them into special COSA interventions, unless the parents are recruited to volunteer the children.
Lack of Honesty in Self-reporting
Even if the child is aware of one or both of their parent's drug or alcohol abuse, they still may minimize the extent of the damage on their family environment or own psychological and emotional status because of the stigma involved in self-reporting this information.
Fear of Reprisal
Child abuse, child neglect, and sexual abuse104 are more common in families in which the parents are alcohol- or drug-involved. Children are not likely to report such information if they fear their parents will be reported to protective services or to the police. Professional clinicians have a “duty to warn” the parents and the children that self-disclosure of abuse during the measurement battery or the treatment or intervention discussions will result in their being reported to authorities. A “mini-Miranda” warning should suffice in which they are told that clinicians have to report sexual abuse or neglect depending on their professional standards and state standards. Then the children or parents can decide what they want to disclose and choose to do so based on complete information on what the consequences will be. Unethical practice in failure to warn children can result in children reporting their parent's drug use to DARE police officers, believing that the police will simply help them to get treatment rather than arrest them.
Developmental Issues in Measurement
The selection of measures used in interventions for COSAs also should match the cognitive and emotional stage of development of the child. Children go through stages of cognitive and social development, therefore, evaluation instruments and methods should be tailored to the developmental level of the targeted population. If young children are included in the evaluation, it is often best to conduct individual interviews with them—reading them the questions and the answers. Depending on their age and fear of disclosure it may be best to have them confidentially record their answers rather than tell the interviewer their answer. If the children can circle the correct answer or put their answer on an optical scan sheet, the best method for data collection may be small group interviews with confidential recording of answers to reduce data collection costs. This is a particularly useful method of data collection if the children are participating in a group intervention.
The reading and conceptual levels of the children and the parents also should be considered. If respondents are expected to read the questions or even program homework, a Woodcock Johnson Reading Test can be used to check on their reading level. Even if the children or parents test at a certain reading level, they may not always understand the words they are reading. Some young children are able to read phonetically, but have little idea of the meaning of the words. Check the published reading levels of measures under consideration for use. It also is wise to field-test, in pilot-testing or focus groups, any proposed instruments to determine whether participants really understand the questions.
Also, consider the length of the test and the attention span or activity level of those being tested. Children's activity levels vary greatly, particularly in COSAs who are at higher risk for hyperactivity or attention deficit disorder. Pilot-test the testing methods and the length of the sessions, and then revise the test accordingly.
It is frequently difficult to get valid and reliable data from children younger than age 9 years. Be sure that selected measures are measures that have reasonable α reliabilities for the youngest children in the intervention study. Always pilot-test the instruments and calculate the Chronbach α reliabilities73 by age groups after the first large scale testing. If the reliabilities are low for the youngest children, it may indicate that it is best not to use that data in the outcome evaluation. These data may be clinically useful, particularly at intake, to help determine whether the children need referrals for additional services or to help the provider become aware of special issues in the children. And to include the younger children in the testing, even if their data will not be able to be used in the evaluation, shows consideration for their feelings and ideas.
Culturally Appropriate Measurement Issues
If ethnicity of the COSAs participating in the intervention is a factor, there are additional conceptual, language, and data collection measurement issues. Unfortunately, few standardized instruments or research instruments have been created for use specifically with minority populations. Consequently, few of the standardized instruments widely available have been tested for cultural appropriateness and sensitivity. Some of these special measurement issues with different minority groups, summarized and discussed by cultural measurement specialists at an NIDA Symposium on Measurement Issues (October 13–15, 1996, Snowbird, UT), organized by Dr Rebecca Ashery and me, are summarized below.
Measurement Experiences With African-Americans
According to William Turner, because language is not a major issue, the primary measurement issue for African-American youth and parents appears to be conceptual differences in the constructs used. For instance, the concept of the family can be very different for an inner city, African-American living with a mother on welfare. Many of these cultural issues are actually issues that accrue because of differences in income, living standards, and community environment. Hence, many risk factors associated with inner city, poor African-American families occur not because of unique cultural differences, but because of the realities of growing up in poverty.105–107 African-American researchers in the field have challenged the notion that African-American youth are high risk for alcohol and drug abuse when the high school senior survey data show they have lower use rates than do white or Hispanic youth.108 Still, the stereotypes of drug-abusing African-American youth persist. Much of the association is more with poverty and need to earn money as a drug dealer than with racial status. Hence, economic status and community climate should be measured in research with African-American youth.
Measurement Experiences With Asian-Americans
At the same NIDA conference, Davis Ja and Shu Cheng explained that with Asian and Pacific-Islanders there are a number of language, conceptual, and lifestyle issues, as well as responding issues. These cultural groups are very heterogenous and in many Asian and Pacific-Islander intervention programs, youth and families from many different cultural and ethnic groups are clustered together. This can make testing very difficult, because it is difficult to have native language speakers or different versions of the testing battery for all ethnic or language groups. If possible, have the test translated into the native language with both forward and backward translation and checking by several other native speakers. Read the questionnaires and have other native speakers available to answer questions on concepts or words. Most Asians with a reasonable education level can record answers on optical scan sheets, but we have found this difficult for Pacific-Islanders.
Because of recent negative experiences with repressive governments, some Asian youth and families are even less trustful than are COAs. They are less likely to divulge negative family or personal information until they have been in the intervention and begin to trust the staff and data collection. One possible solution for this problem is to collect a retrospective pretest at the time of the posttest, or earlier, when the staff feel the clients trust confidentiality. This data collection methodology has been used effectively by researchers for sensitive drug use information with students in schools. If this is not done, the subjects will disclose more negative information on the posttest than on the pretest, making the intervention look like it had negative results. Our experience is that Asian youth and parents are willing to disclose high levels of depression and mental health problems related to stress and acculturation difficulties on the pretest, but will not disclose information about drug use and harsh discipline.
Measurement Experiences With Native-Americans
At the same conference, Dan Edwards presented and emphasized that Native-Americans are not homogeneous. There are >1000 tribes, both officially recognized and not officially recognized. There are many major issues in the collection of data with Native-American youth and families. One is even getting tribal approval for data collection. Although they may allow the COA or COSA intervention to occur, they may ban the collection of data fearing misuse of even the aggregated data. Suspicion of social research is warranted, based on years of exploitation of Native-Americans by researchers. In addition, Native-Americans, like other ethnic groups, may not understand why the researcher needs to know personal, private information. Fortunately for prevention program evaluators, most COSAs participating in the interventions do so because they believe the services have value to them. Thus, they generally are more invested in cooperating and do try to provide valid data. Avoiding collecting personally offensive information in the pretest or having excessively long testing batteries will increase cooperation.
Measurement Experiences With Hispanics
The presenters Martin Arocera and Rose Alvarado said that like the other major ethnic groups mentioned, Hispanics are a diverse ethnic group with major differences in culture from European Spanish families to Caribbean and South American Spanish-speaking youth from indigenous tribes. Because of these major cultural differences, it is difficult to translate testing batteries into a single Spanish language version. Words and concepts differ across different Hispanic cultural groups. Immigrant, migrant, and illegal Spanish-speaking persons also are unlikely to disclose information about parental drug use or family dynamics that could be considered a legal problem.
Like other disenfranchised or traumatized ethnic groups, it is very difficult to get valid and reliable data. The retrospective pretest is one possible solution. Another possible solution is to postpone data collection until the youth or parents are more trusting by holding preintervention get-acquainted sessions.
Another cultural issue in measurement and prevention interventions is the need to get permission for the child to participate from the father and possibly other cultural leaders. It is important to get their approval on the child's participation in the testing even if the father is not participating.
Overall Measurement Issues With Low-income, Low-education Participants
Because of the generally lower income levels of minority families, many measurement issues that derive from low education levels are sometime confused with cultural issues. For instance, it is harder to get high internal consistency of items with low-income children from minority backgrounds. One factor mentioned by Kumpfer and associates41 is that poor physical health, lack of medical care, and poor nutrition (such as lack of breakfast before the testing) can cause inconsistent performance on tests by affecting attention span, concentration, motivation, and even vision and hearing.
Gender-sensitive Issues in Measurement
There has been very little attention to gender issues in prevention intervention for drug use. Unlike in the drug treatment field, which has been perfecting “woman-centered” or women-only treatment strategies, no prevention programs currently exist based on gender relevance. Several NIDA researchers have been funded to conduct research to develop prevention strategies based on women's issues, such as pressure to use drugs in sexual encounters. Other major women's issues to be addressed in women's prevention programs include child and sexual abuse, which is more common in young girls. Measurement issues include an increased tendency in females to respond in socially desirable ways, possibly because of increased denial. One solution for this issue is to include a social desirability scale within the testing battery. Another recommendation is to consider putting less emphasis on risk factors or deficits and focusing more on protective factors or strengths.63 Females are more likely to respond favorably to instruments that measure family strengths rather than deficits. When it was difficult to get disclosure on risk factors with pregnant women for drug use, I had more success in measuring program intervention outcomes by creating a Family Strengths Assessment.109
Issues in the Selection of Recommended Measures
There are a number of issues in the selection of the best measures for interventions and the development of effective testing batteries. Thoughtful selection from available measures can be guided by psychometric principles recommended by Achenbach and McConaughy110 to use 1) standardized measures and procedures; 2) multiple, aggregate items or scales for each hypothesized variable; 3) normed instruments; and 4) instruments with demonstrated high reliability and validity with similar populations. Johnson and associates4 also recommend: 1) following the principles recommended in the APA Standards for Educational and Psychological Testing; 2) using test construction designed only by specialists in this profession; 3) using trained data collectors; and 4) when interviewing, avoiding leading, prejudice, and bias.
Use of Standardized, Core Instruments
Whenever possible, it is better to use instruments that already have been developed and used in similar program evaluation. It is very difficult to develop original instruments. With off-the-shelf instruments, the findings can be compared more easily with those of other intervention programs, a practice encouraged by researchers in this field and by state and federal funders.4,,41,111
Even if major cultural modifications are needed, generally it is better to start with the best known standardized measure or scale and then modify it based on focus groups and pilot-testing to make it more appropriate to the target population. If the decision is made to create an original, working with an experienced instrument development specialist is recommended.
The Development of Testing Batteries
Once the major hypothesized change variables are selected, the next step is to select the shortest and most valid and reliable scale measuring each construct for the target population. Because of the need to measure changes in many different risk and protective factors, intervention researchers are struggling with getting the largest α reliability values with the smallest number of testing questions or items per dependent variable. Unfortunately, the shorter the number of items in a scale, the more difficult it is to get acceptable internal consistency or α values higher than 65. Also, with children younger than age 9, the internal consistencies, or α values, become lower. A low correlation coefficient, which indicates reliability or stability of the measure, indicates greater measurement error or unwanted variation from the true measurement of the respondent. Errors in measurement can be caused by poor instrument design (eg, ambiguity of items, unclear instructions, unclear concepts or wording, confusing formatting, and language or reading difficulties).
To reduce testing burden on the participants, it is important to have them complete only the scales in a multiscale inventory that will be used in the data analysis. Often this means creating testing batteries that include only the specific scales to be used in the intervention outcome study. In the creation of the testing battery, the ordering of testing items is important. It is better generally to begin with positive questions that children like to respond to, such as questions about their friends, their opinions, and their school. Information on critically needed sensitive items is best placed in the middle of the test. Items that are least important can be put at the end of the test, particularly if the test is long. Because there will be more missing data at the ending of the test, these data can become lost; thus, be sure they are not the major data.
Some tests are very easy for children and parents to follow, whereas others are difficult and confusing. Review potential tests with this in mind and then pilot-test with a subsample of evaluation participants. Respondents should be asked to rate the ease of the questions and answer choices and whether the test instructions are clear. Very difficult formatting for participants are those that require two different types of responses for each question. Using the least amount of words to ask and answer the questions also helps if the formatting is clear.
Data Collection Issues
There are a number of data collection issues that must be considered in the selection of the best instrument, such as:
Whether to rely on self-report or to also use direct observation and possibly videotaped behavioral interactions, which then involves selecting the best video coding scheme. Video coding also is very costly but is considered valuable “hard” or objective data in documenting behavioral improvements in children's behaviors and family interactions.
Where to collect the data must be determined. Some researchers prefer to collect data in the intervention group, some in homes, and some in their offices. These interact with whether individual interviews will be conducted or small group interviews supplemented with questionnaires. Take-home questionnaires, if the participants have high reading and motivation levels, is also possible, but generally not considered the best method of data collection.
How to record the answers involves a choice between optical scan sheets and direct marking for either open-ended or closed-ended questions. The fastest to computerize with the most objectivity are the closed-ended questions put on optical scan sheets confidentially by the participants. In ethnic or low-income participants, recording answers on a different sheet from the questionnaire can be confusing. Hence, it is better for them to circle the appropriate answer and to have staff enter the data manually.
Some testing batteries are computerized, with branching programs that allow the respondents to answer only those questions that are applicable to them. However, in program evaluation research, in which the testing battery must be created to match the hypothesized change variables, it is unlikely that any of these computerized programs will be the best measurement methodology. It is possible that in the future, a standardized, computerized instrument will be created that will contain the best core measures for prevention interventions.
HOW TO SELECT THE BEST MEASURES
Issues to consider when selecting the actual scales for each dependent variable include:
Selecting tests for different data sources, or informants, in a multiinformant measurement model. Some tests have different versions with variations in wording depending on the informant—the child, parent, therapist, or teacher.
Length of the test and testing session. The shortest test or scale with the highest internal consistency and validity generally is the best test to select for each variable. If the test is too long when all variables and tests/scales are combined, it may be necessary to prioritize and remove some scales or to conduct the testing in two sessions.
Popularity and previous/current use of the proposed instruments with similar populations. If an instrument is used in other similar intervention outcome studies, it will be easier to compare results. Having norms for a similar target population on non-COSAs also is helpful in data interpretation.
Sensitivity to change. According to Dishion and associates,8 “Over-reliance on the personality assessment strategy has had a deleterious impact on measurement strategies that are sensitive to change.” Most clinical diagnostic instruments are not very change-sensitive, as are testing items that measure “lifetime prevalence.” Having a 5- to 7-point Likert scale rather than just a true/false response allows for more gradation in improvement or change. Still, these measures lack temporal specificity that would permit the researcher or clinician to determine when changes had occurred between measurement points.
Validity of the construct for the target population. The most important characteristic of a test is validity. A test that does not measure what it is supposed to measure is of little use. Review the published data on validity, and also look at the items to see whether the construct or concepts are understandable and valid for their realities.
Language versions. If there are non-English speakers in the intervention research, other language versions will need to be located or created. Even with standardized tests that have versions in other languages, modifications may be needed for the tests because of intraethnic differences in concepts and in wordings or colloquial usage.
Cost of the instrument. Another consideration is the cost of the instrument. Many standardized tests are copyrighted and must be purchased from the publisher or author. However, many equally good measures developed for prevention research may be available from researchers directly, at little or no cost.
Resources for Selection of Best Measures
The best measures to use depend on the type of assessment desired: etiologic research, prevention interventions, diagnosis, treatment planning, or outcome measurement.
Etiology and Prevention Intervention Measures
Currently, the best measurement resource guide for the selection of measures for prevention interventions is Measurements in Prevention: A Manual on Selecting and Using Instruments to Evaluate Prevention Programs,75 which I developed. An updated, computerized CD-ROM version of the manual is planned in conjunction with Dr William Hansen of Tanglewood Research CSAP has convened five task forces to select the best measures for the most critical outcomes in prevention and are developing an Internet-based measures selection system as part of a larger prevention decision support system or expert system.
Chapters by Liddle42 and McMahon76summarizing the results from the NIDA Measurement Symposium on family measures unfortunately are not available. In lieu of this information, see Table 2, which contains many of the best research instruments for many of the variables that one would want to measure in prevention-intervention research.
Table 2 covers measures by constructs/outcome variables by source of information or informant (child, teacher/trainer, and parent). Although the Table is not inclusive of all the best measures, it provides a good selection of a large number of measures for the following constructs:
Best Substance Use/Abuse Measures in areas of incidence and prevalence, expectations to use, family history, parent use, peer use, and more.
Best Child Behavioral Change Measures in areas of conduct disorders, aggression, social withdrawal, anxiety, social skills, depression, self-esteem, child abuse, neglect, and more.
Best Parent Measures in areas of knowledge of discipline principles, discipline style, monitoring and supervision, communication, and more.
Best Family Functioning Measures in areas of family communication, relationships, attachment, conflict, family strengths, organization, and more.
Best Community/Culture Measures in areas of neighborhood cohesion, cultural pride and identity, community problems (crime, norms toward alcohol and drug use), and more.
Diagnosis and Treatment Assessments
Johnson and associates4 also provide a listing of recommended standardized instruments for each of ten areas of functioning included in the Comprehensive Assessment Battery of the NIDA Adolescent Assessment/Referral System (AARS).112These recommended measures were derived from recommendations from national experts in adolescent assessment and treatment. Hence, it should be remembered that these are measures recommended primarily for clinical assessments, not necessarily for prevention interventions. The AARS manual provides descriptions of the instruments, along with information on how to obtain them, administration time, and cost.
Human Subjects Measurement Issues
Before gathering data from children, the parent and possibly the children, should sign an informed consent form that includes a complete disclosure of their rights as human subjects. These rights include the right to not answer all questions, to stop at any time, to choose not to participate at all in the intervention or data collection without loss of other services provided normally, and to have their data remain confidential.
If longitudinal data are collected, as is needed for pre- and posttest data collection for outcome research on interventions, it is necessary to include subject codes. Because of the problems with trust in confidentiality in COSAs, confidentiality in coding must be considered seriously. One way to increase confidentiality is to have the children develop their own code based on a formula developed by the research team. These coding schemes can include the day of the mother's birth date, her middle initial or maiden name, the child's middle initial, and other less than obvious or difficult-to-track data. The problem with using this scheme is that one does have more unmatched testing batteries.
Another scheme is to put the investigator-derived codes for each child on the answer sheets, but then to put their names on the envelopes on post-its that can be removed by the child when they are handed the questionnaire and answer sheet. This procedure enhances that data will match and also the likelihood that the child will believe that the data collected will remain confidential.
Other Ethical Issues
One major ethical issue is locating control or comparison groups for the intervention for COSAs. The hallmark of outcome effectiveness research in clinical trials is random assignment of volunteers either to the intervention or to a no-treatment control or comparison group. If the intervention already has proven effectiveness with the target population to be studied, it is unethical to randomly assign the COSAs to a no-treatment group. However, for purposes of most prevention-intervention research, the intervention would not be studied in a randomized clinical trial if it had already been proven to be effective with the target population.
Recommendations for Practice Guidelines in the Measurement of COSAs
The overall recommendations are that as much attention needs to be paid to the development of valid and reliable measures as to the development of effective prevention interventions. If the measurement model or testing battery and data collection methods are not considered as important as the intervention services, it will not be possible to conduct solid research on program effectiveness to improve the outcomes in the prevention field.
Unfortunately, measurement has been neglected, particularly the development of geographically and culturally valid and reliable measures for minority children and families. Nothing is more important to prevention research now than a major initiative to invest research funding into the establishment of developmentally, culturally, and gender-appropriate measures. Johnson and associates4 aptly summarized recommendations for this area: “What is needed are consensus building workshops with COA researchers, clinicians, and expert advisors to outline the ideal assessment battery for COAs.”
A prevention-intervention subcommittee of the Wolin Consensus Forum on Children of Alcoholics also made this recommendation and expressed willingness to use standardized and shared measures to improve comparison of results. However, no such COA or COSA testing battery has yet been developed.
The other major practice recommendation is to take measurement and evaluation seriously and not to consider it as something that draws down funds for direct services. Although clinicians and prevention practitioners frequently are sure that their work is effective, they also are wrong sometime. In some cases, measurement research can help the practitioner learn how to improve the prevention intervention or identify which modules to drop. Better measurement and research will help professionals determine which interventions work best and help weed out ineffective interventions.
- COSAs =
- children of substance abusers •
- COAs =
- children of alcoholics •
- MTMM =
- multitrait–multimethod •
- SEM =
- structural equation model •
- NIDA =
- National Institute on Drug Abuse •
- FES =
- Family Environment Scale •
- SFP =
- Strengthening Families Program •
- BASC =
- Behavioral Assessment Scale for Children •
- CBCL =
- Child Behavior Checklist •
- MIS =
- management information system
- ↵Kumpfer KL. Special populations: etiology and prevention of vulnerability to chemical dependency in children of substance abusers. In: Brown BS, Mills AR, eds. Youth at High Risk for Substance Abuse. Rockville, MD: National Institute on Drug Addiction Research Monograph; 1987
- ↵Tarter R, Mezzich A. Ontogeny of substance abuse: perspectives and findings. In: Glantz M, Pickens R, eds. Vulnerability to Drug Abuse. Washington, DC: American Psychiatric Association; 1992
- ↵National Institute on Drug Abuse. National Survey Results on Drug use:Monitoring the Future 1975–97. Washington, DC: US GPO; 1998
- ↵Johnson JL, Rolf JE, Tiegel S, McDuff D. Developmental assessment of children of alcoholics. In: Abbott S, ed .Children of Alcoholics: Selected Readings. Rockville, MD: National Association of Children of Alcoholics; 1996
- ↵Institute of Medicine. Reducing Risks for Mental Disorders. Frontiers for Preventive Intervention Research. Washington, DC: National Academy Press; 1994;
- ↵Adger H Jr. Children of alcoholics: the role of the primary care physician. In: Abbott S, ed. Children of Alcoholics: Selected Readings. Rockville, MD: National Association of Children of Alcoholics; 1996
- ↵Kumpfer KL. Selective prevention approaches for drug use pre-vention: overview of outcome results from multi-ethnic replications of the Strengthening Families Program. In: Ashery R, Robertson E, Kumpfer KL, eds. Family Intervention Research for the Prevention of Drug Use in Youth. Rockville, MD: NIDA; 1998
- ↵Dishion TJ, Li F, Spracklen K, Brown G, Haas E. The measurement of parenting practices in research on adolescent problem behavior: a multimethod and multitrait analysis. In: Ashery R, Kumpfer KL, eds. Family Intervention Research for the Prevention of Drug Use in Youth. Rockville, MD: NIDA; In press
- ↵Dwyer JH. Statistical Models for the Social and Behavior Sciences. New York, NY: Oxford University Press; 1983
- ↵Cook TD, Campbell DT. Quasi-experimentation: Design and Analysis Issues in Field Settings. Chicago, IL: Rand–McNally; 1979
- Fiske DW
- ↵Newcomb MD. Drug use etiology among ethnic minority adolescents: risk and protective factors. In: Botvin G, Shinke S, Orlandi MA, eds. Drug Abuse Prevention with Multiethnic Youth. Thousand Oaks, CA: Sage Publications; 1995
- Hawkins JD,
- Arthur MW,
- Catalano RF
- ↵Wright KN, Wright KE. Family Life and Delinquency and Crime: A Policy-Makers Guide to the Literature. Unpublished manuscript. Washington, DC. Department of Justice, Office of Juvenile Justice & Delinquency Prevention; 1992
- ↵Johnson JL, Leff M. Overview of the research on children of substance abusers. Presented at the National Association for Children for Alcoholics Pediatric Initiative Conference; 1997; Alexandria, VA
- ↵Kumpfer KL, Turner CW. The social ecology model of adolescent substance abuse: implications for prevention. Int J Addictions. 1990–1991;25:435–463
- ↵Oetting E. Planning programs for prevention of deviant behavior: a psychosocial model. In: Trimble JE, Bolek CE, Niemcryck SJ, eds. Ethnic and Multi-cultural Drug Use: Perspectives on Current Research. New York, NY: Haworth Press; 1992
- ↵Newcomb MD. Understanding the multidimensional nature of drug use and abuse: the role of consumption, risk factors, and protective factors. In: Glantz MD, Pickens R, eds. Vulnerability to Drug Abuse. Washington, DC: American Psychological Association; 1992
- ↵Zucker R, Kincaid SB, Fitzgerald HE, Bingham CR. Alcohol schema acquisition in preschoolers: differences between children of alcoholics and children of nonalcoholics. Alcoholism: Clin Exp Res. 1995;19:1011–1017
- ↵Gaines L. Children's beliefs about drinking. Presented at NIAAA Research Symposium; 1996; Alcohol and the Family: Opportunities for Prevention. Washington, DC
- ↵Oetting ER, Beauvais F, Edwards RW. Alcohol and Indian youth: social and psychological correlates and prevention. J Drug Issues. 1988;:18:87–101
- Johnson JL,
- Sher KJ,
- Rolf JE
- ↵McCubbin HI, Thompson AI, McCubbin MA. Family resilience and resiliency resources: topologies of patterns of functioning. In: Family Assessment: Resilience, Coping, and Adaptation—Inventories for Research and Practice. Madison, WI: University of Wisconsin System; 1996
- ↵Brook DW, Brook JS. Family processes associated with alcohol and drug use and abuse. In: Kaufman E, Kaufman P, eds. Family Therapy of Drug and Alcohol Abuse. Boston, MA: Allyn and Bacon; 1992
- ↵Brook JS, Cohen P, Whiteman M, Gordon AS. Psychosocial risk factors in the transition from moderate to heavy use or abuse of drugs. In: Glantz M, Pickens R, eds. Vulnerability to Drug Abuse. 1992
- ↵McCord J. The Cambridge–Sommerville Study: A pioneering longitudinal-experimental study of delinquency prevention. In: McCord J, Tremblay R, eds. Preventing Antisocial Behavior. Interventions from Birth to Adolescence. New York, NY: Guilford Press; 1992
- ↵Gerrard M. Psychosocial predictors of rural adolescent alcohol use. Presented at NIAAA Research Symposium: Alcohol and the Family: Opportunities for Prevention; 1996; Washington, DC
- ↵Oregon Social Learning Center. Family Activities Checklist. Unpublished assessment instrument. 1984 (Available from Oregon Social Learning Center, 207 E 5th St, Suite 202, Eugene, OR 97401)
- ↵Hops H. Pathways to alcohol and other drug use. Presented at NIAAA Research Symposium: Alcohol and the Family: Opportunities for Prevention; 1996; Washington, DC
- ↵Tolan PH, Gorman-Smith D, Zelli A, Huesmann R. The Chicago Family Measure. Poster presentation at 51st Annual Meeting of the American Association for Marriage and Family Therapy; 1993; Anaheim, CA
- ↵King J, Beals J, Manson SM, Trimble JE. A structural equation model of factors related to substance use among American Indian adolescents. In: Trimble JE, Bolek CS, Niemcryck SJ, eds. Ethnic and Multi-cultural Drug Abuse. Perspectives on Current Research. New York, NY: Haworth Press; 1992
- ↵Baer P. Alcohol, psychosocial factors, and adolescent development. Presented at the NIAAA Research Symposium. Alcohol and the Family: Opportunities for Prevention; 1995; Washington, DC
- ↵Kumpfer KL, Alvarado R, Turner C, Griffin E. A preliminary predictive model of alcohol and other drug use for Hispanic adolescents. Presented at the Center for Substance Abuse Prevention's 1993 National/International Prevention Conference—New Dimensions in Prevention: Sharing Today, Shaping Tomorrow; 1993; Washington, DC
- ↵Liddle HA. Family measurements for substance abuse programs. In: Ashery R, Kumpfer KL, eds. Family Intervention Research for the Prevention of Drug Use in Youth. Rockville, MD: NIDA; In press
- ↵Dishion TJ. Behavior observations and assessment methods. Presented at the NIDA Conference: Drug Abuse Prevention through Family Interventions; 1996; Gaithersburg, MD
- Hansen WB,
- Malotte CK,
- Collins LM,
- Fielding JA
- ↵Ary D, Duncan T, Biglon A, Metzler C, Noell J, Smolkowski K. Developmental model of adolescent problem behavior: the influence of parents and peers. J Abnorm Psychol. In press
- ↵Dishion TJ, Gardner K, Patterson GR, Reid JB, Thibodeaux S. The Family Process Code: A Multidimensional System for Observing Family Interaction. Unpublished coding manual. 1983 (Available from Oregon Social Learning Center, 207 E 5th Ave, Suite 202, Eugene, OR 97401)
- ↵Dishion TJ, Soberman L. PEN-P. Unpublished training manual 1994 (Available from Oregon Social Learning Center, 207 E. 5th St., Suite 202, Eugene, OR.)
- ↵Kumpfer KL, DeMarsh JP, Child W. Strengthening Families Program: Children's Skills Training Curriculum Manual, Parent Training Manual, Children's Skill Training Manual, and Family Skills Training Manual for Prevention Services to Children of Substance-abusing Parents. Salt Lake City: UT: Social Research Institute, Graduate School of Social Work, University of Utah; 1989
- ↵Kumpfer KL. Strengthening Families Program Parenting Questionnaire; 1989; Available from University of Utah, Department of Health Education, 215 HPER-N, Salt Lake City, UT 84112
- ↵Catalano RF. Raising Healthy Children. Seattle, WA: Social Work Department, University of Washington; 1993
- ↵Kumpfer KL, Alvarado R. Strengthening families to prevent drug use in multi-ethnic youth. In: Botvin G, Schinke S, Orlandi M, eds. Drug Abuse Prevention with Multi-ethnic Youth. Thousand Oaks, CA: Sage Publications; 1995
- ↵Kumpfer KL, Bluth B. Transactional parent/child relationships and impact on resilience for substance abuse. In: Johnson J, McDuff DK, eds. The Chronicity of Substance Abuse. Baltimore, MD: Harcourt; In press
- ↵Kumpfer KL. Factors and processes contributing to resilience: the resiliency framework. In: Glantz M, Johnson J, Huffman L, eds. Resiliency and Development of Positive Life Adaptations. New York, NY: Plenum Press; In press
- ↵Werner EE, Smith RS. Overcoming the Odds: High Risk Children from Birth to Adulthood. Ithaca, NY: Cornell University Press; 1992
- ↵Gottfredson G. Effective Schools Battery. College Park, MD: Department of Criminology, University of Maryland; 1990
- ↵Gresham FM, Elliott SN. Social Skills Rating System Manual. Circle Pines, MN: American Guidance Service; 1990
- ↵Achenbach TM, Edelbrock C. Child Behavior Checklist (CBCL). Burlington, VT: Center for Children, Youth, & Families, University of Vermont; 1988
- Sameroff A,
- Seifer R,
- Barocas R,
- Zax M,
- Greenspan A
- ↵Dunst CJ, Trivette CM. Methodological considerations and strategies for studying the long-term follow-up of early intervention. In: Friedman S, Haywood HC, eds. Developmental Follow-up: Concepts, Domains and Methods. San Diego, CA: Academic Press; 1994
- ↵Rose R. A behavior–genetic perspective: transmission within and across generations. Presented at the NIAAA Research Symposium. Alcohol and the Family: Opportunities for Prevention; 1996; Washington, DC
- ↵Werner MJ, Joffe A, Graham AV. Screening, early identification and office-based intervention with children and youth living in substance-abusing families. Presented at the National Association for Children for Alcoholics' Pediatric Initiative Conference; 1997; Alexandria, VA
- ↵Emshoff JG, Price AW. Prevention and treatment options for children of alcoholics. Presented at the National Association for Children for Alcoholics' Pediatric Initiative Conference; 1997; Alexandria, VA
- ↵Kumpfer KL, Molgaard V, Spoth R. Family interventions for the prevention of delinquency and drug use in special populations. In: Peters R, McMahon R, eds Preventing Childhood Disorders Substance Abuser, and Delinquency. Thousand Oaks, CA: Sage Publications; 1996
- ↵Chronbach LJ. Essentials of Psychological Testing. 4th ed. New York, NY: Harper and Row; 1984
- ↵Cloninger CR, Sigvardsson S, Bohman M. Childhood personality predicts alcohol abuse in young adults. Alcoholism: Clin Exp Res. 1988;12:494–505
- ↵Kumpfer KL, Shur GH, Ross JG, Bunnell KK, Librett JJ, Millward AR. Measurement in Prevention: A Manual for the Selection and Use of Instruments to Evaluate Prevention Programs. Center for Substance Abuse Prevention, Technical Report #8. Washington, DC: US GPO; 1993
- ↵McMahon R. Measures of parenting constructs for family-based programs. In: Ashery R, Robertson E, Kumpfer KL, eds. Drug Abuse Prevention Through Family Interventions. NIDA Research Monograph 177, DHHS, NIDA, Rockville, MD: NIH Pub. No. 97-4135. 1998
- ↵Apsler R. Evaluating the cost-effectiveness of drug abuse treatment services. In: Cartwright WS, Kaple JM, eds. Economic Costs, Cost Effectiveness, Financing, and Community-based Drug Treatment. National Institute on Drug Abuse Research Monograph 113. Washington DC: US GPO; 1991
- ↵Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice-Hall; 1986
- ↵Mayer GR. Preventing antisocial behavior in the schools. J Applied Behav Anal. 1995;34–56
- ↵Bry BH, Greene DM, Schutte C, Fishman C. Targeted Family Intervention: Procedures Manual. Unpublished document; 1991
- ↵Zucker R. Alcohol and the family: early development. Presented at NIAAA Research Symposium: Alcohol and the Family: Opportunities for Prevention; 1996; Washington, DC
- ↵Botvin G, Schinke S, Orlandi M. Drug Abuse Prevention with Multiethnic Youth. Thousand Oaks, CA: Sage Publications; 1995
- ↵Moran J. Alcohol abuse among urban Indian adolescents. Presented at NIAAA Research Symposium: Alcohol and the Family: Opportunities for Prevention; 1996 Washington, DC
- Aktan G,
- Kumpfer KL,
- Turner C
- ↵Kameoka VA, Lecar S. The effects of a family-focused intervention on reducing risk for substance abuse among Asian and Pacific Island youths and families: evaluation of the Strengthening Hawaii's Families Project; Honolulu, HI:Coalition for a Drug-Free Hawaii, University of Hawaii; 1996
- ↵Wamberg K. National search and dissemination of model family programs: how to access resources. Presentation at Office of Juvenile Justice and Delinquency Prevention Conference: Strengthening America's Families Conference; 1996; Snowbird, UT
- Zucker RA,
- Fitzgerald HE
- ↵Walker R, Kumpfer KL, Alder SC, Richardson G. Resilience in adult children of alcoholics: empirical model of spirit, mind, and body precursors of healthy life adaptation. Subst Use Misuse. In press
- ↵Wolin SJ, Wolin S. The Resilient Self. How Survivors of Troubled Families Rise Above Adversity. New York, NY: Villard Books; 1993
- ↵Dunn D. Resilient Reintegration of Married Women With Dependent Children: Employed and Unemployed. Salt Lake City, UT: University of Utah; 1994. Doctoral dissertation
- ↵Walker R. Resilience Instrument: Resilience in Adult Children of Alcoholics. Salt Lake City, UT: University of Utah; 1996. Doctoral dissertation
- ↵Kumpfer KL, Bayes J. Child abuse and alcohol, tobacco and other drug abuse: causality, coincidence or controversy? In: Jaffe JH, ed. The Encyclopedia of Drugs and Alcohol. New York, NY: Macmillan Press; 1995
- ↵Boyd-Franklin N. Black Families in Therapy: A Multi Systems Approach. New York, NY: Guilford Press; 1989
- ↵Boyd-Franklin N. Five key factors in the treatment of Black families. In: Saba GW, Karrer BM, Hardy KV, eds. Minorities and Family Therapy. New York, NY: Haworth Press; 1990
- ↵Boyd-Franklin N. Necessary activities to make family interventions culturally specific. Presented at NIDA Conference: Drug Abuse Prevention through Family Interventions; 1996; Gaithersburg, MD
- ↵Wallace JM, Bachman JG, O'Malley PM, Johnston LD. Racial/ethnic differences in adolescent drug use: exploring possible explanations. In: Botvin G, Schinke S, Orlandi M, eds. Drug Abuse Prevention With Multiethnic Youth. Thousand Oaks, CA: Sage Publications; 1995
- ↵Kumpfer KL. Selective prevention approaches for drug abuse prevention: the Strengthening Families Program. Presented at NIAAA Family Research Symposium. Alcohol and the Family: Opportunities for Prevention; 1996; Washington, DC
- ↵Achenbach TM, McConaughy SH. Empirically Based Assessment of Child and Adolescent Psychopathology: Practical Applications. Thousand Oaks, CA: Sage; 1987
- ↵Goplerud E, ed. Breaking New Ground for Youth at Risk: Program Summaries. Washington, DC: US GPO; 1990
- ↵National Institute on Drug Abuse. The Adolescent Assessment/referral System Manual. Unpublished manuscript available: L. Radhert, NIDA, Division of Clinical Research, 5600 Fishers La, Room 10A-30, Rockville, MD 20857; 1987
- Copyright © 1999 American Academy of Pediatrics