Community-wide Intervention to Improve the Attention-Deficit/Hyperactivity Disorder Assessment and Treatment Practices of Community Physicians
OBJECTIVES. The goals were to implement and to test a quality-improvement intervention aimed at improving community-based primary care providers’ adherence to the American Academy of Pediatrics, evidence-based diagnostic and treatment guidelines for attention-deficit/hyperactivity disorder.
METHODS. Nineteen practices (with 84 primary care providers) from a large urban community were trained by using quality-improvement methods with some academic detailing. Pretraining and posttraining adherence to evidence-based practices was assessed through review of patient charts.
RESULTS. Preintervention rates of guideline usage were uniformly low. After the intervention, primary care providers showed substantial improvement in their use of the guidelines for the assessment and treatment of elementary school-aged patients with newly diagnosed attention-deficit/hyperactivity disorder. Use of parent and teacher assessment rating scales increased from levels of 52% to 55% to levels of nearly 100%. Systematic monitoring of responses to medication improved from a baseline level of 9% to 40%.
CONCLUSIONS. Quality-improvement interventions such as the one used in this study seem quite effective in improving primary care providers’ practices at offices that express interest in improving the quality of care for attention-deficit/hyperactivity disorder. The design of the intervention, problems associated with improving and sustaining treatment monitoring, and issues related to generalizability of the intervention model are discussed.
- attention-deficit/hyperactivity disorder
- American Academy of Pediatrics guidelines
- quality improvement
In 2000 and 2001, the American Academy of Pediatrics (AAP) issued evidence-based guidelines for assessing and treating children with attention-deficit/hyperactivity disorder (ADHD).1,2 Written for primary care providers (PCPs), these guidelines summarized the extant literature and made 10 specific, evidence-based recommendations for practice. Assessment recommendations emphasized the importance of collecting parent and teacher rating scale results, using Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV), criteria as the basis for making ADHD diagnoses, and evaluating children for comorbid conditions. Treatment guidelines focused on specifying targeted treatment outcomes and providing systematic follow-up care, including the collection of follow-up parent and teacher rating scale results for quantitative assessment of responses to treatment. These recommendations represent an important effort to standardize the management of ADHD by addressing 2 documented sets of observations, that is, that most children receive ADHD treatment from their PCPs3,4 and that the majority of PCPs do not provide evidence-based assessment and treatment for their patients with ADHD.3,5
Although the AAP guidelines have been widely disseminated and their adoption actively promoted, guideline adherence in general is known to be poor.6,7 A number of models have been proposed to promote adoption of guideline recommendations of evidence-based practices.8–14 Most of these models use collaborative consultation services,15 which promote collaboration between community-based PCPs, psychologists, and psychiatrists. For example, the San Diego ADHD Project10 provides guideline training, assessment materials, and assistance with collection and scoring of rating scales. Such models have been successful in improving AAP guideline compliance, but their reliance on outside services and grant funding makes sustainability problematic.16 In addition, most collaborative consultation service models have involved communities that are anchored by academic centers with ADHD expertise. These financial and geographic limitations pose potentially significant barriers to the widespread adoption of these models.
An alternative model uses quality-improvement methods to promote the adoption of evidence-based practices. In this model, didactic training focuses on the evidence base that underpins each of the specific guideline recommendations. The process-improvement component provides PCPs and their office staff members with a method for studying, modifying, and redesigning office policies and procedures so that guideline-consistent practices are supported. Polaha et al17 successfully introduced this model in 2 pediatric practices and were able to demonstrate improvements in the accuracy of ADHD assessments across all PCPs. The current study describes the implementation and outcomes of such an intervention, which focused on improving both the assessment and treatment practices of an entire community of PCPs.
Participants and Settings
All PCP offices that were listed in telephone directories and were within a 30-minute radius of Cincinnati Children's Hospital Medical Center (209 practices, with 569 PCPs) were mailed a brochure and letter describing the ADHD Collaborative. The intervention model was described, and the requirements for participation were delineated. Two weeks after the mailing was sent out, practices were called by an ADHD Collaborative staff member to discuss interest in participation in the project. If requested, a local opinion leader (Dr Lichtenstein) traveled to the practice, to answer questions about the project and to solicit participation. Fifty-five practices employing 202 PCPs (158 pediatricians and 44 family physicians) voluntarily enrolled in the ADHD Collaborative. Thirty-eight practices declined participation, whereas the remaining 116 practices were not responsive or had scheduling conflicts. Training for the 55 practices was divided into 11 phases, with each phase consisting of 3 to 10 practices, on the basis of when during the 3-year recruitment period they expressed interest in joining the ADHD Collaborative. The first 4 phases (n = 19 practices) pilot-tested the intervention model. Post–pilot-testing modifications were based on feedback from pilot-testing PCPs and focused on (1) improving PCP adherence in implementing guideline recommendations and (2) condensing the model so that larger numbers of PCPs could be trained with enhanced effectiveness. The intervention model used to train the fifth phase of PCPs was deemed the final model, after the research team determined that it was the most parsimonious but effective intervention version. This model was used without modification for training in all subsequent phases.
The present study reports on data obtained from phase 5 to 9 practices, all of which received the finalized training model. Phases 5 to 9 included 23 practices (97 PCPs). Four practices (13 PCPs) dropped out of the intervention because of scheduling conflicts. No data were collected for those practices. The 19 remaining practices (with 65 pediatricians and 19 family physicians) constituted the sample for this study. Of those PCPs, 63% were female and 87% were white (black, 8%; other, 5%); 36.1% of the practices served primarily (>50% of patients) Medicaid-enrolled populations. An additional 13 practices, with 37 PCPs, have been trained in phases 10 and 11 but have not yet generated sufficient postintervention data for evaluation of outcomes.
The finalized intervention model included 4 training sessions, totaling 5 hours. Two 90-minute didactic sessions given by a practicing, community-based PCP (Dr Lichtenstein) focused on the evidence on which the AAP guideline recommendations are based. These training sessions occurred at a central location and were attended by PCPs and PCP-identified office champions (including nurse practitioners, nurses, medical assistants, front office personnel, and office managers). Each didactic session was followed 1 week later by a 1-hour office-based training session, attended by PCPs and office staff members, which focused on modifying office flow as a means to facilitate the incorporation of the evidence-based guideline practices. These training sessions began with charting office flow as it pertained to how patients with ADHD were treated before the intervention. Offices were then introduced to an idealized office flow diagram (Fig 1), which represented a more-parsimonious system for embedding all AAP guideline recommendations into office operations. Practices were taught to approach the ideal flow with the understanding that each practice faces unique challenges that are defined by differences in patient populations, PCP experience, and support staff composition. Participants were then introduced to a performance-improvement technique that focuses on performing small tests of change or plan-do-study-act cycles.
A variety of tools were provided to practices to support the implementation of evidence-based practices. PCPs were provided with Vanderbilt ADHD Rating Scales,18 which include ADHD symptom ratings, impairment ratings, and comorbidity screening. Offices were given written instructions on how to score and to interpret the Vanderbilt forms, a tracking grid for monitoring sequential Vanderbilt scores as a function of medication dose over time, a written care-management plan outline that encouraged PCPs to identify targeted treatment goals with families, separate formatted scripts for assessing medication responses during telephone interviews with parents or during office follow-up visits, and parent handouts describing ADHD and ADHD treatments.
Practices were taught to use a patient log to track the progress of patients with ADHD through the assessment and treatment process. Every 3 months, ADHD Collaborative staff members conducted chart reviews. Offices were then provided with practice-specific data in a report card format, which identified the extent to which guideline-based process and patient outcome measures were improving.
Finally, PCPs were provided with an algorithm for making ADHD referrals to behavioral health specialists. In general, they were instructed to collect all prescribed assessment information and, if appropriate, to attempt standard treatment for patients with ADHD. If the child presented as a complex case (eg, with comorbid disorders), failed to respond to treatment after 3 months of titration, or experienced deterioration with treatment at any time after initiation of medication treatment, then the PCPs were encouraged to refer the child for specialty assessment services. A fast-track referral service was created by the medical center for PCPs participating in the ADHD Collaborative project, which enabled them to schedule diagnostic and/or treatment consultations with staff psychiatrists, psychologists, and developmental pediatricians within 7 to 14 days.
Design and Analyses
The present project focused on elementary school-aged children. For all participating practices, baseline performance was established by reviewing the charts of up to 10 patients who had been diagnosed as having ADHD in the past 2 years for each PCP. Charts were reviewed for evidence of documentation of 7 specific guideline-related measures. Fewer than 10 patient charts were reviewed for some PCPs because some PCPs had <10 new elementary school-aged patients who had been diagnosed as having ADHD in the past 2 years.
Each of the 19 offices was asked at the onset of the training intervention to begin keeping a log of all patients for whom an ADHD assessment was initiated. Every 3 months after training, charts were reviewed for patients whose names were entered in the office logs. Each patient's chart was examined for evidence of the following practices. (1) Was a parent and/or teacher rating scale that included DSM-IV ADHD items used during the evaluation process? (2) Did the child meet DSM-IV criteria for ADHD on the basis of rating scale symptom criteria? (3) Was a written care-management plan used? (4) If medication was prescribed, was there any contact with the family within 14 days after initiation of medication treatment? (5) If medication was prescribed, was there an office visit within 6 weeks? (6) Was a parent and/or teacher rating scale used during the first 6 weeks of medication treatment, to assess medication response? Finally, (7) for patients for whom follow-up rating scales were completed, a determination was made regarding whether a 25% reduction in ADHD symptoms had occurred between baseline and the 6-week follow-up visit. The 25% symptom reduction criterion was identified as a significant symptom reduction in the Collaborative Charter for ADHD developed by the National Initiative for Children's Healthcare Quality.19 In addition, as a measure of treatment effect magnitude, an effect size was computed by comparing rating scale scores collected at the initial evaluation with those collected at the 6-week follow-up visit. Cohen's d effect sizes were computed by subtracting initial means from follow-up means and dividing the result by the pooled SD.20 An effect size of 1.0 represented a 1-SD improvement.
As noted above, the outcome data that formed the basis for this analysis were obtained from PCPs and practices trained in phases 5 to 9. To test statistically for improved practices, baseline practice levels were compared with postintervention practice levels by using χ2 tests. Because groups of practices were trained in waves that occurred approximately every 3 months, the data are summarized both across all phases and separately according to the individual phases, in accordance with an interrupted time-series design. The institutional review board at Cincinnati Children's Hospital Medical Center approved this study.
Across phases 5 to 9 (19 practices and 84 PCPs), 311 patient charts for elementary school-aged children were reviewed at baseline; 20 of the participating PCPs contributed no baseline data because they had not treated children with ADHD during the time period reviewed. On average, children seen because of ADHD were 7.88 years of age (SD: 1.6 years), and 75% were male. Before intervention, approximately one half of patients had chart documentation indicating that parent (55%) or teacher (52%) rating scale results had been collected as part of the ADHD assessment process. Only 38% of patients with diagnosed ADHD had chart documentation (eg, psychologist report or ADHD rating scale results) verifying that the child's diagnosis satisfied DSM-IV ADHD criteria.
With respect to treatment, PCPs rarely provided patients and families with written care-management plans (1%). Patients had limited contact with their doctors after medication treatment was started. Only 27% of patients had telephone contact with their PCPs within 2 weeks after starting medication treatment, and only 52% were seen in the office for follow-up visits within 6 weeks. Finally, PCPs infrequently obtained follow-up rating scale results from parents (9%) and teachers (9%), as a means of systematically documenting responses to treatment and the presence of medication adverse effects.
After completion of training, charts of elementary school-aged children with newly diagnosed ADHD were reviewed at 3 months (n = 35), 6 months (n = 56), 9 months (n = 83), and 1 year after the intervention (n = 61) for the 84 participating PCPs. Each n value represents the number of elementary school-aged children with newly diagnosed ADHD seen during the preceding 3-month period. A total of 235 new patients were seen with a presenting complaint of ADHD. All children were between 6 and 12 years of age at the time of diagnosis (mean age: 8.14 years; SD: 1.76 years), and 74% were male. Thirty-four PCPs did not record any new patients with ADHD during the study year and thus did not contribute any data to the analysis.
Compared with baseline, PCPs demonstrated significant improvement in targeted practices. For all practices assessed (Table 1), differences between baseline rates and rates at 3, 6, 9, and 12 months after the intervention showed statistically significant (P < .05) improvement. After the intervention, almost all patients were administered parent and teacher ADHD rating scales at the initial evaluation. As a result, much larger proportions of children met DSM-IV diagnostic criteria for ADHD before the initiation of treatment, that is, 77%, 96%, 83%, and 77% at 3, 6, 9, and 12 months, respectively, compared with 38% at baseline. Figure 2 presents a graphical depiction of before/after changes across phases.
After intervention, PCPs used written care-management plans with their patients with ADHD, contacted patients within 14 days after prescribing medication, and had a follow-up visit with patients for whom they prescribed medication within 6 weeks a majority of the time (Table 1). At baseline, PCPs used parent and teacher ratings to assess responses to medication quantitatively 9% of the time. After the intervention, PCP use of follow-up rating scales improved to 45%. Only 40 of the 235 patients from these PCPs were referred to and seen by the fast-track consultation service.
Six-week, follow-up, rating scale results were collected from 70 parents and 73 teachers. With the Vanderbilt total symptom score as the primary dependent measure, 25% decreases in symptom scores were observed in parent and teacher ratings for 77% and 81% of patients, respectively. For this subsample of children, effect sizes for total symptom score changes from initial evaluation (parent-rated total symptom score: 34.66 ± 10.79; teacher-rated total symptom score: 34.68 ± 10.45) to 6-week follow-up evaluation (parent-rated total symptom score: 17.91 ± 10.31; teacher-rated total symptom score: 16.94 ± 10.45) were large for both parent (Cohen's d = 1.59) and teacher (Cohen's d = 1.70) ratings. Because rating scale data were infrequently (9%) obtained by physicians before the ADHD Collaborative intervention, a comparison of medication response rates and effect sizes before and after the intervention was not possible.
Across PCPs, practices, and phases of the study, baseline data collection indicated low rates of AAP guideline implementation, including even the most basic recommendation that DSM-IV-based parent and teacher rating scales be used as part of the assessment and treatment process. Of note, all baseline data were collected at least 3 years after the publication of the AAP guidelines and 2 years after the widespread dissemination of the AAP ADHD Toolkit. Our baseline data suggest that traditional methods of promoting change (eg, guideline publication, provision of toolkits, medical center-based grand rounds, and industry-sponsored seminars) are ineffective means for promoting evidence-based practices in our community.
After participating in the ADHD Collaborative training, all practices demonstrated significant improvement in incorporating the evidence-based practices into the assessment and treatment of their patients with newly diagnosed ADHD. The most striking improvement occurred with the increased use of parent and teacher assessment scales at evaluation, for which compliance rates approached 100%. Use of rating scales resulted in an impressive doubling (from 38% to ∼77%) of the number of children whose new ADHD diagnosis satisfied strict DSM-IV criteria. Rates for use of evidence-based treatment practices (such as periodic monitoring with rating scales) improved, but there was still considerable room for improvement (Table 1).
Overall, the improvements observed in practice behavior were substantial and notably greater than those reported in other quality-improvement implementation research.21 The effectiveness of the ADHD Collaborative intervention model is based on 2 factors. First, the intervention introduced PCPs to the essential systems components of the chronic care model,22 including self-management support, delivery system design, decision support, and clinical information systems. Addressing multiple components of the chronic care model likely improved our outcomes.23 Second, assisting PCPs in incorporating these chronic care components into their office operations resulted in decreased variation in practices among PCPs, more reliance on members of the office support staff to assist with information management (eg, sending out and scoring rating scales), and more-systematic assessment and documentation of responses to therapy.
Although this intervention model produced significant improvement in individual PCP performance, our data revealed clearly that assessment practices were more successfully adopted than treatment practices. This is illustrated by the nearly 100% use of assessment rating scales, compared with the 26% to 66% use of follow-up rating scales to track medication responses. The difficulty of improving the medication maintenance practices of PCPs has been noted in other studies.8 There are several possible reasons for the observed difficulty of changing treatment practices. First, PCPs may be more comfortable using a qualitative, open-ended interview process for measuring treatment responses (eg, “How has your child been doing this past month?”) than using a quantitative rating scale system. Presumably this is not the case with assessment, where there exists a clear understanding by PCPs that they need to use a set of standardized criteria to make a valid diagnosis. Another reason may involve the logistic problems associated with distributing, collecting, scoring, interpreting, and filing multiple sets of rating scales during the medication titration and maintenance phases of patient management. Continued improvement in incorporating the treatment recommendations will require a better understanding of the attitudinal and office systems barriers that interfere with the attainment of targeted treatment process goals.
The current study design did not fully address whether improved use of the ADHD guidelines by PCPs had a meaningful impact on patient outcomes. However, the study did find that, among children who had follow-up rating scales completed within 6 weeks after initiation of medication treatment, 80% had a 25% improvement in symptom scores, as reported by parents (n = 70) or teachers (n = 73). Also, the magnitude of treatment change was quite substantial (effect sizes of >1.5). These response rates and effect sizes equal or exceed those observed in pharmacologic clinical trials24 but likely are inflated because of rater biases that emerge from open, nonblinded, administration of treatment. We have yet to test how these measures of treatment responses compare with those for children treated by PCPs using typical prescribing practices.
Another issue requiring additional study involves the sustainability of process and outcome improvements beyond the 1-year study period reported here. Although gains were observed uniformly across practices in all training phases, it is clear that practices in some phases had difficulty maintaining gains over time (Fig 2). This seemed to be especially true for treatment-related PCP behaviors. However, many practices sustained gains throughout the first study year. This may be attributable, in part, to continuing chart reviews and periodic feedback to practices by ADHD Collaborative staff members. It would be relevant to determine what happens to practice performance over time in the absence of outside oversight and feedback. Polaha et al17 reported sustained improvement with the use of an assessment protocol for up to 3 years after intervention with no continuing feedback. The ADHD Collaborative is currently developing a study to evaluate a variety of methods to address sustaining adherence to both assessment and treatment guidelines (eg, an Internet portal to assist practices with self-monitoring).
Another unanswered question concerns whether the ADHD Collaborative model can be successfully disseminated beyond the greater Cincinnati area. Standardization of PCP practice via the use of evidence-based guidelines is an important national health care goal. Unquestionably, factors inherent in our medical community facilitated this program's success, including the following: (1) Cincinnati Children's Hospital Medical Center is the only primary pediatric medical center in the region and is well respected among community PCPs; (2) PCPs were provided with continuing education credits for their participation in training sessions; and (3) participating PCPs were allowed to use a fast-track, prioritized, consultation system for referral of patients with possible comorbid diagnoses or nonresponsive ADHD to behavioral health specialists for further assessment. The role that each of these factors played in enhancing recruitment and sustained participation of PCPs requires additional analysis. We are currently preparing to study a remote dissemination training model that relies on the use of transportable strategies such as videoconferencing for training and an interactive Internet portal for data collection and self-monitoring.
This study has several limitations, many of which are inherent in community-based, quality-improvement research. First, the study was not designed as a randomized, clinical trial with assignment to intervention and control conditions. Rather, data were examined by using an interrupted time-series design. Unfortunately, without a clinical trial design, intervention effectiveness cannot be truly established. Second, we worked only with PCPs who volunteered to participate; presumably these were the most highly motivated, change-receptive PCPs in the community. It is possible that the observed improvements would not have occurred with more-conservative, change-averse PCP groups. Notably, a number of practices either did not respond to recruitment efforts or indicated that they did not wish to participate. Among practices that provided reasons for nonparticipation, being too busy and attitudinal barriers (eg, not feeling the need to change current ADHD practices) were highly cited rationales for not participating. Future research will need to determine whether this intervention can be abbreviated to accommodate practices that are too busy to participate in the intervention as delivered in this study. For example, it is possible that specific intervention components are not essential for intervention efficacy and could be discarded from the intervention model. Also, for practices that cited attitudinal issues, it will be important to develop intervention strategies (eg, improved marketing and demonstration of benefits for the practice) to engage these practices in quality improvement. Third, our process for collecting outcome data relied on PCPs faithfully entering the names of these patients in an ADHD patient log maintained by a designated log manager. Therefore, the denominator for calculation of all percentages was the number of patient names entered into the office's ADHD log. If PCPs were inconsistent in remembering to enter names of new patients in the log or if they arbitrarily excluded groups of patient names (eg, those whose families refused medication), then the denominator would have been decreased, favorably biasing our results. Finally, our data on patient responses to treatment reflect the performance of only the small number of PCPs who to date have successfully incorporated all of the treatment guidelines.
This study provides an initial demonstration of the effectiveness of an intervention with community-based PCPs that promotes the adoption of the AAP ADHD guidelines for assessment and treatment of children with ADHD. Because this intervention could be implemented across a large diverse sample of PCPs and because the intervention comprehensively addressed both assessment and treatment of children with ADHD, it seems particularly amenable to training large numbers of PCPs to adhere to the AAP ADHD assessment and treatment recommendations. Evidence-based, quality-improvement programs, such as the ADHD Collaborative, may occupy a necessary niche in the current health care emphasis on quality improvement, particularly in the context of recent American Board of Medical Specialties requirements for licensed PCPs to participate in improvement and systems-based initiatives.
This work was supported by a Patient Innovation Fund grant from Cincinnati Children's Hospital Medical Center.
We acknowledge all of the community-based practices that participated in this project. We also acknowledge the helpful comments and feedback on the manuscript provided by Thomas Boat, MD, and Bill Brinkman, MD.
- Accepted November 2, 2007.
- Address correspondence to Jeffery N. Epstein, PhD, Cincinnati Children's Hospital Medical Center, ML 10006, Cincinnati, OH 45229-3039. E-mail:
The authors have indicated they have no financial relationships relevant to this article to disclose.
What's Known on This Subject
Physician's adherence to the AAP ADHD consensus guidelines is poor. Although successful, previous attempts to improve physician behavior have been limited in sustainability and potential for dissemination.
What This Study Adds
The study describes a community-wide, quality-improvement intervention study aimed at improving the ADHD practices of community physicians. The intervention model is shown to be effective. Furthermore, the intervention as designed has strong potential for sustainability and dissemination.
- ↵American Academy of Pediatrics. Clinical practice guideline: diagnosis and evaluation of the child with attention-deficit/hyperactivity disorder. Pediatrics.2000;105 (5):1158– 1170
- ↵American Academy of Pediatrics. Clinical practice guideline: treatment of the school-aged child with attention-deficit/hyperactivity disorder. Pediatrics.2001;108 (4):1033– 1044
- ↵Wasserman R, Kelleher K, Bocian A, et al. Identification of attentional and hyperactivity problems in primary care: a report from Pediatric Research in Office Settings and the Ambulatory Sentinel Practice Network. Pediatrics.1999;103 (3). Available at: www.pediatrics.org/cgi/content/full/103/3/e38
- ↵Leslie LK, Weckerly J, Plemmons D, Landsverk J, Eastman S. Implementing the American Academy of Pediatrics attention-deficit/hyperactivity disorder diagnostic guidelines in primary care settings. Pediatrics.2004;114 (1):129– 140
- Nikles CJ, Mitchell GK, Del Mar CB, Clavarino A, McNairn N. An n-of-1 trial service in clinical practice: testing the effectiveness of stimulants for attention-deficit/hyperactivity disorder. Pediatrics.2006;117 (6):2040– 2046
- ↵Polaha J, Cooper SL, Meadows T, Kratochvil CJ. The assessment of attention-deficit/hyperactivity disorder in rural primary care: the portability of the American Academy of Pediatrics guidelines to the “real world.” Pediatrics.2005;115 (2). Available at: www.pediatrics.org/cgi/content/full/115/2/e120
- ↵National Initiative for Children's Healthcare Quality. Collaborative charter: breakthrough series collaborative on improving care for children with attention deficit hyperactivity disorder. In: Improving Care for Children With Attention Deficit Hyperactivity Disorder. Chapel Hill, NC: Children's Primary Care Research Group;2001:121– 122
- ↵Wright J, Warren E, Reeves J, et al. Effectiveness of multifaceted implementation of guidelines in primary care. J Health Serv Res Policy.2003;8 (3):142– 148
- ↵McMaster University Evidence-Based Practice Center. The Treatment of Attention-Deficit/Hyperactivity Disorder: An Evidence Report. Washington, DC: Agency for Health Care Policy and Research;1998
- Copyright © 2008 by the American Academy of Pediatrics