OBJECTIVE: To determine if implementing attention-deficit/hyperactivity disorder (ADHD) diagnosis and treatment guidelines in a clinical decision support system would result in better care, including higher rates of adherence to clinical care guidelines.
METHODS: We conducted a cluster randomized controlled trial in which we compared diagnosis and management of ADHD in 6- to 12-year-olds after implementation of a computer decision support system in 4 practices.
RESULTS: Eighty-four charts were reviewed. In the control group, the use of structured diagnostic assessments dropped from 50% in the baseline period to 38% in the intervention period. In the intervention group, however, it rose from 60% to 81%. This difference was statistically significant, even after controlling for age, gender, and race (odds ratio of structured diagnostic assessment in intervention group versus control group = 8.0, 95% confidence interval 1.6–40.6). Significant differences were also seen in the number of ADHD core symptoms noted at the time of diagnosis. Our study was not powered to detect changes in care and management, but the percent of patients who had documented medication adjustments, mental health referrals, and visits to mental health specialists were higher in the intervention group than the control.
CONCLUSIONS: The introduction of a clinical decision support module resulted in higher quality of care with respect to ADHD diagnosis including a prospect for higher quality of ADHD management in children. Future work will examine how to further develop the ADHD module and add support for other chronic conditions.
- ADHD —
- attention-deficit/hyperactivity disorder
- CDSS —
- clinical decision support system
- CHICA —
- Child Health Improvement through Computer Automation
- EMR —
- electronic medical record
- RMRS —
- Regenstrief Medical Record System
What’s Known on This Subject:
Despite the existence of authoritative guidelines to assist primary care physicians in identifying and managing attention-deficit/hyperactivity disorder (ADHD), ample evidence demonstrates that they continue to diagnose and treat this disorder suboptimally.
What This Study Adds:
The introduction of a clinical decision support module resulted in higher quality of care with respect to ADHD diagnosis including a prospect for higher quality of ADHD management in children.
Attention-deficit/hyperactivity disorder (ADHD) is the most commonly diagnosed pediatric behavioral disorder.1 The gold standard for diagnosis is standardized assessments of behavior based on specific diagnostic criteria defined in the Diagnostic and Statistical Manual of Mental Disorders.2 Because there is no definitive diagnostic test for ADHD, diagnosis can be difficult. The core symptoms that characterize the disorder (inattention, hyperactivity, and impulsivity) lie on a spectrum; moreover, many other medical, psychiatric, and learning conditions can present with similar symptoms.2,3
Primary care physicians, especially pediatricians, have historically played a large role in the diagnosis and treatment of ADHD.4,5 Despite the existence of authoritative guidelines to assist primary care physicians,6–10 ample evidence demonstrates that they continue to diagnose and treat this disorder suboptimally.2,11–13 This is due, in part, to a lack of training and cumbersome delivery system designs.14 Modern computer decision support strategies offer the best hope of equipping general practitioners to deal with the mental health epidemic of ADHD.
We have developed a novel computer decision support system (CDSS) for implementing clinical guidelines in pediatric practice. The CHICA (Child Health Improvement through Computer Automation) system collects information directly from patients, delivers “just-in-time” patient-relevant guidelines to physicians during the clinical encounter, and accurately captures structured data from all who interact with it. Previous work with CHICA has demonstrated the feasibility of using the system to implement and evaluate clinical guidelines,15,16 In 2010, we enhanced the CHICA system to include the American Academy of Pediatrics guidelines to assist physicians in the diagnosis6 and treatment17 of ADHD. We then conducted a study to see if the delivery of ADHD guidelines via a CDSS would result in better care, specifically higher rates of adherence to clinical care guidelines and improved patient functioning.
This study took place in 4 primary care practices in the Indiana University Medical Group-Primary Care practice network. In 2010, these 4 clinics hosted 37 425 patient visits. Of these, 77% to 88% of visits were for pediatric patients who were on Medicaid. Two of these clinics served as intervention sites, and the other 2 clinics functioned as control sites. This study took place between 2010 and 2012.
Patients between the ages of 5 and 12 years were automatically placed into either the control or intervention group based on which of the 4 pediatric clinics they attended. No potential subjects were contacted by researchers, their physician, or other staff personnel regarding the study.
We conducted a cluster randomized controlled trial in which we compared diagnosis of ADHD in the intervention practices to the control practices. The 2 intervention clinics used the “enhanced” CHICA system, which included guidelines for ADHD diagnosis and management (ie, the ADHD module), and the 2 control clinics used the “traditional” CHICA system that did not include ADHD guidelines. To achieve randomization, the 4 clinics were ranked by their size, where size was defined by the number of physicians and residents in the clinic. At randomization, the first and last ranked clinics were assigned to one group (intervention) and the other 2 clinics were assigned to the other group (control). In total, there were 60 clinicians practicing at the 2 intervention sites, and 50 clinicians practicing at the 2 control sites.
Although randomization at the physician or patient level may have been sample-size efficient, we chose to randomize by clinic because contamination was a major concern. If we randomized at the physician level, physicians in the same clinic who were assigned to different treatment arms might communicate regarding the CHICA ADHD module in terms of its operation and consequences. Similarly, if we randomized at the patient level, the on-and-off use of the CHICA ADHD module might confuse or frustrate the physician and even lead some physicians to think that CHICA was malfunctioning. We have found previously that randomization at the patient level can lead to irritation and inconvenience for both physicians and clinic staff.18
This study was conducted by using the CHICA system, which has been described in significant detail in previous publications.19,20 In short, CHICA is a CDSS coupled with an electronic medical record (EMR) that can be used for pediatric well care and chronic disease management.18,19,21 CHICA produces 2 tailored and scannable paper forms. The prescreener form contains 20 questions that parents answer while in the waiting room. The physician worksheet contains up to 6 prompts to physicians that include check box responses to record the physician’s assessment and actions. Data from these forms are captured in the computer by scanning with optical mark recognition and optical character recognition. The questions and prompts are programmatically chosen according to a child’s age and data stored in the child’s EMR, the Regenstrief Medical Record System (RMRS).16 The RMRS contains additional information, such as diagnostic codes, physician orders, filled prescriptions, and laboratory data from the statewide health information exchange, the Indiana Network for Patient Care.22
Intervention: The CHICA ADHD Module
The functioning of the CHICA ADHD module is described in Fig 1 and is based on the American Academy of Pediatrics guidelines to assist physicians in the diagnosis6 and treatment17 of ADHD. The first step concerned case finding. On the prescreening form, parents of children 5 to 12 years of age were asked the following 3 screening questions:
Does [CHILD’S NAME] OFTEN make careless mistakes or not pay close attention to details, causing problems at home or school?
Does [CHILD’S NAME] OFTEN have difficulty remaining seated when asked to do so, causing problems at home or school?
Does [CHILD’S NAME] OFTEN have a hard time paying attention to tasks or play, causing problems at home or school?
Positive answers to any of these 3 screening questions alerted CHICA to potential ADHD. Additionally, the CHICA system was also programed to determine whether the child had a previous diagnosis of ADHD by looking for the presence of ADHD diagnostic codes in the patient’s EMR. In the control group, screening was left to the discretion of the physician. No screening questions were included on the prescreening form for parents to answer in the control group.
The diagnosis of ADHD requires a combination of symptoms including inattention, distractibility, or hyperactivity that impairs the child’s function in >1 setting. One of the most practical ways to assess these criteria is to have the family complete a standardized instrument, for example, the National Initiative for Children’s Healthcare Quality ADHD Vanderbilt Assessment Scales.23 To achieve this, the CHICA ADHD module automatically printed customized and scannable Vanderbilt assessment scales. The Vanderbilt assessment scales were printed if a child was suspected of having ADHD based on parent answers to the screening questions. Additionally, the Vanderbilt assessment scales were printed at appropriate intervals for those children already diagnosed with ADHD based on the child’s age and based on information contained in their EMR. For example, if a follow-up assessment was needed, CHICA would automatically print the parent and teacher follow-up Vanderbilt forms.
The CHICA ADHD module also instructed the physician in the proper distribution and completion of the Vanderbilt forms. For example, the parent assessment forms were completed by the family in the waiting room and scanned in during the office visit; in rare cases, they were faxed directly to the CHICA system. The teacher Vanderbilt assessment forms were given to the family and could be faxed by the teacher to a standard phone number in the CHICA system. Both the parent and teacher assessment forms were automatically scored by CHICA and recorded in the system. As a convenience for the clinician, CHICA generated a summary sheet with all subscores and interpretations from each assessment form. The interpretations were also made available as prompts on the physician worksheet at subsequent follow-up visits.
The CHICA ADHD module also made treatment recommendations based on established guidelines. This included appropriate starting doses for medications and suggested medication changes if treatment goals were not met. As an aid for the clinician, the patient and family, and teachers, CHICA also provided age-appropriate “just-in-time” handouts that included behavior management suggestions.
We used manual chart abstractions to assess each practice’s adherence to clinical care guidelines before and after implementing the CHICA ADHD module. Data collection began 6 months after the CHICA ADHD Module was turned on in the CHICA system, and data collection occurred until 6 months after the last patient included was diagnosed with ADHD. Data were collected on a revised version of a data abstraction form, which was developed for previous studies by members of our research group.12 Research assistants were trained on the use of this form by the study investigators. Our primary outcome of interest was the percentage of patients who were diagnosed with ADHD who had a completed structured diagnostic assessment, defined as an initial Vanderbilt rating scale completed by both the parent and the teacher, before the diagnosis of ADHD being assigned. The number of core symptoms noted at the time of diagnosis was also recorded during the chart review process. Other data were collected regarding the care and management of ADHD such as reassessment of symptoms, medication adjustments, and mental health referrals. A random sample (20%) of the charts was abstracted twice to assess the reliability of chart abstraction. The agreement on overall chart abstraction in these categories was found to be 89%, with a κ of 0.76.
Characteristics of the study participants at baseline were summarized by using descriptive statistics (ie, frequency counts and percentages by treatment groups). Baseline categorical variables were examined using χ2 tests or Fisher’s exact test, and continuous measures were examined with Wilcoxon rank-sum test.
A logistic regression model including clinic as a cluster variable was used for comparison of dichotomous variables, for example, to examine whether a patient had a structured diagnostic assessment at postintervention. Poisson regression with clinic as a cluster variable was used to model the count variables (eg, the number of inattention behaviors recorded at home). Because we observed overdispersion in the distribution of count variables, we used a generalized Poisson distribution to model these outcomes. Group comparisons in all models were adjusted for age, gender, and race.
All calculations were performed by using the SAS Version 9.2 statistical package.
Sample Size Calculation
Because patients were nested within clinic, responses from the patients within clinic were likely to be correlated. This intracluster correlation was taken into account in the calculation of sample size and in the analyses. By using a χ2 test and setting α at .05, the probability of detecting a statistically significant difference in the adherence rate (20% vs 60%) was 90% with an effective sample size of 35 chart reviews per group. Hence, the total effective sample size was 70. Using 2 clinics in the intervention group and 2 clinics in the control group, a random review of 21 charts per clinic would have 90% power to detect a difference of 40%.
Institutional Review Board
This study was reviewed and approved by the Institutional Review Board of the Indiana University School of Medicine.
We queried the RMRS for patients who received a new diagnosis of ADHD during the study period. One hundred eighty-three patients were identified as being eligible for the study; 101 patients from the intervention clinics and 82 from the control clinics. By using a random number generator, we then selected 42 charts from the intervention and from the control clinics to review.
Characteristics of the patients included in this study can be seen in Table 1. There were no significant differences between the intervention and control groups with respect to gender for structured diagnostic assessment, although some differences were seen in insurance and race because the clinics serve different areas of Indianapolis, Indiana.
Because we wanted to determine a true change in the main outcome of interest (ie, change in the rate of structured diagnostic assessment), we conducted a baseline analysis that looked at the use of structured diagnostic assessments in both the intervention and control clinics before the initiation of our study. Eighty-four charts were pulled for the baseline analysis of children newly diagnosed with ADHD (42 charts from intervention group and 42 from control group). At baseline, the use of structured diagnostic assessment was 50% for the control group and 60% for the intervention group. During the course of our study, the use of structured diagnostic assessment dropped to 38% in the control group (see Fig 2). However, it rose to 81% in the intervention group. In a logistic regression model, this difference was statistically significant, even after controlling for age, gender, race/ethnicity, and insurance (odds ratio of structured diagnostic assessment in intervention group = 8.0, 95% confidence interval 1.6–40.6).
In a Poisson regression model, differences were also seen in the number of ADHD core symptoms noted at the time of diagnosis. Figure 3 shows the average number of inattentive or hyperactive/impulsive symptoms noted at home and school. Although the numbers of hyperactive symptoms at school did not achieve statistical significance (P = .075), increases in the 3 others measured were significant to P < .05 (Fig 3).
Although our study was not powered to detect changes in ADHD care and management, we also did preliminary analyses in these areas (Fig 4). There were no statistically significant differences in the percent of patients who had documented medication adjustments between the intervention and the control groups (45% vs 33%, P = .45). The percent of patients who had documented reassessment of symptoms at the 3-month follow-up visit was also higher in the intervention group, but it was not statistically significant (50% vs 33%, P = .36). More patients in the intervention group also had a mental health referral (74% vs 55%, P = .09) and a visit to a mental health specialist (67% vs 48%, P = .054); however these changes were also not statistically significant.
Evidence-based guidelines recommend that physicians conduct a structured diagnostic assessment and document ADHD symptoms resulting in impaired functioning in 2 settings. This study showed that the introduction of a clinical decision support module for ADHD could significantly improve the use of standardized rating scales from both a parent and teacher for the diagnosis of ADHD. Differences were also seen in the number of ADHD core symptoms noted at the time of diagnosis. This is an important finding because it demonstrates that our automated process can improve the specificity with which parents and teachers are accurately reporting symptoms. In addition, we saw trends toward improvements in areas of management, including timeliness of reassessment and medication titration. We also noted trends indicating that children in the intervention group were referred for comanagement with a mental health specialist more often than children in the control group. This is important because although treatment of ADHD symptoms with stimulant medication can reduce the number of core symptoms, children may continue to have impaired daily functioning, which requires comanagement with a mental health specialist.24
Pediatricians are busy and often overwhelmed in clinical care due to time and resource constraints.25 It is often difficult for them to properly screen for every condition, and things can get missed.26 Even when they try to follow guidelines, care can be suboptimal. Some past CDSS approaches have been specific to mental health issues27 and, as a result, may have been intrusive and disruptive to the usual processes of care.28 The CHICA system developed by our group is unique in that respect. CHICA allows us to introduce guideline-based care into existing clinic practices and prioritize which care modules are most relevant for which children in a tailored fashion. Using a holistic CDSS like CHICA holds much promise for introducing better evidence-based care and ongoing chronic care management into busy pediatric practices. Another benefit to using a CDSS like CHICA is the ability to adjust clinical guidelines easily as the evidence supporting their use changes.
As with all such studies, there are limits to this study that warrant consideration. It is possible that existing clinical practice differences (eg, baseline rates of structured diagnostic assessments in the intervention and control groups) accounted for some of the differences that we found, although we did control for them in our analyses. Randomizing at the clinic level could also lead to biases due to knowledge of ADHD diagnosis and management by physicians, socioeconomic status of patients, and so on. To our knowledge, however, there is no reported significance and scale of the effect of these factors. After weighing the limitations and advantages of each option, we decided to allocate 2 clinics to each of the 2 arms of the study. Generalizability of this study may also be a limitation; this was 1 study of a specific CDSS. However, the CHICA system is built with open-source tools and can be adopted relatively easily and installed cheaply. Although most EMRs may lack the sophistication of CHICA, it is possible to implement the same algorithm with existing commercially available systems. Practices without CDSS would have difficulty implementing this algorithm, but the number of such practices should become fewer as more achieve meaningful use through the HITECH Act.29 Moreover, through CHICA, we are also testing a long-term theory: that the best way to introduce new guideline-based interventions is to make them a part of existing operations in a transparent and unobtrusive manner. We believe that much of the success of the system is based on this. We are not studying the ADHD forms in isolation. Instead we are studying a combination of improved screening, improved document management, and improved CDSS. We were studying this intervention holistically, not pushing any 1 piece above another.
Changing physician behavior can be difficult. This study proved that a CDSS using direct patient input and data-driven algorithms to prioritize and deliver personalized, evidence-based recommendations to physicians at the point of care can accomplish higher quality of care with respect to ADHD diagnosis, as well as the prospect of higher quality for ADHD management. Future work will examine how to develop the CDSS further and support care for other chronic conditions.
- Accepted June 21, 2013.
- Address correspondence to Aaron E. Carroll MD, MS, 410 West 10th St, HITS 1020, Indianapolis, IN 46202. E-mail:
Dr Carroll conceptualized and designed the study and drafted the initial manuscript; Dr Bauer assisted in the conceptualization and design of the study, assisted in acquisition of the data, and critically reviewed the manuscript; Ms Dugan coordinated and supervised data collection and critically reviewed the manuscript; Dr Anand assisted in the conceptualization and design of the study and critically reviewed the manuscript; Dr Saha carried out the initial analyses and reviewed and revised the manuscript; Dr Downs assisted in conceptualization and design of the study and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This project was supported by grant 5R01LM010031 from the National Library of Medicine. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- ↵Diagnostic and Statistical Manual of Mental Disorders. 4th ed. (DSM-IV). Washington, DC: American Psychiatric Association; 1994
- ↵Hoagwood K, Kelleher K, Feil M, Comer D. A national perspective on treatments and services for children with attention deficit hyperactivity disorder. Paper presented at the National Institutes of Health Consensus Conference on Diagnosis and Treatment of ADHD; November 16–18, 1998; Bethesda, MD
- ↵Wolraich M. Current assessment and treatment practices. Paper presented at the National Institutes of Health Consensus Conference on Diagnosis and Treatment of ADHD; November 16–18, 1998; Bethesda, MD
- American Academy of Pediatrics
- Hill P,
- Taylor E
- Goldman LS,
- Genel M,
- Bezman RJ,
- Slanetz PJ,
- Council on Scientific Affairs
- Wolraich M,
- Brown L,
- Brown RT,
- et al.,
- Subcommittee on Attention-Deficit/Hyperactivity Disorder,
- Steering Committee on Quality Improvement and Management
- Leslie LK,
- Weckerly J,
- Plemmons D,
- Landsverk J,
- Eastman S
- ↵Biondich PG, Downs SM, Anand V, Carroll AE. Automating the recognition and prioritization of needed preventive services: early results from the CHICA system. AMIA Annu Symp Proc. 2005;51–55
- ↵Downs SM, Biondich PG, Anand V, Zore M, Carroll AE. Using Arden Syntax and adaptive turnaround documents to evaluate clinical guidelines. AMIA Annu Symp Proc. 2006;214–218
- American Academy of Pediatrics. Subcommittee on Attention-Deficit/Hyperactivity Disorder and Committee on Quality Improvement.
- ↵Downs SM, Zhu V, Anand V, Biondich PG, Carroll AE. The CHICA smoking cessation system. AMIA Annu Symp Proc. 2008;166–170
- McDonald CJ,
- Overhage JM,
- Barnes M,
- et al.,
- INPC Management Committee
- Wolraich ML,
- Lambert W,
- Doffing MA,
- Bickman L,
- Simmons T,
- Worley K
- ↵Sturner R, Howard B, Morrel T, et al. The Child Health and Development Interactive System (CHADIS). Paper presented at the 7th Annual National Institutes of Health SBIR/STTR Conference; July 28–29, 2005; Bethesda, MD
- ↵Office of the National Coordinator for Health Information Technology. HITECH Act. Available at: www.healthit.gov/policy-researchers-implementers/hitech-act-0. Accessed May 29, 2013
- Copyright © 2013 by the American Academy of Pediatrics