BACKGROUND AND OBJECTIVE: Poor adherence to recommended screening for autism spectrum disorder (ASD) and pediatricians’ lack of confidence in providing care for children with ASD reflect quality gaps in primary care. This study aimed to increase the proportion of toddlers screened for ASD and improve physicians’ self-efficacy in providing care to children with ASD.
METHODS: Twenty-six Utah primary care practices participated in a 3 to 6 month learning collaborative (LC) to improve identification and ongoing care of children with ASD. Monthly chart audits assessed whether an ASD screening tool was administered at 18- and 24-month visits. Physicians completed pre-LC and post-LC surveys to assess changes in self-efficacy in providing care and changes in perceived barriers to implementation of screening and caring for children with ASD.
RESULTS: Before the LC, 15% of 18- and 24-month visits had documented ASD screening, compared with 91% during the last month of the LC (P < .001). This rate of ASD screening was sustained 4 years after the LC by most practices. Compared with survey responses before the LC, physicians reported significant improvement in their ability to care for children with ASD and decreases in their perceived barriers to screening and caring for children with ASD.
CONCLUSIONS: The LC was effective in increasing and sustaining recommended ASD screening of toddlers and improving physicians’ perceived self-efficacy in caring for children with ASD. Improving primary care screening, skills, and knowledge may improve the timing of diagnosis, initiation of treatment, quality of care, and outcomes for children with ASD.
- AAP —
- American Academy of Pediatrics
- ADHD —
- attention-deficit/hyperactivity disorder
- ASD —
- autism spectrum disorder
- CAM —
- complementary and alternative medicine
- LC —
- learning collaborative
- M-CHAT —
- Modified Checklist for Autism in Toddlers
- PDSA —
- plan, do, study, act
- QI —
- quality improvement
- UPIQ —
- Utah Pediatric Partnership to Improve Healthcare Quality
Autism spectrum disorder (ASD) is a complex neurodevelopmental disability characterized by deficits in social communication and repetitive or restricted behaviors and interests. Since 2002, the prevalence of ASD has increased by 123%, and ASD now affects 1 in 68 children.1 The earliest symptoms of ASD, although often present during the second year of life, can be subtle and go undetected by developmental surveillance done during brief health supervision visits.2,3 Formal screening with an ASD-specific screening instrument is recommended by the American Academy of Pediatrics (AAP) for all children at the 18- and 24-month visits because implementation of ASD screening lowers the age of ASD diagnosis and entry into appropriate therapy.4–6 The uptake of ASD screening in pediatric primary care has been low, and this may account for the lack of change in the average age of diagnosis of ASD, which remains at 4.5 years.1,7 Identifying and addressing practice level barriers to ASD screening has the potential to improve the rate of screening, which is a necessary first step toward the goal of lowering the age of diagnosis and entry into services.8 Recent studies indicate that intensive behavioral therapy starting as early as 18 months of age is associated with decreases in core symptoms of ASD and improvement in cognition, language, and adaptive skills.9,10
After diagnosis, children with ASD require ongoing care within a medical home to address associated medical conditions, coordinate care with specialists, and provide family support.11 Pediatricians report low self-efficacy in meeting the medical needs of children with ASD and cite poor reimbursement, difficulty with care coordination, and insufficient training as barriers to providing care for children with ASD.12 Despite low self-efficacy, pediatricians desire to improve care for children with ASD, but report lack of adequate knowledge of treatment options, community services, and confidence in addressing common associated conditions, such as gastrointestinal conditions, insomnia, anxiety, and attention-deficit/hyperactivity disorder (ADHD).12–14
To improve the ability of pediatricians to identify children with ASD and to provide high quality health care to this population, the Utah Pediatric Partnership to Improve Healthcare Quality (UPIQ)15 developed and implemented a learning collaborative (LC). A LC is a quality improvement (QI) framework that provides physicians and their teams with the knowledge and tools to change office procedures and practices to achieve their goals related to specific clinical problems. UPIQ has conducted LCs for primary care practices addressing a variety of topics (eg, obesity, asthma, and preventive services).16,17 For this project, UPIQ developed a LC entitled, “Early Autism Detection and Referral in the Medical Home.” The goals of the LC for each practice were to: (1) implement an ASD screening process at the 18- and 24-month visits; (2) use a registry to record referrals made for children identified as at-risk; and (3) implement family-centered changes in processes of care.
We hypothesized that participating practices would increase the percentage of toddlers screened for ASD during recommended visits and that the physicians would report greater self-efficacy and fewer barriers in identifying and caring for children with ASD.
Twenty pediatric and 6 family medicine practice teams were enrolled in the LC. Each practice team consisted of at least 1 physician, a nurse or medical assistant, an office manager, and a parent of a child with ASD. Parents are considered essential team members because they assist practices in creating family-centered changes. Pediatric practices were recruited through e-mail invitations sent to ∼300 primary care pediatricians via the Utah AAP chapter listserv. Rural family medicine practices were recruited via mailings and emails through the Utah Academy of Family Physicians. Pediatricians were offered credit for Part 4 of the American Board of Pediatrics Maintenance of Certification program and American Medical Association Physician’s Recognition Award Category 1 Credit for participation. Family medicine physicians were offered American Board of Family Medicine Part 4 Maintenance of Certification credits and American Medical Association Physician’s Recognition Award Category 1 Credit for participation. Parents received a $150 stipend for team participation.
Three cohorts of practices were enrolled in the LCs: the first from June through December 2009, the second from June through November 2010, and the third from May through July 2011. The first 2 cohorts included 20 urban pediatric primary care practices located in or near Salt Lake City, Utah (8 practices in the first cohort and 12 practices in the second). The third cohort included 2 pediatric and 4 family medicine practices from rural and frontier counties in Utah.
The LC for the first cohort began with a full-day workshop. During this workshop, a general pediatrician member of the AAP Council on Children with Disabilities Autism Subcommittee (P.C.) discussed evidence-based recommendations for ASD surveillance (attending to parents’ concerns, maintaining developmental history, and observing the child for early signs and identifying risk factors for ASD) and ASD screening (use of a standardized tool at specific intervals to support and refine risk).6 We recommended that practices perform ASD screening with the Modified Checklist for Autism in Toddlers (M-CHAT) because of its favorable psychometric properties and ease of use in primary care settings.18 The M-CHAT (and the latest version of this screener, the M-CHAT, Revised, with Follow-Up) has been validated in a low-risk community sample of children 16 to 30 months of age, can be completed by a parent in ∼5 minutes, and is in the public domain.4,5 Participants were trained on proper administration of the M-CHAT and the M-CHAT follow-up questionnaire according to recommendations from the creator of the instrument.19 Practices were encouraged to establish a registry of children identified as at-risk for ASD through surveillance and screening to ensure appropriate evaluation and entry into services. Practices were given examples of paper and electronic registries. Several other tools were presented and demonstrated, including the AAP Autism Toolkit for Primary Care Providers and the Autism Speaks Video Glossary.20,21 A general pediatrician who directs UPIQ and the director of the local family-to-family health information center addressed the need for parent/professional partnerships and the principles of family-centered care for children with ASD. The president of a local ASD advocacy group presented a parent’s perspective on receiving a diagnosis of ASD and the challenges faced in meeting health care needs. A UPIQ QI specialist described a model for QI and the use of plan, do, study, act (PDSA) cycles. Each team developed their specific aims, initial plans for change, and timeframe for meeting to assess progress. At each scheduled team meeting throughout the LC, the teams subjectively rated their progress toward their desired improvements on a scale of 1 to 10. Required aims for all practice teams included increasing the percentage of children screened for ASD at 18- and 24-month well visits and developing a registry of children identified as at-risk for ASD. Teams were invited to participate in monthly conference calls on a variety of ASD-related topics. For example, 1 call addressed referral processes for a child identified as at-risk for ASD and provided contact information for several community-based clinics that perform comprehensive ASD evaluations. Other conference calls reviewed evaluation and management of commonly associated conditions, such as gastrointestinal problems, insomnia, ADHD, and anxiety. A UPIQ QI specialist conducted 2 site visits with each practice to reinforce the need for team meetings, guide the use of PDSA cycles, and address challenges identified by the practices.
The structure of the LC for the second cohort was similar to the first except that the initial workshop was shortened to a half-day to accommodate travel times for practices located farther from Salt Lake City. The content omitted from the initial workshop was presented during 2 additional site visits by a UPIQ QI specialist. The LC for the third cohort lasted 3 months instead of 6 and used webinars instead of in-person workshops to avoid travel for practices located in rural and frontier counties. For the third cohort, the didactic content of the first 2 cohorts’ workshop and conference calls was presented during three 1.5-hour webinars. A UPIQ QI specialist performed 1 site visit and 1 conference call with each practice.
Practice team physicians were invited to participate in the evaluation of the LC; all agreed and provided informed consent. The University of Utah institutional review board approved the project.
After the first workshop (or first webinar), each practice audited at least 5 charts (per participating physician) of children seen in the previous month for an 18- or 24-month visit to determine whether an ASD screening tool had been administered (yes or no for each audited chart). Practice teams conducted similar monthly chart audits throughout, and immediately at the conclusion of, the LC. Based on previous studies of ASD screening in primary care, we set a goal of 80% for ASD screening.22,23 We compared the proportion of patients who were screened at baseline and at the end of the LC using McNemar’s test. To assess sustainability, we contacted each practice team in November 2015 (4 years after the last LC) to find out if routine ASD screening was ongoing, and if so, we asked the practice to conduct an audit of 30 charts from 18- or 24-month visits during the previous 6 months to ascertain if an ASD screening test was competed during the visit.
Registry data of at-risk children identified by practices were evaluated for the numbers of children identified, by what means (eg, by parent/provider concern or by positive screening test), and referrals that were made.
Before the first workshop or webinar, physicians completed a survey that included demographic information, level of confidence in addressing ASD-specific needs and ASD-associated conditions, and perceived barriers to screening and providing care to children with ASD. The survey, consisting of 16 questions with Likert scale responses, had been used for a previous study on physician self-efficacy in addressing the needs of children with ASD.14 The survey addressed 2 constructs: “ASD-specific needs” and “ASD-associated conditions.” We defined ASD-specific needs as questions or service-related issues regarding the diagnosis and treatment of ASD, such as concerns about development, vaccines, and referring to diagnostic and treatment services. We defined ASD-associated conditions as clinical or behavioral issues that frequently accompany ASD, such as sleep, gastrointestinal, and feeding problems, ADHD, and anxiety. At the completion of the LC, participating physicians completed the survey again. Pre- and post-LC survey responses were compared using the Wilcoxon rank test. When appropriate, a multiple comparison adjustment (Bonferroni correction) was performed to adjust the level of significance.
Practice teams’ subjective ratings of progress toward their family-centered care goals were compared at the beginning and during the last month of the LC by using the Wilcoxon rank test.
All 26 practices completed the LC and 43 of 45 physicians completed the pre- and post-LC surveys. Demographic characteristics of participating physicians are shown in Table 1. Nearly 60% practiced in urban settings, and 40% practiced in rural settings. Over 80% had been in practice >5 years, and over one-third reported caring for >10 children with ASD.
At baseline and during the LC, practices audited 1930 charts from 18- and 24-month well visits to assess for documentation of a completed M-CHAT. At baseline, 29% of audited visits in urban practices (cohorts 1 and 2) and 5% in rural/frontier practices (cohort 3) had such documentation. The proportion of monthly audited visits with M-CHAT documentation increased to 95% for urban practices and 84% for rural/frontier practices by the last month of the LC. Combining the data for all cohorts, 16% of audited visits had documented ASD screening at baseline compared with 91% during the last month of the LC (P < .001) (Fig 1). Using the practice as the unit of analysis, only 1 of 26 practices was screening ≥80% of toddlers at baseline compared with 21 of 26 practices during the last month of the LC (P < .001) (data not shown). Four years after the LC, 18 of 26 practices reported sustaining an ASD screening program (5 practices stopped ASD screening, 1 was no longer in business, and 2 did not respond). Combining chart audit data for all cohorts, 91% of 530 audited charts from 18-and 24-month visits that occurred 4 years after the LC had evidence of a completed ASD screening test (Fig 1). Among the 18 practices that continued to screen for ASD, 16 (89%) screened ≥80% of toddlers.
The comparison of pre- and post-LC physician survey responses are shown in Tables 2–5. Compared with the pre-LC survey responses, physicians’ self-efficacy improved significantly on all 7 ASD-specific needs and all 9 ASD-associated conditions (Tables 2 and 3). Compared with pre-LC survey responses, participating physicians’ ratings of lack of guidelines and lack of provider education as barriers to care were lower (Table 4). Similarly, physicians’ ratings of several barriers to ASD screening decreased between the pre-LC and post-LC surveys (Table 5).
Table 6 shows the types of plans that were adopted to improve family-centered care for children with ASD and the number of practices that adopted each. Eleven practices adopted plans to increase awareness of ASD among parents and staff, 9 adopted plans to connect families to community resources, 6 chose to focus on improving chronic care management of children with ASD, and 10 targeted family support. Subjective ratings of progress toward improved family-centered care increased in 14 of the 22 practices for which data were available (P = .001). The average rating of progress, on a scale of 1 to 10, at the end of the LC was 6.50 (SD = 3.02).
All practices reported using a registry to record children in need of further evaluation. The 8 practices in the first cohort identified 16 children as at-risk for ASD. Of these, 10 were identified by M-CHAT and 6 by parent or provider concern. Of these 16 children, follow-up plans were recorded for 9; 5 were referred for evaluation and 4 were only advised to return for the next routine visit. The 12 practices in the second cohort identified 50 children as at-risk for ASD. Among the 49 children with method of identification recorded, 21 were by M-CHAT, 15 by M-CHAT and parent/provider concern, and 13 by parent/provider concern. Forty-two of the 50 at-risk children (84%) were referred for a comprehensive ASD evaluation, 23 (46%) to audiology, and 13 (26%) to an early intervention program. Among the 6 practices from the third cohort (rural/frontier practices), 4 children were found to be at-risk for ASD: 2 by M-CHAT and 2 by M-CHAT and parent/provider concern. Of these 4 children, 2 were referred for comprehensive evaluation, audiology evaluation, and to an early intervention program, and 2 were only told to return for the next well visit. Combining data for all cohorts, 70 children were identified as at-risk for ASD and, of these, 33 (47%) were identified by ASD screening before any concern was noted by parents or providers.
Practice teams that participated in the LC created goals, worked on plans to improve care for children with ASD, and demonstrated, through chart audits, a significant improvement in the proportion of toddlers screened over the course of the LC. Four years after completion of the LC, 70% of practices continue to screen for ASD, and those that do screen the same high proportion (91%) that was reported at the completion of the LC. Given the sustainability of the progress seen during the LC, many participating practices have institutionalized ASD screening programs that were developed during the LC, even after some of the original physicians left these practices. The baseline proportion of toddlers screened for ASD in participating practices (19%) mirrors recent data from the 2011 National Survey of Children’s Health, in which 21% of parents reported that their child had been screened for developmental, behavioral, and social delays with a screening tool during the past 12 months.7 The proportion screened for ASD after the LC (91%) is significantly higher than other estimates of ASD screening within primary care.14,24
Compared with urban practices, rural/frontier practices screened a lower proportion of children at baseline and after the LC, and 3 of the 6 participating practices reported that they were no longer screening 4 years after the LC. We believe there are several possible explanations for these findings. First, the LC for rural/frontier practices was of shorter duration and used distance technology rather than face-to-face interaction for instruction. Second, rural and frontier practices were more likely to be family medicine practices that see relatively fewer children, making the intervention more difficult to sustain given the relatively small number of screen-positive children identified. Further study is needed to assess methods of improving ASD screening in rural/frontier settings.
We believe the overall success of the practices in improving ASD screening was related to the use of QI methods (PDSA) and team engagement and group learning during the LC. The initial workshop allowed time to articulate the importance of ASD screening, to recognize gaps in implementation, and for practices to develop initial plans to improve systems of care (“plan” and “do”). Monthly reporting on a measurable outcome, the proportion of visits with documented ASD screening, gave practices timely feedback on their progress (“study”) and allowed for refinement of the office system of ASD screening (“act”) during regular team meetings. We found that practices were receptive to the information they received during the workshops and were able to use the information to develop a variety of plans to improve care for children with ASD. All practices kept a registry of children found to be at-risk after ASD screening was performed.
The registry data support findings of other studies regarding ASD screening. We found that a high proportion of children identified in the first cohort as at-risk for ASD were not referred for further evaluation, but rather a “wait and see” approach was taken, with a documented plan to follow-up at the next visit. Despite guidelines that recommend prompt referrals for a comprehensive ASD evaluation, audiology evaluation, and to an early intervention program after a positive screen, several recent studies have found that a high proportion of children found to be at-risk for ASD and other developmental delays are not referred for evaluation and treatment in a timely manner.25,26 The QI team emphasized prompt referral of at-risk children in subsequent cohorts and provided practices with a printed referral sheet for families that listed contact information for recommended referrals. The registry data from the second cohort show that this intervention was somewhat beneficial in that 84% of at-risk children were referred for a comprehensive ASD evaluation, although referral rates for audiology and to an early intervention program remained low. System-level interventions that may increase appropriate referrals for ASD evaluations after positive screening include electronic clinical decision support for physicians, phone follow-up of abnormal screenings, and collaboration of pediatric practices with community partners (such as an early intervention program or 2-1-1 information and referral centers) who facilitate navigating the family to the next step in the diagnostic process.26–28
Given that physicians have previously reported a low level of confidence in providing care to children with ASD, the improvement in physician-reported self-efficacy in addressing the needs of children with ASD during the LC is encouraging. The ASD-specific needs in which the physicians showed improvement (addressing early developmental concerns, making appropriate referrals, providing advice about evidenced based therapies, addressing concerns about complementary and alternative medicine [CAM], and having knowledge of community support services) are those identified as essential by the AAP in the ongoing care of children with ASD.11 Physicians also became more comfortable in addressing ASD-associated conditions. Our findings indicate that, with adequate training of primary care providers, these associated conditions, such as constipation, sleep problems, and ADHD, can be effectively addressed within the medical home without the need for specialist referral.29–31
Unlike traditional continuing medical education, the LC focused on improvement of processes of care at the practice level, so it is not surprising that some of the physician-perceived barriers to screening for ASD and caring for children with ASD diminished over the course of the LC. From the physicians’ perspective, the barriers that were most sensitive to improvement were those modifiable by education or changes at the practice level, such as lacking an easy-to-use ASD screening test or an office system to administer screening tests. Barriers that persisted were those that were less within the scope of practice level change, such as lack of community resources and patients’ use of CAM.
Although we were able to accomplish our goal of assisting practices to improve their processes of care for children with ASD, our study was limited in that we do not know if this resulted in improved detection of ASD or improved care of children with ASD. For example, our project did not involve confirmation that at-risk children who were referred were ultimately evaluated for ASD. Another limitation of our study was the self-selection of practices. Practices that chose to participate may have had a relatively high level of motivation to make changes. We also relied on practices to provide data from chart audits, and we did not independently validate the results of the audits. We also limited chart audits to a random selection of 5 per month during the LC and 30 after the LC to respect time limitations on practices to collect data. There is a possibility that this random sample does not accurately reflect the true ASD screening prevalence of the practices. Although the majority of providers who participated in the LC were physicians in private practice and may not be representative of primary care providers in other settings, nearly 40% of the providers were from rural or frontier practices. Despite these limitations, we are confident that improved screening for ASD and better adherence to guidelines in the care of children with ASD offers potential benefits and minimal risk for harm.4,32
A LC using the methods we describe is a successful approach to improving the early identification and ongoing care of children with ASD in primary care practices. Improving primary care screening, skills, and knowledge may improve the timeliness of diagnosis and initiation of therapy, quality of care, and outcomes of children with ASD.
We thank the practice teams throughout Utah who participated in the LC. We are also grateful to UPIQ QI Specialists Sandra DeBry, MBA, Sarah Nae`ole, MSHS, MBA, PMP, and Gabi Baraghoshi, RN, MPH, for their contributions.
- Accepted February 16, 2016.
- Address correspondence to Paul S. Carbone, MD, Department of Pediatrics, University of Utah, 295 Chipeta Way, Salt Lake City, UT 84108. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: The learning collaborative was funded through the “State Implementation Grants for Improving Services for Children and Youth with ASD” from the US Department of Health and Human Services Maternal and Child Health Federal Consolidated Program (grant H6MMC11061AO, HRSA-08-144 CFDA 93.110).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Developmental Disabilities Monitoring Network Surveillance Year 2010 Principal Investigators
- Centers for Disease Control and Prevention (CDC)
- Zwaigenbaum L,
- Bauman ML,
- Stone WL, et al
- Robins DL,
- Casagrande K,
- Barton M,
- Chen CM,
- Dumont-Mathieu T,
- Fein D
- Johnson CP,
- Myers SM; American Academy of Pediatrics Council on Children With Disabilities
- Zwaigenbaum L,
- Bauman ML,
- Fein D, et al
- Myers SM,
- Johnson CP; American Academy of Pediatrics Council on Children With Disabilities
- Golnik A,
- Ireland M,
- Borowsky IW
- ↵Utah Pediatric Partnership to Improve Healthcare Quality. Available at: http://medicine.utah.edu/pediatrics/upiq/. Accessed December 30, 2015
- Young PC,
- DeBry S,
- Jackson WD, et al
- Young PC,
- Glade GB,
- Stoddard GJ,
- Norlin C
- Robins DL
- ↵M-CHAT information. Available at: mchatscreen.com. Accessed December 22, 2015
- American Academy of Pediatrics
- ↵Autism Speaks. Video glossary. Available at: www.autismspeaks.org/what-autism/video-glossary. Accessed December 22, 2015
- Miller JS,
- Gabrielsen T,
- Villalobos M, et al
- Furuta GT,
- Williams K,
- Kooros K, et al
- Malow BA,
- Byars K,
- Johnson K, et al; Sleep Committee of the Autism Treatment Network
- Mahajan R,
- Bernal MP,
- Panzer R, et al; Autism Speaks Autism Treatment Network Psychopharmacology Committee
- Copyright © 2016 by the American Academy of Pediatrics