BACKGROUND: In the United States, up to 20% of children experience a mental health (MH) disorder in a given year, many of whom remain untreated. Routine screening during annual well visits is 1 strategy providers can use to identify concerns early and facilitate appropriate intervention. However, many barriers exist to the effective implementation of such screening.
METHODS: A 15-month quality improvement learning collaborative was designed and implemented to improve screening practices in primary care. Participating practices completed a survey at 3 time points to assess preparedness and ability to promote and support MH issues. Monthly chart reviews were performed to assess the rates of screening at well visits, documentation of screening results, and appropriate coding practices.
RESULTS: Ten practices (including 107 providers) were active participants for the duration of the project. Screening rates increased from 1% at baseline to 74% by the end of the project. For the 1 practice for which more comprehensive data were available, these screening rates were sustained over time. Documentation of results and appropriate billing for reimbursement mirrored the improvement seen in screening rates.
CONCLUSIONS: The learning collaborative model can improve MH screening practices in pediatric primary care, an important first step toward early identification of children with concerns. More information is needed about the burden placed on practices and providers to implement these changes. Future research will be needed to determine if improved identification leads to improved access to care and outcomes.
- AAP —
- American Academy of Pediatrics
- CPT —
- Current Procedural Terminology
- DC —
- District of Columbia
- EMR —
- electronic medical record
- FQHC —
- federally qualified health center
- LC —
- learning collaborative
- MOC —
- Maintenance of Certification
- MH —
- mental health
- MHPRI —
- Mental Health Practice Readiness Inventory
- PCP —
- primary care provider
- PDSA —
- QI —
- quality improvement
- SDQ —
- Strengths and Difficulties Questionnaire
- WCV —
- well-child visit
In the United States, up to 20% of children experience a mental health (MH) disorder in a given year,1 causing impairment and requiring intervention.2 Early identification is essential because the prompt recognition and treatment of MH issues can mitigate social and academic problems into adulthood; however, few youth receive this treatment.3–5 Many factors contribute to unmet MH needs, including the workforce shortage of child MH professionals.4,5 Pediatric primary care providers (PCPs) can play a critical role in ameliorating the MH service gap.6
Routine MH screening during annual well visits, encouraged by the American Academy of Pediatrics (AAP),7,8 is 1 strategy to identify concerns early and facilitate appropriate intervention.2,8–11 However, PCPs face numerous barriers to implementing routine MH screening, including lack of time, insufficient referral resources, inadequate reimbursement,12 and limited partnerships with MH providers.13 Indeed, over half of PCPs report never or rarely using a standardized MH screening tool.14
Little is written about practical strategies to improve MH screening rates, although the literature suggests that quality improvement (QI) can narrow the gap between recommended best practice and current care. QI has been used to improve early identification of and care for children with autism and adolescents and adults with depression when compared to the standard practice of a onetime educational intervention.15–18 Additionally, many states are successfully using QI to improve access to and quality of MH care.19
The QI learning collaborative (LC) is a model that supports large-scale practice change by facilitating learning sessions and coaching for groups of practitioners and organizations to develop, test, and implement sustainable improvement strategies.20 QI LCs have been successfully used to target health issues in the pediatric primary care setting21; however, we know of no researchers applying this approach to MH screening. We therefore sought to determine if participation in a QI LC is associated with improvements in a practice’s capacity to address MH issues through routine screening, coding, and documentation.
Our overall aim was to improve annual MH screening rates by using an approved tool for children aged 1 to 18 years. Specifically, we aimed to do the following: (1) increase practices’ preparedness to address MH concerns through such constructs as education and workflow planning, (2) increase the percentage of annual well-child visits (WCVs) in which an approved MH screening tool is administered to 50%, (3) increase the percentage of WCVs with documentation of MH screening results to 50%, and (4) increase the percentage of WCVs with appropriate Current Procedural Terminology (CPT) coding to 75%. In a preliminary needs assessment, it seemed few providers were regularly conducting MH screening; target aims were determined by content experts, taking into consideration published experience in other states.14
In Washington, DC, 3000 to 20 000 low-income children are estimated to have MH needs22,23; In 2012, the DC (District of Columbia) Collaborative for Mental Health in Pediatric Primary Care (also referred to as the DC Collaborative), a multidisciplinary coalition whose primary aim is to increase the integration of MH services into pediatric primary care, was formed. An early priority was to increase the rates of MH screening at WCVs for youth aged 0 to 21 years. Leveraging local QI infrastructure and funding from state MH and health agencies, the DC Collaborative designed, implemented, and evaluated a citywide longitudinal QI LC.
Recruitment and Enrollment
The LC was planned as a 9-month project (February–October 2014; “round 1”). Because of demand from practices to provide ongoing support, the LC was extended for 6 months (January–June 2015; “round 2”). Round 2 was open both to round 1 and new practices. No other widespread training on this topic occurred locally from February 2014 to June 2015. Practices were invited to participate via electronic communication through multiple outlets, including membership groups, practice networks, government agencies, and personal outreach. Recruitment materials included information about the DC Medicaid managed care requirement that all members receive an annual MH screen using an approved screening tool and encouraged LC participation as a strategy to support the implementation of this requirement in clinical practice. Inclusion criteria included practices who provided medical care to children and adolescents, with no exclusion criteria. Screening tools approved by the DC Department of Behavioral Health included the Ages and Stages Questionnaire: Social-Emotional24 (3–66 months), the Strengths and Difficulties Questionnaire (SDQ) (2–17 years),25 and the Patient Health Questionnaire-926 (≥18 years).
Participation was free to practices in the greater Washington, DC, metropolitan area; there was no cap on the number of practices who could enroll. Individual participants were given the option to earn American Board of Pediatrics or American Board of Family Medicine Part IV Maintenance of Certification (MOC) credits. Hour-for-hour continuing medical education credits could be earned for each individual project component. Providers could only earn MOC credit if their practice completed all of the following requirements during each of the 2 project rounds: (1) monthly chart audit reports, (2) monthly team meeting reports, (3) monthly team leader calls, and (4) 3 plan-do-study-act (PDSA) reports. Within practices that met these requirements, individual providers could earn MOC credit if they fulfilled the following requirements for each round: (1) complete an audit of 12 charts, (2) attend 5 team meetings, and (3) participate in 6 learning sessions.
DC practices were provided with additional incentives, including the AAP Mental Health Toolkit, the Ages and Stages Questionnaire: Social-Emotional Starter Kit (English and Spanish), MH screening posters (in English and Spanish), and access to on-site consultation from an MH QI coach. This project received exempt status from the Children’s National Health System institutional review board.
The overall LC design was adapted from other MOC projects implemented by Children’s National Health System and informed by the experience that practice coaching was an integral part of the model (Fig 1).21 Within a month of registration, a QI coach offered an on-site visit to review program activities and practice responsibilities. Practices and individual providers were registered with QI TeamSpace (projects.upiq.org), a centralized platform that confidentially manages project content and data collection.
A freely available MH resource guide was developed (www.dchealthcheck.net). Additional support, including practice coaching by QI and MH experts, general technical assistance, electronic medical record (EMR) support (eg, assistance developing automated smart forms), and scoring aids (eg, transparencies), was provided.
During months 0 to 3, practices used a structured tool to assess their workflow and preparedness for conducting MH screening and developed a screening implementation plan. Practices conducted a baseline chart audit of 30 patients seen for a WCV during the previous 6 months. Beginning in May 2014, practices performed monthly chart audits (minimum 10–15 charts). Charts for audit were randomly selected by the practice sites. Monthly hour-long webinars provided educational content on QI concepts, MH screening and implementation strategies, methods for engaging families, and management of common MH concerns in primary care. Technical assistance and 1 team leader call were provided between rounds 1 and 2 (November–December 2014). Archived webinars, patient handouts, and additional resources were uploaded to QI TeamSpace.
Practices received monthly report cards based on their chart review data. The report included information about their performance in each domain (eg, percentage of chart reviews in which screening was completed) for that month and the project period, screening targets, and distance from that goal. Each practice designed and implemented individualized PDSA cycles at ∼2-month intervals (3 cycles per round) on the basis of their initial practice assessment, the previous month’s chart audit, or other available data (Table 1). Although practices were not required to seek approval for their PDSA cycles, they were encouraged to consult our expert team.
During monthly team leader calls, deidentified data were used to track progress and compare results. Run charts for practice-specific data and overall LC-wide data were presented and discussed. Practices also shared their experiences with MH screening (including the results of their PDSA cycles, challenges, and successes) to facilitate shared learning.
The AAP Mental Health Practice Readiness Inventory (MHPRI)27 was used to assess practices’ self-reported readiness to address MH issues in 5 areas: community resources, health care financing, support for children and families, clinical information systems and delivery system redesign, and decision support for clinicians.
Chart audit measures, which were selected via expert consensus on the basis of study aims, included the following: (1) MH screening completed by using an approved screening tool, (2) screening scored and documented, and (3) screening billed by using CPT code 96110. Additional components of the chart audit not relevant to the aims included patient age and whether the patient had a previously identified MH issue. Practices were coached on selecting representative, random charts for monthly audits; given that practices selected their own charts, selection was not done in a blinded or randomized fashion.
Practices were coached to implement screening in a stepwise manner in which they started screening a narrow age range of patients and improved workflow processes before expanding. Most practices selected charts for auditing from the age range in which they were targeting screening. As a balancing measure, providers completed a brief survey at the midpoint and end of the project to assess the perceived impact and effectiveness of integrating project activities within practice workflow.
Practices submitted standardized, structured data reporting forms on QI TeamSpace. Data were reviewed for completeness and accuracy both independently and jointly by 3 members of the project team.
Data analyses were performed in Excel. Changes in practice readiness to address MH were examined via practice averages on the 5 domains of the AAP MHPRI reported at pre-, mid-, and end-project time periods. Monthly chart audits examined changes in the percentage of (1) annual WCVs in which an approved screening tool was used compared with the total number of charts reviewed, (2) results documented compared with all instances when a screening tool was used, and (3) appropriate CPT codes used compared with all instances when a screening tool was used. Individual control charts were used to identify process variation and/or trends within overall practice data. Control charts were created by using Statistical Process Control Charts for Excel.
A retrospective chart review of WCVs completed during and 6 months after the LC was conducted at the largest participating practice to determine sustained effects of the LC. This practice was selected because of its large size and consistency in using EMR smart forms, making data collection more feasible. For this practice, data were pulled for all WCVs of 4 to 10-year-olds. Charts were examined for completion of the SDQ, the approved screening tool for children this age. Data abstraction was completed by using structured screening data fields, so only those screens documented in this way in the chart were included in the analyses.
Of the 19 practices enrolled at any point, 8 were academic health centers, 6 were private practices, 4 were federally qualified health centers (FQHC), and 1 was an outpatient specialty clinic. Ten practices, including 6 academic community health centers, 2 FQHCs, and 2 private practices, were considered active participants throughout both rounds of the LC and were included in the analyses. Reasons for exclusion included participation in only 1 LC round (6) and withdrawal from the project (3). Reasons practices withdrew or only completed 1 round included being a single provider practice, not being a primary care clinic, being a practice outside of DC, and internal staffing and infrastructure issues. The mean practice size within the 10 practices was 11 (range, 3–22) providers, with a total of 107 providers participating.
On average, practices submitted 41.4 charts at baseline (SD = 30.6; median = 31.5) and 20 charts per month (SD = 16.2; median = 15.0) for a total of 2721 charts across 10 practices. Providers attended an average of 11 practice QI team meetings (SD = 3.6; median = 11.5). Providers completed a mean 12.33 continuing medical education hours (SD = 8.4 hours; median = 10 hours) for participating in practice team meetings, team leader webinars, learning sessions, and chart audits.
Results from the self-reported AAP MHPRI indicated mean improvement of 17% across all 5 domains from baseline. Overall, practices saw the most improvement in the health care financing (23% increase) and community resources (20% increase) domains (Tables 2 and 3).
Monthly chart audits demonstrated improvement at project end across all domains assessed (Figs 2–4). Data plotted by using the individual x-control chart shows significant improvement within the first 3 months, moving toward a more stable rate of improvement among practices by project end. Overall, the screening and billing rates improved by 73% and 89% from baseline, respectively. There was variability in screening patterns across practices (Table 3). Approximately one-quarter (25.5%) of screens were in the elevated range, suggesting an MH concern, compared with 9% at baseline and slightly higher than national trends. Analysis of the comprehensive chart review from the largest participating practice showed sustained increases in the percentage of MH screens completed and documented in the EMR during the LC, beginning in July 2014, and for 6 months afterward (Fig 5). Data from this practice also elucidated provider follow-up actions after screening indicated a concern, which ranged from management by the pediatric provider (26%) to referral to internal (17%) or external (14%) MH resources. Twenty-five percent were already receiving services of some kind, 10% declined services and/or referral, and 9% were other or not documented.
Response to the balancing measure survey was poor (18%), limiting its usefulness in assessing the impact on other aspects of practice. Anecdotally, on the basis of follow-up communication with participating practices and the state Medicaid agency, we determined that practices’ comfort and efficiency with MH screening improved over time. For example, providers reported on postproject surveys that they felt “much better able to screen and refer to community resources” and felt “more comfortable with some basic management in primary care of anxiety and behavior.” In our follow-up with practices who did not complete the LC, it seems that barriers to participation included lack of investment by senior leadership to prioritize the project, existing challenges in managing patient workflow, and lack of alignment with the project design of supporting primarily DC-based practices.
Overall, the LC was an effective way to support participating practices in their efforts to implement, document, and bill for routine MH screening at WCVs. Substantive improvements were seen across all domains measured and, in at least 1 participating practice, were sustained over time. Additionally, practices demonstrated increased preparedness in several domains to address the MH needs of their patients, as measured by the AAP MHPRI.
Although the LC was associated with improvements in MH screening practices, a greater investment of time and resources was required to achieve this result than is typically dedicated to continuing education in the medical setting. To help offset this investment, practices were offered intensive and ongoing support in a variety of domains, including QI expertise, data analysis, technology support, and content expertise. With the low baseline rates of screening, despite the fact that providers were aware it is a best practice, we suggest that intensive support was needed and a traditional onetime educational approach would not be adequate. The need for intensive practice engagement likely contributed to the fact that some practices were not able to complete both rounds of the project. This is consistent with the findings of researchers conducting a systematic review on the impact of LCs that they have a positive and sustained effect, yet their impact may be difficult to predict and dependent in large part on organizational culture.28 However, for practices who are interested in improving their screening practices and willing to adopt a QI approach to change, these findings should be encouraging.
Although this project did not track reimbursement rates, substantive improvements were seen in CPT code use, which in theory should increase revenue. The 2015 DC Medicaid reimbursement rate was $5.19 for behavioral health screening (96127 CPT code) and $10.30 for developmental screening (96110 CPT code). Of note, there is considerable variability in these rates nationally, with some states lower and some much higher.29 Actual reimbursement could be impacted by any number of factors, including prenegotiated payment agreements and completed submission of claims. A more detailed analysis is needed to determine the actual fiscal impact for individual practices.
As a result of the overall low response rate to the midproject and end-of-project surveys, it was difficult to ascertain the impact that implementing MH screening had on workflow and practice efficiency, which can be a significant barrier to implementing MH screening. Practices that most effectively used multidisciplinary teams seemed to have the greatest success. Screening activities continued postproject for most, suggesting that practices were not unduly burdened. Follow-up qualitative interviews are being conducted with participating PCPs and families who completed MH screens to better understand the impact of screening on workflow and clinical care.
There were several limitations to this project, which may limit its generalizability. First, it was conducted in a locale with QI infrastructure, a screening mandate, and academic resources. Many practices had previously participated in similar projects, so they may have had more comfort and skills with QI methodology than providers in other settings. Chart review was self-reported, and thus accuracy cannot be verified. Participation was voluntary and not randomly selected; therefore, there may be a selection bias, with those who participated being more highly motivated. Additionally, lack of control data from neighboring practices limits our ability to confirm that nonparticipating practices did not make similar improvements. Lastly, we do not have data regarding the proportion of patients with positive MH screen results who received care, outcomes for these patients, or the financial impact of the project.
Despite the limitations, there was evidence for meaningful improvement during the 15-month QI LC, which was sustained in at least 1 practice for 6 months afterward. More information is needed about the burden placed on practices and providers to implement these changes, which will in turn inform recommendations for improvement and additional supports needed by practices. For practices that were uninterested or unable to fully engage in this project, a more individualized approach may be needed, although there is good reason to suggest a QI approach would be helpful. Additionally, cost-effectiveness analysis and expansion of this approach to other geographic regions will be important to determine long-term feasibility and sustainability. With these early findings, we suggest that the LC model can improve MH screening practices in pediatric primary care, an important first step toward early identification of patients with MH concerns. Future research will also be needed to determine if improved identification leads to improvements in access to MH care and patient outcomes.
Support was provided by the Children’s National Health Network. We thank Dr Chaya Merrill for her review and advice.
- Accepted August 23, 2017.
- Address correspondence to Lee S. Beers, MD, Division of General and Community Pediatrics and Child Health Advocacy Institute, Children’s National Health System, 111 Michigan Ave NW, Washington, DC 20009. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by contract RM-014-SAS-165-BY0-DJW from the District of Columbia Department of Behavioral Health and grant CHA.PSMB.CNMC.PGR.M-C.052013 from the District of Columbia Department of Health. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of these agencies.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Foy JM; American Academy of Pediatrics Task Force on Mental Health
- American Academy of Pediatrics
- Weitzman C,
- Wegner L; Section on Developmental and Behavioral Pediatrics; Committee on Psychosocial Aspects of Child and Family Health; Council on Early Childhood; Society for Developmental and Behavioral Pediatrics; American Academy of Pediatrics
- Committee on Psychosocial Aspects of Child and Family Health and Task Force on Mental Health
- Sheldrick RC,
- Merchant S,
- Perrin EC
- Pidano AE,
- Honigfeld L,
- Bar-Halpern M,
- Vivian JE
- Carbone PS,
- Norlin C,
- Young PC
- Larner College of Medicine, University of Vermont
- Delmarva Foundation
- Brink R
- ↵ASQ: SE-2. Baltimore, MD: Brookes Publishing Co; 2015. Available at: www.brookespublishing.com/resource-center/screening-and-assessment/asq/asq-se-2/. Accessed March 15, 2016
- American Academy of Pediatrics
- Schouten LM,
- Hulscher ME,
- van Everdingen JJ,
- Huijsman R,
- Grol RP
- American Academy of Pediatrics
- Copyright © 2017 by the American Academy of Pediatrics