Effectiveness of an Asthma Quality Improvement Program Designed for Maintenance of Certification
OBJECTIVE: Pediatricians are required to perform quality improvement for board recertification. We developed an asthma project within the Pediatric Physicians’ Organization at Children’s, an independent practice association affiliated with Boston Children’s Hospital, designed to meet recertification requirements and improve asthma care.
METHODS: The program was based on the learning collaborative model. We developed practice-based registries of children 5 to 17 years of age with persistent asthma and helped physicians improve processes of asthma care through education, data feedback, and sharing of best practices.
RESULTS: Fifty-six physicians participated in 3 cohorts; 594 patients were included in the project. In all cohorts, improvements occurred in the use of asthma action plans (62.4%–76.8% cohort 1, 50.6%–88.4% cohort 2, 53.0%–79.6% cohort 3) and Asthma Control Tests (4.6%–55.2% cohort 1, 9.0%–67.8% cohort 2, 15.2%–61.4% cohort 3). Less consistent improvements were observed in seasonal influenza vaccines, controller medications, and asthma follow-up visits. The proportion of patients experiencing ≥1 asthma exacerbation within the year declined in all 3 cohorts (37.8%–19.9%, P = .0002 cohort 1; 27.8%–20.7%, P = .1 cohort 2; 36.6%–26.9%, P = .1 cohort 3). For each cohort, asthma exacerbations declined to a greater extent than those of a comparison group.
CONCLUSIONS: This asthma quality improvement project designed for maintenance of certification improved processes of care among patients with persistent asthma. The learning collaborative approach may be a useful model for other board-recertification quality improvement projects but requires a substantial investment of organizational time and staff.
- ABP —
- American Board of Pediatrics
- HEDIS —
- Healthcare Effectiveness Data and Information Set
- MOC —
- maintenance of certification
- PCP —
- primary care pediatrician
- PPOC —
- Pediatric Physicians’ Organization at Children’s
- QI —
- quality improvement
In 2010, the American Board of Pediatrics (ABP) replaced its previous certification requirements with a new 4-part continuous process for recertification. This new maintenance of certification (MOC) process, designed to demonstrate pediatricians’ lifelong commitment to learning, clinical competence, and practice improvement, includes (1) validation of professional standing and licensure, (2) documentation of ongoing lifelong learning and self-assessment, (3) a secure examination, and (4) “Performance in Practice,” a requirement for quality improvement (QI) activities performed within the candidate’s own practice.1,2 It is expected that through ABP-approved QI projects, pediatricians will learn how to measure and improve quality in their own practices. The implementation of this new MOC process was met with some skepticism among physicians who felt these activities would be time-consuming, expensive, and unlikely to yield sustainable improvements in pediatric practice and/or beneficial outcomes for patients.3,4
To facilitate meeting the MOC requirements among its member physicians, the Pediatric Physicians’ Organization at Children’s (PPOC), an independent practice association of more than 200 primary care pediatricians (PCP) affiliated with Boston Children’s Hospital, developed an ABP-approved QI project aimed at enhancing the care of children with persistent asthma. We chose to focus on asthma for this initiative because it is one of the most common chronic disorders in childhood, affecting nearly 7 million children in the United States in 2010.5 Despite such a high prevalence, studies show that the care of pediatric persistent asthma remains inadequate6 and the goals of asthma management aimed at reducing impairment and risk of exacerbations are not being met.7
We designed our program to focus on improving practice performance in the following 5 processes of asthma care suggested by the literature to improve outcomes for children with asthma: (1) receipt of a seasonal influenza vaccine,8,9 (2) completion of ≥2 maintenance visits per year for asthma care,10 (3) filling ≥1 prescription for an asthma controller medication within the year,11–16 (4) completion of at least 1 Asthma Control Test,17 and (5) completing at least 1 updated asthma action plan.18 We also measured the occurrence of asthma exacerbations requiring medical attention to determine if these interventions reduced the risk of exacerbations among participating patients.
The PPOC is an independent practice association of >80 privately owned pediatric practices affiliated with Boston Children’s Hospital, which cares for an estimated 300 000 children in eastern Massachusetts. All PPOC pediatricians who were due for ABP recertification were eligible to participate in the project. Based on the dates of recertification, 3 cohorts were enrolled. The first cohort participated from December 1, 2009, through November 30, 2010, the second from October 1, 2010, through September 30, 2011, and the third from March 1, 2011, through February 28, 2012.
A registry of patients with persistent asthma was developed for each participating PCP in 2 ways. First, the PPOC has access to claims data from 4 commercial insurance companies in Massachusetts, which includes an estimated one-quarter of all PPOC patients. Using this database, children 5 to 17 years of age and who met Healthcare Effectiveness Data and Information Set (HEDIS) criteria for persistent asthma were included in the project. HEDIS criteria for persistent asthma identify patients who have experienced 1 or more of the following events in each of 2 consecutive calendar years: at least 1 inpatient admission or emergency department visit with asthma as the principal diagnosis; at least 4 outpatient asthma visits with asthma as 1 of the listed diagnoses and at least 2 medication-dispensing events; or at least 4 asthma medication–dispensing events (unless all prescribing events were for leukotriene receptor antagonists).19 Second, patients who were not included in the claims database or who were included but did not meet formal HEDIS criteria for persistent asthma were chosen by the PCP for inclusion if the PCP felt they suffered from persistent asthma based on clinical knowledge of the patient or evidence from the practice’s medical record.
The project was organized and run by 2 physician medical directors, the organization’s director of QI, and a QI consultant who was primarily responsible for producing data reports and coaching the participating practices. The goal of the project was for each participating PCP to improve performance on the 5 identified processes of asthma care among their identified patients with persistent asthma. It was planned for each cohort to meet in person 4 times during their respective 12-month intervention period, although because of delays in getting necessary approvals, cohort 1 was able to meet only 3 times. These face-to-face meetings each lasted 2 hours and used the Institute for Healthcare Improvement’s learning collaborative model, a longitudinal learning curriculum bringing together multiple practice teams to seek improvement in performance in a collaborative fashion.20 Approximately one-third of each meeting was devoted to didactic presentations on asthma epidemiology and management and core QI concepts. Asthma concepts covered included the following: recommendations of the 2007 National Heart, Lung, and Blood Institute asthma guidelines; environmental factors in asthma control; longitudinal monitoring of asthma patients; and the application of the chronic care model and care coordination to pediatric asthma. Core QI concepts included use of Plan-Do-Study-Act cycles, defining and measuring process and outcome measures, and creating run charts to track performance over time. The remaining two-thirds of each meeting was devoted to (1) disclosure of each participant’s performance to date; (2) facilitated small group discussions (4–10 participants per group); and (3) large group discussions focused on identification of best practices for patient education, patient activation and adherence, and asthma care delivery.
Between in-person meetings, the QI consultant had at least monthly communication with each participating practice via e-mail or telephone to collect data, review progress, and troubleshoot implementation issues. Throughout the program, the QI consultant devoted ∼10 hours per week to the project. The medical directors and the director of QI attended 1 to 2 meetings per month to review the project’s progress and devoted an average of 1 to 2 hours per week to planning aspects of the project.
For each cohort, the baseline period was defined as the 1-year period preceding the project, and the intervention period was defined as the 1-year period of active project participation. Compliance with asthma care process measures was determined by retrospective chart review and was available for all patients included in the project. Asthma exacerbations were defined as any of the following: (1) hospitalizations or emergency department visits for asthma, (2) outpatient visits coded as asthma with acute exacerbation or with status asthmaticus, and (3) outpatient visits for asthma with an oral steroid prescribed and were determined by analysis of paid insurance claims. Claims data were available only for the subset of the project’s patients who were insured by 1 of 4 commercial insurance companies that provide claims data to the PPOC. The proportion of patients meeting each process measure and the proportion experiencing 1 or more exacerbations over the 12-month period were compared between the baseline and intervention periods for each cohort by the McNemar test.
To determine whether changes in the proportion of patients experiencing 1 or more asthma exacerbations from the baseline to intervention period were associated with participation in the project, proportions were determined for a comparison group for each cohort consisting of PPOC patients who met HEDIS criteria for persistent asthma for the same time periods as the patients enrolled in the project but whose PCP was not participating in the MOC project. The change in the proportion of patients experiencing 1 or more asthma exacerbations from the baseline to intervention periods was compared between the MOC patients and the control patients by logistic regression analysis.
This project met our institution’s definition of QI and was therefore exempt from institutional review board review.
Cohort 1 consisted of 15 participating physicians from 12 different practices and 194 patients, cohort 2 consisted of 24 participating physicians from 21 different practices and 268 patients, and cohort 3 consisted of 17 physicians from 12 different practices and 132 patients. A description of participating physicians is shown in Table 1.
Comparisons of baseline to intervention performance for the 5 program process measures are shown in Figure 1. Cohort 1 exhibited significant improvements in updated asthma action plans (62.4%–76.8%%, P = .001), use of controller medications (86.1%–92.3%, P = .02), and completed Asthma Control Tests (4.6%–55.2%, P < .001). Cohort 2 demonstrated significant improvements in all 5 process measures, including seasonal influenza vaccine (62.9%–83.5%, P < .0001), asthma action plans (50.6%–88.4%, P < .0001), controller medications (72.7%–94.0%, P < .0001), Asthma Control Tests (9.0%–67.8%, P < .0001), and at least 2 asthma visits within the year (75.3%–85.0%, P = .003). Cohort 3 demonstrated improvements in asthma action plans (53.0%–79.6%, P < .0001) and Asthma Control Tests (15.2%–61.4%, P < .0001).
The proportion of patients experiencing 1 or more asthma exacerbations within the year declined from the baseline year to the intervention year in all 3 cohorts. The decline was statistically significant for cohort 1 and of borderline statistical significance for cohorts 2 and 3 (37.8%–19.9%, P = .0002 for cohort 1; 27.8%–20.7%, P = .1 for cohort 2; 36.6%–26.9%, P = .1 for cohort 3; Fig 1). To account for possible secular trends in asthma exacerbations, we compared the baseline to intervention changes within each cohort to a control set of PPOC patients whose primary care physicians did not participate in the MOC program. As shown in Figure 2, for each of the 3 cohorts, asthma exacerbations declined to a greater extent in the MOC group compared with the comparison group, and the comparison was of borderline statistical significance for cohorts 1 and 2 (17.9% decline for MOC participants vs 7.8% decline for controls in cohort 1, P = .07; 7.1% decline vs 0.5% increase in cohort 2, P = .1; 9.7% decline vs 7.1% decline in cohort 3, P = .5).
Fifty-three (94.6%) of the 56 physicians who participated in the program qualified to receive 25 Part 4 credits toward their recertification by ABP.
The implementation of this ABP-approved asthma QI project by 3 cohorts of pediatricians using practice-based registries and a learning collaborative model was associated with significant improvements in processes of asthma care. In each cohort, a higher proportion of patients had updated asthma action plans and had completed at least 1 Asthma Control Test during the intervention phase compared with the baseline period. There were also improvements, although less dramatic and consistent, in receipt of seasonal influenza vaccine, prescription of at least 1 controller medication, and completing 2 asthma office visits within the year. It should be noted that these 3 measures began at higher levels in the baseline period than asthma action plans and Asthma Control Tests and thus had less room for improvement.
An important question for QI work is whether process improvements translate into improved health outcomes for patients. In this case, we measured the proportion of patients experiencing 1 or more asthma exacerbations requiring medical attention because studies have shown that exacerbations are a useful marker for uncontrolled asthma.21,22 We compared the proportion experiencing an asthma exacerbation from the baseline to intervention period in each cohort and found consistent declines in all 3 cohorts, which suggests that the improvements we observed in process measures translated into improved patient outcomes. However, because asthma exacerbation rates can fluctuate seasonally and from year to year, and because there may be a natural trend toward a reduction in asthma severity for some children with persistent asthma over time,23,24 we also compared the baseline to intervention change in exacerbations to an external control group for each cohort. With this analysis, we found a greater decline in exacerbations among patients of MOC participants compared with patients of nonparticipants, although the difference was only of borderline statistical significance for cohorts 1 and 2. This result raises the question of what level of evidence is necessary to conclude that a QI project was successful, compared with that of traditional medical research. The level of “proof” typically required in research studies may be difficult to obtain in QI projects for reasons such as sample size limitations to lack of comparison groups. The most important such limitation in our study was the lack of an ideal comparison group. The most suitable comparison group that we were able to use consisted of patients from our own network whose physicians did not participate in the MOC project. However, because asthma has been an important focus of QI efforts within our organization for many years and because our member physicians interact in a variety of ways, there may have been “contamination” of our comparison group to some extent, making it more difficult to show a difference between our participants and controls.
Perhaps the most relevant lessons from this project revolve around the value of the ABP’s QI requirement for recertification and the suitability of the learning collaborative model for others who wish to emulate it. Beyond the objective evidence of improvements in care processes discussed previously, the staff involved in the project were enthusiastic about the learning collaborative approach, particularly in its ability to allow practices to share lessons they were learning throughout the project with their peers to accelerate one another’s progress. This benefit would be lacking from a QI approach that involved more of a “top-down” model with the central organization teaching practices to improve in a one-on-one manner or indeed that involved individual practices working on their own in isolation. The cost of such an approach, however, is the substantial investment of organizational resources required, particularly the time of staff and medical directors with advanced QI training. It is unclear how many pediatric primary care organizations are equipped to field similar learning collaboratives or how many PCPs are members of an organization willing and able to run them. As for our own organization, in substantial part because of the experience of the asthma project reported here, we are committed to the learning collaborative approach as an effective method of practice-based QI and have embraced it for many of our major QI initiatives. To date, we have successfully obtained ABP MOC credit for our physicians participating in learning collaboratives focused on well-child care, care of the adolescent, patient experience, family engagement, achieving meaningful use of the electronic health record, effective use of the electronic health record to improve care, integrating behavioral health services in the primary care medical home, and group medical visits. We are in the planning stages for additional ABP-approved learning collaboratives on office management of concussions and musculoskeletal injuries and ambulatory risk management.
This asthma QI project demonstrated that an ABP-approved project based on practice-based registries and a learning collaborative approach can improve processes of care among patients with persistent asthma while allowing physicians to obtain necessary MOC credits, albeit with a substantial investment of organizational time and effort. Because of its success, we have expanded this model to other areas of QI and have made it our organization’s central approach to practice-based improvement.
- Accepted February 10, 2014.
- Address correspondence to Louis Vernacchio, MD, MSc, Pediatric Physicians’ Organization at Children’s, 77 Pond Ave, Ste 205C, Brookline, MA 02445. E-mail:
Dr Vernacchio designed the analysis, performed the original analysis, reviewed and revised the analysis, and drafted the original manuscript; Ms Francis, Dr Epstein, Ms Santangelo, and Dr Risko conceptualized and designed the project, supervised all aspects of the implementation, and reviewed and revised the analysis; Ms Trudell performed the original analysis and reviewed and revised the analysis; Ms Reynolds drafted the original manuscript; and all authors reviewed and revised the manuscript, and approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This work was funded by internal funds of the Pediatric Physicians’ Organization at Children’s.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- ↵American Board of Pediatrics. To maintain certification. About maintenance of certification (MOC). Available at: https://www.abp.org/ABPWebStatic/?anticache=0.9492465040985828#murl%3D%2FABPWebStatic%2Fmocabout.html%26surl%3D%2Fabpwebsite%2Fmoc%2Faboutmoc%2Faboutmoc.htm. Accessed June 20, 2013
- Strasburger VC,
- Greydanus DE
- ↵Hertz BT. Where do you stand? Maintenance of certification process sparks debate. Med Econ. 2011;88(23):24–25, 30–21, 36–29
- ↵Halterman JS, Auinger P, Conn KM, Lynch K, Yoos HL, Szilagyi PG. Inadequate therapy and poor symptom control among children with asthma: findings from a multistate sample. Ambul Pediatr. 2007;7(2):153–159
- ↵Institute for Healthcare Improvement. The breakthrough series: IHI’s collaborative model for achieving breakthrough improvement. IHI innovation series white paper. 2003. Available at: www.ihi.org/resources/Pages/IHIWhitePapers/TheBreakthroughSeriesIHIsCollaborativeModelforAchievingBreakthroughImprovement.aspx. Accessed April 18, 2014
- Copyright © 2014 by the American Academy of Pediatrics