OBJECTIVE: Immunization quality improvement (QI) interventions are rarely tested as multicomponent interventions within the context of a theoretical framework proven to improve outcomes. Our goal was to study a comprehensive QI program to increase immunization rates for underserved children that relied on recommendations from the Centers for Disease Control and Prevention’s Task Force on Community Preventive Services and the framework of the Chronic Care Model.
METHODS: QI activities occurred from September 2007 to May 2008 at 6 health centers serving a low-income, minority population in Washington, DC. Interventions included family reminders, education, expanding immunization access, reminders and feedback for providers, and coordination of activities with community stakeholders. We determined project effectiveness in improving the 4:3:1:3:3:1:3 vaccination series (4 diphtheria-tetanus-pertussis vaccines, 3 poliovirus vaccines, 1 measles-mumps-rubella vaccine, 3 Haemophilus influenzae type b vaccines, 3 hepatitis B vaccines, 1 varicella vaccine, and three 7-valent pneumococcal conjugate vaccines) compliance.
RESULTS: We found a 16% increase in immunization rates overall and a 14% increase in on-time immunization by 24 months of age. Improvement was achieved at all 6 health centers and maintained beyond 18 months.
CONCLUSION: We were able to implement a comprehensive immunization QI program that was sustainable over time.
- ACIP —
- Advisory Committee on Immunization Practices
- CCM —
- Chronic Care Model
- CDC —
- Centers for Disease Control and Prevention
- CoCASA —
- Comprehensive Clinic Assessment Software Application
- IIS —
- immunization information system
- MOGE —
- moved or gone elsewhere
- PCV7 —
- 7-valent pneumococcal conjugate vaccine
- QI —
- quality improvement
- WIC —
- Women, Infants and Children
Nearly 1 in 4 young children in the United States are not appropriately immunized.1 Underimmunized children are more likely to be poor,2,3 from single-parent households,2 African American,3–6 and from inner cities.7–9 The Centers for Disease Control and Prevention’s (CDC’s) Task Force on Community Preventive Services has recommended 13 interventions with the potential to improve immunization rates based on a literature review of their efficacy.10,11 In 2008, the recommended interventions related to outpatient pediatric practice included client reminder/recall systems, reduction of out-of-pocket costs, expansion of access in clinical settings, provider reminder/recall, multicomponent educational strategies, assessment and feedback for providers, standing orders, and vaccination programs in Women, Infants and Children (WIC) settings. To our knowledge, no study has evaluated the impact of implementing all the Task Force recommendations related to ambulatory pediatric practice jointly.
Because the Task Force recommendations only include proven effective strategies, they potentially miss beneficial but untested interventions. To examine this possibility, it is useful to consider the fit of the Task Force recommendations to a theoretical framework designed to improve patient outcomes such as the Chronic Care Model (CCM). The CCM includes 6 elements: community resources and policies, patient self-management support, health care organization, delivery system design, decision support, and clinical information systems.12,13 Implementation of the model has been shown to improve patient outcomes for various populations, including children, and various health conditions.14,15 The CCM is applicable to improving immunization coverage because, similar to chronic care, immunization occurs in the outpatient setting and requires a sustained commitment to preventive services by the community and health care system.
Our 6 health centers instituted an immunization quality improvement (QI) program in which we implemented all of the Task Force’s recommended interventions relevant to outpatient pediatric practice. We fit the recommendations to the CCM framework and decided to add another intervention we felt was missing, yet important for success. The aim of this study was to determine the effectiveness of our immunization QI program for increasing compliance with the 4:3:1:3:3:1:3 vaccination series (which includes 4 diphtheria-tetanus-pertussis vaccines, 3 poliovirus vaccines, 1 measles-mumps-rubella vaccine, 3 Haemophilus influenzae type b vaccines, 3 hepatitis B vaccines, 1 varicella vaccine, and 3 doses of 7-valent pneumococcal conjugate vaccine [PCV7]) among children 19 to 35 months old.
Activities occurred from September 2007 to May 2008 at 6 health centers associated with a pediatric hospital in Washington, DC. The largest center, center C, is colocated with the hospital, whereas the others are community-based. Center F includes a permanent site and a mobile van. All centers serve patients who are predominantly poor and publicly insured (Table 1). The majority of patients are African American at all centers, with the exception of center B, which is predominantly Hispanic. Collectively, ∼30 physicians, 3 nurse practitioners, 90 residents, and 42 nurses immunize ∼37% of DC’s pediatric population annually.
Our QI program paid staff consisted of 1 physician 30% time, and 2 full-time administrators. The study protocol was approved by the Children’s National Medical Center institutional review board.
Planning the QI Initiative
We first determined the fit of the Task Force recommendations with the CCM (Table 2).16 Although the fit was good, there remains a deficiency of studies examining the interaction between health care institutions and the community. According to the CCM, health care organization (defined as its goals and values and relationships with other community stakeholders) is a vital pillar of the chronic care edifice. Therefore, we chose to add community partnership to our activities.
We chose not to implement 1 Task Force recommendation—reducing out-of-pocket costs. This intervention did not fit well into the CCM framework and was not an important consideration for our patients because the majority was Medicaid enrollees who did not incur personal costs for immunization.
Implementing the QI Initiative
Table 2 describes the changes we implemented and evaluated. All changes were first implemented at health center C where our program director was located. We implemented each change on a small scale for 1 to 3 weeks before seeking feedback from staff and/or families. With feedback, we altered the interventions as needed. Project leaders met monthly with immunization champions (including physicians, nurses, and administrators) from each health center, representatives of the DC Department of Health (DOH) Immunization Program, and the four DC Medicaid health plans to share ideas and discuss intervention implementation strategies. Afterward, immunization champions introduced the changes at their respective centers, tailoring them to enhance performance improvement for their setting.
Collaboration With Community Stakeholders
The DC DOH helped us to identify incorrect monthly health center vaccine orders (eg, insufficient ordering quantities based on last year’s monthly order) and common practitioner administration errors (eg, administering the third hepatitis B dose too soon). Collaboration with the Medicaid plans enhanced outreach. Our QI staff made 2 phone calls and mailed a postcard to families with children overdue for immunizations. If these attempts were unsuccessful, the health plans could sometimes supply new contact or health practitioner information and, if not, could perform home visits.
Provider Reminder/Recall and Assessment and Feedback for Providers
The DC DOH Immunization Information System (IIS) was used to generate reminders for our practitioners for each patient encounter. Individual patient IIS records listed past immunizations and those that were currently due and overdue. Since 2008, reporting to the IIS has been mandatory for all DC health care providers, licensed daycare providers, and public schools, making a very complete registry. According to the CDC IIS Annual Report, in 2008, the IIS captured 100% of DC’s childhood population between 4 months and 6 years old.17
The IIS was also used to produce immunization coverage summary reports. Monthly immunization compliance run charts (compliance rates graphed over time) were created for each health center individually and overall. Run charts for each center were shared with all practitioners as process performance feedback. All practitioners participated in at least 3 training sessions to learn how to interpret IIS records, strategize intervention implementation, and discuss run chart trends.
Expanding Access in Clinical Settings
To expand access to immunizations, practitioners at all health centers began immunizing at unscheduled “sick” visits in addition to well visits. Another appointment was scheduled for children receiving immunizations at sick visits who were also due for a well examination. Parents were only given health certificates (required for child care and school enrollment) once the child completed a well examination. Access to immunizations was also increased at health centers C and F with immunization-only clinics (held twice during influenza season at center C and monthly at center F).
Because of the medical complexity of some of our patients, we chose to have a physician involved in all immunization-only visits. As a modified version of standing orders, nurses could initiate immunization orders for “shots-only visits,” but physicians approved all orders before immunizations were administered.
Client Reminder/Recall Systems
We sent reminder autodialer calls monthly to children due for immunizations. Because of our large patient population, we focused on children <24 months old. To recall children overdue for immunizations, we made least 2 live phone call attempts. When a parent answered, we attempted to schedule an appointment. If we were unable to reach a family via the telephone, we sent a postcard. Because outreach for overdue immunizations was labor intensive, we limited this activity to children <60 months old.
We created informational posters with pictorial examples of vaccine-preventable diseases, school immunization requirements, and immunization myths and facts. Because physical space varied among our health centers, centers A, B, and C placed posters in all examination and waiting rooms, whereas D had them in the procedure room, and E had them in the triage room. As an additional educational intervention, forms were created for practitioners to document for parents the immunizations their child received that day and the due date for future immunizations.
Vaccination Programs in WIC Settings
WIC is a federally subsidized nutritional program for pregnant women, infants, and children. For our intervention, we prescreened the IIS record for every WIC client <6 years old ahead of WIC appointments at the center colocated with health center C. We then asked WIC personnel to refer underimmunized children to our health center to receive same-day immunizations immediately after their WIC appointment.
We asked immunization champions to assess the frequency with which activities were implemented at their site. Other process measures we tracked included the monthly number of calls for due immunizations that connected to a person or voicemail, and the number of phone calls made and postcards sent for overdue immunizations. We estimated the number of provider reminders generated by using billing data to determine the number of patient visits that occurred in the date range that IIS records were generated.
We measured the outcome of our QI program by using a before-and-after observational approach. Namely, the DC DOH evaluated the immunization coverage of our health centers (done routinely for all DC participants of the Vaccines for Children Program) by using the CDC’s Comprehensive Clinic Assessment Software Application (CoCASA) (earlier versions known as CASA). An immunization was considered valid only if it was administered at the recommended age/interval per the Advisory Committee on Immunization Practices (ACIP). Immunization rates for the health centers were compared at 3 time points: spring 2006 (preintervention), spring 2008 (after all QI activities were deployed), and spring 2009 (18 months postintervention initiation). Data were first imported from the DC IIS into CoCASA. For any child found to be overdue according to the IIS, the health center’s medical chart was reviewed. Any additional immunizations recorded in the chart were added into CoCASA (and the IIS).
Our primary outcome was immunization coverage for 19- to 35-month-old children for the 4:3:1:3:3:1:3 series. Although ACIP recommended four doses of PCV7 at the time, we chose the 4:3:1:3:3:1:3 series because our health centers experienced frequent shortages of PCV7 during the national shortage in 2004.18 The 4:3:1:3:3:1:3 series was a key series assessed by the CDC in 2008 for children ages 19 to -35 months.19
Secondary outcomes were also assessed to determine the timeliness of immunization and sustainability of results. To examine the effect of the QI initiative on timeliness, we compared series completion rates among 19- to 24-month-old children in 2006 and 2008. This age range, which is 0 to 6 months beyond the upper age limit of when the 4:3:1:3:3:1:3 vaccines are due, is how the CDC’s Immunization Program Evaluation reporting requirements define “on time.” To determine sustainability, coverage for 2009 was compared with immunization coverage for 2008. All outcomes were considered as dichotomous variables (series complete or not). Comparisons were made by using the χ2 test.
We performed analyses including all patients for whom 1 of our health centers was the provider of record in the IIS. We also performed analyses excluding all patients we deemed a MOGE (moved or gone elsewhere). We defined MOGE as any child with documentation of (1) a move out of the area, (2) a change to a health care provider not affiliated with one of the study health centers, or (3) no clinical encounter in >12 consecutive months for whom all outreach attempts were unsuccessful (wrong number/phone disconnected and postcard returned to sender/no response). The second set of analyses more accurately reflects immunization coverage for patients who used our health centers for primary care services versus urgent care only, because providers in the IIS are assigned solely according to which institution administered the child’s most recent immunization. Health center D, located at the main hospital, is regularly used as an urgent care clinic by children with other primary care providers. Incorrect IIS provider assignments to health center D became a more common occurrence when we began immunizing children at sick visits (regardless of whether they identified other primary care providers) as part of our QI program.
There were 63 524 total patient visits overall in 2006 vs 71 935 in 2008, an increase of 12% (P < .0001). In 2006, 49 782 (78%) patients were publicly insured compared with 60 921 (85%) in 2008 (P < .0001).
In 2006, IIS records were not preprinted for any patient encounters. In 2008, an IIS record was preprinted for practitioners’ reference for virtually every scheduled patient encounter (41 685 records) at all centers, and ∼25% (7290 records) of all unscheduled “sick visit” encounters (records were not preprinted for all unscheduled encounters because of limited QI staff).
Before the study, in 2006, we did not send any reminder/recall phone messages or mailings. In 2008, we sent out 27 651 autodialer messages for due immunizations, and made 10 253 live phone calls and sent 523 postcards for overdue immunizations.
In 2006, personnel at WIC centers colocated with our health centers did not check their clients’ vaccine compliance, but did ask parents to bring immunization records to every visit that they then forwarded without review to the IIS. In 2008, during a 3-week trial, we checked the IIS record of 370 WIC clients. In this group, 21 children were overdue for immunizations. Of these, 1 child received same-day immunizations, 3 parents refused, and 17 did not present to their WIC appointments. Because of low yield, we did not continue this intervention.
In 2006, the health centers were the primary care providers for 1999 children ages 19 to 35 months compared with 1946 children in 2008 (ie, MOGEs were excluded both years). Immunization coverage for the 4:3:1:3:3:1:3 series improved from 71% to 87% (P < .0001) (Table 3). Coverage increased significantly at all six health centers. In addition, timely vaccination rates improved from 65% to 79% (P < .0001) and increased significantly at 4 of the 6 centers, including the largest and smallest (Table 4).
We also performed data analysis with all patients for whom our centers were the immunizer of record in the IIS (including all MOGEs as not up to date, data not shown). This larger data set included 2232 children aged 19 to 35 months in 2006 and 2250 children in 2008. Immunization coverage overall for the 4:3:1:3:3:1:3 series improved from 64% to 75% (P < .0001). Immunization rates increased significantly at 5 of 6 health centers (P < .05) with the increase at center E reaching borderline significance (P = .07). In addition, timely immunization rates improved overall from 58% to 69% (P < .0001). Rates of timely immunization increased significantly at health centers B, C, D, and G (P < .05), and was borderline significant at center E (P = .07).
To determine whether gains in immunization coverage were sustained, we assessed compliance rates in 2009. In 2009, the health centers were the primary care providers for 2327 children aged 19 to 35 months (excluding MOGEs). Immunization coverage overall was 88%. This 1% improvement over 2008 coverage levels was borderline significant (P = .08) (Table 3). None of the individual health centers experienced a significant decrease in immunization rates, and, in fact, center B improved by 7% (P = .02). Reexamining the data with all patients for whom our centers were the immunizers on IIS record, in 2009, the health centers treated 2766 children aged 19 to 35 months. Overall, with this larger data set, we could not detect a statistically significant change in immunization coverage from 2008 to 2009 (75% vs 74%, respectively, P = .44).
Our health centers implemented a comprehensive immunization best practices program based on an established theoretical framework. The cumulative effect was a 16% increase in immunization rates overall and a 14% increase in timely immunization. This occurred despite immunization rates in DC—whose overall childhood population is similar to our patient population, with the exception that ours includes more racial minorities and is more economically disadvantaged—remaining statistically unchanged (78% in 2006,20 77.6% in 2008,21 and 75.0% in 20091) over the same period.
All 6 health centers improved, including one with 3 practitioners and another with >25 practitioners. Gains were sustained beyond 18 months and were achieved by using the CoCASA software, which excluded any immunization administered before ACIP-recommended minimum ages/intervals. This is a more rigid method than that used in the annual CDC survey, which accepts all doses administered.22,23 We achieved and sustained an 87% coverage rate despite national shortages of PCV7 for 8 months in 200418 and H influenzae type b vaccine for 18 months from 2007 to 2009.24
We implemented 6 Task Force recommendations and supplemented these recommendations with an additional activity: collaboration with community stakeholders. According to the CCM framework, interrelationships among health care institutions and the community are important for improving patient outcomes. However, because provider-community linkages require the cooperation of nonrelated groups, it is the least implemented of all CCM elements.25 Collaborating with the DC DOH enhanced our feedback for practitioners and allowed us to create practitioner reminders. Collaborating with the Medicaid plans improved our outreach abilities.
During program implementation, all our health centers used paper records. Much time was spent reviewing charts to determine patients’ immunization status and printing IIS records for our large patient panel. Since the end of the study, we have transitioned to an electronic medical record system that has facilitated determination of a child’s immunizations status, because the information is all in 1 location. Ideally, an electronic record system would also facilitate searching for unimmunized children and sending automated phone messages to reduce the work burden of implementing client reminders, thereby rendering activities sustainable with less manpower.
We found that most WIC clients at our trial location were already immunization compliant. However, our findings may not be generalizable, because the WIC staff at the trial location, while not specifically checking a child for immunization compliance, requested that parents bring immunization records to every visit to forward to the IIS. Requesting immunization records at every appointment goes beyond what is nationally mandated, which is only to count four diphtheria-tetanus-acellular pertussis vaccinations by 24 months old and only at certification visits.26 However, having WIC personnel request immunization records at every encounter may increase IIS completeness and remind parents of the importance of immunization.
Our improvement rates are consistent with those of similar projects. For instance, Melinkovich et al27 increased immunization compliance 47% among young children over an 11-year span in Denver by using provider reminders and feedback, staff education, standing orders, and patient reminders. Some of the gains may have been due to secular trends, because immunization rates improved 30% in Colorado over the same time frame.20,28 A recent Cochrane review found that patient reminder systems with and without other interventions increased immunization rates from 1% to 20% in randomized controlled trials.29 These studies included various combinations of patient reminders, home visits, door-to-door campaigns, patient tracking, outreach, provider prompts, and client-held records.29
Our study has limitations. We did not include a standard care comparison group. However, immunization rates in DC overall were similar to our baseline rates and remained constant during our study years.1,20,21 Some may consider that we tested the effectiveness of a comprehensive QI program to be a limitation, because we cannot gauge the relative impact of each intervention. However, the efficacy of each intervention has already been well established and summarized by the Task Force.10,11 In our estimation, and in concordance with the CCM, it is a strength of our QI program that interventions were implemented jointly, because they affect different aspects of the patient care spectrum and likely acted synergistically.
We found that our pediatric health centers were able to implement and sustain immunization best practices interventions over 18 months. By instituting these interventions as part of a comprehensive, theory-driven, and evidence-based approach to quality improvement, we increased immunization rates >10% with a predominantly minority and publicly insured patient population.
This work was supported in part by District of Columbia Department of Health grant PO310557. The Department of Health was involved in study design, conduct, data collection, review, and approval of the manuscript. L.Y.F. had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
- Accepted October 10, 2011.
- Address correspondence to Linda Y. Fu, MD, MS, Goldberg Center for General and Community Pediatrics, Children’s National Medical Center, 111 Michigan Ave, NW, Washington, DC 20010. E-mail:
FINANCIAL DISCLOSURE: Drs Fu and Cora-Bramble have served on an advisory board to Pfizer on reducing barriers to immunization among special populations. Dr Cora-Bramble has also been a reviewer of Pfizer grants with this same purpose. In addition, for 3 months, our health centers participated in a program sponsored by Sanofi-Pasteur to receive a reduced rate for placing autodialer calls.
- ↵Task Force on Community Preventive Services. Vaccinations for preventable diseases: universally recommended vaccines. The Community Guide 2002. Available at: www.thecommunityguide.org/vaccines/universally/index.html. Accessed July 7, 2010
- Coleman K,
- Austin BT,
- Brach C,
- Wagner EH
- ↵Centers for Disease Control and Prevention. Percentage of U.S. children >4 months and <6 years with 2+ immunizations in immunization information systems (IIS), 2008. Available at: www.cdc.gov/vaccines/programs/iis/rates/2008-child-map.htm. Accessed August 16, 2011
- Centers for Disease Control and Prevention
- Smith PJ,
- Rao JN,
- Battaglia MP,
- Ezzati-Rice TM,
- Daniels D,
- Khare M,
- Centers for Disease Control and Prevention/National Center for Health Statistics
- ↵Wu S, Pearson M, Schaefer J. Assessing the implementation of the chronic care model in quality improvement collaboratives: does baseline system support for chronic care matter? In: Luczak H, Zink KJ, eds. Human Factors in Organizational Design and Management – VII. 2003. IEA Press, Santa Monica, CA 2003: 595-601
- ↵National WIC-Immunization Workgroup. Implementing the new WIC policy for immunization screening and referral: ideas and suggestions for establishing procedures and conducting training in your state agency. 2002. Available at: www.nal.usda.gov/wicworks/Sharing_Center/Immunizations/Implementation_Guide.pdf. Accessed March 4, 2009
- Copyright © 2012 by the American Academy of Pediatrics