OBJECTIVE: We implemented a quality improvement initiative aimed at reaching a 95% immunization rate for patients aged 24 months. The setting was a hospital-based pediatric primary care practice in Boston, Massachusetts. We defined immunization as full receipt of the vaccine series as recommended by the Centers for Disease Control and Prevention.
METHODS: The initiative was team-based and structured around 3 core interventions: systematic identification and capture of target patients, use of a patient-tracking registry, and patient outreach and care coordination. We measured monthly overall and modified immunization rates for patients aged 24 months. The modified rate excluded vaccine refusals and practice transfers. We plotted monthly overall and modified immunization rates on statistical process control charts to monitor progress and evaluate impact.
RESULTS: We measured immunization rates for 3298 patients aged 24 months between January 2009 and December 2012. Patients were 48% (n = 1576) female, 77.3% (n = 2548) were African American or Hispanic, and 70.2% (n = 2015) were publicly insured. Using control charts, we established mean overall and modified immunization rates of 90% and 93%, respectively. After implementation, we observed an increase in the mean modified immunization rate to 95%.
CONCLUSIONS: A quality improvement initiative enabled our pediatric practice to increase its modified immunization rate to 95% for children aged 24 months. We attribute the improvement to the incorporation of medical home elements including a multidisciplinary team, patient registry, and care coordination.
- quality improvement
- primary care
- organizational innovation
- process assessment (health care)
- system analysis
- CL —
- center line
- EMR —
- electronic medical record
- FTE —
- full-time equivalent
- MIIS —
- Massachusetts Immunization Information System
- QI —
- quality improvement
- QIA —
- quality improvement assistant
Immunizations are among the most important preventive health services for children, and immunization rates reflect the quality of pediatric ambulatory health care.1–3 However, nearly 1 in 4 US young children are incompletely immunized.4 Sporadic outbreaks of vaccine-preventable disease continue to occur, especially in underimmunized populations.5–7 Increasing immunization rates therefore remains a perennial target for quality improvement (QI) in pediatrics.8–12
Our practice began immunization improvement work in 1996 with the development of a home-grown immunization database. This database preceded the Massachusetts Immunization Information System (MIIS), launched in 2011 by the state Department of Public Health.13 In the early 2000s, we began using bar code scanners for vaccine entry into the electronic medical record (EMR). Technological advancements were supplemented with regular education sessions on vaccine schedules and documentation. We trained nurses to identify missing vaccines and to notify ordering providers as part of their clinical workflow. We began providing standardized physical examination forms for well child visits that included the patient’s vaccination record. Based on evidence, we conducted practice surveillance for unvaccinated children and implemented patient immunization reminder systems.14–16 Practice-level immunization rates reached 85% by 2006, exceeding both the national rate (77%) and the Massachusetts state rate (83%).17
Our institution selected immunization rates as a dashboard quality measure in 2007, compelling additional practice interventions aimed at increasing rates. We increased availability of nurse-shot visits, provided individual feedback to clinicians, and implemented all-staff education sessions. These interventions failed to raise immunization rates any more. To our knowledge, there were no previous studies demonstrating increased immunization rates in practice settings with already high rates. Rodewald et al18 showed that patient tracking and outreach were effective interventions, and work by Gannon et al9 supported a team-based approach. Fu et al8 also showed the effectiveness of provider reminder systems as part of a multicomponent intervention. We therefore designed a new initiative incorporating these elements with the aim of raising practice-level immunization rates to 95%. We focused on children aged 24 months because of their vulnerability to vaccine-preventable disease and their potential to serve as vectors for disease transmission.19–21
We defined immunization as full receipt of the 4:3:1:4:3:1:4 combined vaccine series by age 24 months. The series is recommended by the Centers for Disease Control and Prevention and includes 4 diphtheria-tetanus-acellular pertussis vaccines, 3 inactivated poliovirus vaccines, 1 measles-mumps-rubella vaccine, 4 Haemophilus influenzae type B vaccines (3 in times of vaccine shortage), 3 hepatitis B vaccines, 1 varicella-zoster vaccine, and 4 pneumococcal conjugate vaccines.22 We excluded rotavirus because assessment of immunization status began at an age beyond which patients were eligible for rotavirus vaccination.
The initiative began in September 2009 at a hospital-based pediatric primary care practice in Boston, Massachusetts. The practice serves a patient population of 14 000, with more than 40 000 clinical encounters annually. Patients reside primarily in the urban neighborhoods of greater Boston, and 65% are Medicaid insured. The pediatric provider staff includes 65 resident physicians, 25 attending physicians, and 4 nurse practitioners. The nursing to provider ratio is 0.75 nurse full-time equivalent (FTE) to each provider FTE. Nurses routinely screen for immunization needs and administer vaccines.
Multidisciplinary Improvement Team
The improvement team included 1 physician, 1 departmental QI director, 1 nurse manager, 1 staff registered nurse, 1 social worker, and 1 quality improvement assistant (QIA). The QIA was an existing administrative staff member identified to take a leadership role in QI activities. The QIA specifically maintained the patient-tracking registry and performed patient outreach and care coordination activities. Though not part of the core improvement team, residents were a primary referral source for the QIA regarding the tracking registry. We regularly provided educational briefings and opportunities for residents to connect with the QIA during clinic sessions.
Planning the Initiative
Our team conducted monthly practice-wide evaluations of immunization rates for children age 24 months. We created statistical process control charts using monthly rate data, which provided a visual representation of progress over time. We then modified the monthly rates to exclude vaccine refusals and practice transfers. The QI director designed flowcharts to clarify the process of immunization assessment and delivery across the practice. Using the flowcharts, we generated 3 core interventions: systematic identification and capture of target patients, use of a patient-tracking registry, and patient outreach and care coordination. Figure 1 outlines the key drivers, and Fig 2 illustrates specific timeline components of the initiative.
Implementing the Initiative
The initiative consisted of 3 primary interventions targeting underimmunized patients aged 15 to 23 months. We targeted the 15- to 23-month age range as a window of opportunity for immunization by 24 months.
Intervention 1: Systematic Identification and Capture of Target Patients
We used our homegrown immunization database to generate monthly reports for all patients across the practice in the target age range. We manually reviewed the reports and added any patients identified as underimmunized to the tracking registry. To capture patients at the point of care, we programmed the scheduling software (EPIC, Verona, WI) to highlight 15- to 23-month-olds regardless of visit type. The highlighted field activated front desk staff to place a bold “Check Immunizations” stamp on the visit billing sheet. The stamp served as a visual reminder for nurses and providers to assess immunization status and administer necessary vaccines. Any target patients not receiving vaccines at the point of care could be referred to the QIA for entry into the tracking registry. We expanded this process to include urgent care and walk-in visits (Fig 1).
Intervention 2: Use of a Patient-Tracking Registry
We used Excel software (Microsoft Corporation, Redmond, WA) to build a tracking registry for underimmunized patients 15 to 23 months of age. We designed the registry to facilitate care coordination aimed at bringing underimmunized patients up to date. The registry’s initial fields included date of patient identification, action plan for care coordination, and date of action follow-through. We refined the registry to include action plan fields for difficult-to-reach patients and non–English-speaking families. We also honed the action plan fields to incorporate follow-up attempts, their outcomes, and anticipated date of next attempt. The QIA managed the registry and performed care coordination activities. We also trained the QIA on appointment scheduling to facilitate care coordination during patient outreach (Fig 2). Social work and interpreter services were available for consultation as needed. The improvement team met weekly to review progress and troubleshoot challenging cases.
Intervention 3: Patient Outreach and Care Coordination
In general, the QIA attempted outreach 3 times to coordinate care for immunization follow-up. The QIA mailed letters for patients not reachable by phone. Each outreach attempt was entered into the patient-tracking registry along with reminder dates for the next attempt. In successful attempts, the QIA scheduled follow-up appointments for immunizations while on the phone. These appointments often coincided with due or overdue well child care and were scheduled accordingly. The QIA notified pediatric providers via e-mail about the upcoming appointments and offered suggestions about which vaccines to order (Fig 2). The QIA also flagged pending appointments in the scheduling program to include the suggested vaccines (Fig 2). After the appointment, the QIA reviewed the medical record to confirm receipt of vaccines. Patients not receiving the necessary vaccines remained in the tracking registry for subsequent care coordination. The QIA followed up on missing outside immunization records and added patients still designated as underimmunized to the registry. No patient exited the registry until he or she was confirmed as up to date or transferred out of the practice.
QIA Training and Use
We used the QIA at 0.5 FTE, or ∼20 hours per week during early implementation. Although the QIA was a single person, we trained multiple administrative staff on the QIA roles, including registry maintenance and patient outreach. The QIA roles were administrative by nature and therefore easily adopted. Training multiple staff distributed the FTE among existing personnel and created a failsafe in the event of staffing changes, vacation, or sick time. As the initiative progressed, the number of underimmunized patients needing tracking and care coordination decreased. By 6 months we were able to reduce QIA effort to 4 to 5 hours per week, which we have sustained to the present day.
To evaluate the initiative, we measured monthly immunization rates for patients aged 24 months across the practice. We considered patients immunized only if they received the full 4:3:1:4:3:1:4 vaccine series by their second birthday. We measured both overall immunization rates and modified rates, which excluded vaccine refusals and practice transfers. We plotted monthly overall and modified immunization rates on statistical process control charts. We established a mean immunization rate, illustrated as the center line (CL) on each control chart. We also established upper control limits and lower control limits on each control chart. These limits constrain common cause variation, or noise, within the system. We monitored the monthly data for special cause variation, illustrated on the control charts by 6 or more consecutive points above the CL. Special cause variation signals an effect and indicates a system change.
We used χ2 tests to evaluate differences in immunization rates before and after implementation of the initiative. We conducted subanalyses to examine differences in immunization rates based on pediatric provider type, demographic characteristics, and insurance coverage. We used descriptive statistics to characterize actions taken on special cases to bring patients up to date. The initiative was categorized as QI and exempted from review board approval at our institution.
We measured practice-level immunization rates for 3298 pediatric patients aged 24 months between January 2009 and December 2012. Patients were predominantly African American or Latino and publicly insured, as summarized in Table 1.
Figures 2 and 3 show overall and modified immunization rates, respectively, for patients aged 24 months from January 2009 through December 2012. The mean overall immunization rate, as illustrated in Fig 2, was 90%. This rate remained stable through the duration of the study. The mean modified rate was 93% and excludes vaccine refusals and practice transfers. The initiative was implemented September 2009. As illustrated in Fig 3, we observed 6 consecutive points above the CL, indicating special cause variation between September 2010 and February 2011. This finding signals an effect of the initiative on immunization rates and indicates a system change. We therefore revised the mean modified immunization rate to 95% to reflect the change, as illustrated in Fig 3. The lag between implementation and system change reflects time between receipt of vaccines and assessment of immunization status. We found no significant differences in immunization rates before and after implementation using traditional statistics. Subanalyses revealed no significant differences in rates between resident and attending physicians, demographic characteristics, or insurance coverage type.
We identified 328 patients as underimmunized between September 2009 and September 2010. We fully immunized 194 (59%) of these patients by September 2010. Twenty patients (6%) remained underimmunized despite care coordination and outreach attempts. We failed to recover missing outside immunization records on 15 patients (5%). The remaining 99 patients (30%) refused vaccines, transferred care, or were unreachable by phone or mail. For the 194 patients we fully immunized, we made 504 (mean 2.6) total outreach attempts for care coordination. We immunized 176 (91%) of these patients by age 24 months. For the 20 patients who remained underimmunized, we made 113 (mean 5.7) total outreach attempts for care coordination. We continued attempting outreach to immunize these patients even after their second birthday.
A QI initiative enabled our pediatric practice to increase its modified immunization rate for children aged 24 months to 95%. The control charts illustrate that the process was sustainable over time. Traditional statistics revealed no significant differences in immunization rates before and after the initiative, probably because of the high preexisting immunization rates across the practice. Subanalysis revealed that outreach and care coordination activities successfully brought most underimmunized children up to date. The few patients who remained underimmunized received a disproportionate share of outreach attempts and care coordination efforts. Our findings corroborate previous research demonstrating higher immunization rates through tracking- and outreach-based interventions in pediatric primary care.14,18,23 Our study also supports the effectiveness of multicomponent interventions for improving immunization coverage for socioeconomically disadvantaged populations.8,24
The strength of this initiative lies in its alignment with the patient-centered medical home model of primary care. It combined elements of traditional QI, including the automated system for identifying target patients, with population management elements of registry use and care coordination.25,26 Also consistent with the patient-centered medical home model was our multidisciplinary team-based structure that included social work.27,28 Evidence suggests that multidisciplinary teams are a critical element of primary care transformation and improvement.29–31 Patients with high no-show rates faced the greatest risk of fragmented care and underimmunization. Issues of family disorganization or resource barriers were common in this group. The initiative enabled identification of vulnerable families and created opportunities for social work to connect and address unmet needs. Socially disadvantaged patients often need more time than a standard 15-minute primary care visit permits.32 Care coordination and previsit planning for immunizations may have enabled primary care providers to spend time addressing complex or unresolved problems in greater depth.33,34 Taken together, the initiative potentially benefited children and families beyond immunizations alone.
This study had several limitations. First, we examined only immunization rates and did not measure cost-effectiveness. The QIA role was held by existing administrative staff and did not necessitate hiring additional personnel. In our academic pediatric practice, the administrative staff routinely engages in multidisciplinary projects. The QIA role therefore may not be generalizable to smaller practices with fewer support staff. However, once we launched and then streamlined the initiative, we needed progressively fewer resources to sustain QIA activities. As immunization rates rose, the number of patients needing tracking and outreach decreased.
We also focused only on immunization rates and not other aspects of service use. In urgent care, patients with low-urgency complaints were often discharged before an immunization assessment could be made. Higher-complexity complaints often meant patients were too ill to receive vaccines, or parents refused immunizations because their child was ill. Other investigators have reported similar findings.35,36 We concentrated on adding target patients identified in urgent care to the tracking registry for subsequent care coordination. Evidence suggests that patients may not return for timely well child care after a sick visit.37 It was our experience that patients did return for their follow-up immunization appointments, as reflected in our high immunization rates. However, we did not collect data on whether those follow-up appointments were for well child care or immunizations only.
Our initiative preceded the launch of MIIS. We are collaborating with the state Department of Public Health to develop a clinical information system to identify underimmunized children using the MIIS. In our experience, recovering missing outside immunization records inefficiently uses practice resources. A statewide platform enabling shared access to immunization records at the point of care could improve efficiency and decrease unnecessary resource use. Shared access might also facilitate widespread implementation of clinical decision support tools and population-based approaches for immunization assessment and delivery.
We excluded EMR-based decision support as a part of the initiative. Decision support tools for immunizations have shown promise for increasing coverage rates.38 However, we have historically avoided EMR-based decision support because of inaccuracies in assessing immunization status at the point of care. We suspect inaccuracies may be secondary to changing immunization recommendations over time. Inaccurate decision support presents a potential safety issue in practices such as ours with large numbers of trainees. However, continuously improving clinical information systems, including decision support, will probably become critical to delivering efficient, high-quality primary care.
We are applying the patient-tracking registry and outreach components from the initiative to improve our adolescent immunization rates. Research indicates that methods of outreach other than phone, such as text messaging, may be crucial for interventions targeting adolescents.39,40 Additional research is needed to clarify whether the initiative can be replicated across different patient and practice types.
The authors thank the patients and staff of Boston Children’s Hospital Primary Care Center who supported this work and Dr Jonathan Finkelstein for his thoughtful review.
- Accepted January 3, 2014.
- Address correspondence to Ronald C. Samuels, MD, MPH, Division of General Pediatrics, Department of Medicine, Boston Children’s Hospital, 300 Longwood Ave, Boston, MA 02115. E-mail:
Dr Bottino drafted the initial manuscript and reviewed and revised the manuscript; Dr Cox co-conceptualized and designed the study, reviewed and revised the manuscript, and provided critical feedback; Dr Kahlon developed the quality improvement process maps and strategies, conducted statistical process control analyses, and reviewed and revised the manuscript; Dr Samuels co-conceptualized and designed the study, coordinated and supervised data collection, and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: The Department of Medicine at Boston Children’s Hospital provided internal funding support.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Rodewald L,
- Maes E,
- Stevenson J,
- Lyons B,
- Stokley S,
- Szilagyi P
- ↵Massachusetts Immunization Information System (MIIS). Available at: www.mass.gov/eohhs/gov/departments/dph/programs/id/immunization/miis/. Accessed November 20, 2013
- Rodewald LE,
- Szilagyi PG,
- Humiston SG,
- Barth R,
- Kraus R,
- Raubertas RF
- Williams N,
- Woodward H,
- Majeed A,
- Saxena S.
- Hambidge SJ,
- Phibbs SL,
- Chandramouli V,
- Fairclough D,
- Steiner JF
- Coleman K,
- Austin BT,
- Brach C,
- Wagner EH
- Engels Y,
- van den Hombergh P,
- Mokkink H,
- van den Hoogen H,
- van den Bosch W,
- Grol R.
- McAllister JW,
- Cooley WC,
- Van Cleave J,
- Boudreau AA,
- Kuhlthau K
- Bodenheimer T,
- Laing BY
- Robison SG
- Brigham KS,
- Woods ER,
- Steltz SK,
- Sandora TJ,
- Blood EA
- Copyright © 2014 by the American Academy of Pediatrics