Strategies for Improving Vaccine Delivery: A Cluster-Randomized Trial
Abstract
OBJECTIVE: New emphasis on and requirements for demonstrating health care quality have increased the need for evidence-based methods to disseminate practice guidelines. With regard to impact on pediatric immunization coverage, we aimed to compare a financial incentive program (pay-for-performance [P4P]) and a virtual quality improvement technical support (QITS) learning collaborative.
METHODS: This single-blinded (to outcomes assessor), cluster-randomized trial was conducted among unaffiliated pediatric practices across the United States from June 2013 to June 2014. Practices received either the P4P or QITS intervention. All practices received a Vaccinator Toolkit. P4P practices participated in a tiered financial incentives program for immunization coverage improvement. QITS practices participated in a virtual learning collaborative. Primary outcome was percentage of all needed vaccines received (PANVR). We also assessed immunization up-to-date (UTD) status.
RESULTS: Data were analyzed from 3,147 patient records from 32 practices. Practices in the study arms reported similar QI activities (∼6 to 7 activities). We found no difference in PANVR between P4P and QITS (mean ± SE, 90.7% ± 1.1% vs 86.1% ± 1.3%, P = 0.46). Likewise, there was no difference in odds of being UTD between study arms (adjusted odds ratio 1.02, 95% confidence interval 0.68 to 1.52, P = .93). In within-group analysis, patients in both arms experienced nonsignificant increases in PANVR. Similarly, the change in adjusted odds of UTD over time was modest and nonsignificant for P4P but reached significance in the QITS arm (adjusted odds ratio 1.28, 95% confidence interval 1.02 to 1.60, P = .03).
CONCLUSIONS: Participation in either a financial incentives program or a virtual learning collaborative led to self-reported improvements in immunization practices but minimal change in objectively measured immunization coverage.
- AAP —
- American Academy of Pediatrics
- ACIP —
- Advisory Committee on Immunization Practices
- aOR —
- adjusted odds raio
- CI —
- confidence interval
- CIzQIDS —
- Comparison of Immunization Quality Improvement Dissemination Strategies
- ICC —
- intracluster correlation
- P4P —
- pay for performance
- PANVR —
- percentage of all needed vaccines received
- QI —
- quality improvement
- QITS —
- quality improvement technical support
- UTD —
- up-to-date
What’s Known on This Subject:
There is a paucity of evidence about the effectiveness of strategies to disseminate immunization delivery practice guidelines. No studies to date have compared 2 popular dissemination strategies, virtual learning collaboratives and financial incentives, for their impact on immunization coverage.
What This Study Adds:
Standard approaches to implementing both a remote financial incentives program and a virtual learning collaborative may result in improved immunization delivery practices but may not have a high impact on immunization coverage.
New requirements for demonstrating health care quality have increased the need for evidence-based methods to disseminate practice guidelines. Since 2000, the American Board of Pediatrics has included participation in quality improvement (QI) projects as a requisite for maintenance of certification.1 However, methods used in QI programs can vary in adherence to accepted QI principles,2 and evidence for their impact on patient outcomes is sparse and conflicting.3–5 Furthermore, web-based QI learning programs are increasingly common,6 although evidence of impact of virtual or remote learning activities on patient outcomes is particularly weak.7–9
Another approach to improving practitioner adherence to evidence-based guidelines is via financial incentives. Accountable care organizations with provider reimbursement partially determined by quality measurements are currently operating in all 50 states, with 23.5 million covered lives.10 Despite the growth of incentives-based health care models,11 a 2013 Medicare report showed wide variability among accountable care organization quality scores.12,13 Furthermore, prospective trials supporting the effectiveness of pay for performance (P4P) programs for improving patient outcomes are few and methodologically limited.14,15
The Advisory Committee on Immunization Practices (ACIP) has provided immunization guidelines since 1964.16 However, only a few randomized trials have tested the effect of dissemination methods on provider adherence to immunization guidelines. Five trials of financial incentives17–19 and 1 trial of remote provider education20 yielded minimal to no improvement in immunization coverage. To date, we have found no published comparative effectiveness study of financial incentives and remote provider QI education. Furthermore, recent changes in the health care environment (mandated insurance coverage of immunizations,21 electronic medical records,22 immunization registries,23 electronic clinical support tools,22 and changing parental attitudes about immunization24) suggest the need for evidence about dissemination strategies in the contemporary practice environment. The Comparison of Immunization Quality Improvement Dissemination Strategies (CIzQIDS) study was a cluster-randomized, comparative effectiveness trial conducted among geographically dispersed practices comparing a P4P program and a virtual QI learning collaborative for impact on pediatric immunization coverage.
Methods
This study was reviewed and approved by the Children’s National Medical Center and American Academy of Pediatrics (AAP) institutional review boards. The CIzQIDS trial was undertaken and reported in accordance with Consolidated Standards of Reporting Trials (CONSORT) guidelines25 and is registered at clinicaltrials.gov (NCT02432430).
Study Design and Participants
This study was designed as a single-blinded (to outcomes assessor), cluster-randomized, parallel trial. Recruitment occurred from January to June 2013 before the start of the intervention. To gain a diverse participant sample, study advertisements were placed in 4 pediatric journals and emailed to members of 2 pediatric professional associations and 53 primary care, physician, and immunization networks. Advertisements directed individuals to an online eligibility survey.26 Of 300 complete and partial surveys, 42 unique respondents met eligibility criteria (English-speaking primary care pediatricians in practices immunizing at least five 3- to 18-month-olds per week, with daytime internet access, estimated practice immunization coverage of <86% for 3- to 18-month-olds, and the ability to form an improvement team consisting of a provider, clinical support staff, and administrator) and completed informed consent. Of the 42 practices, 7 were excluded after study staff’s random review of 50 medical records per site revealed coverage ≥86%. (The baseline coverage criterion excluded participants with minimal room for improvement.) The remaining 35 practices were block-randomized (via the REDCap data system, which conceals allocation sequence to investigators26) by 2 factors: percent of patients fully up-to-date for age (<75% vs ≥75%) and practice type (federally qualified health center, academic, or other) into 2 interventional study arms: quality improvement technical support (QITS) and P4P.
Outcome Measures
Our primary outcome was percentage of all needed vaccines received (PANVR) during the 12 months preceding assessment. To calculate PANVR, we divided the number of valid doses received by the number of doses that should have been received (ie, denominator included all doses whose maximum recommended age or interval boundary occurred within 12 months preceding the assessment). We also assessed a binary variable representing immunization up-to-date (UTD) status. Patients were considered UTD if they had received all recommended doses for age.
Blinded study staff extracted immunization, encounter, and demographic data from 50 medical records per practice at baseline (March 25, 2013) and postintervention (June 25, 2014) by either site visit or remote electronic record access. Each practice’s patients were stratified into 4 age categories whose boundaries coincided with the minimum and maximum recommended ages for routine receipt of vaccinations (Table 1).27 Patients in each stratum were randomly sorted for sample selection, and larger age strata were intentionally oversampled (3 to 4 months, n = 10; 5 to 6 months, n = 10; 7 to 15 months; n = 16; 16 to 18 months, n = 14). A medical record was excluded if the child had moved or gone elsewhere, had no contact with the practice in ≥12 months, or had <2 encounters. Excluded records were replaced by the next patient record in the corresponding age stratum. At practices in which the total eligible patient population aged 3 to 18 months was <50, all eligible records of patients 3 to 18 months were included. Data were extracted using the Comprehensive Clinical Assessment Software Application for immunizations.19
Vaccine Doses Needed by Age According to Study Definition
Patient vaccination status was determined according to the 2011 ACIP routine and catch-up schedules28 for vaccines included in the Healthy People 2020 goals29: hepatitis B, diphtheria, tetanus toxoids, pertussis, Haemophilus influenzae type b, pneumococcal conjugate, inactivated poliovirus, measles, mumps, rubella, and varicella (Table 1). As per ACIP, doses were considered valid if given within 4 days of the minimum recommended ages or intervals.30 Patients with a documented valid vaccination contraindication were excluded from all analyses. Those who had received any vaccinations elsewhere were excluded from PANVR analyses.
To assess practice demographic information and track changes in immunization practices and attitudes, participants completed online surveys at baseline and monthly throughout the intervention and postintervention.
Interventions
The intervention period spanned June 25, 2013, to June 24, 2014. At baseline, all participants received a Vaccinator Toolkit to support implementation of immunization best practices,31 including examples and worksheets to aid in application of the model for improvement through plan-do-study-act cycles to implement changes based on local needs.32 Practices in the P4P arm additionally participated in a financial incentive program designed to reward improvement in immunization coverage from baseline to postintervention both within individual practices and over the entire study arm. Each P4P practice was eligible to receive $500 if the percentage of patients UTD in their individual practice increased 5 to <10 percentage points and $1000 if it increased ≥10 percentage points. They were eligible to receive an additional $1000 if >90% of practices in the P4P arm achieved immunization coverage of >85%. No restrictions were placed on how practices allocated incentives.
Practices in the QITS study arm participated in a year-long virtual learning collaborative consisting of 6 expert-led, Web-based learning sessions (covering QI science, the immunization schedule, systemic vaccine delivery barriers and solutions, parental vaccine hesitancy, and sustaining improvement), monthly coaching conference calls, and a continuously updated compendium of Web-based resources.33 Each month, QITS practices extracted data (vaccination status, vaccination missed opportunities, and use of patient vaccination reminders) from 10 to 20 patient records. The QI coach used this information to create individual practice and aggregate run charts for performance feedback. QITS participants could earn American Board of Pediatrics Maintenance of Certification Part 4 (QI) credit by meeting participation requirements. Details about the Vaccinator Toolkit and QITS intervention are located on the AAP Quality Improvement Innovation Networks Web site.33
Statistical Analysis
We developed sample size estimates for difference in postintervention PANVR between arms using contemporaneous national early-childhood immunization rates (mean ± SD, 70% ± 5.4%)34 to make assumptions about the expected effect size. We did not find previous reports of intracluster correlation (ICC) by practice to include in the power calculation and conservatively assumed an ICC of 0.20. Setting the 2-tailed type 1 error at 5%, we determined that we would have >90% power to detect a difference equivalent to 0.5 SD units (ie, 2.7 percentage points) between arms if we enrolled 20 practices per arm and extracted data for 50 patients per practice. In actuality, the observed SD for the proportion of expected doses received was larger than anticipated, the ICC in our analytic sample was only 0.07, and we enrolled 16 practices per study arm. Under actual trial conditions, the observed effect size was 0.2 SD units.
Analyses were performed in Stata v13.1.35 Descriptive statistics were used to summarize characteristics of the study population. To account for minor variations in the age distribution of patients sampled from each practice, all practice-level outcome estimates were age-standardized according to the 2011 age distribution of American children.36 All patient-level multivariable analyses accounted for patient clustering within practices by including practice as a random intercept. To model PANVR, a proportional response variable, we used generalized estimating equations (xtgee) with a logit link and the binomial family. To model UTD, we used random-effects logistic regression (xtlogit) and reported adjusted odds ratios (aORs) and 95% confidence intervals (CIs). All patient-level models were adjusted for patient age stratum and insurance status (Medicaid or other). To account for imbalances across groups despite randomization, each of the 2 models of patient-level postintervention outcomes (PANVR and UTD) were also adjusted for baseline practice average PANVR or percentage UTD, respectively.
Results
Thirty-five practices were randomized to P4P (n = 17) or QITS (n = 18) (Fig. 1). One practice in the QITS arm dropped out before the intervention because of unexpected staffing turnover. Data from that practice were excluded from all analyses. During the intervention, 1 practice randomized to QITS dropped out because meeting times were inconvenient, and 1 practice in P4P dropped out because the site project leader left the practice. Data from those 2 practices were included in intent-to-treat but not per-protocol analyses. In this article, we exclusively report per-protocol results, since both dropouts occurred within the first quarter of the intervention period; intent-to-treat findings were virtually identical. Per-protocol analyses consisted of data from 3147 patient records (1576 assessed at baseline and 1571 postintervention) from 32 practices (16 P4P and 16 QITS).
Consolidated Standards of Reporting Trials (CONSORT) flow diagram.
Baseline Characteristics
Practices in the treatment arms were similar with respect to measured baseline characteristics with few exceptions (Table 2). Although randomization balanced arms in terms of the number of practices with <75% of patients UTD, the baseline age-standardized proportion of patients UTD in P4P practices was borderline significantly higher than in QITS practices (mean ± SE, 73.9% ± 7.6% vs. 64.2% ± 19.8%, P = .05). Also, patients sampled from P4P practices at baseline had statistically, but not clinically significantly, fewer health care encounters per month than patients in QITS practices (1.07 ± 0.02 vs 1.14 ± 0.02, P = .04) (Table 3).
Baseline Practice-Level Characteristics
Baseline and Postintervention Patient-Level Characteristics
Quality Improvement Activities
Practices in the 2 study arms reported similar QI activities, with an average of 6 to 7 new immunization improvement strategies initiated during the intervention period (Table 4). The most common changes made were more consistently checking immunization status at point of service and more consistently administering immunizations at sick encounters. Half or more of the practices in both study arms reported distributing new immunization educational materials to patients, eliminating unwarranted per-visit injection limits, increasing use of immunization registries, and having more staff training. Compared with practices in the P4P arm, practices in the QITS arm completed more plan-do-study-act cycles (median 5 [interquartile range 3 to 6.5] vs 2 [1.5 to 3.5]; P = .01), and a greater number of QITS practices implemented new systems to prompt providers about immunizations at point of service (56.3% vs 6.3%, P < .01).
Self-Reported Activities During Intervention Period
Comparisons Between Treatment Arms
In unadjusted analyses of practice-level data, the age-standardized PANVR during the intervention period in the P4P arm was significantly higher than in the QITS arm (mean ± SE, 92.3% ± 1.1% vs 86.9% ± 2.0%, P = 0.01) (Table 5). However, using patient-level data and adjusting for clustering within practice, patient age stratum, Medicaid insurance status, and baseline practice average PANVR, there was no significant difference between groups (90.7% ± 1.1% for P4P vs 86.1% ± 1.3% for QITS, P = 0.46).
PANVR.
In unadjusted analyses of practice-level data, the percentage of patients UTD in practices in the P4P arm trended higher than in the QITS arm (mean ± SE, 75.9% ± 10.2% vs 67.1% ± 16.7%, P = .06). However, on patient-level analysis, after adjustment, there was no difference in the odds of being UTD between study arms (aOR 1.02, 95% CI 0.68–1.52, P = .93).
Changes Within Treatment Arms From Baseline to Postintervention
At the practice level for P4P, we detected an improvement in the unadjusted PANVR over time from 89.3% ± 1.0% (mean ± SE) during the 12 months preceding the baseline assessment to 92.3% ± 1.1% during the 12-month intervention period (P = .05) (Table 5). However, among QITS practices, PANVR did not increase significantly over time (86.0% ± 1.7% vs 86.9% ± 2.0%, P = .72). Results were largely similar in the fully adjusted patient-level models. For patients in the P4P arm, there was a nonsignificant increase in PANVR, from 88.2% ± 1.3% to 90.7% ± 1.1% (P = .12). For patients in the QITS arm, PANVR did not significantly increase from baseline to postintervention (84.5% ± 1.4% vs 86.1% ± 1.3%, P = .30).
At the practice level, the unadjusted percentage of patients UTD did not improve significantly over time in the P4P arm (mean ± SE, 73.9% ± 7.6% to 75.9% ± 10.2%, P = .54) or in the QITS arm (64.2% ± 19.8% to 67.1% ± 16.7%, P = .65). Similarly, results from models predicting the adjusted odds of a patient being UTD revealed nonsignificant or only modest improvement over time in both study arms (P4P: aOR 1.16, 95% CI 0.92–1.46, P = .21; QITS: aOR 1.28, 95% CI 1.02–1.60, P = .03).
In P4P, 7 practices increased in age-standardized percent UTD by 5 to <10 percentage points and 1 practice by ≥10 percentage points. In QITS, 3 practices increased by 5 to <10 percentage points and 4 practices by ≥10 percentage points.
Other Factors
We sought to determine whether factors besides group allocation contributed to immunization improvement by comparing high performers (15 practices that improved percentage UTD by >5% from baseline to postintervention) to low performers (17 practices with ≤5% improvement), regardless of treatment allocation. High and low performers did not differ in demographic characteristics or most self-reported activities; however, high performers tended to have lower baseline percent UTD than low performers (64.1% ± 19.2% vs 73.5% ± 10.2%, P = .09) and were less likely to report using data to give clinicians feedback (27% vs 71%, P = .007).
Discussion
Participation in both a financial incentive program or a virtual QI learning collaborative led to modest improvement in immunization outcomes, as evidenced by temporal gains in mean PANVR of 3.0% among practices exposed to P4P and 28% increased odds of UTD among patients exposed to QITS (nonsignificant 2.9% increase in practice percentage UTD). On direct comparison, the P4P and QITS interventions induced similar QI activities and outcomes.
We found considerable variability among practices in change in immunization coverage over time, with a nonsignificant trend toward an association between lower baseline coverage and greater gains in percent UTD. This supports the intuitive notion that it is easier to improve outcomes when there is more room for improvement. Furthermore, it suggests that local contextual factors, such as organizational constraints, may pose significant barriers to greater improvement after “low-hanging fruit” have already been picked.37–40 Perhaps future guideline dissemination projects should place greater emphasis on helping practices first nurture an environment that is conducive to transformation.41
To our knowledge, this is the first comparative effectiveness trial comparing financial incentives to a virtual learning collaborative for impact on immunization coverage. A meta-analysis of 3 trials to improve adult immunization coverage via provider incentives found no effect compared with the control.17 A trial examining immunization coverage among children with Medicaid insurance found no effect of an intervention that combined provider incentives with performance feedback.18 A trial among inner-city pediatric practices found improvements with incentives but attributed much of the change to improved documentation rather than increased vaccine delivery.19 A trial examining the effect of a virtual immunization learning collaborative found an increase in practices’ self-reported immunization coverage by 4.9% over time, but improvement was not different from that of the control.20 The smaller measured increase in percent UTD among our QITS practices may be attributed in part to our use of random medical record selection and blinded data extraction.
Given the modest effects on immunization rates of both P4P and virtual QI learning collaboratives, the question arises as to whether use of either is worthwhile for improving immunization delivery. This question is timely as several medical boards reconsider their QI participation requirements and as demand grows for value-based care in which providers are financially rewarded for positive patient health outcomes.11,42–44 For individual patients who receive additional protection from vaccine-preventable diseases as a result of their health care practitioners’ stricter adherence to immunization guidelines, the benefit is clear.45 Furthermore, in communities where baseline vaccination rates are low, the benefit to the local population from even incremental improvement in herd immunity is potentially important.46–48 However, benefits must be weighed against the costs in time and expense for both the sponsors of and participants in financial incentive programs and learning collaboratives.
Our study had several strengths, including rigorous study design and exacting assessments of immunization status that incorporated both routine and catch-up schedules. Furthermore, participating practices were unaffiliated and of varying size, structure, and location, allowing for generalizability to many practice settings. In addition, our baseline percentage UTD (69.1%) was similar to national coverage rates (71.6%),49 suggesting that our sample reflects conditions nationwide. Finally, we were able to collect postintervention data from all practices that began the intervention (ie, no loss to follow-up). Our primary limitation was that because of financial and sample size constraints, we were unable to test various intensities of each intervention including a no-treatment control. Nonetheless, the 5 previous trials of financial incentives used widely varying levels of incentives and also found generally negative results.17–19 As for our virtual learning collaborative, the 12-month duration was longer than most QI projects, and the format was consistent with other projects coordinated by the AAP Quality Improvement Innovation Networks. Future studies using factorial study designs to compare varying levels of exposure to each intervention or studies that include a third arm that combines aspects of P4P and QITS could prove illuminating, but recruitment of enough participants for adequate power might be challenging.
Conclusions
Participation in either a financial incentives program or a virtual QI learning collaborative led to self-reported improvements in immunization practices but minimal change in objectively measured immunization coverage. Our findings suggest that immunization guideline dissemination is a complex process requiring further investigation into factors impacting success.
Acknowledgments
The authors thank the AAP Quality Improvement Innovation Networks, a subcontractor on this study, and especially Elizabeth Rice-Conboy for assistance with recruitment and implementation of the QITS intervention. The authors gratefully acknowledge all participants in the CIzQIDS study.
Footnotes
- Accepted March 11, 2016.
- Address correspondence to Linda Fu, MD, MS, Goldberg Center for Community Pediatric Health, Children’s National Health System, 111 Michigan Ave, NW, Washington, DC 20010. E-mail: lfu{at}childrensnational.org
This trial has been registered at www.clinicaltrials.gov (identifier NCT02432430).
FINANCIAL DISCLOSURE: Drs Fu, Gingold, Gillespie, Briccetti, and Moon and Ms Zook received salary support from Pfizer, Inc to conduct this study. Drs Fu and Cora-Bramble have served on advisory boards for Pfizer. Dr Joseph and Ms Haimowitz have no financial relationships relevant to this article to disclose.
FUNDING: Supported in part by Pfizer, Inc (Investigator Initiated Research grant award WS2163043). Financial incentives for the pay-for-performance intervention were supported by Children’s National Health System. Neither Pfizer nor Children’s National Health System had any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, and approval of the manuscript.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
References
- Copyright © 2016 by the American Academy of Pediatrics
