OBJECTIVES: To assess the effectiveness of in-person and webinar-delivered AFIX (Assessment, Feedback, Incentives, and eXchange) consultations for increasing adolescent vaccine coverage.
METHODS: We randomly assigned 91 primary care clinics in North Carolina, serving 107 443 adolescents, to receive no consultation or an in-person or webinar AFIX consultation. We delivered in-person consultations in April through May 2011 and webinar consultations in May through August 2011. The state's immunization registry provided vaccine coverage data for younger patients (ages 11–12 years) and older patients (ages 13–18 years) for 3 adolescent vaccines: tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis (Tdap); meningococcal; and human papillomavirus (HPV) vaccines (≥1 dose, females only).
RESULTS: At the 5-month follow-up, AFIX consultations increased vaccine coverage among younger adolescents. Patients in the in-person arm experienced coverage changes that exceeded those in the control arm for Tdap (3.4% [95% confidence interval (CI): 2.2 to 4.6]), meningococcal (4.7% [95% CI: 2.3 to 7.2], and HPV (1.5% [95% CI: 0.3 to 2.7]) vaccines. Patients in the webinar versus control arm also experienced larger changes for these vaccines. AFIX did little to improve coverage among older adolescents. At 1 year, the 3 arms showed similar coverage changes. The effectiveness of in-person and webinar consultations was not statistically different at either time point (all, P >.05).
CONCLUSIONS: Webinar AFIX consultations were as effective as in-person consultations in achieving short-term increases in vaccine coverage for younger adolescents. AFIX consultations for adolescents need improvement to have a stronger and more durable impact, especially for HPV vaccine.
- adolescent health services
- human papillomavirus infections/prevention and control
- North Carolina
- vaccination/statistics and numerical data
- AFIX —
- Assessment, Feedback, Incentives, and eXchange
- CDC —
- Centers for Disease Control and Prevention
- CI —
- confidence interval
- HBV —
- hepatitis B virus
- HPV —
- human papillomavirus
- MMR —
- NIS —
- National Immunization Survey
- Tdap —
- tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis
What’s Known on This Subject:
The Centers for Disease Control and Prevention recommends that health departments in all 50 states deliver AFIX (Assessment, Feedback, Incentives, and eXchange) consultations to 25% of federally funded vaccine providers each year. AFIX effectively raises vaccination coverage among young children.
What This Study Adds:
AFIX consultations achieved short-term gains in coverage for 11- to 12-year-olds for vaccines in the adolescent platform. No gains occurred for older adolescents or over the long term. Consultations were equally effective when delivered in-person or by webinar.
Adolescent vaccines are a cost-effective way to protect young people from a host of vaccine-preventable diseases, but few adolescents in the United States receive the full benefit. Only 33% of adolescent girls complete the 3-dose human papillomavirus (HPV) vaccine series, and coverage has improved little in recent years.1 Among adolescents of both sexes, coverage for tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis (Tdap) and meningococcal conjugate is higher at 85% and 74%, respectively. Even for these vaccines, however, uptake varies widely by state. Geographic disparities, along with plateauing HPV vaccine coverage nationally, suggest an urgent need for immunization quality improvement.
The experience of early childhood immunization suggests that the Centers for Disease Control and Prevention’s (CDC) AFIX (Assessment, Feedback, Incentives, and eXchange) program is a promising strategy.2,3 Delivered by state and regional health departments to vaccine providers, AFIX consists of a brief consultation in which an immunization specialist evaluates a clinic’s vaccine coverage levels and works with providers to set goals for improvement. For early childhood vaccines, this low-cost strategy of “assessment and feedback” increases coverage by 4 to 7 percentage points3,4 and has received the Community Preventive Services Task Force’s strongest recommendation for use.5 The CDC supports AFIX programs in all 50 states, and it encourages health departments to deliver early childhood AFIX consultations to at least one-quarter of federally funded vaccine providers each year. The existing national infrastructure for AFIX raises the possibility that the program could be used to address underimmunization among adolescents.
We conducted a randomized controlled trial with high-volume primary care clinics in North Carolina to assess the effectiveness of AFIX consultations in increasing adolescent vaccine coverage. We hypothesized that clinics receiving in-person AFIX consultations would have larger increases in coverage for adolescent vaccines than clinics with no consultation. To expand the program’s potential reach, we also assessed whether AFIX consultations delivered using webinars would similarly increase coverage.
Eligible clinics were identified by using the North Carolina Immunization Registry, an online tracking system used by 94% of the 1201 health care facilities in the state’s publicly funded vaccine program. In 2011, an estimated 67% of North Carolina adolescents had at least 2 vaccine doses documented in the registry.6 We identified pediatric and family practice clinics with >200 patients ages 11 to 18 years with active records in the registry; as recommended by the CDC, we targeted high-volume clinics to maximize program reach. One author (A.M.G.) randomly ordered and assigned 481 eligible clinics to 1 of 3 study arms by using a 1:1:1 ratio. For each arm, we selected the first 30 to 31 clinics, yielding 91 clinics in total or the number that many state health departments could reach in the study period (Fig 1). Thus, this study is a pragmatic clinical trial.7 It had power to detect a change in vaccine coverage of ∼2 percentage points, which we considered adequate given that AFIX consultations have been shown to increase early childhood vaccination by 4 to 7 percentage points.3,4 The North Carolina Division of Public Health institutional review board approved the study.
Our intervention has been described in detail elsewhere,8 but briefly, each clinic received 1 in-person AFIX consultation (delivered April–May 2011), 1 AFIX consultation by webinar (delivered May–August 2011), or no consultation. During the consultation, which consisted of a single 60- to 90-minute session, an immunization specialist (A.M.D.) met with the nurse who served as the clinic’s designated vaccine coordinator to evaluate vaccine coverage.
In the “assessment and feedback” component of the consultation, the immunization specialist presented coordinators with separate coverage estimates, specific to their clinic, for each of the following vaccines: Tdap; meningococcal conjugate; 1 and 3 doses of HPV vaccine (female patients only); 2 doses of measles-mumps-rubella (MMR); 3 doses of hepatitis B virus (HBV); and 2 doses of varicella. In the “exchange” component, the specialist helped coordinators gauge their progress by sharing information about average vaccine coverage for their clinic’s county as well as coverage attained by other clinics within the county.
In the “incentives” component, the specialist provided training in immunization best practices. Topics included how to maintain records in the immunization registry, how to use the registry to generate reminders for patients overdue for vaccination, and how to decrease missed opportunities for concomitant vaccination. After completing an evaluation of the clinic’s current strengths and weaknesses, the vaccine coordinator selected several goals from a list of 20 prespecified immunization best practices on which to focus improvement efforts. At the 5-month follow-up, the specialist presented coordinators with updated vaccine coverage estimates so that they could assess their progress.
Webinar AFIX consultations used the same content and one-on-one approach as in-person consultations, except that the immunization specialist communicated with vaccine coordinators via Adobe Connect (Adobe Systems Incorporated, San Jose, CA). This interactive conferencing platform included a screen-sharing function that allowed coordinators to practice registry manipulation.
We assessed vaccination status of 11- to 12-year-old and 13- to 18-year-old patients at 5-month and 1-year follow-up. Patients were stratified according to age because 11- to 12-year-olds constitute the target age for routine administration of adolescent vaccines, whereas 13- to 18-year-olds are eligible for catch-up vaccination. Data were extracted from the North Carolina Immunization Registry by using the CDC’s Comprehensive Clinic Assessment Software Application.9 For each age group, we assessed the number who had received vaccines in the adolescent platform: Tdap, meningococcal, and HPV series initiation and completion (≥1 dose and 3 doses, female patients only). We also assessed the number who received 3 childhood vaccines administered to adolescents on a catch-up basis: HBV (3 doses), MMR (2 doses), and varicella (2 doses). The registry also provided data on how many patients were up-to-date on all 4 of the following vaccines: Tdap, meningococcal, HBV (3 doses), and MMR (2 doses). The CDC’s Comprehensive Clinic Assessment Software Application calculated missed opportunities (ie, instances in which a patient received a vaccine without receiving 1 of the 4 recommended vaccines listed earlier).
The registry was then used to assess vaccine coverage at baseline separately for the 5-month and 1-year patient populations. We established study populations at the follow-up time points to include patients who joined study clinics during the evaluation period.
The registry provided data on clinic characteristics, including total number of adolescent patients; the proportions of adolescent patients who were black, white, or another race; the proportions of male and female patients; and the proportions of publicly versus privately funded vaccine doses. Publicly funded doses were those funded by Vaccines for Children, a program that provides free vaccines to vulnerable populations, including uninsured and Medicaid-eligible youth.10 The registry also provided data on clinic specialty (pediatric or family practice) and location. We defined clinics located within a metropolitan statistical area as “urban or suburban” and others as “rural.”11
To assess whether study arms had different clinic-level characteristics, χ2 tests and analysis of variance models were used. To analyze intervention effects at the level of the patient, we performed mixed-level Poisson regressions for each vaccine, modeling the change in vaccine coverage between baseline and follow-up for each age group. Models included a random intercept to account for unobserved heterogeneity among clinics as well as an offset variable equal to the log of the number of adolescent patients at each clinic. Because trial arms differed on proportion of publicly funded vaccine doses, we controlled for that variable. Using planned contrasts, we compared coverage changes for each intervention arm versus the control arm as well as the intervention arms versus each other. The primary study outcome was 5-month coverage change for Tdap, meningococcal vaccine, and HPV vaccine initiation (≥1 dose, female patients only). We also analyzed coverage changes for other vaccines at 5 months and for all outcomes at 1 year.
A sensitivity analysis was performed to examine whether potential seasonal differences in vaccination rates affected findings for the primary study outcomes. Due to scheduling constraints, study arm activities occurred during different months. We collected 5-month follow-up data for clinics in the control arm in July to September 2011, for clinics in the in-person consultation arm in September and October 2011, and for clinics in the webinar arm in October 2011 to January 2012. To further isolate the effect of the intervention, we repeated our analyses for key findings by using only clinics from the control arm and the in-person arm that fell in the overlapping assessment period.
Vaccine coverage data at follow-up are reported as unadjusted proportions. Vaccine coverage changes are reported as differences in proportions, adjusted for publicly funded vaccine doses and accounting for clustering of data according to clinic. Data were analyzed by using SAS version 9.3 (SAS Institute, Inc, Cary, NC); regression analyses used the NLMIXED procedure. All statistical tests were 2-tailed with a critical α of 0.05.
All 91 clinics selected to be in the study completed AFIX consultations as assigned and had accessible data for 5-month vaccine coverage assessments. At 5 months, study clinics served 107 443 adolescents ages 11 to 18 years (Table 1). Clinic specialties were primarily pediatrics (58%) or family medicine (41%). The mean proportion of publicly funded vaccine doses was higher at clinics that received no AFIX consultation (53%; SD: 0.18) than at clinics that received in-person (42%; SD: 0.21) or webinar (42%; SD: 0.18) consultations (P < .05). Intervention arms did not statistically differ on other characteristics. At 1 year, 90 of 91 clinics had accessible data; the webinar arm lost 1 clinic to follow-up because of closure.
Coverage Change at 5 Months
Among adolescents ages 11 to 12 years, AFIX consultations increased coverage for the 3 vaccines in the adolescent platform at 5 months (Table 2, Fig 2). Patients served by clinics in the in-person arm had coverage changes that exceed those in the control arm for Tdap vaccine (3.4% [95% confidence interval (CI): 2.2 to 4.6]), meningococcal vaccine (4.7% [95% CI: 2.3 to 7.2]), and HPV vaccine initiation (1.5% [95% CI: 0.3 to 2.7]). Patients served by clinics in the webinar versus control arms also had larger coverage increases for Tdap vaccine (3.6% [95% CI: 2.4 to 4.9]), meningococcal vaccine (4.4% [95% CI: 2.0 to 6.8]), and HPV vaccine initiation (1.9 [95% CI: 0.7 to 3.1]). For other vaccines, the webinar arm had larger coverage changes for varicella vaccine only (1.8% [95% CI: 0.8 to 2.8]). There was no evidence of an intervention effect for HPV vaccine series completion, MMR vaccine, or HBV vaccine. Comparing the intervention arms with each other, we found a statistically significant difference for varicella vaccine only, which experienced a higher coverage change among clinics in the webinar arm (1.2% [95% CI: 0.2 to 2.3]).
Among adolescents ages 13 to 18 years, AFIX consultations increased vaccine coverage at 5 months only for the in-person versus control arms for HPV vaccine series completion (0.7% [95% CI: 0.1 to 1.3]). Comparing the intervention arms with each other, we found no evidence of statistically significant coverage changes for any vaccine (all, P > .05).
Missed Opportunities for Vaccination at 5 Months
Among 11- to 12-year olds, changes in combined coverage for 4 vaccines (Tdap, meningococcal, HBV, and MMR) were higher for clinics that received in-person consultations (3.8% [95% CI: 1.6 to 6.0]) and webinar consultations (3.8% [95% CI: 1.6 to 6.0]) compared with the control arm. For patients in the webinar arm, missed opportunities also increased more than for those in control clinics (1.5% [95%: 0.3 to 2.7]). Among 13- to 18-year-olds, neither intervention was associated with combined coverage changes or missed opportunities compared with the control arm.
Sensitivity Analysis of Time-Matched Clinics at 5 Months
The sensitivity analysis identified 9 clinics in the in-person consultation arm (n = 2489) and 9 clinics in the control arm (n = 2961) assessed during the same 1-month time period. Using this subsample, an analysis of 11- to 12-year-olds found associations consistent with the primary analysis for 2 of the 3 vaccines assessed. Adolescents in the in-person versus control arms experienced coverage changes that approached statistical significance for Tdap vaccine (1.9% [95% CI: –0.1 to 3.9]; P = .064) and HPV vaccine initiation (2.6% [95% CI: 0 to 5.3]; P = .053). The intervention effect for meningococcal vaccine was no longer statistically significant (0.8% [95% CI: –8.8 to 10.3]; P = .863).
Coverage Change at 1 Year
At 1 year, the in-person and webinar arms showed no statistically significant coverage changes relative to the control arm for the primary study outcomes (Fig 2) or other vaccines (Supplemental Table 3) in either age group. Statistically significant differences between the intervention arms occurred when comparing 11- to 12-year-old patients in the webinar arm versus the in-person arm for varicella vaccine (1.5% [95% CI: 0.0 to 2.9]). No other comparisons of the intervention arms were statistically significant (all, P > .05).
In this pragmatic, randomized controlled trial in North Carolina primary care clinics, we found that in-person and webinar AFIX consultations improved adolescent vaccine coverage at 5-months postintervention for 11- to 12-year-olds. For these adolescents, the intervention arms achieved relative increases of 3 to 4 percentage points for Tdap vaccine, 4 to 5 percentage points for meningococcal vaccine, and 2 percentage points for HPV vaccine initiation. These incremental improvements are similar to those attained in early childhood AFIX programs, but unlike previous studies, we did not find that such differences persisted over time.3
At 1 year, vaccine coverage levels in the intervention arms were similar to the control arm. This finding may indicate that vaccine providers in the intervention arms were able to initiate, but not sustain, quality improvement efforts. Alternatively, by increasing use of strategies such as reminder/recall, the intervention may have prompted parents in favor of adolescent vaccination to seek such services earlier than they otherwise would have. AFIX strategies may have done less to change the behavior of parents whose decision-making centered on whether, rather than when, to get adolescent vaccines.
In contrast to routine vaccination, AFIX consultations did not improve provision of “catch-up” doses at either time point. Among older adolescents, the intervention did little to change coverage with adolescent vaccines. Furthermore, except in the case of varicella vaccine, we found no intervention effect with regard to childhood vaccines among either age group, even though unvaccinated adolescents were eligible for catch-up doses. These findings suggest that, although the intervention was successful in supporting routine vaccination, the trainings did not prompt providers to thoroughly review patients’ records for missed vaccines.
Taken together, our findings suggest several areas in which further intervention development and evaluation are needed, especially as AFIX for adolescents is already being disseminated nationally. First, given that intervention effectiveness waned after 5 months, clinics may benefit from “booster” quality improvement sessions designed to reinforce the intervention. Expanding the follow-up feedback session, in which vaccine providers receive updated coverage estimates, to evaluate improvements in light of national benchmarks may be an especially promising approach. As assessing clinics’ vaccination coverage by using immunization registries becomes easier, providing more frequent feedback sessions may also become increasingly possible. Second, because AFIX consultations achieved smaller gains in HPV vaccine coverage compared with Tdap and meningococcal vaccines, future iterations of the program should seek specifically to emphasize HPV vaccination. Finally, the AFIX program must be improved with regard to catch-up vaccination, and emphasizing procedures such as chart review may be particularly effective in this regard. By developing AFIX in these key areas, program planners may be successful in sustaining the short-term gains achieved in this study, while also extending the benefits of AFIX to older adolescents.
We were encouraged to find that webinar and in-person AFIX consultations offered similar effectiveness. This finding suggests that an interactive webinar is a viable way to deliver AFIX without incurring travel costs. In a separate process evaluation,8 we calculated the cost of delivering our intervention as $152 per clinic for in-person consultations versus $100 per clinic for webinar consultations. Future studies should seek to replicate our findings in the context of early childhood AFIX because webinar delivery could significantly improve the efficiency of this nationally implemented program.
Study strengths include a strong study design, a large sample size, and the use of a provider-based immunization registry to assess vaccine coverage. Although the completeness and accuracy of registry data have not been well studied, our estimates of vaccine coverage among older adolescents at baseline are in line with state estimates derived from the 2011 National Immunization Survey–Teen for Tdap (77% vs 78%), meningococcal conjugate (66% vs 66%), and HPV vaccine initiation (57% vs 54%).12 This correspondence between National Immunization Survey–Teen and our own findings lends support to the quality of our data for adolescent vaccination. However, registry records likely underestimate coverage for early childhood vaccines, such as MMR, which were most often administered before the establishment of the registry and documented after the fact.
Limitations to the present study include the sequential, rather than simultaneous, delivery of interventions by study condition. Anecdotal evidence suggests that summer months elicit a peak in adolescent vaccination rates, which may have augmented or diminished the intervention effects we report. Although our sensitivity analysis and the localized pattern of our findings support an intervention effect, our results may reflect the timing of intervention delivery rather than, or in addition to, improved vaccine provision. Because our sample was restricted to primary care clinics with more than 200 adolescent patients with active registry records, replication of our findings with smaller clinics, those with specialties other than pediatric or family medicine, or those that do not use an electronic immunization registry is necessary.
Implemented in all 50 states,13 CDC’s AFIX program is well known as an evidence-based quality improvement strategy for increasing vaccine coverage among young children. This study provides early evidence to suggest that AFIX consultations, whether delivered in person or by interactive webinar, may also be effective in raising vaccine coverage levels among adolescents. However, improvements in the intervention are first required to sustain and extend the short-term gains in vaccine coverage achieved in this study. Given that the national infrastructure needed to support program implementation already exists, widespread dissemination of a modified AFIX program represents a unique opportunity to address geographic disparities in adolescent vaccination as well as the lackluster uptake of HPV vaccine nationally. To capitalize on this opportunity, future studies should seek to better understand which of the AFIX quality improvement strategies are most effective for increasing adolescent vaccine coverage, while continuing to explore approaches, such as webinar delivery, for maximizing program efficiency. In the long term, understanding how AFIX compares with other evidence-based strategies, such as centralized reminder/recall,14–16 will also be important for helping state health departments allocate limited funds for immunization quality improvement.
- Accepted April 28, 2014.
- Address correspondence to Noel T. Brewer, PhD, Department of Health Behavior, UNC Gillings School of Global Public Health, CB 7440, Chapel Hill, NC 27599. E-mail:
Dr Gilkey analyzed the data and drafted the methods, results, and discussion sections of the manuscript; Ms Dayton designed the trial, designed the data collection instruments, coordinated and supervised data collection, and critically reviewed the manuscript; Ms Moss prepared data for analysis, contributed to the initial analyses, and critically reviewed and revised the manuscript; Ms Sparks contributed to data collection, drafted the introduction to the manuscript, and reviewed and revised the manuscript; Ms Grimshaw designed the trial, designed the data collection instruments, and critically reviewed the manuscript; Dr Bowling designed the data analysis plan and critically reviewed the manuscript; Dr Brewer designed the trial, participated in survey design, reviewed data collection protocols and progress, supervised data analysis and manuscript preparation, and critically reviewed each draft of the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This intervention trial was conducted by North Carolina Immunization Branch staff using funding from the Centers for Disease Control and Prevention. Evaluation was provided by University of North Carolina staff supported by an unrestricted educational grant from GlaxoSmithKline and the Cancer Control Education Program at the University of North Carolina’s Lineberger Comprehensive Cancer Center (R25 CA57726).
POTENTIAL CONFLICT OF INTEREST: Dr Brewer has received human papillomavirus vaccine–related grants from or been on advisory boards for GlaxoSmithKline and Merck; he also served in the National Vaccine Advisory Committee Working Group on HPV Vaccine. The other authors have indicated they have no potential conflicts of interest to disclose.
- ↵Centers for Disease Control and Prevention. AFIX (Assessment, Feedback, Incentives, and eXchange). Available at: www.cdc.gov/vaccines/programs/afix/index.html. Accessed October 1, 2013
- ↵Community Preventive Services Task Force. Universally recommended vaccinations: provider assessment and feedback. Available at: www.thecommunityguide.org/vaccines/universally/ providerassessment.html. Accessed March 3, 2014
- ↵Centers for Disease Control and Prevention. Immunization information systems: 2011 IISAR data participation rates. Available at: www.cdc.gov/vaccines/programs/iis/annual-report-IISAR/2011-data.html#adolescent. Accessed March 3, 2014
- ↵Centers for Disease Control and Prevention. CoCASA (Comprehensive Clinic Assessment Software Application). Available at: www.cdc.gov/vaccines/programs/cocasa/index.html. Accessed March 3, 2014
- ↵Centers for Disease Control and Prevention. Vaccines for Children Program. Available at: www.cdc.gov/vaccines/programs/vfc/index.html. Accessed March 3, 2014
- ↵US Census Bureau. Metropolitan and micropolitan statistical areas. Available at: www.census.gov/population/metro/. Accessed March 3, 2014
- ↵Centers for Disease Control and Prevention. City/state/territory AFIX staff. Available at: www.cdc.gov/vaccines/programs/afix/contacts/city-state.html. Accessed March 3, 2014
- Copyright © 2014 by the American Academy of Pediatrics