OBJECTIVE: To assess the effectiveness of reminder/recall (R/R) for immunizing adolescents in private pediatric practices and to describe the associated costs and revenues.
METHODS: We conducted a randomized controlled trial in 4 private pediatric practices in metropolitan Denver. In each practice, 400 adolescents aged 11 to 18 years who had not received 1 or more targeted vaccinations (tetanus-diphtheria-acellular pertussis, meningococcal conjugate, or first dose of human papillomavirus vaccine for female patients) were randomly selected and randomized to intervention (2 letters and 2 telephone calls) or control (usual care) groups. Primary outcomes were receipt of >1 targeted vaccines and receipt of all targeted vaccines 6 months postintervention. We calculated net additional revenue for each additional adolescent who received at least 1 targeted vaccine and for those who received all targeted vaccines.
RESULTS: Eight hundred adolescents were randomized to the intervention and 800 to the control group. Baseline rates of having already received tetanus-diphtheria-acellular pertussis, meningococcal conjugate, and first dose of human papillomavirus vaccine before R/R ranged from 33% to 54%. Postintervention, the intervention group had significantly higher proportions of receipt of at least 1 targeted vaccine (47.1% vs 34.6%, P < .0001) and receipt of all targeted vaccines (36.2% vs 25.2%, P < .0001) compared with the control group. Three practices had positive net revenues from R/R; 1 showed net losses.
CONCLUSIONS: R/R was successful at increasing immunization rates in adolescents and effect sizes were comparable to those in younger children. Practices conducting R/R may benefit financially if they can generate additional well-child care visits and keep supply costs low.
- CIIS —
- Colorado Immunization Information System
- HPV —
- human papillomavirus vaccine
- HPV1 —
- first dose of human papillomavirus vaccine
- MCV4 —
- meningococcal conjugate vaccine
- R/R —
- TdaP —
- tetanus-diphtheria-acellular pertussis
- WCV —
- well-child care visit
What’s Known on This Subject:
Rates of coverage for recommended vaccinations in adolescents are substantially lower than Healthy People 2010 goals. Reminder/recall is an evidence-based strategy that is proven to increase immunization rates in both adults and young children.
What This Study Adds:
This study shows that reminder/recall is effective in increasing adolescent immunization rates. Practices may also benefit financially from conducting reminder/recall in this age group if they are able to generate additional well visits and keep supply costs low.
Since 2005, several new vaccines have been recommended for adolescents: tetanus-diphtheria-acellular pertussis (TdaP) vaccine,1 meningococcal conjugate vaccine (MCV4),2 and human papillomavirus vaccine (HPV).3 Because these immunizations are recommended for adolescents, there has been a renewed focus on improving immunization delivery4 by concentrating on the 11- to 12-year-old primary care visit and creating an adolescent vaccination platform.5 Barriers to adolescent immunization include lack of regular preventive care visits at this age,6 record scatter,7,8 difficulty obtaining parental consent,9 and lack of insurance.10
Reminder/recall (R/R) has been proposed as an evidence-based solution to immunizing adolescents. In R/R, patients receive a reminder notification for upcoming immunizations or a recall notice for overdue immunization(s). R/R is generally effective in increasing immunization rates in young children and adults,11,12 but there have been few studies of R/R systems for adolescents.13,14 The objectives of this study were to (1) determine the effectiveness of letter and autodialer R/R in increasing immunization rates for TdaP, MCV4, and first dose of HPV (HPV1; female patients only) among adolescents in 4 private urban/suburban pediatric practices, and (2) measure the costs and revenues associated with R/R. As secondary outcomes, we assessed the effect of the R/R on immunization rates for other childhood vaccines, completion of the 3-dose HPV series among female patients, and well-child care visit (WCV) rates.
Study Setting and Population
The study protocol was reviewed and approved by the Colorado Multiple Institutional Review Board as an expedited protocol, not requiring consent from individual families. We conducted the study in 4 suburban private pediatric practices in metropolitan Denver from February 2008 to August 2009. The practices participate in the Colorado Immunization Information System (CIIS), a statewide immunization registry, and share a common computerized billing system.
The study population was a sample of adolescents aged 11 to 18 years who had been seen at their practice at least once in the preceding 2 years. Adolescents were eligible for study enrollment if they needed 1 or more of the targeted adolescent vaccines (TdaP, MCV4, and HPV1 for female patients only). In the case in which an adolescent had a sibling who also met inclusion criteria, 1 adolescent from the household was randomly chosen to be in the study. Eligible siblings received the same intervention type as the enrolled adolescent, but their data were not analyzed.
To ensure that all immunizations previously given at the practice were represented in CIIS, administrative data from the practices’ electronic billing systems were merged with CIIS data, and these combined data were used to determine which adolescents were eligible for our study. Patient contact information (home phone number and address) was determined from administrative data.
We randomized at the patient-level within each practice by using random number generation (SAS 9.1, SAS Institute, Cary, NC). Providers were blinded to group allocation.
The R/R intervention consisted of up to 2 letters separated by 2 autodialer telephone calls because previous literature has demonstrated that a letter followed by a telephone message is better than either method alone.15 All families were sent a first letter and autodialer telephone call. Adolescents still in need of targeted immunizations 1 month later received a second autodialer telephone call. A final letter was sent to adolescents still needing immunizations 2 months after the initial R/R. The letters and autodialer calls were scripted with the input of the practices and letters were sent on each practice’s letterhead. The letter and phone calls informed parents that their adolescent was due to receive at least 1 immunization (TdaP, MCV4, or HPV1 for female patients), and provided brief information on each immunization. Adolescents in the control group received usual care, which did not include R/R.
The first reminder was sent with a form allowing a family to defer or decline additional R/R for the following reasons: they did not want the patient to receive the vaccine(s), the patient had already received the vaccine(s), the patient already had an appointment scheduled to receive the vaccine(s), or the parent did not wish to receive additional reminders. The individual participating practices decided whether to schedule a WCV or a shot-only visit for a particular child, but all practices had a policy of recommending yearly WCV for adolescent patients.
The primary outcome measures were receipt of at least 1 targeted adolescent vaccine and receipt of all targeted adolescent vaccines at 6 months postintervention. Secondary outcomes included completion of the following vaccine series at 12 months postintervention: hepatitis A, hepatitis B, varicella, and HPV (females only). We also examined WCV rates at 6 months and 12 months postintervention. We used immunization data from CIIS supplemented by billing data to determine all immunization-related outcome measures, and billing data to determine WCV rates. The reach of the intervention was estimated by assuming letters had been received if they were not returned or forwarded and telephone calls received if either a person or voicemail was reached.
This study was powered at 80% to detect an absolute difference of 7 percentage points in postimmunization rates between control and intervention groups. The target for each practice was to enroll and randomize 200 adolescents to each study group. We used intention to treat for all analyses. We calculated descriptive statistics for patient gender and age and compared intervention and control groups for all outcomes by using χ2 tests. We computed unadjusted relative risks and 95% confidence intervals for the primary outcome by using a log binomial regression model by using SAS PROC GENMOD with the binomial distribution and the log link function. We derived adjusted relative risks and 95% confidence intervals for the primary outcome from logistic regression models by using regression risk analysis.16 We included gender, age, and practice site as covariates. We performed all statistical analyses by using SAS software.
Cost Analysis Methods
The net cost analysis in this study took the point of view of the pediatric practices, based on the fact that these practices function as small businesses and that costs and revenues must be considered in their decision-making. We calculated cost and revenue per additional adolescent who received all targeted vaccines and at least 1 targeted vaccine. We calculated total costs for each practice, divided into initial startup costs and operating costs. To calculate net additional revenue, we only used operating costs (personnel and supply costs). Initial startup costs are 1-time costs that would be incurred in developing an R/R system; for this study, they included those related to creating and translating the autodialer script. We excluded startup costs in our net revenue calculations because of uncertainty regarding the period over which these costs would be amortized by different practices. However, startup costs would need to be considered if a practice were to begin an R/R program de novo.
To determine personnel costs, study staff members documented the amount of time they spent on R/R activities, including defining the eligible population, checking immunization records, and generating patient lists for recall. For each activity, we collected the job titles of office staff who would typically perform it. Office staff members recorded their time (1) keeping track of patients who called to make appointments or ask questions after receiving a notice and (2) processing returned mail. We applied national average salaries published by the Bureau of Labor Statistics for each job title to the times recorded by both project and practice staff.17 We determined supply costs by examining invoices for mailing supplies, postage, and autodialer supplies.
We estimated reimbursement for each adolescent within the intervention and control groups for each practice by using patient age, number of visits, and number of shots received. We reviewed Current Procedural Terminology codes to identify WCV and immunizations. Other non-WCV immunization visits were defined as an office visit in which vaccinations were given but no WCV code was observed and included both immunization-only visits and visits for sick or follow-up care. The revenues for all non-WCV only include administration and vaccination costs. Reimbursement data for WCV and immunizations were taken from the American Academy of Pediatrics Analysis of 2005 Medstat MarketScan18 and Colorado Medicaid reimbursement rates.19 We calculated total additional revenue for each practice by subtracting the revenues for the control group from those of the intervention group. We calculated net additional revenue for each practice by subtracting total operating costs from total additional revenues. To make this study useful for pediatric practices, we included revenues resulting from all shots given during the visit prompted by the recall, not only those for the target vaccines. Our analyses did not include either cost or revenue realized from the purchase of vaccines.
Figure 1 is a Consolidated Standards of Reporting Trials diagram detailing selection of the study population. Of the 6775 adolescents who needed ≥1 target vaccines, 1600 were randomly selected, with 800 randomized to the intervention and 800 to the control group. Seven hundred fifty-one (94%) adolescents in the intervention group received at least 1 recall phone call and 1 recall letter, 34 (4%) received recall letters only, 13 (2%) received at least 1 recall phone call only, and 2 (<1%) received neither recall letters nor phone calls. Recalls were stopped for 91 patients (11%) whose parents returned the defer or decline immunization form; these patients were included in the final analysis. Table 1 presents characteristics of enrolled patients including baseline up-to-date rates for the target vaccines for practices. Overall, the intervention and control groups did not differ with respect to gender, age, or insurance status. Baseline up-to-date rates for all targeted vaccines ranged from 32.6% to 53.9% across practices and was 40.3% overall for the study population. Three of the practices were located in suburban areas and the fourth was in an urban, but not inner-city, setting. The number of midlevel or physician providers varied from 8 to 21 providers per practice, and the total number of adolescent patients who had been seen within the past 2 years ranged from 1138 to 3200 per practice.
Effectiveness of R/R in Pediatric Practices
When results for all practices were pooled, a significantly higher percentage of adolescents in the intervention group versus the control group received at least 1 targeted vaccine (P < .001; Fig 2). In 3 individual practices, the intervention group had a significantly higher proportion of adolescents who received at least 1 targeted vaccine compared with the control group (P < .05), with effect sizes ranging from 15.2 to 20.5 percentage points. In 1 practice (Practice 2), no effect was observed. Among the entire study population, the adjusted risk ratio for probability of an adolescent in the intervention versus control group to receive at least 1 targeted vaccine was 1.36 (95% confidence interval 1.21–1.54), adjusting for age, gender, and practice site.
In addition, among all the practices a significantly higher percentage of adolescents in the intervention group versus the control group received all targeted vaccines (P < .001; Fig 2). In 3 of the individual practices, significantly higher proportions of adolescents in the intervention group received all targeted vaccines compared with the control group (P < .05), with effect size ranging from 10.1 to 19.5 percentage points. Again, in Practice 2, no effect was observed. The adjusted risk ratio for probability of an adolescent in the intervention versus control group to receive all targeted vaccines was 1.44 (95% confidence interval 1.25–1.67), adjusting for age, gender, and practice site. Figure 3 shows the percentages of adolescents who received each individual study vaccine, among those who needed that vaccine, in the intervention and control groups. Overall, there was a significant increase in the intervention group compared with the control group for each of the individual vaccines (P < .05). Immunization rates were lower for HPV1 than for the other 2 vaccines.
Table 2 shows revenue generated by WCV and other non-WCV immunization visits, total operating costs, total additional revenues, and net additional revenues associated with the R/R program. The startup costs (autodialer script preparation and transcription) for the 4 practices, which ranged from $327 to $485, are excluded. After subtracting total operating costs from total additional revenues, 3 practices experienced positive net additional revenues and 1 (Practice 2) showed a loss. Three of the practices also experienced positive net revenues per each additional adolescent who received all targeted vaccines (range: $2483–$753) and for each additional adolescent who received at least 1 targeted vaccine (range: $75.24–$21.32). Because in Practice 2 more adolescents in the control group than the intervention group received at least 1 vaccine, it was not possible to calculate net revenue, but it is likely that there was a net loss for this practice.
Table 3 demonstrates the percentages of the intervention and control groups that had not received 2 hepatitis A vaccines, 3 hepatitis B vaccines, and 2 varicella vaccines. The percentage brought up-to-date for each of these series by 12 months postintervention are also shown. The R/R intervention was associated with significant increases in series completion for hepatitis A and varicella but not for hepatitis B. Shown in Table 4 is the percentage of female patients who had not initiated the HPV series at baseline who completed the series among both the intervention and control groups. The percentage of adolescents who attended a WCV was significantly higher in the intervention group compared with the control group (35.3% vs 30.1%, P < .05) at 6 months postintervention. However, differences were no longer significant 12 months postintervention (46.4% vs 44.9%, P = not significant).
The results of this study demonstrate that R/R increased immunization rates among adolescents in 3 of the 4 private pediatric practices, with effect sizes similar to or larger than those seen in previous studies of R/R for infants and adults11 and adolescents.20 The intervention in these private practices also resulted in higher up-to-date rates for other vaccines such as hepatitis A and varicella as well as a higher rate of completion of the HPV series, demonstrating ancillary benefits of conducting R/R in this population. Furthermore, the 3 practices that were successful in their R/R efforts showed positive net revenues associated with the R/R effort when we included revenues associated with visits generated. We also found that the intervention resulted in more timely receipt of WCV at 6 months postintervention, although this effect was no longer seen at 12 months.
Our results show a level of effectiveness of R/R that is at least equivalent to the effectiveness demonstrated in previous trials conducted primarily in practices serving children or adults. A 2009 Cochrane review noted immunization increases in the range of 5% to 20% for all settings,21 and a systematic review in 2000 showed a median absolute change of ∼8.2% in previous trials of recall conducted specifically within private practice settings.22 In a study of a tiered immunization intervention aimed at adolescents that consisted of patient immunization tracking, R/R, and home visits by patient immunization navigators, researchers found an absolute increase in immunization rates in the range of 12% to 16% over control.20 In 3 of our practices, the absolute change ranged from 15% to 20% and, even taking into account the practice in which the recall was ineffective, the overall change was ∼13%. Taken together with these previous studies, our findings that R/R can be effective for increasing rates in adolescents in some settings are encouraging.
The results of our study differ substantially from a previous investigation of R/R in adolescents conducted in 4 urban primary care sites with 35% to 40% of the study population insured by Medicaid.13 In that study, weekly audiotaped reminders were minimally effective at increasing immunization rates in 11- to 14-year-old adolescents. Families with only 1 telephone contact number had a greater response to the recalls than families with multiple telephone contact numbers, suggesting that problems with contacting families contributed to the lack of effect seen. In another trial in which immunization recall for young children conducted in an urban teaching hospital serving a low-income population was not effective at increasing immunization rates, the investigators found they were unable to reach ∼30% of families either by phone or mail.23 The comparison of our findings with these studies underscores the importance of the practice settings and study populations in the success of R/R efforts. Maintaining accurate contact information may be more difficult in patient care settings that serve predominately low-income, urban populations that may more frequently move or lose telephone service. The private practices that participated in our study are located in suburban settings with patient populations that are generally of higher socioeconomic status, with approximately <10% of patient populations being publically insured. In addition, we enrolled adolescents who had been seen in their respective practice in the prior 2 years to ensure the most accurate contact information and that we were excluding those adolescents who were no longer active patients. As a result, only 2 adolescents out of the intervention did not receive either a recall phone call or recall letter in our study.
It is important to note that receipt of HPV1 among those female patients who were unimmunized was just over half as common as receipt of TdaP among those needing these vaccines. This is consistent with recent national data from the National Immunization Survey-Teen, demonstrating significantly lower rates of initiation of the HPV series compared with receipt of TdaP and MCV4 vaccines.24 We did demonstrate a statistically significant increase in completion of the 3-dose HPV series 1 year after the intervention, although the rate of completion in the intervention group was low. In addition, there were ancillary benefits of the targeted R/R on increasing rates of other childhood immunization series that adolescents were missing.
The 3 private practices that successfully raised immunization rates through R/R all generated extra revenue from extra visits and payment for vaccine administration. It is difficult to compare our cost and revenue data with other studies because of the variability in methods of assessing costs, variation in the items included in the cost analyses, and the lack of measurement of both costs and revenues in previous studies.21 Although a recent Cochrane review included 8 cost-effectiveness studies of R/R,15,25–30 to our knowledge, this is the first trial concerning costs and/or effectiveness of R/R that takes the point of view of practicing pediatricians and examines both costs and revenues. This approach allows us to more directly address the likely economic consequences to practices of adopting an R/R system such as that used in the study. In our study, the practices that generated the most additional WCV (Practices 1 and 3) had the highest net additional revenues. However, Practice 4, which scheduled fewer WCV and delivered more vaccinations at immunization-only visits, also experienced a profit. Although scheduling WCV generated the most revenue, the ability of practices to schedule such visits is limited by the fact that some insurance plans do not cover these annual visits31 or that a WCV may not be indicated for an adolescent for that year.32 The operating costs for R/R at the 4 practices ranged from $1087 to $1349. With 1 exception, the greatest single cost for the practices was supplies, including the printing of letterheads and envelopes. The range of supply costs for these practices was wide; the highest practice supply cost 50% more than that for the lowest practice. If this particular cost could be lowered, net additional revenues would increase.
Surprisingly, our R/R intervention had no effect on increasing immunization rates in 1 practice. This practice also had the largest net revenue loss associated with the intervention, mainly because the intervention did not generate additional WCV or vaccinations and the practice had relatively high operating costs. Researchers met with key providers in this practice after the results were analyzed to discuss potential explanations for the negative results. We identified problems with staff turnover and miscommunication about the R/R project among front office staff and providers that may have resulted in cross-contamination between the intervention and control groups. This information provides only limited insight into the lack of effectiveness of the intervention in this practice, although it is interesting to note that this practice had the lowest baseline immunization rates overall, which might suggest more systematic limitations in implementation within this practice.
There are some strengths and limitations to our study. It was a randomized controlled trial with randomization at the patient level. All study participants received their allocated intervention and we were able to obtain postintervention immunization data on close to 100% of participants. Also, although some startup recall activities were performed by the study team, we attempted to simulate practice-based R/R by suggesting that reminder letters be printed on practice letterhead, having staff from practices record autodialer messages, and allowing practice staff to handle all questions and visit-related activities. The primary limitations of our study have to do with the external generalizability of our results. The 4 private pediatric practices in which this study was conducted participate in the statewide, computerized immunization registry and have participated in previous R/R efforts33,34 and therefore may be more motivated to perform R/R than other practices. In addition, their patient populations are generally insured and of higher socioeconomic level. Finally, it is important to note that if R/R were implemented in practices without a functioning immunization information system and easily accessible administrative records, identifying patients to target for R/R would be more difficult and likely associated with increased costs; in such cases, R/R may not be economically advantageous.
Our study shows that R/R is as effective in increasing immunization rates among adolescents as it has been among younger children in practices serving suburban populations. All practices that were successful in increasing rates by using R/R showed net profits and the cost analysis conducted identified modifiable factors that can result in lower costs and higher revenues, such as reducing supply costs and maximizing WCV when administering vaccinations.
- Accepted March 5, 2012.
- Address correspondence to Christina Suh, MD, 13199 Montview Blvd, Suite 300, Aurora, CO 80045. E-mail:
Portions of this work were presented at the 43rd National Immunization Conference; April 2009; Dallas, TX, and the annual meeting of the Pediatric Academic Societies; May 2009; Baltimore, MD.
The findings and conclusions in this report are those of the authors, and do not necessarily represent the views of the Centers for Disease Control and Prevention, US Department of Health and Human Services.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Funded by a grant from the Centers for Disease Control and Prevention (3U01IP000129-02W1) to the University of Colorado Denver Anschutz Medical Campus.
COMPANION PAPER: A companion to this article can be found on page e1446, online at www.pediatrics.org/cgi/doi/10.1542/peds.2011-2921.
- Broder KR,
- Cortese MM,
- Iskander JK,
- et al.,
- Advisory Committee on Immunization Practices (ACIP)
- Bilukha OO,
- Rosenstein N,
- National Center for Infectious Diseases, Centers for Disease Control and Prevention (CDC)
- Markowitz LE,
- Dunne EF,
- Saraiya M,
- Lawson HW,
- Chesson H,
- Unger ER,
- Centers for Disease Control and Prevention (CDC),
- Advisory Committee on Immunization Practices (ACIP)
- McCauley MM,
- Fishbein DB,
- Santoli JM
- Oster NV,
- McPhillips-Tangum CA,
- Averhoff F,
- Howell K
- Szilagyi PG,
- Rand CM,
- McLaurin J,
- et al.,
- Working Group on Adolescent Vaccination in the Medical Home
- ↵Jacobson VJ, Szilagyi P. Patient reminder and patient recall systems to improve immunization rates. Cochrane Database Syst Rev. 2009;CD003941
- ↵Guide to community preventive services. Universally recommended vaccinations: client reminder & recall systems. 2008 February. Available at: www.thecommunityguide.org/vaccines/universally/clientreminder.html. Accessed November 2, 2011
- Lieu TA,
- Capra AM,
- Makol J,
- Black SB,
- Shinefield HR
- ↵Bureau of Labor Statistics. May 2009 national occupational employment and wage estimates, United States. 2010
- ↵Medicaid, Medicare and commercial payments for commonly reported pediatric services, 2004/05: Colorado. 2006
- ↵Medicaid reimbursement survey, 2004/05: Colorado
- ↵Jacobson Vann JC, Szilagyi P. Patient reminder and patient recall systems to improve immunization rates. Cochrane Database Syst Rev. 2009;CD003941
- Rodewald LE,
- Szilagyi PG,
- Humiston SG,
- Barth R,
- Kraus R,
- Raubertas RF
- Rosser WW,
- Hutchison BG,
- McDowell I,
- Newell C
- McDowell I,
- Newell C,
- Rosser W
- ↵US Preventive Services Task Force. Guide to clinical preventive services 2010–2011: recommendations of the U.S. Preventive Services Task Force. 2010
- Daley MF,
- Barrow J,
- Pearson K,
- et al
- Kempe A,
- Daley MF,
- Barrow J,
- et al
- Copyright © 2012 by the American Academy of Pediatrics