Cost-Effectiveness of Childhood Immunization Reminder/Recall Systems in Urban Private Practices
Objective. To assess cost and cost-effectiveness of immunization reminder/recall systems in the private sector.
Methods. A manual postcard system (mail) was compared with a computer-based telephone system (autodialer) and control. Costs included time costs and the cost of equipment and supplies. The cost per child and the incremental cost of the intervention relative to control were computed. Cost-effectiveness ratios were computed for return visits and for immunizations delivered.
Results. The average cost per child was $2.28 for the mail group and $1.47 for the autodialer group. The incremental visit cost relative to the control was higher for the mail group ($9.52) than for the autodialer group ($3.48). The autodialer was more cost-effective in delivering immunizations: $4.06 per extra immunization (autodialer) versus $12.82 (mail).
Conclusions. Excluding start-up costs, the autodialer system was most cost-effective. Including autodialer equipment costs, the autodialer system is more cost-effective only for larger practices.
Low immunization levels in the United States have prompted attention to interventions to improve immunization delivery.1,,2 We undertook a study to evaluate systems designed to remind patients of return visit appointments for children <1 year old in practices of private providers in Houston/Harris County, Texas. The study addresses whether childhood immunization reminder/recall systems could a) increase the rate of patient return visits and b) improve rates of delivery of childhood immunizations as specified in the guidelines of the American Academy of Pediatrics, the American Academy of Family Physicians, and the Advisory Committee on Immunization Practices. Six sites (1138 children) were randomly assigned to a study arm: mail, autodialer, or control. The mail and autodialer interventions increased return visits and immunization delivery compared with control.
This article assesses the cost and cost-effectiveness of the immunization reminder/recall systems on return visits and immunization coverage levels for children <1 year old in the private sector. It compares the cost and cost-effectiveness of: the manual postcard (mail) reminder/recall system, the computer-based telephone (autodialer) reminder/recall system, and the control group.
Other studies have shown that reminder/recall systems improve immunization coverage levels in public clinics and large health maintenance organizations.3–6 However, cost was only roughly estimated in those studies. This study develops a detailed cost estimation methodology to assess the cost of reminder/recall systems when implemented by the study staff, by the physician's offices, or in conjunction with an immunization registry.
Study Population and Interventions
Six sites were selected to participate in a reminder/recall efficacy study using the following criteria: 1) absence of an existing immunization reminder/recall system at the practice site, 2) on-site delivery of childhood immunizations, and 3) participation in Clinic Assessment Software Application (CASA) evaluation. One thousand one hundred thirty-eight children <12 months old presenting to 1 of the 6 participating private pediatric practices in Houston/Harris County, Texas, were enrolled between May 1997 and April 1998. Patients were offered study participation if they were eligible for their first, second, or third diphtheria, tetanus, and pertussis vaccine/diphtheria, tetanus, and acellular pertussis vaccine (DTP/DTaP) at their next visit. Selected practices had similar CASA assessment rates.7 With the exception of age, demographic characteristics of the practice sites were not uniform.
The practices were randomly assigned to a study arm: mail, autodialer, or control. The mail group of patients received a postcard delivered through the US mail reminding them of the date of their return appointments. In the autodialer group, patients were reminded of their return appointment date for immunizations by a computer automated telephone message system.
A validation study was conducted to ensure that immunization reminder receipt did not confound study outcomes, ie, all children received reminder/recall messages within defined study intervals. Children were defined as return visit respondents if a practice return visit (well-child) was documented during the study interval. Nonrespondents were those children without evidence of a practice return visit. All return visit nonrespondents and recall 1 and 2 return visit respondents were included in the validation study. In addition, 10% of children (randomly selected from both study arms) with a study return visit after receipt of their reminder participated in the validation study. Only 7 system failures (5 postcards, 2 incorrect phone numbers) were identified.
The reminder/recall system was conducted from a centralized location for all providers. The methods to implement the reminder/recall systems are described in King et al7 and the events involved in the protocol are summarized below:
Enrollment: Included a chart review to identify eligibility, signing of the consent form, recording demographic data, and providing a point of service reminder. The same enrollment procedures were used in all 3 arms of the study.
Reminder: In the mail group, a postcard was sent 7 days before the target date for the return visit. The autodialer group received a computerized telephone message in the same time range. The control group received no reminder intervention.
Follow-up and recall: After the target immunization date, the study staff reviewed the chart and recorded return visit status. Patient record reviews were conducted in a standardized fashion at 5, 17 and 30 days after the target appointment date. For children with no response to the study reminder, up to 2 recall messages (postcard or computer-generated telephone messages) were delivered 1 to 2 days after the previously defined record review intervals.
Off-study: Children were considered off-study if there was a well-child return visit during the study window (target immunization date plus 30 days) or if they were nonrespondents at the end of the study window. Children were only enrolled for 1 target immunization (DTaP) and then were taken off-study. In addition, children were not eligible for reenrollment.
Cost Estimation Methodology for Study Staff
All expenses for equipment and supplies were recorded and allocated to each intervention. Time spent by the study staff was estimated based on detailed activity logs completed by the study staff. The staff completed the logs for 4 separate typical weeks (week 1: October 6, 1997–October 10, 1997; week 2: January 12, 1998–January 16, 1998; week 3: March 23, 1998–March 27, 1998; week 4: April 27, 1998–May 1, 1998). During week 4, a staff flow analysis was performed to validate the data. Examples of activities included chart review, processing postcards, autodialer submission and deletion (Table 1). Total time spent in each activity was obtained from the logs. To obtain standardized activity time (time per chart review, time per postcard, time per submission or deletion), a denominator was identified for each activity. Denominators were obtained from the records for the children processed during the week when the log was completed plus and minus 3 days to account for delays and lead-times. The standardized activity time in each activity was obtained by dividing total time by the corresponding denominator.
Time costs were imputed based on average study staff salary. Time costs were allocated to standardized activities. Each event in the protocol included a set of activities (Table 2 for a list of events and activities). The cost of each event consisted of the time cost for the activities involved and the cost of supplies. For example, the cost for the event “postcard' included the time cost of generating the label and processing the postcard and the cost of the postcard and mailing. A set of events identified the protocol for a child in each intervention group. The cost per child by return status was computed for each intervention. For example, the cost for a child in the mail group who has a return visit after the first recall included the cost of: enrollment, reminder postcard, follow-up, recall postcard, follow-up, going off-study and miscellaneous. For each intervention, the average cost per child was computed by dividing the total cost of the intervention by the number of children in that arm of the study.
Costs were separated into start-up costs and maintenance costs. The main start-up cost was the autodialer hardware and software. Other start-up costs include equipment and supplies used by the study staff such as laptops, data transfer software, and business cards. A computer at physicians' practices can be included as a start-up cost if the practice does not already have a computer for other purposes, such as billing. The computers were not included in this cost analysis.
Cost Estimation Methodology for Physician Offices and With an Immunization Registry
A large portion of study staff time was spent in study specific activities that would not be required if private providers implemented the recall/reminder systems directly from their offices. We therefore also assessed the cost of implementing the reminder/recall systems directly in the providers' offices by excluding study specific cost. Similarly, the cost of implementing the reminder/recall system in conjunction with a fully working registry was estimated based on the activities required in such setting. We assumed that medical assistants in physicians' offices and registry personnel would implement the reminder/recall system. Median annual salaries for medical assistants in the practices ($24 763) were comparable to the study staff salaries ($24 495). Costs would be lower than those reported in this article if receptionists were to implement the reminder/recall system as their salaries were lower ($20 654). Costs would be higher if nurses were to implement the program (nurses' salaries were $36 608).
The cost per 1000 children was computed for each intervention by multiplying the cost per child in each intervention by 1000. Two outcome measures were used: return visits and immunizations. The incremental cost of the intervention relative to the control per 1000 children was divided by the additional outcomes in the intervention group relative to the control group per 1000 children to obtain the cost-effectiveness ratio for each outcome unit. Two cost-effectiveness ratios were computed: the incremental cost per return visit and the incremental cost per immunization delivered. These 2 ratios were computed for the study cost, the cost for the interventions implemented directly in the physician's office, and the cost for the interventions implemented in conjunction with a fully operational immunization registry. Because start-up costs for the autodialer are very high and the cost per child depends on the number of children for which the autodialer is used, cost-effectiveness ratios were computed for maintenance costs.
The high cost of purchasing the autodialer hardware and software was prorated over the life of the machine to obtain a prorated annual cost. The autodialer hardware and software has a 5-year write off period for Internal Revenue System purposes (personal communication from Teletask Inc, Fair Oaks, CA). Assuming a 5-year life expectancy and 5% discount rate, we calculated the annual number of children (for whom the autodialer is used) that would equate the cost-effectiveness of the mail system to that of the autodialer system when including the autodialer prorated annual cost.
1. Prorating the Autodialer Start-Up Costs
In the main analysis, we considered a 5-year life expectancy for the autodialer. However, in a field where technology is rapidly changing, autodialers can become obsolete in <5 years. On the other hand, the machine can perform its functions for 10 or more years with only minimal maintenance. We therefore computed the prorated value of the autodialer assuming a life expectancy of 3 and 10 years, respectively. We also considered the 3% discount rate as currently recommended by analysts.8
2. Balance Points
We explored the effect of varying several assumptions about the cost of repeated autodialer use when computing the number of children needed for the autodialer to be as cost-effective as the mail system when start-up costs are included.
The time spent in each standardized activity is reported in Table 1 and the cost associated with each activity is reported in Table 2. The average weekly salary for the study staff was $510.31. Assuming that the staff works 40 hours or 2400 minutes a week, the average staff salary was $.21 per minute. The corresponding weekly salary for medical assistants in physicians' offices was $515.89 or $.21 per minute. The time cost was obtained by multiplying the time in minutes for each standardized activity by $.21.
In Table 2, activities are grouped into events, the building blocks of the interventions: enrollment, postcard processing, autodialer submission and deletion, taking the child off-study, and miscellaneous. The study cost for each event is considerably higher than the cost for the same event when the intervention is implemented directly in the physician's office or with the assistance of a fully operational registry.
The average cost per child in each arm of the study depends on the cost of implementing the reminder/recall system, on the number of children who return for a visit, and on when they return (Table 3). The average cost per child in the mail group is higher than the corresponding cost in the autodialer group (the costs of the autodialer hardware and software are not included). When the reminder/recall system is implemented directly at the physician's office, the average cost per child is $2.28 for the mail system and $1.12 for the autodialer system. The costs of the reminder/recall with the registry are slightly lower.
The effectiveness of the interventions relative to control is reported in Table 4. There were more return visits and more children immunized (per 1000 children) relative to the control group in both the mail and the autodialer groups. The autodialer group was more effective than the mail group for both outcomes.
The cost-effectiveness ratios indicate that each additional visit relative to the control costs more in the mail group than in the autodialer group, $9.52 versus $3.48 considering costs at the physician's office (Table 5). The reminder/recall system using the autodialer is more cost-effective than the system using postcards for return visits. Similarly, the cost-effectiveness ratios indicate that the autodialer system is more cost-effective than the mail system in delivering immunizations; $4.06 per extra immunization in the autodialer group versus $12.82 in the mail group considering costs at the physician's office (Table 5).
It is important to notice that these costs do not include the significant start-up costs of the autodialer hardware and software. In our study, the price paid for the autodialer ($17 785) included hardware, software, and maintenance for the life of the autodialer. The prorated annual cost of the autodialer is $3912 considering a 5-year life span and assuming a 5% discount rate. Including this cost, the autodialer would need to be used for 1997 immunizations to children <12 months old to be as cost-effective as the mail system in terms of increasing immunizations. Assuming that a child has 3 immunization visits during the first year of life and the cost and effectiveness of the intervention is the same for each visit, the cost-effectiveness of the autodialer over 1 year is the same as that of the mail system for a practice with at least 666 children <12 months old per year (666 children = 1997 immunizations divided by 3 immunization visits per child).
The autodialer needs to be used annually for 2482 return visits to be as cost-effective as the mail system when the outcome is the number of return visits. Assuming 5 well-child visits in the first year of life and constant intervention cost for each visit, the practice needs to have at least 496 children <1 year old for the autodialer to be cost-effective relative to the mail system (496 = 2482 ÷ 5). All the balance points were computed for costs at the physician's office.
1. Prorating the Autodialer Start-Up Costs
When we assume a 3% discount rate, the number of children <12 months required for the autodialer to be as cost-effective as the mail system for the immunization outcome decreases slightly to 641 from 666. The corresponding numbers of children required assuming the autodialer life expectancy to be 3 years and 10 years (and 5% discount rate) are 1058 and 373, respectively. The corresponding numbers when the outcome is the number of return visits are: 478 (5-year life expectancy and 3% discount rate), 789 (3-year life expectancy and 5% discount rate), 278 (10-year life expectancy and 5% discount rate). Again, costs at the physician's office were considered.
2. Balance Points
When computing the number of children needed to make the autodialer as cost-effective as the mail system, we assumed that the children are due for 5 well-child visits in the first year and that immunizations are delivered during 3 of these visits. The balance points were computed on the assumption that the cost and effectiveness of the intervention were the same for each visit. Although this assumption is probably realistic for the mail system, there are savings arising from the repeated use of the autodialer. The autodialer can be programmed once to deliver messages for all 5 well-child visits so that only 1 submission and 1 deletion is needed for each child. Therefore, the cost of the autodialer intervention over the first year of life is not simply the cost of the autodialer intervention multiplied by the number of visits. Taking these considerations into account we calculated the incremental physician's office cost for the autodialer intervention over the first year of life if used for all 5 well-child visits ($.72) and if used for only the 3 visits when immunizations are delivered ($.04). The corresponding incremental costs for the mail system were $9.52 for the 5 well-child visits and $12.82 if used solely for the 3 immunization visits. We then computed the number of children needed for the autodialer to be as cost-effective as the mail system: 482 children when the autodialer is used solely for the 3 immunization visits (immunization is the outcome) and 317 children when the autodialer is used for all 5 well-child visits (return visit is the outcome).
Cost for First Year of Life
Children should receive 5 well-baby visits, including 3 visits for immunization, during the first year of life. We compute the cost for the first year of life as the cost of the reminder/recall systems for the 3 immunization visits. Based on CASA rates in the Houston community, we assume that for the first visit (2 months) 80% need a reminder only and 20% a reminder and a recall, for the second visit (4 months) and the third visit (6 months) 60% need a reminder only and 40% a reminder and a recall. Delivery of the fourth dose of DTP/DTaP has been problematic and estimates for number of required reminders and recalls for age-appropriate delivery are not easily defined. For this reason, our costs have been determined excluding this vaccine and this may lead to underestimated costs in the first year of life. Based on the costs estimated in this study, the cost of the mail reminder/recall would be $5.64 for the first year of life (the sum for the 3 immunizations of the proportion of children receiving a reminder only multiplied by the cost of the reminder and the proportion of children receiving a reminder and a recall multiplied by the cost of reminder and recall or 80% × $1.46 + 20% × $2.72 + 60% × $1.46 + 40% × $2.72 + 60% × $1.46 + 40% × $2.72). The corresponding cost (excluding start-up cost) for the autodialer system would be $1.12 if the autodialer is programmed to deliver messages for all 3 immunization visits.
Our study is unique in comparing the cost-effectiveness of autodialer and mail reminder/recall systems in independent, private physician offices. It is also unique in assessing the reminder/recall systems for children <12 months old. Other studies have estimated the cost of reminder/recall systems for immunization purposes in children up to 2 years old in the public sector3,,4 and in health maintenance organizations.5,,6
The autodialer reminder/recall system was more effective than a system involving postcards in bringing children <12 months old back for return visits and in getting children immunized in private providers' offices. Excluding start-up costs, the autodialer system had lower cost per child than the mail system. Comparing the autodialer and mail systems with the control group, the autodialer system was more cost-effective than the mail system, when start-up costs were excluded.
The study costs were considerably higher than the costs for the interventions implemented directly in the physician's office or with the assistance of a fully operational registry because study-related costs were high. For example, enrollment involved study staff preparing enrollment packets with consent forms, reviewing charts at the physician's office to identify the eligible children, attaching a pocket to the charts of eligible children, training and assisting the physician's office staff in requesting parents of eligible children to fill in the consent form, bringing back data and documents to the project office and transferring the data to the project computer. When the reminder/recall system is part of the regular practice at a physician's site, the medical assistant only has to identify a child in need of reminder/recall as he or she comes in for a visit. With a fully operational registry, the registry can be programmed to identify children automatically so that there are no enrollment costs involved. There will still be some low costs to include new children migrating to the area and to assign medical homes to children who frequently change physicians, but we ignore these costs. Similarly, there are no follow-up costs and costs associated with taking the child off-study when the reminder/recall takes place at the physician's office or with the registry. On the other hand, the costs of processing the postcards and of submitting/deleting autodialer entries are the same whether the intervention is implemented by the project staff, the physician's office, or with the registry. The activities listed in the miscellaneous category apply only to the project staff. The results of this study support the use of the autodialer for reminder/recall purposes in private physicians' offices or in conjunction with a fully functional immunization registry.
The costs of the autodialer hardware and software are substantial, and the autodialer needs to be used for a large number of children to make it cost-effective compared with the mail system that has practically no start-up investment. Considering costs at the physician's office in our study, the autodialer is cost-effective in 1 year for a practice providing immunizations to at least 482 children or well-child visits to at least 317 children <12 months old. As the average pediatric practice follows about 443 children between 1 and 12 months old during 1 year,9 the autodialer will be cost-effective compared with the mail system. The balance points above are likely to overestimate the number of children required because the autodialer can also be used for the vaccines that are given later in childhood but were not included in this study, such as the DTP/DTaP4 and the vaccines given during the second year of life. In addition, new and more immunogenic vaccines are continually added to the immunization schedule; 7 new antigens and 16 additional doses in the last 9 years (1990– 1999).10 Additional uses of autodialer systems could include delivery of other interventions not included in the childhood immunization schedule (influenza vaccine, respiratory syncytial virus prophylaxis). Furthermore, pediatricians and family physicians can also use the autodialer for delivery of other prevention messages about lead toxicity, dental care, ophthalmologic care and trauma/accident prevention (car seat use). All these uses would decrease the number of children <12 months old that are required to reach the previously determined balance point.
Balance points are even lower when considering costs in conjunction with the registry. When the autodialer is used with the registry there are no deletion costs, further reducing the cost of the autodialer system compared with the mail system. For private practices, opportunities to link with existing immunization registries or managed care reminder/recall systems would significantly decrease start-up costs.
Saved Morbidity Dollars
It is important to note that childhood vaccines are highly cost-effective.11–14 For example, it has been estimated that every dollar spent on DTP/DTaP saves $29.00.15 Based on these estimates, the use of the autodialer reminder/recall system with an incremental cost of $4.06 per immunization is cost-effective from a societal view point because the saved morbidity dollars per immunization delivered are higher than the cost of the additional immunization. However, it is not possible to know from this study whether the additional immunizations attributed to the reminder/recall system would have been delivered at a later date. Therefore, the saved morbidity dollars per immunization delivered might be overestimated.
Comparisons With Other Studies
A major contribution of our study is the detailed methodology used in estimating the cost of 2 reminder/recall systems in different settings: the study setting, the physician's office setting, and in conjunction with a fully working registry. Linkins et al3estimate that the auto-dialing equipment would pay for itself in postage savings within 2 years in clinics serving >3500 children <2 years. Dini et al,4 using cost estimates similar to Linkins,3 compute the cost per additional appointment at $5.20 in the first year and $1.04 in subsequent years. These studies however computed a very rough estimate of cost based on approximate hardware, software, and maintenance costs and assigned personnel time costs.
Lieu et al5 evaluated the cost-effectiveness of computer-generated recall letters to parents of 20-month-olds who had not yet received a measles-mumps-rubella immunization. The cost per additional child appropriately immunized was estimated at $4.04, a figure consistent with our results even though the target populations differ. In a later study, Lieu et al6 considered the cost-effectiveness of a mail reminder, a phone reminder, and a reminder followed by a recall to underimmunized 20-month-olds. In contrast to our study that found the autodialer system was more effective than the mail system at increasing return visits as well as immunization delivery, they found no difference in immunization rates between those receiving a phone message or a letter. Only those receiving a letter followed by a phone message had statistically higher immunization rates by 24 months. The results of the 2 studies are not directly comparable because our reminder/recall systems involved 1 reminder and up to 2 recalls. The effectiveness of the intervention in the Lieu study6 was lower probably because the underimmunized 20-month-olds are harder to bring up to date compared with children <12 months old who may or may not be underimmunized. This may explain also the lower immunization rate in the Lieu6 control group (36%) compared with our control group (67%). The costs were higher overall in the Lieu study compared with our study after adjusting for the number of reminder/recalls sent.
This study has several limitations that need to be considered. First, study practices rather than children were randomized to intervention arms using CASA rates to control for immunization delivery practices. Therefore, the balance of race/ethnicity and payor status is not equivalent across study arms and could have impacted results. Second, the study population included only children <12 months old. It is well-known that this population has the highest utilization of services. It is therefore not possible to extrapolate the effectiveness and cost-effectiveness of the reminder/recall systems to older children, in particular children up to 2 years old. However, Linkins et al3 have shown that computer-generated telephone messages were most effective in getting patients to return for visits during the second year of life.
Our reported costs are local costs as only practices in Houston or Harris County in Texas were included in the study. However, this is unlikely to greatly affect the conclusions drawn from the cost-effectiveness ratios.
Only 6 practices were included in the study and it is possible that unobserved practice characteristics affected the results, although efforts to control for practice confounders were implemented across practices (similar CASA rates for participating practices, delivery of point of service reminders at all practices). In addition, it is possible that patient accessibility/mobility (change of address or phone number) may differ across practices and have a negative impact on cost efficacy. The study was not powered to evaluate such factors.
The cost of the reminder/recall systems when operated directly from the physicians' office or when used in conjunction with a fully operational registry were constructed by adding all nonstudy specific costs. No direct measurements of costs in those settings were made, so that the actual costs may differ based on the cost of using medical assistants, receptionists, or nurses in physician offices or registry personnel to implement the intervention. However, the constructed costs in this study are the most detailed available to date.
Immunization reminder/recall systems are best utilized in large patient populations where system costs are balanced by savings in reduced morbidity and/or mortality.6 Our data suggest that immunization reminder/recall systems are cost-effective in small, private pediatric practices. Excluding start-up costs, the autodialer system was most cost-effective. If autodialer equipment costs are included, the autodialer system is more cost-effective in larger practices only. Cost efficacy of reminder/recall systems is likely to increase as practices continue to shift to managed care and a greater number of private practices participate in immunization registry efforts.
This research was supported by the Association of Teachers of Preventive Medicine (Grant TS 83-11/13) and the National Centers for Disease Control, National Immunization Program.
- Received December 28, 1999.
- Accepted April 13, 2000.
Reprint requests to (C.H.) Baylor College of Medicine, 6621 Fannin, MC 1-3291, Houston, TX 77030. E-mail:
- CASA =
- Clinic Assessment Software Application •
- DTP/DTaP =
- diphtheria, tetanus, and pertussis vaccine/diphtheria, tetanus, and acellular pertussis vaccine
- ↵Centers for Disease Control and Prevention. Impact of missed opportunities to vaccinate preschool-aged children on vaccination coverage levels—selected US sites, 1991–1992. MMWR Morb Mortal Wkly Rep. 1994;43:709–711, 717
- ↵Lieu TA, Capra AM, Makol J, Black SB, Shinefield HR. Effectiveness and cost-effectiveness of letters, automated telephone messages, or both for underimmunized children in a health maintenance organization. Pediatrics. 1998;101(4). URL: http://www.pediatrics.org/cgi/content/full/101/4/e3
- ↵King H, Evans D, Balderas L, et al. Improving return visits in private provider offices through immunization reminder/recall. Pediatric Research. American Pediatric Society; New Orleans, LA; May 1–5, 1998;43:113A
- ↵Drummond MF, Stoddart GL, Torrance GW. Methods for Economic Evaluation of Health Care Programes. 2nd ed. Oxford, England: Oxford University Press; 1997
- ↵Rosenthal J, Brink E, Rodewald L. Immunization practices of private providers in the US in 1995 and 1997–1998. Meeting of the Pediatric Academic Societies; San Francisco, CA; May 1999
- American Academy of Pediatrics, Committee on Infectious Diseases
- Willems J,
- Sanders C
- Ginsberg G,
- Shouval D
- ↵Centers for Disease Control and Prevention. CDC Webpage available at: http://www.cdc.gov/nip/publication/niiw/PDF/FactsheetChildhoodImmun.pdf, 1999
- Copyright © 2000 American Academy of Pediatrics