Abstract
Background. Immunization rates have improved in the United States, but are still far from the national 90% goal for the year 2000. There is scant evidence about the effectiveness and costs of automated telephone messages to improve immunization rates among privately insured children.
Objective. To evaluate the effectiveness and cost-effectiveness of sending letters, automated telephone messages, or both to families of underimmunized 20-month-olds in a health maintenance organization (HMO).
Methods. In this randomized trial, underimmunized 20-month-olds identified by the HMO's computerized immunization tracking system were assigned to one of four interventions: 1) an automated telephone message alone; 2) a letter alone; 3) an automated telephone message followed by a letter 1 week later; and 4) a letter followed by an automated telephone message 1 week later. The primary outcome was receipt of any needed immunization by 24 months of age. Decision analysis was used to evaluate the projected cost-effectiveness of the alternative strategies.
Results. A total of 648 children were randomized. A letter followed by a telephone message (58% immunized) was significantly better than either a letter alone (44% immunized) or a telephone message alone (44% immunized). A telephone message followed by a letter (53% immunized) also was more effective than either alone, although the differences were not statistically significant. Among a similar comparison group that received no systematic intervention, 36% were immunized. The estimated cost per child immunized was $7.00 using letters followed by automated telephone messages, $9.80 using automated telephone messages alone, and $10.50 using letters alone. Under alternative cost assumptions for automated telephone messages and mailed messages, the cost per child immunized ranged from $2.20 to $6.50.
Conclusions. For underimmunized 20-month-olds in this HMO setting, letters followed by automated telephone messages were more effective and cost-effective than either message alone. The cost-effectiveness of automated telephone messages and letters may vary widely depending on the setting, and choices among strategies should be tailored to the populations being served.
- HMO =
- health maintenance organization •
- HEDIS =
- Health Plan Employer Data Information Set •
- CAAPS =
- Customer-Activated Appointment Processing Services
Immunization coverage rates of US preschool children have improved in the past 5 years, but remain far from the 90% goal set in national health objectives for the year 2000.1 2Various interventions to improve immunization delivery have been suggested, but there is limited evidence about their cost effectiveness in actual practice.
It is well-established that mailing reminder or recall messages to patients improves adherence to immunizations and other preventive health practices.3 However, there is limited information about automated telephone messages and whether they should be used instead of or in addition to mailed messages for childhood immunizations. In addition, most immunization reminder research in toddlers has been conducted in low-income populations, which typically have immunization coverage rates of 50% to 75%.7 More than two thirds of the nation's children are privately insured, and increasing numbers of children are being enrolled in managed care plans.13 In these plans, and for pediatricians who are increasingly practicing in large groups, computerized immunization tracking systems are a potentially powerful tool to improve immunization delivery.
A previous randomized trial in our health maintenance organization (HMO) found that letters to families of underimmunized 20-month-olds increased immunizations compared with no intervention.14Another study in public immunization clinics found that automated telephone reminder and recall messages increased immunization visits, and suggested that such an approach would be cost-effective.12 To our knowledge, automated telephone messages for immunizations have not been studied, nor has their cost effectiveness been formally evaluated, among privately insured toddlers. This study's aim was to evaluate the effectiveness and cost effectiveness of sending letters, automated telephone messages, or both to families of underimmunized 20-month-olds in an HMO setting.
METHODS
Overview
In this randomized trial, underimmunized 20-month-olds identified by a computerized immunization tracking system were assigned to one of four intervention groups: 1) an automated telephone message alone; 2) a letter alone; 3) an automated telephone message followed by a letter 1 week later; and 4) a letter followed by an automated telephone message 1 week later. The primary outcome was receipt of any needed immunization(s) by the 24-month birthday.
Study Population
This study included underimmunized 20-month-olds who lived in the residence areas of ten northern California medical centers of the Kaiser Permanente Medical Care Program of Northern California, a nonprofit, group-model HMO. Underimmunized children who turned 20 months old during the study period (September 1996 through January 1997) were identified using the regional Immunization Tracking System following the rules shown in Table 1. These rules were similar to the criteria used in the Health Plan Employer Data Information Set (HEDIS) Version 3.0 quality indicator for childhood immunizations. However, as Table 1 shows, our rules for underimmunization were based on the children's ages when they enrolled in the health plan. This was because some children who enrolled in Kaiser Permanente after 42 days of age had received prior immunizations, and these were not always recorded in the health plan's computerized tracking system.
Definition of Underimmunization Status Among 20-Month-Old Children, Northern California Kaiser Permanente, October 1995 Through January 1996
Interventions
Families of underimmunized children were equally randomized to receive one of four interventions: 1) an automated telephone message alone; 2) a letter alone; 3) an automated telephone message followed by a letter 1 week later; and 4) a letter followed by an automated telephone message 1 week later. The clinics participating in the study were asked to suspend any other efforts to contact these families about their children's immunizations. Messages were sent from the HMO's regional offices during the month after the child's 20-month birthday. The letters were personalized; printed in English, Spanish, and Cantonese; and included a list of which immunizations were needed by 24 months of age. To enable a clean evaluation of the study interventions, when the current study was initiated in September 1996, the participating clinics were asked to temporarily suspend any local efforts to telephone or send letters regarding immunizations to those families scheduled to receive messages as part of the study.
Telephone messages were sent on Tuesdays between 5 pmand 9 pm by the Customer-Activated Appointment Processing Services (CAAPS), an automated telephone message system currently used by Northern California Kaiser Permanente to generate appointment messages to its members. A prerecorded message approximately 1 minute long was sent to each family. It stated that the child was overdue for immunizations, and provided the telephone numbers of the advice/appointment lines at the nearest Kaiser Permanente clinics. The message was personalized to the extent that the child's first name was spoken by software that generated the name from text. The system prompted the listener to choose the language in which the message was to be delivered (English, Spanish, or Cantonese), asked him or her to confirm that the correct family had been reached, and also enabled him to replay the message if desired. The system kept records of the results of each call. Telephone numbers that could not be reached because there was no answer either by a person or an answering machine were called again the following evening. Up to six calls were made each evening.
Comparison Group
The current study did not randomize patients to no intervention because a previous randomized controlled trial in our setting had found that letters increased immunization relative to no intervention.14 However, to estimate the proportion of underimmunized 20-month-olds who would receive immunizations with no intervention, we evaluated a comparison group of similar patients who turned 20 months old during January 1996.
Follow-up Interviews
For 4 months after the study was complete, we continued to randomly assign families of underimmunized 20-month-olds to automated telephone messages and/or letters after the procedure described above. Among families who were assigned to automated telephone messages in this poststudy group, a convenience sample was asked to participate in telephone interviews 2 to 4 weeks after they were sent the messages. The 12-question structured, closed-ended interviews asked parents' opinions of the automated telephone messages and/or letters they had been sent. These interviews were conducted with patients outside of the study group so that the telephone interviews would not affect immunization rates in the study group.
Statistical Methods
We attempted to randomize approximately 160 children to each intervention group. This sample size was expected to have 80% power to detect a 16% difference in the percentage of children receiving any immunization during the 4 months after their families were sent the message(s). The primary analysis classified patients on the basis of intention-to-treat, ie, a family assigned to receive an automated telephone message or letter was analyzed as part of the assigned group regardless of whether our records indicated they received a completed message or letter.
The χ2 test was used to evaluate the association between the message method and the main outcome: receipt of any neeeded immunization between age 20 and 24 months according to the HMO's computerized immunization tracking system. A secondary outcome, being fully immunized by age 24 months, was also analyzed. When the relationship between the message method and the outcome variable was statistically significant (ie, had a P value of .05 or less), pairwise comparisons were conducted using appropriate contrasts of parameters in a logistic regression model.
Cost-effectiveness Analysis
Decision analysis, a quantitative modeling method that evaluates the results of alternative strategies, was used to project the cost effectiveness of the possible interventions.15 The decision tree for this study is shown in Fig1. Assumptions about effectiveness were based on the current study's findings and on other public-domain data about the health plan's population. Cost estimates were based on consultation with Kaiser Permanente administrators and study personnel. Because Kaiser Permanente's costs may differ from those of other settings, sensitivity analyses were conducted using alternative cost assumptions, including estimates from a study of automated telephone messages in public immunization clinic populations.12
Decision tree illustrating possible strategies and outcomes in cost-effectiveness analysis of messages for underimmunized 20-month-olds in an HMO. The square represents the choice among alternative message strategies. Circles represent chance events that are not under the direct control of the decisionmaker. Probabilities in parenthesis are taken from Kaiser Permanente data and from data in the current study (see text).
RESULTS
Of the 752 patients initially randomized, 67 (9%) were excluded because they had a gap in health plan enrollment between 20 and 24 months of age. Because the Kaiser Permanente immunization tracking system does not consistently include information about immunizations given after a patient has left the health plan, these 67 patients were excluded because their follow-up data may have been incomplete. A total of 648 children who were randomized among the four intervention groups had continuous enrollment between 20 and 24 months of age and were included in analyses.
Effectiveness of Message Strategies
Table 2 shows the percentage of underimmunized children who received any needed vaccinations by age 24 months. A single automated telephone message (44% immunized) and a single letter (44% immunized) were equally effective. However, a letter followed by a telephone message (58% immunized) was significantly better than either a letter alone (P = .01) or a telephone message alone (P = .02). A telephone message followed by a letter (53% immunized) also was more effective than either message alone, although the differences only approached statistical significance (P = .09 with letters;P = .10 with phone messages.)
Percent With Any New Vaccination and Percent Fully Immunized by 24 Months of Age Among Children Who Received Letter and/or Phone Immunization Messages, Northern California Kaiser Permanente, October 1995 Through January 1996
As shown in Table 2, the four message methods showed similar patterns using the secondary outcome measure, the percentage of children who became fully immunized by 24 months of age. Although the differences among the four groups were not statistically significant (P = .11), the data suggest that 1) a phone message was similar to a letter, and 2) two messages 1 week apart are better than either a letter or a phone message alone.
There were 219 underimmunized 20-month-olds in the comparison group, who received no systematic regional intervention for immunizations. Of these, 78 (35.6%) received a needed immunization by their 24-month birthday.
Outcomes of Telephone Messages
During the study, 486 families were assigned to be sent automated telephone messages with or without letters. Among these families, 429 (88%) actually received a call that was answered by either a person or an answering machine. Calls to 57 families (12%) resulted in dialing failures (ie, busy signal or no answer) after multiple attempts. Of the 429 calls that were received, 412 (96%) were considered complete in that the message about the child being underimmunized was noted to be delivered by the computerized system. An average of 2.4 calls per family were attempted, and the mean length of answered calls was 96.8 seconds.
Information about whether a person or an answering machine responded was available for only the last month of the study. Among families who received calls, 61% had the call answered by a person (based on a requested touch tone response being given) and 39% had the telephone message successfully left on an answering machine. Among persons who answered calls, 6% chose Spanish and 0.7% chose Cantonese. The respondent confirmed that this was the correct telephone number for the child's family in 93% of calls answered by a person.
Parent Perceptions of Telephone Messages
Of the 101 parents who were sent automated telephone messages and who completed follow-up telephone interviews, 69% remembered receiving the telephone message and most of the rest said that someone else in the family may have received it. Of those who received it, 91% found it very easy or somewhat easy to understand. Most respondents (81%) said they liked the idea of the health plan sending them automated telephone messages, 14% said they disliked the idea, and 5% were neutral. Those who did not most commonly said they did not like getting messages from a computer or dislike the fact that they cannot ask questions during such messages. If offered the choice, 63% of parents said they would prefer to receive immunization messages by telephone as well as letter, 25% would prefer a letter alone, and 11% would prefer a telephone message alone.
Projected Cost-effectiveness
Baseline Analysis
The decision tree in Fig 1 illustrates the possible immunization message strategies: Automated Telephone Messages Alone, Letters Alone, Letters plus automated telephone message, and No Intervention. Table 3 shows the cost and effectiveness assumptions used in the analysis. The baseline analysis assumed that the health plan already owned hardware for automated telephone messages, but would need to invest start-up costs in software programming; these costs were amortized over 10 years in the analysis.
Projected Annual Costs, Effectiveness, and Cost-effectiveness of Alternative Message Strategies for Underimmunized 20-Month-Olds, Kaiser Permanente Medical Care Program, 1997
It was estimated that 30 000 children turn 24 months old in the health plan each year, and that 22% of these are underimmunized according to HEDIS 3.0 criteria (S. Platt, PhD, unpublished data). The current study found that in the comparison group, which received no intervention, 35.6% of children who were underimmunized at 20 months of age received an immunization by 24 months of age. Thus, it was estimated that the families of 34% (22%/[100% − 35.6%]) of 20-month-olds in the health plan, or 10 248 patients, would need to be sent reminders each year.
Relative to No Intervention, the cost per child immunized by 24 months of age was $9.80 using Automated Telephone Messages Alone and $10.50 using Letters Alone. For Letters plus automated telephone message relative to phone messages alone, an additional 1435 children would be immunized for an additional cost of $7824. Thus, the incremental cost effectiveness of adding letters to an existing automated telephone strategy would be $8.30 per child immunized by 24 months of age.
Sensitivity Analyses
The costs of the automated telephone message strategy used in the baseline analysis were substantially higher than estimates from a previous study in a public immunization clinic population.12 Under the public immunization clinic cost assumptions (Table 3), automated telephone messages would cost $2.20 per child immunized. As shown in Table 3, postcards could be substituted for letters at an estimated 16 cents each for mailing and 2 cents each for printing and paper costs. If postcards were equally effective as letters, they would cost $6.50 per child immunized.
DISCUSSION
Main Findings
For underimmunized 20-month-olds in this HMO setting, letters followed by automated telephone messages were more effective and cost-effective than either message alone. In addition, compared with letters alone, automated telephone messages alone were equally effective and more cost-effective.
Comparisons With Other Studies
These findings are unique in comparing the effectiveness of automated telephone messages and letters in privately insured children. The result that automated telephone messages alone work equally well with letters alone is in accord with a previous study that evaluated automated telephone messages and letters to increase adherence to tetanus booster vaccination among adults.16 Another study of public immunization clinic patients found that among children overdue for immunizations during the second year of life, automated telephone recall messages doubled the proportion making an immunization visit from 12% to 25%.12
We are not aware of other published studies that have evaluated the effects of using the two different message approaches in tandem. In this study, the proportion of underimmunized 20-month-olds immunized during the next 4 months increased from 36% with no intervention to 44% with either a letter or telephone message alone to 58% with letter followed by telephone messages. Thus, relative to no intervention, the use of both types of message resulted in a more than double absolute percentage increase in the proportion immunized compared with using either reminder alone (22% vs 8%). There are several possible explanations for this. First, some families who would not have received one type of message due to an incorrect address or telephone number may have received the other type of message when the dual-reminder system was used. Second, some parents may attend to one type of message but not to the other. For example, some families do not routinely open mail promptly and others may not be able to understand an automated telephone message due to language barriers.
Third, it is possible that receiving a second message soon after receiving the initial message reinforces the importance of the message and the recipient's propensity to act. The Health Belief Model, a widely used theory of preventive health behavior, includes cues to action as a determinant of preventive health actions.17 18The dual-reminder approach may serve to increase the salience of the need for immunizations among multiple competing priorities that a parent faces.19 We do not know whether the finding that letters plus telephone messages are more effective than either alone will be generalizable to other types of preventive care, such as mammography or sigmoidoscopy; this would be an important area for future research. It would also be useful to conduct a trial that included two automated telephone messages versus two mailed reminders.
Automated telephone messages have been routinely used by the HMO in this study to remind patients about upcoming clinic appointments, but limited information exists about patients' opinions of such messages when they are used as preventive service reminders. Immunization messages are more complex than appointment reminders both because the information that a child is overdue for shots may be unexpected, and because the message asks the recipient to proactively call the health plan about the issue. Similar to a previous public clinic-based study, most respondents in the current study said they would like to receive telephone messages for immunizations.12 However, 14% did not like being sent the automated telephone messages and 25% said they would prefer to be sent a letter alone.
This study did not randomize patients to no intervention because a previous randomized trial in this population found that letters to families of underimmunized 20-month-olds increased immunizations relative to no intervention.14 In that study, 35% of 20-month-olds overdue for the measles-mumps-rubella vaccine who had no intervention received the vaccine during the next 4 months. The fact that 36% of the no intervention comparison group in the current study received an immunization is consistent with the previous estimate.
Published information on the cost-effectiveness of interventions to improve immunization rates is scant. The projected cost per child immunized in our analysis ranged from $2.20 to $10.50. This cost was low compared with the cost in a study of outreach visits to inner-city families in Los Angeles.20 Among the approaches we studied, the letters plus automated telephone message strategy would be considered the most desirable using standard cost-effectiveness methods because it had what is termed extended dominance: it was both more effective and had a lower average cost per child immunized than less effective strategies.21
The costs of the automated telephone message strategy in this study's setting were higher than those estimated for a public clinic setting, but were realistic for the HMO in this study.12 This underscores the fact that as US health care becomes consolidated into large managed care organizations, the costs of new computer-driven projects may actually increase because they must be integrated with existing hardware and software systems. The cost-effectiveness analysis shows that the optimal immunization message strategies are likely to vary in different settings and depend most on start-up and maintenance costs of hardware and software. The decision analytic model in this study can easily be modified to incorporate alternative assumptions about effectiveness and costs.
Limitations
Various alternative approaches to immunization messages could be adopted. In this study, recall messages were directed only to those families whose 20-month-olds had already missed immunizations. Another possible approach would be to send messages to all patients in advance of their due dates for immunizations. However, this approach would be more costly, and might not be more effective than targeting only those children who are overdue.12
Another alternative strategy would be to include in each mailing a slip that a parent could bring to the injection clinic to obtain immunizations without an appointment. In this study, we did not attempt such an approach because many clinicians believe that parents should telephone the advice nurse to confirm that the child truly needs immunizations and to schedule a preventive clinic visit if one is needed. It is possible that needing to call the advice nurse, or having to wait for an appointment, could have hindered immunization for some children.
Some clinics in this study's HMO have nurses, medical assistants, or clerical staff attempt to call families and schedule appointments for underimmunized children. However, such an approach might be less efficient than letters or automated telephone messages, in part because personnel who can attempt calls tend to be available during weekdays when parents are often not home. Our previous studies have suggested that although recall messages result in increased immunization rates, there is no single, easily correctable barrier to immunization for this population.14
Finally, the measure of cost-effectiveness in this study was dollars per additional child immunized, relative to no intervention. In cost-effectiveness analyses, life-years or quality-adjusted life-years saved are more standard denominators,21 but these cannot be easily calculated for individual shot doses. Alternatively, the analysis could be extended to incorporate savings in the future costs of disease prevented by immunization. Although such an analysis is beyond the scope of this study, one model for vaccine selection that incorporates the value of future disease prevention is being developed.22
CONCLUSIONS
We conclude that for underimmunized 20-month-olds in this HMO setting, letters followed by automated telephone messages were more effective and cost-effective than either message alone. The cost- effectiveness of automated telephone messages and letters may vary widely depending on the setting, and choices among strategies should be tailored to the populations being served.
ACKNOWLEDGMENTS
This study was supported by the Innovation Program of Northern California Kaiser Permanente and by the Vaccine Safety Data-link Project of the Centers for Disease Control and Prevention.
We are grateful to Paul Kander, Mary Gleeson, Ronald Price, and Norman Toy of CAAPS, Jim Blackburn of DIS Consulting, and Margaret Fodor of Fodor Consulting for their invaluable work in designing and implementing the phone message system.
Footnotes
- Received November 3, 1997.
- Accepted December 23, 1997.
- Address correspondence to: Tracy Lieu, MD, MPH, Division of Research, Kaiser Permanente, 3505 Broadway, Oakland, CA 94611.
↵‖ Members of the Immunization Message Study Group are all with Permanente Medical Group, Northern California and all with Departments of Pediatrics unless otherwise noted: Kathleen Martin (Division of Research, Oakland); Hervey Froehlich, MD (Fresno); Steven Gotanda, MD (Fairfield); Spencer Larsen, MD (Hayward); Gail Udkow, MD (Oakland); Wayne Easter, MD (Redwood City); Melvin Burman, MD (Richmond); Michael Applebaum, MD (San Francisco); Paul Phinney, MD (Sacramento); Jeff Miller, MD (Santa Rosa); and Robert Maddox, MD (Vallejo).
REFERENCES
- Copyright © 1998 American Academy of Pediatrics
In this issue
