Objective. To compare and measure the effects and cost-effectiveness of two interventions designed to raise immunization rates.
Settings. Nine primary care sites serving impoverished and middle-class children.
Subjects. Complete birth cohorts (ages 0 to 12 months; n = 3015) from these sites.
Interventions. Two 18-month duration interventions: 1) tracking with outreach [tracking/outreach] to bring underimmunized children to their primary care provider office, and 2) a primary care provider office policy change to identify and reduce missed immunization opportunities (prompting).
Design. Randomized, controlled trial, randomizing within sites using a two-by-two factorial design. Subjects were allocated to one of four study groups: control, prompting only, tracking/outreach only, and combined prompting with tracking/outreach. Outcomes were obtained by blinded chart abstraction.
Measures. Immunization status for age; number of days of delay in immunization; primary care utilization; and rates of screening for occult disease.
Results. Out of 3015 subjects, 274 subjects (9%) transferred out of the participating sites or had incomplete charts and were excluded. The 2741 (91%) remaining subjects were assessed. At baseline, study groups did not differ in age, gender, insurance type, or immunization status. Of the remaining subjects, 63% received Medicaid. Final series-complete immunization coverage levels were: control, 74%; prompting-only, 76%; tracking/outreach-only 95%; and combined tracking/outreach with prompting, 95%. Analysis of variance showed that: 1) tracking/outreach increased immunization rates 20 percentage points; 2) tracking/outreach decreased mean immunization delay 63 days; 3) tracking/outreach increased mean health supervision visits 0.44 visits per child; 4) tracking/outreach increased mean anemia screening 0.17 screenings per child and mean lead screenings 0.12 screenings per child; 5) impact of tracking/outreach was greatest for uninsured and impoverished patients; and 6) the prompting intervention had no impact on the studied outcomes, and its failure was caused by inconsistent use of prompts and failure to vaccinate ill children when prompted.
Using tracking/outreach, the cost per additional child fully immunized was $474. Each $1000 spent on the tracking/outreach intervention resulted in: 2.1 additional fully vaccinated children and 668 fewer child-days of delayed immunization; 4.6 additional health supervision visits and 5.9 additional other visits to the primary care provider; and 1.8 additional anemia screenings and 1.3 additional lead screenings.
Conclusions. Outreach directed toward children not up-to-date on immunizations improves not only immunization status, but also health supervision visit attendance and screening rates. The cost per additional child immunized was high, but should be interpreted in view of the spillover benefits that accompanied improved immunization. Effective means to improve coverage by reducing missed immunization opportunities still need to be identified. immunization, primary care, randomized, controlled trial, missed immunization opportunities, outreach.
- DTP =
- diphtheria, tetanus, and pertussis vaccine •
- ANOVA =
- analysis of variance
Despite record high immunization coverage levels,1 barriers to the timely delivery of immunizations to preschool children cause a 14 percentage point gap between the current series-complete coverage of 76% and the Health People 2000 goal of 90% coverage by 2 years of age. These barriers have been intensively investigated because the measles resurgence of 1989 to 1991, resulting in several consistent findings. First, studies have noted the strong association between low socioeconomic status and underimmunization.2–10 This association is probably mediated through the many factors associated with poverty, such as transportation difficulties and inadequate access to or utilization of primary care. The result is an 11 percentage point gap in immunization coverage levels between impoverished children and those above the poverty level.11
Second, both parents and providers tend to overestimate the immunization status of their children or patients. Most parents of underimmunized children do not know that their child is not up-to-date on immunizations.12–14 Few providers know the immunization status of their patients accurately, operate a practice-based tracking system, or routinely assess the immunization status of their patients.15 When asked, providers overestimate the immunization coverage among their patients.16 ,17 Third, missed opportunities for immunization are associated strongly with immunization status18–23 and with risk of disease.24Although missed opportunities may be difficult to eliminate, their reduction does represent a provider-controlled intervention.25 ,26
Only some of the barriers to immunization are amenable to intervention. Logical barriers to address are parent and provider knowledge about the immunization status of their children and patients, and provider practices relating to missed immunization opportunities. In designing appropriate interventions, we felt that flexibility would be needed to overcome the multiple types of barriers that poverty places between children and their timely receipt of vaccinations. Thus, we chose to have outreach workers facilitate recall of underimmunized children into primary care practices. Because providers are the critical component in the immunization process,27 we chose to base our interventions in the provider offices, rather than centrally in a research office or a health department. Because poverty is a major factor related to inadequate immunization, we chose to set out study among providers caring for impoverished children.
We report findings from a randomized, controlled trial to measure the impact on immunization coverage and utilization of other preventive care services of two interventions designed to improve immunization coverage: 1) a practice-based tracking and outreach program, and 2) a policy change to reduce missed opportunities by prompting physicians when a vaccination is needed for a patient.
This study was approved by the Research Subjects Review Board of the University of Rochester (Rochester, NY) and the respective review boards of the primary care practice settings.
Settings and Subjects
The settings were nine primary care practices in the Rochester area that collectively serve >50% of the city preschool children: two pediatric urban group practices, two family medicine neighborhood health centers, one pediatric neighborhood health center, one hospital-based clinic, and three rural health centers. All practices accepted patients regardless of insurance status. The practices agreed to allow their patients to participate in the study, conduct the interventions in their offices, and provide space in their offices for the outreach worker.
In Monroe County, <1% of immunizations are provided by the health department, and these are primarily for school enterers or for school-aged children. The Monroe County Health Department has a policy of notifying primary care providers in writing when they vaccinate one of his or her patients so that the vaccination record remains intact at the primary care provider office. This notification policy, coupled with low utilization of health department clinics for routine infant and toddler vaccination, provide assurance that the outcomes record reviews capture all vaccinations given to the subjects.
The subjects were all the children in the practices who were born between March 1, 1993 and February 28, 1994. The interventions were conducted over the 18-month period from March 1, 1994 to August 31, 1995. Computer billing or encounter files were used to obtain names and identifiers for the subjects. Children in the inclusion date-of-birth range who transferred into the study practices during the intervention period were included in the study.
Subjects who changed primary care providers to another provider participating in the study were retained in the study and kept their original study group assignment. Subjects who changed to a nonparticipating provider or moved from Monroe County, NY, were excluded from the analyses.
Two interventions were tested. In the first intervention, tracking with outreach (tracking/outreach), lay outreach workers were assigned to one or more study sites and were provided with a list of subjects for whom they were responsible. The outreach workers were recruited from the neighborhoods in which the practices were located; all had some college education, but none had an advanced degree. The outreach workers set up tickler-file immunization tracking systems for their primary care sites and determined the immunization status of their assigned subjects using the medical charts. They did not document their activities in the medical chart.
Using a protocol that was based on the immunization status of the subjects, the outreach workers worked with parents of the underimmunized children in their cohort to bring them to the primary care office. Outreach workers determined immunization status in accordance with each practice's interpretation of the harmonized immunization schedule. The use of postcards and telephone calls was the primary means of recalling underimmunized children to the primary care provider offices. Telephone calls were made during days and evenings. For nonresponding parents, the outreach workers made home visits. The outreach workers received back-up support from Strong Memorial Hospital's Social Work Department in case they required assistance with an unexpected situation, such as encountering child abuse or neglect in the home. The caseload averaged 300 subjects per full-time equivalent outreach worker.
The second intervention was a program to reduce missed immunization opportunities (prompting) in the primary care offices. The program involved: 1) conducting discussions with the practice-site physicians on the epidemiology of missed immunization opportunities and their contribution to underimmunization; 2) having the practices agree to immunize prompting group participants that were not up-to-date at a visit, regardless of the visit type (health supervision, illness, or follow-up); 3) placing a distinct marker on the charts of the children in the prompting group; 4) having the triage nurse assess the immunization status of children in the prompting group on presentation to the practice, and place a “missed opportunity card” on the chart as a prompt, if the child was in need of immunization; and 5) providing bimonthly feedback on provider-specific rates of missed opportunities. The missed opportunity card indicated needed vaccines and requested a valid reason for not immunizing the child if the provider decided not to do so.
The study was a randomized, controlled trial using a two-by-two factorial design to test the impact of the two interventions. Children were allocated randomly into one of four study groups using a computer program. The study groups were: 1) tracking/outreach and prompting, 2) tracking/outreach only, 3) prompting only, and 4) control. Children in the first group received both interventions; children in the second and third groups received single interventions; and the control group received the standard of care, which was neither intervention. Randomization was done within site and siblings were not split between study groups.
Baseline and outcomes data were collected by an independent research information group at the University of Rochester that was not involved in the conduct of the interventions. All data were obtained by medical chart abstraction in which the chart reviewers were blind to study group assignment. Markers in the chart indicating study group assignment were removed before outcomes review. Quality control checks included dual, independent review of 10% of the charts. In all cases, only provider-validated immunization histories were accepted.
The primary outcomes were measures of immunization status, which was either “up-to-date” or “not up-to-date” for age appropriate, series-complete coverage. The immunization schedule we used was the most liberal interpretation of the combination of the recommendations by the American Academy of Pediatrics Committee on Infectious Diseases and the recommendations of the Advisory Committee on Immunization Practices, in conjunction with the specific vaccines in use in the community. A 1-month grace period was allowed before counting a child as not up-to-date. Thus, for example, the “due date” for the fourth diphtheria, tetanus, and pertussis vaccine (DTP) shot would be 18 months, and with a 1-month grace period, a patient would be considered up-to-date until the fourth DTP was delayed past 19 months. Series-complete coverage included age-appropriate coverage with DTP, trivalent oral polio vaccine, measles, mumps, and rubella vaccine, and Haemophilus influenzae type b vaccine immunizations.
The immunization status was determined for each child for each day of the intervention period. Two outcome measures were defined: 1) immunization status on the last day of the intervention period, which was used to calculate the rate of age-appropriate immunization at the end of the study in each group; and 2) the number of days of delay in immunization during the intervention. For an individual patient, this was the cumulative number of days during the intervention that the immunization status for that patient was not up-to-date. For subjects who entered the study after the start of the intervention, the immunization delay was prorated so that all means and estimates pertain to an 18-month basis. Thus, a child entering a practice site 10 months after the study started would contribute days of delay for only an 8-month period.
Additional outcome measures included: 1) the number and type of visits to the primary care provider offices during the intervention period, 2) the number of appointments not kept at the provider offices, and 3) the number of screenings for lead and anemia.
A descriptive and categorizing factor was insurance type. Based on the insurance type of the child, the insurance type was classified into three groups: 1) uninsured for immunization, which included children with no insurance and with insurance that did not cover vaccinations; 2) insured by Medicaid; and 3) insured commercially for immunizations.
χ2 tests and one-way analysis of variance (ANOVA) were used to compare the four study groups with respect to demographics and baseline immunization status. Two-way ANOVA was used to test for effects of each of the interventions on the outcome measures, and for interactions between the interventions.28 Additional analyses added insurance type to determine if intervention effects differed by insurance coverage.
The cost-effectiveness of the tracking/outreach intervention was calculated as the ratio of the cost of the intervention per child, to the impact of the intervention on each outcome measure. The cost of the intervention included outreach worker salaries and fringe benefits, the cost of supervising the outreach workers, and the materials used in their work. The impact of tracking/outreach on outcomes was estimated by the main effect for tracking/outreach in the ANOVA results. As an alternative way to describe cost-effectiveness, we also calculated the total (rather than per child) improvement in outcome measures per $1000 spent on the intervention.
Study Group Characteristics
A total of 3015 infants was enrolled and randomized into the four study groups: 732 into the tracking/outreach-prompting group, 715 into the tracking/outreach-only group, 801 into the prompting-only group, and 767 into the control group. Among those enrolled, 272 (9%) transferred out of the study practices and were not included in the analysis, and two had incomplete medical charts; the remainder had their outcomes assessed. The numbers of subjects completing the study (including outcomes assessment) were: 648 (89%) for the tracking/outreach-prompting group, 630 (88%) for the tracking/outreach group, 744 (93%) for the prompting group, and 719 (94%) for the control group.
Table 1 compares the study groups on their demographic characteristics and on their immunization status measures at study entry. Among all subjects, 63% received Medicaid insurance, 14% were not insured for immunization services, and 23% were insured for immunizations by commercial insurers. At entry into the study, the average cumulative delay in immunizations was 34 days and the point prevalence of age appropriate immunization was 82%. There were no significant differences among the four study groups in age, gender, insurance type, or baseline immunization status. There was a statistically significant difference in racial composition (P = .03), but race was not recorded in the medical charts for almost half of the subjects.
Impact on Immunization Status
The vaccination coverage levels at the end of the study were 95% for the tracking/outreach-prompting group, 95% for the tracking/outreach group, 76% for the prompting group, and 74% for the control group. The tracking/outreach-prompting group accumulated an average of 69 days of delay in immunization per patient (12.6% of the intervention period); the tracking/outreach group accumulated 76 days (13.9% of the intervention period); the prompting group accumulated 133 days (24%); and the control group accumulated 140 days (26%). Differences between the two tracking/outreach groups and the control group were significant (P < .001), but differences between the prompting-only group and the control group were not significant.
ANOVA showed that tracking with outreach increased immunization coverage by 20 percentage points (P < .001), and decreased immunization delay by 63 days (P < .001)(Table 2). The prompting intervention did not have a measurable impact. There was no significant interaction between tracking/outreach and prompting in their effects on these outcomes (P = .40 and .99, respectively). However, the effects did differ significantly by insurance type (Table 3), as effects of tracking/outreach were largest among uninsured children.
Impact on Primary Care
Tracking with outreach increased the mean number of health supervision visits by .44 visits per child, a relative increase of 14% (P < .001); increased primary care visits by 1.0 per child, a relative increase of 11%; increased anemia screenings by .17 screenings per child, a relative increase of 11% (P = .006); and increased lead exposure screening by 0.12 screenings per child, a relative increase of 8% (P = .019). The prompting intervention had no significant impact on primary care visits or screening for occult conditions. Neither intervention had any significant effect on the mean number of appointments not kept (data not shown).
This section describes details of the implementation of the prompting intervention. During the study, 28 351 visits were made to the primary care provider sites. Of the 7794 visits in which the child was not up-to-date at the start of the visit (determined by the outcomes chart review), reminder cards were placed on 1276 (16%). The presence of a reminder card increased the proportion of these visits that resulted in an immunization being administered from 55% without a card to 78% with a card. This impact of cards varied by visit type. For health supervision visits by children in need of vaccination, 94% of those having cards were vaccinated compared with 88% of those without cards. For acute care visits of children in need of vaccination, 33% with a card were vaccinated compared with 9% without a card. For follow-up visits, 62% with a card were vaccinated and 33% without a card were vaccinated. The P values for these comparisons were <.001.
The mean number of missed immunization opportunities during the study period varied by study group. There were 1.8 missed opportunities per child in the combined tracking/outreach-prompting group, 1.9 per child in the tracking/outreach group, 2.3 per child in the prompting group, and 2.2 per child in the control group. The prompting intervention did not reduce missed opportunities; its impact (<.1 per child), as determined through ANOVA, was not statistically significant (P = .75). The tracking/outreach intervention did reduce the mean number of missed opportunities by 0.4 per child (P = .002). This reduction in missed opportunities because of tracking/outreach was mediated through improved vaccination status in the tracking/outreach study groups. Because a missed opportunity is defined as a visit of a vaccine-eligible child to a provider, having fewer children in need of vaccination in the tracking/outreach study groups reduced the mean number of missed opportunities.
Tracking/Outreach Process and Cost-effectiveness
The use of outreach was triggered by the immunization status of the patient. The protocol was staged so that less intensive outreach, such as telephone calls and recall letters, was done in preference to more intensive forms, such as home visits. Of the subjects in the tracking with outreach groups, 27.5% had no attempted telephone contacts; 15.6% had a single contact; 11.8% had two completed contacts; 10.3% had three completed contacts; and the remaining 34.8% had more than three completed contacts. For the same groups, 69.6% had no contacts attempted by letter; 13.4% had a single letter; 6.0% had two letters; and the remaining 11% had more than two letters sent. The trend was similar for home visits. A total of 88.2% had no attempted home visits; 5.9% had a single home visit; 3% had two attempts at home visits; and the remaining 2.9% had more than two attempted home visits.
During four of the home visits, the outreach workers encountered social situations that they felt needed to be reported to Child Protective Services. With assistance from the Division of Social Work, these four cases were reported and subsequently “indicated” by Child Protective Services. Two of the cases resulted in foster care placement for the children.
The cost of the tracking/outreach intervention was $70 290 for the outreach worker effort and materials and $51 120 for their supervision, for a total cost of $121,410. The number of children in the tracking/outreach groups was 1278, so the cost was $95 per child over 18 months, or $5.27 per child per month. Table 2 indicates the improvements in outcomes associated with this cost: 1) a 20 percentage point increase in the proportion of children fully vaccinated at 18 to 30 months of age; 2) reduction by 63 days in mean immunization delay; 3) increase of 0.44 in the mean number of health supervision visits and 1.0 in mean total visits to the primary care provider; and 4) increase of 0.17 in mean number of anemia screenings and 0.12 in lead screenings.
Increasing the proportion of fully vaccinated children in the tracking/outreach groups by 20 percentage points amounted to 256 additional fully vaccinated children at the end of the study, compared with what would have been expected without the intervention. This implies a cost of $474 over 18 months per additional fully vaccinated child. However, this calculation ignores the simultaneous improvements in other outcomes. An alternative is to calculate the amount of improvement in each outcome per dollar spent on the intervention in total, rather than on a per child basis. $1000 spent on the tracking/outreach intervention was estimated to result in: 1) 2.1 additional fully vaccinated children and 668 fewer child-days of delayed immunization; 2) 4.6 additional health supervision visits and 5.9 additional other visits to the primary care provider; and 3) 1.8 additional anemia screenings and 1.3 additional lead screenings, all over 18 months.
This study provides strong evidence that a tracking and outreach program designed to bring underimmunized children to their primary care provider will significantly improve not only immunization coverage, but also other important and measurable aspects of primary care. The outreach efforts were based on immunization status; the improvement in health supervision visit attendance and screening for occult conditions follows from their known association with immunization status,29–32 and is consistent with another study showing a spillover impact of an immunization-targeted intervention.33
This study provides somewhat weaker evidence that physician prompting fails to improve immunization status. The weakness of this evidence is a result of the difficulties in conducting an intervention in which study group allocation was done within primary care provider sites, rather than by primary care provider site. Thus, the sites had to treat subjects differently, depending on their study group allocation.
Despite that limitation in the study design, it still appears that the prompting intervention did not work. The expected effect of contamination across study groups within a primary care site would be to have the intervention conducted among both groups. However, we found that very few potential missed opportunities were used to immunize subjects, regardless of study group allocation—ie, opportunities to vaccinate persisted in both study groups. The modest proportion of vaccination opportunities at acute care visits that were used to vaccinate was consistent with our previous trial that attempted to reduce missed opportunities,34 and with another study that attempted to reduce missed opportunities through standing orders.35 The difference between this study and our previous one was the presence of bimonthly feedback to providers regarding performance of the intervention, the number of missed opportunities, and the visits at which missed opportunities occurred. However, the addition of feedback in this study failed to improve compliance.
Although the study failed to show improvement in immunization coverage attributable to the prompting intervention, the presence of cards indicating the need for vaccination did cause more children to be vaccinated at all visit types. A large part of the failure of this intervention was lack of identification of children in need of vaccination at acute visits. Despite a quality improvement approach using measurement and feedback, the intervention did not result in a meaningful change in provider behavior, perhaps due in part to a lack of change in the “prompter” behavior. These results are consistent with Davis' structured review of interventions to change provider behavior that showed greater impact of reminder systems than of audit and feedback.36 Of the two interventions, the prompting intervention clearly demanded more change in office routine and provider behavior than the tracking/outreach intervention.
Vaccinating ill children has not been an easy task for providers, even although it is safe and efficacious.37 Several studies describe the difficulty of routinely vaccinating children brought to the clinician for an acute condition.15 ,38 Even although very few illnesses are valid contraindications to vaccination, illness visits have some constraints that make “converting” them to vaccination visits difficult,23 despite the theoretical improvements in coverage that could result from doing so.20 ,22 Illness visits tend to be short; parent and provider focus for the visits is usually diagnostic and curative, not preventive; routine determination of vaccination status for acute care visits is seldom systematized (although that was one point of this study); and many parents and clinicians believe that it is not safe to vaccinate an ill child.
The tracking with outreach used in this study was similar to the recall portion of “reminder and recall” systems, which have been shown in a number of studies to improve immunization coverage rates. A recall-based system is one that recalls subjects to the provider only after they are past due for a recommended action, such as immunization, whereas a reminder system is proactive but less specific in target, as it reminds parents of upcoming visits. Recall systems intervene on a smaller portion of the population (those not up-to-date on immunizations, for example), but compared with reminder systems, they require the additional effort of verifying immunization status before the recall—an effort that involves the assessment of the entire population. Thus, one method of making a recall system more efficient is to decrease the cost of conducting assessments—for example, by use of a billing computer-based assessment.
One might expect that a reminder system conditioned on health supervision visits (rather than immunization status) would improve key components of health supervision care. They certainly do improve health supervision visit attendance,39 ,40 but, for example, Campbell's randomized, controlled trial that measured the impact on immunization coverage of a reminder system conditioned on health supervision visit attendance failed to show improvement of immunization status. Thus, it is interesting that this study showed that the converse can be true—that is, a recall system conditioned solely on immunization status improved preventive care attendance rates and key components of health supervision care: lead exposure screening and anemia screening.
A key feature of the tracking with outreach intervention is that it was situated in the providers' offices. This afforded the outreach works access to the medical records that contained immunization histories and contact information, and is in contrast interventions that are located outside of provider offices, such as a central research office or a health department.
We found that the majority of the outreach worker effort was invested in office-based activities, such as making telephone calls, sending reminder letters, remaking broken appointments, and meeting parents of patients in the primary care offices. Relatively few subjects received home visits. However, other social needs, such as home necessities and clothing were also identified, and children were referred for more intensive intervention when needed. In four cases, child abuse or neglect was identified, leading to foster care placement on two occasions.
The cost per additional child completely immunized was high. The secondary benefits associated with the tracking/outreach intervention were substantial, however, and are reflected in the results showing the outcomes per cost spread over an 18-month period. These nonimmunization outcomes were due to a spill over effect of the intervention, because the application of outreach was conditioned on immunization status.
Intervention cost was overestimated by an unknown amount because a portion of the intervention cost was the study-related cost of obtaining information on outreach worker activities to learn how best to implement the intervention and how to balance the work load among the outreach workers. We do not have a measurement of steady state intervention costs. Also, the benefits of the intervention were underestimated because we did not measure additional benefits such as referrals to day care or community social services. However, the Monroe County Health Department contracted with the University of Rochester to take over management of all aspects of the tracking/outreach intervention after the study period. They added more provider sites and were able to increase the average outreach worker case load to approximately 1000 children while sustaining the immunization coverage levels of the participating children above 90%.41 If the same intervention impact is assumed, this would bring the cost per additional child completely vaccinated to $160, and would reduce the cost of other attributable outcomes, such as increased health supervision visits and anemia screenings, proportionately. The cost of tracking/outreach per child is $32 over 18 months, or $1.80 per child per month. This is the best current estimate of the maintenance cost of tracking/outreach. Ultimately, however, it is difficult to determine what an acceptable cost to the health care system would be. To make such a determination, one would need a more complete assessment of the benefits of the intervention and a measure of the costs of providing comprehensive health care to populations with varying degrees of social or medical risk.
Knowing the cost of bringing a child completely up to date on their immunizations helps one compare interventions. Other point estimates for intervention cost have been published recently. Two studies by Lieu and colleagues42 ,43 indicate very low costs (from $4.04–$10.50) per additional child vaccinated using a recall systems based in an health maintenance organization that has a tracking system in place. A Women, Infants, Children's Supplemental Nutrition Program-based approach that links immunization status with the frequency of visits to Women, Infants, Children's Supplemental Nutrition Program clinics (less frequent visits for fully vaccinated children) had an estimated cost of $38.00 per additional child vaccinated.44 A recent study by Wood and colleagues on a centrally based outreach intervention showed a cost of $12 022 per additional child vaccinated.45–47 Although both our study and Wood's study used outreach workers, there were two differences between them that may relate to the large differences in cost-effectiveness. Ours used provider-based outreach rather than a centrally based outreach; and our outreach protocol used home visits only as a last resort to bring in a recalcitrant patient, rather than routine part of the protocol (four home visits per child).
In addition to the limitation mentioned above of the randomization frame for the prompting intervention, some additional limitations apply to the interpretation of these results. With regard to internal validity, it should be noted that the use of outreach worker-based tracking systems makes it more likely to be able to determine the outcomes among the tracking with outreach arms of the study. We attempted to minimize this ascertainment bias by 1) not having the outreach workers document their activities in the medical record, and 2) using outcome assessors who were blind to study hypotheses and study group assignment.
With regard to external validity, several points are relevant. First, the study was conducted in Rochester, NY, a community with relatively high immunization coverage levels. However, the control group's immunization coverage level was similar to current, national estimates of coverage (76%, but measured with less stringent criteria48), indicating that the baseline coverage level in this study is applicable to the current national situation. Second, the effective intervention in this study (tracking with outreach) was implemented at the primary care sites—not centrally from a public health department or research office. This facet of the implementation was important for two reasons: 1) it demonstrated a degree of acceptance and support from the practices involved in the study, and 2) it greatly simplified the tracking by making the medical charts readily available. Thus, our study results may not be applicable to a centrally operated tracking and outreach program. And, third, the increase in immunization coverage due to a similar intervention is likely to vary by baseline value. For example, this study demonstrated a 20 percentage point improvement in immunization coverage from a baseline coverage of between 70% to 75%. We speculate that a larger improvement in coverage would be realized by the same intervention if the community baseline coverage levels were lower, but more work would be needed because more patients would need to be contacted.
The results of this study support several recommendations. First, primary care providers should consider implementing a recall system that brings children in their practice who are falling behind in their immunizations in for preventive care. There is sufficient evidence in the literature on the effectiveness of this type of intervention to support such a recommendation, and the added incentive of improved preventive care attendance and screening for occult diseases helps justify implementing a recall system. Policy makers should consider supporting tracking with outreach for children receiving publicly funded health care; however, the high cost per child completely vaccinated requires additional research to discover less expensive models for recall systems than outreach worker-based recall systems. Lieu's studies in the health maintenance organization setting, mentioned previously, represent models to emulate.42 ,43
Additional research is also recommended. First, studies are needed to determine methods of reducing illness visit missed immunization opportunities. A different study design than a within-site randomization is warranted—perhaps either a by-site randomization, which would be expensive because of the large number of practices required, or a before-and-after study design. Second, studies are needed to investigate further the impact of immunization status-driven interventions on primary care utilization and child health status. Expansion of these research efforts to the adolescent adult populations is indicated.
This project was supported by the Centers for Disease Control and Prevention (Contract No. 200–90–0869).
We thank the physicians, nurses, and staff at the participating primary care sites. We thank Martha Sandler for her excellence in social work supervision and consultation.
- Received July 29, 1997.
- Accepted June 4, 1998.
Reprints request to (L.E.R.) National Immunization Program, Centers for Disease Control and Prevention; 1600 Clifton Rd, NE, Mailstop E-52; Atlanta, GA 30333.
- Williams I,
- Milton J,
- Farrell J,
- Graham N
- Bobo J,
- Gale J,
- Thapa P,
- Wassilak S
- Wood D,
- Donald-Sherbourne C,
- Halfon N,
- et al.
- Miller LA,
- Hoffman RE,
- Baron AE,
- Marine WM,
- Melinkovich P
- Zell E,
- Peak R,
- Rodewald L,
- Ezzati-Rice T
- ↵Suarez L, Simpson D, Smith D. Errors and correlates in parental recall of child immunizations: effects on vaccination coverage estimates. Pediatrics. 1997;99(5). URL: http://www.pediatrics.org/cgi/content/full/99/5/e3
- Szilagyi PG,
- Rodewald LE,
- Humiston SG,
- et al.
- ↵Watt J, Kahane S, Smith N, et al. The difference between measured and estimated vaccination coverage among private physicians in California. Presented before the Ambulatory Pediatric Association, May 1, 1998; New Orleans, LA
- Guyer B,
- Hughart N,
- Holt E,
- et al.
- McConnochie K,
- Roghmann K
- Szilagyi P,
- Rodewald L,
- Humiston S,
- et al.
- ↵Wood D, Pereyra M, Halfon N, Hamlin J, Grabowsky M. Vaccination levels in Los Angeles public health centers: the contribution of missed opportunities to vaccinate and other factors. Am J Public Health. 1995;:85:850–853
- Holt E,
- Guyer B,
- Hughart N,
- et al.
- Hutchins S,
- Escolan J,
- Markowitz, et al
- ↵Armitage P, Berry G. Statistical Methods in Medical Research. 2nd ed. Oxford, England: Blackwell Scientific Publications; 1987:227–239
- Bordley W,
- Margolis P,
- Lannon C
- Fairbrother G,
- Friedman S,
- DuMont K,
- Lobach K
- ↵Adams W, Geva J, Coffman J, Palfrey S. Anemia and elevated lead levels in underimmunized inner-city children. Pediatrics. 1998;101(3). URL: http://www.pediatrics.org/cgi/content/full/101/3/e6
- Campbell JR,
- Szilagyi PG,
- Rodewald LE,
- Doane C,
- Roghmann KJ
- Szilagyi PG,
- Humiston SG,
- Rodewald LE,
- Kraus RM,
- Hillengas M,
- Barth RD
- ↵Lieu TA, Capra AM, Makol J, Black SB, Shinefield HR. Effectiveness and cost-effectiveness of letters, automated telephone messages, or both for underimmunized children in a health maintenance organization. Pediatrics. 1998;101(4). URL: http://www.pediatrics.org/cgi/content/full/101/4/e3
- ↵Hutchins SS, Rosenthal J. Results from WIC demonstration projects. Proceedings of the 29th National Immunization Conference. Washington, DC: Department of Health and Human Services; 1994:1–4
- Schuster MA,
- Wood DL,
- Duan N,
- Mazel RM,
- Sherbourne CD,
- Halfon N
- Copyright © 1999 American Academy of Pediatrics