PEDIATRICS Vol. 103 No. 4 April 1999, pp. 889-897

Immunization Performance Measurement in a Changing Immunization Environment

Lance Rodewald, MD^*, Edmond Maes, PhD^*, John Stevenson, MA^*, Bridget Lyons, BS^*, Shannon Stokley, MPH^*, and Peter Szilagyi, MD, MPH

From the ^* National Immunization Program, Centers for Disease Control and Prevention, Atlanta, Georgia; and the  Department of Pediatrics, University of Rochester, Rochester, New York.

	ABSTRACT

Top Abstract Conclusion References

Objective.  The measurement of performance in the delivery of recommended vaccinations for children is used frequently as a marker for quality of care and as an outcome for studies of interventions to improve immunization coverage levels. The critical element of immunization performance measurement is the determination of immunization status. This methodologic review 1) discusses immunization status as a measure of quality of primary care for children, 2) describes immunization status measures used in immunization intervention studies, and 3) examines selected technical issues of immunization status measurement.

Methods and Topics.  1) Description of the characteristics of immunization status measurements obtained by a systematic review of studies published between 1980 and 1997 on interventions to raise immunization coverage, and 2) illustration of technical considerations for immunization status measurement using one local database and one national database of immunization histories. Technical issues for immunization status measurement include 1) the need to use documented immunization histories rather than parental recall to determine immunization status, 2) the need to link records across providers to obtain complete records, 3) the sensitivity of immunization status to missing immunization data, and 4) the potential of measures incorporating combinations of immunizations to underestimate the degree of vaccination in a population.

Conclusions.  Immunization performance measurement has many characteristics of a robust quality of care measure, including high acceptance by primary care providers of routine vaccination, association of immunization status with the conduct of other clinical preventive services, agreed-on technical and programmatic standards of care, and legislative requirements for medical record documentation. However, it is not without challenges. Careful attention to technical issues has potential to improve immunization delivery health services research.  Key words:  immunization, vaccination, performance measurement, quality of care, children.

The President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry stated in 1998 that "the purpose of the health care system must be to continuously reduce the impact and burden of illness, injury, and disability, and to improve the health and functioning of the people of the United States."¹ Consistent with this viewpoint, outcome-oriented goals in child health define the ultimate test for the adequacy of efforts to improve child health care. The numerous disease-related objectives in Healthy People 2000 and the proposed Healthy People 2010 provide additional emphasis on outcomes.²

However, serious adverse health outcomes are infrequent events in the general population of children, making the testing of interventions to improve child health difficult and expensive to conduct when the primary outcome measure is reduction of disease morbidity or mortality. Health services research studies of modest size testing innovative interventions seldom have adequate power to detect clinically meaningful differences in disease incidence, health status, or morbidity.

Process measures are frequently used as intermediate measures to overcome limitations in study power to detect differences in disease outcomes. The relative ease of demonstrating improvement in the process of health care (compared with outcomes) provides substantial advantage, albeit with the need to infer indirectly from process to outcome. For immunizations, this inference is on solid ground because the process of vaccination is highly associated with both protection from vaccine-preventable disease (VPD) and reduction of VPD rates.³ This strong link between process and outcome makes immunization status one of the most commonly used measures of child health status and the adequacy of preventive care.

Despite the existence of an accepted immunization schedule, no standardized measures exist for program evaluation or health services research related to childhood immunization. Several factors account for variation in immunization status measures, such as 1) the vaccines included in combination measures, 2) the timing and definitions of "due" and "overdue" for vaccination, 3) the age or age range used to calculate immunization status, and 4) the sources and validation of immunization histories. This variation makes selection of immunization status measures for studies difficult and complicates interpretation of study results and comparison across studies.

The purpose of this methodology article is to 1) discuss immunization status as a measure of quality of primary care for children, 2) describe immunization status measures used in studies testing the impact of interventions to raise childhood vaccination coverage levels, and 3) examine key technical issues of immunization status measurement.

	IMMUNIZATION STATUS AND QUALITY OF CARE

Routine vaccination is the most cost-effective clinical preventive service for children, saving both lives and dollars.⁴ Universal childhood vaccination has wide acceptance among primary care providers, with an accepted technical standard of care that is endorsed by the three major recommending groups: the federal Advisory Committee on Immunization Practices, the American Academy of Pediatrics' Committee on Infectious Diseases, and the American Academy of Family Physicians. This standard is the routine immunization schedule, and it has been harmonized among the three recommending groups since 1994, thus reducing confusion about the acceptable timing of vaccinations. Because of the licensing of new vaccines and new scientific information about vaccine safety and efficacy, the harmonized schedule changes frequently and is updated formally once per year. Changes in the schedule challenge researchers by making the standard a moving target.

An additional set of standards of care exists for the delivery of childhood vaccination: The Standards for Pediatric Immunization Practices.⁵ These standards also are endorsed by the major provider groups; they describe recommended practices for delivery of vaccinations, including vaccinating at all visit types, reducing cost barriers, operating recall and reminder systems, routine assessment of coverage, and handling vaccines properly.

Because these standards are detailed and prescriptive, they make useful benchmarks against which to measure the quality of services delivered. Additionally, federal law requires documentation in the provider's medical chart of vaccines in the National Childhood Vaccine Injury Compensation Program (which includes all the vaccines recommended for universal use), and the Vaccines for Children program (VFC) requires medical record documentation of VFC eligibility status. These reporting requirements have meant that vaccines delivered by a provider generally can be found by medical chart review. Providers frequently have a designated page in their medical charts to document the vaccines and their dates of administration, additionally simplifying chart review.

The combination of agreed-on standards and legislatively required documentation ensures the feasibility of measuring the adherence to these quality standards. Furthermore, primary care providers generally regard deviation from the quality standards as unsatisfactory care. These attributes make performance measurement of immunizations popular among those with a need to assess quality of care. For example, immunization performance is a prominent part of the effectiveness of care portion of the Healthplan Employer Data and Information Set (HEDIS) that measures the quality of service delivery for managed care organizations;⁶ almost all the Child Health Insurance Program plans will use immunization as a quality of care measure; and national-level vaccination coverage is a widely cited Healthy People 2000 objective.

Although immunization is important by itself, a number of studies have shown an association of the adequacy of immunization delivery with the adequacy of the delivery of other routinely recommended clinical preventive services and with adherence to the number of recommended health supervision visits by children. Compared with children who are up to date (UTD) on vaccinations, underimmunized children have much lower rates of screening for lead exposure and anemia and have fewer health supervision visits.^7-10 Furthermore, children not UTD with immunizations have higher rates of anemia or lead-exposure than completely vaccinated children.¹¹ Interventions that cause underimmunized children to go to their primary care provider improve not only immunization coverage but also the performance of other clinical preventive services.^12-14

In summary, agreed-on standards for a service that is recommended and widely accepted for universal implementation and associated with the conduct of other health supervision activities make immunization performance measurement a robust indicator of quality of preventive care. To fulfill the promise of a high-quality measure, however, the use of immunization measurement and the technical aspects of measurement deserve close attention.

	IMMUNIZATION STATUS MEASUREMENT: 1980-1997

To describe the characteristics of immunization measures used in the childhood health services research literature, we conducted a systematic review of literature on studies of interventions to raise immunization coverage levels. We reviewed studies that were accepted by the Task Force for Community Preventive Services to make recommendations about techniques to raise community immunization coverage levels. The Task Force, an independent advisory body to the Department of Health and Human Services, makes recommendations on methods to improve community health to public health officials, providers, managed care organizations, and other policy- and decision-makers.¹⁵

Immunization studies accepted by the Task Force met the following characteristics: publication or presentation between 1980 and 1997, in the English language and conducted in a developed country, testing an intervention to raise coverage levels, and having a control group. The Task Force rated each study on the fit between the design and the hypothesis and on the adequacy of the study execution.¹⁶ The studies having the least appropriate design for the hypothesis and the weakest execution were excluded from consideration. For this article, we restricted further our analysis to the 48 studies concerning children that had a primary outcome measure of immunization coverage.^1417-63 By using the Task Force-accepted studies, we were able to review good-quality studies, but we were limited in our review to intervention studies. Studies on barriers to immunization, for example, were not included in the review, although most of them include an assessment of immunization status.

Each study was reviewed to determine 1) the date the study was conducted; 2) the vaccines assessed; 3) which combination(s) of vaccines was reported; 4) whether individual vaccination status was reported; 5) the actual measure used (eg, age-appropriate, UTD, of a combination); 6) whether a "grace period" was allowed before counting a child delayed in immunizations; 7) whether minimum spacing intervals were used to determine which doses are valid (eg, explicitly not counting a fourth diphtheria, tetanus, and pertussis (DTP) if given <6 months after the third DTP); 8) whether minimum age criteria were used to determine which doses are valid (eg, not counting measles, mumps, and rubella (MMR) given before first birthday); 9) the age range of the study participants; 10) the data source (parent recall, provider records, immunization registry, shot cards, school or day care records, or billing files); and 11) whether parents were contacted to identify the sources of immunization for their child.

Table 1 shows the characteristics of the studies reviewed. None of the studies used parental recall as the source of immunization data, whereas most relied on provider records or immunization registries without parent contact to identify all immunization providers. The vast majority of studies used UTD immunization status measures rather than age-appropriate vaccination status. Few studies used minimum spacing intervals or minimum age criteria to determine whether to count administered doses as valid.

New vaccines tended to be excluded from combination measures. There were 24 studies conducted after 1990 that measured coverage of at least two vaccines. Of those studies, 11 (46%) included Haemophilus influenzae (Hib) vaccine coverage in the measured outcomes. There were 12 studies conducted after 1992 that included at least two vaccines, and only 2 (17%) included hepatitis B (HBV) vaccine. No studies included varicella vaccine. Combined-series measures were used frequently, with a total of 24 different combination series measures used. The most common combined series was the four DTP, three polio, and one MMR combination series (eight studies), followed by a combination with three DTP and two polio vaccinations (six studies).

	KEY TECHNICAL ISSUES

This section describes measurement issues to consider when planning an investigation. Measurement issues include the immunization status function, data sources, the relationship between missing data and misclassification, and the use of combination series measures. An important consideration that is beyond the scope of the article is sample frame for provider-based immunization performance measurement. Darden and Taylor have reviewed this topic recently.^64,65

Characteristics of Immunization Status Measures

The two most commonly used immunization status measures are "age-appropriate" and "up-to-date." Age-appropriate is applied to a population or cohort of children that varies in age, and the measure indicates the proportion of children who have adequate vaccination for their age.⁶⁶ UTD generally indicates the proportion of children in a population with adequate immunizations at a certain age threshold or age range. With the current emphasis on the Healthy People 2000 objectives, UTD usually means the proportion UTD at 24 months of age. UTD measures also can have a range of ages rather than an age threshold at which immunization status is calculated. For example, the National Immunization Survey and the National Health Interview Survey (NHIS) report the immunization status of children 19 to 35 months of age as the proportion UTD at the time of the interview.

Immunization status measures use an immunization status function that determines the immunization status of an individual child. This function relies on an immunization schedule, an individual's immunization history, an age for calculation, and a "grace period" (during which a child may be vaccine-eligible but not "past due") to determine the individual's immunization status. The immunization schedule is a vector of pairs of vaccinations and their recommended ages of administration; the immunization record is a vector of pairs of vaccinations and the actual ages of administration; and the immunization status usually is a categorical variableeither UTD or not UTD at the given age, using the given grace period. Many studies use combinations of vaccines in the schedule when calculating immunization status. Because underimmunization is the condition being measured, it sometimes is convenient to use the inverse of immunization status, the underimmunization status, as the primary measure (Fig 1).

View larger version (10K): [in this window] [in a new window]   Fig. 1.   Immunization status function for an individual, showing the relation between the schedule, the administration age for vaccines, and immunization status.

Depending on the purpose for which the status function is used, the schedule can be more complex than a simple vector of vaccination and age pairs. For example, most of the vaccine recommendations have a minimum spacing requirement that, if violated, invalidates at least one of the doses administered. These spacing requirements may have clinical importance, but comparisons between study groups frequently ignore them. For example, Table 1 shows that only 15% of the intervention studies used minimum spacing intervals to invalidate administered doses. In terms of coverage estimates, however, the consequence of ignoring or using the minimum spacing intervals or minimum age recommendations may be substantial. Vivier and colleagues demonstrated a 27 percentage point difference (50% to 77%) in UTD combined-series coverage when minimum spacing intervals and minimum age criteria were adhered to, compared with a calculation that ignored interval and age criteria.⁶⁷

An additional complication is that the harmonized immunization schedule recommends age ranges for many vaccines, rather than specific ages. Thus, acceptable practice can vary from provider to provider. The Advisory Committee on Immunization Practices is currently studying methods to standardize immunization status algorithms. The Centers for Disease Control and Prevention has developed a standardized methodology and computer program to determine the immunization coverage level of a clinic. This Clinic Assessment Software Application is available at http://www.cdc.gov/nip/casa. This application has provided the basis for several published studies,^61,21 and is the standard method for assessing health department clinics.

When immunization status is summarized for a population, age-appropriate and UTD vaccination status have different characteristics. Figure 2 shows the nonlinear relationship between age and age-appropriate immunization status for a hypothetic population. Age-appropriate measures can have more variation in the ages of the study participants than can UTD measures. For example, age-appropriate coverage has been applied to the lifetime immunization history of school enterers⁶⁸ and emergency department patients.⁶⁹ In contrast, UTD measures tend to have narrow age ranges and are best suited to cross-sectional studies.

View larger version (6K): [in this window] [in a new window]   Fig. 2.   Immunization status for a populations using an age-appropriate measure. The percent of a hypothetic population that is UTD for age is shown.

Because of the emphasis on reducing the delay in immunization, some studies have used a direct measure of days of delay. Days of delay relates closely to age-appropriate vaccination coverage because it uses age-appropriate status in the calculation. Using the immunization status function, the days of delay in immunization is the sum of the number of days during the study period in which the subject is not UTD (Fig 3). This number can be divided by the subject's age or by the duration of an intervention to calculate a proportion indicating the relative time spent unprotected by vaccination. However, as a proportion, days of delay is difficult to interpret. Comparison across studies also is difficult using days of delay, but the measure has some intuitive appeal because it represents the quantity of time spent incompletely protected from VPD.

View larger version (10K): [in this window] [in a new window]   Fig. 3.   Days of delay function for an individual showing the relation between the schedule, the administration age for vaccine, the immunization status, and the cumulative number of days of delay in vaccination.

Data Sources: Parents, Charts, and Hand-held Records

One might assume that the gold standard for immunization status would be serologic tests of immunity, but serology falls short on a number of counts. First, it may not distinguish between disease and vaccination. Second, it cannot determine the number of doses of a vaccine if the number is >1. Third, serology may be negative where the antibody is no longer present despite adequate immunization. Fourth, no serologic test for immunity exists for some vaccines, such as pertussis.

The current gold standard for measuring the process of vaccination uses parent-linked and provider-validated immunization status. Parent-linked means asking parents to name all immunization providers for their child; provider-validated means obtaining the immunization records for each child and combining them into a single record. The CDC's National Immunization Survey⁷⁰ and the immunization component for 19- to 35-month-old children in the NHIS⁷¹ currently use this method. The major disadvantage of the method is that the researcher must obtain information from at least two sources (the parent and the provider), a process more time-consuming and expensive than simple chart review.

Multiple studies document the inaccuracy of parents as the sole source of immunization histories.^72-75 For example, 78% of parents of underimmunized children in the 1994 NHIS immunization supplement incorrectly thought that their child was UTD.⁷⁶ In the same survey, half of the children whose parents said received no vaccinations actually had complete immunizations. The complex and ever-changing immunization schedule most likely explains much of the inaccuracy of parental recall. It is difficult for providers to keep up with the schedule, even more so for parents.

The single provider record check method lies intermediate between parental recall of vaccination and a parent-linked, provider-verified measure. The validity of this method depends on the frequency and reliability with which medical records include immunizations administered in other sites, such as health department clinics or other primary care providers. Because the proportion of vaccinations given at health department clinics varies geographically, by specialty type,⁷⁷ and by insurance type, the accuracy of this method depends on factors specific to any given study and must be determined individually.

To illustrate the impact of scattered immunization records, we analyzed the 1995 NHIS Provider Record Check Study, comparing the number and type of vaccinations recorded in the chart of the most recent immunization provider and the first immunization provider with the number and type of vaccinations recorded by all providers for children having more than one immunization provider. Immunization histories of children 19 to 35 months of age collected during the 1995 NHIS were verified by the providers reported to have administered the child's vaccines. Up to three providers were contacted per child. The providers were asked to report all immunizations in their records for the child surveyed, regardless of whether they or another provider administered the vaccinations. We restricted our analysis to children whose parents reported more than one immunization provider.

Of the 1352 children 19 to 35 months of age in the 1995 NHIS who had provider-verified immunization histories, a total of 304 (22%) had more than one provider. Table 2 shows impact of missing immunization histories. Depending on the specific vaccine, the immunization coverage levels determined by using only immunizations recorded by the most recent provider were from 9.6 to 13.4 percentage points lower than the coverage levels determined by using all provider-verified immunizations. This discrepancy is similar in magnitude to the effect sizes of many impact studies.¹⁶ Furthermore, this measure of impact on record scattering is a conservative estimate because the researcher conducting medical chart reviews may not know whether the provider is the most recent immunization provider. If the first immunization provider record is used, the discrepancy between true coverage and measured coverage is as high as 35 percentage points. Yawn and colleagues showed similar discrepancies attributable to record scattering.⁷⁸

Data Completeness and Misclassification

Routine immunization covers a series of events rather than an individual event. For example, recommendations include a minimum of 15 vaccinations during the first 2 years of life. If one uses a combination series-complete measure of immunization status, a child missing at least one vaccination is considered not UTDan extreme dichotomization of immunization status that is seen commonly in the literature. Because missing data are indistinguishable from missing vaccinations, a single missing datum will misclassify an UTD child as not UTD. The direction of the error introduced by missing data is to decrease measured coverage below true coverage. The magnitude of the error is much greater than generally appreciated.

To illustrate the biasing effect of missing data on coverage estimates, one can consider a population of 2-year-old children completely UTD. If the dataset includes all immunizations with no missing data, the measured coverage and the true coverage would both equal 100%. However, if missing immunizations are independently and randomly distributed throughout the dataset, then the misclassification rate is equal to 1 minus the data completeness rate raised to the number of vaccinations required for UTD status. For a 15-vaccination series, when 1% of the vaccinations are missing (ie, the data are 99% complete), the proportion of children UTD who are misclassified as not UTD is (1 .99¹⁵) = 14%. Thus, only 1% missing data will misclassify 14% of UTD children. If 2% of the data are missing, 26% of the population will be misclassified; 3% missing data cause 37% to be misclassified; 5% missing data cause greater than half (54%) of the UTD population to be misclassified. If missing data are assumed to cluster around vaccination visits, the exponent becomes 7 (for the seven vaccination visits) instead of 15, but the resulting relationship still shows a dramatic rise in misclassification with missing data (1% missing data cause 7% to be misclassified; 2%  13%; 5%  30%; 10%  52%).

This sensitivity of series-complete coverage to missing data can jeopardize the validity of conclusions drawn from comparisons of study groups that differ only modestly in terms of missing vaccine information. For example, consider the situation in which one arm of an intervention trial involves tracking immunization status (eg, a test of a recall system) and the control arm does not track immunizations. The data collection might bias the outcomes if the intervention arm documents in the medical record vaccinations given elsewhere whereas the control arm does not. The exponential relation between misclassification and missing data then would magnify the introduced bias. This particular problem arises because many immunization interventions involve some method of gathering, tracking, or assessing vaccination records.

Measures of Complete Combination Series Versus Individual Vaccine Measures

Many intervention studies report vaccination coverage as combination series-complete measures that combine into a single status measure all the vaccines for which the study subject was eligible. Because the epidemiology for each disease in the combination measure differs, the appropriate disease prevention strategies must be tailored to balance effectiveness with cost, feasibility, and safety. Ideally, immunization status measures should align with currently accepted disease prevention strategies (eg, eradication of polio; elimination of measles, rubella, and invasive Haemophilus influenzae disease; and control of mumps, pertussis, diphtheria, tetanus, varicella, HBV, and rotavirus). However, when immunization coverage is reported as a combination series measure, the coverage of specific vaccines cannot be determined, which in turn creates a problem in the translation of the process of vaccination into the outcome of protection from VPD. Single-vaccine measures can be linked more easily to their corresponding diseases, the strategies for their prevention, and their incidence goals for the year 2010 than can combination measures.

The two primary problems with combined-series measures are 1) the historical relegation of new vaccines to the status of "second-class citizens" because of failure to become part of generally accepted series, and 2) the nearly complete uncoupling of disease prevention strategies and goals from series measures that results from expansion of small series, such as the 4:3:1 series into more comprehensive series, such as the 4:3:1:3:3:1 series. This section presents the pros and cons of using combined-series measures instead of single-vaccine measures.

Combined-series measures tend to exclude newer vaccines. Hib vaccine was recommended for routine administration more than a decade ago; however, less than half of the studies in the past decade that use combination-series measures included Hib vaccine. Failure of the commonly used combined series to include HBV and varicella vaccines may have had the effect of taking the pressure off evaluating their individual performances. With such low incidences of the diseases covered by the 4:3:1 series, the new vaccines, such as varicella and rotavirus, likely provide the most important opportunities for reducing the burden of VPD.

The combined-series measure can be misleading, because the vaccine with the lowest coverage dominates the results, primarily reflecting the lowest coverage of a single vaccine in the series. For example, the current 78% coverage rate for the 4:3:1 series primarily represents the coverage level for the fourth dose of DTaP (81%), the vaccine with the lowest coverage in the series. Coverage for each of the other doses is 90% or greater. Adding varicella coverage to the series today would lower the coverage rate for the combined series from 78% to <26%.⁷⁹ The misleading nature of long series of combination measures will increase with the introduction of newly recommended vaccines into the series.

Combined-series coverage levels do not correspond well with degree of population protection from VPD. The combined-series measure ignores the fact that most underimmunized children have received most of their vaccines. Combination measures count a child as underimmunized whether he or she is missing one dose of one vaccine or all vaccines. To illustrate this effect, we reanalyzed a set of immunization histories obtained for the control group of an intervention trial described elsewhere.¹⁴ Although the combination-series complete coverage using DTP, polio, MMR, and Hib vaccines was 74.7% for this group, 93.7% of the recommended vaccines actually were administered to the subjects (Fig 4). When restricting the analysis to children not UTD for the combination of vaccines, one can identify the vaccines administered to children who were vaccinated incompletely. Greater than half of the not UTD children had received MMR vaccination, greater than half had received three or four DTP vaccines, and greater than half had received three HBV vaccines. Only 6.3% of the children who were not UTD for the combination series (1.6% of the study population) had received no vaccination at all.

View larger version (32K): [in this window] [in a new window]   Fig. 4.   Vaccination performance for a cohort of children (n = 719) in the control group of an immunization intervention trial14 comparing performance by child and by vaccination. The left pie chart shows a child-level analysis defining UTD as coverage with DTP, polio, MMR, and Hib vaccines. The right pie chart shows a vaccine-level analysis for the same cohort of children.

Combination measures do not pinpoint problems with specific vaccines and therefore fail to identify specific prevention opportunities. For example, the quickest way to improve the 4:3:1 coverage might be to improve DTaP4 coverage among children missing only the fourth (and least important) dose of DTaP. However, better prevention opportunities may be found among severely underimmunized vulnerable populations at risk for many VPD or with underutilized new vaccines.

The combined-series measure may not be valid for historical or cross-national comparisons. The current combined-series measures includes polio vaccination, which will be discontinued after certifying the world as polio-free. Thus, the 4:3:1 measure will become a "4:1" measure in a few years, and after measles eradication it will be a "4" measure. Other developed countries report vaccination coverage by individual vaccine.

Combination measures, however, have certain advantages, primarily the convenience and simplicity of a single-number interpretation of vaccination coverage that is familiar to researchers, providers, nonhealth care professionals, and policy-makers. For example, HEDIS uses a series measure, rather than individual vaccines. This use of a combined measure also reflects the clinical interpretation that would be used in a recall system. Underimmunized children normally would be recalled if they needed any recommended vaccines.

The primary disadvantage of a set of individual vaccine measures is the confusion resulting from too much information. Coverage rates of generally accepted vaccines are closely correlated, and reporting many similar numbers measuring similar processes may confuse the consumer of the information. It is encouraging, however, that greater than half of the intervention studies reviewed reported single-vaccine coverage as an outcome.

	CONCLUSIONS

Top Abstract Conclusion References

Child health services research commonly uses immunization performance as a measure of quality of care. Although it has many qualities of a robust measure, such as high acceptability of vaccination by primary care providers, association of immunization status with the conduct of other clinical preventive services, agreed-on technical and programmatic standards of care, and legislative requirements for medical record documentation, its use provides challenges. Key technical issues include using only provider-validated or hand-held immunization records rather than parental recall, linking records across providers in settings with multiple providers per child, and using single-vaccine measures when possible. The most difficult challenge for immunization performance measurement is the exponential relationship between missing immunization data and misclassification of immunization status. Because some study designs can introduce ascertainment bias, the tendency of series measures to magnify the impact of missing immunization data will, in turn, magnify the ascertainment bias. Careful attention to technical issues can improve the accuracy of immunization health services research and evaluations.

Implications

The gold standard for immunization status measurementthe parent-linked, provider-validated measureis expensive. It corresponds most closely to true vaccination coverage level because it is the measure least susceptible to missing data caused by record-scattering or incomplete records. Selection of immunization status measures has to be individualized to the purpose and design of the study and to the availability of resources. Measured immunization coverage levels in studies set in communities where children receive immunizations from multiple sources of care must be interpreted cautiously if linked records are not used.

An essential function of immunization registries is their linkage of records from multiple providers into a single record for each child. With this capability, they will become the new gold standard measure of immunization status. A major benefit of registries is their ability to provide a complete immunization record that can be used for clinical decision-making. Widespread use of immunization registries will enhance greatly the ability to conduct studies of barriers to timely vaccination, interventions to improve coverage, vaccine safety, and vaccine efficacy.

	FOOTNOTES

Received for publication Dec 31, 1998; accepted Jan 5, 1999.

This work was presented in part at the Primary Care Research Methods and Statistics Conference; December 5, 1998; San Antonio, TX.

Address correspondence to Lance Rodewald, MD, National Immunization Program, Mailstop E-52, Centers for Disease Control and Prevention, 1600 Clifton Rd, NE, Atlanta, GA 30333.

	ABBREVIATIONS

VPD, vaccine-preventable disease; VFC, Vaccines for Children program; HEDIS, Healthplan Employer Data and Information Set; UTD, up-to-date; DTP, diphtheria, tetanus, and pertussis; MMR, measles, mumps, and rubella; Hib, Haemophilus influenzae vaccine; HBV, hepatitis B vaccine; NHIS, National Health Interview Survey.

	REFERENCES

Top Abstract Conclusion References

1. The President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry. Quality First: Better Health Care for All Americans. Washington, DC: US Government Printing Office; 1998
2. US Department of Health and Human Services, Office of Public Health and Science. Healthy People 2010 Objectives: Draft for Public Comment. Washington, DC: US Government Printing Office; 1998. Also at http://web.health.gov/healthypeople/2010Draft
3. US Preventive Services Task Force. The Guide to Clinical Preventive Services. 2nd ed. Alexandria, VA: International Medical Publications; 1997
4. Miller MA, Hinman AR. Cost-benefit and cost-effectiveness analysis of vaccine policy. In: Plotkin S, Orenstein W. Vaccines. 3rd ed. Philadelphia, PA: WB Saunders Co; 1999
5. National Vaccine Advisory Committee Standards for pediatric immunization practices. JAMA. 1993; 269:1817-1822 [Abstract/Free Full Text]
6. National Committee for Quality Assurance. HEDIS 3.0The Health Plan Employer Data and Information Set. II. Technical Specifications. Washington, DC: National Committee for Quality Assurance; 1997
7. Fairbrother G, Friedman S, DuMont K, Lobach K Markers for primary care: missed opportunities to immunize and screen for lead and tuberculosis by private physicians serving large numbers of inner-city Medicaid-eligible children. Pediatrics. 1996; 97:785-790 [Abstract/Free Full Text]
8. Bordley W, Margolis P The delivery of immunizations and other preventive services in private practices. Pediatrics. 1996; 97:467-473 [Abstract/Free Full Text]
9. Dietz V, Lewin M, Zell E, Rodewald L Evaluation of failure to follow vaccination recommendations as a marker for failure to follow other health recommendations. Pediatr Infect Dis J. 1997; 16:1157-1161 [CrossRef][Medline]
10. Rodewald L, Szilagyi P, Shiuh T, Humiston S, LeBaron C, Hall C Is underimmunization a marker for insufficient utilization of preventive and primary care? Arch Pediatr Adolesc Med. 1995; 149:393-397 [Abstract/Free Full Text]
11. Adams W, Geva J, Coffman J, Palfrey S Anemia and elevated lead levels in underimmunized inner-city children. Pediatrics. 1998; 101:1-5 [Abstract/Free Full Text]
12. Fairbrother G, Friedman S, Hanson K, Butts G Effect of the Vaccines for Children program on inner-city neighborhood physicians. Arch Pediatr Adolesc Med. 1997; 151:1229-1235 [Abstract/Free Full Text]
13. Shefer A, Fritchley J, Stevenson J. Improvement in immunization coverage and other health outcomes following implementation of immunization activities in WIC, Milwaukee, 1995-97. Presented at the 32nd National Immunization Conference; July 22, 1998; Atlanta, GA
14. Rodewald LE, Szilagyi PG, Humiston SG, Barth R, Kraus R, Raubertas RF A randomized study of tracking with outreach and provider prompting to improve immunization coverage and primary care. Pediatrics. 1999; 103:31-38 [Abstract/Free Full Text]
15. Pappaioanou M, Evans C. Development of the Guide to Community Preventive Services: A US Public Health Service initiative. J Public Health Manage Practice. 1998;4:48-54. See also http://web.health.gov/communityguide
16. Shefer A, Briss P, Rodewald L. Improving immunization coverage rates: an evidence-based review of the literature. Epidemiol Rev. In press
17. Cutts FT, Orenstein WA, Bernier RH Causes of low preschool immunization coverage in the United States. Ann Rev of Public Health. 1992; 13:385-398 [CrossRef][Medline]
18. Paunio M, Virtanen M, Peltola H, Increase of vaccination coverage by mass media and individual approach: intensified measles, mumps, and rubella prevention program in Finland. Am J Epidemiol. 1991; 133:1152-1160 [Abstract/Free Full Text]
19. Hand JS, Anderson D, Feffer D, Day C A successful school immunization programor not? J School Health. 1980; 50:50 [Medline]
20. Oeffinger KC, Roaten SP, Hitchcock MA, Oeffinger PK The effect of patient education on pediatric immunization rates. J Fam Pract. 1992; 35:288-293 [Medline]
21. Pierce C, Goldstein M, Suozzi K, Gallaher M, Dietz V, Stevenson J The impact of the standards for pediatric immunization practices on vaccination coverage levels. JAMA. 1996; 276:626-630 [Abstract/Free Full Text]
22. Waterman SH, Hill LL, Robyn B, et al. A model immunization demonstration for preschoolers in an inner-city barrio, San Diego, California, 1992-1994. Am J Prev Med. 1996;12:8-13. Supplement
23. Alemi F, Alemagno SA, Goldhagen J, et al. Computer reminders improve on-time immunization rates. Med Care. 1996;34:OS45-OS51. Supplement
24. Alto WA, Fury D, Condo A, Doran M, Aduddell M Improving the immunization coverage of children less than 7 years old in a family practice residency. J Am Board Fam Pract. 1994; 7:472-477
25. Bell JC, Whitehead P, Chey T, Smith W, Capon AG, Jalaludin B The epidemiology of incomplete childhood immunization: an analysis of reported immunization status in outer western Sydney. J Paediatr Child Health. 1993; 29:384-388 [Medline]
26. Campbell JR, Szilagyi PG, Rodewald LE, Doane C, Roughmann KJ Patient-specific reminder letters and pediatric well-child-care show rates. Clin Pediatr. 1994; 1994:268-272
27. Ferson MJ, Fitzsimmons G, Christie D, Woollett H School health nurse interventions to increase immunisation uptake in school entrants. Public Health. 1995; 109:25-29 [CrossRef][Medline]
28. Lieu TA, Black SB, Ray P, Computer-generated recall letters for underimmunized children: how cost-effective? Pediatr Infect Dis J. 1997; 16:28-33 [CrossRef][Medline]
29. Stehr-Green PA, Dini EF, Lindegren ML, Patriarca PA Evaluation of telephoned computer-generated reminders to improve immunization coverage at inner-city clinics. Public Health Rep. 1993; 108:426-430 [Medline]
30. Tollestrup K, Hubbard BB Evaluation of a follow-up system in a county health department's immunization clinic. Am J Prev Med. 1991; 7:24-28 [Medline]
31. Yokley JM, Glenwick DS Increasing the immunization of preschool children: an evaluation of applied community interventions. J Appl Behav Anal. 1984; 17:313-325 [CrossRef][Medline]
32. Young SA, Halpin TJ, Johnson DA, Irvin JJ, Marks JS Effectiveness of a mailed reminder on the immunization levels of infants at high risk of failure to complete immunizations. Am J Public Health. 1980; 70:422-424 [Abstract/Free Full Text]
33. Combs SP, Walter EB, Drucker RP, Clements DA Removing a major barrier to universal hepatitis B immunization in infants. Arch Pediatr Adolesc Med. 1996; 150:112-114 [Abstract/Free Full Text]
34. Lurie N, Manning WG, Peterson C, Goldberg GA, Phelps CA, Lillard L Preventive care: do we practice what we preach? Am J Public Health. 1987; 77:801-804 [Abstract/Free Full Text]
35. Rodewald LE, Szilagyi PG, Holl J, Shone LR, Zwanziger J, Raubertas RF Health insurance for low-income, working families: impact on the delivery of immunizations to preschool children. Arch Pediatr Adolesc Med. 1997; 151:798-803 [Abstract/Free Full Text]
36. Taylor JA, Darden PM, Slora E, Hasemeier CM, Asmussen L, Wasserman R The influence of provider behavior, parental characteristics, and a public policy initiative on the immunization status of children followed by private pediatricians: a study from pediatric research in office settings. Pediatrics. 1997; 99:209-215 [Abstract/Free Full Text]
37. Zimmerman RK, Janosky JE Immunization barriers in Minnesota private practices: the influence of economics and training on vaccine timing. Fam Pract Res J. 1993; 13:213-224 [Medline]
38. Szilagyi PG, Rodewald LE, Humiston SG, Effect of 2 urban emergency department immunization programs on childhood immunization rates. Arch Pediatr Adolesc Med. 1997; 151:999-1006 [Abstract/Free Full Text]
39. Crittenden P, Rao M. The immunisation coordinator: improving uptake of childhood immunisation. Commun Dis Rep CDR Rev, 1994;7:R79-R81
40. Moore BJ, Morris DW, Burton B, Kilcrease DT Measuring effectiveness of service aides in infant immunization surveillance program in North Central Texas. Am J Public Health. 1981; 71:634-636 [Abstract/Free Full Text]
41. Rosenberg Z, Findley S, McPhillips S, Penachio M, Silver P. Community-based strategies for immunizing the "hard-to-reach" child: the New York State immunization and primary health care initiative. Am J Prev Med. 1995;11:14-20. Supplement
42. While AE Health visitor contribution to pre-school child prophylaxis. Public Health. 1987; 101:229-232 [CrossRef][Medline]
43. Birkhead GS, LeBaron CW, Parsons P, The immunization of children enrolled in the Special Supplemental Food Program for Women, Infants, and Children (WIC). The impact of different strategies. JAMA 1995; 274:312-316 [Abstract/Free Full Text]
44. Wood D, Halfon N, Donald-Sherbourne C, Increasing immunization rates among inner-city, African American children. JAMA. 1998; 279:29-34 [Abstract/Free Full Text]
45. Golden RE. Evaluation of three immunization interventions among families participating in the special supplemental nutrition program for women, infants, and children in South Central and South East Los Angeles. UCLA; 1997; p 1. Dissertation
46. O'Mara LM, Isaacs S Evaluation of registered nurses follow-up on the reported immunization status of children attending child care centres. Can J Public Health. 1993; 2:124-127
47. Unti LM, Coyle KK, Woodruff BA, Boyer-Chuanroong LB Incentives and motivators in school-based hepatitis B vaccination programs. J School Health. 1998; 67:265-268
48. Rodewald LE, Szilagyi PG, Humiston SG Effect of emergency department immunizations on immunization rates and subsequent primary care visits. Arch Pediatr Adolesc Med. 1996; 150:1271-1276 [Abstract/Free Full Text]
49. Carlson JA, Lewis CA Effect of the immunization program in Ontario schools. Can Med Assoc J. 1985; 133:215-216 [Medline]
50. Nelson DB, Layde MM, Chatton TB Rubella susceptibility in inner-city adolescents: the effect of a school immunization law. Am J Public Health 1982; 72:710-713 [Abstract/Free Full Text]
51. Robbins KB, Brandling-Bennett D, Hinman AR Low measles incidence: association with enforcement of school immunization laws. Am J Public Health 1981; 71:270-274 [Abstract/Free Full Text]
52. Scheiber M, Halfon N Immunizing California's children. Effects of current policies on immunization levels. West J Med. 1990; 153:400-405 [Medline]
53. Schulte EE, Birkhead GS, Kondracki SF, Morse DL Patterns of Haemophilus influenzae type b invasive disease in New York State, 1987 to 1991: the role of vaccination requirements for day-care attendance. Pediatrics. 1994; 94:1014-1016 [Abstract/Free Full Text]
54. van Loon FP, Holmes SJ, Sirotkin BI, et al. Mumps surveillanceUnited States, 1988-1993. MMWR. CDC Surveillance Summaries. 1995;44:1-14
55. Bell LM, Pritchard M, Anderko R, Levenson R A program to immunize hospitalized preschool-aged children: evaluation and impact. Pediatrics. 1997; 100:192-196 [Abstract/Free Full Text]
56. Brink SG Provider reminders. Changing information format to increase infant immunizations. Med Care. 1989; 27:648-653 [CrossRef][Medline]
57. Carlin E, Carlson R, Nordin J Using continuous quality improvement tools to improve pediatric immunization rates. Jt Comm J Qual Improv. 1996; 22:277-288 [Medline]
58. Harper PG, Murray DM An organizational strategy to improve adolescent measles-mumps-rubella vaccination in a low socioeconomic population. A method to reduce missed opportunities. Arch Fam Med. 1994; 3:257-262 [Abstract/Free Full Text]
59. Reading R, Colver A, Openshaw S, Jarvis S Do interventions that improve immunisation uptake also reduce social inequalities in uptake? Br Med J. 1994; 308:1142-1144 [Abstract/Free Full Text]
60. Szilagyi PG, Rodewald LE, Humiston SG, Reducing missed opportunities for immunizations. Easier said than done. Arch Pediatr Adolesc Med. 1996; 150:1193-1200 [Abstract/Free Full Text]
61. LeBaron CW, Chaney M, Baughman AL, Impact of measurement and feedback on vaccination coverage in public clinics, 1988-1994. JAMA. 1997; 277:631-635 [Abstract/Free Full Text]
62. Christy C, McConnochie KM, Zernik N, Brzoza S Impact of an algorithm-guided nurse intervention on the use of immunization opportunities. Arch Pediatr Adolesc Med. 1997; 151:384-391 [Abstract/Free Full Text]
63. Schum TR, Nelson DB, Duma MA, Sedmak GV Increasing rubella seronegativity despite a compulsory school law. Am J Public Health. 1990; 80:66-69 [Abstract/Free Full Text]
64. Darden PM, Taylor JA Assessing immunization rates in office practice. Pediatr Ann. 1998; 27:411-416 [Medline]
65. Darden PM, Taylor JA, Slora EJ, Hasemeier CM, Asmussen L, Recknor JC, Wasserman RC Methodological issues in determining rates of childhood immunization in office practice. A study from pediatric research in office settings (PROS). Arch Pediatr Adolesc Med. 1996; 150:1027-1031 [Abstract/Free Full Text]
66. Ross A, Kennedy A, Holt E, Guyer B, Hou W Initiating the first DTP vaccination age appropriately: a model for understanding vaccination coverage. Pediatrics. 1998; 101:970-974 [Abstract/Free Full Text]
67. Vivier PM, Alario AJ, Simon PR, Flanagan P, O'Haire C, Peter G. The importance of the timing of vaccine administration in the assessment of the immunization status of children enrolled in a hospital based Medicaid managed care practice. Pediatr Res. 1998;42. Abstract 697
68. Ehresmann K, White K, Hedberg C, A statewide survey of immunization rates in Minnesota school age children: implications for targeted assessment and prevention strategies. Pediatr Infect Dis J. 1998; 17:711-716 [CrossRef][Medline]
69. Rodewald L, Szilagyi P, Humiston S, Is an emergency department visit a marker for undervaccination and missed vaccination opportunities among children who have access to primary care? Pediatrics. 1993; 91:605-611 [Abstract/Free Full Text]
70. Centers for Disease Control and Prevention National, state, and urban area vaccination coverage levels among children aged 19-35 monthsUnited States, April 1994-March 1995. MMWR. 1996; 45:1-6 [Medline]
71. Peak RR, Cadell DM. Overview of the National Immunization Provider Record Check Study. Proceedings of the Section on Survey Research Methods. Am Stat Assoc. 1996:332-334
72. Suarez L, Simpson D, Smith D Errors and correlates in parental recall of child immunizations: effects on vaccination coverage estimates. Pediatrics. 1997; 99:e3 [Abstract/Free Full Text]
73. Goldstein K, Kviz F, Daum R Accuracy of immunization histories provided by adults accompanying preschool children to a pediatric emergency department. JAMA. 1993; 270:2190-2194 [Abstract/Free Full Text]
74. Watson M, Feldman K, Sugar N, Sommer C, Thomas E, Lin T Inadequate history as a barrier to immunization. Arch Pediatr Adolesc Med. 1996; 150:135-139 [Abstract/Free Full Text]
75. Fierman AH, Rosen CM, Legano LA, Lim SW, Mendelsohn AL, Dreyer BP Immunization status as determined by patients' hand-held cards vs medical records. Arch Pediatr Adolesc Med. 1996; 150:863-866 [Abstract/Free Full Text]
76. Zell E, Peak R, Rodewald L, Ezzati-Rice T. Reliability of vaccination cards and parent-derived information for determining immunization status: lessons from the 1994 National Health Interview Survey (NHIS) Provider Record Check (PRC) study. Presented at the Society for Pediatric Research; May 1997; Washington, DC
77. Maes EF, Rodewald L, Coronado VG, et al. Who is immunizing children in the US: public or private providers? Presented at the Pediatric Academic Societies Meeting; May 1998; New Orleans, LA
78. Yawn B, Edmonson L, Huber L, Poland G, Jacobson R The impact of a stimulated immunization registry on perceived childhood immunization status. Am J Managed Care. 1998; 4:185-192 [Medline]
79. Centers for Disease Control and Prevention National, state, and urban area vaccination coverage levels among children aged 19-35 monthsUnited States, 1997. MMWR. 1998; 47:547-554 [Medline]