PEDIATRICS Vol. 107 No. 1 January 2001, pp. 91-96

The Impact of Record Scattering on the Measurement of Immunization Coverage

Shannon Stokley, MPH^*, Lance E. Rodewald, MD^*, and Edmond F. Maes, PhD

From the ^* National Immunization Program and the  Epidemiology Program Office, Centers for Disease Control and Prevention, Atlanta, Georgia.

	ABSTRACT

Top Abstract Methods Results Discussion References

Background.  Lack of a consolidated immunization record may lead to problems with determining individual immunization needs at office visits as well as measuring vaccination coverage levels of a clinician's practice or a community's population.

Objectives.  For children with multiple immunization providers, evaluate the difference in coverage levels using data from all responding immunization providers compared with: 1) the most recent immunization provider's records, 2) the first immunization provider's records, and 3) a randomly selected immunization provider's records. Identify characteristics of the most recent provider that may be associated with reporting incomplete immunization histories.

Methods.  Data from the 1995 National Immunization Provider Record Check Study (NIPRCS) were used for analysis. The NIPRCS is a provider validation study of the household reported immunization histories of all children 19 to 35 months of age included in the National Health Interview Survey (NHIS). Providers identified by the child's parent during the NHIS interview are mailed a 2-page survey to report all immunizations (type and date) the child received, regardless of the provider who administered the shots, and child's first and most recent visit dates to the practice.

Results.  Of the 1352 children with provider data, 304 (22%) had received immunizations from more than one provider. Compared with information from all providers and depending on the vaccine, the most recent provider records underestimated coverage by 9.6 to 13.4 percentage points; the initial provider records underestimated coverage by 15.6 to 34.6 percentage points; and the randomly selected provider records underestimated coverage by 10.0 to 20.7 percentage points. Public facilities and having an immunization summary sheet in the patient's chart were associated with having complete records.

Conclusion.  Scattered immunization records significantly compromise the ability of clinicians to determine the immunization status of their patients who received immunizations at other sites of health care. Routinely assessing immunization coverage levels at the practice level, implementing a recall system, and developing community-wide immunization registries are some strategies to reduce the problem of scattered immunization records.immunization, assessment, provider validation, record scattering.

Routine childhood vaccination is one of the major public health success stories in this century,¹ currently producing the lowest incidence rates of traditional vaccine-preventable diseases² attributed to the highest vaccination coverage levels for the corresponding diseases.^3,4 However, the continued success of vaccination is being challenged by an immunization schedule that is increasing in size and complexity and by the lack of accessible, unified immunization records. Already, children are recommended to receive a total of 15 to 19 vaccine doses by 18 months of age, compared with only 8 doses by 20 years ago. New vaccines will continue and perhaps accelerate this trend.

A child usually has 2 sources of immunization history information, the parent and the health care provider. Because parental records of their child's immunization history have been shown to be unreliable^5-8 and because health care professionals are required by law to record information about immunizations given in their offices, the health care provider's records are typically viewed as being the most accurate and reliable. However, recent studies have shown that many children are vaccinated away from the primary care office,^9,10 either at a previous provider office or at the health department; most new patients do not bring adequate documentation of their immunization history to the initial visit to a new primary care provider^8,11; and communication among immunization providers is frequently poor,¹² all of which leads to a lack of unified records at the primary care provider office and an inability to determine vaccination needs accurately.

Lack of a consolidated record is problematic not only for determining individual immunization needs at office visits, but also for measuring the vaccination coverage levels of a clinician's practice or a community's population. Measuring coverage levels at the practice or community level is an important strategy to improve and sustain high vaccination coverage levels.^13,14 In theory, the relation between missing vaccinations and misclassification of an up-to-date (UTD) child as not UTD is exponential, with small amounts of missing information having a very large impact on the accuracy of coverage assessments.¹⁵ Scattered records are a potential source of missing vaccination information at the provider and community level.

We report the results of a national survey to measure the impact that the scattering of vaccination records has on the assessment of coverage for US preschool children. We use the 1995 National Immunization Provider Record Check Study (NIPRCS) to: 1) determine the proportion of children having more than one immunization provider and 2) compare coverage assessments using records available at individual providers' offices with a consolidated immunization record.

	METHODS

Top Abstract Methods Results Discussion References

Data from the 1995 NIPRCS, a component of the National Health Interview Survey (NHIS), were analyzed. The NHIS is a cross-sectional, household, face-to-face interview of the civilian, noninstitutionalized population of the United States. Beginning in 1991, the NHIS included immunization-related questions to produce national vaccination coverage estimates of children <6 years of age.¹⁶ During the immunization section of the NHIS, the respondent could choose to report from the child's immunization record all immunizations (type and date) the child received or the respondent could report from memory the number of doses (all is an acceptable response) of each immunization the child had received.

The NIPRCS was introduced in 1994 to reduce response bias of reporting immunizations from memory to the NHIS.¹⁷ For each data collection year, immunization histories of children 19 to 35 months of age collected during the NHIS interview were verified by the provider(s) reported by the parent to have administered the child's vaccines. An immunization history questionnaire was mailed to each identified provider. The first page of the questionnaire included an immunization grid for the provider to record the date and type of each immunization that the child received and to indicate whether the vaccine was given by a different provider. The second page included questions about the child's first and most recent visit dates to the practice, the source of immunization information from the patient's chart used to complete the questionnaire (ie, immunization summary sheet, physician's orders from patient's chart, or an immunization summary sheet plus an additional source), and characteristics of the practice (public or private facility).

For children with more than one provider, we compared the most recent visit dates of each provider to identify the most recent provider for each child in the survey. We compared the first visit dates of each provider to identify the first provider. If the information was missing for either the first or most recent visit date, we used the information in the immunization grid to identify the provider sequence. The date of the earliest immunization given by the provider (the vaccine was not indicated to be given elsewhere) was recorded as the first visit date and the most recent immunization date reported by the provider (even if the provider indicated that the vaccine was administered elsewhere) was recorded as the most recent visit date.

We determined the immunization coverage level for each vaccine and vaccine series using: 1) data from all provider records (referred to as the consolidated record), 2) data from only the most recent provider's records, 3) data from only the first provider's records, and 4) data from only one randomly selected provider's records. To determine the immunization coverage levels based on one randomly selected provider, a bootstrap simulation of 1000 samples of one randomly selected provider per child was performed. The immunization coverage levels from the randomly selected provider's records represent the average immunization coverage levels taken from the 1000 samples. Differences in coverage levels were calculated by comparing data from the most recent provider, the first provider, or a randomly selected provider to data from the consolidated record. The vaccines and combination of vaccines evaluated include: 4 or more doses of diphtheria-tetanus-pertussis (DTP), 3 or more doses of polio, 1 or more doses of measles-mumps-rubella (MMR), 3 or more doses of Haemophilus influenzae type b (HIB), 3 or more doses of hepatitis B, the 4:3:1 series (4-DTP/3-polio/1-MMR), the 4:3:1:3 series (4-DTP/3-polio/1-MMR/3-HIB), and the 4:3:1:3:3 series (4-DTP/3-polio/1-MMR/3-HIB/3-hepatitis B). Varicella was not licensed at the beginning of the 1995 NHIS data collection and was not included in our study.

Sensitivity, specificity, and predictive value analysis of the most recent provider was performed to determine whether the ability to identify undervaccinated children differed by provider characteristics. The characteristics of interest were facility type and the source of information used to complete the immunization history questionnaire. The analysis was limited to children who received immunizations from more than one provider and the true immunization status was determined from the overall immunization history resulting from the consolidated record. The condition being tested was the recent provider's ability to accurately identify a child as not UTD for the recommended immunizations as described previously.

The specificity of the most recent provider's records represents the ability to correctly identify children who were truly UTD and, therefore, not in need of additional vaccination. We defined specificity as the proportion of children UTD according to the consolidated record who were also identified as UTD based on the most recent provider's records. Low specificity implies that children not in need of vaccination would be incorrectly classified as needing vaccination, which might result in overvaccination.

The positive predictive value (PPV) of the most recent provider's records represents the probability that a child classified as not UTD was truly not UTD. We define PPV as the proportion of children not UTD based on the most recent provider's records, who were truly not UTD based on the consolidated record.

Because the most recent provider's records are a subset of the consolidated record, by definition, children not UTD according to the consolidated record cannot be UTD according to the most recent provider's records. Therefore, the sensitivity and negative predictive value (NPV) to identify underimmunized children will always be 100%. Thus, we do not report sensitivity or NPV as results.

	RESULTS

Top Abstract Methods Results Discussion References

Of the 2089 children 19 to 35 months of age included in the 1995 NHIS, 1352 (65%) had provider immunization data, of which 304 (22%) went to more than one provider (91% went to 2 providers, 8% went to 3 providers, and 1% went to 4 providers). Overall, there were 769 children (57%) with provider data who had immunization dates reported from a written record during the NHIS interview, whereas 202 children (66%) who went to more than one provider had immunization dates reported from a written record during the NHIS interview.

Differences in coverage levels for each vaccine and vaccine series for children with more than one provider can be seen in Table 1. Compared with information from the consolidated record, the most recent provider records underestimated coverage by 9.8 percentage points for 3 or more doses of hepatitis B to 12.3 percentage points for 4 or more doses of DTP. The first providers' records underestimated coverage by 27.4 to 45.8 percentage points and the randomly selected providers underestimated coverage by 18.7 to 29.1 percentage points, depending on the vaccine or vaccine series. Vaccine series that require a vaccine dose in the second year of life (eg, DTP) had larger differences in coverage levels between the consolidated record and the first, most recent, and randomly selected provider, compared with vaccine series that do not require a vaccine dose in the second year of life (eg, HIB).

To determine how record scattering could affect immunization coverage levels of all children included in the NIPRCS, differences in coverage levels based on the most recent providers records compared with the consolidated record were calculated for all children with provider data regardless of the number of providers visited. For children with only one provider, that provider was considered the most recent provider. Compared with the consolidated record, the most recent provider underestimated by 3.5 percentage points for 3 or more doses of HIB, 4.2 percentage points for 4 or more doses of DTP/DTaP, and 4.5 percentage points for the 4:3:1:3:3 series.

Figure 1 presents a closer look at how accurately the most recent provider classified a child as UTD for a particular vaccine or series. Of the children with more than one provider who were UTD for a particular vaccine or series based on information from the consolidated record, the most recent provider classified 15.8% of the children as behind schedule for 4+ DTP/DTaP and 22.9% of the children as behind schedule for the 4:3:1:3:3 series.

View larger version (12K): [in this window] [in a new window]   Fig. 1.   Proportion of UTD children classified as not UTD by the most recent provider, children with more than one provider (n = 304), 1995 NIPRCS.

Of the children who went to more than one provider, 64% of the recent providers practiced in a private facility, 23% in a public facility, 6% were "other" (hospital outpatient, military, other), and 6% were unknown. Because a small proportion of facilities were other or unknown, analysis by facility type was limited to private and public facilities. Table 2 shows the specificity and predictive value analysis of the most recent provider by facility type for children who went to more than one provider. Differences in specificity were found between public and private providers. Although public facilities had a higher probability of correctly classifying a child who was UTD for all vaccine and vaccine series compared with private facilities, a significant difference was found only for the 4:3:1 series. In addition, significant differences in PPV were found between private and public facilities for 4+ DTP/DTaP, 3+ polio, the 4:3:1 series, and the 4:3:1:3:3 series. If a public facility classified a child as not UTD for the 4:3:1 series, there was a 90% chance that the child was truly not UTD, whereas if a private facility classified a child as not UTD, there was only a 58% chance that the child was not actually UTD.

With respect to the source of information the most recent provider used to complete the immunization history questionnaire, 60% used an immunization summary sheet, 10% used physician's orders as documented in the patient's chart, 22% used a combination of a summary sheet and another source, 5% used an other source, and 3% did not specify. Because of the small sample size, specificity and PPV analysis (Table 3) excluded the most recent providers who used an other source or did not specify. Comparison of the most recent providers who used physician's orders from the patient's chart to the recent providers who used the immunization summary sheet shows significant differences in specificity for all vaccines and vaccine series, except for 3+ hepatitis B. Significant differences in PPV were also found between the 2 groups, except for 3+ HIB, 3+ hepatitis B, and the 4:3:1:3:3 series. No significant differences in specificity and PPV were found between providers who used the immunization summary sheet plus another source and providers who used the immunization summary sheet only except for 1+ MMR.

Analysis of the source of information used to complete the immunization history questionnaire by facility type indicates that for public facilities, 68.5% used an immunization summary sheet only, 14.2% used a combination of an immunization summary sheet and another source, and 3.4% used physician's orders. Of the private facilities, 60% used an immunization summary sheet only, 25.4% used a combination of an immunization summary sheet and another source, and 10.8% used physician's orders.

	DISCUSSION

Top Abstract Methods Results Discussion References

We have shown that scattered immunization records significantly compromise the ability of clinicians to determine the immunization status of their patients who received immunizations at other sites of health care. Nationally, 22% of children received their early preschool vaccinations from more than one health care professional. Among children having more than one immunization provider: 1) the records of the child's most recent provider mistakenly indicated that 23% of completely vaccinated children were in need of vaccination, 2) a record from the most recent provider indicating that the child needed additional vaccinations for the 4:3:1:3 series was incorrect 38% of the time for private practitioners and 19% of the time for health department clinics, and 3) the presence of a summary immunization record in the chart was associated with more complete records.

Relation to Other Literature

Findings from our study provides a national perspective to a problem that has been studied mostly at the local level. Hamlin and colleagues¹⁸ showed that records were scattered between 2 clinics located together in Los Angeles County. One clinic was a health department clinic and the other was a community health center; both were on the same floor of the same building. Murphy and colleagues¹⁹ demonstrated that for children who visited a Dallas county public clinic, incomplete documentation of immunizations in both the public clinic and parent's record was associated with a 45% rate of unnecessary immunization. Yawn and colleagues²⁰ demonstrated the high degree of record scattering in Olmstead County, Minnesota, and showed that if the records could be combined by an immunization registry, the ability to correctly classify children's immunization status would increase. Our study shows that scattered records is a problem that is national in scope, resulting in the misclassification of over 150 000 completely vaccinated US children as being in need of vaccination and that the problem is more serious among private practitioners.

Watson and colleagues¹¹ showed one source for scattered recordsonly 22% of parents brought their immunization records to an initial visit to a new primary care clinician. Many studies have shown that parents do not know the immunization status of their children. In general, parents believe that their children are fully vaccinated when they may not be UTD.^5-8,11 Thus, parents cannot be relied on to know the vaccination needs of their children. Our study did not investigate parent recall or the causes for scattered records. Thus, these other studies provide potential explanations for our findings.

Strengths and Weaknesses

The main strengths of the study are that it is national in scope, uses a household area probability survey to identify children, and uses a parent-linked provider record check to determine immunization status. There are 2 main weaknesses of the study. The first is that there is an unknown degree of misclassification of whether the immunization provider was the first, second, or most recent provider. The study methods used the first and last dates of visits to each provider to make that determination, and in 19% of cases, that information was missing. However, the majority of children had only 2 providers, therefore, increasing the degree of certainty when identifying the most recent provider. The second weakness is one of scope. We were limited to the questions available on the NHIS and its immunization provider record check study. Thus, reasons for the scattered records could not be determined.

Implications of the Study

There are several reasons for needing complete immunization records at the offices of immunization providers. As a result of the studies that demonstrated the potential to improve immunization coverage by reducing missed opportunities,^21,22 providers are strongly encouraged to vaccinate at every opportunity. For individual children, the records are essential to determine their need for vaccination at the time of the office visit. Failure to assess accurately implies failure to make a correct clinical decision whether to vaccinate and with what vaccines.

For health care providers, complete records are needed to assess accurately the immunization coverage of their patientssomething that all providers, public and private, are being asked to do.¹³ Without complete records, the assessments show substantially lower coverage than may actually be the case, and information that would help clinicians improve their immunization practices might be rendered inaccurate. However, a benefit of these assessments is to quantify the degree of missing records for clinicians, which, in turn, should lead to more complete records.

This study also supports previous recommendations that all immunization providers operate a recall system to bring children in need of vaccination back to the office for vaccination and other clinical preventive services.^14,23-25 Recall systems list patients belonging to a practice that cannot be documented to be UTD on immunizations. To avoid recalling fully vaccinated children whose medical records are missing vaccinations that were actually administered, the office staff needs to make a judgment about the completeness of the record. If the staff determines that the record may be incomplete (for example, if the patient is relatively new to the practice or has been recently referred to the health department because his or her insurance does not cover vaccinations), an attempt to determine the true immunization history could precede the recall of the patient.

The final implication of the study is that the data support the development of community immunization registries that communicate with registries of other communities.²⁶ Once a system of registries is in place, the problem of scattered immunization records could be greatly reduced in magnitude.

Unanswered Questions and Further Research

This research raises 3 types of questions: What are the causes of scattered records?; What are the consequences of scattered records?; and What can be done to reduce their impact? Causation might best be answered by looking at the relation between mobility, changing providers, and insurance-related referrals to health departments. Consequences might include both undervaccination and overvaccination. For example, a reluctance to vaccinate when there is uncertainty about the completeness of vaccination records might lead to missed opportunities to vaccinate. Conversely, vaccinating children with incomplete records may lead to overvaccination, as was seen in the study by Murphy and colleagues¹⁹ and Feikema and colleagues.²⁷ Just as undervaccination exposes children to unnecessary risk of vaccine-preventable diseases, overvaccination exposes children to unnecessary risk of adverse events from vaccines. Finally, research to reduce the scattering of records will lead to improved efficiency of the immunization delivery system. This work will support and guide the development of immunization registries, which will ultimately help clinicians keep track of the immunization status and needs of their patients.

	FOOTNOTES

Received for publication Sep 2, 1999; accepted Mar 31, 2000.

Presented in part at the Pediatric Academic Societies Annual Meeting; May 1, 1999; San Francisco, CA.

Reprint requests to (S.S.) Centers for Disease Control and Prevention/National Immunization Program, 1600 Clifton Rd, E-62, Atlanta, GA 30333. E-mail: sstokley{at}cdc.gov

	ABBREVIATIONS

UTD, up-to-date; NIPRCS, National Immunization Provider Record Check Study; NHIS, National Health Interview Survey; DTP, diphtheria-tetanus-pertussis; MMR, measles-mumps-rubella; HIB, Haemophilus influenzae type b; PPV, positive predictive value; NPV, negative predictive value.

	REFERENCES

Top Abstract Methods Results Discussion References

1. Centers for Disease Control and Prevention Impact of vaccines universally recommended for childrenUnited States, 1900-1998. MMWR Morb Mortal Wkly Rep 1999; 48:243-248 [Medline]
2. Orenstein WA, Hinman AR, Rodewald LE. Public health considerationsUnited States. In: Plotkin SA, Orenstein WA, eds. Vaccines. 3rd ed. Philadelphia, PA: WB Saunders Co; 1999:1024
3. Centers for Disease Control and Prevention Notice to readers: national vaccination coverage levels among children aged 19-35 monthsUnited States, 1998. MMWR Morb Mortal Wkly Rep 1999; 48:829-830 [Medline]
4. National trends in vaccination coverage. Available at: http:/www.cdc.gov/nip/coverage/data.htm
5. Suarez L, Simpson DM, Smith DR. Errors and correlates in parental recall of child immunizations: effects on vaccination coverage estimates. Pediatrics. 1997;99(5). URL: http://www.pediatrics.org/cgi/content/full/99/5/e3
6. Bolton B, Holt E, Ross A, Hughhart N, Guyer B Estimating vaccination coverage using parental recall, vaccination cards, and medical records. Public Health Rep 1998; 113:521-526 [Medline]
7. Zell E, Peak R, Rodewald L, Ezzati-Rice T Vaccine coverage. Public Health Rep 1999; 114:3-4 [CrossRef][Medline]
8. Goldstein KP, Kviz FJ, Daum RS Accuracy of immunization histories provided by adults accompanying preschool children to a pediatric emergency department. JAMA 1993; 270:2190-2194 [Abstract]
9. Zimmerman R, Medsgar A, Ricci E, Impact of free vaccine and insurance status on physician referral of children to public vaccine clinics. JAMA 1997; 278:996-1000 [Abstract]
10. Bordley WC, Freed GL, Garrett JM, Factors responsible for immunization referrals to health departments in North Carolina. Pediatrics 1994; 94:376-380 [Abstract/Free Full Text]
11. Watson MA, Feldman KW, Sugar NF, Sommer CJ, Thomas ER, Lin T Inadequate history as a barrier to immunizations. Arch Pediatr Adolesc Med 1996; 150:135-139 [Abstract]
12. Santoli JM, Barker LE, Gandhi NB, Lyons BH, Rodewald L. Health department clinics as immunization providers for pediatric patients in urban and rural settingsa 1998 national survey. Presented at the 1999 Pediatric Academic Societies Annual Meeting; May 1-4, 1999; San Francisco, CA. Abstact No. 764
13. Ad Hoc Working Group for the Development of Standards for Pediatric Immunization Practices Standards for pediatric immunization practices. JAMA 1993; 269:1817-1822 [CrossRef][Medline]
14. Shefer A, Briss P, Rodewald L, Improving immunization coverage rates: an evidence-based review of the literature. Epi Rev 1999; 21:96-142
15. Rodewald L, Maes E, Stevenson J, Lyons B, Stokley S, Szilagyi P Immunization performance measurement in a changing immunization environment. Pediatrics 1999; 103:889-897 [Abstract/Free Full Text]
16. Massey JT, Moore TF, Parsons VL, et al. Design and Estimation for the National Health Interview Survey, 1985-1994. Hyattsville, MD: US Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention; 1989. Vital and Health Statistics Series 2, No. 110
17. Peak RR, Cadell DM. Overview of the national immunization provider record check study: proceedings of the section on survey research methods. Am Stat Assoc. 1996:332-334
18. Hamlin JS, Wood D, Pereyra M, Grabowsky M Inappropriately timed immunizations: types, causes, and their relationship to record keeping. Am J Public Health 1996; 86:1812-1814 [Abstract/Free Full Text]
19. Murphy TV, Pastor P, Medley FB Factors associated with unnecessary immunization given to children. Pediatr Infect Dis J 1997; 16:47-52 [CrossRef][Medline]
20. Yawn BP, Edmonson L, Huber L, Poland GA, Jacobson RM, Jacobsen SJ The impact of a simulated immunization registry on perceived childhood immunization status. Am J Manage Care 1998; 4:185-192 [Medline]
21. Dietz VJ, Stevenson J, Zell ER, Cochi S, Hadler S, Eddins D Potential impact on vaccination coverage levels by administering vaccines simultaneously and reducing dropout rates. Arch Pediatr Adolesc Med 1994; 148:943-948 [Abstract]
22. Szilagyi PG, Rodewald LE, Humiston SG, Reducing missed opportunities for immunizations. Arch Pediatr Adolesc Med 1996; 150:1193-1200 [Abstract]
23. Udovic S, Lieu T Evidence on office-based interventions to improve childhood immunization delivery. Pediatr Ann 1998; 27:355-361 [Medline]
24. Task Force on Community Preventive Services Recommendations regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med 2000; 18:92-96 [CrossRef][Medline]
25. Briss PA, Rodewald LE, Hinman AR, Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med 2000; 18:97-123 [Medline]
26. Linkins R, Feikema S Immunization registries: the cornerstone of childhood immunization in the 21st century. Pediatr Ann 1998; 27:349-354 [Medline]
27. Feikema S, Klevens R, Washington M, Barker L Extra immunization among US children. JAMA 2000; 283:1311-1340 [Abstract/Free Full Text]