 |
|
 |
|
 |
 |
|
 |
 |
 |
 |
 |
 |
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
PEDIATRICS Vol. 106 No. 3 September 2000, pp. 493-496
From the * Immunization Services Division, National Immunization Program, Centers for Disease Control and Prevention, Atlanta, Georgia.
 |
ABSTRACT |
|---|
Top
Abstract
Methods
Results
Discussion
References
|
|---|
Background. Accurate identification of underimmunized children is needed to determine which children need vaccination. Previous studies have found the accuracy of manually determining the immunization status from a personal vaccination record to be low (<50%).
Objective. To determine the accuracy of manual immunization status assessment for preschool-aged children.
Subjects and Setting. Children 
32 months old
(n = 21 263) seen over 1 year at 12 women,
infants, and children (WIC) sites in San Diego, California. Age at
evaluation was between 0 and 24 months.
Methods. Paraprofessional immunization specialists conducted manual immunization status assessment using the WIC client's personal vaccination record. Immunization status as recorded in the WIC record was compared with computerized assessment (the gold standard).
Measures and Results. For all patient encounters, 29 078 (80%) of 36 368 were assessed correctly; manual assessment outcome was not recorded in the WIC record for 2171 (6%) of encounters. Accuracy varied by WIC site (range: 70%-90%). The sensitivity at correctly identifying an underimmunized child per encounter was 53.6%; the specificity at correctly identifying a fully vaccinated child per encounter was 89.4%. The 3 most common vaccines that were incorrectly assessed in identifying an underimmunized child were Haemophilus influenzae type b (43%), hepatitis B (37%), and diphtheria-tetanus toxoids and (cellular or acellular) pertussis vaccine (24%). Children with no outcome as recorded in the WIC record were 5 times as likely to be up-to-date.
Conclusions. Manual immunization assessment was specific but only moderately sensitive at identifying underimmunized children. Thus, many underimmunized children will by missed but only 10% of children will be referred inappropriately. Key words: immunization, vaccination, assessment, WIC.
The widespread use of vaccines has led to record low levels
of vaccine-preventable diseases in the United States. Because of school
immunization laws enforced since the 1980s, immunization coverage rates
for students entering kindergarten or first grade have exceeded 95%.
Despite success in immunizing school-aged children, >1 million
preschool-aged children remain underimmunized and at risk for disease.
Preschool-aged children continue to remain delayed in age-appropriate
immunization because of a number of factors. Underimmunization may
occur because of an inadequate number of health visits to the
immunization provider or because of missed opportunities during the
visit. Even when children receive the correct number and timing of
health visits, missed opportunities during the visit may result in a
delay in adequate immunization.1,2 Providers may fail to
follow valid contraindications, with subsequent inappropriate delay of
immunizations.3,4,5 An increasingly more important missed
opportunity may result when providers fail to recognize what
immunizations are needed by the child at the visit. Several factors
contribute to this problem: the addition of new vaccines, an
increasingly complex vaccination schedule, and frequently changing
guidelines. At public health clinics in Los Angeles, nurses'
assessment of the child's immunization status were accurate at only
27% of well-child visits.3 Additionally, a survey of
providers in the public and private sector in Los Angeles showed
significant deficits in their knowledge of the immunization schedule;
private physicians, public physicians, and public nurses were correct
about the immunization schedule on average (64%, 71%, and 78%,
respectively).6
Accurate identification of underimmunized children is needed to
determine which children need vaccination. A previous study found the
accuracy of nurses manually determining the immunization status from a
vaccination record to be low.3 We report a study to
determine the accuracy of manual immunization status assessment
conducted by paraprofessional staff in the Special Supplemental
Nutrition Program for Women, Infants, and Children (WIC) program,
capturing a low-income population known to be at risk of acquiring
vaccine-preventable diseases.7 The use of
nonprofessional staff to identify vaccine-eligible children has
important implications in many settings where staff or resources do not
permit the use of more highly trained health personnel.
Subjects and Setting
The study was conducted in San Diego, California. To improve the
immunization and general health status of WIC participants <2 years
old, the San Diego County Infant Immunization Initiative collaborated
with all 5 WIC agencies in January 1995. The 12 participating WIC sites
in San Diego County serve approximately 20 000 low-income women,
infants, and children. In May 1996, the ethnicity of infants and
children in the study was 54% Hispanic, 18% white, 16% black, 6%
Asian, and 6% other. Clients routinely visit WIC every 2 months to
receive nutrition education and pick up food vouchers; high-risk clients who need closer follow-up visit WIC every month. Vaccination services were not provided on-site.
The San Diego Immunization Program consisted of assessment of
the child's vaccination status and referral to a provider for children
in need of 1 or more vaccinations. A full-time paraprofessional immunization specialist at each WIC site conducted the manual assessment; staff essentially "eyeballed" the vaccination record and decided what vaccinations were due. These WIC staff had no previous
formal training in immunizations but had received 2 to 4 weeks of
training at the start of their job and ongoing monthly training by the
San Diego Immunization Program. There was no official certification
process that staff had to complete. Training consisted of comprehensive
reviews of vaccine-preventable diseases, provider and patient-oriented
educational materials, vaccination algorithms, updates on new
recommendations, and frequently encountered assessment problems. These
materials were available for staff to use, as needed, to conduct the
manual assessment.
Assessment was based on vaccination dates obtained from a personal
vaccination record brought in by the client. An official state-approved
personal vaccination record has been in use for many years in
California and is distributed to all public and private providers in
the state. The parent was requested to bring in the child's personal
vaccination record during every WIC visit. Demographic and immunization
information were collected at the time of the WIC visit and entered
into a computer database from which reminder postcards were generated.
This computer database is referred to as the WIC record in this study.
WIC staff also documented if the parent had brought in the child's
personal shot record at the time of the WIC visit. Even though
immunization dates were entered into the WIC record or database at the
time of the WIC visit, there was no computer-assisted assessment; the computer software did not have the capability to forecast which vaccines were due.
Staff training for immunization record assessment and computer data
entry was provided by the immunization program, which also conducted
quality assurance measures and feedback to staff and provided
instructions and information on referral for children needing
additional doses. Quality assurance was conducted quarterly by staff in
the San Diego Immunization Program and consisted of manually reviewing
a selection of immunization records and vaccination status assessments
that had previously been conducted by the WIC staff and entered into
the WIC record. The San Diego Immunization Program reported
immunization assessment error rates of <5% for each WIC site during
the previous 12 months. Computer-assisted immunization status
assessment was not used to conduct these quality assurance checks.
All immunization activities were started in March 1995. By July 1996, all WIC sites had been fully implementing the intervention for at least
6 months. Full implementation was slower at some WIC sites because of
other factors that may have been occurring at the same time at the WIC
site, including new personnel or staff taking maternity leave.
Outcome Measures
To determine the accuracy of manual immunization status
assessment, immunization status as recorded in the WIC record was compared with immunization status using computerized assessment. Computerized assessment was based on the actual immunization dates contained in the WIC record. Immunization dates in the WIC record were
downloaded into a customized immunization module developed at the
Centers for Disease Control and Prevention to be used
specifically for this analysis. In August 1997, the computerized WIC
record was obtained from WIC. The WIC record contained children 0 to 32 months old who had been seen over a 1-year period at the 12 participating WIC sites in San Diego. Information obtained in the WIC
record included date of birth, client encounter dates, immunization
dates, presence of vaccine record at WIC visit, and outcome of manual
immunization status assessment (up-to-date [UTD] or not UTD). Only
client encounters at which a personal vaccine record had been brought
in to that visit and only children with an age at evaluation between 0 and 24 months at the time of the WIC visit were selected for analysis
in this study.
Eligible children were evaluated for up-to-date status per encounter
(UTD-E). Even though children had multiple encounters, we chose to
evaluate UTD-E rather than UTD status per child. UTD-E best reflected the accuracy of the manual assessment that was conducted
at each patient encounter. Calculations of UTD for age were
based on the child receiving the correct number of doses of
diphtheria-tetanus toxoids and (cellular or acellular) pertussis vaccine (DTP/DtaP), poliovirus vaccine (OPV), measles, mumps, and
rubella vaccine (MMR), Haemophilus influenzae type B
vaccine (Hib), and hepatitis B vaccine (Hep-B) according to the
recommendations of the Advisory Committee on Immunization
Practices8 as modified to be consistent with the algorithm
used to train staff in WIC. Children were classified as UTD if they
were behind schedule but were as UTD as possible, for example, if a
child was not fully immunized but had received the second dose series of DTP/DTAP, OPV, and Hib 1 week prior, the child was not eligible for
another vaccination until the minimal interval had passed. Minimal
intervals for the accelerated schedule were only applied retrospectively.
To evaluate the accuracy of manual immunization status assessment, a
2-way table was constructed showing UTD status based on manual
assessment versus the gold standard of computerized assessment. The
sensitivity and specificity of manual assessment to identify
underimmunization was based on the immunization status assessment as
recorded in WIC. Sensitivity refers to the ability of manual assessment
to identify undervaccinated children. It was defined as the proportion
of truly undervaccinated children also classified as undervaccinated by
manual assessment. Specificity refers to the ability of manual
assessment to identify fully vaccinated children. It was defined as the
proportion of truly fully vaccinated children also classified as fully
vaccinated by manual assessment. Positive predictive value (PPV) refers
to the ability of manual assessment to correctly identify an
underimmunized child. It was defined as the percentage of encounters
classified as not UTD by manual assessment that were actually not UTD.
The negative predictive value (NPV) refers to the ability of manual
assessment to correctly identify a fully immunized child. It was
defined as the probability that an encounter classified as UTD by
manual assessment was actually UTD.
Outcome Measures
The immunization database contained 19 092 children. Most
immunization status evaluations were performed when the client was between 0 to 12 months old: 1 to 6 months (32%); 7 to 12 months (26%); 13 to 18 months (18%); and 19 to 24 months (11%). Of
36 368 patient encounters, immunization status was correctly assessed at 29 078 (80%), incorrectly assessed at 5087 (14%), and unknown at
2203 (6%). Accuracy ranged from 70% to 90% by WIC site. Encounters with unknown status had no outcome (UTD or not UTD) recorded in the WIC
record even though current vaccination dates were available at the time
of the WIC visit. These were encounters in which vaccination dates were
present in the WIC record based on a vaccination record brought to that
encounter but no manual assessment had been conducted. Encounters with
an unknown status were 4.8 times as likely to be not UTD at the time of
the encounter based on current vaccination dates in the WIC record.
Encounters with unknown status were excluded from further analysis.
Of encounters at which the child was actually not UTD as determined by
computerized assessment, the manual assessment process identified a
little over half of the children as not UTD, for a sensitivity of
53.6% (Table 1). Of encounters at which the child was actually UTD as determined by computerized assessment, a
high proportion were identified as UTD by manual assessment, for a
specificity of 89.4%. The PPV of manual assessment was low at
41%. The NPV was much higher at 93%.
TABLE 1
METHODS
Top
Abstract
Methods
Results
Discussion
References
RESULTS
Top
Abstract
Methods
Results
Discussion
References
Accuracy of Manual Immunization Status Assessment Showing UTD Status
Based on Manual Assessment Versus the Gold Standard of Computerized
Assessment (n = 34 165
Encounters)
For all encounters combined, the immunization status of the population calculated by manual assessment compared with computer assessment slightly underestimated the true UTD status (84.2% vs 87.9%, respectively). This relatively well-immunized population led to more fully immunized children being missed by the manual assessment than underimmunized children being incorrectly classified by the manual assessment.
Sensitivity and Specificity Errors
Sensitivity errors were evaluations where the computer assessment showed the child was not UTD and the manual assessment showed the child was UTD. The 3 most common antigens incorrectly assessed were Hib (43%), Hep-B (37%), and DTP (24%). For 77% of the errors, only 1 antigen was incorrectly assessed: Hep-B (31%) > Hib (24%) > polio (10%).
Specificity errors were evaluations where the computer assessment showed the child was UTD and the manual assessment showed the child was not UTD. A simple random sample of 100 encounters where this type of error occurred demonstrated that most of these errors were attributable to WIC staff classifying the child as not UTD before the end of the due month.
| |
DISCUSSION |
|---|
|
Top
Abstract
Methods
Results
Discussion
References
|
|---|
We have shown that manual immunization status assessment conducted by paraprofessionals can be quite specific (~90%) but only moderately sensitive (~55%) in identifying an underimmunized child. This means that many underimmunized children will be missed by these assessments, but only 10% of children will be referred inappropriately for vaccination. Since the UTD status of the population in our study was quite high (>80%) accuracy remained at 80%; in populations with less complete coverage, we expect the accuracy to decline due to more underimmunized children being incorrectly assessed. Indeed, in a study conducted in Los Angeles among a population with low coverage (27% at 2 years of age), accuracy of manual assessment was only 27% when conducted by staff with presumably higher training (nurses).3
The accuracy of manual immunization status assessment varied by WIC site (70%-90%). The manual assessment was conducted by a different person specifically assigned to do this activity at each site. Therefore, the variation in accuracy was probably attributable to differences in levels of expertise in conducting the assessment by the personnel at the sites.
Although the immunization status of children at relatively few encounters (6%) were not manually assessed at the time of their WIC visit (no manual assessment status was recorded in the WIC record), these children were actually more likely to be not UTD according to the gold standard of computerized assessment using the actual vaccination dates. One reason for this may be that these records were more difficult to assess (ie, child may have been on a accelerated schedule), and therefore the staff in WIC were unsure of the immunization status and did not record it in the WIC record.
We do not believe data entry errors played a significant role in affecting the measured accuracy of the assessment; however, we cannot exclude the possibility that data entry errors may have played some role. Data entry errors could occur when WIC staff transcribed the dates from the personal vaccination record into the WIC record. Because an assessment that was done incorrectly would be just as likely to be data entered wrong as an assessment that was done correctly, this potential misclassification would be unlikely to affect our results.
Although the staff in our study had the availability of a vaccine algorithm tool (Baby Shots Ruler Referral Guide) which had been developed by the San Diego Health Department to improve efficiency and accuracy of forecasting UTD status, it was rarely, if ever, used. The tool was a clear plastic sheet that fit over the child's vaccination record and indicated which vaccines were due. Staff reportedly felt comfortable reviewing the vaccine record on their own and felt that the process was faster without the use of the tool. Therefore, vaccination assessment for the purpose of this evaluation was essentially manual, and was achieved by "eyeballing" the vaccination record. It is unknown whether the accuracy of manual assessment could be improved with the use of an assessment tool that would be required to be used at every encounter.
Although many primary care provider sites, immunization clinics, and other sites such as WIC conduct assessment of the child's vaccination status using a computer to determine which vaccinations are due (computer-assisted assessment), the vast majority of locations do not have the computers/software or computer program support to perform this type of assessment. In 1998, only 25% of WIC sites nationwide were reported to use computer-assisted assessment to determine immunization status (Centers for Disease Control and Prevention, unpublished data, 1999). In programs that must rely on manual immunization status assessment, we recommend extensive training and updates on the vaccine schedule, evaluation of assessment accuracy, and periodic feedback of error rates and problems to staff conducting the assessment. In San Diego, program personnel have recommended modification of the already existing computer software so computer-assisted forecasting can be conducted in the future.
Even though periodic quality assurance reviews of the data were conducted by trained staff in the San Diego Immunization Program, the lack of availability of computer-assisted assessment to perform these quality assurance reviews probably resulted in continued problems with the accuracy of the WIC-based assessments being performed. Even for trained staff, the complexity of the immunization schedule makes it difficult for accurate quality control to be conducted using only manual assessment techniques. At least until a computerized assessment module is available for WIC staff, it was recommended that error rates for quality assurance reviews be calculated using computer-assisted assessment.
More widespread use of immunization registries could potentially have a positive impact on improving the efficiency and accuracy of assessing a child's immunization status. Decreasing duplication efforts for data entry would improve the efficiency of the process and ensure accuracy of the immunization history (less transcription errors). In addition, standardized algorithms to determine UTD status could be incorporated into the forecasting abilities of a registry.
Accurate identification of underimmunized preschool-aged children is needed if we are to improve vaccination coverage. Paraprofessional staff may have difficulty interpreting vaccination records without rigorous quality control to ensure accuracy of the process and/or computer-assistance to assist in conducting the assessment.
| |
ACKNOWLEDGMENTS |
|---|
We would like to thank Roger Friedman for his excellent data analysis skills and Wendy Wang for her assistance with data collection and analysis.
| |
FOOTNOTES |
|---|
a Formerly of the San Diego Immunization Program, San Diego, California.
This was presented, in part, at the Ambulatory Pediatric Association Meeting; May 3, 1998; New Orleans, LA.
Received for publication Nov 9, 1999; accepted Feb 8, 2000.
Reprint requests to (A.S.) MS E-52, CDC, Atlanta, GA 30333. E-mail: ams7{at}cdc.gov
| |
ABBREVIATIONS |
|---|
WIC, Special Supplemental Nutrition Program for Women, Infants, and Children; UTD, up-to-date; UTD-E, up-to-date status per encounter; DTP/DtaP, diphtheria-tetanus toxoids and (cellular or acellular) pertussis; OPV, poliovirus vaccine; MMR, measles, mumps, and rubella; Hib, Haemophilus influenzae type b; Hep-B, hepatitis B; PPV, positive predictive value; NPV, negative predictive value.
| |
REFERENCES |
|---|
|
Top
Abstract
Methods
Results
Discussion
References
|
|---|
United States, 1996. MMWR Morb Mortal Wkly Rep. 1997;46(41):963-969
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||
|
||
Home | My Pediatrics | Journal Information | Current Issue | Past Issues | Subscriptions & Services | Contact Us Click here for faster international access © 2000 American Academy of Pediatrics. All rights reserved. |
||
|
||
