pediatrics
July 2003, VOLUME112 /ISSUE 1

The Immunization Delivery Effectiveness Assessment Score: A Better Immunization Measure?

  1. James H. Glauber, MD, MPH
  1. From the Neighborhood Health Plan, Boston, Massachusetts

Abstract

Background. Childhood immunization measures, such as the Health Employer Data Information Set (HEDIS) or the National Immunization Survey, assess the percentage of children up-to-date for a specified series of vaccinations. In particular, the HEDIS assesses immunization delivery to children enrolled in managed care organizations (MCO). Such measures do not assess the timeliness of immunization delivery with reference to recommended age standards. To achieve maximal protection against vaccine-preventable diseases, children should receive all immunizations within recommended age intervals—fully “on-time.”

Objective. The Immunization Delivery Effectiveness Assessment (IDEA) is a novel immunization measure that assesses, on a continuous scale, the timeliness of administration of each vaccination with reference to recommended age intervals. Specifically we ask: 1) Do absolute immunization rates differ between HEDIS and IDEA? 2) Does relative MCO performance differ when assessed by the 2 performance measures? 3) How well do MCOs perform relative to the standard of fully on-time immunization? The health services implications of using the timeliness standard to assess childhood immunization delivery is discussed.

Methods. A vaccine-dose IDEA score was developed for each of the 14 vaccination events in the 4:3:1:3:3 combination series (4 diphtheria-pertussis-tetanus:3 polio:1 measles-mumps-rubella:3 Haemophilus influenzae type B:3 hepatitis B). Assessing the actual age of administration with reference to the recommended age of administration generates the vaccine-dose IDEA score. A child’s composite IDEA score is obtained by averaging the 14 vaccine-dose IDEA scores. These composite IDEA scores, when averaged among children sampled within the MCO, constitute the MCO’s immunization score.

Setting. Retrospective analysis of childhood immunization datasets from a convenience sample of 6 MCOs in 5 states.

Results. HEDIS rates ranged from 57% to 75%. IDEA scores ranged from 80% to 90%. Relative rankings of MCO immunization performance were different using HEDIS rates and IDEA scores, respectively. At most, 16% of children in any of these MCOs received all of their immunizations fully on-time. From 47% to 77% of children experienced at least 3 delayed immunizations.

Conclusions. An immunization measure based on timeliness of administration yields both absolute and relative differences in MCO childhood immunization performance when compared with HEDIS rates. By assessing delivery of each component vaccination, the IDEA score permits more detailed analysis of immunization patterns within an MCO and focuses improvement efforts.

  • immunization
  • measurement
  • quality
  • managed care

Childhood immunizations are among the most cost-effective health care interventions.1 Given their value, assessing the quality of childhood immunization delivery is crucial to maximize individual and population protection against vaccine-preventable diseases (VPDs). Immunization status of young children in the United States is predominantly assessed through up-to-date combination measures, such as those articulated by the Health Employer Data Information Set (HEDIS) and the National Immunization Survey (NIS).24 These performance measures assess immunization adequacy with a categorical determination of whether a child has received all of a specified series (ie, 4:3:1:3:3) of immunizations at a given age: 4 diphtheria-pertussis-tetanus (DPT), 3 polio, 1 measles-mumps-rubella (MMR), 3 Haemophilus influenzae type B (HIB), and 3 hepatitis B (HBV). The HEDIS immunization measure, which assesses up-to-date status at 2 years old, is the predominant preventive health quality measure for children enrolled in managed care organizations (MCOs).

However, up-to-date combination measures may not adequately assess the delivery of recommended early childhood immunizations for the following reasons:

  1. Combination measures consider a child as not up-to-date whether they have missed 1 or all recommended immunizations.

  2. Overall up-to-date rates are determined by the specific immunization with the lowest delivery rate.4 For example, the most recent NIS up-to-date rate for the 4:3:1:3:3 series was 74%.3 However, the lowest coverage rate for a specific vaccination was 82% for the DPT series; the other vaccination coverage rates ranged from 89% to 93%.3 The DPT series coverage rate, in turn, is largely driven by failure to deliver the fourth, least biologically important, dose.4

  3. By assessing immunization status at 24 months, the HEDIS up-to-date measure overlooks the timeliness of immunization delivery. According to the recommendations of the American Academy of Pediatrics (AAP) and the Advisory Committee on Immunization Practices (ACIP), the primary immunization series should be completed no later than 18 months of age.5 HEDIS, however, assesses immunization status categorically at 24 months. Initiation and subsequent delivery of recommended immunizations may be markedly delayed with reference to recommended age intervals yet completed by 24 months. Despite this, children experiencing such delay would be considered fully up-to-date by HEDIS criteria. Delayed delivery of recommended immunizations extends a young child’s period of vulnerability to VPD.6 Immunization measures that fail to assess timeliness may, as an unintended consequence, tolerate inefficient immunization practices such as nonsimultaneous administration of needed doses, missed opportunities, acceptance of false contraindications, and nonuse of recall/reminder systems.79

The Immunization Delivery Effectiveness Assessment (IDEA) score is a novel immunization measure that assesses the timeliness of delivery of each recommended early childhood immunization with reference to AAP/ACIP-recommended age intervals. A vaccine-dose IDEA score for each recommended early childhood immunization is generated on a continuous scale from 0 to 100. Averaging vaccine-dose IDEA scores generates a child’s composite IDEA score. By creating a continuous scale for childhood immunization, the IDEA score avoids the bluntness of the HEDIS measure, where a single missed immunization renders the child “not up-to-date.” Moreover, the IDEA score assesses immunization delivery against the performance standard of on-time delivery. Children considered up-to-date by HEDIS criteria have the effectiveness of immunization delivery quantitatively assessed with reference to this higher performance standard. By assessing the timeliness of immunization delivery on a continuous scale, we hypothesize that the IDEA score offers a more sensitive measure of immunization performance than HEDIS and other up-to-date measures.

This paper describes the methodology for generating the IDEA score, which is derived from databases used to generate HEDIS rates and does not require additional data collection. In creating an alternative immunization quality measure we address 3 central questions:

  1. Do absolute HEDIS scores and IDEA scores differ within the same MCO?

  2. Do the relative rankings between MCOs differ when assessed by HEDIS rates and IDEA scores respectively?

  3. Would immunization improvement strategies differ if immunization status were assessed against the standard of “on time” delivery?

METHODS

Development of IDEA Score

The IDEA score is strictly based on the AAP/ACIP-recommended age intervals for immunization in the first 2 years of life. A vaccine-dose IDEA score is generated for each recommended immunization that comprises the HEDIS combo 1 measure (4 DTP, 3 polio, 1 MMR, 2 HIB, 3 HBV). The vaccine-dose IDEA score ranges from 0 to 100 for each recommended immunization; the child’s composite IDEA score is the average of vaccine-dose IDEA scores. The vaccine-dose IDEA score is generated in the following way:

  1. For each vaccine dose during the first 2 years, the recommended age range-of-administration is defined based on AAP/ACIP criteria. The minimal age of administration if the dose is the first in a series (ie, 42 days for DPT1, 365 days for MMR) or minimal intervaccination intervals if a booster dose (ie, at least 182 days between DPT3 and DPT4) are also specified (Table 1).

  2. For each immunization the actual age-of-administration (a-AOA) in days-of-life (DOL) is determined by calculating the date of administration with reference to the child’s date of birth.

  3. If the immunization is delivered within the recommended age range-of-administration a vaccine-dose IDEA score of 1 is assigned. If the vaccine is not given by 24 months or is given too early, a vaccine-dose IDEA score of 0 is assigned.

  4. To align with ACIP/AAP recommendations, the latest acceptable age of administration (L-AOA) is defined as the DOL after which the immunization is considered delayed. We grant a 14-day grace period for immunizations recommended at a specific age (ie, DPT1 No. 1 recommended at 2 months old). For immunizations whose recommended age of administration is an interval (ie, MMR at 12–15 months), the L-AOA is defined as the end of the interval; no grace period is granted. If the immunization is delivered later than the L-AOA, but by 24 months, a vaccine-dose IDEA score between 0 and 1 is assigned based on the following calculation: Embedded Image, where a-AOA = a-AOA, measured in DOL, actual age of administration L-AOA = L-AOA, measured in DOL, latest acceptable age of administration 730 = DOL at the second birthday. For delayed immunizations, the vaccine-dose IDEA score is simply the ratio of the number of days overdue when the vaccine is delivered and the number of potential overdue days (if the vaccine is not delivered by 24 months). The inverse of this ratio is taken so that the vaccine-dose IDEA score is scaled so that “1” represents optimal performance and “0” indicates either too early or nondelivery of the vaccine. For example, if the MMR vaccine is delivered at 18 months (548 DOL) the vaccine-dose IDEA score is calculated as: Embedded Image L-AOA = 455 DOL a-AOA = 548 DOL a-AOA − L-AOA = 548 − 455 = 93 days late Embedded Image

  5. The composite IDEA score for each child is obtained by summing and averaging the vaccine-dose IDEA scores. The composite IDEA score is multiplied by 100 so that it is scaled from 0 to 100.

  6. The IDEA score for an MCO, in turn, is obtained by averaging the composite IDEA scores for the children comprising the sample pulled to generate HEDIS childhood immunization rates.

TABLE 1.

Recommended Age of Administration and Intervals Between Boosters for IDEA Score

Analysis of HEDIS Rates and IDEA Score

We obtained HEDIS databases of Medicaid-enrolled children from a convenience sample of 6 MCOs in 5 states (Massachusetts, Rhode Island, Pennsylvania, California, and Hawaii). These MCOs participated in improvement collaboratives targeting the pediatric Medicaid managed care population, called Best Clinical and Administrative Practices, sponsored by the Center for Health Care Strategies.10 Member-identifying information was deleted from the databases before analysis.

We calculated the IDEA score assessing the delivery of 4 DTP, 3 polio, 1 MMR, 3 HIB, and 3 HBV vaccines from the immunization databases used to calculate MCO HEDIS scores. Within each MCO we compared the IDEA score to its HEDIS combo 1 rate (4 DPT, 3 polio, MMR, 2 HIB, 3 HBV). We adhered to the more stringent requirement of 3 to 4 HIB vaccines in order that the IDEA score strictly aligns with ACIP/AAP recommendations. We then ranked the 6 MCOs based on their HEDIS rates and IDEA scores, respectively, to determine if relative rankings changed when assessed by each measure.

Timeliness of Immunization on the Child Level

We next describe how the IDEA methodology can be used to assess the effectiveness of immunization delivery with reference to recommended age intervals. Immunization effectiveness can be viewed from 2 perspectives: that of the child or a specific immunization (or immunization series). From the child’s perspective we describe the percentage of children within an MCO that received all immunizations on time, received no immunizations, or had 1 or more immunization delays.

Timeliness of Immunization at the Immunization Level

From the immunization perspective, the immunization series that are most susceptible to the greatest magnitude of delayed delivery are described. Further specificity is provided regarding which vaccine-dose within a immunization series is least efficiently delivered.

RESULTS

Analysis of HEDIS Rates and IDEA Score

Figure 1 presents IDEA scores and HEDIS rates, respectively, for the 6 MCOs. In all cases the MCO IDEA score was higher than its HEDIS rate. The range of increase was 9 points for the MCO with the highest HEDIS rate (75%) to 25 points for the MCO with the lowest HEDIS rate (57.7%). The range of MCO IDEA scores (8.7) was smaller than the range of MCO HEDIS rates (17.2).

Fig 1.

Comparison of Health Plan Immunization Performance Assessed by HEDIS and IDEA Score.

The relative performance of the MCOs differed when assessed by HEDIS rates and the IDEA score. The MCO with the highest HEDIS rates (Fig 1) ranked fourth by the IDEA score; conversely, the fourth-ranked MCO by HEDIS rates was the top performing MCO by the IDEA score (Fig 1).

Timeliness of Immunization at the Child Level

Figure 2 presents the percentage of children in each MCO that were fully immunized on time, received no immunizations, or had delayed immunizations. Only 16.3% of children in the best performing MCO received all the recommended immunizations on time. From 47% to 77% of Medicaid-enrolled children in these MCOs experienced at least 3 delayed immunizations.

Fig 2.

Distribution of age-appropriate and age-inappropriate immunization.

This analysis reveals that few children in these MCOs went completely unimmunized (Fig 2). No more than 7.7% of children in any of these MCOs went completely unimmunized, whereas from 25% to 42.3% of children were not up-to-date.

The MCO with the highest IDEA score immunized 8% of children fully on time and 42.1% of children had >3 delayed immunizations. The lowest performing MCO immunized only 0.42% of children on time and 66.2% of children had >3 delayed immunizations (Fig 2).

Timeliness of Immunization at the Immunization Level

Figure 3 summarizes immunization effectiveness by vaccine series for MCO E, with the lowest IDEA score. It demonstrates adequate on-time delivery of the polio series, hepatitis series, and MMR, but predominantly delayed delivery of components of the DPT series (66%) and HIB series (98%). Closer examination of vaccine-dose IDEA scores from MCO E (Fig 4) reveals an uneven pattern of HIB delivery, with HIB1 being delivered more efficiently (IDEA score = 89) than HIB2 (47), HIB3 (40), and HIB4 (47). By contrast, MCO D, with the highest overall IDEA score, did not demonstrate as marked a drop-off in efficient delivery of HIB booster doses (HIB1–4 scores: 98, 95, 89, 72).

Fig 3.

Timeliness by vaccine series.

Fig 4.

Vaccine-dose IDEA scores for MCO E.

DISCUSSION

Comparison of HEDIS and IDEA Scores

The IDEA score offers a novel assessment of childhood immunization status based on timeliness of individual vaccine delivery. The IDEA score references a different performance standard for immunization delivery, on-time delivery, rather than the HEDIS up-to-date standard. This alternative standard yields different absolute and relative performance scores when compared with HEDIS.

Although IDEA scores were consistently higher than HEDIS rates, only a small minority of children in any of these MCOs received all their immunizations fully on-time. Additionally, in most MCOs, a majority of children experienced at least 3 delayed immunizations. This apparent paradox arises because the composite IDEA score is a continuous scale derived by averaging vaccine-dose IDEA scores. Each of the 14 immunizations comprising the 4:3:1:3:3 series contributes only 6.7% to the composite IDEA score. Thus, a single missed or delayed immunization has limited impact on the overall score. In contrast, a single missed immunization determines the entire HEDIS score by rendering the child not up-to-date.

Additional Information Provided by IDEA Score

Most children not up-to-date by HEDIS criteria are nonetheless partially immunized. At most, 7.7% of children in any of the MCOs went completely unimmunized. The IDEA score credits partial immunization, whereas HEDIS does not. Thus, MCO IDEA scores eclipse HEDIS rates by counting the immunizations received by partially immunized children. Partially immunized children are partially protected, and in some cases almost fully protected, against VPDs.4 The HEDIS score for these children thus presents a misleading assessment of their vulnerability to VPD.

The biological outcome of immunization is protection against VPD. A valid immunization performance measure should accurately assess this outcome. In a recent discussion of immunization measurement in this journal, Rodewald et al4 argued: “When immunization coverage is reported as a combination series measure, the coverage of specific vaccines cannot be determined which in turn creates a problem in the translation of the process of vaccination into the outcome of protection from VPD.”

By giving specific vaccine doses equal weights, the IDEA score may offer a better assessment of protection against VPD, on both the individual and population level, than the HEDIS measure.

This is not strictly an academic debate, as MCO effectiveness of care relating to pediatric preventive services is primarily assessed through HEDIS childhood immunization rates. Both absolute and relative MCO performance scores changed when assessed by the IDEA score. The National Commission for Quality Assurance encourages consumers and purchasers to utilize HEDIS performance measures to inform choice among MCOs.11 Because quality measurement impacts both the marketplace and the resources allocated to performance improvement, the divergent profiles yielded by alternative immunization standards merits concern.12 For example, MCO D may undertake improvement activities in response to its HEDIS rate of 67.7. However, it may chose to devote its limited resources to improve quality elsewhere in response to its IDEA score of 90. This divergence in both absolute and relative performance scores should stimulate vigorous debate over the most suitable method for measuring immunization performance.

IDEA Score as a Diagnostic Tool for Performance Improvement

In this debate, 2 criteria for quality measures are paramount: 1) intelligibility to relevant audiences and, 2) the ability to guide performance-improvement strategies. The HEDIS measure is more readily understandable than the IDEA score. The HEDIS measure is a simple rate, whereas the IDEA score is an index. Indices are inherently more complex than rates but may become meaningful with familiarity (consider the body mass index as a measure of obesity). Unlike a simple rate, however, the timeliness methodology behind the IDEA score facilitates a diagnostic assessment of immunization delivery that can direct quality improvement activities. For example, MCO E may chose to send immunization reminder cards regarding the fourth dose of DPT and HIB to parents of 15-month-olds after noting that these doses had particularly low vaccine-dose IDEA scores. The HEDIS measure, in contrast, neither describes how efficiently up-to-date children were immunized nor adequately describes the immunization status of children who were not up-to-date.

Widespread adoption of the HEDIS measure may, in fact, have the unintended consequence of discouraging proven strategies to improve immunization delivery.15 By defining quality as having children up-to-date by 2 years old, rather than fully on time, the HEDIS measure tolerates provider behaviors known to contribute to immunization delay (missed opportunities, accepting false contraindications, nonsimultaneous administration of due vaccines).8,1315 Having a 6-month window with which to catch up delayed children yet remain fully compliant with the immunization standard may lessen the imperative to improve these practices.

The timeliness standard, in contrast, quantitatively assesses and penalizes practices that engender immunization delay. The IDEA score gives particular emphasis to timely initiation of immunizations because markedly delayed onset lowers IDEA scores both for the late initial dose and subsequently delayed booster doses. This relative emphasis on timely initiation of immunizations is appropriate because early infancy is a period of heightened vulnerability to VPD.

Limitations of the IDEA Score

The composite IDEA score weighs the 14 individual vaccinations that comprise the series equally. The DPT series, comprising 4 doses, has 4-fold greater contribution to the composite IDEA score than MMR, comprising 1 dose, although each vaccine confers protection against 3 VPD. This bias toward vaccine series with more doses is also inherent in the HEDIS measure. The composite IDEA score could be calculated so that each vaccine series, rather than each vaccine dose, contributes equally to the overall score. For example, the composite IDEA score for MCO E, whose vaccine-dose IDEA scores appear in Fig 2, is 83.6 when each vaccine series is weighted equally (as compared with a composite IDEA score of 80.9 when vaccine doses are weighted equally).

Another bias of the IDEA score derives from the methodology for generating the vaccine-dose IDEA score. Two vaccines delayed by the same absolute magnitude of days (for example, if both the first dose of DPT and MMR are delayed 90 days) will yield different vaccine-dose IDEA scores. In this example, the vaccine-dose IDEA score for DPT No. 1, 86.3, exceeds that for MMR, 67.2, despite the same absolute magnitude of delay. This result derives from the different denominators used to generate the vaccine-dose score. The denominator is derived from the number of days that a vaccine could potentially be delayed relative to the second birthday. However, the vaccine-dose IDEA scores for DPT Nos. 2 and 3 will also be lowered by the delayed delivery of DPT No. 1, amplifying the impact of delayed initiation on the composite IDEA score.

The IDEA score references a higher performance standard for immunization delivery: fully on-time immunization. From this standard, MCO performance was deficient, with at most 16% of children being immunized fully on-time and 47% to 77% experiencing at least 3 delayed immunizations. Enhanced focus on these diagnostic metrics, provided by the timeliness methodology, might encourage MCOs and providers to improve the aforementioned practices contributing to immunization delay.8,15

Prior Research on Immunization Measurement

Key policymakers have advocated for a different strategy of assessing childhood immunization status. In July 2000, the National Scientific Panel on Immunization Measurement Standards produced recommendations for harmonizing assessments of immunization coverage between HEDIS and the NIS (T. A. Lieu, personal communication, July 2000). Among their 6 recommendations, the panel agreed to emphasize single-antigen measures and reduce reliance on combination measures. Their rationale was that “single antigen measures were more sensitive to variation in delivery of new vaccines, and potentially less misleading, than combination measures. Combination measures were felt to present a falsely low sense of immunization rates” (T. A. Lieu, personal communication, July 2000).

Researchers have expressed immunization status in a manner that is more responsive to timeliness of vaccination. Davis et al16 defined a “cumulative up-to-date” measure at 12 and 24 months by “assessing up-to-date status for each day over a given time interval and summing the days that were spent up-to-date.” Rodewald et al17 developed a similar metric, cumulative number of days not up-to-date, in a randomized trial of outreach and provider prompting to improve immunization status. Although these create continuous measures of immunization status, they nonetheless assess immunization status through daily application of the combination up-to-date standard rather than at 24 months.

Vivier et al18 assessed early childhood immunization status with an “interval method” by which an immunization was considered valid only if it adhered to minimal age and interval recommendations of the AAP and the ACIP. By including this timeliness element to the assessment standard, overall up-to-date rates declined from 80%, using counts of the 4:3:3:1 series vaccinations received, to 53%.20 However, the aforementioned measurement strategies retain categorical up-to-date status as the assessment standard. More recently, Luman et al19 used NIS data to describe the timeliness of delivery of vaccine doses and vaccine series with reference to recommended age intervals. Timely delivery of immunizations varied by vaccine dose, ranging from 45.4% (HIB4) to 93.1% (HBV1). Similar to our sample of MCO-enrolled Medicaid children, a distinct minority (12.7%) of children in the NIS survey received all doses of the 4:3:1:3:3 series fully on-time.19 Luman and colleagues19 concluded that focusing on vaccination status at 2 years fails to detect the extent of underimmunization during the first 2 years of life, but did not propose an alternative measurement strategy. The current work extends this descriptive analysis by developing an alternative measure on a continuous scale based on timeliness of administration for each recommended early childhood immunization.

Study Limitations

The findings regarding the performance characteristics of the IDEA score are based on analysis of HEDIS data from a convenience sample of 6 MCOs enrolling Medicaid recipients. Thus, my conclusions must be considered preliminary. The ranges of HEDIS scores for these plans were fairly narrow (58% to 75%). The IDEA scores for these plans were consistently higher than their HEDIS rates. It bears emphasis that HEDIS combo 1 requires only 2 HIB doses, whereas the IDEA score assesses 3 to 4 HIB doses. Because each measure has different specifications, strict comparison of absolute rates must be interpreted cautiously. Had the HEDIS standard required 3 or 4 doses of HIB, it is likely that the MCO HEDIS rates would have been lower and, by consequence, the divergence in MCO IDEA scores and HEDIS rates would have been greater.

It is uncertain if IDEA scores would be higher for MCOs with substantially higher or lower HEDIS rates. Conceivably, MCOs providing care to children with a greater number and magnitude of immunization delay, on the one hand, and more missed immunizations, on the other, may have lower IDEA scores than HEDIS rates. MCOs with very high HEDIS rates may also have lower IDEA scores if up-to-date children nonetheless experienced substantial immunization delay. Larger studies from more diverse settings will be needed before firm conclusions are possible regarding the relative performance characteristics of the HEDIS standard and IDEA score. These limitations notwithstanding the IDEA score yields a different assessment of the immunization status of young children and, in doing so, may stimulate more efficient clinical practices that reduce early childhood underimmunization.

Acknowledgments

This work was made possible, in part, through a grant from the Center for Healthcare Strategies. Kara Costello provided assistance with preparation of the manuscript.

Footnotes

    • Received June 12, 2002.
    • Accepted February 27, 2003.
  • Address correspondence to James H. Glauber, MD, MPH, Neighborhood Health Plan, 253 Summer St, Boston, MA 02210. E-mail: jim_glauber{at}nhp.org
HEDIS, Health Employer Data Information Set, MCO, managed care organization, IDEA, Immunization Delivery Effectiveness Assessment, VPD, vaccine-preventable disease, NIS, National Immunization Survey, DPT, diphtheria-pertussis-tetanus, MMR, measles-mumps-rubella, HIB, Haemophilus influenzae type B, HBV, hepatitis B, AAP, American Academy of Pediatrics, ACIP, Advisory Committee on Immunization Practice, a-AOA, actual age-of-administration, DOL, days-of-life, L-AOA, latest acceptable age of administration

REFERENCES