Impact of Electronic Health Record-Based Alerts on Influenza Vaccination for Children With Asthma
OBJECTIVE: The goal was to assess the impact of influenza vaccine clinical alerts on missed opportunities for vaccination and on overall influenza immunization rates for children and adolescents with asthma.
METHODS: A prospective, cluster-randomized trial of 20 primary care sites was conducted between October 1, 2006, and March 31, 2007. At intervention sites, electronic health record-based clinical alerts for influenza vaccine appeared at all office visits for children between 5 and 19 years of age with asthma who were due for vaccine. The proportion of captured immunization opportunities at visits and overall rates of complete vaccination for patients at intervention and control sites were compared with those for the previous year, after standardization for relevant covariates. The study had >80% power to detect an 8% difference in the change in rates between the study and baseline years at intervention versus control practices.
RESULTS: A total of 23 418 visits and 11 919 children were included in the study year and 21 422 visits and 10 667 children in the previous year. The majority of children were male, 5 to 9 years of age, and privately insured. With standardization for selected covariates, captured vaccination opportunities increased from 14.4% to 18.6% at intervention sites and from 12.7% to 16.3% at control sites, a 0.6% greater improvement. Standardized influenza vaccination rates improved 3.4% more at intervention sites than at control sites. The 4 practices with the greatest increases in rates (≥11%) were all in the intervention group. Vaccine receipt was more common among children who had been vaccinated previously, with increasing numbers of visits, with care early in the season, and at preventive versus acute care visits.
CONCLUSIONS: Clinical alerts were associated with only modest improvements in influenza vaccination rates.
Receipt of influenza vaccination is a benchmark of quality care for children and adolescents with asthma and other high-risk conditions. Despite recommendations from both the Advisory Committee on Immunization Practices (ACIP) and the American Academy of Pediatrics,1 rates throughout the nation remain low for children with asthma and have been found to be even lower for teens.2–4 In this setting, the Centers for Disease Control and Prevention called on the medical and public health communities to develop plans for improving vaccine delivery for the 2006–2007 season.5 The American Academy of Pediatrics emphasized that improving vaccination rates among children with high-risk conditions and extending vaccination throughout the influenza season were 2 of the highest priorities.6
To address low vaccination rates, strategies may target public demand, access to vaccines, or vaccine delivery in medical settings.7 For the latter, successful interventions have included use of reminder/recall systems,8–11 year-round scheduling of visits for influenza vaccine,12 and feedback to providers regarding rates.13 Even with labor-intensive measures, however, improvements in rates often have been modest. Electronic health records (EHRs) are increasingly common in ambulatory pediatric settings14 and are capable of identifying children at high risk and reminding providers that influenza vaccine is due.15 However, no published randomized trial has evaluated the benefit of EHR-based clinical alerts at the point of pediatric primary care for improving influenza vaccine delivery.
This cluster-randomized, multisite study using an EHR-based clinical alert system was designed to fill this gap in the understanding of strategies to improve influenza vaccine rates by focusing on those with asthma. The study had 2 specific aims, that is, to assess the impact of EHR-based alerts for influenza vaccination on captured opportunities for influenza vaccination among children and adolescents with asthma between 5 and 19 years of age in the pediatric primary care setting and to improve up-to-date influenza vaccination rates in this population.
Setting and EHR
This study was conducted at the first 20 practices in the Children's Hospital of Philadelphia Pediatric Research Consortium that implemented the ambulatory EHR EpicCare (Epic Systems, Verona, WI). All practices that were approached agreed to participate. The Pediatric Research Consortium is a multistate, hospital-owned, primary care practice-based, research network including >235000 children and adolescents. Study practices included 4 urban teaching practices, where <35% of patients had private insurance, and 16 primarily suburban practices not involved in resident teaching, where >80% of children were privately insured.
Study Design and Patient Population
We conducted a prospective, 20-primary care site, cluster-randomized, decision-support trial between October 1, 2006, and March 31, 2007 (the study period). For each site, captured opportunities for influenza vaccination and influenza vaccination rates were compared with those for the same period in the previous year. Subjects included all children with asthma who were >60 months of age as of October 1, 2005, for the baseline year and October 1, 2006, for the study year and <20 years of age as of March 31, 2006, and March 31, 2007, for the baseline and study years, respectively. Children <60 months of age were excluded because of recommendations for universal vaccination and the use of alerts at all intervention and control sites for this population.
Children with asthma were identified on the basis of International Classification of Diseases, Ninth Revision codes for asthma on their chronic problem list or visit diagnoses. We included mild intermittent asthma, mild persistent asthma, moderate persistent asthma, and severe persistent asthma, consistent with national guidelines at the time.16 Because the intervention required an office visit with a clinician to be effective, all children and adolescents in the study were required to have ≥1 visit during the influenza season.
Following a previously published approach, influenza vaccine alerts appeared prominently at the top of the computer screen, in bold and highlighted text, whenever a patient encounter was opened within the EHR for a study subject at an intervention site who was due for this vaccine; a link was provided to simplify vaccine ordering.17 According to ACIP recommendations for children with asthma, alerts prompted the use of trivalent influenza vaccine, conformed to the need for 2 doses ≥1 month apart for children <9 years of age who had not been vaccinated previously and 1 dose for others, and did not appear for children with egg allergy.1 The clinician, in consultation with the family, decided to order vaccine. In addition, a 30-minute, Internet-based, slide presentation describing influenza morbidity and mortality rates, as well as current recommendations and contraindications for influenza vaccination, was delivered by 2 expert primary care pediatricians to practicing clinicians and interested staff members at all sites. At intervention sites only, the presentation included information on use of the alert system for children with asthma. All clinicians were e-mailed a copy of the presentation that their site received.
Practices were divided into 2 groups on the basis of practice structure and patient demographic features, that is, urban resident-teaching practices and non–resident-teaching practices. Practices within these groups were assigned randomly to receive either the influenza vaccine clinical alerts or routine care during the study year.
This study was designed to test the effects of influenza vaccine alerts on 2 main outcomes, that is, (1) rates of captured opportunities for influenza vaccination (visit-level analysis) and (2) up-to-date influenza vaccination among patients with asthma. Rates of captured influenza vaccination opportunities measured the proportion of all eligible visits at which influenza vaccine was administered (if due). Eligible visits included preventive and acute care visits that were conducted by physicians, nurse practitioners, or trainees (residents or fellows).
Children were considered up to date if they were fully vaccinated according to ACIP guidelines with 1 or 2 doses, depending on age and previous vaccination status. In a secondary analysis, the proportion of children receiving at least 1 dose of influenza vaccine was assessed.
Covariates were chosen to control for demographic, clinical, and influenza vaccine-specific factors potentially influencing the effect of clinical alerts on influenza vaccination (Tables 1 and 2). The presence of chronic medical problems except for asthma was determined by using a previously published list.18
To measure the effectiveness of the site randomization, the proportions of the population with each covariate of interest were compared between intervention and control practices for the study year, by using a model-based approach with robust variance estimates that accounted for clustering. In particular, we used logistic regression in Stata 9 (Stata, College Station, TX) for these analyses, with the robust option that implements the sandwich estimator.19 Because the performance of each practice was compared with that of the same practice during the previous year, the stability of the populations at control and intervention practices was assessed.
The children included in our study might be the same or different for the baseline and study years. The analysis accounted for clustering within site, which addresses the lack of independence of subjects within each practice. Because the children did differ from year to year, we treated this analysis as a repeated cross-sectional design, which provides conservative (larger) estimates of variance.
Unadjusted odds ratios, with 95% confidence intervals (CIs), for the association of each covariate with receipt of vaccine (at both the patient and visit levels) were determined by using logistic regression with the selected covariate as the only independent variable and adjusting variances to account for the clustered design.20 Following this approach, the interaction of the intervention with time was used to test the overall effectiveness of the alerts.
In a multivariate logistic regression analysis that accounted for the clustered design, changes over time in the proportions of captured immunization opportunities at visits or in the proportions of patients with up-to-date vaccinations at intervention and control sites were adjusted for included covariates. We also conducted a secondary analysis with patient receipt of a single dose of influenza vaccine as the outcome. For both patient-level and visit-level analyses, linear models confirmed that results were robust to model specification. To assess the impact of shortages on the intervention, a secondary analysis limited to visits on days when vaccination was actively occurring (≥2 doses of vaccine were administered at a given site) was performed. Because previous work demonstrated that alerts for routine pediatric vaccines were effective in resident-teaching practices,17 we conducted an analysis at the patient level that was limited to these sites and then we tested the significance of the interactions of the intervention with time and practice type (teaching versus nonteaching), to confirm the results.
Key comparisons from logistic regression analyses were transformed into standardized proportions by using marginal standardization.21 These standardized proportions are interpreted as the expected outcome (immunization) if the entire sample were alternatively subjected to the intervention or monitored as a control group. This analysis controls for all covariates. Bias-corrected CIs were estimated by using bootstrap resampling. The study had >80% power to detect an 8% difference in the change in rates between the study and baseline years at intervention versus control practices. Power calculations were performed through simulation and were based on both the number of sites and the number of subjects at each site. The anticipated effect size and the variability across both subjects and sites were considered in the process. Assumptions used in the simulation were based on both pilot data from our network and published literature findings.
Analyses were performed by using Stata 9, Stata 10, and SAS 9.1 (SAS Institute, Cary, NC). The Children's Hospital of Philadelphia institutional review board approved the study.
Study Population and Cluster Randomization
A total of 5338 children with asthma were included at control sites for the baseline year, with 5809 for the study year. At intervention sites, 5329 children with asthma were included in the baseline year and 6110 in the study year (Table 1). Only 65% of children with asthma who were seen for any reason at a study site during the previous year had ≥1 sick or well visit during the study year influenza season. A majority of children were male, were between 5 and 9 years of age, had received influenza vaccine previously, and had private insurance (Table 1). Mild intermittent asthma was most common among those with documented severity.22
For the study year, no significant differences in patient characteristics were observed between control and intervention sites (Table 1). A nonsignificant 8% difference was seen in the proportions of black children at intervention versus control sites. Although 2 urban teaching practices with large proportions of black patients were in each arm of the study, this difference arose because, by chance, the larger practices were assigned randomly to the intervention group. Overall, the characteristics of the study populations in control and intervention practices were stable between the baseline and study years.
Visit-Level Characteristics and Outcomes
Visit-level characteristics are presented in Table 2. There were no significant differences in visit characteristics between the intervention and control sites for the study year, and the population accruing visits was stable over time.
Overall rates of captured opportunities for vaccination increased 3.8%, from 12.3% to 16.1%, at control practices and 4.8%, from 14.4% to 19.2%, at intervention sites, a difference of 1% (95% CI: −2.4% to 4.9%). With standardization for selected covariates, captured vaccination opportunities increased from 14.4% to 18.6% at intervention sites and from 12.7% to 16.3% at control sites, a 0.6% (95% CI: −1.9% to 2.5%) greater improvement. When the analysis was limited to visits on days when individual visit sites administered ≥2 doses of influenza vaccine (to explore the impact of shortages), the impact of the intervention was similar.
We also examined covariates affecting the receipt of influenza vaccine at visits (Table 3). Factors significantly associated with improved vaccination in both the crude and adjusted analyses included care during the study versus baseline year, documented asthma severity on a child's problem list, care at urban resident-teaching practices, visit for preventive versus acute care, and previous receipt of influenza vaccine. Girls, nonblack children, children with fever, and children with visits later in the influenza season were significantly less likely to be immunized. The monthly decline in vaccination was especially steep, with rates of captured opportunities decreasing steadily from 28% in October to 4% in March at control and intervention sites during the study year. The alerts resulted in minor improvements in late-season captured opportunities at intervention versus control practices of 4.4, 3.3, and 0.3 percentage points in January, February, and March, respectively (P = .63 for the impact of the intervention in the first versus last 3 months of the influenza season, with all P values accounting for clustering).
Rates of up-to-date influenza vaccination increased from 44.2% to 48.2% at control sites and from 45.0% to 53.0% at intervention sites, a 4.0% (95% CI: −1.3% to 9.1%) greater but not statistically significant improvement (Table 4). With standardization for selected covariates, up-to-date vaccination rates increased similarly by 3.4% (95% CI: −1.4% to 9.1%), a statistically nonsignificant improvement. The 4 practices with the greatest improvements in rates (between 11% and 15%) were all in the intervention group (Fig 1) and, on the basis of the standardized improvement in rates with the intervention, 208 additional children with asthma at intervention sites alone were fully vaccinated as a result of the alerts. Results were similar when the outcome was receipt of a single dose of influenza vaccine, with a statistically nonsignificant 4.0% (95% CI: −1.1% to 10.7%) relative improvement at intervention versus control sites. When only the urban resident-teaching practices were considered and the outcome was up-to-date vaccination, standardized rates increased from 47.8% to 53.8% at control practices and from 47.1% to 58.5% at intervention practices, a statistically significant 5.4% (95% CI: 1.6%–9.7%) relative improvement (Table 4). However, when the impact of practice on the intervention was considered in a model including all practices, results were no longer significant (P = .23).
Covariates affecting up-to-date influenza vaccination also were considered (Table 5). Factors significantly associated with complete vaccination in both the crude and adjusted analyses included Asian race (versus black), severe persistent asthma, chronic medical problems other than asthma, care at urban teaching practices, increasing numbers of office visits during the influenza season, and previous receipt of influenza vaccine. Rates for children with 1, 2, 3, or 4 visits to the office during the influenza season improved by 5, 5.4, 9.8, and 7.5 percentage points more, respectively, at intervention practices than at control practices (P = .38). Similarly, at intervention sites, compared with control sites, the relative improvement in rates was 6.5% for children who had received the influenza vaccine previously, compared with just 3.2% for those who had not (P = .61).
We found that influenza alerts for children and teens with asthma resulted in relative improvements in up-to-date influenza vaccination rates of 3.4% overall and 5.4% in the urban setting. Because rates increased at both intervention and control practices, the before/after design proved especially effective for determining the independent contribution of the alerts. The rates of improvement were similar to those in a trial of tailored interventions for children at high risk 2 to 6 years of age23 and a controlled trial of reminder recall for a general population of children 6 to 24 months of age10 but smaller than the 17% improvement seen with reminder recall in a population of high-risk Colorado children between 6 months and 6 years of age.11 A smaller number of sites, existing provider interest in immunization research at each of those locations, a younger population with more routine physician visits, and lower baseline rates (25%) might have contributed to the larger impact of the intervention in that study. Although we did observe higher baseline influenza vaccination rates, nearly one half of all children were not immunized, which suggests that the baseline rate is unlikely to fully explain the small impact of this intervention.
Despite improvements, influenza vaccination rates nationally remain much lower than rates for routine pediatric immunizations, are only slightly better for children with asthma than for other children, and leave many children at risk.2 In a reflection of the importance of the initial acceptance of influenza vaccine by families, we found that children who had received influenza vaccine previously had 8 times greater odds of up-to-date vaccination by the end of the influenza season. Concern about vaccine side effects is known to limit influenza vaccine receipt, however, and as many as 70% of parents think that the vaccine can cause influenza disease.8,24,25 In addition, many parents consider influenza vaccine less safe in high-risk populations than in healthy populations.26 Our results suggest that simply reminding clinicians to vaccinate children in the office is an inadequate strategy for overcoming these barriers.
Our visit-level data indicate that better capturing of available immunization opportunities remains an essential strategy for improving rates. The EHR-based clinical alerts resulted in a minimal 0.3% better improvement in rates of captured vaccination opportunities at intervention practices, compared with control practices, and >80% of eligible children with asthma failed to receive the vaccine at visits with clinicians even with the alerts active. These rates of missed opportunities are similar to those found among children 6 months to 6 years of age with asthma at practices in metropolitan Denver, Colorado, where vaccination, as in the current study, was less common at acute care than preventive visits.27 Our analysis suggested that shortages, which represent a known barrier to influenza vaccination,28 had minimal impact on the effectiveness of this intervention.
By activating the alerts through the end of March, we anticipated increased vaccination in the last 3 months of the influenza season, which is a public health goal.6 Relative improvements of up to 4.4% were seen for intervention versus control practices in the last 3 months of the influenza season. However, the decrease in influenza vaccination rates throughout the influenza season, which was also seen in previous studies,27 indicates that expanding late-season vaccination remains a challenge. These decreases occurred despite the continued prevalence of influenza disease through April.29,30 Because clinician recommendation has been found repeatedly to be the strongest predictor of influenza vaccination acceptance by parents,31–33 these results underscore the need for ongoing training for clinicians regarding the importance of their role in preventing missed opportunities throughout the influenza season. Our finding that children with more visits during the influenza season benefited more from the clinical alerts suggests that repeated discussions with families at each visit may be helpful.
The use of additional strategies in conjunction with the alerts might have increased the effectiveness of the intervention. The alerts appeared at the point of care among children known to be at high risk, offered a specific recommendation, and then facilitated compliance, all of which are suggested approaches for maximizing decision-support effectiveness.34,35 However, combining in-office, EHR-based alerts with reminder recall and educational communication, all of which are possible through health information technology,9 might have improved vaccination rates more effectively.
This study had several limitations. We had no specific information on why influenza vaccine was not given at individual visits. Data on whether alerts triggered clinicians to address influenza vaccine and, if they did, whether parents refused the vaccine would have been helpful for better understanding the impact of the intervention. Our previous results with the same system suggested that clinicians did act on vaccine alerts, at least in the resident-teaching practices at visits with young children who were due for routine pediatric vaccinations.17 Furthermore, although the EHR offered a robust source of information on vaccine receipt at study sites and the receipt of vaccine outside study practices was abstracted if reported, we might have missed some children who were vaccinated in other settings. None of these limitations is likely to have changed the overall results.
The results of this study contrast with the success of the alert system in improving vaccination rates and reducing missed opportunities for routine pediatric vaccinations among young children.17 Given the complexity of and frequent changes in the routine immunization schedule, clinicians might have benefited more from reminders for routine vaccines. Furthermore, routine pediatric vaccinations for young children may be more accepted by families than influenza vaccination for older children and adolescents, because of effective requirements for routine vaccine receipt before entry into day care or school.36,37 Finally, the study of the routine vaccination alerts occurred exclusively in resident-teaching practices, where the influenza vaccine alerts also had an impact on up-to-date vaccination rates.
Although practices with EHR-based alerts performed better than control practices, relative improvements at intervention versus control practices were modest and by themselves would not justify the implementation of an immunization alert system. If practices are already using such a system for routine pediatric vaccinations, however, then the addition of an influenza reminder system may be helpful, especially in the setting of urban resident-teaching practices. More broadly, alternative or complementary strategies will be needed to address the ongoing challenge of better capturing missed opportunities and ensuring that children and teens visit the office and receive influenza vaccine throughout the entire season. These approaches will be especially important because new national recommendations call for universal influenza vaccination for youths between 5 and 18 years of age, the age group targeted in this study.38
Research support was provided by institutional development funds from the Children's Hospital of Philadelphia.
We thank the network of primary care physicians, their patients, and the patients’ families for their contributions to clinical research through the Pediatric Research Consortium at the Children's Hospital of Philadelphia. In addition, we recognize Jonathan Crossette, MPH, and Svetlana Ostapenko for their assistance in assembling data, Maggie Swietlik, MSN, CRNP, for her help in implementing the intervention in the Pediatric Research Consortium, and Lovlei McKinnie for her administrative support.
- Accepted December 22, 2008.
- Address correspondence to Alexander G. Fiks, MD, MSCE, Pediatric Generalist Research Group, Children's Hospital of Philadelphia, 3535 Market St, Room 1546, Philadelphia, PA 19104. E-mail:
Financial Disclosure: The authors have indicated they have no financial relationships relevant to this article to disclose.
What's Known on This Subject:
Influenza vaccination rates remain low for children and adolescents with asthma. EHR-based alerts have proved effective in improving vaccine delivery for routine pediatric vaccines but have not been thoroughly examined for influenza vaccine.
What This Study Adds:
This study is the first trial of EHR-based clinical alerts to improve influenza vaccine delivery in pediatric primary care.
- ↵American Academy of Pediatrics, Committee on Infectious Diseases. 2006 Red Book: Report of the Committee of Infectious Diseases. Elk Grove Village, IL: American Academy of Pediatrics; 2006
- ↵Advisory Committee on Immunization Practices. Prevention and Control of Influenza: Recommendations of the Advisory Committee on Immunization Practices. Atlanta, GA: Centers for Disease Control and Prevention; 2006
- ↵American Academy of Pediatrics, Committee on Infectious Diseases. Prevention of influenza: recommendations for influenza immunization of children, 2006–2007. Pediatrics.2007;119 (4):846– 851
- ↵Shefer A, Briss P, Rodewald L, et al. Improving immunization coverage rates: an evidence-based review of the literature. Epidemiol Rev.1999;21 (1):96– 142
- ↵Szilagyi PG, Rodewald LE, Savageau J, Yoos L, Doane C. Improving influenza vaccination rates in children with asthma: a test of a computerized reminder system and an analysis of factors predicting vaccination compliance. Pediatrics.1992;90 (6):871– 875
- ↵Kempe A, Daley MF, Barrow J, et al. Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall. Pediatrics.2005;115 (1):146– 154
- ↵Daley MF, Barrow J, Pearson K, et al. Identification and recall of children with chronic medical conditions for influenza vaccination. Pediatrics.2004;113 (1). Available at: www.pediatrics.org/cgi/content/full/113/1/e26
- ↵Kemper AR, Uren RL, Clark SJ. Adoption of electronic health records in primary care pediatric practices. Pediatrics.2006;118 (1). Available at: www.pediatrics.org/cgi/content/full/118/1/e20
- ↵Hak E, van Essen GA, Stalman WA, de Melker RA. Improving influenza vaccination coverage among high-risk patients: a role for computer-supported prevention strategy? Fam Pract.1998;15 (2):138– 143
- ↵National Asthma Education and Prevention Program. Expert Panel Report 2: Guidelines for the Diagnosis and Management of Asthma. Bethesda, MD: National Institutes of Health; 1997
- ↵Fiks AG, Grundmeier RW, Biggs LM, Localio AR, Alessandrini EA. Impact of clinical alerts within an electronic health record on routine childhood immunization in an urban pediatric population. Pediatrics.2007;120 (4):707– 714
- ↵Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC, Burney PGJ, Donner A. Methods in health service research: evaluation of health interventions at area and organisation level. BMJ.1999;319 (7206):376– 379
- ↵Zimmerman RK, Hoberman A, Nowalk MP, et al. Improving influenza vaccination rates of high-risk inner-city children over 2 intervention years. Ann Fam Med.2006;4 (6):534– 540
- ↵Daley MF, Crane LA, Chandramouli V, et al. Misperceptions about influenza vaccination among parents of healthy young children. Clin Pediatr (Phila).2007;46 (5):408– 417
- ↵Hodinka RL. Clinical Virology Laboratory 2000–2006 Cumulative Respiratory Virus Activity Report. Philadelphia, PA: Children's Hospital of Philadelphia; 2006
- ↵Hodinka RL. Clinical Virology Laboratory 2006–2008 Cumulative Respiratory Virus Activity Report. Philadelphia, PA: Children's Hospital of Philadelphia; 2008
- ↵Poehling KA, Speroff T, Dittus RS, Griffin MR, Hickson GB, Edwards KM. Predictors of influenza virus vaccination status in hospitalized children. Pediatrics.2001;108 (6). Available at: www.pediatrics.org/cgi/content/full/108/6/e99
- Daley MF, Crane LA, Chandramouli V, et al. Influenza among healthy young children: changes in parental attitudes and predictors of immunization during the 2003 to 2004 influenza season. Pediatrics.2006;117 (2). Available at: www.pediatrics.org/cgi/content/full/117/2/e268
- ↵Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc.2003;10 (6):523– 530
- ↵Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ.2005;330 (7494):765
- ↵Lopez AS, Kolasa MS, Seward JF. Status of school entry requirements for varicella vaccination and vaccination coverage 11 years after implementation of the varicella vaccination program. J Infect Dis.2008;197 (suppl 2):S76– S81
- ↵Morita JY, Ramirez E, Trick WE. Effect of a school-entry vaccination requirement on racial and ethnic disparities in hepatitis B immunization coverage levels among public school students. Pediatrics.2008;121 (3). Available at: www.pediatrics.org/cgi/content/full/121/3/e547
- Copyright © 2009 by the American Academy of Pediatrics