New! Video Abstracts -- brief videos summarizing key findings of new articles
Trainees, enter the Registry-Linked Electronic Influenza Vaccine Provider Reminders: A Cluster-Crossover Trial
Abstract
OBJECTIVE: To determine the impact of a vaccination reminder in an electronic health record supplemented with data from an immunization information system (IIS).
METHODS: A noninterruptive influenza vaccination reminder, based on a real-time query of hospital and city IIS, was used at 4 urban, academically affiliated clinics serving a low-income population. Using a randomized cluster-crossover design, each study site had “on” and “off” period during the fall and winter of 2011–2012. Influenza vaccination during a clinic visit was assessed for 6-month to 17-year-old patients. To assess sustainability, the reminder was active at all sites during the 2012–2013 season.
RESULTS: In the 2011–2012 season, 8481 unique non-up-to-date children had visits. Slightly more non–up-to-date children seen when the reminder was ‘on’ were vaccinated than when ‘off’ (76.2% vs 73.8%; P = .027). Effects were seen in the winter (67.9% vs 62.2%; P = .005), not fall (76.8% vs 76.5%). The reminder also increased documentation of the reason for vaccine non-administration (68.1% vs 41.5%; P < .0001). During the 2011–2012 season, the reminder displayed for 8630 unique visits, and clinicians interacted with it in 83.1% of cases where patients required vaccination. During the 2012–2013 season, it displayed for 22 248 unique visits; clinicians interacted with it in 84.8% of cases.
CONCLUSIONS: An IIS-linked influenza vaccination reminder increased vaccination later in the winter when fewer vaccine doses are usually given. Although the reminder did not require clinicians to interact with it, they frequently did; utilization did not wane over time.
What’s Known on This Subject:
Frequency of influenza vaccination is low, partially because of missed opportunities to vaccinate. Barriers to implementing successful influenza vaccination reminders in the electronic health record include alert fatigue and incomplete vaccination information due to scattered records.
What This Study Adds:
A noninterruptive, immunization information system–linked influenza vaccination reminder can increase vaccination late in the winter when fewer vaccine doses are usually administered. Tailoring the reminder to clinicians’ needs can increase its use.
Influenza vaccination is important both for a child’s own health and for preventing infection in households and communities.1,2 Yet coverage is far below that of other childhood vaccinations.3–6 One way to increase influenza vaccination is to reduce “missed opportunities” to vaccinate.7–10
Computerized clinical decision support tools, such as reminders in electronic health records (EHR), may improve adherence to clinical guidelines.11–16 Several studies have reported beneficial results from electronic vaccination reminders in both adults and children17–21; however, electronic reminders may also adversely affect clinician workflow (eg, contributing to “alert fatigue”).22–25
Reminders are dependent on accurate, timely data.26 When a decision support system generates incorrect recommendations, clinician trust in the system diminishes. One barrier to implementing successful influenza vaccination reminders is a lack of up-to-date vaccination information for a patient.27 This is a notable concern for low-income and minority children, who are especially susceptible to vaccination record fragmentation due to care from multiple providers.28,29 One can address this challenge by merging local vaccination data with data from a regional immunization information system (IIS), a population-based system that centrally collects vaccination data at a regional or state level.30 Electronic vaccination reminders in the EHR, however, have historically acted on local vaccination data.20,21
The objective was to determine the impact on vaccine administration of EHR influenza vaccination reminders that are linked to a city IIS. A secondary goal was to assess their effect on documentation of reasons for vaccine nonadministration. We hypothesized that an electronic reminder would increase influenza vaccination and documentation of reason for nonadministration. Secondarily, we sought to assess the sustainability of reminder implementation.
Methods
Setting and Subjects
In the 2011–2012 influenza season, we assessed the impact of an influenza vaccination reminder in a cluster randomized crossover trial in 4 community-based pediatric clinics affiliated with New York–Presbyterian (NYP) Hospital/Columbia University Medical Center in New York City. For the entire 2012–2013 season, the reminder was implemented at all 4 sites to assess sustainability. These clinics are staffed by a single pediatric group practice, but providers are assigned to 1 clinic for their primary patient care. They serve a predominantly Latino and publicly insured population; ∼95% are eligible for free vaccines through the Vaccines for Children program. There were no known vaccine shortages during the study. The Columbia University Medical Center Institutional Review Board approved the study with a waiver of consent.
Each site used a common commercial EHR (Allscripts Sunrise, Allscripts, Chicago, IL), including structured visit notes. The EHR is integrated with the institution’s local vaccination registry, which includes all patients receiving care at NYP hospitals and affiliated clinics.31 The registry extracts information directly from the provider order entry/nurse sign-off EHR module. It also securely retrieves vaccination information for NYP patients from New York City’s Immunization Information System (IIS), the Citywide Immunization Registry (CIR). Reporting to the CIR is mandated for children aged <19 years old,32 and ∼94% of facilities that vaccinate children report regularly.33
Design of the Reminder
The reminder was designed with clinician and parent focus group and β-tester input.27 When a medical or nursing provider opened the EHR note-authoring module, the reminder automatically retrieved vaccination information from the hospital registry, which synchronized data real-time with the CIR. The reminder window appeared in a non-interruptive fashion in the top-right corner of the EHR screen, floating above the note entry form. The reminder displayed both a color-coded message based on the influenza vaccination status and the most recent seasonal and 2009 H1N1 vaccination dates (Supplemental Fig 3). An orange background indicated the child was non–up-to-date and allowed vaccine ordering using age-specific options or documentation of reason for vaccine nonadministration. Any action taken was recorded within the note’s “Plan” section. The reminder window also included a more information button, opening a window with all influenza vaccinations, allergies, and medical problems, as well as talking points based on information focus group parents suggested might help with influenza vaccination decision-making. For a child seen solely by a nurse whose family did not want vaccination, the nurse could document why the vaccine was not being ordered. If the family wanted vaccination, the child was seen by an ordering provider per existing site protocols. A green background indicated the child was up-to-date. A yellow background indicated a child was up-to-date but needed another dose after a prescribed waiting period. A red background indicated an egg allergy regardless of vaccination status.
The reminder required no mandatory action; a provider could interact any time during note-writing or ignore it. The reminder window opened once per visit for each provider for up-to-date patients, but opened every time a note was opened for non–up-to-date patients and an action was not yet performed. At the conclusion of note-writing, if a provider neither ordered the vaccine nor documented a reason for nonadministration for a non–up-to-date patient, an additional reminder prompt was displayed.
Care providers, including attending and resident physicians, nurse practitioners, and nurses, were given a short in-person in-service as well as visual aids. The reminder was pilot-tested at all 4 sites during a 10-week period in the 2010–2011 season.
Intervention
During the 2011–2012 influenza season, we conducted a cluster randomized crossover trial. To account for seasonal variations in vaccination,6,34 the season was divided into 2 phases (Fig 1), fall and winter, each with a 28-day “on” and a 28-day “off” period. Each clinic was randomized to a cluster with an on and an off period in each phase. This design allows for differences in outcomes to be attributed to the intervention, rather than due to preexisting site differences.35 During the on period, the reminder appeared when an electronic note was authored or edited for well-child, follow-up, acute care, Supplemental Nutrition Program for Women, Infants, and Children, or vaccine-only visits. During the off period, the reminder was not displayed. The switch every month and switch pattern were selected to minimize the number of transitions while allowing for sites to have both an on and an off period in the fall and winter. In the 2012–2013 season, the reminder was on for all 4 clinic sites through the season.
Study flow diagram: 2011–2012 cluster-crossover trial. During “on” periods, the electronic influenza vaccination reminder appeared for health care providers in the electronic health record.
Measures
The primary outcome was the percentage of non–up-to-date children vaccinated at a visit. A secondary outcome was documentation of reason for nonadministration. Process measures were also assessed including the number of reminder firings by vaccination status, proportion of reminders acted on for non–up-to-date patients, and actions taken.
Data Sources
Influenza vaccination status was obtained from the hospital registry, which includes data from the local EHR as well as the CIR. To assess documentation of vaccine nonadministration, a random sample of children who remained non–up-to-date after their first visit to an MD provider during the study period was stratified by on/off status, age, site, and seasonality (fall or winter). The chart reviewer, who was blinded to on/off status, used an abstraction form to identify (1) any mention in the assessment/plan of influenza vaccine or reason for vaccine nonadministration or (2) evidence of a vaccine order, even if the vaccine was ultimately not given.
Statistical Analyses
For analyses related to receipt of influenza vaccination in the 2011–2012 season, the season was divided into the following analytic periods based on the on/off periods: period 1 (October 12–November 8); period 2 (November 9–December 13 excluding Thanksgiving week, a shortened week with different visit structure); period 3 (January 4–January 31); period 4: (February 1–February 28). Final vaccination status after clinic visits for non–up-to-date children aged 6 months to 17 years were compared when the reminder was on versus off using the χ2 test, and the relative risk of remaining unvaccinated (or having a missed opportunity) was determined. Children who required 2 doses in a given season were included in the analysis, focusing on the dose needed as of the start of the analytic period. The 365 children (5.2% of the total non–up-to-date) who were non–up-to-date at both on and off periods were removed from analyses to avoid any potential carryover effects. Additionally, for the 975 children (14.8% of the remaining non–up-to-date) who had multiple non–up-to-date visits, analyses were limited to the last visit in the season when the child was non–up-to-date. The last visit was chosen to best reflect the patient’s final vaccination status for the season. A sensitivity analysis was also conducted assessing the impact of the reminder at the first visit a child was seen in the analytic period; this included all children.
Multivariable logistic models accounting for seasonality, site, and age were conducted, and interactions were assessed. Analyses were then stratified by seasonality. Fall and winter analyses included the last visit when a dose was due in the fall and winter, respectively.
Documentation of reason for nonadministration was compared between on and off periods. In addition, to better understand the sustainability of reminder implementation, process measures were described for the 2011–2012 (October 12–March 31) and 2012–2013 (October 1–March 31) seasons.
All analyses were performed by using SAS v.9.3 (SAS Institute, Inc, Cary, NC).
Results
During the analytic period in 2011–2012, 8481 unique children had visits, and median age was 6.5 years (interquartile range 2.1–10.2). Nearly two-thirds came from Spanish-speaking families, and half were female (Table 1). At the time of their first visit during the analytic period, 1478 (17.4%) had already been vaccinated that season and did not require another dose; 66 (0.8%) were vaccinated but were in need of a second dose, 21 of whom returned when they were due for that dose. A total of 6958 children had a visit in the analytic period for which they were not up-to-date, including those who were not up-to-date at their first visit (n = 6937) and the 21 indicated earlier.
Characteristics of Study Population in 2011–2012
After excluding 365 children seen in both on and off periods while non–up-to-date, most (85.2%) had only 1 visit when they were non–up-to-date, 11.8% had 2. Most (78.3%) presented for a well-child visit, with the remainder presenting for acute care (14.9%), vaccine only (6.5%), and Supplemental Nutrition Program for Women, Infants, and Children visit (<1%). There were no significant differences in gender, language, or age among those who were non–up-to-date at a visit when the reminder was on versus off.
Overall, more non–up-to-date children seen when the reminder was on were vaccinated than when off (Table 2). In a regression model adjusted for seasonality, age, and clinic site, the impact of the reminder on reducing the number of children who remained unvaccinated, that is, had a missed opportunity, was significant (adjusted relative rates 0.90, 95% confidence interval 0.83–0.98). There were no interactions by site (P = .64)
Proportion of Children Receiving Needed Influenza Vaccination During Clinic Visits When the Reminder Was On Versus Off, 2011–2012 Season
Due to an interaction between on/off status and seasonality (P = .06), stratified analyses were conducted. Reminder effects were seen in the winter (67.9% vs 62.2%; P = .005), not the fall (76.8% vs 76.5%; Table 2). Although most visits were preventative, those whose last non–up-to-date visit in the fall was an acute care visit when the reminder was on were more likely to be vaccinated than when it was off (47.7% vs 38.9%; P = .027). In the winter, after adjusting for clinic and age, a child seen when the reminder was on for any visit was less likely to remain unvaccinated (adjusted relative rates 0.83, 95% confidence interval 0.74–0.93). In a sensitivity analysis, the impact of the reminder on the first non–up-to-date visit of any kind was significant in the winter (65.5% vs 60.2%, P = .01) but not fall.
There were 319 children who received their first dose during the analytic period and subsequently returned for a second dose; of those, more received that dose when the alert was on versus off (73.0% vs 65.4%; P = .14). Of the 365 children who were excluded because they were seen when they were non–up-to-date at both on and off periods, more were vaccinated if their last non–up-to-date visit occurred when the alert was on vs off (48.0% vs 39.2%; P = .09), regardless of whether their previous period was on or off.
In the 2011–2012 season, 1830 children remained non–up-to-date after their first clinic visit with an MD during the analytic period. In a random stratified sample (n = 624), documentation of the reason for vaccine nonadministration was more likely to occur when the reminder was on versus off (68.1% vs 41.5%; P < .0001; Fig 2). A similar pattern was observed in the fall (73.2% vs 45.6%; P < .0001) and winter (60.0% vs 36.6%; P < .0001).
Proportion of unvaccinated children whose medical records contained documentation related to influenza vaccination decision-making, 2011–2012 season. All P values <.0001.
In the 2011–2012 season, the reminder displayed 13 859 times for 8630 unique visits; 61.0% involved non–up-to-date children, for whom care providers acted on the reminder 83.1% of the time (Table 3). In the 2012–2013 season, the reminder displayed 37 379 times for 22 248 unique visits, 61.7% for non–up-to-date children, for whom care providers acted on the reminder in 84.8% of cases.
Vaccination Status and Care Provider Actions While the Influenza Vaccination Reminder Was Active
Discussion
In this cluster randomized crossover trial, an influenza vaccination reminder in the EHR linked with city IIS data resulted in modest but significant increases in influenza vaccination, primarily in the January–February period. The majority of influenza vaccine doses nationally are given in the October–December period, and providers may be more likely to remember to offer vaccine then. Despite low national coverage, there are few increases in vaccine coverage after December.36 Yet in recent years, influenza has been present in the community through the spring, making the continued vaccination of patients later into the winter valuable.37,38 Therefore, an intervention that can have an impact on influenza vaccination after December may be particularly important. This reminder also had an effect during acute care visits in the fall, when missed opportunities may occur.39
In EHRs, “hard-stop” alerts preventing users from proceeding until addressed, are highly unpopular with care providers. Nevertheless, hard-stop alerts are often used to enforce compliance.24,40,41 Reported override rates of non–hard-stop alerts in EHRs are as high as >90%.25,42,43 By contrast, in this study, care providers acted on the non–hard-stop reminder in the vast majority of cases; this behavior was sustained over multiple years. The success of the reminder may be attributed to involving care providers from the project’s inception.27 The reminder also fit within clinicians’ workflow and provided them with information at the time and place needed.44
A previous study of influenza vaccine EHR reminders not linked to an IIS for children with asthma demonstrated only a 0.3% improvement in captured vaccination opportunities.21 Although it is not known if this lack of linkage to an IIS played a role, best practices should ensure that data used to trigger a clinical reminder are accurate and reliable.44 One of the key features of our reminder was the integration with a city IIS, enabling the decision support logic to use the best available vaccination information.27 This may have led to greater trust in the alert, increasing its use. All 50 states, 5 cities, and the District of Columbia operate an IIS, and ∼86% of US children ≤6 years old have vaccinations in one.30 The Centers for Medicare and Medicaid Services EHR Incentive Program provides financial incentives for meaningful use of certified EHR technology to improve patient care.45 Stage 2 of this program includes the use of clinically relevant information to identify patients who should receive reminders for preventive or follow-up care. One of the proposed measures for Stage 3 is transfer of vaccination information from an IIS to an EHR. The results of this study lend support to this Stage 2 and proposed Stage 3 measure.
Finally, the reminder had an important impact on documentation of reasons for vaccine nonadministration. At the care-provider level, it is useful to know why a vaccine was not given at a previous visit. For example, encouraging vaccination for a child whose family previously refused the vaccine may require a different approach than for one who deferred due to illness. Improved documentation of nonvaccination may also be important at an institutional level for purposes of compliance and quality improvement.46 In this study, the most common reason for nonadministration was patient illness, followed by family refusal. It is possible that provider education regarding the acceptability of vaccinating during mild illness is needed. In addition, parents’ perceptions of influenza and the vaccine may also need to be addressed.47,48
There were limitations to this study. The study took place in 1 low-income, urban, academic setting using a commercial EHR; results may not generalize to other environments. Latino children have higher influenza vaccination rates nationally (66.0%) versus the overall population (58.9%).6 Outcomes were assessed based on EHR documentation; the details of what actually transpired at each visit were not available. Additionally, it is possible that there may have been incomplete vaccination documentation. However, vaccine administrations are automatically transferred from the EHR to the hospital registry, and incomplete records should affect sites equally, regardless of reminder on/off status. Although we did not a priori indicate what would constitute a clinically significant difference, we believe a relative reduction in undervaccination of 10% to 17% and improvement in documentation of 61% to 64% are significant benefits, especially for an intervention that is straightforward to implement. Another limitation is that a minority (5.2%) of children had a visit during both an on and off period and were excluded from the primary analysis. Likewise, 14.8% of children had multiple visits that did not include both on and off periods. We chose to assess the last visit to best reflect the patient’s final vaccination status; we also conducted a sensitivity analysis assessing the first visit, which included all children. Also, some children required 2 doses in a given season. Children were included in the analysis for the first dose they needed during the analytic period. Finally, randomization was not performed at the individual patient level due to potential carryover effects from 1 patient to another seen by the same provider. The study used a crossover trial giving each site on and off comparison periods. We cannot, however, control for the rare possibility that the reminder sites independently had better vaccination practices at the times the reminder was on versus off. The on/off design might also have felt disruptive to providers.
Conclusions
An influenza vaccination reminder in an EHR linked to a regional IIS was effective in increasing vaccination, particularly later in the winter when fewer vaccine doses are usually given. Although care providers were not forced to act on the reminder, they did so the vast majority of the time. Utilization did not wane over time. The reminder also had a significant effect on documentation of why a vaccine was not administered.
Acknowledgments
We thank Ken Kitayama, Lauren Sonnabend, Oscar Peña, Balendu DasGupta, and Karthik Natarajan for their help with the study. We also thank NewYork–Presbyterian Hospital for its support of the EzVac Immunization Information System and NewYork–Presbyterian Hospital Ambulatory Care Network and its staff and patients.
Footnotes
- Accepted October 20, 2014.
- Address correspondence to Melissa S. Stockwell, MD, MPH, Division of Child and Adolescent Health-Columbia University, 622 W. 168th Street-VC 417, New York, NY 10032. E-mail: mss2112{at}columbia.edu
Dr Stockwell conceptualized and designed the study, analyzed the data, and drafted the initial manuscript; Drs Catallozzi, Findley, Kukafka, Hofstetter, and Vawdrey contributed to the conceptualization and design of the study and reviewed and revised the manuscript; Mr Camargo implemented the reminder application, provided data management and data collection tools, and critically reviewed the manuscript; Dr Ramakrishnan and Mr Holleran contributed to the design of the study, aided in data analysis, and critically reviewed the manuscript; Dr Fernandez aided in data collection and critically reviewed the manuscript; and all authors approved the final manuscript as submitted.
This manuscript’s contents are solely the responsibility of the authors and do not necessarily represent the official views of the Agency for Healthcare Research and Quality. The funding agency had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript.
This trial has been registered at www.clinicaltrials.gov (identifier NCT01146899).
FINANCIAL DISCLOSURE: Dr Hofstetter receives support from the Pfizer Medical Education Group for a different investigator-initiated study and also received support through the Investigator-Initiated Studies Program of Merck, Sharp & Dohme Corp for a different study; Dr Stockwell is a coinvestigator but receives no financial support; and the other authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This study was supported by grant R18HS018158 (Stockwell) from the Agency for Healthcare Research and Quality.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
References
- Copyright © 2015 by the American Academy of Pediatrics
