PEDIATRICS Vol. 106 No. 3 September 2000, pp. 489-492

Immunization Entry at the Point of Service Improves Quality, Saves Time, and Is Well-Accepted

William G. Adams, MD, William P. Conners, Adriana M. Mann, and Sean Palfrey, MD

From the Division of General Pediatrics, Boston University School of Medicine, Boston, Massachusetts.

	ABSTRACT

Top Abstract Methods Results Discussion References

Objective.  Computer-based immunization tracking is a routine part of many pediatric practices; however, data quality is inconsistent and entry often relies on dedicated data entry personnel and is time-consuming, expensive, or difficult. The purpose of this study was to evaluate data quality, nursing satisfaction, and reduction in documentation burden after the introduction of a point-of-service immunization entry system in an inner-city pediatric primary care center.

Design.  Prospective preintervention and postintervention study.

Methods.  Visit records from all pediatric nonurgent care visits for patients <5 years old were collected during a 2-week period before (preintervention) and after (postintervention) the introduction of a computer-based immunization entry system. Nurses used software designed to allow rapid entry during immunization preparation followed by printing 2 adhesive labels for documentation. Satisfaction was evaluated using an 8-question survey administered 3 months after the intervention.

Results.  One hundred forty-seven (63.6%) of 231 preintervention and 132 (51.4%) of 257 postintervention children received at least 1 immunization (immunized) during the study visit. Gender and mean age were similar for immunized children in the 2 groups. In the preintervention group, 56 (37.9%) of 147 immunized children had at least 1 dose missing (a total of 128 of 343 doses administered) from the immunization tracking database compared with none in the postintervention group. Medical record review showed that 92.6% of preintervention and 91.4% of postintervention children were on-schedule after the study visit. However, missing data lead to the misclassification of preintervention childrenonly 68.4% were reported by the database to be on-schedule. All 9 nurses reported using the program all the time to enter immunizations, 89% said that the program required somewhat or a lot less time, and 100% strongly recommended continued use of the program. All 9 nurses also reported that they would be somewhat or very unenthusiastic about the system if labels were not available. During the 12 months after introduction of the system, 8273 forms containing immunization information were printed, preventing nurses from having to write >101 000 dates.

Conclusions.  Immunization entry by nurses at the time of immunization preparation improves the quality of tracking data, reduces misclassification of immunization needs, saves time, and can be well-accepted. It is likely that poor data quality in some tracking systems has led to falsely low immunization coverage estimates. Systems such as the one in this study can improve quality and should be integrated into routine clinical practice.  Key words:  immunization tracking, immunization delivery, point of service.

Immunization tracking with reminders and recalls has been shown to improve immunization delivery,¹ as well as delivery of primary care services.² Computer-based immunization tracking is now a routine part of practice for many child health care providers. However, the data quality in many immunization tracking systems is inconsistent, and entry often relies on dedicated data entry personnel and is time-consuming, expensive, or difficult. Published guidelines for the use of immunization tracking systems recommend that providers routinely evaluate the accuracy of such systems and the few studies available have shown substantial variation in data quality with 10% to 40% of doses given missing from the tracking database.^3-7 Despite the limitations of these systems, a number of benefits have already been realized, including enhanced outreach efforts, improved immunization rates, and reduced clinical workload related to form completion.^1-3,8,9

Collection of medical data at the point of service (POS) has become the focus of many new medical software packages. POS systems require clinical personnel to record information at the time that the patient is being seen. These systems offer the best opportunity for accurate information collection because the clinician providing the service records the desired data, which is then immediately available for clinical decision-making, form printing, or other real-time process. Integration into clinical workflow is a critical requirement of any successful clinical information system¹⁰ and POS systems are no exception. Any system that is unacceptably slow, or difficult to use will not be accepted in clinical settings, especially busy pediatric centers with high patient volumes. Innovations in information technology have finally made possible the development of user-interfaces that can substantially improve the speed and ease of data collection at the point of service while not disrupting the demanding clinical workflow of ambulatory care centers. Automated immunization tracking systems have begun to use these interfaces either as isolated programs or in conjunction with computer-based patient records (CPR). No published studies have evaluated the data quality and acceptability of POS immunization tracking systems.

The purpose of this study was to evaluate data quality, nursing satisfaction, and reduction in documentation burden after the introduction of a POS immunization entry system in an inner-city pediatric primary care center.

	METHODS

Top Abstract Methods Results Discussion References

Setting

This study was conducted at the Boston Medical Center Pediatric Primary Care Center (PPCC). The PPCC is located in inner-city Boston and has over 24 000 primary care patient visits annually. The clinic primarily serves poor patients of minority ethnicity (35% African American, 20% Haitian American, 30% Latino, 15% other groups).

Immunization Tracking Before POS Entry

Before introduction of POS immunization entry, tracking in the clinic had been labor-intensive and the data quality inconsistent. Immunization entry required a full-time immunization coordinator positioned at the end of a paper trail that began in the examination room with the clinician recording immunizations given on a billing sheet. Clerical staff would then collect and transfer the billing sheet to the immunization coordinator in a separate building. The coordinator then entered the data into the city of Boston Immunization Information System (BIIS).¹¹ Finally, the data were imported into the PPCC clinical data repository. Laboratory screening results (hemoglobin, hematocrit, mean corpuscular volume, red cell distribution width, lead, and zinc protoporphyrin) from the central hospital computer were also imported into the repository monthly allowing clinicians to print immunization and laboratory screening data on standardized forms for school, camp, immunization certification, and the Women, Infants, and Children (WIC) program. Software for form printing was available throughout the clinic. The immunization coordinator was not aware of the study during the preintervention period.

POS Immunization Entry

As the first step in the implementation of a CPR, we developed and implemented immunization entry software to be used at the time of immunization preparation. The software was designed to allow data entry without using a keyboard via a point-and-click interface and used a vaccine lot number inventory to speed entry. Users were required to log-in, select a patient's electronic record using the unique medical record number, select immunizations to be given for the day from a standardized list, select site of administration and lot number from vaccine specific lists, and finally, to print 2 labels to record doses given in the patient's paper record. Software was placed on a single networked computer in the clinic medication room. Nurses received 15 minutes of personalized training from the study research assistant.

Study Design

The study used a preintervention/postintervention design. The intervention was implemented on June 15, 1998. Data quality was evaluated during a preintervention phase (May 24-June 12, 1998) and a postintervention phase (August 15-August 28, 1998). All nonurgent care progress notes for children <5 years old were reviewed to determine which doses had been given during the visit. Immunization summary pages located in the front of all patient records were also reviewed. A dose was considered missing from the database if it was recorded on either the progress note or summary page, but not in the BIIS database. A child was considered to be on-schedule if no additional immunizations were due until the next regularly scheduled health maintenance visit. To assess nursing satisfaction with the system, an 8-question survey was given to each nurse 3 months after implementation. All full-time clinic nurses⁹ were involved in the study.

Statistical Analysis

Statistical significance testing was performed using Yates corrected ² test for comparison of categorical data and the t test for comparison of continuous data. Differences were considered statistically significant for P < .05.

	RESULTS

Top Abstract Methods Results Discussion References

A total of 488 visits occurred during the 2 study periods. In the preintervention group, at least 1 immunization was administered in 147 (63.6%) of 231 visits. In the postintervention group, at least 1 immunization was administered in 132 (51.4%) of 257 visits. The mean age of preintervention children was 16.8 months (range: .3-59) compared with 20.2 months (range: .2-59) in the postintervention group. Forty-six percent of preintervention children were male compared with 53% of postintervention children (P = .9).

The intervention was effectivereducing the number of visits with at least 1 dose given but missing from the database from 37.9% to 0% (Table 1). A total of 128 (37.3%) of 343 preintervention doses were missing from the database. In the preintervention group, 12.4%, 4.1%, 19.3%, and 2% of visits had 1, 2, 3, or 4 doses missing respectively. No doses were missing from the postintervention database; however, 1 dose of varicella vaccine had been entered at the time of vaccine preparation but not removed after the decision not to administer the dose. Missing data caused a substantial number of children to be classified as not on-schedule. According to the medical record, a similar proportion of children in the 2 groups were on-schedule at the end of each visit (Table 1), however, missing data lead to only 64% of preintervention children being classified appropriately according to the database. Preintervention children were 3.7 times more likely than postintervention children to be misclassified as not on-schedule (Table 1).

Six of the 9 study nurses were <40 years old, and 6 had been clinically active for over 10 years. Eight classified themselves as comfortable or very comfortable with computers. Nursing satisfaction with the program was very high. All 9 nurses reported using the program all the time to enter immunizations, 89% said that the program required somewhat or a lot less time, and 100% strongly recommended continued use of the program. All 9 nurses also reported that they would be somewhat or very unenthusiastic about the system if labels were not available.

Funding for the immunization coordinator ended in October 1998. Hence, we had the opportunity to determine if the POS system could be used to collect data without support from a dedicated data-entry person. After the loss of the coordinator, the number of immunization doses entered monthly did not decrease and, in fact, increased substantially (Fig 1).

View larger version (24K): [in this window] [in a new window]   Fig. 1.   Doses entered monthly by age group, January 1997-July 1999.

During the 11 months after the intervention (July 1998-May 1999), a total of 16 525 doses were entered compared with only 10 415 during the same period 1 year earliera 58% increase. This occurred during a time when the number of visits to the clinic remained constant (personal communication, clinic medical director). Before POS immunization entry, only children <10 years old were in the system because of limited data entry time. In the 11 months after the intervention, 12 626 doses were entered for children <10 years old compared with 9799 in the same period 1 year earlier. For children 10 years, the number increased from 616 to 3899 during the same periods (Fig 1).

During 1998 through 1999 the number of forms printed from the system increased substantially. A total of 8273 forms were printed during July 1998 through June 1999, preventing nurses from having to write >101 000 dates. School/camp forms were the most popular form (4933), with the WIC program (2014) and Immunization Certification Forms (1326) also frequently printed.

	DISCUSSION

Top Abstract Methods Results Discussion References

This study demonstrates that immunization entry at the POS can improve data quality and be well-accepted. User-friendly may be one of the most overused words in medical informatics; however, the software in this study did not take long to learn and was quickly implemented. Although the study did not evaluate reasons for the relatively poor data quality before introduction of the intervention tracking software, any of the following problems would have prevented successful entry of a dose into the database: failure of the provider to check the appropriate dose given on the clinic billing sheet; failure of the sheet to be placed into the billing sheet receptacle; failure of the administrative staff to transfer all sheets to the immunization coordinator; and finally failure to enter the doses recorded into the tracking software. Each of these issues is ameliorated by POS entry. Although the study was not able to determine the relative importance of the point-and-click interface and the fact that entry was done at the time of immunization preparation, we believe that both factors contributed to the success of the system.

In this study, 37.3% of doses given were missing from the tracking system database in the preintervention group. This represents a high rate compared with other studies. Our data also showed a substantial amount of variability during the time that the clinic coordinator did immunization entry, with the preintervention study period being one of the lowest months for doses entered. Even with a substantially lower rate of missing doses, however, the probability of being misclassified as not-up-to-date would be very high after 8 routine health care visits (the number recommended before age 2 years). For example, if even 5% of doses were not entered at each visit, the probability of a correctly immunized child being classified as up-to-date (not missing a single dose by age 2 years) would only be 66.3% (.95 to the 8th power) without ongoing data quality reviews.

In this study, the pendulum swung from a large amount of missing data to a single excess dose. Although welcomed by those who have struggled with the work of collecting immunization data, doses entered at the time of preparation that are not removed when the order is cancelled are potentially dangerous. These patients may be mistakenly considered up-to-date but continue to be at risk from vaccine-preventable diseases. To address this problem, periodic reminders to clinicians as well as periodic audits will be necessary.

A limitation of this study is that nurses reported a high level of computer familiarity. Eighty-nine percent of nurses reported being familiar with computers. The nurse who reported not being familiar with computers required approximately 30 minutes of training with 2 refresher sessions. Increased training time may be required in settings where the staff has lower levels of computer proficiency. The study is also limited by its preintervention/postintervention design. This design cannot control for factors that may have changed during the study period. The short time interval between the preintervention and postintervention evaluation should serve to minimize this effect. A randomized trial could be used to confirm our findings. Finally, the possibility of a Hawthorne effect must be considered. Although nurses did not know that their completeness would be evaluated, the novelty of the system could improve attention to detail. Data quality will need to be monitored in the future.

Increasing immunization rates continues to be a high priority in many countries. With several new vaccines added recently, and more to follow, tracking of immunization data will be critical to identify those children in need of missing doses. If we had relied entirely on the coordinator's data, we would have substantially overestimated the number of underimmunized children in our clinic. POS systems such as the one described in this article will be necessary to maintain the quality of immunization data and when designed to fit into clinical workflow can improve quality and save time. In our clinic, we had provided clinicians with immunization summaries before each visit. The summaries were reviewed, updated, and returned to the immunization coordinator to recover missing data. The process was labor-intensive, and not possible without a dedicated immunization coordinator. Ongoing data entry for new patients with previous immunizations and for patients missing data from before implementation of the tracking system requires <4 hours per week and has been included in the clerical responsibilities of a full-time employee.

The CPR is an essential technology; however, CPRs that can be integrated into clinical workflow are still elusive. Many believe that successful pediatric CPRs are just around the corner. POS immunization entry systems such as the one described in this study could provide a nucleus around which pediatric information systems could be built. Integration of POS immunization entry into routine clinical practice will lead to improved immunization data quality and substantial savings of administrative time, making them an essential technology for clinicians providing high-quality preventive services to children.

	ACKNOWLEDGMENTS

The study was supported by grants from the Robert Wood Johnson Foundation Generalist Physician Faculty Scholars Program and the David and Lucile Packard Foundation.

We thank Howard Bauchner, MD, for his assistance in reviewing this manuscript.

	FOOTNOTES

Received for publication Sep 29, 1999; accepted Dec 28, 1999.

Reprint requests to (W.G.A.) Boston University School of Medicine, 91 E Concord St, Maternity 412, Boston, MA 02118. E-mail: badams{at}bu.edu

	ABBREVIATIONS

POS, point of service; CPR, computer-based patient record; PPCC, Pediatric Primary Care Center; BIIS, Boston Immunization Information System; WIC, Women, Infants, and Children.

	REFERENCES

Top Abstract Methods Results Discussion References

1. Lieu TA, Capra AM, Makol J, Black SB, Shinefield HR. Effectiveness and cost-effectiveness of letters, automated telephone messages, or both for underimmunized children in a health maintenance organization. Pediatrics. 1998;101(4). URL: http://www.pediatrics.org/cgi/content/full/101/4/e3
2. Rodewald LE, Szilagyi PG, Humiston SG, Barth R, Kraus R, Raubertas RF A randomized study of tracking with outreach and provider prompting to improve immunization coverage and primary care. Pediatrics. 1999; 103:31-38 [Abstract/Free Full Text]
3. Wilton R, Pennisi AJ Evaluating the accuracy of transcribed computer-stored immunization data. Pediatrics. 1994; 94:902-906 [Abstract/Free Full Text]
4. Richards A, Sheridan J Reasons for delayed compliance with the childhood vaccination schedule and some failings of computerised vaccination registers. Aust N Z J Public Health. 1999; 23:315-317 [Medline]
5. Payne T, Kanvik S, Seward R, Development and validation of an immunization tracking system in a large health maintenance organization. Am J Prev Med. 1993; 9:96-100 [Medline]
6. Hobbs FD, Hawker A Computerised data collection: practicability and quality in selected general practices. Fam Pract. 1995; 12:221-226 [Abstract/Free Full Text]
7. Davis RL, Black S, Vadheim C, Immunization tracking systems: experience of the CDC Vaccine Safety Datalink sites. HMO Pract. 1997; 11:13-17 [Medline]
8. Lieu TA, Black SB, Ray P, Computer-generated recall letters for underimmunized children: how cost-effective? Pediatr Infect Dis J. 1997; 16:28-33 [CrossRef][Medline]
9. Norman LA, Hardin PA, Lester E, Stinton S, Vincent EC Computer-assisted quality improvement in an ambulatory care setting: a follow-up report. Jt Comm J Qual Improv. 1995; 21:116-131 [Medline]
10. Shiffman RN, Liaw Y, Brandt CA, Corb GJ Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc. 1999; 6:104-114 [Abstract/Free Full Text]
11. LeBaron CW, Mercer JT, Massoudi MS, Changes in clinic vaccination coverage after institution of measurement and feedback in 4 states and 2 cities. Arch Pediatr Adolesc Med. 1999; 153:879-86 [Abstract/Free Full Text]