Context. Preventing loss of vaccine potency during storage and handling is increasingly important as new, more expensive vaccines are introduced, in at least 1 case requiring a different approach to storage. Little information is available about the extent to which staff in private physicians' offices meet quality assurance needs for vaccines or have the necessary equipment. Although the National Immunization Program at the Centers for Disease Control and Prevention (CDC) in 1997 developed a draft manual to promote reliable vaccine storage and to supplement published information already available from the CDC and the American Academy of Pediatrics, the best ways to improve vaccine storage and handling have not been defined.
Objectives. To estimate the statewide prevalence of offices with suboptimal storage and handling, to identify the risk factors for suboptimal situations in the offices of private physicians, and to evaluate whether the distribution of a new National Immunization Program draft manual improved storage and handling practices.
Design. Population-based survey, including site visits to a stratified, random sample of consenting private physicians' offices. At least 2 months before the site visits, nearly half (intervention group) of the offices were randomly selected to receive a draft CDC manual entitled, “Guideline for Vaccine Storage and Handling.” The remainder was considered the control group. Trained graduate students conducted site visits, all being blinded to whether offices were in the intervention or control groups. Each site visit included measurements of refrigerator and freezer temperatures with digital thermometers (Digi-thermo, Model 15-077-8B, Control Company, Friendswood, TX; specified accuracy ± 1°C). Their metal-tipped probes were left in the center shelf of cold storage compartments for at least 20 minutes to allow them to stabilize. The type of refrigerator/freezer unit, temperature-monitoring equipment, and records were noted, as were the locations of vaccines in refrigerator and freezer, and the presence of expired vaccines. Other information collected included the following: staff training, use of written guidelines, receipt of vaccine deliveries, management of problems, number of patients, type of office, type of medical specialty, and the professional educational level of the individual designated as vaccine coordinator.
Participants. Two hundred twenty-one private physicians' offices known by the Georgia Immunization Program in 1997 to immunize children routinely with government-provided vaccines.
Outcome Measures. Estimates (prevalence, 95% confidence interval [CI]) of immunization sites found to have a suboptimally stored vaccine at a single point in time, defined as: vaccine past expiration date, at a temperature of ≤1°C or ≥9°C in a refrigerator or ≥−14°C (recommended for varicella vaccine) in freezer, and odds ratios (ORs) for risk factors associated with outcomes. We performed χ2 analysis and Student's ttests to compare the administrative characteristics and quality assurance practices of offices with optimal vaccine storage with those with suboptimal storage, and to compare the proportion of offices with suboptimal storage practices in the groups that did and did not receive the CDC manual.
Results. Statewide estimates of offices with at least 1 type of suboptimal vaccine storage included: freezer temperatures measuring ≥−14°C = 17% (95% CI: 10.98, 23.06); offices with refrigerator temperatures ≥9°C = 4.5% (95% CI: 1.08, 7.86); offices with expired vaccines = 9% (95% CI: 4.51, 13.37); and offices with at least 1 documented storage problem, 44% (95% CI: 35.79, 51.23). Major risk factors associated with vaccine storage outside recommended temperature ranges were: lack of thermometer in freezer (OR: 7.15; 95% CI: 3.46, 14.60); use of freezer compartment in small cold storage units (OR: 5.46; 95% CI = 2.70, 10.99); lack of thermometer in refrigerator (OR: 3.07; 95% CI: 1.15,8.20); and failure to maintain temperature log of freezer (OR: 2.70; 95% CI: 1.40, 5.23). Offices that adhered to daily temperature monitoring for all vaccine cold storage compartments, compared with those that did not, were 2 to 3 times more likely to assign this task to staff with higher levels of training, have received a recent visit from the state immunization program, and be affiliated with a hospital or have Federally Qualified Health Center status. In addition, sites using >1 refrigerator/freezer for vaccine storage were more likely to have at least 1 cold storage compartment outside recommended temperature ranges. We found no significant differences in the data reported above between the intervention group (received copy of the draft manual) and the control group (did not receive copy of draft manual), even when controlling for the annual number of immunizations given or the type of office.
Conclusions. Problems with vaccine storage are common and mainly relate to inadequate monitoring of cold storage units or use of freezer units in inappropriate, small refrigerator/freezer units. A modest outlay to purchase equipment and/or train staff could avoid these problems. These results support the following steps: 1) do not store frozen vaccines in freezer compartments in less than full-sized refrigerators (<18 cu ft); 2) monitor temperatures in both the refrigerator and freezer compartments to ensure that setting the freezer compartment control to <−15°C does not lower the refrigerator compartment to <2°C and thereby freeze vaccines that may be damaged by such exposure; 3) prepare a written job description for the duties of vaccine coordinator; 4) review temperature-monitoring practices; 5) follow standard procedures when vaccine temperatures are out of range or a power outage occurs; 6) inventory and rotate vaccines in cold storage each time new vaccines are delivered; and 7) train all vaccine-handling staff in the above and ensure that all have access to the latest authoritative guidance on vaccine storage and that all understand the meaning of temperature range, negative temperatures, Celsius and Fahrenheit scales, and conversion.
Vaccine potency and shelf life depend on correct storage temperatures.1–3 To maintain potency of vaccines, manufacturers include storage guidelines on package inserts with each dose, based on studies of product stability. This is especially relevant as new vaccines are introduced.4–6 Public health departments in many countries have established quality assurance programs for their staff who maintain the vaccine cold chain.7–9 However, in countries where vaccines are primarily administered by private providers, such systems may not be in place, and refrigerator temperatures may be above or below recommended ranges for vaccines.10–13
Given the increasing role private physicians play in immunizing children in the United States, the National Immunization Program (NIP) at the Centers for Disease Control and Prevention (CDC) in 1997 developed a draft manual to promote reliable vaccine storage.14 In addition, information is available in the guidelines of the CDC entitled, “Vaccine Management” (the guidelines, Vaccine Management: Recommendations for Handling and Storage, are also posted at:http://www.cdc.gov/nip/publications/vac_mgt_book.pdf) and theRed Book of the American Academy of Pediatrics, which recommends that immunization providers establish a systematic approach to vaccine storage and quality control.6,,15
We designed this study for 2 purposes: 1) to assess vaccine storage and handling in private provider offices; and 2) to evaluate whether the distribution of a new NIP draft manual had an effect. Unlike previous studies, we examined a large stratified sample of randomly selected sites, obtained information about personnel, and included observations of freezers as well as refrigerators.
Sample and Data Collection
In 1997 we classified all 851 private providers known by the Georgia Immunization Program (GAIP) in 1997 to immunize children with government-provided vaccines as high volume and low volume, depending on whether they had received <400 live oral trivalent polio vaccine (OPV) doses or ≥400 OPV doses during 1996. We then selected 315 sites from this group, including all group practice and solo providers in the high-volume groups, and all Federally Qualified Health Centers (FQHCs) and hospital-affiliated practices. In each of the larger strata of low-volume group practices and solo providers, we selected a 20% random sample (Table 1). The final sample included 159 group practices, 80 solo providers, and 76 community health centers and hospital-affiliated practices. Of the 315 sites thus selected, 52 were found not to be eligible for the study, either because the office no longer existed (ie, mail was returned with no forwarding address or telephone was disconnected) or because our telephone contact informed us that the office was no longer immunizing children. Of 263 eligible offices, 221 agreed to participate, for an overall response rate of 84%. Offices that refused to participate were not significantly (ie, P > .05) different from those that did, judged by the following characteristics: having a pediatrician as lead physician, being a solo practice, or being located in an urbanized county (defined as those with >80 000 residents).
In January 1998, we mailed a letter to all offices in the sample informing them of plans to conduct site visits. Nearly half of the offices (intervention group) were randomly selected to receive a draft CDC manual entitled, “Guidelines for Vaccine Storage and Handling” at least 2 months before the site visit. The remainder was considered the “control group.” Informed consent was obtained in writing, including an agreement to allow site visitors to contact the GAIP for guidance if they found potentially mishandled vaccines. Five pairs of trained graduate students conducted site visits. All site visitors were blinded as to whether offices were in the intervention or control groups. We asked members of the Georgia Health Department's immunization advisory committee to review the information to be collected and conducted pilot tests at offices not included in the sample before beginning the main study.
Each site visit included measurement of refrigerator and freezer temperatures with digital thermometers (Digi-thermo, Model 15-077-8B, Control Company, Friendswood, TX; specified accuracy ± 1°C). Their metal-tipped probes were left in the center shelf of cold storage compartments for at least 20 minutes to allow them to stabilize. Digital thermometer readings were, on average, 0.6°C lower than the office refrigerator thermometers and 0.2°C warmer than the office's own freezer thermometers. The type of refrigerator/freezer unit, temperature-monitoring equipment, and records were noted, as were the locations of vaccines in refrigerator and freezer and the presence of expired vaccines.
During site visits, we asked office respondents ∼50 questions about the following: staff training, use of written guidelines, handling of vaccine deliveries, vaccine storage, handling of vaccines at time of use, offsite vaccination, and management of problems. Additionally, we recorded the number of patients, type of office, type of medical specialty, and the professional educational level of the individual designated as vaccine coordinator. At the conclusion of each office visit, site visitors briefly stated the major findings and gave the office respondent a copy of the CDC/NIP wall chart6 and a free training video. We subsequently mailed each office a summary of recommendations for proper vaccine storage and handling that incorporated a checklist of corrective measures based on recorded observations.
We coded all observations made at the time of site visits and entered the data into EpiInfo, Version 6.04b for analysis with SAS, Version 6.12 (SAS, Cary, NC). We defined the outcome, “optimal storage,” unless otherwise stated, as “all vaccines being stored at temperatures within 1°C of recommended ranges (to allow for error within certifiable limits of thermometer accuracy) and no instances found of unlabeled vaccines being past their expiration dates.” We performed χ2 analysis, Student's t tests, and multiple logistic regression using unweighted frequencies to compare the administrative characteristics and quality assurance practices of offices with optimal vaccine storage to those with suboptimal storage. We used SUDAAN, Version 7.53 (Research Triangle Institute, Research Triangle Park, NC) and weighted frequencies to estimate statewide prevalences of different practices.
Staff Training and Guidance
Most providers (83%) designated a specific person in the office to be responsible for vaccine storage and handling, with a backup in 63% of cases. Approximately two thirds of vaccine coordinators had not attained an educational level of a bachelor's degree or the equivalent. One half were able to locate relevant written reference materials (CDC, American Academy of Pediatrics, GAIP, or vaccine manufacturer) in the office. Most office respondents (73%) could state the correct temperature range for storing refrigerated vaccines. Written instructions on how to take vaccines out of cold storage in preparation for immunizing patients existed in 26% of provider offices, and written instructions for taking care of vaccines in the event of power outages existed in 5%. Sixty-one percent of respondents reported that their training (either on-the-job or professional), specific to vaccine storage and handling, had used some written materials published by an authoritative source. When asked about temperatures, several office respondents were unable to read the mercury thermometer in their office, and some failed to understand the term range. Some thought that colder temperatures were safer for vaccines, and consequently kept the refrigerator controls turned low, increasing the risk for freezing vaccines that were intended to be maintained between 2°C and 8°C.
Vaccine Monitoring and Observed Storage Conditions
Most offices (83%) stored vaccines in a single refrigerator/freezer unit. The refrigerators on average were 4.8 years old. More than half of refrigerators were “full-sized kitchen-style” units (>18 cu ft) with a separately sealed top-mounted freezer. One quarter of offices stored vaccines in smaller refrigerator/freezer units, and of these 36% lacked a separately sealed freezer compartment. Thermometers were more frequently missing in freezer compartments (20%) than in refrigerators (7%; Table 2). More than one quarter of all offices (29%) had at least 1 missing thermometer, and 63% failed to have a thermometer or keep a complete, up-to-date temperature log for all compartments (Table 3). Three of every 4 offices rotated vaccines in all cold storage compartments so that dates closest to expiration were in front (data not shown).
Several offices had temperature logs that indicated temperatures had been outside recommended ranges for weeks. We also observed numerous instances of vaccines being stored above or below recommended temperature ranges or past expiration dates. Table 3 displays the most common problems observed with storage conditions: freezer temperatures above those recommended for storing varicella and/or oral polio vaccine (17%); refrigerator temperatures ≤1°C (15%); and expired vaccines (9%). Altogether, we observed or documented at least 1 vaccine storage problem in 44% (95% confidence interval [CI]: 35.79, 51.23) of all offices visited, using temperature thresholds of 1°C and 9°C in refrigerators, −14°C in freezers of offices receiving Varivax (Merck & Co, Inc, West Point, PA) and −2°C in freezers of offices that received OPV but not Varivax.
We found no significant differences in the data reported above between the intervention group (received copy of draft manual) and the control group (did not receive copy of draft manual), using χ2 analyses, even when controlling for the annual number of immunizations given or the type of office.
Mean freezer temperature readings were significantly higher (−15.12°C vs −18.72°C, respectively, t statistic = −4.25; P = .0001) in less than full-sized (ie, small) refrigerators compared with full-sized units (≥18 cu ft). Freezers lacking temperature logs, freezer compartments in small-sized refrigerators, and freezers without thermometers were between 2.5 and 7 times as likely to have temperatures higher than the recommended range for Varivax, compared with those in full-sized refrigerators (Table 2). Using refrigerators as the unit of analysis, we found that failure to keep a thermometer in the refrigerator was marginally associated with vaccines stored at temperatures outside recommended ranges (P = .020), using thresholds of 2°C and 8°C.
Daily vaccine temperature monitoring was 2 to 3 times more likely to occur in offices (P < .05) that had the following indicators of training, knowledge, or outside inspection: 1) vaccine coordinators (when a coordinator was designated) had at least bachelor's level professional training; 2) office respondent reported a recent visit from the GAIP; 3) office respondent demonstrated knowledge of acceptable temperature ranges for childhood vaccines; and 4) office was affiliated with a hospital or had FQHC status.
However, respondents who could locate copies of national guidelines for vaccine storage and handling did not adhere to better temperature-monitoring practices, compared with those who could not (data not shown).
We found no significant associations (P > .01) between suboptimal vaccine storage and the following variables: number of annual vaccine doses received by the office, medical specialty of lead physician, inability to locate written guidelines, failure to designate a vaccine coordinator, vaccine storage in the door of the refrigerator compartment, or presence of food in the refrigerator. Offices with optimal vaccine storage at the time of the site visit were significantly (P < .01) more likely to store vaccines in only 1 full-sized refrigerator/freezer (compared with multiple units), to keep a thermometer in the freezer, and to maintain a temperature log for each cold storage compartment containing vaccines (data not shown).
This is the first systematic, population-based study of vaccine storage and handling in private medical practices and is the first to evaluate storage of frozen vaccines. Temperature readings were taken in the same manner each time using certified digital thermometers by a group of site visitors trained at the same time. The results can be generalized to an entire state and offer a baseline against which to compare improvements in quality assurance practices. The sample size and design enabled statistical analysis of office risk factors. Key risks for improper vaccine storage found in private physicians' offices were the use of freezer compartments in small refrigerators for Varivax, failure to place thermometer in the freezer or refrigerator, and failure to monitor temperatures in all cold storage compartments.
Our study had several limitations. First, observations of quality assurance practices at the time of the visits may have been biased because the office administrators received advance notice about our visit, and half of the offices had received a vaccine storage and handling manual in draft form by mail before the visit. If such bias exists, actual practices would then be worse than our statewide estimates of vaccine storage problems. Second, our findings in Georgia might not be generalizable to other states. Third, we observed vaccine storage temperatures at a single time point. Obtaining cumulative temperature exposures through continuous recording thermometers or by cold and heat monitors might be expected to show higher overall incidence of vaccines stored outside recommended temperature ranges. Finally, our documentation of suboptimally stored vaccine does not permit any inference about the potency of the observed vaccine. Testing vaccine vials to determine potency when found to be improperly stored is expensive, was beyond the scope of this study, and is inconsistent with the overall concept of quality assurance systems.
Observed reduction in vaccine preventable disease shows overall clinical effectiveness of the US vaccine delivery system. However, the expense of vaccines and vaccinations is sufficiently great that simple and affordable quality assurance systems should be universal. Using thermometers, recording temperatures, and basic training of staff can prevent waste and minimize the chance of any person failing to get the maximum benefit from vaccines received. Overall, our study findings strongly indicate a need to improve the knowledge and quality assurance practices related to vaccine storage and handling in private offices. These results support the following steps:
Do not store frozen vaccines in freezer compartments in less than full-sized refrigerators (<18 cu ft).
Monitor temperatures in both the refrigerator and freezer compartments to ensure that setting the freezer compartment control to <−15°C does not lower the refrigerator compartment to <2°C and thereby freeze vaccines that may be damaged by such exposure.
Prepare a written job description for the duties of vaccine coordinator that includes daily temperature monitoring and recording of all cold storage compartments used for vaccines.
Implement a procedure to review vaccine temperature monitoring practices.
Prepare standard procedures to follow when vaccine temperatures are out of range or a power outage occurs.
Inventory vaccines in refrigerator and freezer on a regular basis and rotate vaccines in cold storage each time new vaccines are delivered.
Train all vaccine handling staff in the above and ensure that all have access to the latest authoritative guidance on vaccine storage and that all understand the meaning of temperature range, negative temperatures, Celsius and Fahrenheit scales, and conversion. All should be able to demonstrate and interpret temperature readings using thermometers available in the practice.
Improving quality assurance for vaccine storage and handling in private offices will require commitment from key office decision-makers such as business managers and physicians to an ongoing process that institutes and maintains these steps.
The support for this study was obtained from the NIP, CDC, through Cooperative Agreement S189-15/17 with the Association of Schools of Public Health.
As project director, K.N.B. contributed to study design, supervised all fieldwork and data collection, analyzed the data, and drafted the article. C.J.R.H. was the principal investigator, designed and directed the overall study, including data analysis, and contributed substantially to the article. As project statistician, C.M. created the final data set and performed all statistical analyses using SAS and SUDAAN. A.P.K served as senior technical advisor, providing support to study design, data collection, interpretation of findings, and revisions of the final article.
We thank Mike Chaney, director of the GAIP, and his staff for their outstanding collaboration and support in every aspect of this study.
- Received August 4, 2000.
- Accepted January 22, 2001.
Reprint requests to (K.N.B.) Women's and Children's Center, Rollins School of Public Health, 1518 Clifton Rd, NE, Emory University, Atlanta, GA 30322. E-mail:
- NIP =
- National Immunization Program •
- CDC =
- Centers for Disease Control and Prevention •
- GAIP =
- Georgia Immunization Program •
- OPV =
- live oral trivalent polio vaccine •
- FQHC =
- Federally Qualified Health Center •
- CI =
- confidence interval •
- OR =
- odds ratio
- ↵Galazka A, Milstien J, Zaffran M. Thermostability of Vaccines: Global Programme for Vaccines and Immunization. Geneva, Switzerland: World Health Organization, Global Programme for Vaccines and Immunization; 1998
- ↵Georgia Immunization Program. Provider Agreement, Vaccines for Children Program. Atlanta, GA: Georgia Immunization Program; 1997
- ↵Centers for Disease Control and Prevention. Prevention of varicella: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Morb Mortal Wkly Rep. 1996;45(RR-11):1–25
- ↵Centers for Disease Control and Prevention. Vaccine Management: Recommendations for Handling and Storage of Selected Biologicals. Atlanta, GA: US Department of Health and Human Services, Public Health Service; 1999
- Casto D,
- Brunell P
- Woodyard E,
- Woodyard L,
- Alto W
- Jeremijenko A,
- Kelly H,
- Sibthorpe B,
- Attewell R
- ↵Centers for Disease Control and Prevention. Guidelines for Vaccine Storage and Handling. Atlanta, GA: US Department of Health and Human Services; 1997
- ↵Pickering L, ed. 2000 Red Book: Report of the Committee on Infectious Diseases. 25th ed. Elk Grove Village, IL: American Academy of Pediatrics; 2000:9–16
- Copyright © 2001 American Academy of Pediatrics