Steps of Signal Refinement

1Check the dataExamine both observed and expected counts and rates. Compare with incidence and prevalence estimates from the literature.MMRV and thrombocytopenia: the original background rate was from all person-time due to so few cases after MMR, a more appropriate post-MMR rate from a recent publication was substituted
2Examine descriptive statisticsTabulate descriptive statistics according to age, gender, and study site. Compare vaccine utilization and outcome rates at the different study sites. Look for secular and seasonal trends.MMRV and ataxia: only 1 site had an excess number of cases, which was because of an increase in miscoding
3Check the computer code and do an equivalent nonsequential analysisCheck both the analysis code and the data-generation code at each study site. Use the same data to perform an equivalent nonsequential analysis, which should provide similar risk estimates.MMRV and seizure: a standard nonsequential logistic regression analysis was performed, and similar results were obtained
4Look for patterns in time from exposure to outcomeLook at the time from vaccine exposure to the outcome by using descriptive histograms. If there is no relationship between vaccination and outcome, the cases should be roughly uniformly distributed. Consider different risk windows. Formal statistical inference can be performed by using the temporal scan statistic, which adjusts for the multiple testing inherent in the many different potential risk windows evaluated.MMRV and seizure: a highly statistically significant cluster was found on days 7–10 after vaccination
5Adjust for additional confoundersAdjust for a different and larger set of potential confounders by using standard nonsequential pharmacoepidemiologic methods on the same data set, which may include more detailed age adjustments, adjustments for seasonal trends using months or sinusoidal curves, adjustments for secular trends, adjustments for day-of-the-week effects, adjustments for chronic disease conditions, adjustments for concomitant vaccines or medications, etc.RotaTeq and gastrointestinal bleeding, first signal: the expected number of cases needed to be adjusted by week of age
6Use other comparison groupsConduct nonsequential analyses by using different comparison groups than the one used in the sequential analysis, which may include historical comparison groups from different time periods, matched controls using different criteria, and different time periods in self-control designs.RotaTeq and gastrointestinal bleeding, second signal: a standard nonsequential logistic regression analysis was done using other-vaccine visits as a concurrent comparison group; no elevated risk was seen
7Conduct chart reviewConduct chart review to exclude erroneously coded cases. This could be a complete review or a review of only a random subsample of the exposed and/or unexposed people. Re-perform the nonsequential analyses with only the chart-confirmed cases.MCV4 and GBS: all the postvaccination cases that had appeared in the automated data were ruled out by chart review
8Compare results for similar outcomesCompare the signal generated by 1 vaccine-outcome pair with results for subdiagnostic groups and with results for similar vaccines and outcomes. For example, if there is a signal indicating an increased risk of febrile seizures, check to determine if there is also an increased risk of fever, even if that by itself would not be of interest.MMRV and seizure: there was an excess number of fever cases in the same period after vaccine as for seizures
9Compare results with other existing dataCompare the results with those from other existing data sets, such as phase III clinical trials, phase IV postmarketing trials, spontaneous adverse-event reporting systems such as the VAERS, and other observational data sets such as electronic health records from a different health plan.MMRV and seizure: results were compared and found to be generally consistent with those of a phase IV postmarketing trial conducted at an independent VSD site
10Collect more data and/or conduct a new studyContinue the prospective monitoring of a vaccine-outcome pair even after a statistical signal has been generated to determine if the effect size increases or decreases over time. Conduct a completely new study designed from scratch, such as a case-control study or a postmarketing randomized trial.HPV and appendicitis: after a very early signal in the adult group, surveillance continued, and the excess relative risk disappeared
  • VAERS indicates Vaccine Adverse Event Reporting System.

  • Source for all but the example column: Kulldorff M. Sequential statistical methods for prospective post-marketing safety surveillance. In: Strom BL, Hennessy S, Kimmel SE, eds. Pharmacoepidemiology. 5th ed. Oxford, England: Wiley; 2011: In press.