Statistical, Epidemiological, and Risk-Assessment Approaches to Evaluating Safety of Vaccines Throughout the Life Cycle at the Food and Drug Administration

Single-Arm Studies

Early-phase clinical trials may consist of several uncontrolled studies aimed at evaluating a vaccine in relatively healthy populations or, sometimes, in certain special populations. Should unexpected, serious AEs occur, assessing possible relation to the vaccine may be difficult, because there is no randomized control group to serve as a standard for comparison (see “Randomized Clinical Trials” below). In these circumstances, however, it may be possible to compare the observed AE rate obtained by combining the studies, if combining is feasible (see “Meta-analysis” below), to a known background rate derived from external sources. This evaluation could potentially detect a signal to be monitored further in subsequent, larger studies. An obvious drawback to this approach is the potential for biases against which randomization guards. Also, how well a background rate could serve as a comparator would depend heavily on how accurate and reliable that rate is and its applicability to the study populations.

Randomized Clinical Trials

The gold standard in study design for assessing intervention effect is the randomized clinical trial. Studies in which subjects are randomly assigned to an intervention or as controls are more sensitive to small-to-moderate intervention effects than are nonrandomized designs, the latter of which can typically detect only large effects. Randomization promotes balance between groups being compared regarding both known and unknown prognostic factors so that the only real difference between the groups is the intervention administered. Thus, when such designs are accompanied by additional features, such as double-blinding, randomized designs are far less subject to “noise” or bias compared with other designs. Consequently, randomized clinical trials permit direct assignment of causality. It is understandable that randomized clinical trials are used before licensure to evaluate the safety of vaccines, especially because that is the time when randomization is still ethical and, thus, feasible. After a vaccine has been licensed, and especially if it is recommended for use by the Centers for Disease Control and Prevention (CDC) Advisory Committee on Immunization Practices, it may be considered unethical to randomize using a placebo control and possibly even when using an active control. However, despite the fact that prelicensure clinical trials can be randomized, their size is generally more limited than that of postmarketing studies. Only if such trials are very large can there be a reasonable chance of observing less common AEs (see “Large Simple Trials” below).

Commonly Used Approach to Evaluating Safety

Phase III vaccine randomized clinical trials typically have been designed to estimate vaccine efficacy on the basis of clinical disease case definitions. Then, safety is assessed by using the same data set. An exception to this process is when efficacy is indirectly inferred from immune response data gathered from a few hundred participants. In this case, a few hundred subjects might be considered inadequate for evaluating safety, so a larger safety trial might be needed. In many instances, the smaller immunogenicity study could be nested within the much larger safety trial. Sample size is often calculated on the basis of the desired efficacy estimation, rather than safety, although the size of the total safety database needed for licensure is considered (the total safety database is the total number of subjects who received the vaccine, in both controlled and uncontrolled studies combined, before licensure). More consideration is now given to how much safety data are needed from randomized trials before licensing a vaccine.

The typical analysis compares AE rates between the vaccine and control groups. Often, a P value of <.05 is viewed as a statistical safety signal, which does not mean that there is necessarily a safety issue but rather that the AE needs further investigation by subject-matter experts. Note also that the figure .05, although commonly used, is quite arbitrary, and other values such as .025 or .10, for example, might be used as well if the sponsor and the FDA reach agreement. The value chosen is determined by the amount of certainty required to infer a causal relationship. Even relatively large trials will usually not have sufficient power to detect rare AEs.

Large Simple Trials

The concept of large simple trials (LSTs) has existed at least since the 1980s,^2,3 but these trials were used mainly to evaluate efficacy. With the more recent focus on safety evaluation throughout the product life cycle, LSTs are being considered as one of many approaches to assess safety. Before licensure, an LST might afford the last opportunity to evaluate safety in a randomized setting and with all the benefits that randomized designs provide. Applied to safety ascertainment, LSTs would collect data only on serious AEs, because efficacy, immunogenicity, and common AEs would have already been studied in earlier trials. Keeping data collection to a minimum would keep the trial simple.

There are several advantages to using LSTs to investigate the safety of a vaccine. First, as already mentioned, such studies would permit randomization and all the benefits thereof. Second, the simplicity of the data collection would promote rapid assessment of safety. Finally, the minimal data collection would greatly offset the expense of conducting a large trial. The latter advantage would be expected to increase manufacturers' willingness to undertake such large studies.

In submissions to the Office of Vaccine Research and Review (in the FDA's Center for Biologics Evaluation and Research [CBER]), 2 types of LSTs for evaluating safety have been the fixed-sample-size design and the group-sequential design. In the former, the intended final sample size is specified in advance of trial initiation. In the latter type of LST, the maximum sample size is prespecified with the understanding that the trial may stop earlier if a stopping boundary is crossed or if there is a high probability of it being crossed. Prespecified stopping boundaries for safety must be in place for sequential monitoring. This group-sequential design was used to conduct prelicensure surveillance for intussusception in Merck's RotaTeq rotavirus vaccine trial⁴; this vaccine was licensed in 2006.

Types of “Hypotheses” Tested

The International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use E9 document “Guidance on Statistical Principles for Clinical Trials”⁵ refers to 2 general approaches to the statistical analysis of safety data: (1) often called the “conventional” approach and (2) the “noninferiority” approach. The conventional method is what is referred to in the “commonly used approach” described above. In this approach, a vaccine is assumed to be safe (the null hypothesis), and the goal is to discover if it is not safe (the alternative hypothesis) by detecting signals. If we define the relative risk (RR) as the incidence rate in the vaccinated group relative to that in the control group, then a signal is detected if the lower limit of the 2-sided confidence interval excludes a RR of 1, which is analogous to observing a P value of <.05 if the confidence level is 0.95. Note that if the sample size is too small, this approach will tend not to detect signals because of insufficient power. On the other hand, if the sample size is too large, the consequent high power may detect too many false signals. These 2 features should be considered when this conventional approach is contemplated for safety evaluation.

In the noninferiority approach,⁶ the null and alternative hypotheses are the reverse of the conventional hypotheses: the vaccine is assumed to be unsafe (the null), and the goal is to find evidence that it is safe (the alternative) by ruling out signals. With this approach, a signal is ruled out if the upper limit of the confidence interval for RR excludes the prespecified value d, where d is the increase in RR (caused by the vaccine) that is deemed unacceptable. In this case, if the sample size is too small, a noninferiority design will tend not to rule out signals because the confidence intervals for the RR will tend to be too wide to exclude d. Again, insufficient power is the problem. Conversely, as the sample size increases, the more likely this approach is to rule out false signals in contrast to the conventional approach. This latter feature is an asset when using a noninferiority design in a large trial setting.

A disadvantage of this approach, however, is that sponsors and the FDA must agree prospectively on the value of d (the increase in RR deemed unacceptable). However, it should be noted that in safety evaluation, much more so than for efficacy, such statistical criteria should be viewed as guideposts to aid safety assessment and not as rigid criteria to override medical and subject-matter expert judgments.

Meta-analysis

Meta-analysis is another tool for evaluating safety that may be useful in vaccine-licensure decisions. Meta-analysis is a statistical method for combining results from multiple independent studies designed to answer the same question to synthesize all the information into 1 summary statistic.^7,–,9 Individual studies will likely be too small to achieve precise estimates of RR of less common or rare AEs. In addition, sometimes results from multiple studies may be conflicting and demonstrate very different associations. In such cases, meta-analysis might be able to increase statistical power, improve precision of estimates, and provide an overall summary measure from conflicting results.

Although meta-analysis can contribute to the overall safety evaluation of a vaccine, this class of analyses is not without limitations. The quality of a meta-analysis depends, to a large extent, on the quality of the individual studies and the integrity of the process of combining them. Note that it will rarely be appropriate to simply pool all the studies' data and analyze as if they are from a single study. If the separate studies are poorly designed, or compliance is not good, then a meta-analysis of such studies cannot be a panacea to remedy those problems. Also, if the pooling process does not account, in statistically appropriate ways, for the fact that the data come from distinct trials, the results may be questionable. Even if the separate studies are of high quality, there may be a lack of compatibility among them. For example, if the enrolled populations, doses, case definitions, and intensity of surveillance for AEs are not comparable, then a meta-analysis of these high-quality studies may not be advisable. If a series of trials occurred over a long period of time, there may be time-trend differences such as changes in knowledge of the disease, AE definitions, and medical practice. There is also a potential for publication bias if studies with positive findings are more likely to be published, because the meta-analyst can combine only studies that are known. There is also a distinction between posthoc and preplanned meta-analyses. The former are more likely to be fraught with the problems just mentioned, but preplanned meta-analysis, as part of an overall product-development plan, can permit certain studies to be planned so that they are as compatible as reasonably possible.

Signal-Detection Tools

Signal Confirmation (Hypothesis Testing)

Once the existence of a signal has been verified, more definitive epidemiologic studies are implemented. Because conclusions of the FDA/CBER regarding the safety of medical products have high potential impact (ie, affect physician and public confidence in a vaccine program and contribute to decisions about whether a product remains on the market), rigorous methods for hypothesis testing are used to ensure high data quality. Generally, these methods include medical record review to confirm diagnoses, onset date, confounding factors, and exposure to the vaccine. These studies use logistic regression and other analytical methods to estimate RR and attributable risk.

Although the most common AEs have usually been identified during the clinical trials, serious AEs are generally uncommon and require large study populations for rigorous hypothesis testing. For this purpose, the FDA/CBER and the CDC use networks of databases from managed care organizations, including federal partners (the Department of Defense, the Veteran's Administration, and the Indian Health Services), private partners such as the Vaccine Safety Datalink, and, via research contracts, other outside institutions. The FDA/CBER also participates in multinational investigation efforts such as the World Health Organization's pilot international collaborative analytical study of the risk of Guillain-Barré Syndrome after H1N1 influenza vaccination. Such collaborative studies are expected to become important contributors to the armamentarium for global vaccine-safety monitoring.

In addition to these efforts to improve the ability of the FDA/CBER to directly monitor vaccine safety after licensure, sponsors are requested to submit a pharmacovigilance plan at the time of license application in accordance with the 2005 International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use E2E guidance.¹⁹ Typically, sponsors will commit to conducting large observational safety-monitoring studies, and in some cases the Food and Drug Administration Amendment Act gives the FDA authority to require studies to evaluate serious safety concerns.