PEDIATRICS Vol. 104 No. 4 October 1999, pp. 997

Observer Bias in Acellular Pertussis Vaccine Trials

To the Editor.

We wish to comment on the recently published article¹ that evaluates the impact of observer bias on the estimate of clinical efficacy of one acellular pertussis vaccine. The authors report that, in the Erlangen trial, the observed incidence in unblinded unvaccinated children (DT) was only 13%, compared with an expected figure of approximately 40%. To determine potential observer bias, the authors analyzed the trial physicians' cough referral rates and used these rates to divide the physicians into three categories indicating their compliance to the trial protocol, which required referral of any cough lasting >7 days. Physicians who had reported an at least 20% incidence of cough were classified as "highly compliant." The authors then recalculated vaccine efficacy for each compliance category and found that the estimated efficacy greatly varied among the categories and that it was inversely proportional to the physician's referral rate, concluding that a bias had occurred.

Because no correlation was found between the cough referral rate and the proportion of unvaccinated (DT) children in each physician's study subject group, the authors concluded that the bias did not depend on unblinded conditions and that a similar bias was thus likely to have occurred in all of the recently conducted pertussis trials. Although the evidence for the Erlangen trial is convincing, extending the conclusions to other studies may be inappropriate, given that the studies differed in terms of the means of conducting cough surveillance, the sensitivity of cough detection, and the type of observer. In particular, we would like to emphasize that the Italian trial differed from the Erlangen trial for the following aspects:²

1. Children were randomized to study vaccine groups, including a DT group, in double-blind conditions.
2. Surveillance of cough among the 15 601 randomized study children was performed exclusively by 121 full-time study nurses specifically hired for this trial; each nurse was responsible for maintaining continuous contact with about 130 study families and was specifically trained to detect and immediately investigate any cough episode lasting at least 7 days.
3. Because vaccine recipients contracting pertussis were expected to present with an atypically mild clinical picture, cough episodes were detected and investigated irrespectively of the physician's evaluation (who, having being trained to diagnose illnesses, is more likely to have a personal opinion about the cause of each episode and to select episodes worthy of microbiologic investigation).
4. The observed rate of cough detection in the blind unvaccinated study group (DT) was 30% (620 episodes out of 2078 person-years of observation). This rate is higher than that reported in the Erlangen study (13%) and is quite close to the expected rate (40%) quoted by Cherry et al.¹
5. The uniformity of surveillance was demonstrated by the observation that the four regions included in the study reported similar cough detection rates, ranging from 29% to 41% of enrolled children.
6. Of the cough episodes detected, 96% were microbiologically investigated through culture of nasopharyngeal aspirate and paired sera samples. Acute specimens were taken on the eighth day of cough (median value) and 68% of all cases were confirmed by culture.

In conclusion, we believe that there is a greater chance of observer bias if the sensitivity of the entire surveillance system is low. In the Italian trial, the fact that the personnel was specifically hired and trained greatly contributed to the high sensitivity of the trial, which was also confirmed by the fact that this trial, compared with the others, had the highest proportion of cases (20%) removed with the use of World Health Organization (WHO) clinical criterion. Thus, we believe that the estimates of vaccine efficacy of the two three-component acellular vaccines studied in the Italian trial should be regarded as valid and not as representing an overestimate. In fact, randomized, double-blinded placebo-controlled clinical trials provide the best conditions for properly evaluating both absolute and relative vaccine efficacy, whereas trials lacking a randomized unvaccinated control group are designed to determine the efficacy of one vaccine compared with another³ and estimates of absolute efficacy are more subject to bias.

Stefania Salmaso, Alberto E. Tozzi, and Marta L. Ciofi degli Atti
Italian Pertussis Trial
Istituto Superiore di Sanità
00161 Rome, Italy

REFERENCES

1. Cherry JD, Heininger U, Stehr K, Christenson P The effect of investigator compliance (observer bias) on calculated efficacy in a pertussis vaccine trial. Pediatrics. 1998; 102:909-912 [Abstract/Free Full Text]
2. Greco D, Salmaso S, Mastrantonio P, A controlled trial of acellular vaccines and one whole cell vaccine against pertussis. N Engl J Med. 1996; 334:341-348 [Abstract/Free Full Text]
3. Fine PEM Implications of different study designs for the evaluation of acellular pertussis vaccines. Dev Biol Stand. 1997; 89:866-883

In Reply.

I enjoyed reading the letter by Salmaso et al regarding our article and their review of observer bias in their study.^1,2 Clearly, their trial was excellent and their methods tended to minimize observer bias and this I noted in a previous publication.³ However, human nature being what it is leads me to stand by our opinion that, "It is likely that all recently completed efficacy trials have been effected by this type of observer bias and all vaccines have considerably less efficacy against mild disease than published data suggest."¹

In our article we suggested that a minimum of 42 children per 100 person-years (42%) should be evaluated for possible pertussis. Salmaso et al note that their rate of work-up in the DT group was 30% whereas our rate of work-up in DT recipients was only 13%. However, the rate of illnesses with coughs of 7 days is markedly dependent on population factors such as family size and day care attendance. Therefore, because family sizes and perhaps day care use are higher in Italy than in our German population, these differences (30% vs 13%) may not be relevant.

Salmaso et al also point out that in their study they had "the highest proportion of cases (20%) removed with the use of the WHO clinical criterion." However, using the same laboratory criteria our percent removed (18%, unpublished data) was similar. The data of both trials suggest less observed bias than the other two trials with available data.³

A strength of our study compared with all other recently completed pertussis vaccine efficacy trials was the prospective use of phone calls every 2 weeks compared with  every 4 weeks. This procedure should have led to less observer bias by parents in our trial compared with the other trials.

Finally, much emphasis has been placed upon the necessity of totally randomized, double-blinded placebo controlled trials.⁴ However, although our unblinded DT group was different from the blinded DTP and DTaP groups, careful correction for the differences had minimal effect on efficacy.^1,2 In contrast, unrecognized bias probably related to observer (parent or study personnel) compliance can result in major differences in reported efficacy. For example, the same lot of the same DTP vaccine was evaluated in two double-blind placebo-controlled trials (Sweden, Stockholm, and Italy) and using the same WHO case definition the efficacy differed by 12%.^2,5

Again the study group in Italy should be complemented for their high level of investigations and their efforts, which tended to reduce observer bias.

James D. Cherry, MD, MSc
UCLA School of Medicine
Los Angeles, CA 90095

REFERENCES

1. Cherry JD, Heininger U, Stehr K, Christensen P The effect of investigator compliance (observer bias) on calculated efficacy in a pertussis vaccine trial. Pediatrics. 1998; 102:909-912
2. Greco D, Salmaso S, Mastrantonio P, A controlled trial of two acellular vaccines and one whole-cell vaccine against pertussis. N Engl J Med. 1996; 334:341-348
3. Cherry JD. Comparative efficacy of acellular pertussis vaccines: an analysis of recent trials. Pediatr Infect Dis J. 1997;16:90-96. Supplement
4. Fine PEM Implications of different study designs for the evaluation of acellular pertussis vaccines. Dev Biol Stand. 1997; 89:866-883
5. Gustafsson L, Hallander HO, Olin P, Reizenstein E, Storsaeter J A controlled trial of a two-component acellular, a five-component acellular, and a whole-cell pertussis vaccine. N Engl J Med. 1996; 334:349-355 [Abstract/Free Full Text]