Standard 2: Containing Risk of Bias
- Lisa Hartling, PhDa,
- Michele Hamm, BSc, MSca,
- Terry Klassen, MDb,c,
- An-Wen Chan, MD, DPhil, FRCPCd,
- Martin Meremikwu, MBBS, MSc, FMCPede,
- Virginia Moyer, MD, MPHf,
- Shannon Scott, RN, PhDg,
- David Moher, PhDh, and
- Martin Offringa, MD, PhDi for the StaR Child Health Group
- aDepartment of Pediatrics, Alberta Research Centre for Health Evidence, University of Alberta, Edmonton, Alberta, Canada;
- bManitoba Institute of Child Health, Winnipeg, Manitoba, Canada;
- cDepartment of Pediatrics and Child Health, Winnipeg, Manitoba, Canada;
- dDepartment of Epidemiology and Community Medicine, Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Canada;
- eDepartment of Paediatrics, University of Calabar, Nigeria;
- fTexas Children's Hospital, Houston, Texas;
- gFaculty of Nursing, University of Alberta and Alberta Heritage Foundation for Medical Research, Edmonton, Alberta, Canada;
- hClinical Epidemiology Program, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Canada; and
- iChild Health Evaluative Sciences, Research Institute, The Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada
- StaR Child Health
- outcome reporting
- CONSORT —
- Consolidated Standards of Reporting Trials
- ITT —
- intention to treat
- SPIRIT —
- Standard Protocol Items: Recommendations for Interventional Trials
- StaR Child Health —
- Standards for Research in Child Health
There is a crisis of credibility facing the child health research community because of the paucity of reliable estimates of the effects of interventions in children. Associations between risk of bias assessments and treatment effect estimates have important implications, for the clinician, and the families face important challenges as decision-makers stemming from results that exaggerate treatment effectiveness or safety. Consequently, interventions that are not efficacious and potentially harmful may be prescribed, whereas interventions that truly are efficacious may be withheld.1–5
Positive trends in pediatric research have been observed since the first trial was published in 1948. Specifically, there has been a substantial increase in the number of trials published over time, the proportion of randomized to nonrandomized controlled trials, and the proportion of child to adult trials.6 Reporting of methods has also improved; however, methodological quality remains modest.6
Three studies have specifically examined risk of bias in pediatric trials by using the Cochrane Risk of Bias tool.7–9 The results are summarized in Table 1 by risk of bias domain. In 2 reviews, the overall risk of bias was unclear or high for the vast majority of trials.7,8 Both of these articles also revealed that trials at high or unclear risk of bias had exaggerated treatment effects compared with those at low risk of bias. Sequence generation and allocation concealment appear to be the domains that are consistently problematic. Importantly, several variables have been found to be associated with risk of bias including source of funding (industry-sponsored research revealing higher risk of bias), nature of the interventions (behavioral/educational interventions having higher risk of bias), and number of authors (higher risk of bias with fewer authors).9
These analyses demonstrate that there is substantial room for improvement in the methodological and reporting quality of pediatric trials. The association between risk of bias assessments and treatment effect estimates has important implications for decision-making both in terms of false-positive and false-negative results. In practice, this may result in unrealistic expectations of treatment benefit and safety. Inadequate reporting can also have an impact on systematic reviews, which are being used increasingly as a basis for informed decision-making. Although systematic reviews aim to be comprehensive and include all relevant studies, the number of studies contributing to various analyses, be they qualitative or quantitative, is often considerably smaller due to inadequate reporting.
Ensuring methodological rigor and complete reporting is essential for informed clinical decision-making. It is critical that issues of bias are recognized and addressed by every person who conducts trials, reviews trials, funds trials, or uses trials to guide practice.
This standards article was motivated by 2 key factors: (1) emerging evidence demonstrating methodological flaws and weaknesses in child health research and (2) a growing base of empirical evidence quantifying the association between design features in randomized trials and estimates of treatment effects. The following sections address the domains of bias including (1) sequence generation and allocation concealment, (2) blinding, (3) missing outcome data, (4) selective outcome reporting, and (5) other sources of bias.
Distorted Interpretation of Treatment Effects
The types of bias that may occur in randomized controlled trials can generally be classified as selection, performance, detection, attrition, and reporting bias.10Figure 1 illustrates the progression of a trial and where bias may occur. The biases in a given trial can have varying impact on the magnitude and direction of the treatment effect estimates. A growing body of literature provides empirical evidence quantifying the association between specific methodological features of randomized trials and treatment effect estimates. This evidence forms the basis of the Cochrane Risk of Bias Tool, which assesses potential risk of bias in randomized trials based on 6 domains. Table 2 shows the relationship between these domains and the different types of bias. The Cochrane Handbook provides an exhaustive review of the empirical evidence.10 The following is a summary of this evidence.
1. Sequence Generation and Allocation Concealment
Appropriate methods for generating the randomization sequence and concealing the allocation sequence are essential to minimize selection bias. The goal of randomization is to create study groups that are balanced with respect to both known and unknown confounders, whereas allocation concealment ensures that the randomization sequence is unknown to the participants and the person enrolling participants into a trial until allocation to a study group has occurred. On average, inadequate sequence generation results in overestimation of treatment effects by 12%,11–15 whereas inadequate allocation concealment can result in an overestimate of treatment effects by 18%.11–14,16–18 The degree of bias may vary based on the nature of the outcome (eg, less effect for all-cause mortality).19,20
Blinding has long been considered a methodological characteristic of importance.21 Blinding of key individuals in a trial (ie, study participants, study personnel, and outcome assessors) can minimize performance and detection bias. Studies not described as double-blind reveal a 9% overestimate in treatment effect.11–14,16–18 Other studies, however, have revealed no significant association between blinding and effect estimates.16,20 One reason for the discrepancy may be variations in the definition of blinding used in different studies. Experts now maintain that it is important to consider who is blinded in a trial22 and the consequences of inadequate blinding.23
3. Missing Outcome Data
The effects of missing outcome data and how missing data are managed have been investigated in a number of studies. Some suggest more favorable treatment estimates from per protocol analyses*,24–26 or “modified” intention-to-treat† (ITT)27 analyses compared with true ITT analyses. However, other studies have provided no evidence of an association between missing outcome data and effect estimates.11,13,14,18 A meta-epidemiologic study revealed no significant difference in effect sizes overall for studies with adequate (ie, ITT) versus inadequate or unclear approaches to analysis, but results varied across meta-analyses according to the degree of between-trial heterogeneity.28
4. Selective Outcome Reporting
Selective outcome reporting occurs within published studies and is defined as “the selection of a subset of the original variables recorded for inclusion in publication of trials.”29 The most apparent source of bias is when outcomes measured in a trial are deliberately not reported based on their statistical significance; however, other sources of selective outcome reporting exist, such as how the outcome is analyzed, how and when the outcome is measured, and reporting of different subsets of data or subgroups.28–30 A recent systematic review summarized 5 studies that followed inception cohorts from protocol to full publication to examine selective reporting of outcomes.30–35 Four studies “that examined the association between outcome reporting bias and statistical significance found that statistically significant outcomes were more likely to be completely reported than non-significant outcomes.”30 The studies also revealed discrepancies in the primary outcomes proposed and those reported. Other studies have revealed discrepancies in statistical methods reported in protocols and subsequent publications (eg, planned versus reported subgroup analyses).36,37 A study examining the impact of selective outcome reporting on the results of meta-analysis revealed that approximately half of the trials identified as relevant to a systematic review did not contribute to the meta-analysis of patient-important outcomes.38 Further, the effect estimates decreased as the proportion of relevant studies contributing to the meta-analysis increased. Other research has investigated discrepancies due to unpublished versus published scales39 and handling of baseline and end-point data.40
5. Other Sources of Bias
The final domain within the Cochrane tool includes an assortment of study characteristics that may lead to biased results, including factors associated with specific designs (eg, cross-over trials, cluster trials), blocked randomization in unblinded trials, and baseline imbalances.41,42 Sample size is not included in the tool; however, some evidence suggests “small study bias,” whereby trials with few participants may be associated with exaggerated effect estimates.13,43 This variable may be relevant within child health research given the preponderance of trials with small samples.44,45
The following is a brief overview of methodological features that pediatric trialists should consider when designing, conducting, and reporting their research (Table 3). A mnemonic such as presented in Table 4 may also be helpful when designing and reporting a trial.
Appropriate randomization ensures that each study participant has the same probability of being assigned to the respective study groups. The most commonly used tools to generate a randomization sequence are readily available computer programs (eg, Microsoft Excel) or random numbers tables. Any method that does not ensure the same probability is not random, such as assigning patients by odd or even numbers, timing of presentation (eg, day of the week), or based on clinician judgment, patient preference, test results, or treatment availability. For stratified or blocked randomization, trialists should consult specialized sources or statisticians.49
Allocation concealment ensures that the randomization sequence is not known to the participants or those enrolling participants until assignment to study groups has occurred. Allocation concealment is possible in all trials, even those that are not blinded. Moreover, allocation concealment is often mistaken for blinding in trials. A key distinction is that allocation concealment occurs until the point of assignment to study groups, whereas blinding occurs following the point of assignment and for the duration of the trial. The most commonly used methods for allocation concealment are sequentially numbered, opaque, sealed envelopes; sequentially numbered drug containers that are identical in appearance; or central allocation, wherein the individual enrolling the participant consults a central source (eg, pharmacy) for treatment assignment once the patient has consented to participate. A key point is that allocation concealment is not possible if inappropriate methods, such as alternation, have been used to generate the randomization sequence.
Blinding of key study personnel, patients, and outcome assessors should be considered. One common fallacy is that blinding is not applicable when the nature of the intervention precludes blinding of study personnel or patients (eg, educational intervention). In such cases, trialists should have independent, blind outcome assessors and use objective outcomes as well as reliable, valid measurement tools.
The outcome analysis plan should be prespecified, including specification of the following: the primary outcome and how it will be assessed, including the statistical tests to be used; subgroup or adjusted analyses, including variables of importance and how and when subgroup and adjusted analyses will be performed; interim analyses, including when and how they will be performed, what decisions will be made based on these, and how these will be made. Further, before beginning a study, trialists should conduct sample size calculations based on their primary outcome of interest with justification for the parameters used.
Numbers and reasons for withdrawal or loss to follow-up by study group should be tracked. Trialists should specify a priori how they will handle missing data in the analysis. Current evidence supports ITT analyses as less prone to bias than per protocol analyses. Sensitivity analyses using best case and worst case scenarios can add to the interpretation and robustness of the study findings.
All outcomes should be detailed (including benefits and harms), including how and when they will be assessed (eg, which tools, measurement time points) at the outset of the trial. In addition, primary versus secondary outcomes should be specified a priori. All prespecified outcomes should be reported on in the final report. Justification should be provided for changes in outcomes between protocol and publication.
Trialists should declare any financial support received to conduct the trial as well as the role of the sponsor in the design, conduct, analysis, or reporting. Agreements should be formalized at the outset to ensure independence of the researchers to analyze and report the study findings.
Specialized resources should be consulted for issues related to specific trial designs, such as cross-over trials and cluster randomized trials.
Treatment of baseline imbalances should be prespecified. Imbalances can occur despite appropriate randomization. Analyses accounting for baseline imbalances should be reported.
There are several other items that trialists should consider when planning and reporting a trial.
Trial registration. The World Health Organization and medical journal editors worldwide have endorsed prospective registration of trials, and many journals will not publish trials if they have not been publicly registered before recruitment.50 Trial registration enhances the transparency of research. Specifically it ensures accountability in terms of what is planned and done and what is reported. Trial registration can also provide a source for identifying all research in a given area (particularly unpublished and ongoing research), raise awareness of ongoing initiatives, avoid duplication, foster collaboration (eg, to increase potential recruitment sites), and identify gaps in research. The International Committee of Medical Journal Editors has a list of approved registries.51
Trial protocols. The Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) initiative provides guidance regarding items to consider and report when developing a trial protocol.52 The SPIRIT initiative stemmed from empirical evidence demonstrating major discrepancies between trial protocols and publications, including “data suppression, misrepresentation, and manipulation.”53 SPIRIT offers a checklist to improve the quality of protocols and interpretation of study results.54 Closely tied to this initiative is growing interest and a forum for the publication of trial protocols with the intent of providing a permanent, public record of a trial and reduce the potential impact of publication bias.55,56 The journal Trials has a specific remit to publish trial protocols.
Reporting guidelines. The Consolidated Standards of Reporting Trials (CONSORT) Statement provides guidance on reporting a randomized trial.57 It is recommended that trialists review the reporting guidelines even at the protocol development stage to identify important elements: once the trial is complete, clear reporting cannot eliminate bias due to deficient study design. For designs other than the 2-group parallel design, trialists should consult the CONSORT Web site.58 There is increasing evidence that use of reporting guidelines is associated with improved quality.59 Moreover, transparent and accurate reporting should be considered a moral and ethical responsibility.
Three key areas for additional research include (1) quantitative research to improve the evidence related specifically to pediatric research methodology, (2) qualitative research to understand barriers to high methodological quality in pediatric trials, and (3) knowledge translation efforts to reduce the knowledge to practice gap in pediatric trial methodology.
First, the majority of research quantifying the association between methodological characteristics and treatment effect estimates has been based on adult trials. There is growing evidence that interventions may work differently for children than adults due to physiologic and developmental differences and different disease pathophysiology.60–62 Design features may lead to differences in estimates of effect, such as greater response among children to placebo.63,64 The finding of differences between children and adults in their response to treatment may extend to the influence of bias on estimates of treatment effectiveness. Further, there are areas where the empirical evidence is weak or inconsistent, such as the impact of missing outcome data.
Second, there is a need to identify reasons for the suboptimal methodological quality of pediatric trials and to understand the challenges faced by pediatric trialists when conducting research. We have recently completed a survey of a sample of international trialists to determine the following: (1) researcher knowledge and awareness of bias; (2) perceived barriers and facilitators in conducting clinical trials; (3) awareness of existing methodological initiatives, and (4) the perceived utility of potential strategies to use in knowledge translation interventions. We will also assess how researchers’ beliefs and values related to working with children and their caregivers intersect with issues of study design. The findings of this work will help clarify and contextualize the discrepancy between methodological rigor and trial conduct.
Third, there is a need to develop knowledge translation mechanisms to bridge the gap between what is known and how trials are conducted. Accordingly, StaR Child Health is committed to bridging this gap by not only raising awareness of appropriate trial design, conduct, and reporting but also committed to assisting with implementation of evidence-based standards and becoming a “global centre for resources and training relating to the design, conduct, and reporting of clinical research with children.”65
Studies have revealed the methodological shortcomings of pediatric trials. Meta-epidemiologic research provides evidence of the association between methodological features and exaggerated treatment effect estimates. Several tools exist to guide the conduct, design, and reporting of pediatric trials including the Cochrane Risk of Bias tool, the SPIRIT initiative, and the CONSORT Statement. Trialists need to adhere to sound methodological principles in designing and conducting their trials including appropriate sequence generation, adequate allocation concealment, blinding of key study personnel particularly outcome assessors, adequate follow-up and handling of missing outcome data, and reporting of all prespecified outcomes. Further research is needed to quantify the association between methodological characteristics and treatment effect estimates, identify barriers and facilitators to the implementation of sound methodological principles, and develop knowledge translation tools to ensure the effective dissemination and uptake of these principles in child health research.
- Address correspondence to Martin Offringa, MD, PhD, Senior Scientist and Program Head, Child Health Evaluative Sciences, Research Institute, The Hospital for Sick Children, 555 University Ave, Toronto, Ontario, Canada M5G 1X8. E-mail:
Dr Hartling, Ms Hamm, and Dr Klassen wrote the first draft of the article; Dr Chan, Dr Meremikwu, Dr Moyer, Dr Scott, Dr Moher, and Dr Offringa contributed to the writing of the article; Dr Hartling, Ms Hamm, Dr Klassen, and Dr Scott participated in regular conference calls, identified the issues, and drafted the article; Dr Hartling, Ms Hamm, Dr Klassen, Dr Moher, and Dr Offringa participated in identifying the evidence base for Standards for Research in (StaR) Child Health standards; and all authors agree with the final version.
Members of the Star Child Health Risk of Bias Standard Development Group include the above authors as well as Drs Jamie Brehaut, Jeremy Grimshaw, and Prathap Tharyan.
This is the second in a series of standard articles resulting from an ongoing process in which a group of invited experts called a standard development group from StaR Child Health assembles and exchanges information about methods for pediatric trial design, conduct, and reporting. More detailed information about this topic can be found in the introductory article of this supplement or at the StaR Child Health Web site (www.starchildhealth.org).
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
* Per protocol analysis refers to “an analysis of the subset of participants from a randomised controlled trial who complied with the protocol sufficiently to ensure that their data would be likely to exhibit the effect of treatment.” (The Cochrane Collaboration Glossary)
† In an ITT analysis “All participants are included in the arm to which they were allocated, whether or not they received (or completed) the intervention given to that arm.” (The Cochrane Collaboration Glossary)
- Copyright © 2012 by the American Academy of Pediatrics