The article in this issue of Pediatrics electronic pages by Bhuta and Henderson-Smart1is an example of an important advance in the science of reviewing research. Readers who wish to keep up-to-date with current evidence should pay particular attention to such systematic reviews. This commentary addresses the methodology of a systematic review, the place of such reviews as a foundation of evidence-based practice, and the role of the Cochrane Collaboration in fostering such reviews.
WHAT ARE SYSTEMATIC REVIEWS AND WHY DO WE NEED THEM?
The practice of evidence-based neonatology requires the synthesis of up-to-date, valid evidence concerning the efficacy and safety of treatments, and timely dissemination of that evidence to practitioners. For therapeutic maneuvers, effects of only moderate size can be clinically very important and, to be able to detect such modest but important effects, all the available studies must be considered. Avoidance of bias is essential, so assessments of benefits and risks of therapy must be based primarily on evidence from randomized, controlled trials.
Almost 3000 randomized trials have been reported in the field of neonatology. Busy clinicians, policymakers, or patients cannot readily keep abreast of the evidence from so many trials. Moreover, neonatology is a rapidly changing field where there are frequent shifts in the weight of accumulating evidence. Therefore, there is a pressing need for continuously updated and rapidly disseminated reviews of the results of randomized trials. These can provide efficient access to current best evidence from research and avoid a long gap between the time when the efficacy of a treatment has in fact been established and the time when the treatment is routinely recommended.
A systematic review of the results of randomized, controlled trials uses methods designed to produce unbiased and precise estimates of the effect of a treatment on each of the major outcomes of clinical importance, along with some measure of confidence about the precision of the estimate. The term “systematic review” is used to distinguish its explicit methodology from that of a nonsystematic review where the methodology of the review process may not be expressly stated and opinion may be mixed with evidence.2 When the studies and data permit, the use of statistical methods to aggregate and summarize the results of independent studies (meta-analysis) increases the power to detect treatment effects and can provide more precise estimates of the size of treatment effects than are provided by an individual trial.3
Reviewers conducting a systematic review use a standard method4,5 that can be described under five headings: 1) specifying the objectives of the review, 2) identifying and selecting studies, 3) assessing validity, 4) combining results of independent studies, and 5) making inferences.
Specifying the Objectives of the Review
The main objectives of the review are formulated a priori in the form of focused research questions that specify the population, the intervention, and the outcomes (both beneficial and harmful) of interest. The reviewer frequently will set not only a main objective but also secondary objectives—for example, assessment of the effect of the intervention within subgroups defined by subcategories of population or intervention. A protocol is developed that states the objectives of the review and describes the methods that will be used at each stage of the review. To minimize errors and increase the reliability of the results, an independent co-reviewer is recruited.
Identifying and Selecting Studies
The reviewer states the criteria to identify and select studies that will be considered for inclusion, based on characteristics of their experimental design, the population included, the intervention tested, and the outcomes assessed. Exclusion criteria used to reject studies are also stated. The search strategy used to detect all relevant trials is described. This typically includes electronic searching of Medline, the Cochrane Library, and other bibliographic databases, and the use of additional sources such as trial registries, conference/symposia proceedings, previous reviews, dissertations, expert informants, granting agencies, industry, and personal files. If completed but unpublished trials or ongoing trials are identified, these are noted for inclusion in future updates of the review.
Randomized trials, by their design, offer maximum protection against selection bias at entry. However, opportunities for bias arise at multiple points in the design, conduct, analysis, and reporting of randomized trials. Therefore, the validity of each primary trial is assessed, and the methodologic features of each trial are included as part of the review. Systematic attention is given to the degree of rigor in blinding of the randomization process, blinding of the intervention, completeness of followup, and blinding of the outcome measurement.
Combining the Results of Independent Studies
Outcome data are extracted and tabulated for each included trial. For categorical outcome data, the measures of treatment effect include both relative estimators (relative risk—also known as event rate ratio—and odds ratio) and absolute estimators (risk difference—also known as event rate difference). Derived from these are the relative risk reduction (1-relative risk) and the number needed to treat (1/risk difference). For outcome data measured on a continuous scale, the effect of treatment is expressed as the mean difference. For each outcome, standard statistical methods3 are used to calculate the treatment effect (point estimate and 95% confidence interval) as shown in each trial and in the set of included trials as a whole. This latter estimate, the “typical effect,” is a weighted average, the weights being the inverse of the variance of the estimate provided by each participating trial. Thus, large trials and/or trials in high-risk populations have the greatest influence on the typical effect. These methods are based on either fixed effect or random effects models.4,5 The neonatal reviews generally use a fixed effect model.
The statistical significance of a treatment effect is demonstrated when the confidence interval around the point estimate of treatment effect excludes “no difference.” This would occur when the confidence interval of the estimate for relative risk or odds ratio does not include one, or when the confidence interval for absolute risk reduction or mean difference does not include zero. By convention, the proportion of adverse events (rather than favorable outcomes) is tabulated. In graphical displays of categorical outcomes, treatment effects that lie to the left of the “no effect” vertical line (eg, to the left of a relative risk or odds ratio of 1) indicate a result that favors the experimental group.
Reviewers consider not only the direction and size of the typical treatment effect, but whether the results are reasonably consistent within a set of trials. If there is substantial heterogeneity of the results, the reviewer may elect not to calculate a typical effect.
The strength of inference concerning the probable effects of a treatment depends on the methodologic quality of the primary trials on which the review was based, the degree of consistency of results among the trials contributing to the review, and the degree of confidence that the search for all trials relevant to the review was comprehensive. When the methodologic quality of the included trials is high, the results are consistent within the set of trials, and there is high confidence that all relevant trials have been detected, there is a basis for strong inference regarding the statistical significance and clinical implications of the results.
USES OF SYSTEMATIC REVIEWS
The results of systematic reviews are important in informing providers, consumers, policymakers, and funders of health care about the likely efficacy and safety of treatment, and to aid investigators and funders of research in identifying priorities for new research.
Implications for Practice
In deciding whether to use a treatment, the physician faces four questions that must be distinguished: 1) What are the effects of this treatment, both positive and negative? 2) Is the assessment of these probable treatment effects valid? 3) Are these treatment effects likely to be important? 4) Should I use this treatment in my patient?
Systematic reviews can answer the first two questions. They can contribute to but do not in themselves provide a complete answer to the last two questions. Why is this? A systematic review that is well-done provides valid and precise estimates of the probable benefits and risks of the treatment, including the probable size of these effects. When the size of the benefits outweighs the unwanted side effects, the results of the review provide important evidence that, on average, the treatment is beneficial. However, physicians treat individual patients whose risks and values need to be taken into account in deciding whether to prescribe a treatment in the individual case. Thus, consideration must be given to the relation between the likely benefits and risks when a treatment is prescribed in individual patients at higher or lower risk of developing the adverse event without treatment. For example, when the patient's expected event rate is low, the adverse effects of treatment may outweigh the benefits, whereas when the patient's expected event rate is high, the opposite may be the case. Secondly, individual patients vary in the importance they attach to treatment outcomes, both beneficial and adverse. Incorporation of these values into the treatment decision is essential if treatment is to produce net benefits that are seen as such by the patient.6
The practice of evidence-based medicine requires efficient access to systematic reviews of the best evidence available concerning the probable effects of treatment. These reviews inform, but do not singlehandedly determine, the decision as to whether to prescribe a treatment. Thus, systematic reviews of the evidence are a necessary but not sufficient guide to evidence-based clinical practice.
Implications for Research
Systematic reviews identify research questions which, on the basis of an exhaustive summary of the best evidence, are well-answered. In such cases, further research that addresses the already-answered question is not warranted. Conversely, systematic reviews identify questions that remain important because they have not been answered clearly (in which case further trials may be warranted), or questions that remain important in only certain populations (in which further trials in selected populations are warranted). New research questions may be identified by systematic reviews—for example, the testing in future trials of hypotheses generated by a posteriori analyses undertaken in the review.
PRODUCTION OF NEONATAL SYSTEMATIC REVIEWS: ROLE OF THE COCHRANE COLLABORATION
The Cochrane Collaboration is an international effort to prepare, maintain, and disseminate systematic reviews of the effects of health care. It is named to honor Archie Cochrane who, in a seminal book,7 argued that because resources are limited they should be used to provide health care interventions that have been shown to be effective in properly controlled research. Within the Cochrane Collaboration, members of Collaborative Review Groups prepare systematic reviews based mainly on results from randomized, controlled trials.
One of these review groups is the Neonatal Collaborative Review Group,8 which commissions reviews of randomized trials of treatments in the field of neonatology. These reviews, along with reviews in other fields of health care prepared by members of over 28 other Collaborative Review Groups, are published electronically in the Cochrane Database of Systematic Reviews,9 along with criticisms and comments on the reviews. The neonatal reviews also appear on a web page maintained by the National Institute of Child Health and Human Development (http://silk.nih.gov/silk/cochrane/). Cochrane reviews are updated frequently as new evidence from randomized trials becomes available. Readers who wish to join in the effort of preparing systematic reviews in neonatology should contact the editorial office of the Cochrane Neonatal Collaborative Review Group.8
Authors of Cochrane neonatal systematic reviews are encouraged to prepare papers based on their reviews for publication in peer-reviewed journals. The review by Bhuta and Henderson-Smart1 is the first of an intended series of such reviews to be published inPediatrics electronic pages. Pediatrics electronic pages is particularly well-suited for the publication of systematic reviews. Pediatrics electronic pages is available on the world wide web (http://www.pediatrics.org) in over 190 countries. Extensive tables and figures can be included without the usual space limitations of the printed journal. Readers can browse the text sections of the reviews on-line, and display enlarged versions of selected meta-analytic tables and figures as desired. The Medline links available in Pediatrics electronic pages allow the reader to click on a specific reference and quickly access and display the abstract of the study and search for related studies through Medline. High-quality formatted copies of the review including all text, tables, and figures can be downloaded and printed on the reader's own personal printer. Hyperlinks from Pediatrics electronic pages also provide rapid access to other sites of interest. The Other Resources section on Pediatrics electronic pages provides links to web resources in evidence-based medicine, including links to neonatal reviews published on the National Institute of Child Health and Human Development web site.
The Cochrane reviews published in Pediatrics electronic pages can be found in several other electronic formats, including primary publication in the Cochrane Database of Systematic Reviews, available on CD-ROM and through the Internet10 and the web site maintained by the National Institute of Child Health and Human Development. Because the provenance of each review should be fully disclosed and the same reviewer-authors are involved with each publication, the spectre of plagiarism should not arise, but other questions involving secondary publication may be legitimately raised. These include issues of copyright, academic citation, and how to handle review updates.
Perhaps more than in any other area of scientific inquiry, publication of quantitative reviews of clinical medicine that bear directly on the safest and most effective ways to treat patients demand widespread and rapid dissemination. The growth of evidence-based health care, at the same time that electronic publication has evolved, may be coincidental but is indeed fortuitous. One of the principles underlying the Cochrane Collaboration's foundation was that copyright privileges should not hinder broad publication of reviews. No one venue for publication differs with respect to the degree of background material reviewed or in the synthesis of results. Although each electronic publication may demand some changes in format, all reviews are fundamentally the same in their facts and conclusions and evolve directly from the parent Cochrane neonatal review.
The requirements for acceptable secondary print publication have been recently promulgated by an International Committee of Medical Journal Editors in their uniform requirements for manuscripts submitted to biomedical journals. Secondary publication in the electronic media should, presumably, follow the same criteria for acceptability that are summarized as: 1) authors receive approval from all editors, 2) priority of primary publication is by publication interval of at least 1 week, 3) secondary publication is intended for different readers and an abbreviated version may suffice, 4) secondary publication faithfully reflects data and interpretation of primary version, and 5) readers of secondary publication are notified of primary source.11 TheBritish Medical Journal, Lancet, and Annals of Internal Medicine are among those journals that publish reviews from the Cochrane Database of Systematic Reviews.
The problem of academic citation is more readily resolved. After listing the primary review, authors should mention other publication sources as “also published in,” which clearly identifies the second publication as not being another original piece of work. An analogy might be how one lists the foreign publication of a paper originally published in English in the Journal of the American Medical Association.
Review updates pose a particular problem for the traditional standards of academic publishing. An update of a Cochrane review may be as minimal as adding data from one randomized trial with extremely minor changes to text and none to any conclusions from the review. Particularly egregious examples of plagiarism occur when the updated review is done by another author who may or may not cite the original review. However, even when the same authors are involved concern remains for editors and academic promotion committees that updates be appropriately identified. This is not a new problem as new editions of books and other scholarly work have always ranged from being completely revised to something bordering on the trivial.
Within the Cochrane Neonatal Collaborative Review Group, updates are identified by date and some notification that the review is the nth edition, and the new review identifies changes to the prior edition. It also appears important to archive older reviews for future historical research into the development of therapy. Whether or not updates merit publication in Pediatrics electronic pages will depend on the judgment of authors who wish to submit them and the willingness of editors to accept them, just as occurs in the print media.
- Received March 27, 1997.
- Accepted August 12, 1997.
Reprint requests to (J.C.S.) Department of Pediatrics, McMaster University Medical Center, Room 4G40, 1200 Main St, Hamilton, Ontario, Canada L8N 3-Z5.
- ↵Bhuta T, Henderson-Smart DJ. Elective high-frequency oscillatory ventilation versus conventional ventilation in preterm infants with acute pulmonary dysfunction: systematic review and meta-analyses. Pediatrics. 1997;100(5). URL: http://www.pediatrics.org/cgi/content/full/100/5/e6
- ↵Sinclair JC. Assessing evidence concerning treatment and prevention of diseases of the newborn. In: Sinclair JC, Bracken MB, eds. Effective Care of the Newborn Infant. Oxford, England: Oxford University Press; 1992:3–12
- ↵Bracken MB. Statistical methods for analysis of effects of treatment in overviews of randomized trials. In: Sinclair JC, Bracken MB, eds. Effective Care of the Newborn Infant. Oxford, England: Oxford University Press; 1992:13–18
- ↵Mulrow CD, Oxman AD, eds. Cochrane Collaboration Handbook [updated 21 October 1996]. Available in The Cochrane Library [database on disk and CD-ROM]. The Cochrane Collaboration; Issue 3. Oxford: Update Software; 1996. Updated quarterly. Available from: British Medical Journal Publishing Group, London, England
- ↵Cochrane AL. Effectiveness and Efficiency: Random Reflections on Health Services. London, England: The Nuffield Provincial Hospitals Trust; 1972
- ↵Neonatal Collaborative Review Group, Cochrane Collaboration, Dr. John C. Sinclair, Coordinating Editor, McMaster University Medical Centre, 1200 Main Street West, Hamilton, Ontario, Canada, L8N 3Z5. Telephone: (905) 521-2100 (ext 5611), e-mail:
- ↵Cochrane Library, Cochrane Database of Systematic Reviews. Update Software, Summertown Pavilion, Middle Way, Summertown, Oxford 0X2 7LG, UK. Telephone: 44 1865 513903, Fax: 44 1865 516918
- ↵Synapse Publishing, Inc, PO Box 52146, Edmonton, Alberta, Canada T6G 2T5. E-mail:Telephone: (403) 492-2985, Fax: (403) 492-7253
- Copyright © 1997 American Academy of Pediatrics