Comparative Effectiveness of Interventions for Children Exposed to Nonrelational Traumatic Events
OBJECTIVES: To assess the effectiveness of interventions targeting traumatic stress among children exposed to nonrelational traumatic events (eg, accidents, natural disasters, war).
METHODS: We assessed research on psychological and pharmacological therapy as part of an Agency for Healthcare Research and Quality–commissioned comparative effectiveness review. We conducted focused searches of Medline, Cochrane Library, Embase, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, International Pharmaceutical Abstracts, and Web of Science. Two trained reviewers independently selected, extracted data from, and rated the risk of bias of relevant trials and systematic reviews. We used qualitative rather than quantitative analysis methods because of statistical heterogeneity, insufficient numbers of similar studies, and variation in outcome reporting.
RESULTS: We found a total of 21 trials and 1 cohort study of medium or low risk of bias from our review of 6647 unduplicated abstracts. We generally did not find studies that attempted to replicate findings of effective interventions. In the short term, no pharmacotherapy intervention demonstrated efficacy, and only a few psychological treatments (each with elements of cognitive behavioral therapy) showed benefit. The body of evidence provides little insight into how interventions to treat children exposed to trauma might influence healthy long-term development.
CONCLUSIONS: Our findings serve as a call to action: Psychotherapeutic intervention may be beneficial relative to no treatment in children exposed to traumatic events. Definitive guidance, however, requires far more research on the comparative effectiveness of interventions targeting children exposed to nonrelational traumatic events.
- posttraumatic stress disorder
- comparative effectiveness research
- intervention studies
Approximately two-thirds of children and adolescents younger than age 18 years will experience at least one traumatic event, creating a critical need to identify effective child trauma interventions. Although some children exposed to trauma do not experience long-term negative consequences in terms of psychological and social functioning, many later develop traumatic stress syndromes, including posttraumatic stress disorder (PTSD).1,2 Studies have indicated that childhood traumatic stress syndromes are associated with a high degree of impairment during childhood that can carry into adolescence and adulthood. For example, childhood PTSD increases the risk of several comorbid mental disorders such as depression, substance abuse, conduct disorder, and suicidality.3,4 Studies have observed decreased functioning in several domains (social, home, school, relational) by children and adolescents with PTSD.5 Although several guidelines on the treatment of PTSD during childhood and adolescence exist, the recommendations are inconsistent and largely not based on evidence from high-quality clinical trials or comparative effectiveness reviews (CERs).
This article presents the results of a CER on the topic of interventions for children exposed to traumatic events. An overarching goal of this review is to identify gaps in the current scientific literature and highlight important areas for future research to build the evidence base. It is 1 of 2 CERs on the topic of child trauma sponsored by the Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program. The first CER examined the evidence for interventions addressing exposure to the relational trauma of maltreatment in children aged 14 years and younger.6,7 The current review examines the evidence for interventions that target traumatic stress symptoms and syndromes associated with nonrelational (noninterpersonal) trauma such as accidents, natural disasters, war, and political instability. Specifically, this review focuses on children and adolescents aged 0 to 17 years who are (1) exposed to nonrelational, often single-incident, traumatic events or traumatic experiences and/or (2) experiencing symptoms after such traumatic events. For the sake of brevity, we refer to children and adolescents as “children.”
The reason for conducting 2 companion reviews was to limit clinical heterogeneity through a separate analysis of literature evaluating interventions for children exposed to or affected by nonrelational traumatic events and those addressing the complex relational trauma of child maltreatment. Thus, critical differences in these types of trauma exposures affect the intervention type and delivery method. Although we sought to focus on nonrelational trauma in this review, the inclusion of unspecified or mixed trauma exposures may have covered children exposed to maltreatment, thus creating some unavoidable overlap with the AHRQ review on child maltreatment.6
The research team included a clinical psychologist, a child psychiatrist, a family physician specializing in child trauma, a psychiatric epidemiologist, a developmental psychologist, and several researchers with extensive expertise in AHRQ CER methodology. The protocol and full review are available at http://effectivehealthcare.ahrq.gov.
We defined inclusion and exclusion criteria for this review using the populations, interventions, comparators, outcomes, timing, settings (PICOTS) framework for organizing CERs (Table 1). This article presents results for interventions addressing a prescribed set of outcomes, whereby at least one needed to be traumatic stress symptoms or syndromes. We also sought to determine whether evidence exists for differences in the efficacy of these interventions by specific child or treatment characteristics or by setting of the intervention and to identify adverse events associated with the interventions. However, we found very little information regarding these topics and refer the reader to the AHRQ full review available at www.effectivehealthcare.ahrq.gov for more information.8
Literature Search and Review Strategy
We systematically searched, reviewed, and analyzed the scientific evidence for articles that met our PICOTS criteria. We began with a focused PubMed search on traumatic stress disorders and psychological and pharmacological therapies using a variety of terms, medical subject headings, and major headings. We also searched the Cochrane Library, Embase, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, International Pharmaceutical Abstracts, and Web of Science by using analogous search terms. We limited results to studies published from 1990 onward to ensure current applicability of the therapeutic modalities. Because of limited resources, we restricted the search to studies published in English; this may bias the report to include more studies from English-speaking countries. We conducted quality checks to verify that we had identified studies cited in the articles identified by our search strings and revised and reran our searches accordingly. In addition, we reviewed scientific information packets (SIPs) requested from the developers and distributors of the interventions identified in the literature review. We also examined other gray literature identified from peer and public review comments and manual searches to identify studies that our systematic searches may have otherwise missed and to assess publication bias. Our review of the SIPs and gray literature found no evidence of publication bias.
Two trained members of the research team independently reviewed each of the titles and abstracts against the inclusion/exclusion criteria listed in Table 1. For each article that one or both reviewers chose to include based on the abstract review, a third and fourth reviewer reviewed its full text for eligibility against our inclusion/exclusion criteria. During full-text review, if both reviewers agreed that a study did not meet eligibility criteria (including designation of high risk of bias), the study was excluded. Reviewers resolved conflicts by discussion and consensus when possible; they consulted a senior member of the review team for unresolved conflicts. We applied the same criteria to systematic reviews and primary studies.
For studies meeting our inclusion criteria, a trained reviewer abstracted information into structured evidence tables; a second senior member of the team reviewed all data abstractions for completeness and accuracy. As with review of the full-text articles, we resolved conflicts by discussion and consensus when possible, consulting a senior member of the review team for unresolved conflicts.
Two independent reviewers assessed risk of bias (internal validity) for each study using predefined criteria described in the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews,9 using questions specified in the RTI Item Bank10 and the Cochrane Risk of Bias tool.11 We resolved disagreements between the 2 reviewers by consulting an experienced member of the team. We selected items based on relevance to the topic and anticipated sources of bias and rated each study as having low, medium, or high risk of bias for individual outcomes. In general, a study with a low risk of bias had a strong design, measured outcomes and used statistical and analytical methods appropriately, reported low attrition, and reported methods and outcomes clearly and precisely. Studies with a medium risk of bias did not meet all criteria required for low risk of bias. Although these studies had flaws in design or execution (eg, imbalanced recruitment, high attrition), they provide information (eg, through sensitivity analysis) to allow the reader the ability to evaluate and determine that those flaws did not likely cause major bias. Missing information often led to ratings of medium as opposed to low risk of bias. Studies with a high risk of bias had at least one major flaw that probably caused significant bias, invalidating the results. Major flaws precluded the ability to draw causal inferences between the intervention and the outcome. Examples include poor randomization for randomized controlled trials or failure to control for confounding for observational studies, as well as overall attrition ≥20% or differential attrition ≥15% without appropriate handing of missing data such as the use of intention-to-treat analyses.
Strength of Evidence Grading
We graded strength of evidence (SOE) for all available outcomes in our prespecified list based on the guidance established for the AHRQ EPC program.12 This approach incorporates 4 key domains: risk of bias (including study design and aggregate quality), consistency, directness, and precision of the evidence. We used the SOE grades high confidence, medium confidence, low confidence, or insufficient evidence, as defined by Owens and colleagues.12
Two reviewers assessed each domain for each key outcome and resolved differences by consensus. We used a qualitative process, considering each of the domains, to determine the overall SOE grade for each relevant outcome. For outcomes with only a single study to provide evidence, we evaluated consistency as not applicable. When a study had estimates of effects with confidence intervals that permitted clinically distinct conclusions, we rated that domain as imprecise. When studies provided sufficient information (ie, SD or SE) to calculate confidence intervals around between-group changes, without making assumptions about the correlation between available measures of variance, we calculated confidence intervals for the difference in the change in outcomes for the study groups. For studies that did not provide estimates of variance for between-group differences in outcomes, we relied on measures of statistical significance from between-group adjusted analyses where available or unadjusted analyses if no other data were available.
We also considered whether studies were adequately powered when evaluating precision. For outcomes with a single study with imprecise results and for which power was not ensured, we considered this to be insufficient evidence that the estimate from the single study was sufficiently robust to have any confidence in the finding. For a single study with precise results, we graded it as low. Therefore, although effectiveness is not synonymous with precision or with SOE, individual studies that showed an effect generally merited a rating of low SOE.
We report results from direct comparisons of different interventions. Quantitative analysis was not appropriate (eg, because of heterogeneity, insufficient numbers of similar studies, insufficiency or variation in outcome reporting); thus, we synthesized the data qualitatively. We report magnitude of effect data provided by authors in the studies reviewed. We did not perform additional calculations, with the exception of one study that provided the effect size without the statistical significance level. We did not attempt indirect comparisons, given the heterogeneity of usual care comparators. We present results categorized by population and intervention type.
We assessed the applicability of the evidence according to guidance from Atkins and colleagues.13 We used the PICOTS framework to explore factors that affect or limit applicability.
Results of Literature Searches
Figure 1 presents literature search results. Literature searches through August 3, 2012, identified 6647 unduplicated citations. We excluded 6141 at the title and abstract review stage. For the 506 articles reviewed at the full-text stage, we eliminated 446 before risk-of-bias review based on our PICOTS criteria. After assessing risk of bias for all included studies (before data abstraction), we eliminated 35 studies that we rated high risk of bias. We eliminated studies because of selection bias (eg, poor randomization and lack of allocation concealment for trials, failure to control for confounding factors for observational studies), performance bias (eg, not controlling for concurrently occurring or unintended interventions), attrition bias (eg, substantial loss to follow-up of ≥20%) or differential loss to follow-up of ≥15% without appropriate handling of missing data, detection bias (eg, in outcome assessment), and performance bias (eg, not controlling for concurrently occurring or unintended interventions). We dropped all except 1 of the 35 for multiple reasons. We eliminated only 1 study with a single reason for the high risk of bias rating that invalidated all findings (a 77% dropout rate). Having a study design less rigorous than a controlled trial did not drive our decision to drop the study for high risk of bias; only 4 of the 35 excluded studies had observational (prospective cohort) study designs. Most of the studies removed for high risk of bias tested interventions similar to those included in our review.
The 25 articles included in this review represent 23 studies testing 20 interventions. One study was identified from the SIPs and 2 studies were from additional review of the gray literature. In terms of study design, 16 were randomized controlled trials (RCTs), 6 were cluster RCTs, 2 were prospective cohort studies, and 1 was a systematic review. We assessed 19 included articles as being of medium risk of bias and 5 as being of low risk of bias. We did not assess the risk of bias for the single systematic review that met our criteria because tools such as AMSTAR cannot easily be applied to systematic reviews with no included studies. No other systematic reviews could be used in our review in their entirety because their inclusion/exclusion criteria did not match ours, although we evaluated the citation lists for several systematic reviews for additional studies.
Overall, the evidence from 21 trials and 1 observational study (25 articles) evaluated 6 types of interventions targeting children identified as exposed to trauma (7 studies, 8 articles)14–21 and 13 types of interventions targeting children with trauma exposure who already have symptoms (15 studies, 16 articles).22–37 Psychotherapeutic interventions were marked by substantial heterogeneity in components, dose, frequency, involvement of family members, and mode and method of delivery. The wide variety of approaches presented challenges in how to best combine and categorize the psychotherapeutic interventions in particular.
Although we identified numerous potential interventions in our protocol, very few studies examining these interventions met our inclusion criteria; often the interventions had not yet been implemented among children with trauma from sources other than exposure to maltreatment or sexual abuse.
Treatment Based on Exposure
Seven studies (in 8 articles) on 6 different interventions (trauma-focused cognitive-behavioral therapy [TF-CBT], child and family traumatic stress intervention [CFTSI], 2 different school interventions with elements of CBT, early psychological intervention, and propranolol) met our inclusion criteria (Table 2).14–21 The propranolol study20 and the early psychological intervention study21 found no improvement in any outcomes. All other interventions reported some improvement in ≥1 outcomes.14–19 Notably, 3 of the 4 interventions showing evidence of benefit (TF-CBT and both school group interventions, ERASE Stress and Overshadowing the Threat of Terrorism) compared outcomes from interventions with outcomes from wait-list controls or no intervention.14,15,17–19 The CFTSI trial was the only study showing evidence of benefit with an active group comparator.16
Five studies (4 treatment types) evaluated PTSD diagnosis16–20; of these, 3 studies (2 treatment types, CFTSI and school group ERASE Stress) found evidence of improvement favoring intervention.16–18 Four studies (3 treatment types) evaluated severity of PTSD symptoms17–20; 3 studies, representing 2 treatments, found evidence of improvement favoring intervention (both school group interventions).17–19 Three studies (1 study presented in 2 publications) evaluating PTSD symptoms found evidence of improvement14–16,19; 1 study found no benefit (early psychological intervention).21
Some of the included studies also examined other mental health, physical health, and functional outcomes. Both studies that also evaluated anxiety16,19 reported improvement. Three interventions that also evaluated depression14,15,17,18 reported improvement, whereas the early psychological intervention found no improvement in depressive symptoms.20 All 3 studies that evaluated somatic complaints found evidence of benefit favoring the intervention,17–19 but a single study evaluating physiologic reactivity found no evidence of benefit.20 All 3 studies that evaluated functional impairment found evidence of benefit.17–19 The single study that evaluated behavior problems found no evidence of benefit.21
Table 3 presents detailed findings by outcome, specifying type(s) of exposure, for interventions with some evidence of benefit. We rated the evidence as low for all these outcomes, based on the limited number of studies (generally no more than 1 study per intervention) and small sample sizes.
Treatment Based on Exposure and Already-Occurring Symptoms
We also sought evidence of the effectiveness of interventions designed to treat already-occurring symptoms in children exposed to trauma. Fifteen studies reporting on a subset of outcomes for 13 different interventions met our inclusion criteria: TF-CBT, cognitive processing therapy, narrative exposure therapy (NET), grief- and trauma-focused intervention (GTFI) group and GTFI with coping skills and narrative processing, emotional regulation therapy, eye movement desensitization and reprocessing (EMDR), cognitive-behavioral intervention for trauma in schools (CBITS), trauma and grief component therapy; school group with elements of CBT; imipramine; fluoxetine; and sertraline (Table 4).22–29,31–33,35,36–38
Ten of 13 interventions (presented in 12 studies22–29,35,36–38) evaluated a variety of psychotherapeutic approaches; of these interventions, 5 reported in 7 studies compared outcomes with wait-list controls,22,23,26,27,29,36,37 and 2 compared outcomes with usual care.28,35 Three interventions used active comparators: 1 trial compared outcomes for NET with meditation-relaxation therapy outcomes,24 1 GTFI study compared group therapy with individual therapy,25 and 1 trial examined outcomes for GTFI with coping skills and narrative processing versus GTFI with coping skills only.38
As with the cluster of studies reporting on interventions targeting children exposed to trauma, no pharmacological interventions found any evidence of benefit, and the sertraline study suggested that children in the intervention arm fared worse than those in the control arm.31–33 Three studies with active arms (NET and both GTFI treatments) did not report evidence of benefit for any outcome.24,25,38 All other interventions that compared outcomes with wait-list controls found some evidence of benefit for one or more outcomes.22,23,26,27,29
Four studies evaluated PTSD diagnosis22,24–26,38; of these, 2 (TF-CBT and EMDR) found evidence of improvement favoring intervention arms.22,26 Fifteen studies evaluated PTSD symptoms, but only 4 interventions were graded as having low SOE of improvement.22,23,26,28 One study reported evidence of worse outcomes for sertraline compared with placebo for parent-rated PTSD symptoms and clinician-rated PTSD severity.33
Twelve studies representing 10 interventions also evaluated mental health outcomes, specifically anxiety, depression, and internalizing symptoms.22,23,25–29,33,35–38 Six studies reported no improvement in one or all outcomes evaluated.25,26,29,33,35,38 One22 of 5 interventions reported in 6 studies22,26,29,35,37,38 evaluating anxiety symptoms showed improvements (TF-CBT); 4 interventions reported in 5 studies22,23,27,29,36 of 10 interventions reported in 12 studies22,23,25–29,33,35,36–38 showed improvement in depression; 2 studies found no improvement in internalizing behaviors.26,38Two studies evaluated physical symptoms or general health outcomes; neither found evidence of benefit.24,26
Five studies also evaluated a range of other outcomes. One study suggested evidence of no benefit for quality of life for the intervention arm (sertraline) compared with the placebo arm.33 Two24,26 of 3 studies evaluating general functioning did not find evidence of benefit. A third study found mixed results.29 One study found evidence of benefit for the intervention arm on psychosocial dysfunction.27 One37 of 3 studies29,37,38 found evidence of benefit for the intervention arm on externalizing/conduct problem behavior. No studies found any evidence of benefit for acting out or aggression, shyness, learning problems, quality of life, externalizing/conduct problem behaviors, global distress, anger, and supernatural complaints.
Table 5 presents detailed findings by outcome for interventions with some evidence of benefit. We rated the evidence as low for all these outcomes, based on the limited number of studies (generally no more than 1 study per intervention and no intervention having >2 studies combined) and small sample sizes.
We found 21 trials and 1 cohort study (reported in 25 articles) of medium or low risk of bias from our review of 6647 unduplicated abstracts. We did not find studies that attempted to replicate findings of effective interventions; rather, studies tested unique interventions. No pharmacotherapy intervention demonstrated efficacy. Authors typically evaluated short-term outcomes. The body of evidence provides no insight into how interventions targeting children exposed to traumatic events, with or without symptoms, might influence long-term development. Also, studies demonstrating improvement in outcomes generally compared results of interventions with wait-list controls. With a single exception (CFTSI versus supportive therapy),16 studies comparing interventions with active controls did not show benefit. Some psychotherapy interventions targeting children exposed to traumatic events appear promising based on the magnitude and precision of effects. These interventions were school-based treatments with elements of CBT. We found some evidence, although less compelling, regarding potentially promising interventions targeting already existing symptoms; each also had elements of CBT.
Our exclusions served to focus the review and to control for sources of heterogeneity; nonetheless, these exclusions necessarily limited the scope of this review. First, our outcome criteria limited our review. We required studies to report change in traumatic stress symptoms or syndromes as an outcome to align with our primary objective of examining intervention effectiveness on these outcomes. The criterion requiring traumatic stress symptoms or syndromes as at least 1 study outcome resulted in the exclusion of 16 articles that were identified through our search strings but did not report on traumatic stress symptom outcomes. The nature of trauma interventions targeting other mental health conditions and functioning, such as suicide or conduct problems, may differ in objectives, design, and delivery from trauma interventions targeting traumatic stress symptoms or syndromes. We included these other types of outcomes as secondary outcomes of interest for studies that did examine traumatic stress symptoms or syndromes as an outcome because of the importance of identifying other benefits that may result from a single intervention.
Second, we limited the synthesis to trials and observational studies with low and medium risk of bias. Given the limitations of the included studies and their applicability to other contexts, including high-risk-of-bias studies would likely have increased the pool of evidence without resulting in more actionable evidence.
Data on pharmacological interventions are sparse and marked by methodological limitations. Only 1 trial targeted children exposed to trauma, and 3 trials focused on treatment trials for children already experiencing symptoms. These pharmacologic interventions were small trials and none had findings of benefit. Other types of medications routinely used to treat traumatic stress in adults and children exposed to maltreatment and family violence have not been adequately tested in this population.
Finally, the setting and type of trauma exposure limit the applicability of our findings. Nearly half of the included studies (11 of 23) were conducted outside the United States. Findings may not translate across setting, culture, economic conditions, and trauma type.
Future Research Directions
Future studies on interventions targeting children exposed to trauma other than maltreatment, some of whom already have symptoms, are warranted for several reasons. First, the evidence base for well-designed interventions that lack sufficient bias is small. We found almost no replication of interventions in this population. Also, we found no evidence for several interventions commonly used to treat children with trauma exposures. Although most psychotherapy interventions were manualized for delivery, several did not assess treatment fidelity. Finally, only 4 pharmacotherapy trials met our inclusion criteria, and those trials did not study many of the most commonly prescribed medications for trauma-exposed children.
Second, the sample sizes of the studies included in this review were small to medium. Identifying children with trauma exposure and obtaining informed consent limits the feasibility of recruiting large sample sizes for RCTs. Insufficient funding also may contribute to small sample sizes. The difficulty of conducting studies in this population suggests that future research may require focus on well-designed observational studies, including heightened attention to research involving registry data.
Third, previous studies have used outcome assessment methods that relied largely on child self-reports. Few studies used a clinical interview to assess PTSD diagnosis or other mental health outcomes. Although controversy exists regarding whether PTSD is an appropriate diagnosis for children, determining whether an intervention can affect clinically meaningful syndromes of traumatic stress symptoms requires future research. As noted earlier, few included studies assessed longer-term outcomes.
Our findings support several conclusions and serve as a call to action. Psychotherapeutic intervention may provide benefit relative to no treatment in children with traumatic stress symptoms or exposed to traumatic events and appears not to have associated harms. In addition, definitive guidance requires far more research on the comparative effectiveness of psychotherapeutic or pharmacological interventions targeting children exposed to nonrelational or single-incident traumatic events, with or without symptoms.
- Accepted December 19, 2012.
- Address correspondence to Valerie Hoffman, RTI International, 3040 Cornwallis Rd, Research Triangle Park, NC 27709. E-mail:
- CBT —
- cognitive-behavioral therapy
- CER —
- comparative effectiveness review
- CFTSI —
- child and family traumatic stress intervention
- GTFI —
- grief- and trauma focused intervention
- KIDNET —
- narrative exposure therapy for children
- NET —
- narrative exposure therapy
- PICOTS —
- populations, interventions, comparators, outcomes, timing, settings
- PTSD —
- posttraumatic stress disorder
- RCT —
- randomized controlled trial
- SIP —
- scientific information packet
- SOE —
- strength of evidence
- TF-CBT —
- trauma-focused CBT
Dr Hoffman conceptualized and designed the study, analyzed and interpreted the data, drafted the article, critically revised the article for important intellectual content, gave final approval of the article, provided statistical expertise and technical support, and collected and assembled data; Drs Zolotor, McKeeman, and Blanco conceptualized and designed the study, analyzed and interpreted the data, drafted sections of the article, critically revised the article for important intellectual contributions, gave final approval of the article, and collected and assembled data; Ms Knauer provided administrative, technical, logistic support, collected and assembled data, and gave final approval of the article; Ms Lloyd helped conceptualize the study, assisted in data collection, critically reviewed and revised the manuscript, and approved the final manuscript as submitted; Dr Fraser helped conceptualize the study, critically revised the article for important intellectual contributions, and approved the final manuscript as submitted; and Dr Viswanathan conceptualized and designed the study, analyzed and interpreted the data, critically revised the article for important intellectual contributions, gave final approval of the article, provided statistical, administrative, technical, and logistic support, obtained funding, and collected and assembled data.
FINANCIAL DISCLOSURE: The authors have indicated that they have no financial relationships relevant to this article to disclose.
FUNDING: All phases of this study were supported by Agency for Healthcare Research and Quality contract 290-200-10056-I.
- ↵Goldman Fraser J, Lloyd SW, Murphy RA, et al. Child Exposure to Trauma: Comparative Effectiveness of Interventions Addressing Maltreatment. Comparative Effectiveness Review No. 89. (Prepared by the RTI-UNC Evidence-based Practice Center under Contract No. 290 2007 10056 I.) AHRQ Publication No. 13-EHC002. Rockville, MD: Agency for Healthcare Research and Quality. In press
- Goldman Fraser J,
- Lloyd SW,
- Murphy RA,
- et al
- ↵Forman-Hoffman V, Knauer S, McKeeman J, Zolotor A, Blanco R, Lloyd S, Tant E, Viswanathan M. Child and Adolescent Exposure to Trauma: Comparative Effectiveness of Interventions Addressing Trauma Other Than Maltreatment or Family Violence. Comparative Effectiveness Review No. 107. (Prepared by the RTI International-University of North Carolina at Chapel Hill Evidence-based Practice Center under Contract No. 290-2007-10056-I.) AHRQ Publication No. 13-EHC054-EF. Rockville, MD: Agency for Healthcare Research and Quality. February 2012. Available at: www.effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed January 17, 2013
- ↵Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; February 2011. AHRQ Publication No. 10(11)-EHC063-EF
- ↵Higgins JPT, Altman DG. Assessing Risk of Bias in Included Studies. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons, Ltd; 2008:187–241
- Owens DK,
- Lohr KN,
- Atkins D,
- et al
- ↵Atkins D, Chang S, Gartlehner G, et al. Assessing the Applicability of Studies When Comparing Medical Interventions. Rockville, MD: Agency for Healthcare Research and Quality; January 2011. AHRQ Publication No. 11-EHC019-EF
- Goenjian AK,
- Walling D,
- Steinberg AM,
- Karayan I,
- Najarian LM,
- Pynoos R
- Kemp M,
- Drummond P,
- McDermott B
- Lawrence S,
- De Silva M,
- Henley R
- Copyright © 2013 by the American Academy of Pediatrics