Effectiveness of Quality Improvement in Hospitalization for Bronchiolitis: A Systematic Review
- Shawn Ralston, MDa,b,c,
- Allison Comick, RNa,c,
- Elizabeth Nichols, BSc,
- Devin Parker, BAc, and
- Patricia Lanter, MDa,b,c
BACKGROUND: Bronchiolitis causes nearly 20% of all acute care hospitalizations for young children in the United States. Unnecessary testing and medication for infants with bronchiolitis contribute to cost without improving outcomes.
OBJECTIVES: The goal of this study was to systematically review the quality improvement (QI) literature on inpatient bronchiolitis and to propose benchmarks for reducing unnecessary care.
METHODS: Assisted by a medical librarian, we searched Medline, Cumulative Index to Nursing and Allied Health Literature, and the Cochrane Library. Studies describing any active QI intervention versus usual care in hospitalized children <2 years of age were included. Data were extracted and confirmed by multiple investigators and pooled by using a random effects model. Benchmarks were calculated by using achievable benchmarks of care methods.
RESULTS: Fourteen studies involving >12 000 infants were reviewed. QI interventions resulted in 16 fewer patients exposed to repeated doses of bronchodilators per 100 hospitalized (7 studies) (risk difference: 0.16 [95% confidence interval: 0.11–0.21]) and resulted in 5.3 fewer doses of bronchodilator given per patient (95% confidence interval: 2.1–8.4). Interventions resulted in fewer hospitalized children exposed to steroids (5 per 100), chest radiography (9 per 100), and antibiotics (4 per 100). No significant harms were reported. Benchmarks derived from the reported data are: repeated bronchodilator use, 16%; steroid use, 1%; chest radiography use, 42%; and antibiotic use, 17%. The study’s heterogeneity limited the ability to classify specific characteristics of effective QI interventions.
CONCLUSIONS: QI strategies have been demonstrated to achieve lower rates of unnecessary care in children hospitalized with viral bronchiolitis than are the norm.
- AAP —
- American Academy of Pediatrics
- ABC —
- achievable benchmark of care
- CI —
- confidence interval
- QI —
- quality improvement
Acute viral bronchiolitis, a clinical entity resulting from several different acute viral lower respiratory tract infections, is one of the most common infectious diseases of childhood.1 Bronchiolitis causes nearly 20% of all acute care hospitalizations for children aged <1 year in the United States.2 The disease generally occupies 1 of the top 3 spots for inpatient medical expenditures in young children annually, and costs are continuing to rise. Unfortunately, despite the high volume and significant economic impact of the disease, clinical trials have failed to establish any specific therapy as effective; therefore, supportive care is the mainstay of an evidence-based approach to the disease.3–7 Nevertheless, overuse of ineffective therapy remains common in bronchiolitis, with overuse of β-agonists, corticosteroids, antibiotics, viral testing, and chest radiography all well documented.8–11
In response to the problem, many hospitals have adopted quality improvement (QI) strategies, such as implementing clinical practice guidelines, intended to standardize the approach to bronchiolitis or to operationalize recommendations made in national evidence-based guidelines.12–15 There are a wide range of published approaches and outcomes, and it remains unclear which strategies are superior. Furthermore, bronchiolitis is a special case in that evidence-based guidelines for this disease do not recommend specific interventions but rather are intended to prevent overtreatment given the lack of effective therapy. Waste due to overtreatment, defined as “waste that comes from subjecting patients to care that, according to sound science and the patients’ own preferences, cannot possibly help them,” has been estimated to cost between $158 billion and $226 billion per year in the United States.16 Unfortunately, much less research has been directed toward evaluating strategies for reducing waste than toward strategies for improving underuse of effective care.17
A further hindrance to improvement efforts in bronchiolitis is the absence of benchmarks for proposed quality measures. Few quality metrics have clearly defined “all or none” goals. Even if a system perfectly eliminated all unwarranted variation, some variation due to individual patient circumstances and preferences would remain.18 The amount of acceptable variation depends on the specifics of each measure, and benchmarks may therefore be difficult to identify. The average performance within a network or a health system has been used as a benchmark, although the downside of such a strategy for systems seeking substantive improvement is obvious. Achievable benchmarks of care (ABC) is a benchmarking method that overcomes this problem by deriving benchmarks from a Bayesian adjusted top 10%.19
Despite the frequency of bronchiolitis and the well-documented problem of unnecessary care, there have been no previous attempts to review QI interventions for the disease. For this systematic review, our specific research question was: in children hospitalized for acute viral bronchiolitis, how well do QI strategies reduce unnecessary care? In a secondary analysis, we sought to derive benchmarks for hospitals attempting to reduce unnecessary care by applying ABC methods to the published literature.
Study Eligibility Criteria
This systematic review was conducted and reported by following recommendations outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.20 We created a review protocol and searched for studies of varying design describing QI interventions directed at improving the care of acute inpatient bronchiolitis. The following inclusion criteria were prespecified: (1) the design was one of the following: cluster randomized trials, before-and-after study studies, cohorts, and QI reports; (2) the exposure was an active QI intervention aimed at improving or standardizing care of inpatients with acute bronchiolitis; (3) the comparison received usual care; and (4) ≥1 of our prespecified outcomes were targeted. We defined QI interventions to include: educational campaigns, creation or active dissemination of clinical pathways or guidelines, implementation of an evidence-based order set, and/or adoption of a respiratory score to clinically manage patient care.
We prespecified β-agonist usage as our primary outcome of interest. This factor was measured 2 ways: the percentage of patients receiving repeated dosing (to allow for a test dose to be given as per the 2006 American Academy of Pediatrics [AAP] guideline) and the mean volume of usage (number of doses) per patient. Because bronchiolitis clinically resembles asthma, β-agonists are the most commonly overused, nonevidence-based therapy.3,10,12 The negative consequences of β-agonist usage in bronchiolitis are unnecessary costs and adverse effects (primarily tachycardia, irritability, and muscle tremors).
Corticosteroid usage, antibiotic usage, and chest radiography (all measured as dichotomous outcomes) were secondary outcomes for this study. Corticosteroids are another of the most commonly overused, nonevidence-based therapies in bronchiolitis, for reasons similar to those provided for bronchodilators.10,12 The negative consequences of steroid usage are adverse effects such as hyperglycemia and immunosuppression. There is ample clinical evidence showing antibiotic overuse in pediatric patients with acute viral bronchiolitis.12 The negative consequences of unnecessary antibiotic use are costs, allergic reactions, potential induction of antimicrobial resistance, and diarrhea (including serious illness from Clostridium difficile). Chest radiography is described in the literature as overused and demonstrated to only rarely contribute to management in this disease.8,12 Cost and exposure to radiation are the primary consequences of overuse of chest radiography. Each of these therapies or tests is explicitly not recommended in the AAP guideline. We did not categorize route of delivery for steroids or antibiotics in terms of oral, intravenous, or intramuscular. Inhaled corticosteroids were not considered.
Readmission rates and length of stay were assessed as balancing measures, which substitute for measures of potential harm of the intervention in QI literature. Although these outcomes could be expected to be improved with standardized, evidence-based management, implementing guidelines could theoretically result in underuse of necessary care as well. The costs of implementation of a guideline or other QI strategy, if reported, could also be considered as a potential harm. Although cost savings are likely to occur due to discontinuation of unnecessary care, these savings will often accrue to the payer and not to the hospitals or physicians involved, and are less commonly reported.
We searched Medline (1946–2013), Cumulative Index to Nursing and Allied Health Literature (1981–2013), and the Cochrane Library. Each search was last updated in October 2013. In collaboration with a professional librarian, we used exploded Medical Subject Headings and key words to generate sets for 2 themes: bronchiolitis and QI interventions. We used the Boolean “OR” to combine terms related to guideline adherence, practice guideline, clinical pathways, QI, and evidence-based medicine; we then used the Boolean “AND” to the intersection between that and the bronchiolitis set. We used no limits but only reviewed English-language abstracts. Our full search strategy can be found in Supplemental Appendix 1.
We manually reviewed the bibliographies of published studies meeting inclusion criteria to locate any studies that were not identified by our electronic database searches. In addition, we searched ClinicalTrials.gov by using the clinical conditions search with the topic “bronchiolitis” and a child (age 0–17 years) age limit for potentially relevant unpublished or ongoing studies.
Study Selection/Data Collection
A single reviewer screened studies according to title and abstract and excluded clearly irrelevant records. Two reviewers independently assessed each potentially eligible study by using full-text review, and discrepancies were resolved by group consensus of all authors. Two reviewers independently extracted data from each included study by using a standardized data collection form. Data were extracted when reported regardless of whether they were prespecified as primary outcomes, adverse events, or balancing measures by the study authors.
Assessment of Methodologic Quality
We developed our own methodologic quality assessment tool by using domains of before-and-after study quality as defined by the Agency for Healthcare Research and Quality21 blended with elements taken from the Standards for Quality Improvement Reporting Excellence guidelines,22 a published rubric for reporting QI studies (Supplemental Appendix 2). Two reviewers independently applied the tool to included studies, and discrepancies were resolved by group consensus.
Measure of Treatment Effect
For the dichotomous outcomes, a summary statistic was generated by using a weighted risk difference and for continuous variables, a weighted mean difference. We converted the pooled risk differences for dichotomous outcomes into a rate per 100 patients hospitalized to facilitate more intuitive understanding of the effect size of the intervention. Comprehensive Meta-Analysis 2.023 was used to calculate summary estimates for each outcome for which appropriate data were available. A priori, a decision was made to use a random effects model due to anticipating study heterogeneity given that these studies were not randomized trials of therapy nor did they purport to be representative samples.
Because the interventions were not uniform, they were qualitatively summarized based on the components of the intervention and the known elements of effective interventions.24
Benchmarks for overuse measures in bronchiolitis were derived by using ABC methods.19 ABCs were calculated as follows: (1) an adjusted performance fraction was calculated for each study by using published methods19; (2) studies were ranked based on the adjusted performance fraction; (3) a second sample of studies was created by moving down this ranked list until a sample approximating 10% of the total sample size was reached; and (4) the achievable benchmark was calculated as the average performance of the second sample.19
Dealing With Duplicate or Missing Data
Several studies reported on overlapping time periods at the same institution, and we specifically extracted only nonduplicate data for our analyses (further described in the Results). Studies defined repeated use of bronchodilators differently. To compare the largest number of studies on this metric, repeated bronchodilator use was defined as either the proportion of patients receiving >1 or >2 bronchodilators, whichever was reported. In cases in which both metrics were reported, we used the number of patients receiving >1 dose.
We prespecified several subgroup analyses aimed at exploring the impact of content and intensity-related characteristics of the reported QI interventions on their efficacy for reducing unnecessary care. Based on the information available from the included studies, we ultimately performed a single subgroup analysis by dividing studies into the 2 categories described here.
Results of Search
Our initial search identified 821 articles, and 14 studies met final inclusion criteria. Figure 1 describes our study selection process in detail.
The characteristics of the 14 studies25–38 in this review are described in Table 1, including our qualitative assessment of the categorical type of intervention (described later in further detail): primarily educational (providing voluntary guidelines, presenting conferences); involving significant process change (implementing standardized order sets); and involving significant process change plus the use of a respiratory score. A respiratory score was loosely defined as any multicomponent numerical representation of respiratory effort. These studies were published over a span of 16 years, from 1998 to the present; however, all but 4 were published before the 2006 AAP bronchiolitis guideline. Sample sizes varied dramatically, and 3 of the studies were from outside the United States.
Two studies were excluded after full-text review, 1 for failure to provide adequate detail to categorize the QI interventions39 and 1 for failure to provide adequate data on outcomes of the intervention.40
Several studies presented duplicate data. Simpson et al32 presented data included in a larger study (Kotagal et al31) and is described but was excluded from the data analysis. Data presented in the 1999 study by Perlstein et al26 was also included in the 2000 study by Perlstein et al27 and are analyzed in our report primarily from the 2000 article, except for one measure (doses per patient of bronchodilator), which is only reported in the 1999 article. Muething et al33 presented data included in the 2 earlier articles by Perlstein et al; for the present analysis, we extracted the data from Muething et al from 2002 to 2004, which were not included in the Perlstein et al articles of 1999 or 2000.
The majority of trials were before-and-after study interrupted time series. Only 3 studies used some form of external control.29,30,34 The lack of control subjects in before-and-after studies raises concerns that the change reported could have been due to secular trends rather than the intervention described. All studies did well on evaluation of whether the before-and-after study population shared similar characteristics. One possible characteristic of high-quality descriptions of QI interventions is providing a description of the intervention with sufficient detail to allow for replication. Eight of our studies met this criterion. We hypothesized that the respiratory score reported in several studies served as a forcing function in QI terms because it provided an automated mechanism for cessation of unproven interventions, and 6 of our studies were assessed as having a forcing function. Six of our studies were deemed exempt from institutional review board review, and the remainder did not address the issue. Institutional review board approval for publication of QI interventions has not always been uniformly required. Appendix 2 provides a summary of selected aspects of study methodologic quality.
Effects of Interventions
Primary Outcome (Bronchodilator Utilization)
The reported QI interventions resulted in a significant decrease in the percentage of patients receiving repeated dosing of bronchodilators. Figure 2 presents the summary estimate for the decrease in patients receiving repeated doses of bronchodilators. The pooled risk difference of 0.16 indicates that 16 fewer patients were exposed to repeated bronchodilators per 100 patients hospitalized after QI interventions (95% confidence interval [CI]: 11–21). Studied patients also received 5.3 fewer doses of bronchodilators (95% CI: 2.1–8.4) (Fig 3).
Secondary outcomes evaluated were systemic corticosteroid use (Fig 4), chest radiography (Fig 5), and antibiotic use (Fig 6). Risk differences for these measures indicate that QI strategies reduced the proportion of patients who received antibiotics by 4 fewer patients exposed per 100 hospitalized (95% CI: 0.6–8); chest radiography by 9 fewer patients exposed per 100 hospitalized (95% CI: 2–15); and systemic corticosteroids by 5 fewer patients exposed per 100 hospitalized (95% CI: 1–9).
Length of hospital stay was the only balancing measure that was reported in a uniform enough manner to assess systematically, and the interventions were associated with a small decrease of 0.2 hospital day in 10 studies analyzed (Fig 7). We also collected data on readmission rates and costs when presented. Readmission rates were reported based on widely differing intervals, from 72 hours to 30 days. No individual study reported a negative impact on readmission rates, and 2 studies reported a statistically significant positive impact of the intervention.27,34 Costs were analyzed in 4 studies, all using differing methods. Four studies reported decreased charges or costs associated with the intervention,26,27,30,36 and 1 study reported no change.25
Qualitative Summary of the Interventions
Because the interventions were not uniform, we sought to create a system of classification to better analyze the effect of the interventions. The studies were qualitatively organized based on the reported elements of the intervention described. We developed 2 categories of intervention, which are described here (and presented in Table 1):
Educational interventions/guidelines/pathways: The basic premise for educational interventions is that providers lack up-to-date information on appropriate disease management for bronchiolitis. Such interventions relied on voluntary participation of clinicians at educational conferences or voluntary clinician usage of pathways or guidelines. We classified evidence integrated into physician order entry systems as primarily educational.
Interventions involving a forcing function: In the case of bronchiolitis, the forcing function in the included studies was a respiratory score. A forcing function is a method of introducing the intervention into a provider’s normal workflow, and it has been described as a particularly successful QI method.41 Respiratory scores are more traditionally thought of as research tools; however, in the case of overuse of bronchodilators in bronchiolitis, they have been adapted to address the phenomenon. Some clinical protocols specified a minimum respiratory score as a threshold value (establishing a minimum severity of illness necessary to justify further intervention). Protocols also allowed the continued use of bronchodilators only if the patient had an improvement in the respiratory score after receiving the medication. This strategy was described in one of the studies as “a prove it or don’t use it” approach.29 We labeled either type of usage of a respiratory score as a forcing function because they were both built-in to the usual workflow. The majority of the studies in this review used the same score, developed and published by Cincinnati Children’s Hospital and Medical Center (CCHMC) and known as the WARM score.
We performed subgroup analysis for the primary outcome measure based on the classification of the interventions described here. There were no differences in outcomes between groups that used a forcing function and those which used educational interventions alone. Subgrouping did resolve heterogeneity in the group of studies using a forcing function (data not presented).
Given that we expected study heterogeneity and prespecified only random effects analysis, we have not presented measures of heterogeneity. The issue of heterogeneity is paramount when combining randomized controlled trials or when seeking a summary estimate of a population-based measure but may not apply to the QI literature in the same manner. Sensitivity analysis removing one study per pass did not resolve heterogeneity. Subgroup analysis according to intensity of intervention resolved heterogeneity for the subgroup of studies by using a forcing function with I2 = 9% and P > .10 for our bronchodilator measure. The validity of our summary estimates must be interpreted with caution given our inability to fully address the issue of study heterogeneity.
We calculated benchmarks from the postintervention data presented in the reviewed studies. Table 2 provides the study-specific postintervention rates of use of nonevidence-based therapies, which we then used to derive the ABC for each measure where appropriate.
Our review found that a variety of QI interventions successfully decrease unnecessary care in acute viral bronchiolitis, although we were unable to designate a particular type of QI intervention as more effective. The varied methods of reporting hindered our ability to compare all studies across all interventions and outcomes. Our qualitative assessment of the studies suggests that interventions incorporating a respiratory score were the most successful at decreasing unnecessary bronchodilator usage. This assessment is consistent with findings in other QI studies; educational initiatives alone seem less successful in effecting change.42 All of the interventions involved establishing local evidence-based guidelines or pathways, with varying levels of process change incorporated.
Benchmarks derived from this literature are significantly lower than current average rates of utilization for these measures based on a recent large multicenter study in a similar population.43 Specifically, the average rate of steroid use was 16%, whereas our proposed achievable benchmark was 1%; average rate of antibiotic use was 33%, and our proposed benchmark was 17%; and average rate of chest radiography was 52%, and our benchmark was 42%. Of particular note, the average bronchodilator utilization rate was 58% and, although we limited our analysis to repeated usage, our benchmark was 16%.
The issue of identifying and benchmarking unnecessary bronchodilator usage merits further discussion because it has plagued pediatricians for decades. In general, the QI protocols we reviewed attempted to exclude patients who were believed to have bronchodilator-responsive lung disease because the goal of these interventions was reduction in bronchodilator use. However, there remains a lingering concern that patients who might benefit from bronchodilators will be missed if the drugs are not trialed, at least in some situations. The QI literature contributes useful knowledge to this debate. Although respiratory scores have not been convincingly validated for predictive utility in the research setting, improvement in a respiratory score has no clinical significance in terms of risk of hospitalization, length of stay or other studied outcomes.44,45 Nevertheless, the literature we reviewed consistently shows that absence of improvement in a score can be used as a QI tool to “weed out” unnecessary bronchodilator use without compromising outcomes. Scores work to decrease bronchodilator usage, either because they actually tell us something about the patient or because they do something to the psychology of the bedside caregiver. The Centers for Disease Control and Prevention estimate that ∼10% of children have asthma.46 Ideally, the rate of repeated bronchodilator use in bronchiolitis would not exceed that number, although our proposed benchmark, at 16%, is close.
There was some evidence for a positive impact of QI interventions on costs and readmission rates, but these findings were not sufficiently uniform to combine for analysis. There were no reports of negative impacts on these measures. For this study, we chose to treat length of stay as a balancing measure, although it was also presented as a primary outcome in several studies. Length of stay was significantly decreased according to QI methods. All of the available evidence on balancing measures favors QI; therefore, it seems unlikely that we have missed significant harms.
Overall Completeness and Applicability of Evidence
With 14 studies located, we expected to find adequate evidence to determine characteristics of successful QI strategies for this disease entity, but variability in outcome reporting hindered our ability to make fine distinctions. From a scientific standpoint, there is likely publication bias affecting all of the QI literature because it is rare to see a study describing an entirely unsuccessful QI project; however, in our cohort we had multiple interventions showing limited impact on subsets of the reported outcomes. Almost all of the studies were performed in freestanding children’s hospitals. Given that the majority of children in the United States are hospitalized outside of freestanding children’s facilities,47 these studies may not be applicable to all settings in which children are hospitalized.
Most of the studies were before-and-after assessments of the QI interventions, with all of the limitations inherent to such study methods, the most salient being the lack of a control for secular trends. Only 1 study controlled for secular trends by using an external database. A national guideline12 was published in the United States in 2006 and may have begun a secular trend toward decreasing utilization. A meta-analysis demonstrating lack of utility for bronchodilators in bronchiolitis was first published in 199748; hence, evidence was available for local QI projects long before it was incorporated into a national guideline. A recent large database analysis43 suggests a decline in diagnostic testing and medication use in bronchiolitis in the period after the guideline publication. A smaller study performed outside of freestanding children’s hospitals49 supports this assertion by showing declining utilization even in the absence of specific QI interventions. However, most of the studies in our review were performed before 2006 and would not have been subject to the secular influence of the guideline.
Our review did not include Embase or Google Scholar searches and did not include any foreign language literature. It did include 3 studies from outside of the United States (Canada, Australia, and Scotland) as well as studies from the nursing literature. We did not perform a formal assessment for publication bias, and we acknowledge that such bias almost certainly exists in this arena. The literature on QI interventions is in its early stages and has yet to confront the issue of publication bias. It is exceptionally rare to see a “negative” QI report, as one of the premises behind publication is to provide information to others hoping to implement similar changes, although information on unsuccessful attempts at change may also be useful. Our only effort to address this issue was to include data on all reported measures in a study, regardless of whether the authors named the measure as a target of the intervention.
Agreements and Disagreements With Other Studies or Reviews
There has been a longstanding discussion in medicine on the topic of guideline uptake, or the lack thereof.50,51 Our review of the QI strategies used in bronchiolitis care revealed mostly local adoption of guidelines and pathways and provides modest evidence for effective uptake and utility of guidelines. Furthermore, our review adds to the decidedly sparse literature on interventions intended to reduce unnecessary care in disease entities characterized by overuse.17 Finally, our proposed benchmarks for usage of each of the nonevidence-based therapies may provide useful targets for any hospital attempting to improve its own performance.
QI strategies can be recommended to reduce unnecessary care in children hospitalized for acute viral bronchiolitis. Benchmarks derived from the literature suggest hospitals can do significantly better than current average rates of utilization. Further research may be needed to determine the most effective elements of an intervention and whether the proposed benchmarks are appropriate outside of freestanding children’s hospitals.
This study was originally conceived and developed as part of a collaborative QI project on inpatient bronchiolitis care sponsored by the AAP Quality Improvement Innovations Network funded in part by the AAP Friends of Children Fund. The authors thank medical librarians Heather Blunt and Tom Mead for assistance with the search strategy. The authors also thank Dr Robin Larson for her guidance and feedback during the development of the manuscript.
- Accepted May 30, 2014.
- Address correspondence to Shawn Ralston, MD, Dartmouth Hitchcock Medical Center, 1 Medical Dr, Lebanon, NH 03745. E-mail:
Dr Ralston conceptualized and designed the study and drafted the initial manuscript; Ms Comick, Ms Nichols, Ms Parker, and Dr Lanter designed the data collection instruments, performed the data collection, conducted the initial analyses, and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: A portion of the work for this study was funded by the American Academy of Pediatrics Friends of Children Fund.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Copyright © 2014 by the American Academy of Pediatrics