Establishing Benchmarks for the Hospitalized Care of Children With Asthma, Bronchiolitis, and Pneumonia
BACKGROUND AND OBJECTIVES: Asthma, pneumonia, and bronchiolitis are the leading causes of admission for pediatric patients; however, the lack of accepted benchmarks is a barrier to quality improvement efforts. Using data from children hospitalized with asthma, bronchiolitis, or pneumonia, the goals of this study were to: (1) measure the 2012 performance of freestanding children’s hospitals using clinical quality indicators; and (2) construct achievable benchmarks of care (ABCs) for the clinical quality indicators.
METHODS: This study was a cross-sectional trial using the Pediatric Health Information System database. Patient inclusions varied according to diagnosis: asthma (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] codes 493.0–493.92) from 2 to 18 years of age; bronchiolitis (ICD-9-CM codes 466.11 and 466.19) from 2 months to 2 years of age; and pneumonia (ICD-9-CM codes 480–486, 487.0) from 2 months to 18 years of age. ABC methods use the best-performing hospitals that comprise at least 10% of the total population to compute the benchmark.
RESULTS: Encounters from 42 hospitals included: asthma, 22 186; bronchiolitis, 14 882; and pneumonia, 12 983. Asthma ABCs include: chest radiograph utilization, 24.5%; antibiotic administration, 6.6%; and ipratropium bromide use >2 days, 0%. Bronchiolitis ABCs include: chest radiograph utilization, 32.4%; viral testing, 0.6%; antibiotic administration, 18.5%; bronchodilator use >2 days, 11.4%; and steroid use, 6.4%. Pneumonia ABCs include: complete blood cell count utilization, 28.8%; viral testing, 1.5%; initial narrow-spectrum antibiotic use, 60.7%; erythrocyte sedimentation rate, 3.5%; and C-reactive protein, 0.1%.
CONCLUSIONS: We report achievable benchmarks for inpatient care for asthma, bronchiolitis, and pneumonia. The establishment of national benchmarks will drive improvement at individual hospitals.
- ABCs —
- achievable benchmarks of care
- CPG —
- clinical practice guideline
- CXR —
- chest radiograph
- ICD-9-CM —
- International Classification of Diseases, Ninth Revision, Clinical Modification
- PHIS —
- Pediatric Health Information System
What’s Known on This Subject:
With the publication of evidence-based guidelines for asthma, bronchiolitis, and pneumonia, numerous efforts have been made to standardize and improve the quality of care. However, despite these guidelines, variation in care exists.
What This Study Adds:
This study establishes clinically achievable benchmarks of care for asthma, bronchiolitis, and pneumonia. Using a published method for achievable benchmarks of care, we calculated average utilization among the high-performers, which can serve as achievable goals for local quality improvement.
Clinical practice guidelines (CPGs) are systematically developed statements that can guide providers in decision-making.1 CPGs are intended to reduce variation, which in turn is expected to lower costs and improve outcomes. Despite the availability of national CPGs for 3 of the most common pediatric inpatient conditions (asthma, bronchiolitis, and pneumonia), wide variation in their management continues across US hospitals, leading to excess resource utilization and cost of care.2–7 Improvements have been modest at best; 1 possible reason is that although the guidelines make recommendations, they appropriately do not prescribe specific courses of action in specific patients, leaving those decisions to individual clinicians. To preserve physician autonomy and patient preference, metrics cannot be either extreme (0% or 100%), but we still need achievable goals for these metrics to help clinicians or hospitals measure their performance.
Because there are no currently accepted benchmarks for what constitutes best in class performance for quality measures, hospitals that wish to improve their performance are faced with inventing goals for their improvement projects. In the present article, we offer benchmarks to define what is possible for hospitals to reasonably achieve. We chose to focus on bronchiolitis, asthma, and pneumonia because these 3 conditions account for 10% of all pediatric admissions; each of these conditions also ranks among the top 10 in terms of inpatient costs.8 Although national evidence- and consensus-based CPGs also exist for each of these conditions,9–11 there continues to be widespread variation in care and resource utilization for each.2,4,5
Achievable benchmarks of care (ABCs) were described in 1999 as a way to systematically create clinically relevant benchmarks. ABCs use process-of-care indicators to measure and analyze performance, and derive benchmarks that: (1) represent a measurable level of excellence; (2) are attainable; and (3) are derived from data in an objective, reproducible, and predetermined fashion.12,13 ABC methods have been used to systematically study performance in a variety of clinical conditions.14 In the absence of objective benchmarks for treatment of pediatric inpatient conditions, the objectives of the present study were to measure the 2012 performance of freestanding children’s hospitals using clinical quality indicators and to construct ABCs for the clinical quality indicators for healthy children hospitalized with bronchiolitis, asthma, and pneumonia.
The retrospective cohort study used the Pediatric Health Information System (PHIS) database (Children’s Hospital Association, Overland Park, KS). The PHIS database contains de-identified administrative data, detailing demographic, diagnostic, procedures, and daily billing data (including pharmacy, laboratory testing, imaging, supplies, clinical, and room/nursing) from 42 tertiary care children’s hospitals. This database accounts for ∼20% of all annual pediatric hospitalizations in the United States. Data quality is ensured through a joint effort between the Children’s Hospital Association and participating hospitals, as described previously.15
Patient Population by Diagnosis
PHIS data were used to evaluate hospital-level resource utilization for children requiring hospital-based care (either inpatient or observation) for each of the 3 diagnoses (asthma, bronchiolitis, and pneumonia) from January 1, 2012, to December 31, 2012.
For asthma, children 2 to 18 years of age with a primary discharge diagnosis of asthma were included. To avoid clinical misclassification between asthma and bronchiolitis, we opted to only include children >2 years of age given the prevalence of bronchiolitis in children <2 years of age who are hospitalized with acute wheezing. In addition, patients with secondary diagnoses of bronchiolitis and/or pneumonia were excluded to establish an asthma patient cohort without concomitant lung infection. For bronchiolitis, children 2 months of age to 2 years of age with a primary discharge diagnosis of bronchiolitis were included. This age range was selected because it represents the range addressed in the bronchiolitis guidelines published by the American Academy of Pediatrics.11 We excluded patients with a secondary diagnosis of asthma and/or pneumonia. For pneumonia, children 2 months to 18 years of age with a primary diagnosis of pneumonia were included. Patients with a secondary diagnosis of bronchiolitis and/or asthma were excluded. The International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes used for inclusion are listed in Table 1.
Our goal was to identify hospital-based care for patients with uncomplicated asthma, bronchiolitis, and pneumonia involving previously healthy children with no significant comorbid conditions. Therefore, patients with complex chronic conditions (based on an ICD classification scheme developed and validated by Feudtner et al16) and patients who required any ICU management were excluded.
Clinical Quality Indicators
A total of 15 potential clinical quality indicators were considered: 3 for asthma, 5 for bronchiolitis, and 7 for pneumonia (Table 1). For moderate to severe asthma, the 2007 Expert Panel Report 3 discourages use of antibiotics and a routine chest radiograph (CXR), encourages use of systemic corticosteroid therapy, and, for children >5 years old, recommends albuterol metered-dose inhalers and recommends restricting the use of ipratropium bromide to the initial hours after admission.10 Given the well-documented adherence for bronchodilators and steroids for asthma encounters,5,17 and the bundling of pharmacy bills for some hospitals in the database, we opted to exclude these treatment options as clinical quality indicators. As a result, 3 clinical quality indicators for resource utilization were included for asthma: antibiotic, CXR, and ipratropium bromide utilization. For bronchiolitis, the 2006 evidence-based guidelines discourage the use of routine viral testing, CXR, steroids, antibiotics, and bronchodilators; thus, the 5 clinical quality indicators focus on decreasing these therapeutic and diagnostic treatments.11 Finally, for pneumonia, the 2011 evidence-based guidelines were used to guide the following 5 clinical quality indicators: discourage ancillary testing (complete blood cell count, erythrocyte sedimentation rate, C-reactive protein, and viral testing) and support use of narrow-spectrum antibiotics (amoxicillin, ampicillin, or penicillin).9 Two additional measures from the national pneumonia guidelines were included for pneumonia (ie, encourage blood cultures for moderate to severe pneumonia, the judicious use of macrolides). However, these recommendations are not supported by strong evidence and are therefore not recognized as a clinical quality indictor in the present article. For certain medications, trends in utilization over the course of the hospitalization are clinically relevant (eg, ipratroprium bromide for asthma and bronchodilators for bronchiolitis), and utilization rates over time were therefore calculated. The database only captures medication use by calendar day; thus, to observe trends in utilization over the course of the hospitalization, we calculated utilization rates on days 0, 1, and 2 of admission.
The demographic characteristics of the 3 populations were summarized by using frequencies and percentages. For each measure, we computed each hospital’s use rates and displayed the results in box-plots. Hospitals in which utilization was outside of the fences of the box plots (lower fence: 25th percentile – [75th percentile – 25th percentile]; upper fence: 75th percentile + [75th percentile – 25th percentile]) were considered outliers.
ABCs were computed by using data for the clinical quality indicators. A 3-step method was used to define ABCs: first, hospitals were ranked in order based on the desired performance on a measure; second, the best performing hospitals were selected that comprise 10% of the total patient population for each clinical quality indicator; third, from these hospitals, the benchmark was computed as the sum of the numerators divided by the sum of the denominators.14 For small sample sizes, it is important that the rate be adjusted into an adjusted performance fraction ([x + 1]/[d + 2]), but this adjustment was unnecessary for our large samples.12
Finally, we computed the number of percentage points that each hospital was away from the ABC on each metric and categorized hospitals as: (1) meets benchmark; (2) 0% to 10% of benchmark; (3) 11% to 20% of benchmark; or (4) >20% of benchmark. Results are displayed in a heat map. All statistical analyses were performed by using SAS version 9.3 (SAS Institute, Inc, Cary, NC).
During the study period, 50 051 patient encounters met inclusion criteria: asthma, n = 22 186; bronchiolitis, n = 14 882; and pneumonia, n = 12 983 (Table 2). Selected ABC measures according to diagnosis are listed in Table 3, along with median rates for each clinical quality indicator. Calculated ABC measures for macrolide and blood culture utilization for pneumonia were difficult to interpret given the lack of strong evidence supporting these guideline recommendations, so we opted to not present these data in the included tables and figures. If evidence is interpreted to support blood culture and macrolide utilization, ABC metrics were calculated to be 69.2% and 57.4%, respectively; however, if evidence is interpreted to reduce utilization, calculated ABCs were 0.13% for blood culture and 18.0% for macrolide utilization for pneumonia. Hospital performance on the selected quality indicators are summarized in Fig 1, demonstrating the variability with each indicator. Unlike previous reports of utilization, this study lists the average utilization of listed clinical quality indicators as well as the variation of practice and, more importantly, a goal established by the top performing hospitals.
Figure 2 shows how individual hospitals performed against the established benchmark for selected indicators. When evaluating performance according to condition, performance of asthma at the participating hospitals was closest to the established benchmarks, followed by bronchiolitis and then pneumonia. With respect to the asthma clinical indicators, ipratropium discontinuation after 1 day was the most closely adhered measure, and CXR was the least adhered. Bronchiolitis ABCs were more varied, compared with asthma, with the most success in approaching the ABC for CXR utilization at 32.4%. Overall, pneumonia ABC measures exhibited the lowest performance. In general, pneumonia clinical indicators discourage routine testing and limiting ancillary treatments, and encourage the use of narrow-spectrum antibiotics.
Many hospitals have adopted CPGs and/or order sets in asthma, bronchiolitis, and pneumonia in an attempt to adhere to the published evidenced-based recommendations for diagnosis and management. However, there remains marked variability of care among pediatric hospitals. Unlike previous studies that report median values, we report achievable goals for each of the clinical indicators, which are lower than reported median values of all clinical indicators, except for blood culture and narrow-spectrum antibiotic use for pneumonia. With the advent of achievable benchmarks, an established goal may facilitate guideline adherence locally and nationally.
We demonstrated an ABC of 6.6% for antibiotic use in asthma treatment, a meaningful benchmark. In 2008, inappropriate antibiotic use for asthmatic subjects was specifically investigated in the emergency department setting. In 2 different databases, 22% and 18% of patients with acute asthma without concomitant infection received an antibiotic.18 In 2011, nearly 1 in 6 pediatric ambulatory care visits for asthma ended with a prescription for an antibiotic without another ICD-9 code to justify its use, raising the concern for unnecessary antibiotic use.19 With concern about antibiotic overutilization for asthma, along with previous reports of antibiotic use in asthmatic subjects of ∼20%, an ABC of 6.6% seems to be an appropriate goal. We also demonstrated an ABC of 24.5% for CXR use in asthma management, which is consistent with literature that supports the belief that CXR is not routinely indicated for asthma exacerbations but only in circumstances with clinical suspicion for alternate diagnosis (eg, pneumothorax, foreign body, pneumonia).20
The interquartile range for CXR utilization was greater than that for antibiotic utilization for asthma, and more hospitals were closer to the antibiotic ABC compared with the CXR ABC. Children with asthma who receive a CXR are at increased risk of inappropriately being diagnosed with pneumonia and consequently receiving antibiotics.21 However, in this study, we show that although CXR utilization had a median of 46%, the median for antibiotic utilization was only 15%. Antibiotic utilization may be a good indicator to focus local quality improvement given that there were a few outliers, indicating utilization greater than the 75th percentile; similarly, CXR utilization may be an important clinical quality indicator for local quality improvement efforts given the wide range of performance in these hospitals.
We demonstrated decreasing utilization of ipratropium bromide over the course of hospitalization for asthma, which is consistent with guideline recommendations of limiting ipratropium bromide use to the first hours of hospital evaluation. These data show overall high fidelity to limiting ipratropium usage to the early portion of the hospitalization for most hospitals. The ABC measures for asthma (specifically, CXR and antibiotic utilization) diverge from how most hospitals are practicing and therefore may provide opportunities for local quality improvement initiatives.
The 2006 evidence-based guidelines for bronchiolitis discourage the use of routine viral testing, CXR, steroids, antibiotics, and bronchodilators.11 However, we demonstrated a high use of nonrecommended tests and continued high practice variability across children’s hospitals nationwide. Several factors may contribute to the high variability in bronchiolitis treatment. Although large meta-analyses have failed to find substantial benefit from albuterol22 or racemic epinephrine23 in the inpatient management of bronchiolitis, differentiation between bronchiolitis and wheezing with an atopic phenotype remains challenging.24 The American Academy of Pediatrics guidelines recommend a “carefully monitored trial” of bronchodilators to assess for response, and if no response, medication should be discontinued. Median bronchodilator utilization decreased from 74.4% to 30.3% to 11.4% at day 0, day 1, and day 2 of hospitalization, respectively, which may indicate a trial of bronchodilator use on admission and cessation of use if no benefit is noted. The ABC for >2 days dropped to 0%, possibly representing that the top performers embraced the findings of the available literature. The American Academy of Pediatrics guidelines do not recommend corticosteroids for bronchiolitis, and the calculated ABC for this clinical quality indicator is low at 6.4%; however, median use was almost triple the benchmark, and there were 3 hospital outliers with utilization rates above the 75th percentile. Nine of the 42 hospitals had utilization rates >20% over the ABC measure, and they would potentially benefit from local quality improvement work. Our established ABC for corticosteroid use was lower than recently reported utilization values of 11% to 16% but does represent a reasonable achievable goal given the strong data demonstrating lack of benefit.3,25
The ABC measure for antibiotics in bronchiolitis is 18.5%, almost one-half the recently reported utilization of 32%.3 Antibiotic utilization in bronchiolitis should not be 0% because we know that coinfections such as urinary tract infections or acute otitis media may occur. Therefore, it is helpful to have a benchmark as a clinical quality indicator. Specifically, 16 of the 42 hospitals had utilization rates >20% higher than the ABC measure, again potentially providing an opportunity for local quality improvement work.
Our findings indicate widespread institutional variation in the use of diagnostic testing, such as viral testing and CXR. Although knowledge of the causative virus does not alter clinical outcomes,26 >31 hospitals had viral testing utilization rates >20% over the ABC measure of 0.6%. It may be that some hospitals use mandatory viral testing as a method of grouping patients into a cohort for infection control purpose; however, the quality of evidence for this action is poor, particularly given the high rate of coinfection and/or different serotypes.27 CXR is another clinical quality indicator with tremendous variability; only 13 of the participating hospitals were within 10% of the established benchmark (32.4%). Previously reported utilization has ranged from 45% to 50%, again demonstrating that the achievable benchmark is a goal to lower utilization.3,25
Although routine testing for uncomplicated pneumonia is discouraged, there was large variation of use among the hospitals. Erythrocyte sedimentation rate use was infrequent, but it is noteworthy that its use was more common in the lower performing hospitals. C-reactive protein has a higher utilization rate in both low and high performers, and complete blood cell count was consistently overused in lower performing hospitals. Disappointingly, initial use of narrow-spectrum antibiotics was one of the worst performing clinical indicators. Even using the best performers, the ABC for initial narrow-spectrum antibiotic use was only 60.7%.
Although blood culture testing in severe, complicated, or worsening pneumonia is strongly recommended with moderate-quality evidence,9 there is recent debate regarding the need to perform blood cultures in children hospitalized for pneumonia who are “nontoxic” and fully immunized.28 The British Thoracic Society guidelines, updated in 2011, recommend blood culture only in the setting of severe pneumonia sufficient to require PICU admission and do not recommend routine performance of blood culture tests in milder disease.29 In the era of evidence-based medicine guiding care to reduce utilization and improve outcomes, the benefit of blood cultures in children with mild to moderate, uncomplicated pneumonia, even when hospitalized, is unclear. The rationale for obtaining blood cultures includes the difficulty in determining illness severity at presentation, the changing epidemiology of pneumonia with increasing prevalence of pneumonia-associated complications, and the fact that children with uncomplicated pneumonia at presentation may subsequently develop pneumonia-associated complications such as empyema. In this latter case, the blood culture obtained before antibiotic initiation may be the only opportunity to identify the causative pathogen. Among 330 patients reported by Heine et al,28 almost one-half had a blood culture drawn, and the overall rate of bacteremia was just 1.5% (3.2% of those tested). Of those with positive blood culture results, all would have been identified according to the local guideline, which recommended blood cultures for patients with complicated pneumonia. Although blood culture utilization for pneumonia in hospitalized patients remains a topic of debate, Myers et al30 recently found that the prevalence of bacteremia in children with pneumonia may be higher than previously reported. In that multicenter retrospective study, 56% of children who required hospitalization for pneumonia received blood cultures, and of those, 7% (4.7%–10.1%) had bacteremia. Although studies have shown a low rate of bacteremia in uncomplicated pneumonia, prevalence of bacteremia is higher in children with pneumonia complicated by effusion or empyema, ranging from 13% to 26%.30,31 Deciding early on if a blood culture is required in a patient hospitalized with pneumonia is difficult, particularly because it is not always clear which patients may subsequently develop complicated pneumonia. The ABC for blood culture utilization in pneumonia reported here reflects the current guidelines to obtain a blood culture on admission; however, as with other guideline recommendations, new evidence may alter this benchmark, making it difficult to interpret at this time.
The present study has several limitations. First, and of true significance, many children are cared for in non–tertiary care hospitals and because our data are based on outcomes at tertiary care hospitals, these results may not be generalizable in the non–tertiary care hospital setting. Future efforts should compare these ABCs with benchmarks derived from non-freestanding hospitals. Second, the data were limited to charges incurred in the participating hospital, and resource utilization occurring in the outpatient setting or referring hospitals could not be identified. We were also unable to identify if the utilization occurred in the emergency department or in the inpatient setting. The PHIS database works on calendar days, and it is therefore difficult to identify utilization within 24 hours of admission. For example, although the evidence suggests that ipratropium bromide utilization should be limited to the first hours of admission for a patient with asthma, we were unable to capture that time frame with PHIS. As a result, we used calendar days as a proxy measure. Although we were able to identify the number of days the bronchodilators were used, we were unable to determine the total number of doses administered, to distinguish trial from prolonged utilization. Again, we used calendar days as a proxy measure. Finally, these quality indicators were extracted from guidelines for these common respiratory illnesses in an effort to standardize care; however, they do not reflect length of stay, readmissions, or patient/family satisfaction. It is noteworthy that short lengths of stay,32 along with low rates of condition-specific readmission for asthma, bronchiolitis, and pneumonia,33 make use of length of stay and readmissions as quality measures challenging.
Even for the most common pediatric conditions, in which there are clear evidence-based guidelines for care, there continues to be significant variability in how well hospitals follow these guidelines. We have demonstrated that administrative data can be used to calculate ABCs for the top 3 admission diagnoses in pediatric hospital care. These ABCs represent measurable and attainable goals for standardization of care, and they can be the starting point for individual hospitals to evaluate their performance to a national standard. If the use of ABCs becomes institutionalized, it would allow for integrated, national efforts to decrease resource utilization and enhance the quality of care for children admitted to the hospital with these common diagnoses.
- Accepted May 30, 2014.
- Address correspondence to Kavita Parikh, MD, MSHS, Division of Hospitalist Medicine, Department of Pediatrics, Children’s National Medical Center and George Washington School of Medicine, 111 Michigan Ave NW, Washington, DC 20010. E-mail:
Dr Parikh conceptualized the study, led data interpretation, and drafted the initial manuscript; Dr Hall conducted the statistical analyses and reviewed and revised the manuscript; Drs Mittal, Montalbano, Mussman, Morse, and Shah aided in study design, performed data interpretation, and drafted, reviewed, and revised the manuscript; and Drs Hain and Wilson aided in study design, performed data interpretation, and reviewed and revised the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Bero LA,
- Grilli R,
- Grimshaw JM,
- Harvey E,
- Oxman AD,
- Thomson MA,
- The Cochrane Effective Practice and Organization of Care Review Group
- Christakis DA,
- Cowan CA,
- Garrison MM,
- Molteni R,
- Marcuse E,
- Zerr DM
- Willson DF,
- Horn SD,
- Hendley JO,
- Smout R,
- Gassaway J
- Nkoy FL,
- Fassl BA,
- Simon TD,
- et al
- Florin TA,
- French B,
- Zorc JJ,
- Alpern ER,
- Shah SS
- ↵Bradley JS, Byington CL, Shah SS, et al. The management of community-acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis. 2011;53(7):e25–e76
- American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis
- ↵Kiefe CI, Weissman NW, Allison JJ, Farmer R, Weaver M, Williams OD. Identifying achievable benchmarks of care: concepts and methodology. Int J Qual Health Care. 1998;10(5):443–447
- ↵Vanderweil SG, Tsai CL, Pelletier AJ, et al. Inappropriate use of antibiotics for acute asthma in United States emergency departments. Acad Emerg Med. 2008;15(8):736–743
- Paul IM,
- Maselli JH,
- Hersh AL,
- Boushey HA,
- Nielson DW,
- Cabana MD
- ↵Lozano JM. Bronchiolitis. Clin Evid. 2005;(14):285–297
- Heine D,
- Cochran C,
- Moore M,
- Titus MO,
- Andrews AL
- Harris M,
- Clark J,
- Coote N,
- et al.,
- British Thoracic Society Standards of Care Committee
- ↵Byington CL, Spencer LY, Johnson TA, et al. An epidemiological investigation of a sustained high rate of pediatric parapneumonic empyema: risk factors and microbiological associations. Clin Infect Dis. 2002;34(4):434–440
- Bardach NS,
- Vittinghoff E,
- Asteria-Peñaloza R,
- et al
- ↵Morse RB, Hall M, Fieldston ES, et al. Children’s hospitals with shorter lengths of stay do not have higher readmission rates. J Pediatr. 2013;163(4):1034–1038.e1
- Copyright © 2014 by the American Academy of Pediatrics