Impact of a National Guideline on Antibiotic Selection for Hospitalized Pneumonia
BACKGROUND: We evaluated the impact of the 2011 Pediatric Infectious Diseases Society/Infectious Diseases Society of America pneumonia guideline and hospital-level implementation efforts on antibiotic prescribing for children hospitalized with pneumonia.
METHODS: We assessed inpatient antibiotic prescribing for pneumonia at 28 children’s hospitals between August 2009 and March 2015. Each hospital was also surveyed regarding local implementation efforts targeting antibiotic prescribing and organizational readiness to adopt guideline recommendations. To estimate guideline impact, we used segmented linear regression to compare the proportion of children receiving penicillins in March 2015 with the expected proportion at this same time point had the guideline not been published based on a projection of a preguideline trend. A similar approach was used to estimate the short-term (6-month) impact of local implementation efforts. The correlations between organizational readiness and the impact of the guideline were estimated by using Pearson’s correlation coefficient.
RESULTS: Before guideline publication, penicillin prescribing was rare (<10%). After publication, an absolute increase in penicillin use was observed (27.6% [95% confidence interval: 23.7%–31.5%]) by March 2015. Among hospitals with local implementation efforts (n = 20, 71%), the median increase was 29.5% (interquartile range: 19.6%–39.1%) compared with 20.1% (interquartile rage: 9.5%–44.5%) among hospitals without such activities (P = .51). The independent, short-term impact of local implementation efforts was similar in magnitude to that of the national guideline. Organizational readiness was not correlated with prescribing changes.
CONCLUSIONS: The publication of the Pediatric Infectious Diseases Society/Infectious Diseases Society of America guideline was associated with sustained increases in the use of penicillins for children hospitalized with pneumonia. Local implementation efforts may have enhanced guideline adoption and appeared more relevant than hospitals’ organizational readiness to change.
- CI —
- confidence interval
- CPG —
- clinical practice guideline
- IDSA —
- Infectious Diseases Society of America
- IQR —
- interquartile range
- ORCA —
- Organizational Readiness to Change Assessment
- PHIS —
- Pediatric Health Information System
- PIDS —
- Pediatric Infectious Diseases Society
- PRIS —
- Pediatric Research in Inpatient Settings
What’s Known on This Subject:
The 2011 consensus national guideline for childhood pneumonia recommends narrow-spectrum penicillins for most children hospitalized with pneumonia. The impacts of this recommendation and subsequent hospital-level implementation efforts on antibiotic prescribing for children hospitalized with pneumonia are unknown.
What This Study Adds:
Publication of the national guideline was associated with sustained increases in the use of penicillins for children hospitalized with pneumonia. Local implementation efforts may have enhanced guideline adoption and appeared more relevant than hospitals’ organizational readiness to change.
Pneumonia is one of the most common reasons for pediatric hospitalization1,2 and accounts for more antibiotic use than any other condition in US children’s hospitals.3 Reducing inappropriate antibiotic use for this condition could have a substantial impact on slowing the progression of antibiotic resistance. In 2011, the Pediatric Infectious Diseases Society (PIDS) and Infectious Diseases Society of America (IDSA) published a consensus guideline for the management of community-acquired pneumonia in children.4 Recommendations for antibiotic use emphasized the selection of narrow-spectrum penicillins, such as ampicillin or amoxicillin, as first-line treatment of the majority of children in both inpatient and outpatient settings. Before the release of the national guideline, use of these antibiotics accounted for <25% of prescribing for children with pneumonia in ambulatory care settings and <10% in hospitalized children.5–7 Previous studies among children hospitalized with pneumonia have reported similar outcomes, including duration of hospitalization, intensive care admission, readmission rates, and costs, for those treated with narrow-spectrum penicillins compared with broader-spectrum antibiotics.8,9
Studies assessing the early impact of the guideline on antibiotic selection for children hospitalized with pneumonia noted substantial variation in prescribing across hospitals.10,11 Overall, guideline-concordant prescribing increased modestly and use of broader-spectrum antibiotics declined. Whether these initial trends have been sustained remains unclear. Furthermore, although several studies have demonstrated the importance of local, institutional-level efforts to increase guideline-concordant prescribing,12–14 the critical determinants of rapid uptake and adherence to evidence-based recommendations at the hospital level, including elements of “organizational readiness,”15 remain largely unexplored.
In this study, we assessed whether the publication of the 2011 PIDS/IDSA national guideline was associated with a sustained change in initial antibiotic selection across a national sample of children’s hospitals. We also examined the impact of hospital-level interventions targeting antibiotic prescribing for pneumonia implemented after the release of the national guideline.
Data Sources and Patient Population
We used data from the Pediatric Health Information System (PHIS) database (Children’s Hospital Association, Overland Park, KS). The PHIS administrative database contains clinical and billing data from 48 tertiary care children’s hospitals in the United States. Data quality is ensured through a joint effort between the Children’s Hospital Association and participating hospitals.16 Data from PHIS were supplemented with surveys distributed to a representative at each hospital (detailed below) to obtain details regarding hospital-level activities related to antibiotic prescribing for pneumonia and perceptions of organizational readiness to implement evidence-based practice changes.
Hospitals that did not continuously submit PHIS data throughout the study period (n = 12) or that declined to participate in the study surveys (n = 8) were excluded. Among participating hospitals, children 6 months to 18 years of age were eligible for inclusion if they were: (1) hospitalized between August 1, 2009 and March 31, 2015 with an International Classification of Diseases, Ninth revision, Clinical Modification–coded diagnosis of pneumonia using a validated algorithm,17 and (2) received oral or parenteral antibiotics within the first 2 calendar days of hospitalization. We excluded children who received a pleural drainage procedure and those admitted to an ICU during the first 2 calendar days of hospitalization because narrow-spectrum antibiotic therapy may be inappropriate for these children. Similarly, we excluded children at risk for health care–associated infections, including children with ≥1 complex chronic conditions,18 interhospital transfers, and hospitalization at the same PHIS hospital within 30 days before the admission date. To ensure complete ascertainment of initial antibiotic use, we also required children to have at least 2 calendar days of hospitalization.
The primary exposure was time after the release of the PIDS/IDSA national guideline, expressed in monthly time units. The electronic publication of the guideline occurred on August 30, 2011 followed by print publication on October 1, 2011. The quarter in which the guideline was published (fall 2011) was excluded as a transition period.
The primary outcome was the percentage of children treated with penicillin, ampicillin, or amoxicillin (hereafter referred to as penicillins) with or without a macrolide in each month of the study period aggregated across all hospitals. For each month, this percentage was calculated by dividing the number of children receiving a penicillin during the first 2 days of hospitalization by the total number of included children. As a secondary outcome, we also evaluated the percentage of children treated with a second- or third-generation cephalosporin (hereafter referred to as cephalosporin).
Hospital-Level Interventions and Organizational Readiness to Change
Each participating hospital was surveyed to assess local activities targeting prescribing for pneumonia and organizational readiness to change. Hospitals were recruited through the Pediatric Research in Inpatient Settings (PRIS) Network, an inpatient-based, open research network comprising >50 pediatric centers across the United States and Canada. PRIS Network site leads from each hospital designated a site investigator who attended a webinar detailing the study’s objectives and survey content. Two surveys were applied. The first survey queried the site investigator regarding the presence and content of local clinical practice guidelines (CPG) and order sets related to antibiotic prescribing for pneumonia, as well as antimicrobial stewardship activities both before and after national guideline publication. The timing of implementation and changes in the antimicrobial stewardship activities were also assessed. The second survey used a validated instrument, the Organizational Readiness to Change Assessment (ORCA),15 to assess each hospital’s readiness to adopt the PIDS/IDSA guideline recommendations for antibiotic use in pneumonia. The ORCA instrument assesses 3 scales: strength and extent of evidence (3 subscales), quality of organizational context (6 subscales), and capacity for internal facilitation (9 subscales). The ORCA survey was distributed to 5 faculty providers at each hospital (identified by site investigators), including 2 faculty members representing hospital medicine and 1 each representing emergency medicine, infectious diseases, and antimicrobial stewardship (or a second infectious diseases faculty) because individuals from these specialties routinely care for children with pneumonia. Both surveys were administered and managed by using REDCap.19
Hospital-level characteristics were summarized by using frequencies and percentages for categorical variables and medians and interquartile ranges (IQR) for continuous variables. The impact of the national guideline on antibiotic choice was assessed by using interrupted time-series analyses, the reference standard for evaluating the impact of policies or programs.20,21 Segmented linear regression models were used to determine trends in the monthly percentages of children receiving penicillins in the periods pre- and postguideline. Autoregressive integrated moving average models were used to account for first-order autocorrelation in the error terms of consecutive observations. A term for calendar month was also included due to known seasonal variation in pneumonia incidence.22 To quantify the cumulative impact of the guideline on antibiotic choice, we compared the observed proportion of children receiving penicillins (aggregated across all hospitals) in the last month of the study period with the expected proportion at this same time point had the guideline remained unpublished (estimated projection of trend from the preguideline period). An age-stratified analysis (<5 years and ≥5 years) was also conducted for this outcome. To identify variation across hospitals, individual time-series models were also created for each hospital by using quarterly estimates as the unit of analysis. All of the above analyses were repeated for the secondary outcome of cephalosporin use.
We next characterized each hospital according to hospital-level antibiotic recommendations contained in CPGs and order sets for pneumonia implemented after the release of the national guideline. Because the timing of hospital-level antibiotic recommendations varied across hospitals, it was not possible to estimate the cumulative impact of these activities at the end of the study period as was done for the main analysis. Instead, we assessed the short-term impact of each intervention (eg, national guideline, local CPG, local order set) by comparing the observed proportion of children receiving penicillins at 6 months after each intervention with the expected proportion estimated from the preintervention trend at the same time point. Hospitals that did not implement or modify CPGs or order sets in accordance with the national guideline and interventions with <6 months of follow-up time were excluded from this analysis.
The 3 scales of the ORCA instrument were scored as follows: for each respondent, item scores were averaged for each subscale, then the subscale scores were averaged to generate the evidence, context, and facilitation scores (range: 1–5). Hospital-level ORCA scores for each scale were generated by averaging the individual respondent scores within each hospital. We assessed for correlation between hospital-level ORCA scores and overall impact of the national guideline estimated from the primary analysis using Pearson’s correlation coefficient. All analyses were performed by using SAS version 9.4 (SAS Institute, Inc, Cary, NC), with P < .05 considered statistically significant. This study was approved by the Institutional Review Board at Vanderbilt University and the institutional review boards at each of the participating hospitals.
The study population included 58 559 patient encounters from 28 hospitals (Fig 1). The median age was 3 years (IQR: 1–6 years); 52.5% of children were boys; 21.1% were non-Hispanic white, 43.3% were non-Hispanic black, and 23.6% were Hispanic. The median hospital length of stay was 3 days (IQR: 2–4 days). The baseline characteristics of the included children in the periods before and after release of the national guideline are presented in Table 1.
Cumulative Impact of the PIDS/IDSA Guideline on Antibiotic Choice
During the period before the release of the national guideline, cephalosporin use was common, accounting for ∼60% of prescribing for pneumonia across all hospitals. During this same period, an increasing trend was noted for penicillin prescribing (P = .047), although use remained low (<10% of encounters). After the release of the guideline, penicillin use additionally increased, whereas cephalosporin use declined. By the end of the study (March 2015), and compared with the expected antibiotic use based on preguideline trends, this corresponded to an absolute increase of 27.6% (95% confidence interval [CI]: 23.7–31.5) in penicillin use and an absolute decline of 27.8% (95% CI: 23.0–32.6) in cephalosporin use (Figs 2A and 2B). Modest differences were noted in the magnitude of prescribing changes according to age (<5 years vs ≥5 years, Supplemental Table 2), whereas substantial variability was noted across hospitals (Figs 3A and 3B).
Hospital-Level Interventions and Organizational Readiness To Change Assessment
In the preguideline period, 10 of the 28 included hospitals (36%) reported use of a local CPG for pneumonia, 13 (46%) reported use of a pneumonia order set, and 14 (50%) reported the presence of an antibiotic stewardship program; 3 hospitals reported the presence of a local CPG, order set, and antibiotic stewardship program. Only 2 hospitals with a CPG or order set recommended penicillin-based therapy as a first-line treatment of pneumonia. After the release of the national guideline, 19 hospitals (68%) implemented a new or revised CPG, 20 (71%) implemented a new or revised order set (17 [61%] reported the presence of both), and all reported the presence of a stewardship program. All but 1 CPG and 2 order sets recommended penicillin-based therapy. Most respondents perceived that local activities at their institution were somewhat important or very important for influencing antibiotic prescribing for pneumonia (CPG, 81%; order set, 85%; stewardship program, 54%).
By the end of the study period (March 2015), the median absolute increase for penicillin use was +29.5% (IQR: +19.6 to +39.1) and the respective decrease for cephalosporin use was –25.8% (IQR: –41.5 to –15.7) among hospitals that implemented a new or revised CPG or order set recommending penicillin-based therapy, and was +20.1% (IQR: +9.5 to +44.5) for penicillin use and –20.3% (IQR –46.0 to +9.7) for cephalosporin use among hospitals without such recommendations, although these differences between hospitals with or without local implementation activities were not statistically significant.
Of the 140 ORCA surveys distributed, 131 (94%) were completed. The mean ORCA scores across hospitals were highest for the evidence scale (3.99, range: 3.39–4.58), followed by the context scale (3.85, range: 3.06–4.68), and were lowest for the facilitation scale (3.63, range: 2.70–4.46) (Fig 4). Cumulative changes in penicillin and cephalosporin use at the hospital level were not correlated with mean evidence score (ρ = 0.03, P = .89), context score (ρ = 0.31, P = .11), or facilitation score (ρ = 0.09, P = .64).
Short-term Impact of Hospital-Level Interventions
Among the hospitals that implemented a new or revised CPG or order set, we measured the independent, short-term impact of these hospital-level interventions. At 6 months after the first hospital-level intervention (variable timing for each hospital), penicillin use additionally increased (median absolute difference: +6.5% [IQR: –3.5 to +13.8]) and cephalosporin use declined (–5.7% [IQR: –15.5 to +0.7]) compared with the trend estimated after the release of the national guideline. For comparison, the short-term impact of the national guideline was also assessed at each hospital. At 6 months after the release of the national guideline (February 2012), penicillin use increased (median absolute difference: +4.3% [IQR: +1.2 to +11.8]) and cephalosporin use declined (–5.9% [IQR: –15.2 to –0.3]) compared with estimated use from the preguideline period.
Within a national sample of 28 US children’s hospitals that included >58 000 hospitalizations for pneumonia, we identified significant changes in antibiotic prescribing after the release of the 2011 PIDS/IDSA national pneumonia guideline. In accordance with guideline recommendations,4 empirical use of penicillins continued to increase, whereas cephalosporin use steadily declined. The magnitude of these changes, however, varied across individual hospitals. Most hospitals implemented a new or revised CPG or order set targeting antibiotic prescribing for pneumonia after the release of the national guideline. Our findings suggest that these local activities contributed to the observed changes in antibiotic prescribing. In contrast, a measurement of hospitals’ organizational readiness to change was not correlated with changes in prescribing at the local level.
In a study conducted among 18 hospitals after the release of the national guideline, cephalosporin use was not significantly impacted, whereas penicillin use increased modestly 18 months after guideline publication.11 In our study, which included nearly 2 additional years of follow-up, penicillin use progressively increased and exceeded cephalosporin use by the end of the study period. In both studies, striking variation was noted across hospitals. Although our findings suggest that guideline publication directly influenced management decisions, the national guideline likely also had indirect effects by facilitating the implementation of local activities targeting specific recommendations. Over two-thirds of the hospitals included in our study reported the use of a CPG or order set recommending penicillin-based therapy after the release of the national guideline compared with <10% of hospitals before the release of the guideline. It is also likely that other unmeasured factors (eg, increased knowledge and attention to guideline recommendations and other less tangible activities) contributed. Thus, the prescribing changes observed in our study likely represent a mixture of both direct and indirect effects of the national guideline.
Changes in antibiotic prescribing occurred quickly among hospitals that implemented a local CPG or order set targeting guideline-concordant antibiotic use. Most hospitals observed changes in prescribing patterns within 6 months of implementation of local strategies, highlighting the importance of such interventions. In addition, most survey respondents perceived that their hospital’s activities were important determinants of prescribing locally. These findings are consistent with previous studies documenting the effectiveness of local efforts targeting antibiotic prescribing.10,12,13,23 Nonetheless, we did not observe substantial cumulative differences in prescribing trends between hospitals that implemented local activities and those that did not by the end of the study period. Several reasons may explain this apparent discrepancy. Differences in the design, content, or provision of local activities (eg, paper or Web-based guideline versus recommendations integrated into provider order entry) may have influenced provider awareness or reliable use.23 In addition, delayed implementation of local activities (eg, ≥3 years after the release of the national guideline) at some hospitals could have diluted the estimated cumulative impact of local interventions. Conversely, increased national focus on antibiotic overuse and resistance, evidenced by the spread of antimicrobial stewardship programs to all of the participating hospitals in the postguideline period, could have increased guideline-concordant prescribing even in the absence of more formal institutional activities.
The variation in the adoption of guideline recommendations was not associated with organizational readiness to change as measured by using a validated tool.15 Although the scores for the 3 ORCA scales were comparable, the 2 highest scores were noted in the evidence and context scales, signifying general support of the guideline recommendations and positive perceptions of a change-oriented culture across hospitals. The facilitation scale, which measures the capacity to implement proposed practice changes, had the lowest scores with more variation across hospitals. Although not correlated with changes at individual hospitals, this latter observation may help explain why prescribing changes did not differ dramatically 3 years after the publication of the national guideline. Others have demonstrated the potential to induce rapid and reliable change using quality improvement methodology.12,24,25 Review of those successful interventions demonstrate the importance of internal facilitation, including effective implementation teams with adequate resources, progress tracking, and evaluation, as well as a focus on both education/awareness and interventions with high reliability. Without widespread dissemination of similar initiatives, the promise of rapid evidence adoption on a national scale will likely remain elusive.
Several limitations of our study merit discussion. The ecological design allowed us to identify prescribing trends in relation to the release of the national guideline and local activities targeting guideline adoption, but it is possible that other unmeasured factors also contributed to the observed changes. The variable timing in which hospitals implemented local interventions precluded a more robust assessment of their cumulative impact. Similarly, it was not possible to fully assess the independent, short-term impact of local CPGs versus order sets because most hospitals implemented both within a short period of time. In addition, details regarding the content and design of local activities were not collected. Although the ORCA surveys were completed by up to 5 faculty members at each hospital from specialties that routinely provide care for children with pneumonia, a larger systematic sample of respondents from each hospital may have yielded different results. Finally, our study was conducted among 28 freestanding children’s hospitals, and results in regard to the impact of the national guideline and hospital-level interventions may not be generalizable to other populations.
The publication of the 2011 PIDS/IDSA national pneumonia guideline was associated with significant and sustained increases in guideline-concordant antibiotic prescribing across a large sample of US children’s hospitals. Despite strong endorsement of the guideline recommendations across hospitals, wide variability in antibiotic choice was observed. The presence of CPGs and order sets may enhance evidence adoption and seem more relevant than hospitals’ organizational readiness to change. Future prospective assessments will be important to confirm our study’s findings and to more precisely identify critical elements for producing reliable changes in prescribing in local environments.
The PRIS Network collaborators include: Rey Gomez, MS; Susan Wu, MD; Nivedita S. Srinivas, MD; Julia G. Arana, MD; Teresa A. McCann, MD; Jen Weyandt, APRN, CNP, MS; Chris Miller, MD; Sandra R. Arnold, MD; Meghan E. Hofto, MD; Andrea Green Hines, MD; Judith M. Martin, MD; Jim O'Callaghan, MD; Samantha M. Gunkelman, MD; Suchitra Rao, MD; Daxa P. Clarke, MD; Matthew B. Johnson, MD; Jared Beavers, MD; Rebecca Wallihan, MD; Sowdhamini S. Wallace, MD; Luis A. Castagnini, MD, MPH; Susan S. Woo, MD; Joanne C. Mendoza, MD; Heather C. Pierce, MD; Gian Musarra, MD; Kelly B. Flett, MD, MMSc; Sonal Kalburgi, DO, MSHS; Elaine G. Cox, MD; Clifford Chen, MD; T. Shea Osburn, MD; and Ean Miller, PharmD.
- Accepted January 3, 2017.
- Address correspondence to Derek J. Williams, MD, MPH, Vanderbilt University Medical Center, CCC5324 Medical Center North, 1161 21st Ave S. Nashville, TN 37232. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This work was supported by Clinical and Translational Science Award UL1TR000445 from the National Center for Advancing Translational Sciences. This work was also supported by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health under award K23AI104779 (to Dr Williams) and the Agency for Healthcare Research and Quality under award R03HS022342 (to Dr Grijalva). The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the National Center for Advancing Translational Sciences, the National Institute of Allergy and Infectious Diseases, or the Agency for Healthcare Research and Quality. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- ↵Agency for Healthcare Research and Quality. National estimates on use of hospitals by children from the HCUP Kids' Inpatient Database (KID). Available at: http://hcupnet.ahrq.gov/. Accessed Jan 12, 2014
- Bradley JS,
- Byington CL,
- Shah SS, et al; Pediatric Infectious Diseases Society and the Infectious Diseases Society of America
- Kronman MP,
- Hersh AL,
- Feng R,
- Huang YS,
- Lee GE,
- Shah SS
- Williams DJ,
- Edwards KM,
- Self WH, et al
- Ross RK,
- Hersh AL,
- Kronman MP, et al
- Newman RE,
- Hedican EB,
- Herigon JC,
- Williams DD,
- Williams AR,
- Newland JG
- Johnson DP,
- Lind C,
- Parker SE, et al
- Copyright © 2017 by the American Academy of Pediatrics