OBJECTIVES: To determine if an improvement collaborative of 33 children’s hospitals focused on reliable best practice implementation and culture of safety improvements can reduce hospital-acquired conditions (HACs) and serious safety events (SSEs).
METHODS: A 3-year prospective cohort study design with a 12-month historical control population was completed by the Children’s Hospitals’ Solutions for Patient Safety collaborative. Identification and dissemination of best practices related to 9 HACs and SSE reduction focused on key process and culture of safety improvements. Individual hospital improvement teams leveraged the resources of a large, structured children’s hospital collaborative using electronic, virtual, and in-person interactions.
RESULTS: Thirty-three children’s hospitals from across the United States volunteered to be part of the Children’s Hospitals’ Solutions for Patient Safety collaborative. Thirty-two met all the data submission eligibility requirements for the HAC improvement objective of this study, and 21 participated in the high-reliability culture work aimed at reducing SSEs. Significant harm reduction occurred in 8 of 9 common HACs (range 9%–71%; P < .005 for all). The mean monthly SSE rate decreased 32% (from 0.77 to 0.52; P < .001). The 12-month rolling average SSE rate decreased 50% (from 0.82 to 0.41; P < .001).
CONCLUSIONS: Participation in a structured collaborative dedicated to implementing HAC-related best-practice prevention bundles and culture of safety interventions designed to increase the use of high-reliability organization practices resulted in significant HAC and SSE reductions. Structured collaboration and rapid sharing of evidence-based practices and tools are effective approaches to decreasing hospital-acquired harm.
- CAUTI —
- catheter-associated urinary tract infection
- HAC —
- hospital-acquired condition
- HRO —
- high-reliability organization
- QI —
- quality improvement
- SPS —
- Solutions for Patient Safety
- SSE —
- serious safety event
- SSER —
- serious safety event rate
- VTE —
- venous thromboembolism
Hospitalized patients are harmed at concerning rates. In 1999, the National Academy of Medicine (then called the Institute of Medicine) reported that medical errors cause up to 98 000 patient deaths annually,1 with more recent studies suggesting that number may be closer to 400 000.2 Nonfatal harm is estimated to occur in up to one-third of hospitalized patients.3,4 Hospitalized children are equally vulnerable; adverse events in children’s hospitals occur at a rate of 40 per 100 discharges, with almost 50% of these harms deemed preventable.5
Isolated examples demonstrate that harm reduction is achievable. Several pediatric quality improvement (QI) collaboratives have demonstrated significant harm reduction by implementing high-reliability organization (HRO) principles6 in combination with QI methodologies.7–9 These principles come from an analysis of characteristics of industries that exist in extremely high-risk and complex environments but have much lower than expected rates of harm. Despite skepticism about study design and methodology used in QI collaboratives to assess effectiveness,10,11 the collaborative approach has frequently yielded valid harm reduction in hospitalized patients.6,12–14
One collaborative that demonstrated sustained harm reduction was the Ohio Children’s Hospitals’ Solutions for Patient Safety (SPS) network. Guiding principles for this collaborative of 8 children’s hospitals included application of HRO principles, data transparency, and sharing best practices to maximize the synergy between improving processes and institutional safety culture. This collaborative reduced surgical site infections by 60%12 and by 2011 had reduced serious safety events (SSEs) and serious harm events by 55% and 40%, respectively.6 With support from the Centers for Medicare and Medicaid Services’ Partnership for Patients initiative, the Ohio network added 25 children’s hospitals from around the United States in 2012 to form the Children’s Hospitals’ SPS hospital engagement network. This study describes the collaborative interventions and impact on hospital-acquired conditions (HACs) and SSE rates (SSERs).
To evaluate the effect of networkwide implementation of HAC- and SSE-reduction practices, we conducted a prospective cohort study of 33 SPS children’s hospitals (Supplemental Table 4) that ranged in size, clinical scope, and geographical region. Each had independent experience with patient safety improvement work and HAC-reduction efforts. This project was reviewed, and consent was waived by the Cincinnati Children’s Hospital Institutional Review Board. Each hospital paid $15 000 as an annual participation fee, which helped offset a ∼$2 500 000 annual SPS network budget that covered in-person and virtual learning, data management and analysis, QI expertise, and staff.
The SPS leadership oversaw identification and dissemination of best practices related to HAC and SSE reduction. Each participating hospital was asked to incorporate these best practices by using local QI methods, such as the Model for Improvement15 or Lean Six Sigma,16 and leverage the structure and resources provided by the collaborative to accelerate and sustain improvement. At the collaborative level, improvement strategies were designed by using the Model for Improvement, including aims, metrics, and key drivers.
The 9 HACs targeted for improvement are listed in Table 1. A leadership team for each HAC was responsible for adopting a specific aim and identifying evidence-based best practices, which were bundled together to produce a cadre of practices (a “bundle”) to disseminate to participating hospitals.17 Hospitals were encouraged to establish local improvement teams for each HAC and instructed to target 90% compliance to the bundles. As such, hospitals identified team leaders for each HAC, assembled multidisciplinary teams, developed HAC-reduction goals that aligned with the collaborative’s goals, and then planned, implemented, and measured tests of change designed to reduce hospital-acquired harm. These tests primarily focused on the institution and reliable use of the evidence-based bundles identified at the network level.
Culture of Safety Improvement
Culture of safety best practices recommended by SPS leadership were similar to those successfully used by the Ohio collaborative to reduce Ohio’s statewide SSERs.6 These collective tools and techniques, known as the “culture work,” aimed to change attitudes and behaviors of hospital personnel within the following 4 domains: error-prevention behaviors, leadership methods, cause analysis, and patient–family engagement. Several of these practices were modeled after commercially available tools,24 and others were developed by SPS leaders.
Error-prevention behaviors were taught to all hospital personnel in classes facilitated by hospital educators trained by SPS. In addition, SPS hospitals developed a peer-to-peer safety coach program and a hospital leader training program. Leaders were taught to engage in structured unit-based safety rounds, implement routine safety huddles, and establish daily organization-wide safety briefings. SPS experts trained a team from each hospital in a standardized root cause analysis process. The collaborative supported recruitment and inclusion of parents on hospital board committees, hospital-level quality and safety committees, and local HAC teams. The Child Health Patient Safety Organization afforded participating hospitals the ability to transparently discuss serious harm event details, subsequent analysis, and follow-up actions with colleagues at other hospitals.
Adhering to the philosophical tenet of “all teach, all learn,” multiple opportunities for collaboration across the network were offered. Opportunities included semiannual, in-person learning sessions; monthly webinars on topics including safety culture, QI science, and HRO theory; discussions on a password-protected Web site; outcome and process data reports for chief executive officers and quality leaders; and published reports that highlighted which hospitals were achieving the best results in each category. The SPS support staff consisted of quality consultants, project managers, and data specialists who provided guidance and analytics support to the network. Informational webinars for each HAC occurred monthly, and webinars focusing on the implementation of HRO principles occurred weekly.
Main Outcome Measures
The primary outcome measures were HACs and SSERs over time. HACs were identified by the Centers for Medicaid and Medicare Services and defined when possible by using established standards from national organizations (Table 1). When standard definitions were not available, operational definitions were established by expert pediatric consensus. All operational definitions were consistent throughout the study period. Each hospital submitted monthly HAC outcome data for the following 3 time periods: a 12-month “pre-SPS” period (April 1, 2011–March 31, 2012); a 3-month “ramp-up” period (April 1, 2012–June 30, 2012); and a 33-month “post-SPS” period (July 1, 2012–March 31, 2015). The pre-SPS period included retrospective baseline data for the 12 months before the beginning of SPS. Only hospitals that submitted baseline data were included in the analysis. During the ramp-up period, hospitals organized prospective data collection processes and began bundle implementation.
SSEs were defined as serious patient harm events resulting from standard of care deviation.16 Examples include a surgical procedure performed on the wrong patient or 10-fold opiate overdose resulting in respiratory arrest. Hospitals were trained to detect SSEs by all means, including incident reports, legal activity, patient complaints, and quality-assurance processes at the department and unit levels. SSER reduction has been associated with organizational safety culture improvement.25–29 The monthly SSER was defined as the number of SSEs divided by the number of adjusted patient days in a given month. In addition, a 12-month SSER rolling average was calculated with the following: the number of SSEs in the last 12 months divided by the total number of adjusted patient days. Hospitals were included in the SSE analysis if they submitted SSE data beginning in 2011 and completed culture training by December 31, 2013. In alignment with previous work by researchers using this metric, improvement was measured from the maximum study period rate and thereby accounting for rate changes that occurred as a result of better detection processes.17,24–27
Participating hospitals were expected to submit monthly bundle compliance data for each HAC. Because several SPS-proposed HAC bundles did not have a comprehensive evidence basis in 2011, hospitals were permitted initially to either use SPS bundles or develop their own. However, after June 2014, SPS endorsed specific bundles for central line-associated blood stream infections, surgical site infections, catheter-associated urinary tract infections (CAUTIs), pressure ulcers, and ventilator-associated pneumonia. Hospitals were expected to adopt them and instructed to submit at least 20 bundle compliance audits per month for each HAC. Bundle compliance was defined as adherence to all elements of the bundle. Central line-associated blood stream infections and CAUTIs had 2 compliance metrics that correlated with dual prevention bundles, one for device insertion and one for device maintenance.
Collaborative participation was estimated by using the frequency of attendance on webinars, frequency of presentations on webinars, and attendance at both face-to-face learning sessions and 1 or more in-person board training sessions.
HAC Outcome and Process Analysis
Monthly HAC outcome data were collected from each hospital using a Web-based form and were defined as the number of HAC events per month per 1000 relevant exposure units (eg, patient days and catheter days). Monthly process data reflecting compliance with the specific HAC bundles were defined as the number of observations of perfect bundle execution divided by the total number of observations. Hospitals were additionally categorized based on whether they were reporting pre-SPS compliance data to determine the impact of this level of organizational attention on their initial outcome rates or rate change over time.
Monthly data were analyzed from April 2011 to March 2015. To examine the HAC outcome data and account for temporal correlation within individual hospitals, generalized Poisson mixed effects modeling with a log link was used. Random hospital effects were included to address any temporal correlation of observations within hospitals. The analytic model included indicator variables for SPS periods and allowed the estimation of HAC rates pre- and post-SPS centerlines as well as the estimation of relative rate changes from pre- to post-SPS through exponentiated regression coefficients interpreted as relative rates. Computed P values tested the null hypothesis of no change in HAC outcome rates from pre- to post-SPS. To illustrate monthly change in HAC rates during the 3 study periods, monthly mean profile plots of HACs over the study period and estimated lowess (locally weighted scatterplot smoother) curves for nonparametric data were produced.30
To illustrate hospital-level variation in HAC rate changes from pre- to post-SPS and to identify the percentage of hospitals that improved after SPS initiation, we computed pre- and post-period HAC rates for each hospital and graphically display the change. We did this for each HAC overall and stratified by reporting of bundle compliance data.
The SSER was calculated in the following 2 ways: by using a 12-month retrospective rolling average and by using a monthly rate. To estimate improvement, 12 months of data from before the peak rate (December 2013) were used to establish the baseline, which was then compared with the most recent 12 months (postintervention period) by using a Wilcoxon rank test of the change. For analysis of monthly rates, an overall baseline and postintervention period of SSEs were estimated, and the change between periods through a Wilcoxon rank test was compared. A u-chart based on count data with a Poisson distribution to visually describe the trend for monthly SSEs was created. All statistical tests were 2-sided, and significance was defined as a P < .05. Statistical analyses were performed by using SAS v.9.3 and R v.3.1.1.
Thirty-three hospitals participated in the SPS collaborative. One site was excluded from the analyses because of failure to provide baseline data. Of the remaining 32, 21 (63.6%) were freestanding children’s hospitals, and 29 (90.6%) had academic affiliations. The mean number of hospital beds was 330 (range = 75–628), and the mean number of admissions varied from 13 016 (range = 4083–31 262) in 2011 to 15 713 (range = 1952–35 054) in 2014. The hospitals represented 19 states and the District of Columbia, 15 of which were in the Midwest, 5 in the Northeast, 7 in the South, and 5 in the West.
Significant reductions occurred in 8 of the 9 HACs after initiation of the SPS collaborative (Fig 1, Supplemental Fig 4–7). The relative rate reduction from pre- to post-SPS for the 8 improved HACs ranged from 9% to 71% (Table 2). Depending on the HAC, between 50% and 69% of hospitals had significant HAC reductions. Monthly individual HAC rates decreased after the ramp-up period for all HACs except venous thromboembolism (VTE).
Twenty-one of the 32 eligible hospitals participated in the culture work to reduce SSEs. Of the original 32, 10 hospitals had not completed the required culture training by December 31, 2013, and thus were excluded from the SSE outcome analysis. Additionally, 1 hospital did not submit SSE data in 2014 and was excluded. The 12-month rolling average SSER of the remaining 21 hospitals decreased by 50% from a peak of 0.82 per 10 000 adjusted patient days in March 2012 to 0.41 in March 2015 (P < .001) (Fig 2A). Of the 21 participating hospitals, 15 had an SSER reduction (percent reduction range = 28–100; mean 62), and 6 had an increase (percent increase range = 3–211; mean 78). The monthly number of SSEs per 10 000 adjusted patient days also decreased from a mean of 0.77 to 0.52 (P < .001), a 32% reduction (Fig 2B).
Among hospitals that reported both pre- and post-SPS compliance data, compliance to HAC bundles increased (Table 3; range = 0.4%–26.6%). No hospitals had complete pre-SPS compliance data for every HAC, and no hospitals were initially measuring obstetric adverse events and VTE compliance. Excluding obstetric adverse events, because this HAC is only relevant to 6 of the participating children’s hospitals, the number of hospitals reporting bundle compliance to any HAC increased from a pre-SPS mean of 7.3 (range = 0–17) to a post-SPS mean of 27.8 (range = 19–32).
To examine the relationship between bundle compliance and outcomes, pre- and post-SPS HAC outcome data were divided into categories based on bundle compliance rates (Supplemental Table 5). Hospitals that were not reporting bundle compliance data in the pre-SPS period but did in the post-SPS period had the most significant reductions in HAC rates. For example, improvement in adverse drug event rates for the 20 hospitals without baseline compliance data but with post-SPS compliance data was 41% (P < .001), whereas improvement in adverse drug event rates across all hospitals was 34%. A similar trend was observed for 6 other HACs. For hospitals that did not report bundle compliance in either the pre- or post-periods, no change was observed from pre- to post-SPS HAC outcomes.
Hospital-Level Variation in HAC Change From Pre- to Post-SPS
To examine the level of variation in HAC rates between hospitals before SPS initiation and the variation in improvement after SPS, we computed pre- and post-period HAC rates for each hospital and graphically display the change (Fig 3, Supplemental Fig 8–11). Hospitals varied in their pre-SPS performance across all HACs. The percent of hospitals that improved after SPS initiation varied across HACs from 44% (VTE) to 69% (CAUTIs). Overall, the largest improvements in HAC rates were seen for hospitals that did not have bundle compliance reporting before SPS and began reporting after starting SPS. For example, the overall percentage of hospitals that improved for CAUTIs was 69%, with hospitals that started reporting bundle compliance after SPS initiation reporting 83% improvement. Lastly, the denominator for each HAC was averaged across the study period, and we report the median and interquartile range of the average denominators across hospitals in Fig 3 and Supplemental Fig 8–11. Denominators varied slightly across hospitals but didn’t vary drastically across compliance bundle groups.
Participation in Collaborative Activities
An average of 20 (62.5%) hospitals attended each of the 205 available webinars, and 100% of hospitals served as teachers of 1 or more webinars or learning sessions. From 2012 to 2014, 97% of hospitals had at least 1 representative at each of the 4 national learning sessions. Sixty-six percent of hospitals sent at least 1 board member to an in-person board training session. During the intervention period, >600 discussions occurred on the SPS password-protected online discussion board.
To our knowledge, this is the first report in which researchers describe a significant reduction in patient harm (HACs and SSEs) in a cohort of children’s hospitals participating in a structured collaborative network. Our results suggest that implementing and measuring compliance to best-practice bundles, implementing HRO principles, and robustly participating in a multihospital learning collaborative can significantly reduce harm in hospitalized children on a large scale. These data support the hypothesis that participation in a hospital engagement network can result in significant harm reduction. Comparing baseline to study-end rates for each HAC and by using previously published data on average costs of each individual HAC, we estimate that this work resulted in >5000 children being spared harm with an estimated savings to the health care system of $93 000 000.26,31–35
One important finding in this study is that significant HAC reduction can occur at the hospital and collaborative levels when compliance to best-practice bundles is emphasized, measured, and reported. Collaborative leadership aggressively communicated the importance of reliable adherence to all bundle elements. By using this approach, 8 of 9 targeted HACs in the SPS network showed collaborative-wide improvement. Although the vast majority of hospitals experienced improvement during the study period, the most dramatic HAC reduction came from the subset of hospitals that did not report bundle compliance data at the onset but did during the post-SPS phase of the study. We believe the acts of measuring, reporting, and comparing local best-practice compliance to defined 90% targets contributes substantially to improvement. Notably, this rate reduction occurred despite the fact that increased focus on harm is frequently associated with increased event detection.
The activities of the participating hospitals that are associated with successful harm reduction are as follows: engagement of executive leadership, identification of HAC leadership, development of harm-reduction goals and teams focused on implementing collaborative bundles aimed at maximizing compliance, and incorporating tactics designed to improve the organization’s culture of safety through basic error-prevention training, leadership methods, and cause analysis. Although this intervention highlights the value of the collaborative model in supporting hospital participation, institutions that cannot participate in the collaborative model may benefit from implementing similar strategies on an individual basis.
A second interesting study finding is that SSEs, which are typically generated by complex system failures unique to local hospital environments, can be reduced collectively by implementing tools based on HRO principles. Both the 12-month rolling average and the monthly SSER significantly decreased in this study. We believe that this robust SSER reduction strongly suggests that targeting improvements in organizational culture can be a highly effective and necessary adjunct to focused process improvement on particular HACs.
Our study addresses several of the concerns identified in previous studies of hospital engagement networks.11 First, rather than using a simple “pre-post” study design, we used the more rigorous autoregressive integrated moving average (ARIMA) time series analysis to determine more confidently the impact of our intervention. Second, a well-demarcated baseline period was established a priori. The timing of the intervention was easily identified based on this defined baseline period. Finally, all HAC and SSE metrics in this study were based on standard, previously published definitions and remained consistent throughout the study period.
Our study had several limitations. First, because several hospitals were not measuring processes or outcomes related to some HACs before the SPS intervention, baseline data collection for these hospitals was retrospective. This could bias the results in an unknown direction. Second, although the available SPS data demonstrated the impact of multielement HAC-prevention bundles and culture of safety interventions, more detailed data would be necessary to appreciate the effect of individual bundle elements. Third, we did not require hospitals to formally communicate their compliance to the culture work, resulting in an inability to determine the extent of implementation of best practices across the collaborative. This limitation would likely bias the results toward the null, so the impact of these practices would most likely be underestimated in our study. Finally, because this was a cohort study with historical controls, we were unable to determine the effect of secular trends on our outcomes. However, it is unlikely that concurrent interventions occurred at the exact time we launched our interventions at both the hospital or collaborative levels. Additionally, the statistical approaches we chose minimize the risk of secular trends impacting the outcomes.
Participation in a structured collaborative dedicated to implementing HAC-related best practice prevention bundles and culture of safety interventions designed to increase the use of HRO practices resulted in significant HAC and SSE reductions. The potential implications of these findings on national patient harm reduction are substantial. Future research should focus on determining which bundle and cultural elements have the most impact on outcomes, finding novel and effective ways to drive toward zero harm, and formally evaluating the cost effectiveness of implementing these interventions.
- Accepted May 31, 2017.
- Address correspondence to Anne Lyren, MD, MSc, UH Rainbow Babies & Children’s Hospital, Departments of Pediatrics and Bioethics, Case Western Reserve University School of Medicine, 11100 Euclid Ave, Mail Stop 6001, Cleveland, OH 44106. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: The Centers for Medicare and Medicaid Services Partnership for Patients program provided financial support for the operation of the collaborative; the Cardinal Health Foundation provided financial support for the in-person boards training program that was part of the collaborative intervention; the Children’s Hospital Association provided financial support for the operation of the collaborative and participating children’s hospitals.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Institute of Medicine
- Classen DC,
- Resar R,
- Griffin F, et al
- Stockwell DC,
- Bisarya H,
- Classen DC, et al
- Weick K,
- Sutcliffe K
- Miller MR,
- Niedner MF,
- Huskins WC, et al; National Association of Children’s Hospitals and Related Institutions Pediatric Intensive Care Unit Central Line-Associated Bloodstream Infection Quality Transformation Teams
- Billett AL,
- Colletti RB,
- Mandel KE, et al
- Toltzis P,
- O’Riordan M,
- Cunningham DJ, et al
- Langley GJ,
- Moen RD,
- Nolan KM,
- Nolan TW,
- Norman CL,
- Provost LP
- Ponde PS,
- Neuman RP,
- Cavanagh RR
- Children’s Hospitals Solutions for Patient Safety
- National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP)
- Centers for Disease Control and Prevention
- Centers for Disease Control and Prevention
- Centers for Disease Control and Prevention
- National Database of Nursing Quality Indicators (NDNQI)
- Centers for Disease Control and Prevention
- Healthcare Performance Improvement LLC
- Cleveland WS
- Goudie A,
- Dynan L,
- Brady PW,
- Fieldston E,
- Brilli RJ,
- Walsh KE
- Copyright © 2017 by the American Academy of Pediatrics