Abstract
Objective. To make measurable improvements in the quality and cost of neonatal intensive care using a multidisciplinary collaborative quality improvement model.
Design. Interventional study. Patient demographic and clinical information for infants with birth weight 501 to 1500 g was collected using the Vermont Oxford Network Database for January 1, 1994 to December 31, 1997.
Setting. Ten self-selected neonatal intensive care units (NICUs) received the intervention. They formed 2 subgroups (6 NICUs working on infection, 4 NICUs working on chronic lung disease). Sixty-six other NICUs served as a contemporaneous comparison group.
Patients. Infants with birth weight 501 to 1500 g born at or admitted within 28 days of birth between 1994 and 1997 to the 6 study NICUs in the infection group (n = 3063) and the 66 comparison NICUs (n = 21 509); infants with birth weight 501 to 1000 g at the 4 study NICUs in the chronic lung disease group (n = 738).
Interventions. NICUs formed multidisciplinary teams that worked together under the direction of a trained facilitator over a 3-year period beginning in January 1995. They received instruction in quality improvement, reviewed performance data, identified common improvement goals, and implemented “potentially better practices” developed through analysis of the processes of care, literature review, and site visits.
Main Outcome Measures. The rates of infection after the third day of life with coagulase-negative staphylococcal or other bacterial pathogens for infants with birth weight 501 to 1500 g, and the rates of oxygen supplementation or death at 36 weeks' adjusted gestational age for infants with birth weight 501 to 1000 g.
Results. Between 1994 and 1996, the rate of infection with coagulase-negative staphylococcus decreased from 22.0% to 16.6% at the 6 project NICUs in the infection group; the rate of supplemental oxygen at 36 weeks' adjusted gestational age decreased from 43.5% to 31.5% at the 4 NICUs in the chronic lung disease group. There was heterogeneity in the effects among the NICUs in both project groups. The changes observed at the project NICUs for these outcomes were significantly larger than those observed at the 66 comparison NICUs over the 4-year period from 1994 to 1997.
Conclusion. We conclude that multidisciplinary collaborative quality improvement has the potential to improve the outcomes of neonatal intensive care.
In an attempt to improve the quality and increase the value of their services, health care organizations are adopting a variety of process improvement tools originally developed for use in other industries.1–6 The resulting improvement efforts have typically been conducted by unit or service-based teams working within single institutions. Recently, improvement models based on interinstitutional collaboration have been introduced.7–9
The Institute for Healthcare Improvement has successfully applied the collaborative improvement model to such diverse topics as adult critical care, asthma care in children and adults, cesarean section rates, delays and waiting times for medical services, adverse drug events, and medication errors.7,10–13 O'Connor and colleagues14 applied collaborative improvement to adult cardiothoracic surgery. After a 3-part intervention that included performance feedback, quality improvement training, and site visits to the other medical centers, hospital mortality rates for coronary artery bypass graft surgery decreased by 24% at the 5 cardiac surgical centers in northern New England.
Despite these examples, many health professionals remain unconvinced about the value of these quality improvement methods. Without additional evidence of clinical benefit and a reduction in cost, many organizations may be unwilling to invest the human and financial resources required to participate in collaborative improvement efforts. A major strength of our study is the inclusion of a prospectively chosen group of comparison hospitals that did not receive the quality improvement intervention.
In this report, we describe the results of a collaborative quality improvement project for very low birth weight infants. This cooperative effort, referred to as the Neonatal Intensive Care Collaborative Quality (NIC/Q) Project, was completed at 10 NICUs in the Vermont Oxford Network and sought to make measurable improvements in the quality and cost of neonatal intensive care using a collaborative quality improvement model. This report describes the project and its impact on the incidence of nosocomial infection and chronic lung disease. A companion report describes the treatment costs at the 10 participating neonatal intensive care units (NICUs) over the course of the project.15 Together these reports provide unique evidence for the potential benefits of collaborative quality improvement.
METHODS
The Vermont Oxford Network
The Vermont Oxford Network is a voluntary group of health professionals committed to improving the quality of medical care for newborn infants and their families. The Network facilitates a coordinated program of research, education, and quality improvement.9,16 To support this program, the Network maintains a Database for infants weighing 401 to 1500 g at birth who were born at or transferred to participating NICUs within 28 days of birth.17,18 The Network Database was used to generate customized reports on practices and outcomes for the 10 NICUs in the project, to identify best performing or benchmark NICUs, and to monitor and evaluate the results of the quality improvement interventions.
Recruitment of Participating NICUs
Written descriptions of the proposed collaborative quality improvement project and applications to participate were mailed to all NICUs in the Vermont Oxford Network. The project was discussed at the Network Annual Meeting and was the subject of an article in the Vermont Oxford Network Newsletter. The overall goals and structure of the project were described. It was explained that the participating institutions would be responsible for funding all travel expenses for their team members and for supporting the internal staff time required by the project. Eleven institutions signed up for the project. One institution withdrew before the first meeting of the collaborative because of difficulty with funding. The 10 project sites and their key personnel are listed in the “Appendix.”
Collaborative Quality Improvement
In preparation for the project, participating institutions were instructed to create multidisciplinary project teams, including a neonatologist, nurse manager, administrator, and quality improvement coach. These teams were to direct local improvement efforts and attend project meetings. The actual composition of the teams varied somewhat among institutions and over time within institutions. Although all teams included a neonatologist and a NICU nurse or nurse manager, the professional disciplines of other team members varied.
At the first meeting of the NIC/Q Project in January 1995, data were reviewed from the Network Database on practices and outcomes at the 10 project sites. Participants at individual NICUs agreed to share their aggregated data openly among collaborative members, chose the primary clinical improvement goals for the project, and began detailed analyses of the processes of care related to the improvement goals. At no time were personally identifiable data on individual infants shared publicly.
Six NICUs elected to focus on reducing the incidence of nosocomial infection; 4 NICUs elected to focus on reducing the incidence of chronic lung disease. Three subgroups were formed. Two subgroups of 3 NICUs each worked on nosocomial infection; the other subgroup of 4 NICUs worked on chronic lung disease. After choosing improvement goals, the subgroups developed “potentially better practice” concepts aimed at achieving the desired improvements. These practices were based on review of the medical literature, detailed analyses of the processes of NICU care, site visits to other participating NICUs, and benchmarking visits to superior performing units in the Vermont Oxford Network.
Each NICU team performed a detailed analysis of their targeted care processes. From this analysis, a list of key questions was developed that served to create a common knowledge base among collaborating institutions. Each NICU shared answers to the key questions with the other NICUs. Each center's self-analysis report became a central part of the next step, which was a series of site visits by the multidisciplinary teams. Both visitor and host prepared carefully for the visits. The teams received instructions and suggestions regarding how to conduct a site visit.19 A detailed schedule for the visits was prepared. A primary goal was to allow visitors to observe the routine operation of the host NICU and to provide an opportunity for individuals to observe and interact with colleagues at the host NICU in their same professional discipline. The visitors completed brief site visit reports immediately after the visit. These reports were circulated and discussed by the subgroups.
The members in the 2 infection subgroups requested that the Network identify NICUs in the Network Database with low nosocomial infection rates that would be willing to serve as benchmark sites and act as hosts for visiting project teams. Two sites were identified and both agreed to participate. These benchmark NICUs were visited using procedures similar to those described above.
Each subgroup prepared a document describing its “potentially better practice” concepts. The documents included a review and an evaluation of the strength and quality of the published evidence. The evidence was graded using the following classification: 1) The practice is supported by well-designed, randomized control trials. If more than one trial has been performed, the results are homogeneous or consistent. 2) The practice is supported by observational data, such as cohort or case–control studies or by randomized trials that are not considered definitive. This may occur because of small sample size, defects in design or the presence of multiple trials with conflicting or heterogeneous results. 3) The practice is supported by causal theory of disease or pathogenesis. 4) The practice is based on experience or intuition. These documents were presented at the second meeting of the collaborative in November 1995. The concepts of the 2 infection subgroups were similar and were merged to create one common list.
There was no attempt to create uniform treatment protocols at all NICUs. Each NICU was encouraged to choose those concepts that were most relevant to their unit and to adapt the concepts to their unique situation. In June of 1997, participating teams documented when specific “potentially better practices” had been implemented at their sites. The timing and intensity of the implementation strategies varied across the NICUs.
Throughout the project an attempt was made to foster a collegial atmosphere. Social events and group outings were included in the schedule for all of the large group meetings. Between meetings contact was maintained primarily through conference calls. These calls each had a designated leader, a formal agenda, and lasted from 1 to 2 hours. There were a total of 4 large group meetings of the collaborative (January 1995, November 1995, June 1996, and June 1997). These meetings included didactic sessions on quality improvement, group exercises focused on specific improvement skills or tasks, presentations by the subgroups and individual hospital teams, and open discussion. There were 14 site visits in 1995, and 2 benchmarking visits to NICUs outside the project with superior performance during 1996. The subgroups participated in 64 conference calls during the project. Implementation of “potentially better practices” and collaboration within and among the NICU teams continued throughout 1997.
Additional improvement goals were chosen at subsequent meetings of the collaborative; there will be the subject of a future report. These goals included reducing chest radiographs, blood gas tests, and length of stay. The participating institutions also submitted detailed patient bills for infants 501 to 1500 g enrolled in the Network Database during the course of the project. The cost of NICU care at each institution was calculated based on the charge information in the detailed bills and the cost-to-charge ratios reported by the institutions in their Medicare Cost Reports.20 The results of the cost analyses are presented in a companion article.15
Definitions of Outcomes and Improvement Goals
The infection subgroups identified its quantitative improvement goal as a reduction in the nosocomial infection rate for infants 501 to 1500 g to the 25th percentile for NICUs in the Network with 70 or more very low birth weight admissions per year. Analysis of the nosocomial infection rates in 1994 demonstrated that 25% of the eligible NICUs had rates of 15% or less.
The chronic lung disease subgroup chose as its outcome measure the combined measure of either death before 36 weeks' postconceptional age or the requirement for supplemental oxygen at 36 weeks' postconceptional age. The date of 36 weeks' postconceptional age is based on the neonatologist's best estimate of gestational age recorded in the Network Database.17
The chronic lung disease group identified its quantitative improvement goal as an absolute reduction in the rate of death or oxygen supplementation at 36 weeks' postconceptional age by 10% for infants 501 to 1000 g with gestational ages of 34 weeks or less.
Nosocomial infection was defined as the occurrence of one or more infections after the third day of life with either coagulase-negative staphylococcus or another bacterial pathogen from a predefined list included in the Database Manual of Operations.17 Blood cultures could be drawn from peripheral veins or through central venous lines. Coagulase-negative staphylococcal infection required the recovery of the organism from blood or spinal fluid, signs of systemic illness, and treatment for 5 or more days with antibiotics. Infections with a bacterial pathogen required recovery of a pathogen other than coagulase-negative staphylococcus from blood or spinal fluid. The specific organisms that are considered bacterial pathogens for the Vermont Oxford Network Database are listed in an appendix to the Manual of Operations and represent organisms unlikely to represent contaminants.17
Statistical Methods
The primary analyses, planned prospectively, involved comparing outcomes in 1996, the first postintervention year, with those occurring in 1994, the baseline year before the initial meeting of the project. These analyses included pre–post comparisons within the project subgroups as well as comparisons of changes at the project subgroups with those at a group of other Network NICUs that did not participate in the project. The criteria for inclusion of a NICU in the comparison group were that the NICU was in North America, participated in the Network Database from 1994 through 1996, enrolled 25 or more cases in the Database in 1994, and was not a participant in the NIC/Q Project. A secondary analysis of trends in outcomes during 1994 to 1997 for infants at project NICUs and those in the comparison NICUs was also performed when data for 1997 became available. Statistical analyses were performed using SAS, Version 6.11 (SAS Institute, Inc, Cary, NC).
Primary Analyses
The rates of nosocomial infection and chronic lung disease in 1996 were compared with the rates in 1994 among the 6 infection subgroup NICUs and the 4 chronic lung disease subgroup NICUs. The comparisons were performed using the Mantel-Haenszel χ2 test, stratified by hospital and 250-g birth weight category. The Breslow-Day test was used to determine whether there was heterogeneity among the NICUs.
The statistical significance of the differences in the magnitude of change observed at the NIC/Q Project sites and the comparison NICUs was determined as follows. A logistic regression model was created for each outcome of interest. The covariates in the model were birth weight, location of birth (inborn or outborn), multiple birth, assisted ventilation, and year of birth (1994 or 1996). Terms for group (NIC/Q Project subgroup hospital or comparison group hospital), and group-by-year interaction were also included in the models. Model fit was assessed using the Hosmer-Lemeshow goodness-of-fit test. The significance of the coefficient of the interaction term was used as a test of whether the change over time was significantly different between the NIC/Q subgroup of interest and the comparison group, after controlling for potentially confounding differences with respect to other covariates.
Data regarding supplemental oxygen administration at 36 weeks' postconceptional age are not routinely collected for the Network Database when infants are discharged before that time point. Because of the importance of this outcome to their improvement goal, the 4 NICUs in the chronic lung disease subgroup identified all infants 501 to 1000 g discharged before 36 weeks' postconceptional age and determined their status with respect to oxygen supplementation.
Because similar data were not available for the comparison NICUs, we developed an algorithm for imputing the 36-week oxygen status for infants discharged before that time. This algorithm was created before collecting or analyzing the data for infants discharged before 36 weeks. It classified infants discharged before 36 weeks based on their gestational age at discharge and whether they were receiving supplemental oxygen at discharge, data that are available routinely in the Database. Infants discharged before 36 weeks who were not in oxygen at the time of discharge were classified as not in oxygen at 36 weeks. Infants discharged between 34 and 36 weeks' postconceptional age were classified based on their requirement for supplemental oxygen at the time of discharge. Infants discharged before 34 weeks who were in oxygen at the time of discharge were classified as unknown. The algorithm was tested against the gold standard of the actual 36-week data collected by the 4 NICUs in the chronic lung disease subgroup.
Secondary Analyses
The trends in outcome over the period 1994 to 1997 for each of the project subgroups were compared with those at the comparison NICUs using logistic regression models similar to those described above. Additional analyses were performed restricted to the project subgroups to determine the odds ratios for the outcomes of interest associated with the variable, year of birth, as a measure of the magnitude of changes over time.
RESULTS
Comparison of Rates in 1994 and 1996 at Project NICUs
Nosocomial Infection
The 17 “potentially better practice” concepts developed by the multidisciplinary teams at the 6 NICUs in the infection subgroups are shown in Table 1. By June of 1997, the total number of these “potentially better practice” concepts that had been either partially or fully implemented at the 6 institutions ranged from 10 to 16. From 0 to 5 of these practices were already in place before the start of the project at some sites.
Potentially Better Practices for Preventing Nosocomial Infection
There were 745 infants born in 1994 and 772 infants born in 1996 treated at the 6 NICUs in the infection subgroup who weighed 501 to 1500 g and who were hospitalized for >3 days. The overall rate of nosocomial infection at the 6 NICUs in the subgroup declined from 26.3% in 1994 to 20.9% in 1996 (P = .007). The rate of coagulase-negative staphylococcal infection declined from 22.0% in 1994 to 16.6% in 1996 (P = .007); the rate of infections with other bacterial pathogens did not change (Fig 1). There was evidence of heterogeneity among the 6 NICUs in the infection subgroup (P = .04). The observed rate of coagulase-negative staphylococcal infections decreased at 4 and increased at 2 of the 6 NICUs in the subgroup between 1994 and 1996.
The rates of coagulase-negative staphylococcal infection, infection with other bacterial pathogens, and any nosocomial bacterial infection in 1994 (black bars) and 1996 (gray bars) for infants 501 to 1500 g hospitalized >3 days at the 6 NICUs in the infection subgroup. The rates in 1994 and 1996 were significantly different for coagulase-negative staphylococcal infection (P = .007) and for combined nosocomial infections (P = .007).
Chronic Lung Disease or Death
The 9 “potentially better practice” concepts developed by the multidisciplinary teams at the 4 NICUs in the chronic lung disease subgroup are shown in Table 2. By June of 1997, the total number of these “potentially better practice” concepts that had been either partially or fully implemented at the 4 institutions ranged from 6 to 9. None of the “potentially better practices” had been in place before the start of the project at 3 of the sites, whereas 3 of the practices had been in place at one of the sites.
Potentially Better Practices for the Prevention of Chronic Lung Disease
The analysis of chronic lung disease or death was restricted to infants with birth weights 501 to 1000 g and gestational ages of 34 weeks or less. There were 188 infants born in 1994 and 187 infants born in 1996 treated at the 4 NICUs in the chronic lung disease subgroup who met these criteria. The overall rate of death or supplemental oxygen at 36 weeks' postconceptional age declined from 55.9% in 1994 to 47.6% in 1996 (P = .039). The rate of supplemental oxygen administration at 36 weeks for infants alive at 36 weeks decreased from 43.5% in 1994 to 31.5% in 1996 (P = .03). The mortality rate did not change significantly (21.8% to 23.5%;P = .98; Fig 2). There was evidence of heterogeneity among the 4 NICUs in the chronic lung disease subgroup (P = .01). The observed rate of supplemental oxygen administration at 36 weeks' postconceptional age decreased at 2 of the 4 NICUs in the subgroup and increased at the other 2 NICUs between 1994 and 1996.
The rates of supplemental oxygen at 36 weeks' adjusted gestational age (oxygen at 36 weeks), death at 36 weeks' adjusted gestational age (death), and either supplemental oxygen or death at 36 weeks' adjusted gestational age (death or oxygen) in 1994 (black bars) and 1996 (gray bars) for infants 501 to 1000 g with gestational ages of 34 weeks or less at the 4 NICUs in the chronic lung disease subgroup. The rates in 1994 and 1996 were significantly different for supplemental oxygen at 36 weeks (P = .04) and for death or supplemental oxygen combined (P = .04).
Comparison of Changes From 1994 to 1996 at Project and Comparison NICUs
There were 66 NICUs that met the criteria for inclusion in the comparison group. The number and characteristics of infants 501 to 1500 g enrolled in the Database by NICUs in the infection subgroup, the chronic lung disease subgroup and the 66 NICU comparison group in 1994 are shown in Table 3. Infants in the comparison group had a slightly higher average birth weight and a slightly lower rate of assisted ventilation; a lower proportion of infants in the infection subgroup were inborn. None of these differences was statistically significant.
Characteristics of Infants 501 to 1500 Grams Born in 1994 and Treated at the NICUs in the NIC/Q Project Subgroups and the Comparison Group
Nosocomial Infection
There were 5108 infants born in 1994 and 5528 infants born in 1996 treated at the 66 comparison NICUs who weighed 501 to 1500 g and who were hospitalized for >3 days. The overall nosocomial infection rate at the 66 NICUs in the comparison group declined from 22.6% in 1994 to 21.1% in 1996 (P = .002). The rate of coagulase-negative staphylococcal infection declined from 15.4% in 1994 to 14.5% in 1996 (P = .025); the rate of infections with other bacterial pathogens decreased from 10.3% to 8.9% (P = .002).
The percentage change in nosocomial infection rates from 1994 to 1996 at the 6 NICUs in the infection subgroup was larger than that at the 66 comparison NICUs (−5.5% vs −1.6%; P = .058). The percentage change in the rates of coagulase-negative staphylococcal infections was significantly greater at the 6 infection subgroup NICUs than at the 66 comparison NICUs (−5.4% vs −.8%; P = .026). The percentage change in infections with other bacterial pathogens (−.9% vs −1.4%; P = .96) was not significantly different between the groups. None of the logistic regression analyses for these outcomes indicated lack of fit.
Death or Supplemental Oxygen at 36 Weeks
The validity of the algorithm developed for classifying cases discharged before 36 weeks' postconceptional was tested. The chronic lung disease subgroup obtained data on actual oxygen need at 36 weeks in those infants discharged before 36 weeks. These results were compared with the imputed results using the algorithm. The classification using the algorithm agreed well with the classification using the actual data (κ = .98). The classification based on the algorithm correctly classified 99% of the cases. Furthermore, the algorithm resulted in a more accurate estimate of the percentage of infants receiving supplemental oxygen at 36 weeks than would have been obtained if infants discharged before 36 weeks were omitted from the analysis. The true percentage of infants receiving supplemental oxygen at 36 weeks at the 4 subgroup NICUs was 21.8%; the estimate obtained by applying the algorithm to cases with missing data was 21.7%; the estimate obtained if cases with missing data were omitted was 32.1%. Based on these results, it was decided to use the algorithm to classify infants discharged before 36 weeks at the 66 comparison NICUs. Before imputation using the algorithm, 21.1% of cases at the comparison sites were missing data for supplemental oxygen at 36 weeks; after imputation, data were missing for 5.7% of cases.
There were 2411 infants who weighed from 501 to 1000 g and had a gestational age of 34 weeks or less born in 1994 and 2653 infants born in 1996 treated at the 66 comparison NICUs. The overall rate of death or supplemental oxygen at 36 weeks' postconceptional age decreased from 54.5% in 1994 to 53.3% in 1996 (P = .09). The rate of supplemental oxygen administration at 36 weeks for infants alive at 36 weeks did not change significantly between 1994 and 1996 (36.4% vs 36.3%; P = .57); the death rate decreased from 27.4% in 1994 to 25.4% in 1996 (P = .017).
The percentage change in death or supplemental oxygen at 36 weeks from 1994 to 1996 was larger at the 4 NICUs in the chronic lung disease subgroup, compared with change at the 66 NICUs in the comparison group (−8.3% vs −1.2%; P = .14). The percentage change in supplemental oxygen at 36 weeks was significantly greater at the 4 subgroup NICUs than at the 66 comparison NICUs (−12.1% vs −.1%;P = .045). The change in death rates was not significantly different between the groups (+1.7% vs −2.1%;P = .44). Lack of fit was not observed for the logistic regression model for the combined measure, death or supplemental oxygen at 36 weeks, or when supplemental oxygen at 36 weeks was analyzed separately. Lack of fit was observed when death at 36 weeks was modeled as the outcome. Fit was improved for this outcome when a quadratic term for birth weight was included in the model (fit, P = .10), and the test for year by death at 36 weeks was consistent with previous results (P = .18).
Analysis of 1997 Data and Trends From 1994 to 1997
Nosocomial Infection
One of the comparison NICUs did not participate in the Vermont Oxford Network Database in 1997, leaving 65 NICUs in the comparison group for that year. There were 789 infants treated at the 6 NICUs in the infection subgroup and 5572 infants treated at the 65 comparison NICUs who were born in 1997, weighed 501 to 1500 g, and who were hospitalized for >3 days. In 1997, the rate of coagulase-negative staphylococcal infection was 12.3% at the 6 NICUs in the project subgroup (vs 22.0% in 1994) and 16.5% at the 65 NICUs in the comparison group (vs 15.4% in 1994). The rate of infection with other bacterial pathogens was 7.2% at the 6 NICUs in the project subgroup (vs 10.3% in 1994) and 10.4% at the 65 NICUs in the comparison group (vs 10.3% in 1994). The overall nosocomial infection rate in 1997 at the 6 NICUs in the project subgroup was 16.7%, compared with 22.8% at the 65 comparison NICUs.
Logistic regression modeling showed a significantly greater change over the time period 1994 to 1997 for the project NICUs relative to the comparison NICUs for the overall rate of nosocomial infection (P = .001), the rate of coagulase-negative staphylococcal infection (P = .001), but not for the rate of infection with other bacterial pathogens (P = .68).
Additional analyses of the changes in infection rates from 1994 to 1997 were performed for the data from the 6 NICUs in the infection subgroup. The rates of nosocomial infection and of coagulase-negative staphylococcal infection decreased significantly at the 6 NICUs; the rate of infection with other bacterial pathogens did not. The odds ratio associated with the variable, year of birth, was .82 (95% confidence interval [CI]: .75–.89) for nosocomial infection, .79 (95% CI: .72–.86) for coagulase-negative staphylococcal infection and .95 (95% CI: .84–1.08) for infection with other bacterial pathogens. Thus, the decline in infection rates from 1994 to 1997 can be explained by the decrease in the rate of infections with coagulase-negative staphylococcal infections. The rates of coagulase-negative staphylococcal infection from 1994 to 1997 at NICUs in the infection group and the comparison group are shown in Fig 3.
Rates of infection with coagulase-negative staphylococcal infection for infants 501 to 1500 g hospitalized >3 days from 1994 to 1997 at the 6 NICUs in the infection subgroup (black bars) and those NICUs in the comparison group (gray bars).
Chronic Lung Disease or Death
There were 185 infants born in 1997 who weighed 501 to 1000 g and were 34 weeks' gestational age or less and treated at the 4 NICUs in the chronic lung disease subgroup and 2719 infants treated at the 65 comparison NICUs. In 1997, the rate of supplemental oxygen at 36 weeks was 34.0% at the 4 NICUs in the project subgroup (vs 43.5% in 1994) and 38.7% at the 65 NICUs in the comparison group (vs 36.3% in 1994). The rate of death before 36 weeks was 22.2% at the 4 NICUs in the project subgroup (vs 21.8% in 1994) and 25.7% at the 65 NICUs in the comparison group (vs 27.4% in 1994). The combined rate of supplemental oxygen or death at 36 weeks in 1997 was 48.9% at the 4 project NICUs and 55.2% at the 65 comparison NICUs. Logistic regression modeling showed that the change at the project NICUs over time was significantly different from that at the comparison NICUs for the rate of supplemental oxygen at 36 weeks (P = .049), but not for the rate of death by 36 weeks (P = .55) or for the combined outcome death or supplemental oxygen (P = .14).
Additional analyses of the changes in rates of oxygen supplementation and death from 1994 to 1997 were performed for the data from the 4 NICUs in the chronic lung disease subgroup. The rate of supplemental oxygen at 36 weeks decreased significantly at the 4 NICUs; the rates of death and the combined outcome, death or supplemental oxygen, did not. The odds ratio associated with the variable, year of birth, was 0.85 (95% CI: .72–.99) for supplemental oxygen at 36 weeks, .97 (95% CI: .81–1.15) for death and .87 (95% CI: .76–1.01) for the combined outcome death or supplemental oxygen. Thus, only supplemental oxygen at 36 weeks declined significantly over the period 1994 to 1997. The rates of oxygen supplementation at 36 weeks from 1994 to 1997 at NICUs in the chronic lung disease group and the comparison group are shown in Fig 4.
Rates of supplemental oxygen at 36 weeks' adjusted gestational age for infants 501 to 1000 g with gestational ages of 34 weeks or less from 1994 to 1997 at the 4 NICUs in the chronic lung disease subgroup (black bars) and those NICUs in the comparison group (gray bars).
DISCUSSION
We have observed measurable decreases in 2 important morbidities, coagulase-negative staphylococcal infection and oxygen supplementation at 36 weeks, for critically ill preterm infants at the group of NICUs participating in the Vermont Oxford Network NIC/Q collaborative quality improvement project. The improvement intervention was based on intensive collaboration within and among multidisciplinary teams of health professionals from different institutions over a period of 3 years. These teams worked together to develop and implement “potentially better practices” aimed at achieving improvement goals that they set as a group. The teams received instruction in quality improvement methods, performed in-depth analyses of the processes of care related to their improvement goals, participated in a series of site visits to other institutions, and reviewed the published evidence. The teams received performance feedback and openly shared information about their practices and outcomes. Each institution selected the “potentially better practices” that it believed were applicable to its unique situation and then implemented them in ways uniquely suited to its own NICU environment. The Vermont Oxford Network Database provided a common measurement tool. It was used to provide feedback about practices and outcomes to the participating sites, to create a contemporaneous comparison group, and to identify benchmark sites in the wider Network. A skilled consultant with experience in quality improvement in health care provided ongoing instruction in quality improvement, facilitated collaborative learning, and provided guidance to the group.
O'Connor and colleagues14 used a 3-component intervention in their regional study of coronary artery bypass graft surgery in Northern New England. They speculated that their intervention, which included data feedback, quality improvement training, and site visits, might have applications in other clinical settings. Our observations suggest that they were correct. We included similar components in our collaborative quality improvement intervention. The Vermont Oxford Network Database was used to provide feedback to participants about their practices and patient outcomes. Quality improvement training was a major component of project meetings. Finally, our intervention included site visits among participating NICUs as well as to other NICUs in the Network with superior performance.
Our work also represents an important extension to that of O'Conner and colleagues.14 Like the work of O'Conner and colleagues, our analysis shows a statistically significant improvement on the selected clinical indicators in a preintervention and postintervention comparison for the group of centers taken as a whole. Going beyond O'Conner and colleagues, we also show a statistically significant improvement in the group of centers versus a contemporaneous comparison group. Advances in quality naturally occur at some rate in the environment because of a variety of factors. The major value of this comparison group is to protect against the danger of misinterpreting a secular trend in outcomes as a result of our intervention. Our analysis indicates that the group of NIC/Q centers improved at a faster rate than did the secular trend.
What inferences can be drawn from our observations? First, it is important to stress that our results cannot be interpreted to demonstrate that any specific “potentially better practices” were effective in improving outcomes. Although the participating teams developed a common list of practices, they selected and applied these practices in unique ways customized to fit their local situations. This project, therefore, evaluated a global approach to collaborative quality improvement, not individual clinical practices. We did not attempt to monitor the specific practice changes implemented at the participating sites. It is important to recognize that we are not recommending any specific clinical practices because this study was not designed to test whether any of the “potentially better practices” were effective. It would be a misinterpretation of our results to think otherwise. We purposefully termed the practices “potentially better” rather then “better” because we did not gather any additional evidence for their effectiveness. If any inference can be drawn, therefore, it is that active participation in structured multidisciplinary, cross-institutional collaborative learning that leads to focused changes in local practice can lead to improvements in clinical outcomes. Such participation may lead to widespread changes in organizational structure and performance that may be as or more important than the specific clinical practices implemented. We are currently developing tools to measure these organizational changes in domains, such as medical and nursing leadership, staff coordination and satisfaction, conflict resolution, and unit culture.
Alternative explanations must, of course, be considered. First, chance is an unlikely explanation for the observations, because statistically significant differences were seen between the preintervention and postintervention periods for key improvement indicators chosen prospectively by the groups. Second, we have no evidence that differences in underlying patient risk either over time or between the project and comparison NICUs could account for our observations. Birth weight, a major risk factor for infection, mortality, and oxygen supplementation, was used to adjust for risk in analyses comparing the outcomes in 1994 with those in 1996. Other known risk factors were included in the multivariate models used to compare changes at the project NICUs with the comparison group. Third, bias in ascertainment of cases, choice of comparison group, or choice of time periods are also unlikely explanations. It is unlikely that cases with infection, death, or supplemental oxygen at 36 weeks' postconceptional age were selectively omitted from reporting to the Database. Although we cannot exclude the possibility that systematic changes in measurement occurred at project sites after their improvement goals were identified, the application of uniform definitions from the Database Manual of Operations provides protection against ascertainment bias. Prospective evaluation of our algorithm for imputing 36-week oxygen status for infants discharged before that time is desirable and would strengthen our result. Additional protection against bias was provided by prospectively planning the primary analyses including the choice of comparison time periods and definition of criteria for inclusion in the comparison group of NICUs. The comparison group includes all of the NICUs in the Vermont Oxford Network that met predefined eligibility criteria. Finally, secular trends in nosocomial infection or chronic lung disease are unlikely to explain our results because the changes observed at the units in the NIC/Q Project were significantly greater than those seen in the comparison group.
Although both subgroups in the NIC/Q Project demonstrated improvement, neither group achieved their quantitative goals. The 21% nosocomial infection rate achieved in 1996 by the infection subgroup was higher than the goal of 15%. By 1997, the rate of nosocomial infection at the 6 NICUs in the infection subgroup had declined to 16.7%. The 8.3% decrease in the combined outcome of death or oxygen supplementation at 36 weeks achieved by the chronic lung disease subgroup in 1996 was also less than the initial goal of a 10% decrease. In 1997, the combined rate of death or oxygen supplementation at 36 weeks had increased to 48.9% from 47.6% in 1996. When we originally planned this study, we did not know how long it would take participants to discover and implement changes. We do not know whether the observed improvements will be maintained or change in magnitude over time. Additional follow-up will be required.
Both subgroups included combined outcome measures in their stated improvement goals. The improvements that occurred were, however, seen only with respect to one of the outcomes for each group; coagulase-negative staphylococcal infection in the case of infection and oxygen supplementation at 36 weeks in the case of chronic lung disease. We do not have data to explain these findings. We do not know how to identify clinical outcomes that may be amenable to the collaborative improvement strategy that we used. Therefore, one should be cautious about generalizing our findings to clinical outcomes other than those that we studied.
The NICUs in this project were a self-selected, highly motivated, interested, and willing group of participants. Because of this, one must also be cautious in generalizing our findings to the wider universe of NICUs or hospitals. The Vermont Oxford Network is currently performing a randomized, controlled trial of evidence-based quality improvement at 114 hospitals to address this concern. Many organizations chosen at random and forced to participate in collaborative improvement would fail. In our study, we observed heterogeneity in the results among the NICU teams; the observed results in some centers improved, whereas others showed degradation.
Heterogeneity of results is not surprising given the multiple factors that contribute to the successful implementation of improvements and the likelihood that commitment and motivation varied among the sites. With such a small number of centers and the relatively small number of cases within each center, our study design lacks the statistical power to justify a center-by-center analysis. We cannot exclude the possibility that our findings could result from a few centers with the largest improvements. We will be better able to explore center-to-center variation in a follow-on study involving 34 NICUs that is already underway.9 A key goal of future research will be to identify organizational factors associated with successful improvement so that individual organizations can be helped to increase the magnitude and pace of improvement. It is our conjecture that active involvement in improvement efforts such as the one we describe here leads to a cultural change in an organization that makes future improvement more likely. Creative research designs will be necessary to address these questions. New tools for assessing the structural characteristics and culture of health care organizations will be needed.
Improvement collaboratives can complement and support research efforts in several other practical ways. A formal, randomized trial on the use of skin care to prevent nosocomial infection grew out of observations and unresolved questions in the infection subgroup. The chronic lung disease subgroup observed that the 2 centers with observed improvement in rates used different styles of mechanical ventilation, but each used its preferred style consistently. The 2 centers with observed degradation in rates displayed more variation in the selection of ventilation method. This leads to the testable hypothesis that establishing consistency in practice might be more important than the choice of one mechanical ventilation method over another. Reduction of variation is a foundational principle in quality improvement and this situation provides an opportunity to test this concept in a more rigorous way. Another researchable question is whether focused improvements in one or a few selected indicators comes at the expense of reductions in quality in other areas. This can be examined in improvement collaboratives in which each organization tracks several indicators in addition to those chosen for focus. We suggest that large improvement collaboratives that are supported by good operational definitions of a suite of indicators, such as our follow-on study involving 34 NICUs, can provide an important substrate for practical, relatively low-cost research.9
Health care organizations are examples of complex adaptive systems. Thus, ideas from complexity theory may have direct application to the management and improvement of health care.21Generalizations from the study of evolution in complex biological systems may allow the creation of formal explanatory models for the evolution of social systems and the emergence of technological innovation.22 In the future, such models may even provide a theoretical framework for understanding collaborative quality improvement.
CONCLUSION
Existing evidence suggests that interinstitutional, multidisciplinary collaborative quality improvement is effective in a variety of clinical settings. The findings of the NIC/Q Project strengthen that evidence and extend it to the field of neonatal intensive care. The findings reported in this article on clinical outcomes and the companion article on costs15 suggest that multidisciplinary collaborative quality improvement has the potential to improve patient outcomes while also reducing costs.
APPENDIX: NIC/Q PROJECT INVESTIGATORS
Jeffrey D. Horbar, MD Jeannette Rogowski, PhD Joseph H. Carpenter, MS Kathy Leahy, RN, NNP Principal Investigator Chief Executive and Scientific Officer Vermont Oxford Network Co-principal Investigator Statistician Project Coordinator Steering Committee Eugene Lewit, PhD Jerold F. Lucey, MD Paul E. Plsek, MS Patricia Shiono, PhD Staff Nancy Allen Stephanie Williamson NICU Teams Children's Hospital Medical Center of Akron, Akron, Ohio: Lela Bartley, RNC, BSN; Anand Kantak, MD; Joann Lindeman, RNC; and Judy Ohlinger, RN. Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire: William Edwards, MD; Karen Shea-Ricci, RN; and Karen Vergura, RNC. Legacy Emanuel Children's Hospital, Portland, Oregon: Jan Genke, RN, MSN; Patrick Lewallen, MD; Pat Seifert, RN; and Karen Waske, RN. Fletcher Allen Health Care, Burlington, Vermont: Marcia Patterson, RN; Roger F. Soll, MD; and Greg Ward, RRT. Miami Valley Hospital, Dayton, Ohio: Beatrice Harris, RN; William Johnson; and Connie McCarroll, DO. Milton S. Hershey Medical Center, Hershey, Pennsylvania: Cindy Banta, RN; Jane Ebersole, NNP; Dennis Mujsce, MD; and Moira Winstanley, NNP. Minneapolis Children's Medical Center, Minneapolis, Minnesota: Jo Crosby, RNC, MS; Kathy Johnson, RN; Carol Miller, RN; Kristin Nelson, RN, MS; and Nathaniel R. Payne, MD.
ACKNOWLEDGMENTS
This work was supported by a grant from the David and Lucile Packard Foundation.
We thank all of the staff members at participating hospitals for their commitment and dedication to improvement and for their hospitality during the site visits. We especially thank the staff at those institutions that served as Benchmark sites and hosted visits by the NIC/Q Project teams: Sutter Memorial Hospital, Sacramento, California; Tacoma General Hospital, Tacoma, Washington; and University of Kentucky Children's Hospital, Lexington, Kentucky.
Footnotes
- Received April 21, 2000.
- Accepted August 16, 2000.
Reprint requests to (A.L.S.) Vermont Oxford Network, 444 S Union St, Burlington, VT 05401.
- NIC/Q =
- Neonatal Intensive Care Collaborative Quality •
- NICU =
- neonatal intensive care unit •
- CI =
- confidence interval
REFERENCES
- Copyright © 2001 American Academy of Pediatrics
In this issue
Jump to section
Related Articles
Cited By...
- Survival Without Major Morbidity Among Very Low Birth Weight Infants in California
- Outcomes and care practices for preterm infants born at less than 33 weeks gestation: a quality-improvement study
- Racial/Ethnic Disparities in Neonatal Intensive Care: A Systematic Review
- Implementing Standardized Feeding Guidelines, Challenges, and Results
- Are quality improvement collaboratives effective? A systematic review
- The History and State of Neonatal Nursing Quality Improvement Practice and Education
- Decreasing infection in neonatal intensive care units through quality improvement
- The power of improvement
- Choosing Wisely in Newborn Medicine: Five Opportunities to Increase Value
- Improvement in Interstage Survival in a National Pediatric Cardiology Learning Network
- NICU Bedside Caregivers Sustain Process Improvement and Decrease Incidence of Bronchopulmonary Dysplasia in Infants < 30 Weeks Gestation
- Building Perinatal Case Manager Capacity Using Quality Improvement
- Longitudinal adoption rates of complex decision support tools in primary care
- Improving the quality of tracheostomy care
- Health Care-Associated Infections Among Critically Ill Children in the US, 2007-2012
- A Multicenter Cohort Study of Treatments and Hospital Outcomes in Neonatal Abstinence Syndrome
- Changes in ventilator strategies and outcomes in preterm infants
- Standardizing Umbilical Catheter Usage in Preterm Infants
- Decreased Incidence of Pneumothorax in VLBW Infants After Increased Monitoring of Tidal Volumes
- Mortality and Neonatal Morbidity Among Infants 501 to 1500 Grams From 2000 to 2009
- Improved Outcomes in a Quality Improvement Collaborative for Pediatric Inflammatory Bowel Disease
- The Safety Attitudes Questionnaire as a tool for benchmarking safety culture in the NICU
- Sustained Reduction in Neonatal Nosocomial Infections Through Quality Improvement Efforts
- The Safety Attitudes Questionnaire as a tool for benchmarking safety culture in the NICU
- Strategies for Addressing Barriers to Publishing Pediatric Quality Improvement Research
- Translating Evidence into Practice, Policy, and Public Health in Perinatal Medicine
- Effectiveness of collaborative improvement: evidence from 27 applications in 12 less-developed and middle-income countries
- Impact of Sepsis on Neurodevelopmental Outcome in a Swiss National Cohort of Extremely Premature Infants
- Ohio Statewide Quality-Improvement Collaborative to Reduce Late-Onset Sepsis in Preterm Infants
- Neonatal infections in England: the NeonIN surveillance network
- Intercenter Differences in Bronchopulmonary Dysplasia or Death Among Very Low Birth Weight Infants
- Using quality-improvement methods to reduce variation in surfactant administration
- National quality campaigns: who benefits?
- NICU Practices and Outcomes Associated With 9 Years of Quality Improvement Collaboratives
- Improving the quality of care for infants: a cluster randomized controlled trial
- Rapid quantitative procalcitonin measurement to diagnose nosocomial infections in newborn infants
- Decreasing Trends in Malnutrition Prevalence Rates Explained by Regular Audits and Feedback
- Construction of a Parent-Derived Questionnaire to Measure End-of-Life Care After Withdrawal of Life-Sustaining Treatment in the Neonatal Intensive Care Unit
- BAPM/NNS: Infection and Gut
- Black/White Differences in Very Low Birth Weight Neonatal Mortality Rates Among New York City Hospitals
- Overtreatment in threshold and developed countries
- Implementing Pay-for-Performance in the Neonatal Intensive Care Unit
- A Cluster-Randomized Trial of Benchmarking and Multimodal Quality Improvement to Improve Rates of Survival Free of Bronchopulmonary Dysplasia for Infants With Birth Weights of Less Than 1250 Grams
- Evaluation and Development of Potentially Better Practices to Reduce Bronchopulmonary Dysplasia in Very Low Birth Weight Infants
- Overcoming Barriers to Oxygen Saturation Targeting
- Reduction of Bronchopulmonary Dysplasia After Participation in the Breathsavers Group of the Vermont Oxford Network Neonatal Intensive Care Quality Improvement Collaborative
- Standardizing Nasal Cannula Oxygen Administration in the Neonatal Intensive Care Unit
- Developing and implementing new safe practices: voluntary adoption through statewide collaboratives.
- The Association of Third-Generation Cephalosporin Use and Invasive Candidiasis in Extremely Low Birth-Weight Infants
- The nursery needs discoveries
- Relationship between probable nosocomial bacteraemia and organisational and structural factors in UK neonatal intensive care units
- Primary Care Practice-Based Research Networks: Working at the Interface Between Research and Quality Improvement
- Follow-up Care of High-Risk Infants
- Collaborative quality improvement to promote evidence based surfactant for preterm infants: a cluster randomised trial
- Variations In The Quality Of Care For Very-Low-Birthweight Infants: Implications For Policy
- Bias in Reported Neurodevelopmental Outcomes Among Extremely Low Birth Weight Survivors
- Marginal Increase in Cost and Excess Length of Stay Associated With Nosocomial Bloodstream Infections in Surviving Very Low Birth Weight Infants
- Voluntary Anonymous Reporting of Medical Errors for Neonatal Intensive Care
- Center Differences and Outcomes of Extremely Low Birth Weight Infants
- The Vermont Oxford Network: Improving Quality and Safety Through Multidisciplinary Collaboration
- Measuring the Quality of Children's Health Care: A Prerequisite to Action
- Analysing differences in clinical outcomes between hospitals
- Improving Growth of Very Low Birth Weight Infants in the First 28 Days
- NIC/Q 2000: Establishing Habits for Improvement in Neonatal Intensive Care Units
- Evaluation and Development of Potentially Better Practices for Improving Family-Centered Care in Neonatal Intensive Care Units
- Implementing Potentially Better Practices to Improve Neonatal Outcomes After Reducing Postnatal Dexamethasone Use in Infants Born Between 501 and 1250 Grams
- Using Organizational Assessment Surveys for Improvement in Neonatal Intensive Care
- Evaluation and Development of Potentially Better Practices to Prevent Neonatal Nosocomial Bacteremia
- Evidence-Based Approach to Change in Clinical Practice: Introduction of Expanded Nasal Continuous Positive Airway Pressure Use in an Intensive Care Nursery
- Nosocomial Infections in the Neonatal Intensive Care Unit
- Outcomes for high risk New Zealand newborn infants in 1998-1999: a population based, national study
- Progress of unit based quality improvement: an evaluation of a support strategy
- Intersite Differences in Weight Growth Velocity of Extremely Premature Infants
- Quality collaboratives: lessons from research
- Late-Onset Sepsis in Very Low Birth Weight Neonates: The Experience of the NICHD Neonatal Research Network
- Trends in Mortality and Morbidity for Very Low Birth Weight Infants, 1991-1999
- Epidemiological, Clinical, and Microbiological Characteristics of Late-Onset Sepsis Among Very Low Birth Weight Infants in Israel: A National Survey
- Quality in Pediatrics: A Progress Report for the American Academy of Pediatrics
- Economic Implications of Neonatal Intensive Care Unit Collaborative Quality Improvement