PEDIATRICS Vol. 103 No. 1 Supplement January 1999, pp. 203-214

SECTION 1: EVIDENCE-BASED QUALITY IMPROVEMENT, PRINCIPLES, AND PERSPECTIVES:
Quality Improvement Methods in Clinical Medicine

Paul E. Plsek

From Paul E. Plsek and Associates, Inc, Roswell, Georgia.

	ABSTRACT

Top Abstract References

This article surveys the methods and tools of quality improvement used today in health care. Specifically, we describe how clinicians can use these methods to impact the clinical practice of medicine. Improvement teams from a variety of health care organizations have reported the successful use of basic methods such as group work, flowcharting, data collection, and graphical data analysis. In addition to these incremental, problem-solving methods borrowed from the industrial practice of improvement, we have also seen the use of specific process design methods in health care applications such as care path development. The pace of change in health care has also led to the practical development of newer methods for rapid cycle improvement. We will review the basic approach behind these methods and illustrate key elements such as the ideas of change concepts and small-scale tests of change. Unfortunately, whereas these methods have been very successful and highly appealing to improvement practitioners, they may also have inadvertently widened a gulf between these practitioners and traditional health-services and clinical researchers. We offer an assessment of this issue and suggest ways to narrow the communication gap. Measurement has also traditionally been a part of the thinking about quality assurance and improvement in health care. We review the new philosophy of measurement that has emerged from recent improvement thinking and describe the use of control charts in clinical improvement. Benchmarking and multiorganizational collaboratives are more recent innovations in the ways we approach improvement in health care. These efforts go beyond simple measurement and explore the why and how associated with the widespread variation in performance in health care. We explore a variety of health care examples to illustrate these methods and the lessons learned in their use. We conclude the article with an overview of four habits that we believe are essential for health care organizations and individual clinicians to adopt to bring about real improvement in the clinical practice of medicine. These are the habits for: 1) viewing clinical practice as a process; 2) evidence-based practice; 3) collaborative learning; and 4) change.

Structured quality improvementthe use of systems thinking, data analysis, and teams to bring about better outcomes, higher satisfaction, improved processes, and reduced variationis now widely practiced in health care. Whereas early efforts were primarily directed at administrative and service processes, the techniques are increasingly being applied to clinical issues as well.^1-5

In this article we will examine how clinicians can use the methods of quality improvement to impact the practice of medicine. We will survey the methods of improvement science from general industry and describe the newer models of rapid cycle improvement. We will also show how clinicians are using the statistical methods associated with control charts to improve care for patients. Finally, we will describe how health care leaders are using benchmarking and multiorganizational collaboratives to take advantage of existing variation in practice as an opportunity for learning.

	MODELS AND TOOLS IMPORTED INTO HEALTH CARE FROM INDUSTRIAL QUALITY MANAGEMENT

The modern approach to the management of quality in health care borrows heavily from the quality management science in use for decades in general industry.^6,7 Industrial quality management sciencealso known as continuous quality improvement or total quality managementis an eclectic collection of techniques borrowed from the fields of systems theory, statistics, engineering, psychology, and others. Many of these improvement methods have been in use in general industry for more than 50 years, and new techniques are constantly being developed.

Improvement Teams and Project Models

Many health care organizations have used multidisciplinary teams as a mechanism for bringing about improvements in quality. The use of teams for quality improvement goes back to the 1930s and the work of industrial quality experts such as Joseph Juran.⁸ Improvement teams typically consist of 3 to 9 people who routinely work in the care process under investigation. For example, an improvement team at HealthPartners (Minneapolis, MN) that increased pediatric immunization rates consisted of two pediatricians, a family practitioner, a pediatric head nurse, a clinic manager, and an improvement facilitator.⁹ The makeup of improvement teams explicitly recognizes that good care often depends on the coordinated activities of many different people; it cannot be the result solely of the action of a single clinician, however well-intentioned or skilled.

A quality improvement model, such as the one in Fig 1, usually guides the work of these improvement teams. Models provide a high-level road map to remind the team to explore thoroughly the work process under study, involve key staff, and rely on the scientific method to guide decisions.¹⁰ In addition to guiding improvement efforts, such models also establish a common approach and vocabulary for improvement; making it easier for people from different disciplines and backgrounds to work together.

View larger version (71K): [in this window] [in a new window]   Fig. 1.   Quality improvement project model. Such models provide high-level, problem-solving guidance to improvement teams. This model is based on the work of Joseph Juran, as described in Plsek PE, Onnias A, Early J. Quality Improvement Tools. Wilton, CT: Juran Institute; 1989.

The thinking process that lies behind improvement models should be quite familiar to clinicians. Systematic process improvement is conceptually the same as the scientific method used in research and clinical decision-making. Joseph Juran,¹¹ a leading expert in the field of quality, explicitly acknowledged this connection back in the 1940s when he used the terms "diagnostic journey" and "remedial journey" to describe his idealized approach to industrial quality improvement.

The work of improvement teams is also aided by a set of simple engineering and statistical tools. These methods too have been in wide use in general industry for more than 50 years and are described fully in a variety of references.^12-15

Tools for Process Description

A flowchart graphically depicts the sequence of steps in a work process. It is a chronological description of a process. In health care, flowcharts might describe the flow of patients (the process for admitting); information (the handling of lab orders and test results); materials (the flow of supplies from receiving to the operating theater); or thought (the clinical algorithm for the treatment of low back pain). Sometimes, particularly in early process improvement efforts, the flowchart is the only tool needed. As teams document the sequence of activities, they often uncover redundant steps, wasted effort, and unnecessary complexity. In such cases, improvement can be a simple matter of common sense.

Whereas a flowchart describes the process as a chronological sequence, cause-effect analysis seeks to understand the process as a system of causal factors. A cause-effect diagram can be constructed around a clinical area of interest (for example, "what are the factors that lead to good diabetes care?") or a problem area (for example, "what causes nosocomial infections?"). To ensure comprehensive thinking, cause-effect analysis is often guided by explicit consideration of generic categories of factors such as people, equipment, supplies, information, methods, measurements, and environment.^16,17

Tools for Data Collection

The scientific method underlies classic industrial improvement methodology and calls us to be objective in our thinking. Therefore, improvement teams also often use simple tools for data collection.¹⁸

Data collection begins with the formulation of a specific question for which we are seeking an answer. For example, a team working to reduce the incidence of pneumothorax in neonates might ask: "what percentage of infants born in our delivery room at less than 30-week's gestational age received prophylactic surfactant?"

Having formed specific questions, the improvement team typically gathers data using simple check sheets, data sheets, interviews, and surveys. A check sheet is a form for gathering data that enables one to analyze the data directly from the form. For example, a team working to improve the timeliness of the administration of thrombolytic therapy to chest pain patients in the emergency department might construct a check sheet to study the distribution of times from presentation in the emergency room to administration of the thrombolytic. The check sheet might consist of a sheet of graph paper with a horizontal axis labeled in 10-minute intervals from 0 to 180 minutes. Each time thrombolytic therapy is administered, a responsible nurse places an X in the appropriate 10-minute interval column to indicate the elapsed time since the patient presented. After some time, a histogram of administration times emerges. The mean, spread, and characteristic shape of the time distribution can be read directly from the data collection form. In contrast, a data sheet is a form for recording data for which additional processing is required. In the thrombolytic therapy improvement case, this might be a simple logbook with text and numerical entries for each patient. Interviews and surveys are used when the question of interest involves perceptions. For example, the thrombolytic therapy team might want to monitor patient satisfaction as they change the processes in the emergency department.

Tools for Data Analysis

Data collection leads naturally to data analysis. Keeping in mind that the purpose of the data is to stimulate constructive change, this analysis must be simple enough for everyone in the care process to understand. So, although quality management practitioners do use computational statistical methods such as analysis of variance and tests of hypothesis (for example, t tests), simple graphical analysis methods are preferred because more people understand them. Experience in health care has shown that although the physicians, nurses, and secretaries on a team might not be equally facile with an analysis of variance table, they can all learn to look for certain departures from the classic bell-shaped curve in a simple histogram. Therefore, simple tools such as bar charts, histograms, line graphs, and scatter diagrams are the staples of data analysis in quality improvement projects.

It is important to note here that although the classic methods of industrial quality improvement are based on the scientific method, they were never intended to stand up to the rigor demanded in a full-scale medical research study. Berwick¹⁹ has suggested that improvement efforts represent a new type of knowledge-generating mechanism that lies somewhere between the currently unacceptable option of simply living with the rampant variation in daily practice, and the full rigor of relatively expensive and time-consuming traditional research. Others have made similar points about the potential pitfalls of over-reliance on classic statistical methods in quality improvement and other decision-making situations.^20,21

Tools for Collaborative Work

Widespread staff involvement and unprecedented cooperation across traditional boundaries are key principles of quality improvement whose roots again go back to the 1940s industrial practice of quality management.^22,23 Collaborative work is especially critical to success in improving health care because health care is a system of interdependent resources, the functioning of which depends primarily on how well we communicate with one another.²⁴ To support this collaborative work, quality management practitioners have borrowed several tools (ie, brainstorming, nominal group technique, and conflict resolution methods) from fields such as psychology and organizational development.^25-27

Models and Tools for Process (Re)Design

Whereas the graphical and team tools cited above are most often used to improve existing processes, industrial quality science also includes models and tools for planning or redesigning processes.^28-31

The first step in these design efforts typically involves defining the aim of the new process based on analysis of customer needs. Two illustrative health care examples are Gustafson and colleagues'³² use of the critical incident technique (also called moments of truth analysis) to guide the redesign of information-giving processes in breast cancer care at the University of Wisconsin Medical Center (Madison, WI) and Niles and colleagues'³³ use of focus groups to guide the redesign of cardiac care at the Dartmouth-Hitchcock Medical Center (Lebanon, NH).

Process (re)design project teams typically proceed by constructing flowcharts of an ideal process to meet the customers' needs. These flowcharts lead naturally to focused planning for the internal hand-offs between individuals and departments that are often the sources of process breakdowns. Next, the design team reviews the ideal process step-by-step and plans for realistic contingencies using techniques such as failure modes and effects analysis.³⁴ Finally, the design team plans for the measurements and controls that are needed to assure quality.

The importance of process design has led to the development of new tools specifically adapted for health care. Critical paths (also called care paths, clinical paths, care maps, and other names) are multidisciplinary, high-level process design efforts that specify key milestones in the care process for patients in a given diagnostic category.^35,36 These explicit milestones aid coordination of care, help reduce length of stay, improve quality of care, increase patient and family involvement, and enhance cross-departmental cooperation. Similarly, clinical guidelines and algorithms outline the process of clinical decision-making and thereby help focus the design of clinical practice.^37,38 Both critical paths and guidelines are used widely in the design of health care processes; with good reported results over several clinical topics.^39-41 Both are described further in an article by Bergman⁴² in this collection.

	RAPID CYCLE IMPROVEMENT IN HEALTH CARE

Although most improvement teams that use the classic, industrial quality management methods described above find them effective in making continuous improvement in health care processes, many leaders complain about the slow pace of such efforts. This frustration has led to the recent development of enhancements to these traditional methods that can be generally described under the umbrella of rapid cycle improvement.

Based on empirical observation in four health care organizations that had accelerated their improvement efforts, Alemi and colleagues⁴³ offer several suggestions for speeding up the improvement process. They suggest that the pace of improvement efforts can be accelerated by a combination of such things as:

Being thoughtful about topic selection.

Using good meeting skills and optimizing time management.

Focusing on testing the change you wish to make, rather than detailed analysis of the current process.

Collecting only the data you really need. (The authors go so far as to suggest that a team identify the data elements it thinks it needs, concoct a set of data, analyze that data fully, and then reflect on whether all the data requested is really essential to forming conclusions!)

Thinking about replication of changes throughout the organization from the very beginning of the project by widely sharing information about the project in progress.

Model for Rapid Cycle Improvement

Whereas Alemi and colleagues⁴³ offer general advice for accelerating improvement, Nolan, Langley, and their colleagues^44,45 have formalized much of this learning in a new model for improvement (see Fig 2). The theory behind this model, supported by experience, suggests that organizations able to make rapid gains have an underlying capacity to answer three questions:

View larger version (30K): [in this window] [in a new window]   Fig. 2.   A model for rapid-cycle improvement. Based on concepts described in Langley GJ, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco, CA: Jossey-Bass; 1996.

What are we trying to accomplish? (aim)

Example: Reduce respiratory distress syndrome (RDS) and intraventricular hemorrhage (IVH) by 50% in infants 501 to 1500 g.

How will we know if a change is an improvement? (measurement)

Example: Incidence of RDS and IVH in infants 501 to 1500 g.

What changes can we make that will result in an improvement? (change concept, explained below)

Example: Adherence to National Institutes of Health Consensus Conference recommendations for antenatal corticosteroid treatment.⁴⁶

Based on the answers to these three questions, the theory and experience suggests that successful organizations then set in place small-scale tests of change in Plan-Do-Study-Act (PDSA) cycles. That is, they systematically plan a specific adaptation of the change concept that they think will move them closer to the stated aim, they do it, they study the effect of the intervention through measurements, and they act on their learning to start another cycle. (The concept of building knowledge through PDSA cycles has a rich and long history. Shewhart⁴⁷ and Deming⁴⁸ brought the concept into industrial quality management, but its roots can be traced to philosopher John Dewey. The similar models of Donald Schön⁴⁹ and David Kolb⁵⁰ in the fields of education and organizational development represent a parallel development.)

Figure 3 provides an example of a small-scale test of change for the aim stated above. The team's plan is to work with one obstetrics/gynecology (OB/GYN) practice to implement a specific process to identify at-risk women and administer the recommended treatment. Note that the small-scale nature of the change involves working with a single OB/GYN practice. The team will learn something by doing this. Of course, several cycles will be needed to accomplish the overall aim; but the key point is that nothing in a system of care will change until something starts changing. Rapid, small-scale PDSA cycles build the momentum of change; something that is often missing in large-scale change efforts based on comprehensive data collection and one time, all-or-nothing implementation.

View larger version (71K): [in this window] [in a new window]   Fig. 3.   Sample report from a rapid cycle improvement effort. Hypothetical, for illustration purposes only.

It is important to note that measurement must also be tailored to fit the small scale of a particular cycle. By working with only a single OB/GYN practice, the team cannot expect to significantly alter the overall RDS and IVH rates in its neonatal intensive care unit. Instead, they have chosen an appropriate measurement for this specific test of change: a before-and-after comparison of the percentage of at-risk women from this particular OB/GYN practice who have properly received antenatal corticosteroid treatment. In perhaps a few weeks' time the team can learn enough about the process it has developed with this one OB/GYN practice to begin planning the next cycle of change.

In rapid cycle improvement, pace is crucial. The theory suggestsand experience confirmsthat it is better to run small cycles of change soon, rather than large ones after a long time. The reason is that each cycle, properly done, is informative, and provides a basis for further improvement. The more cycles, the more learning.⁵¹

The simple tools of classic, industrial quality management science can also be useful in these rapid cycle improvement efforts. For example, the team might find it useful to construct a flowchart of the new process to be followed in the OB/GYN practice. They might also jointly develop a simple check sheet to record data on at-risk women and display these data on bar charts. The caution here goes back to the advice of Alemi and colleagues⁵² cited above. To maintain an appropriate pace in the improvement effort, the team should optimize its use of meeting time, focus on flowcharting the future process rather than consuming time documenting the existing process, and question how much data are really needed to develop a level of comfort that the new procedures are effective enough to warrant moving on to the next cycle of change.

Change Concepts

The notion of a change concept is an important aspect of rapid cycle improvement. A change concept is a general idea that responds to the third question in the model of Fig 2: "What changes can we make that will result in an improvement?" Nolan and Schall⁵³ note:

"Ideas for change can come from a variety of sources: critical thinking about the current system, creative thinking, observing the process, a hunch, an idea from the scientific literature, or an insight gained from a completely different situation. A change concept is a general idea, with proven merit and sound scientific or logical foundation, that can stimulate specific ideas for changes that lead to improvement."

In the above example, "adherence to National Institutes of Health Consensus Conference recommendations for antenatal corticosteroid treatment" is a change concept. It is a generally good idea, in this case based on high-grade evidence from the scientific literature, which should lead to the accomplishment of the aim.

Change concepts are generally good ideas, not specific ideas ready-to-implement. The task of the rapid cycle improvement team is to adapt the change concept to their specific context and map out the specifics of implementation. This adaptation to local context is key. It increases commitment to the change and explicitly recognizes that the local practice of medicine has some inherent level of appropriate variation.

The notion of change concepts has a slipperiness to it that should be acknowledged but should not be allowed to get in the way of its practical use. Concepts can be expressed at different levels of abstraction.^54,55 For example, a higher level of abstraction for the change concept above might be the statement: "Adhere to evidence-based, consensus guidelines." This is a more general statement of a good idea for change that could be applied to many clinical issues. The change concept that the team used is simply a more specific statement in reference to the team's aim; but still not specific enough to say exactly what tests of change the team should try in its local context. How abstract or specific we should make our statement of change concepts is primarily a matter of style and situation. In reality, there is always a range of possible concept statements from the very abstract to the very specific. As we move up the ladder of abstraction, the concepts should be based on sound logic and good sense. We could say that change concepts should fit one of the grades of evidence from the literature on evidence-based medicine.⁵⁶ As we move down the ladder, the concepts and specific tests of change should be logically connected, increasingly sensitive to local context, and increasingly concrete.

The fact that change concepts can be expressed at various levels leads to the notion that change concepts can be catalogued and made generally available to aid improvement work. Such a catalog should greatly accelerate improvement efforts by avoiding long searches for potentially useful ideas to try in PDSA cycles. Several such catalogues have been proposed; covering topics such as waits and delays, medical errors, asthma care, and intensive care.^57-62 It is probably only a matter of time before a unified catalogue of change concepts appears.

To illustrate the utility of a catalogue of change concepts, consider the experience of two teams involved in rapid cycle improvement through participation in Breakthrough Series Collaboratives sponsored by the Institute for Healthcare Improvement (IHI, Boston, MA). The IHI regularly assembles change concept catalogues in specific clinical areas based on literature reviews and recommendations from a panel of experts. A team at The Mayo Clinic (Rochester, MN) had as its overall aim to improve the outpatient care provided to asthma patients in the Family Medicine Clinic. Through participation in the IHI project, the team learned about the following change concepts: build capacity for routine assessment of patient outcomes (through the use of standardized assessment tools); reduce unintended variations in care (through simple guidelines and in prescribing practices); streamline the process of care (as it applies to both anticipatory patient management and acute episodes); and build information systems capacity (to establish routine feedback of data on outcomes). By implementing these concepts through a dozen or so small-scale cycles of change, the Mayo team reduced hospitalization rates in various age groups by 23% to 47% and reduced emergency department visits by 22%.⁶³ Similarly, a team from Centura-St. Anthony Central Hospital (Denver, CO) implemented 3 change concepts from a list of 11 concepts developed by an expert group on adult intensive care. By adapting these 3 concepts to their local contextcreate a formal process, consider people to be in the same system, and reach agreement on expectationsthe team decreased readmission to the intensive care unit from 15.6% to 9.8%, increased overall patient and family satisfaction from 70% to 95%, decreased costs for sedation drugs by 80%, and decreased average direct cost per case from $5900 to $4100.⁶⁴

Future Directions and Critical Assessment

With the ever-increasing pressure for change in health care, rapid cycle improvement methods and catalogues of change concepts are welcomed enhancements to classic quality improvement methods. The model of Fig 2 and the concept of small-scale tests of change to establish momentum stresses implementation of what is known to improve care. Health care desperately needs to implement what it knows.

The current practice of rapid cycle improvement could be strengthened, however, by the more thoughtful use of evidence grading in the presentation of change concepts. Several evidence-grading systems exist and most allow a category for expert opinion based on experience, observation, and common sense.⁶⁵ The current catalogues of change concepts do not always explicitly grade their recommendations, although some of the recommendations are based on sound studies published in peer-reviewed journals. The absence of explicit grading of evidence and the fact that some change concepts are obviously only opinion, has unnecessarily allowed critics of rapid cycle improvement to label the entire effort as unscientific (which it is not).

Advocates of rapid cycle improvement could also help spread the methods more widely in health care by having the minimal discipline to use simple tests of significance (or control charts, explained below) in reporting overall results. The small-scale, rapid cycle nature of the PDSA cycles that are at the heart of the method obviously (and correctly, in my opinion) precludes the use of tests of significance in each cycle of change. However, since the aim and overall measurement do remain fixed during a period of months as the various cycles of change take place, data accumulate throughout time allowing overall tests of significance in at least a before and after comparison. Most reports on rapid cycle improvement projects, such as those cited above, merely state the overall improvement (such as "from 15.6% to 9.8%") or show a time series of measures in run chart form. To avoid getting bogged down in massive data collection at every cycle of change, proponents of rapid cycle improvement are sometime unnecessarily defiant in avoiding the use of statistical techniques; even when they have accumulated enough data during several cycles of change to enable the use of such techniques. The lack of reports of statistical significance, even when such tests would be easy to do, widens the gulf between practitioners of practical, real-time improvement of health care and those demanding more experimental rigor; unnecessarily delaying the diffusion of good ideas that will improve health care for all.

	THE USE OF MEASUREMENT TO SUPPORT IMPROVEMENT IN HEALTH CARE

Measurement of performance has always been an integral part of the quality management sciences. Measurement provides the feedback loop in the system that helps one maintain performance at a desired level (quality control and assurance) and signals the need for change or the accomplishment of productive change (quality improvement). Similarly, there is a long tradition of measurement in health care. The introduction of quality management science in health care organizations should be seen as building on, not tossing out, the rich tradition and current efforts in the measurement of performance in health care.

A New Philosophy of Measurement

Although quality management science builds on the tradition of measurement in health care, it also encourages us to seek three new objectives. First, quality management science encourages us to expand the scope of our thinking about what is important to measure by prominently featuring the perceptions of patients and families as valid indicators of quality, in addition to clinical and professionally based views of performance. Second, quality management science focuses on cross-functional processes and suggests that we view measurements as integrated systems that must be managed by cross-functional teams rather than having one set of measures tracked by medical staff, another set by nursing, and still another by administration. Third, quality management science calls into question the traditional use of measurement as a way of allocating rewards and punishments to individuals. Berwick's⁶⁶ seminal article on this topic, in which he described the "search for bad apples" that seems to characterize our traditional approach to measurement, is widely cited as the trigger that initiated the introduction of industrial quality management science into health care.

Control Charts

A specific measurement tool from the quality management sciences that does warrant mention here in this overview of improvement methods for clinical medicine is the control chart.^67-69 A control chart is a line graph of data with superimposed horizontal lines indicating statistically derived upper and lower control limits (Fig 4). These upper and lower limits indicate the range of variability that one would expect if the variation is subject only to small, randomly occurring factors that are inherent in the process (so-called common cause variation). If the measured data points fall randomly within the control limits, we say that the process is stable and predictable; performance will continue to fall within the limits as long as the process remains as it is. Furthermore, we can also assert that further improvements in performance can only come about through fundamental changes in the process itself. Reacting to individual ups and downs in the data within the control limits (for example, "infection rates were higher this month, I'll speak to the staff and admonish them to do better") is called tampering and is likely to be counterproductive.^70,71

View larger version (22K): [in this window] [in a new window]   Fig. 4.   Control chart. This is an example of what control chart practitioners call a "p-chart." The data indicate the percentage of orders in a daily, random sample of 150 at an outpatient pharmacy for which patients had to wait more than 10 minutes. The mean (average) percentage of untimely orders during the 20-day period was 18.6% (0.186). This is indicated by the solid, horizontal center line on the chart. The standard deviation of this time-series of data were calculated using the formula SD = SQRT {[mean percentage × (1  mean percentage)]/[sample size]}; where here the mean percentage is 0.186 and sample size is 150. This is the normal approximation of the standard deviation of the binomial distribution and indicates the expected variation because of random causes (what control chart practitioners call "common cause variation"). The upper and lower control limits (UCL and LCL, respectively) are set by convention at ±3 SD beyond the mean. "X" on the control chart indicates nonrandom (special cause) variation. Special cause variation is defined by run rules based on normal distribution theory. Data points for days 8 and 9 are marked because 2 out of 3 consecutive points lie more than 2 SD beyond the mean. Data points for days 15 to 18 are marked because 4 out of 5 consecutive points lie more than 1 SD beyond the mean. Both of these are indications of statistically significant shifts in the performance of the process.

When data values fall outside the control limits, or exhibit certain unnatural patterns within the control limits, there is statistical evidence of a so-called special cause. The evidence in the data suggests that the variation is not random and we should, therefore, be able to isolate the source and remove it from the system.

Carey and Lloyd⁷² provide several case studies of the use of control charts for clinical quality measurement of laboratory turn-around time, cesarean section rate, patient falls, use of restraints, and medication errors. Nelson and colleagues⁷³ give examples of the use of control charts in monitoring length of stay, time to therapy, and duration of therapy for community-acquired pneumonia (Dartmouth Medical School, Lebanon, NH). Use of a control chart explicitly recognizes that such indicators will naturally vary from reporting period to reporting period. Both wringing our hands in the Quality Assurance committee because the numbers are a little worse this month, and congratulating ourselves when they are a little better might well be a waste of effort.

Control charts can also be useful at the level of care to individual patients. Carey and Lloyd⁷⁴ provide an example of the use of a control chart to help a physician and her cancer patient who has undergone an autologous bone marrow transplant make sense of the variation in platelet counts from numerous daily blood draws. Gibson and colleagues⁷⁵ (John Hunter Hospital, New South Wales) report on the use of control charts to help physicians understand the degree of variability in individual asthma patient's peak expiratory flow rates. The analysis of the control charts in this case enabled physicians and patients to work together to individualize care plans to manage asthma exacerbations at an earlier stage. Several clinicians have reported on the use of control charts to respond appropriately to variation in blood sugar readings for diabetics.^76,77 Laffel and colleagues⁷⁸ have analyzed continuous data streams from intensive care unit patients that suggest that physicians may be over-medicating patients in response to natural variation in readings. Further clinical applications of control charts will surely develop in the coming years.

	BENCHMARKING AND ITS APPLICATION TO CLINICAL IMPROVEMENT

Benchmarking is the process of comparing one's performance to that of others.^79-81 The formal practice of benchmarking is a relatively recent innovation in the quality management sciences, having been developed in the 1980s at Xerox. Benchmarking begins with standardized, comparative measurement, but true benchmarking goes deeper to understand why there are performance differences between seemingly similar processes.

Building Process Knowledge

It is important to underscore the phrase "true benchmarking" in the preceding paragraph. Many activities that are called benchmarking in health care are actually only comparisons of performance indicators across clinicians or organizations. Although such indicator benchmarking provides information about where one stands relative to peers, it gives little information about why someone else's performance is better. Knowing why something is better is the key to improving one's own processes. O'Conner and colleagues⁸² point out the inherently "insular nature of clinical medicine" and go on to note that "the inadequacy in the detail of the information on current clinical practice prevents knowledge of the fine structure of care and makes studies linking practice to outcomes difficult." Process benchmarking explicitly seeks to uncover knowledge about this fine structure; knowledge that can lead to improvement.

Specific methods for uncovering this knowledge vary. Usually, but not always, these details are uncovered through site visits. For example, the members of the Northern New England Cardiovascular Disease Study Group⁸³ conducted site visits to each others' locations to observe the processes of cardiac surgery. Multidisciplinary teams consisting of surgeons, nurses, profusionists, and industrial engineers spent a day or two at the host institution observing the entire cardioarterial bypass graft process from the catheterization conference through postoperative care. The hosts conducted business as usual, while the visitors focused on identifying similarities and differences compared with their own processes. The visitors and hosts then worked together to document the observations from each site visit in reports that Kasper et al⁸⁴ describe as "candid and high in information content." Information gleaned from these visits led to a 24% reduction in in-hospital mortality after cardioarterial bypass graft surgery among the 23 cardiothoracic surgeons who participated in the effort (P < .001).

The Vermont-Oxford Network NIC/Q Project used a similar series of site visits to study practices that led to reductions in nosocomial infection rates and better care for infants with chronic lung disease.⁸⁵ Before embarking on day-long visits, the multidisciplinary visiting teams conducted internal studies of their own local practices and identified specific questions to explore during the visit. Benchmarking experts cite such careful preparation as one of the keys to successful benchmarking. It is the essence of the distinction that Garvin⁸⁶ makes between true benchmarking and "industrial tourism."

The drawback of benchmarking studies, such as those cited above, is the time and cost associated with site visiting. However, there are examples of successful benchmarking efforts that address this drawback. For example, the SunHealth Alliance, a partnership of more than 200 hospitals, employs a central council to identify topic areas and collect relevant performance indicators from its members.⁸⁷ From these data, the council selects a group of 10 to 20 organizations; including some poor performers, some superior performers, and some in between. Then, instead of a series of site visits, each organization brings its flowcharts and other process documents to a multiday meeting, during which they pour over the documents and data to cull the best practices. Finally, members of the collaborative select the best practices that fit well with their own environment, develop an action plan, implement changes, and report back on results.

Although this approach eliminates the time and cost of numerous site visits, it does presume a relatively high level of sophistication in the use of the process analysis and data collection tools of quality management. This can be a significant drawback as it is not uncommon for clinicians and staff to be unaware of why their performance is superior or inferior to peers. The quality of the benchmarking analysis is also limited by the information that groups bring with them to the meeting, whereas in a site visit this information might be more routinely uncovered by the probing of outsiders not steeped in the traditions of the host organization. Despite the potential for drawbacks in this approach, SunHealth has reported improvements through group benchmarking in the clinical areas of circulatory disorders with cath (DRG 124), pneumonia (DRG 89), total hip replacement (DRG 209), acute myocardial infarction (DRG 122), and angioplasty (DRG 112).⁸⁸ Using a similar approach of group meetings, the 12 hospitals of the Benchmarking Effort for Networking Children's Hospitals (BENCHmark) also reported significant reductions in waiting times and costs.⁸⁹

UniHealth, another large, multiorganization consortium, further streamlines the benchmarking process by using a full-time staff department to assess comparative data and conduct site visits at the best performing organizations.⁹⁰ The investigative staff then share the best practice concepts with other UniHealth organizations through a series of regional conferences. Implementation is at the discretion of the individual facilities.

In a project on joint replacements, the UniHealth benchmarking effort with superior performing institutions identified three best practice concepts: early referral to physical therapy, preadmission visits by home health staff, and standardization of surgical prosthetics.⁹¹ Note that good benchmarking studies, regardless of the format used, produce change concepts that could feed rapid, small-scale PDSA cycles for implementation. Replication of the joint replacement change concepts by one UniHealth hospital resulted in reduced costs per case of $2000 and a 50% length of stay reduction (9 days to 4.5 days).

The potential drawback in this method of using centralized staff to conduct the site visits is the lack of involvement of local care-process staff who will need to make the changes necessary to bring about improvement. Visiting an organization and actually seeing a different way to do something is much more personally convicting in regard to the need for change than is reading about something or hearing a report.

Potentially Better Practices

The term "best practices" from the benchmarking literature carries an unfortunate connotation that can impede the deployment of benchmarking as a clinical improvement technology in health care. Because of the strong research tradition in health care, the term best practices implies to some that the identified practices are the result of an exhaustive search and rigorous experimental verification. This is not the intent of benchmarking. Benchmark is practical and action-oriented in its analysis; it is not a rigorous research methodology. Such rigor is not demanded in the industrial world where the concepts of benchmarking and best practices were developed.

Although it is awkward, I prefer the phrase "potentially better practices" to describe what health care benchmarking efforts are after. This phrase suggests that we are looking for good improvement ideas that have both the logical appeal and experience in practice to suggest they might lead to improvements in our own organization. We will not know if they are truly better until we adapt them to our local context, implement them, and measure the results. Because we cannot be sure, we must guard against unintended adverse outcomes, as we would in introducing any variation in practice. We will also not know whether such better practices are generalizable across all health care organizations until we conduct a formal, controlled trial. This may not be practical and, so, we may never know whether the practices are generalizable; we will only know if they worked or failed when we tried them. Finally, we should hold no illusions about these practices being the best in any absolute sense. We are limited by the organizations and experts we have consulted; even better practices may be out there somewhere and will most certainly appear in the future.

Benchmarking as a way to get ideas for improvement is a promising technology that breaks through the isolation that many clinicians report as the underlying cause for the well-documented variation in clinical practice in health care. However, benchmarking can be a time-consuming and expensive tool. It is certainly not indicated when appropriate change concepts are already available in the literature or through common knowledge. On the other hand, benchmarking may be the only way we will ever uncover new knowledge for improvement in some clinical areas where a program of systematic, randomized trials is impractical because of such things as the large number of potential factors to study, the rapid pace of technological change, or the nature of the intervention itself. Doing benchmarking well requires good observational skills, rather sophisticated knowledge of one's own local practices, and the ability to transform the learning from a visit into real change in local practice. Benchmarking may not be the easiest way to start with an improvement effort, but on some topics it may ultimately be the only way to go.

	CLINICAL IMPROVEMENT THROUGH MULTIORGANIZATION COLLABORATIVES

The history of the improvement methods that we have discussed so far is that they were developed primarily to assist individuals and individual organizations in meeting improvement goals. Even in benchmarking, which obviously requires at least one other partner to learn from, the improvement effort could be in only one of the partners.

More recently, in general industry but especially in health care, practitioners are discovering the power of collaborative efforts across multiple organizations that have banded together under a common improvement aim.⁹² Within these collaborative groups, organizations can pool data and information resources, learn from the rich set of experiences represented by the variation in practice, think outside the box of their local culture and customs, discuss ways to overcome common barriers, and mutually encourage one another through the tough processes of change.

We have already seen some examples of this type of collaborative effort. The IHI Breakthrough Series Collaboratives, the Northern New England Cardiovascular Study Group, and the Vermont-Oxford Network NIC/Q Project are illustrative of the many improvement collaboratives that have emerged in health care.

Key Concepts Behind Collaboratives

Figure 5 lists the key concepts behind these collaboratives.⁹³ To be successful, collaborative members must have enough variability in practice among them to make coming together for comparisons useful. Further, members must have the willingness to share this variation openly. Unfortunately, the trend toward mergers and acquisitions in the health care industry is placing stress on multiorganizational improvement collaboratives. Organizations can be understandably reluctant to share the details of their performance on quality indicators if they fear that these data might be used to put them at a competitive disadvantage. To avoid this difficulty, some collaboratives are formed solely from organizations within an existing health care system, other collaboratives explicitly select organizations that are not direct competitors, and others require members to agree formally to full disclosure within the group but strict nondisclosure to parties outside the group.

View larger version (28K): [in this window] [in a new window]   Fig. 5.   Key concepts behind successful improvement collaboratives.

Collaborative members must also have or acquire the skills both to know their own internal processes deeply and to inquire wisely into the processes of others. The tools of classic quality managementflowcharts, data collection, group working, benchmarking, and so oncan be valuable here. Collaborative improvement efforts do not replace an organization's quality management efforts; rather, they depend and build on them.

Finally, collaborative members must have the commitment and skills to implement what they learnreplicate the potentially better practices the collaborative uncoversand determine through measurement whether there has been an actual improvement in the local context. An improvement collaborative cannot be all talk and no action.

Current Results and Future Trends

The Northern New England Cardiovascular Study Group was the first large collaborative to report statistically significant clinical results with the lowering by 24% of in-hospital mortality after cardiac surgery.⁹⁴

Organizations in the IHI's Breakthrough Series collaboratives have made many dramatic improvements in waiting times, asthma care, cardiac surgery, cesarean section rates, adverse drug events, adult intensive care, and others. The article by Kilo⁹⁵ in this collection provides more details on these efforts.

Similarly, Horbar, Rogowski, and Plsek⁹⁶ describes significant reductions in infection rates and chronic lung disease among the 10 centers who participated in the Vermont-Oxford Network NIC/Q Project collaborative. Importantly, this group is the first major effort to report not only significant results in a before and after comparison within participating organizations, but also a statistically significant result when the group of 10 centers who participated in the collaborative were compared with a larger database of neonatal intensive care units not in the study. More comparisons to secular trends such as this are needed to boost confidence in the effectiveness of these methods.

Improvement collaboratives seem to be springing up throughout health care, sponsored by a variety of organizations. I have been personally involved in collaboratives sponsored by The HMO Group (a trade association of managed care plans, New Brunswick, NJ); the Centers for Disease Control (Atlanta, GA); the peer review organizations in Massachusetts and New York; The Health Care Forum (San Francisco, CA); the Voluntary Hospitals of America (Dallas, TX); the Ontario CQI Network (Toronto, Canada); the National Health Service in Britain (Buckinghamshire, England); and the Health Care System of Sweden (Stockholm, Sweden). Undoubtedly, there are many more than these in all corners of the United States and around the world.

Collaboration to improve health care represents the finest traditions of medicine. It remains to be seen whether the emergence of these many collaborative groups is a true reflection of those deep traditions, or merely another passing fad in health care. It also remains to be seen whether the increasing pressures of competition and intellectual property protection in the health care industry will close off future collaboration on clinical issues.

The building of new knowledge through multiorganizational collaboratives benefits us all. Patients obviously benefit from improved outcomes and reduced risk of complications. Clinicians and other health care professionals also benefit through the intrinsic reward of contributing to improved care. Clinicians involved in collaborative improvement efforts almost universally report enjoying the experience of interacting with colleagues on focused clinical issues. It is one way to rise above the ever-encroaching business issues and refocus on the care in health care.

	CONCLUSION: DEVELOPING FOUR HABITS FOR IMPROVEMENT

Although not wide-spread yet, there is considerable evidence building that the methods surveyed here are effective approaches for clinical improvement. Certainly, there is a great deal of case-by-case evidence of effectiveness. However, it would be naive to ignore the many anecdotal reports of improvement efforts that fail. These failed efforts do not often appear in the literature; there is considerable positive reporting bias.

At the same time, we should not be surprised that we are not able to report the generalizable effectiveness of interventions such as quality management with the same degree of certainty that we can a new drug therapy applied to a patient population. The difference lies in understanding the underlying stability and predictability of the systems in which we are intervening.

In the drug therapy case, our intervention is in human body systems. Human body physiology is remarkable stable across the population, at least at the macrolevel. If a certain drug therapy is effective in patients in an experimental study group, we can be reasonably certain that it will be effective in the population as a whole.

Such is not the case when we talk about interventions in human organizations and care process systems. Such systems can be vastly different from site to site. The effectiveness of an intervention is, therefore, inherently context-dependent. The methods of improvement must be adapted for use in the local setting. The fact that there is evidence of the effectiveness of improvement methodologies in at least some settings indicates that these methods can be effective. However, it says little about whether they will be effective in a given situation. Such is the nature of organizational and process interventions.

Nevertheless, we can speculate on the characteristics of organizations that are more likely to be successful most of the time with the improvement methods described here. Such organizations will likely demonstrate four key habits:

The Habit of Viewing Clinical Practice as a Process Good health care depends on the complex coordination of many factors, and the efforts of many people. Clinicians and health care organizations that are able to improve care routinely will inherently begin from this premise and will increasingly reject the more restrictive, discipline-based view of care.

The Habit for Evidence-based Practice Although it is probably not appropriate to completely eliminate variation in health care, clinicians and health care organizations that routinely improve will accept the fact that there is rampant unintended variation in health care. A significant proportion of the care delivered daily is not consistent with what we know to be most effective. Clinical improvement is primarily about the continuous effort to bring the daily practice of health care more in line with our knowledge of what works.

The Habit for Collaborative Learning Although there is a great deal of knowledge for improvement available in the literature as a result of standard research techniques such as randomized controlled trials, the knowledge of what makes for good care processes is currently locked up in unexamined variation in practice. The only way we may ever get at this knowledge is through collaborative learning with others. Improvement oriented individuals and organizations will start from the premise that it is better to be open and curious, than defensive.

The Habit for Change No matter how much we know, improvement only comes about when we do something differently. Clinicians and health care organizations that are successful at improvement know that improvement requires change. Although we certainly do not want to do harm nor make change for change's sake, holding fast to "the way we have always done it" is a prescription for mediocrity. Successful practice requires continual change.

	FOOTNOTES

Mr Plsek is an independent consultant in the field of quality management, with special focus in health care. He is also a Senior Fellow at the Institute for Healthcare Improvement in Boston, MA.

Received for publication Sep 9, 1998; accepted Sep 9, 1998.

Address correspondence to Paul E. Plsek and Associates, Inc, 1005 Allenbrook Lane, Roswell, GA 30075.

	ABBREVIATIONS

RDS, respiratory distress syndrome; IVH, intraventricular hemorrhage; PDSA, Plan-Do-Study-Act; OB/GYN, obstetrics/gynecology; IHI, Institute for Healthcare Improvement.

	REFERENCES

Top Abstract References

1. Blumenthal D Total quality management and physician's clinical decisions. JAMA. 1993; 269:2775-2778 [Medline]
2. Carlin E, Carlson R, Nordin J Using continuous quality improvement tools to improve pediatric immunization rates. Jt Comm J Qual Impr. 1996; 22:277-288
3. Heckman M, Ajdari SY, Esquivel M, Quality improvement principles in practice: reduction of umbilical cord blood errors in the labor and delivery suite. J Nurs Care Qual. 1998; 12:47-54 [Medline]
4. Berwick DM, Nolan TW Physicians as leaders in improving healthcare: a new series in Annuls of Internal Medicine. Ann Intern Med. 1998; 128:289-292 [Medline]
5. Lowers J Reducing variation in cataract surgery technique cuts complications. The Quality Letter for Healthcare Leaders. 1998; 10:19
6. Berwick DM, Godfrey AB, Rossener J. Curing Health Care: New Strategies for Quality Improvement. Jossey-Bass: San Francisco, CA; 1990
7. Laffel G, Blumenthal D The case for using industrial quality management science in health care organizations. JAMA 1989; 262:2869-2873 [Medline]
8. Juran JM. Managerial Breakthrough. New York, NY: McGraw-Hill; 1964
9. Carlin E, Carlson R, Nordin J. op. cit.
10. Plsek PE Quality improvement models. Quality Management in Health Care 1993; 2:69-81
11. Juran JM. op. cit.
12. Plsek PE, Onnias A, Early JF. Quality Improvement Tools. Wilton, CT: Juran Institute, Inc; 1989
13. Plsek PE Techniques for the management of quality. Hospital and Health Services Administration. 1995; 40:50-79
14. Ishikawa K. Guide to Quality Control. New York, NY: UNIPUB; 1985
15. Wadsworth HM, Stephens KS, Godfrey AB. Modern Methods for Quality Control and Improvement. New York, NY: John Wiley & Sons; 1986
16. Ishikawa K. op. cit.
17. Smith GF Determining the cause of quality problems: lessons from diagnostic disciplines. ASQ Qual Management J. 1998; 5:24-41
18. Plsek PE Planning for data collection. Quality Management in Health Care 1994; 2:69-81
19. Berwick DM Harvesting knowledge from improvement. JAMA. 1996; 275:877-888 [Medline]
20. Cohen J The earth is round (P < .05). Am Psychol. 1994; 49:997-1003
21. Jordan HS How rigorous need a quality improvement study be? Jt Comm J Qual Improv. 1995; 21:683-691 [Medline]
22. Juran JM. op. cit.
23. Ishikawa K. What Is Total Quality Control: The Japanese Way. Translated by David J. Lu. Englewood Cliffs, NJ: Prentice-Hall; 1985
24. Ferguson S, Howell T, Bataldin P Knowledge and skills needed for collaborative work. Quality Management in Health Care 1993; 2:1-11
25. Scholtes PR. The Team Handbook. Madison, WI: Joiner Associates; 1988
26. Mosel D, Shamp MJ Enhancing quality improvement team effectiveness. Quality Management in Health Care 1993; 1:47-57
27. Clemmer TP, Spuhler VJ, Berwick DM, Nolan TW Cooperation: the foundation for improvement. Ann Intern Med. 1998; 128:47-57
28. Juran JM. Juran on Quality By Design. New York, NY: The Free Press; 1992
29. Plsek PE The systematic design of health care processes. Quality in Health Care 1997; 6:40-48
30. Larson JA. Healthcare Redesign Tools and Techniques. New York, NY: Quality Resources; 1997
31. Lenz PR, Sikka A. Reengineering Health Care: A Practical Guide. Tampa, FL: American College of Physician Executives; 1998
32. Gustafson DH, Taylor JO, Thompson S, Chesney P Assessing the needs of breast cancer patients and their families. Quality Management in Health Care 1992; 2:6-17
33. Niles N, Tarbox G, Schults W, Using qualitative and quantitative patient satisfaction data to improve the quality of cardiac care. Jt Comm J Qual Improv. 1996; 22:323-335 [Medline]
34. Plsek PE. FMEA for process quality planning. Proceedings of the 1989 Annual Quality Congress of the American Society for Quality Control. Milwaukee, WI: ASQC; 1989
35. Mosher C. Upgrading practice with critical pathways. Am J Nurs. 1992:41-44
36. Coffey RJ, Richards JS, Remmert CS, An introduction to critical paths. Quality Management in Health Care 1992; 1:45-54
37. Woolf SH Practice guidelines: a new reality in medicine. Part 1: Recent developments. Arch Intern Med. 1990; 150:1811-1817 [Medline]
38. Green E, Katz JM Practice guidelines: a standard whose time has come. J Nurs Care Qual 1993; 8:23-32 [Medline]
39. Murrey KO, Gottlieb LK, Schoenbaum SC Implementing clinical guidelines: a quality management approach to reminder systems. Qual Rev Bull 1992; 18:423-433 [Medline]
40. Gregor C, Pope S, Werry D, Dodek P Reduced length of stay and improved appropriateness of care with a clinical path for total knee or hip arthroplasty. Jt Comm J Qual Improv. 1996; 22:617-628 [Medline]
41. Rudisill PT, Phillips M, Payne CM Clinical paths for cardiac surgery patients: a multi-disciplinary approach to quality improvement outcomes. J Nurs Care Qual. 1994; 8:27-33 [Medline]
42. Bergman D Evidence-based guidelines and critical pathways for quality improvement. Pediatrics. 1999; 103(suppl):225-232
43. Alemi F, Moore S, Headrick L, Rapid improvement teams. Jt Comm J Qual Improv. 1998; 24:119-129 [Medline]
44. Nolan TW, Schall MW. Reducing Delays and Waiting Times Throughout the Healthcare System. Boston, MA: Institute for Healthcare Improvement; 1996
45. Langley GJ, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco, CA: Jossey-Bass; 1996
46. NIH consensus development panel on the effect of corticosteriods for fetal maturation on perinatal outcomes. JAMA. 1995;273:413-418
47. Shewhart WA. Economic Control of Quality of Manufactured Product. New York, NY: Van Nostrand; 1931
48. Deming WE. Out of the Crisis. Cambridge, MA: Center for Advanced Engineering Study; 1986
49. Schön DA. Educating the Reflective Practitioner. Toward a New Design for Teaching and Learning in the Professions. San Francisco, CA: Jossey-Bass; 1988
50. Kolb DA. Experiential Learning. Experience as the Source of Learning and Development. Englewood Cliffs, NJ: Prentice Hall; 1984
51. Berwick DM Developing and testing changes in delivery of care. Ann Intern Med. 1998; 128:651-656 [Medline]
52. Alemi et al. op cit.
53. Nolan TW, Schall MW. op. cit.
54. Provost LP, Langley GJ The importance of concepts in creativity and improvement. Quality Progress. 1998; 31:31-38
55. Plsek PE. Creativity, Innovation, and Quality. Milwaukee, WI: ASQ Quality Press; 1997
56. US Dept of Health and Human Services, Public Health Service. Clinician's Handbook of Preventative Services: Put Prevention into Practice. Washington, DC: US Government Printing Office; 1994
57. Langley GJ, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco, CA: Jossey-Bass; 1996
58. Altov H. The Art of Inventing: And Suddenly the Inventor Appeared. Translated and adapted by Lee Shulyak. Worcester, MA: Technical Innovation Center; 1994
59. Leape LL Error in medicine. JAMA. 1994; 272:1851-1857 [Medline]
60. Nolan TW, Schall MW. Reducing Delays and Waiting Times Throughout the Healthcare System. Boston, MA: Institute for Healthcare Improvement; 1996
61. Weiss KB, Mendoza G, Schall MW, Berwick DM, Roessner J. Improving Asthma Care in Children and Adults. Boston, MA: Institute for Healthcare Improvement; 1997
62. Rainey TG, Kabcenell A, Berwick DM, Roessner J. Reducing Costs and Improving Outcomes in Adult Intensive Care. Boston, MA: Institute for Healthcare Improvement; 1996
63. Weiss et al., op. cit.
64. Rainey et al., op. cit.
65. Guyatt HG, Sackett DL, Sinclair JC, User's guide to the medical literature. IX. A method for grading health care recommendations. JAMA. 1995; 274:1800-1804 [Medline]
66. Berwick DM Continuous improvement as an ideal in health care. N Engl J Med. 1989; 320:53-56 [Medline]
67. Shewhart W. op. cit.
68. Plsek PE Introduction to control charts. Quality Management in Health Care 1992; 1:65-74
69. Carey RG, Lloyd RC. Measuring Quality Improvement in Healthcare: A Guide to Statistical Process Control Applications. New York, NY: Quality Resources; 1995
70. Deming WE. op. cit.
71. Berwick DM Continuous improvement as an ideal in health care. N Engl J Med. 1989; 320:53-56 [Medline]
72. Carey, Lloyd, op. cit.
73. Nelson EC, Splaine ME, Batalden PB, Plume SK Building measurement and data collection into medical practice. Ann Intern Med. 1998; 128:460-466 [Medline]
74. Carey, Lloyd, op. cit.
75. Gibson PG, Wlodarczyk J, Hensley MJ, Murree-Allen K, Olson LG, Saltos N Using quality-control analysis of peak expiratory flow recordings to guide therapy for asthma. Ann Intern Med. 1995; 123:488-492 [Medline]
76. Nelson EC, Splaine ME, Batalden PB, Plume SK Building measurement and data collection into medical practice. Ann Intern Med. 1998; 128:460-466 [Medline]
77. Oniki TA, Clemmer TP, Arthur LK, Linford LH. Using statistical quality control techniques to monitor blood glucose levels. Proc Ann Symp Comput Appl Med Care. 1995;586-590
78. Laffel G, Luttman R, Zimmerman S Using control charts to analyze serial patient-related data. Quality Management in Health Care. 1994; 3:70-77
79. Tucker FG, Zivan SM, Camp R How to measure yourself against the best. Harvard Bus Rev 1987; 65:8-10
80. Camp RC. Benchmarking: The Search for Industry Best Practices That Lead to Superior Performance. Milwaukee, WI: Quality Press; 1989
81. Gift RG, Mosel D. Benchmarking in Health Care: A Collaborative Approach. New York, NY: American Hospital Publishing; 1994
82. O'Connor GT, Plume SK, Wennberg JE Regional organization for outcomes research. Ann N Y Acad Sci 1993; 703:44-51 [Medline]
83. O'Connor GT, Plume SK, Olmstead EM, A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. JAMA 1996; 275:841-846 [Medline]
84. Kasper JF, Plume SK, O'Connor GT A methodology for QI in the coronary artery bypass grafting procedure involving comparative process analysis. Qual Rev Bull 1992; 18:129-133 [Medline]
85. Horbar JD, Rogowski J, Plsek PE, Collaborative quality improvement for neonatal intensive care. Pediatr Res 1998; 43:177A
86. Garvin DA Building a learning organization. Harvard Bus Rev 1993; 71:78-92
87. Patrick M, Alba T Health care benchmarking: a team approach. Quality Management in Health Care 1994; 2:38-47
88. ibid.
89. Porter JE The benchmarking effort for networking children's hospitals (BENCHmark). Jt Comm J Qual Improv. 1995; 21:395-406 [Medline]
90. Kennedy M Benchmarking clinical processes. Qual Lett Healthcare Leaders. 1994; 6:6
91. ibid.
92. Plsek PE Collaborating across organizational boundaries to improve the quality of care. Am J Infect Control 1997; 25:85-95 [Medline]
93. ibid
94. O'Connor GT, Plume SK, Olmstead EM, et al. op. cit.
95. Kilo C Improving care through collaboration. Pediatrics. 1999; 103(suppl):384-393
96. Horbar JD, Rogowski J, Plsek PE, et al. op. cit.
97. Horbar JD The Vermont-Oxford Network: evidence-based quality improvement for neonatology. Pediatrics. 1999; 103(suppl):350-359