Evidence-Based Guidelines and Critical Pathways for Quality Improvement

PRACTICE GUIDELINES AND THE IMPROVEMENT OF CARE

I will attempt to cut through this confusion by clearly stating that the primary goal of practice guidelines in pediatrics is to improve the health of infants and children. This is a premise that is readily acceptable to most pediatricians and clearly puts aside the notion that practice guidelines should be primarily used to reduce cost or protect against litigation. This is not to say that cost reduction and reduced malpractice litigation are not laudable goals—they are. There are, however, more effective ways of reducing costs in health care through programs designed to reduce waste and improve coordination of care in the treatment of illness.⁷ Similarly there are more effective ways to reduce litigation through such means as improved patient communication, better documentation, and root-cause analysis of adverse clinical events.⁸

If the primary purpose of practice guidelines is to improve health care, what role do they play in this process? First and foremost, we have to recognize that practice guidelines do not provide a complete model for improving health care outcomes. Numerous studies have shown that the development and dissemination of a practice guideline does not guarantee any improvement in the outcome of care.^9–11 We propose a simple model for improving health care: there exists for many health care conditions in this country a gap between what medical science has shown to be effective practice and what is actually done. Health care outcomes will improve for infants and children when current practice incorporates these more effective practices. The role of practice guidelines in this process is to provide a tool for reviewing, synthesizing, and communicating evidence-based practices to the pediatrician. Practice guidelines can then be defined as evidence-based recommendations for care. Practice guidelines should not be confused with critical pathway or care maps. These are implementation tools for the evidence-based recommendations found in practice guidelines. They provide a means of translating the recommended changes in care into a more detailed set of recommendations of who provides what care and when, as well as what resources should be used.

THE MEASUREMENT OF OUTCOMES AND PERFORMANCE

Because practice guidelines are used to improve outcomes of care, they are often associated with a set of measures that document this improvement. For example, the AAP hyperbilirubinemia guideline¹² is designed to reduce the complications of jaundice in the newborn, decrease rehospitalizations for hyperbilirubinemia, and to decrease the hospital length of stay and the cost of care. Each of these outcomes can be associated with one or more outcome measures that together define an outcome measurement set. The outcome measurement set comprises one of the most important features of a practice guideline.

Frequently a practice guideline will be found to vary in effectiveness between settings and groups of patients. One reason for this variability is that the guideline has not been modified to meet patient and site-specific needs. By collecting outcome data in their own setting, physicians are able to customize the guideline through the collection and feedback of this data on their own patients. For example, a practice guideline for the treatment of hyperbilirubinemia may need to have the thresholds for further assessment and treatment modified to address the ethnic makeup of the patient population and the resources available to the family. Customization and documentation of the guideline that can meet the needs of the targeted patient population are a critical step in successful implementation.

Our model for improvement presupposes that better clinical outcomes will occur when the recommended clinical practices are adopted. The development of performance measures in association with the practice guidelines provides a means to document and inform clinicians that these changes in practice have indeed occurred. A performance measure can be defined as an adherence rate to one or more key recommendations in the practice guideline. For example, a practice guideline that recommends antenatal corticosteroids for prevention of respiratory distress syndrome in infants may have as a performance measure the percentage of eligible mothers that receive corticosteroids. In summary, practice guidelines can be best understood in the context of an improvement model that ensures infants and children receive evidence-based care. They are an important and necessary tool for improvement. They are not, however, sufficient to achieve improvement. They are best used in association with a process that collects outcome information and a set of performance measures that provides feedback to clinicians about their adherence to the recommendations for care. This feedback then allows the clinician to modify the guidelines to more effectively meet the needs of the local patient population.

THE QUALITY OF EVIDENCE IN EVIDENCE-BASED GUIDELINES

Webster's Dictionary defines evidence as something that furnishes proof. This definition is subjective in interpretation and leaves a wide latitude as to what constitutes sufficient proof. This subjectivity is unfortunately reflected in the type and quality of evidence that supports practice guidelines. This level of proof ranges from evidence obtained from properly randomized controlled studies to opinions from respected authorities. In their review of the evidence for appropriateness of care guidelines, health services researchers at the RAND Corporation (Personal communication, Elizabeth M. Glynn, PhD) found that only 11% of the evidence from their reviews for pediatric conditions came from randomized clinical trials. Seventy-two percent of the evidence came from expert opinion, 10% from clinical panels, and 11% from nonrandomized clinical trials.

Even when high quality studies such as randomized controlled trials are available, there is still the subjective translation of the evidence into recommendations. Recommendations from these studies can vary in strength from good evidence to support or reject the recommendation to insufficient evidence to either support or reject the recommendation. There are also vexing issues involved in interpreting the evidence from the perspective of diverse patient groups and from the perspective patient and family. For example, studies done on an all white male population may not be valid for a group of patients that is more diverse in gender and ethnicity. Similarly, when a recommendation presents options for care the preferred option may depend on patient preference. For example, a practice guideline may conclude that there is not sufficient evidence to choose between a medical and surgical intervention. The AAP practice parameter for the diagnosis and treatment of otitis media with effusion offers the option of antibiotic treatment or surgical intervention after a defined period of watchful waiting.¹³ In this case, the best choice may depend on the preference of the family for one alternative. Perhaps the best effort to date to classify the quality of medical evidence and to rank order the strength of the recommendation has been done by the United States Preventative Services Task Force in their Guide to Clinical Preventive Services.¹⁴ This classification schema is presented in Table 1. Here the quality of evidence is ranked according to the study design with randomized controlled trials being the gold standard and expert opinion occupying the other end of the spectrum. The evidence is also classified as to whether or not the results are sufficiently compelling to lead to a recommendation to adopt or reject the intervention, or if the evidence cannot discriminate between options for care. Even this effort involves a considerable degree of subjective assessment. Despite these shortcomings, practice guideline developers must include statements about the quality of evidence and the strength of the recommendations. These qualifiers enable the individual clinician to decide where a recommendation requires 100% adherence or where a weaker recommendation can allow the choice of several options. Where options are available, decisions can be made in concert with patient preferences with attention to issues of diversity.

DO PRACTICE GUIDELINES IMPROVE HEALTH CARE OUTCOMES?

The evidence supporting the notion that practice guidelines improve health care outcomes has been mixed. Several investigators have shown that adherence to evidence-based guidelines leads to improvement in the quality of care provided.^15–18 Studies of the impact of critical pathway implementation in neonatology have shown mixed results. Wirtschafter et al¹⁶ was able to demonstrate that written clinical guidelines when used in conjunction with an educational program were able to improve clinical practice patterns and reduce variability in outcomes between institutions within a large health maintenance organization.¹⁰ Thompson and Maringer¹⁹ reported that the use of guidelines that impacted on nursing interventions were able to reduce the length of stay in the neonatal intensive care unit (NICU) by 8% and charges by 14% for selected infants. Several studies, however, have not been able to show a significant impact on health care outcomes. In one of the few controlled trials involving practice guideline intervention, Bergman showed that the implementation of a critical pathway for very low birth weight infants had no impact on clinical outcomes, practice patterns, cost, or utilization of services. Jones²⁰ studied the implementation of NeoMAP critical pathways in extreme premature infants and premature infants with major problems and was not able to demonstrate a statistically significant reduction in length of stay after the implementation. In summary, there is a dearth of well-designed studies in this area. Those studies that do exist are conflicting in their results, and frequently use a predesign–postdesign without a control group. Very few studies have measured the impact of the intervention on clinical outcomes, choosing instead to study adherence rates to the recommendation or the impact on cost and utilization.

EXAMPLES OF PRACTICE GUIDELINES IN PERINATOLOGY

Practice guidelines currently in use by perinatologists and neonatologists vary in scope, the quality of evidence, and the methodology used in their development. Some of the guidelines with the best evidence base involve recommendations for neonatal screening. Those conditions recommended by the United States Preventive Services Task Force are listed in Table 2.¹⁴ In general, these guidelines are substantiated by a strong evidence base. The baseline prevalence of the condition and the expected utility of screen-no screen strategies have been used to develop a cost-effectiveness analysis to substantiate the recommendation. Of interest is that some of the screening guidelines for which there is a strong evidence base have not been universally mandated (eg, hypothyroidism), whereas for others there is considerable controversy concerning the results of a cost-effectiveness analysis (eg, cystic fibrosis) that have been mandated by some states.²¹ For some conditions (eg, HIV) screening has been mandated but incompletely implemented.

Some of the practice guidelines for antenatal and/or neonatal therapy that are currently in use are listed in Table 3. In general, these guidelines have a strong evidence base but vary considerably in the degree that they have been implemented. For some conditions such as antenatal steroid treatment, the work of Horbar¹¹ and others have demonstrated that a significant proportion of mothers and fetuses do not receive this effective treatment. For other conditions such as maternal and neonatal group B streptococcus infection detection, there is considerable variability in the construction and interpretation of the cost-effectiveness analyses and consequential variation in the type and strength of the recommendations.^22,,23 There have also been guidelines that are concerned with appropriate usage of resources. These guidelines have addressed maternal and infant stay after childbirth, and length of stay and usage of resources for specific pediatric disease-related groups such as extreme prematurity with and without complications. There have been numerous anecdotal reports from concerned families and physicians about the safety of early discharge of infants and mothers, but little evidence from controlled trials to support the notion that early discharge is an unsafe practice. The evidence supporting the effectiveness of critical pathways that address utilization has been conflicting at best.^24–26 Although it has been demonstrated that for some of the NICUs studied these interventions can reduce utilization,¹² there is little evidence, either for or against, that these pathways significantly impact clinical outcomes or the family experience of care. When considered in the aggregate, there are probably more clinical guidelines in active use in perinatology and neonatology than any other area in pediatrics. Yet this group of guidelines varies considerably in the quality of the scientific evidence supporting the recommendations. This is somewhat surprising because the evidence base for perinatology and neonatology is probably greater than in any other area of pediatrics. Moreover, this evidence has been better reviewed and summarized than most other areas.²⁷ With this comprehensive and easily accessible evidence base, clear opportunities exist to define and disseminate new practice guidelines for the care of mothers and newborns.

MAKING PRACTICE GUIDELINES WORK

As we noted earlier, practice guidelines in and of themselves do not provide a means to improve health care outcomes. To achieve a favorable impact on health outcomes, guidelines need to be disseminated and accepted by clinicians and patients alike. For this to occur, data need to be collected on performance and outcomes and used to modify the guideline to meet the specific needs of different patient populations. Dissemination of new knowledge to the level of the individual practitioner often takes years. The seminal work of Everett Rogers²⁸ done more than 50 years ago showed that the time it took for new technology to be disseminated to the everyday user was 6 to 10 years. Working with the dissemination of new agricultural technologies, he was able to show that this process followed an S-shaped curve with adoption taking place slowly during the first and third periods of the process and rapid dissemination occurring during the middle third period. The duration and shape of the process have been remarkably consistent between different groups and different technologies. Everett's work also provide insights into possible strategies to shorten this time interval. Through his research, he has defined the key groups that play an important role in the innovation process. The initiating groups in the dissemination process are the innovators. These are the individuals who create the technology and do the early testing. The next group is the early adopters. These individuals look outside their group or institution to innovators for new and innovative practices. They tend to be risk takers who have the opportunity to be exposed to new ideas from outside the group or organization. The early adopters are responsible for convincing the early majority within their group or institution that this new practice or technology is a good idea worth trying. The early majority are also risk takers willing to try new ideas, but they differ from early adopters in that they look within rather than outside their organizations for these ideas. More widespread adoption by the early majority convinces the next group, the late majority, to try the new practice. The late majority serves the important function of resisting too rapid adoption of new practices before they have been thoroughly tested and evaluated. The final group is the nonresponders. These individuals resist change and will only adopt new ideas when they are mandated. Considerable resources can be wasted trying to convince these individuals to change. The steep part of the S-curve, or the period of rapid change, occurs once the key leaders in the early majority try the new practice. When the early majority has adopted the practice, the dissemination process becomes self-sustaining.

This research has many important implications for the implementation of practice guidelines. These results tell us that the presentation of practice guidelines to a group of physicians without first working with the early adopters and the early majority is much more likely to fail. This is true even if the guideline's recommendations are evidence based. Some of the early majority and most of the late majority will not dispute the validity of the evidence, but will believe that the recommendations do not directly apply to their patients or will not work effectively in their setting. These concerns travel quickly through an organization and can potentially poison the well for future change. Successful guideline implementation is best accomplished by first identifying and working with the early adopters to pilot the changes and using the preliminary data to modify and refine the intervention. The second step is to insure these changes are implemented in a manner that is very visible to the rest of the organization. This allows the early majority to evaluate the impact of the changes and incorporate them into their practice. This can be facilitated by having key physicians in the early majority act as observers for the pilot physicians. Once the changes have begun in the early majority, the dissemination process is on the steep part of the curve and is usually self-sustaining at this point.

IMPLEMENTING PRACTICE GUIDELINES IN A NICU OR WELL-BABY NURSERY

As we saw from the slow adoption of antenatal steroids to prevent respiratory distress syndrome, dissemination and adoption of new knowledge in neonatology can often take years. This long delay occurs despite strong scientific support behind the recommendations. This delayed implementation takes place at two levels: first, between NICUs with respect to how rapidly they adopt new guidelines; and second, between neonatologists within a given NICU. One answer to the first challenge of how to more rapidly implement practice guidelines between NICUs has been the development of networks or collaboratives. The Vermont Oxford Network (VON) is an example of a group of neonatologists and NICUs that came together to develop common processes for the measurement and improvement of health care outcomes.²⁹ VON engaged 10 NICUs to participate in a quality improvement effort designed to improve practices and outcomes in the rate of nosocomial infections in the NICU and the incidence of chronic lung disease in premature infants. These units developed a common set of measures to document improvement in outcomes and used a combination of evidence-based practices from the medical literature and their own benchmarking process to change practices. They were ultimately able to show significant improvement for both of their target conditions. In the context of dissemination theory, these 10 practices could be considered early adopters or part of the early majority. The challenge for VON will be to see if they can now rapidly disseminate these changes through the rest of the network and ultimately through NICUs nationwide. What the VON experience has demonstrated is that collaboratives or networks can provide an important setting for early adoption and refinement of new interventions and their eventual rapid dissemination.

The adoption and dissemination of a practice guideline within a NICU presents a similar yet distinctive challenge. Even within a small group of neonatologists, there are likely to be early adopters and representatives of the early and late majority. Adding to this complexity are groups of nurses, respiratory care practitioners, and other ancillary personnel who cannot be expected to follow in lock-step the directions of the physicians, and who represent the same spectrum from early adopters to late majority. The first step to effective implementation in such a setting is the formation of a multidisciplinary team that involves the key stakeholder in the clinical process. For example, a team targeting the implementation of the guidelines for the use of antenatal steroids must include neonatologists, perinatologists, neonatal and obstetric nurses, as well as other involved personnel. Depending on the guideline, it may be useful to involve the patient or family at some point in the process. A focus group of new parents may highlight the barrier to implementing guidelines in your institution. For example, parents may have difficulty in making an appointment with the primary care physician after early discharge from the nursery because no appointments are available. Once the team is formed, the next step is to agree on a common set of aims or outcomes for the improvement effort and a set of measures that will let you know that you have achieved the aim. Each of these outcomes can be associated with one or more outcome measures that together define an outcome measurement set. A portion of a clinical pathway for the care of extremely low birth weight infants is presented in Fig 1. The outcome measures associated with this project included: discharge disposition; achievement of certain clinical milestones such as time to reaching a stable state on feeds and getting off inotropes; complication rates for sepsis, pneumothorax, and pulmonary interstitial emphysema; mortality rates; and resource use as measured by length of stay and hospital charges. The outcome and performance measurement set comprise one of the most important features of a practice guideline.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 1.

Extreme prematurity clinical pathway.

The goal here is to be very specific in your aims, and realize that you may need to divide your initial aim into smaller and more specific aims as the project progresses. For example, your initial aim may be to have 100% of eligible mothers receive antenatal steroids. However, you may realize before you can reach this aim you need to solve the problem of how to get 100% of eligible mothers identified. A useful tool in this process is the critical pathway. A critical pathway is a matrix with time or phase of care along one axis and the type of care and/or accountable group of individuals along the other axis. Each cell is filled with the recommended practice. An example of a portion of a critical pathway for the care of extremely low birth weight infants is shown in the Figure. Critical pathways allow for the definition of all the components of care and the accountable groups or individuals. More importantly it can breakdown a large aim into a set of smaller aims directed at what needs to be achieved before the larger aim can be realized.

The next step is the definition of a set of measures that enables you to know that you have achieved your aims. If we look again at the clinical path presented in the Figure, the performance measures are the adherence rates to key recommendations found in the cells of the matrix such as the percentage of infants having their blood urea nitrogen monitored every week during the acute unstable phase or the percentage of infants receiving an eye examination during the maintenance phase. The key to this step is to keep your measurement set as parsimonious as possible and easy to collect. The propensity of many physicians is to look on measurement as a one time opportunity to gather as much data as possible, create large computerized data bases, and analyze them using sophisticated statistical packages. Increasing the size and complexity of the data collection and analysis process can create a significant impediment to rapid, iterative data collection and process revision cycles. If possible keep the measurement set small and collectible with paper and pencil technology. For most improvement projects, entering the data on a standard spreadsheet program will provide sufficient analytic power.

The next step for the improvement team is the assessment and possible revision of the recommended change. Even if the recommendation is supported by data from properly controlled randomized trials, there may be real concerns in translating these changes in practice to your patient population and to your practice setting. In most cases the practice guideline will need to be customized to meet these site-specific needs. This is best accomplished through a series of small, iterative pilot studies of the intervention. This allows for problems to be identified and solved and the guidelines revised before the next pilot. This rapid cycle quality improvement model has been described by Langley et al³⁰ and has been used by the Institute for Health Care Improvement^31,,32 to bring about improvement in asthma care and reductions in cesarean section rates. A discussion of the specific aims, measures, and improvement cycles of a project to reduce medication errors in a NICU can be found in their breakthrough series report on reducing medication errors.³³ An added benefit of using rapid cycle quality improvement is that you can use your early adopters and early majority physicians to conduct the pilots. If these pilot studies are widely visible and the data shared with members of the health care team, it will increase the sense of ownership of the interventions and will decrease the time needed to effectively implement the guideline.

Once a guideline has been successfully implemented in a NICU there is still the concern that these recommended practices will become codified and as intractable to change as the previous practices. It is important that the quality improvement team stay together to continue to monitor practices and outcomes. It is also important that new innovations, no matter how small, be encouraged and tested. Some medical centers have developed an internal granting program for innovations in patient care that have spawned the development of new, cost-effective patient care practices.³⁴ One of the greatest resources that a tertiary care nursery possesses is the collective knowledge and creativity of the staff. Yet this is a resource that is often squandered. Creativity and innovation do not magically occur. They are processes like other processes that need to be supported and facilitated. Current work by Plsek³⁵ and others offer an excellent framework for facilitating creativity in your organization.