Background. The increasing complexity of medical care and a desire to increase quality and control costs have led to growing use of clinical practice guidelines (CPGs). It is unclear how helpful these guidelines are to the practitioners expected to use them. We surveyed pediatricians about their knowledge and impressions of four well-publicized CPGs: the American Academy of Pediatrics' “Practice Parameter for Hyperbilirubinemia in Newborns” (hyperbilirubinemia), “A Guideline for the Management of Febrile Infants” (fever), the Agency for Health Care Policy and Research's “Guideline for Otitis Media With Effusion” (otitis), and the US Preventive Services Task Force Guide to Clinical Preventive Services (preventive care).
Objectives. 1) What percentage of practicing pediatricians are aware of these guidelines? 2) How helpful do they find them? 3) What are practitioners' perceived limitations of these guidelines? 4) Have these guidelines affected provider behavior? 5) Are there features of a provider's training or practice that are associated with changing practice as a result of guidelines?
Design. A national survey of 600 pediatricians selected at random from the American Medical Association master file.
Results. A total of 300 of 555 eligible participants (54%) returned surveys. Of the respondents, 66% were aware of the hyperbilirubinemia guideline, 64% of the fever guideline, 50% of the otitis guideline, but only 16% knew of the preventive care guidelines. Mean helpfulness scores (1 to 10 scale, where 1 = “not at all helpful” and 10 = “extremely helpful”) ranged from 3.67 to 6.67 for the different guidelines. In terms of limitations, 15% to 33% of respondents reported that CPGs were “too cookbook,” 6% to 19% reported that they were “too time-consuming,” and 4% to 16% reported that they were “too cumbersome.” Additional reported limitations were believing that a guideline left no room for personal experience and judgment, concern of increased liability risk, and poor parental acceptance of CPG recommendations. The proportions reporting change in management as a result of a CPG were 28% for the hyperbilirubinemia guideline, 36% for the fever guideline, 19% for the preventive care guidelines, and 28% for the otitis guideline. Mean helpfulness scores reported by nonuniversity-affiliated physicians were significantly higher than those reported by university-affiliated physicians. In a regression model of respondents aware of a particular guideline, more recent graduation from medical school and increased helpfulness scores were associated with guideline-related behavior change.
Conclusion. In their present form, CPGs are not perceived as very helpful by most practitioners. More recent medical school graduates and nonuniversity-affiliated physicians are more likely to find them helpful and more likely to change their behavior because of them.
Clinical practice guidelines (CPGs) are being promulgated avidly. Spawned in part by the increased complexity of medical care and by the desire to improve quality and control costs, as many as 1500 guidelines have been issued in the United States alone.1 Many of these guidelines have pertained to internal medicine, although increasingly they are being devised for pediatric conditions. What benefits these guidelines offer to busy clinicians who are to use them are not well understood. Some studies have documented poor knowledge of and compliance with guideline recommendations,2-8 but few studies have analyzed what practicing clinicians think of existing guidelines and what they perceive as their limitations.9-11 If guidelines are to serve their putative purpose of helping clinicians improve patient care through the use of evidence-based medicine, then practicing clinicians' impressions of guidelines should be elicited and addressed.
This study was designed to solicit pediatricians' feedback on four CPGs: the Agency for Health Care Policy and Research (AHCPR) “Guideline for Managing Otitis Media With Effusion in Young Children” (otitis),12 the US Preventive Services Task Force (USPSTF) Guide to Clinical Preventive Services(preventive care),13 the American Academy of Pediatrics (AAP) “Practice Parameter for Hyperbilirubinemia in the Healthy Term Newborn” (hyperbilirubinemia),14 and the “Practice Guideline for the Management of Infants 0–36 months With Fever Without a Source” (fever).13 These guidelines were selected for three reasons. First, they deal with important, common pediatric issues. Second, they were well-publicized and distributed and have been excerpted and referenced in various forms in many journals, thereby increasing the likelihood that clinicians would be aware of them. Although not as widely publicized as the others, the Guide to Clinical Preventive Services was included because it represents an enormous effort and serves as a comprehensive reference on guidelines for preventive care. Third, all of the guidelines we studied used evidence-based criteria to varying degrees to develop their recommendations.
We had four primary questions: 1) What percentage of practicing pediatricians are aware of these guidelines?; 2) How helpful do they find them?; 3) What are practitioners' perceived limitations of these guidelines?; and 4) Have these guidelines affected provider behavior?
In addition, we had two secondary questions: 1) Are there features of a provider's training or practice that are associated with changing behavior as a result of guideline recommendations?; and 2) How might CPGs be improved?
We recruited our sample from the American Medical Association's (AMA) master list of licensed physicians in the United States. This database is not limited to AMA members and is acknowledged to be the most complete repository of physicians' names, addresses, and specialties. Medical Marketing Services is a private vendor that dispenses names from this database under an agreement sanctioned by the AMA. A total of 600 pediatricians practicing medicine in the United States were selected randomly for participation. In addition to names and addresses, Medical Marketing Services provided information on board certification, gender, and age.
Participants were sent a four page questionnaire, requiring ∼10 minutes to complete, and a prepaid return envelope. As an inducement to participate, 1 dollar was attached to each survey in the first mailing. Physicians were assured that participation was voluntary and that their responses would be confidential. Those who had not responded within 3 weeks were sent the survey again.
The study was approved by the University of Washington institutional review board.
We asked for demographic data including practice type (staff-model HMO, multidisciplinary group, pediatric group, solo practice, university-based); practice location (rural, urban, suburban); and year of graduation from medical school. In addition, we asked what percentage of respondents' time was devoted to clinical duties and whether any of their practice locations had explicit CPGs.
All participants were asked whether they were aware of the four guidelines under study. Those who knew of a particular guideline were asked how helpful they found it to be and what they perceived as its limitations. For reporting on how helpful providers found a particular guideline, participants were asked to rate its helpfulness on a 10-point scale, with 1 being “not at all helpful” and 10 being “extremely helpful.” This response constituted a helpfulness score for each guideline. Several options for limitations were given for all guidelines including “too cookbook,” “too difficult to access,” “too confusing,” “not applicable to my patients,” “too time-consuming to apply,” “too cumbersome,” and “don't believe in guidelines.” These options were derived from informal conversations with providers and a review of the literature regarding barriers to guideline implementation. In addition to these a priori options, an “other” option was provided, and participants were invited to submit their own perceived limitations with a given guideline. Finally, respondents were asked whether each guideline had changed their management of the particular condition. At the survey's conclusion, there was an open-ended section for comments.
χ2 Analysis was used for comparing categorical variables. Student's t test was used to compare means on continuous variables. Multivariate logistic regression was used to model dichotomous dependent variables. SPSS 6.1 (SPSS, Chicago, IL) for the Macintosh computer was used for all analyses.
Of 600 surveys distributed in the initial mailing, 33 were returned with no forwarding address, 8 were returned by people who reported that they do not provide medical care for children, 3 were returned by retired physicians, and 1 was not completed because the provider was out of the country. Of the 555 eligible participants, 148 responded to the first mailing and 152 responded to the second. Three hundred surveys were returned (response rate, 55%). Because of missing data, not all totals equal 300. Also, because providers were allowed to select more than one practice type and location, these totals exceed 300. Respondents were not different from nonrespondents according to age, board certification, or gender (the three demographic variables provided by Medical Marketing Services). Similarly, respondents to the first mailing did not differ significantly from respondents to the second mailing on these three characteristics.
Demographic information on survey recipients and their practices is summarized in Table 1. Respondents had a mean age of 48 years. Sixty percent were male, and 26% percent reported that they had some subspecialty training. On average, the respondents spent 88% of their time on clinical duties (range, 0% to 100%). Thirty-three percent of respondents reported that some of their practice sites made use of explicit CPGs.
Awareness of Guidelines
Fifty percent of respondents knew of the AHCPR otitis media with effusion guideline; 66%, the AAP practice parameter for hyperbilirubinemia in the newborn; 64%, the practice guideline for management of infants and children with fever without source; but only 16% knew of the USPSTF Guide to Clinical Preventive Services. Because we were concerned that respondents might overreport their awareness of guidelines, we included a misrepresentation index by asking whether participants had heard of the “Cochrane collaboration.” We expected awareness of this effort to be quite low, and in fact it was. Only 2% of respondents reported that they knew of the Cochrane collaboration, a plausibly low number suggesting honesty in responses.
Mean helpfulness scores varied from 3.74 to 6.10. The AHCPR otitis media with effusion guideline and the Guide to Clinical Preventive Services were judged significantly less helpful than both the fever and the hyperbilirubinemia guidelines (P < .01 for each comparison). The mean helpfulness scores for nonuniversity-based physicians were higher than were university-based physicians for the otitis, hyperbilirubinemia, and fever guidelines by 1 to 2.5 points (P < .01 for each comparison). The helpfulness scores for all guidelines are shown in the Figure.
Of respondents who were familiar with a given guideline, 28% reported changing their management as a result of the otitis guideline, 19% as a result of the preventive care guidelines, 28% as a result of the hyperbilirubinemia guideline, and 36% as a result of the fever guideline. The denominator of these proportions is limited to those that knew of a guideline and hence overestimates overall guideline-related behavior change. Of all survey respondents, only 14% reported change in management as a result of the otitis guideline, 3% as a result of the preventive care guideline, 18% as a result of the hyperbilirubinemia guideline, and 23% as a result of the fever guideline.
Perceived Problems With Guidelines
There was a wide variety of perceived problems with the guidelines. The most common limitation reported by 15% to 33% of respondents was that a given guideline was “too cookbook” (Table2). Between 15% and 23% of respondents filled in the “other” box. Although there was no consistency in these written responses, problems cited included being familiar with the specified condition and not needing additional help, the fact that the guideline left no room for personal experience and judgment, that a guideline's existence increased their liability risk, and that parents would not accept guideline recommendations. In addition, 15% of respondents wrote some general comments at the survey's conclusion.
Using logistic regression, we sought to determine what variables were associated with guideline-related behavioral change. Clinical time, board certification, subspecialization, and university-based practice did not affect whether the guidelines resulted in behavior change. Increased helpfulness scores were strongly associated with reported behavior change for all guidelines. More recent graduation also was associated with guideline-related behavior change for both the hyperbilirubinemia and the otitis media with effusion guidelines. These results are summarized in Table 3.
The majority of respondents were aware of three of the four guidelines we studied (otitis, fever, and hyperbilirubinemia guidelines). However, only a minority were aware of the USPSTFGuide to Clinical Preventive Services. Why the Guide to Clinical Preventive Services fared considerably worse than the others is unknown. Perhaps this disparity is because much of the content of the Guide to Clinical Preventive Services does not pertain to the care of children. Alternatively, it may be because the preventive care guidelines were not published or referenced in the pediatric literature to the same extent as the others. Indeed, all of the guidelines could have benefited from improved dissemination efforts because even the best known, the AAP's hyperbilirubinemia guideline, left 34% of practitioners untouched. How to reach these practitioners is unclear.
That none of the four guidelines we studied were perceived as very helpful is disconcerting, considering the effort and money devoted to their development. How cost-effective guideline development and dissemination are remains difficult to assess. The costs of creating guidelines are themselves elusive because a considerable amount of time is frequently donated by guideline committee members. Even when costs are reasonably well accounted for, as was the case with the creation and dissemination of the AHCPR otitis media with effusion guideline (estimated to have cost ∼$1.3 million),15 these costs must be put into proper perspective. If CPGs lead to more effective medical care, for example by significantly reducing the number of unnecessary tympanostomy tubes placed, they could prove to be cost-saving. In our study, nonuniversity-based providers and more recent graduates of medical school were more likely to find guidelines helpful, and finding them helpful was, in turn, associated with reported behavior change. What the final balance sheet is for a given CPG is an important and underaddressed issue.
That more recent graduates were more likely to report changing their behavior as a result of guidelines may reflect the fact that more recent trainees are in greater need of external guidance than more seasoned practitioners. Still, one other theory must be considered. It could be that recent trainees are more receptive to the theory underpinning guideline construction and implementation. That is, they might put greater stock in the recommendations that come from an evidence-based approach to medical care, or they may be more willing to codify medical management. Whether more recent graduates will inexorably come to view guidelines skeptically and with recalcitrance requires a longitudinal study. Nevertheless, if guidelines are here to stay and if they represent an opportunity to compile and distill evidence for certain appropriately selected conditions, then we must train practitioners to evaluate them and use them during their early years of training.16-18 This may not be happening at present because university-based practitioners were less likely to find guidelines helpful (and one might infer) less likely to model use appropriately.
Multiple respondents commented on the potential adverse legal ramifications of CPGs. Specifically, some providers were concerned that guidelines could be used against them during malpractice litigation. As one respondent stated, “These guidelines tend to push physicians into a corner when it comes to potential malpractice situations, as we are forced to choose between what we believe is best versus safest medical practice.” Their concerns are not without some merit. Although CPGs have been touted as having the potential to decrease providers' liability, there is mounting evidence that they are being used disproportionately to bolster plaintiffs' cases.19-23Issuers of guidelines should be aware of this possibly valid perception by physicians.
The limitation to the guidelines reported most commonly was their promotion of cookbook medicine. This finding is not surprising to anyone who has discussed guidelines with clinicians, but its implications warrant additional consideration. That cookbook medicine might pose an obstacle to guidelines' use suggests (to extend the metaphor) that physicians view themselves as master chefs—culinary artists who improvise recipes with each trip to the clinical kitchen. The existence of a recipe is perceived as threatening their artistry, as one respondent explicitly stated: “Medicine is 60% art, 40% science. Make it cookbook and then you don't need me.” It is not the intention of guidelines to eliminate the art of medicine. In fact, mindful of the importance of provider discretion, these guidelines, more than any cookbook we have ever seen, make explicit mention of the need for individual physician input and assessment. For example, the hyperbilirubinemia practice parameter repeatedly calls on providers to make subjective clinical assessments (eg, “if patient exhibits signs of lethargy, apnea, poor feeding”). What constitutes lethargy is subject to the interpretation of the treating physician. In a similar manner, the febrile infant guideline has as its first branch point the clinician's subjective assessment of the infant's appearance. The role for the art of medicine in assessing toxicity, for example, is made explicit. Nevertheless, some health care organizations are using conformity with guidelines as a performance measure for their providers, which does impinge on physician's autonomous decision-making and may represent misuse of the guidelines themselves.
It may be that what is perceived as cookbook for some providers is the flow charts that many guidelines (these included) use. Paradoxically, these algorithms are intended to help busy clinicians by presenting the gist of the guideline in a readily accessible way. Although they too make explicit mention of the role for provider assessments, their format may be perceived as demeaning inasmuch as the chart itself suggests concreteness. Although relatively few respondents (4% to 8%) reported that they outright “did not believe in guidelines,” several volunteered that they prefer review articles to guidelines, presumably because they allow their readers to draw their own conclusions, modulating the information conveyed with their own experience. This suggests that attempts to circumvent the process barriers to guideline implementation by making them easier to comprehend or use24-26 may be at odds with attempts to make some physicians more receptive to guidelines. Diversity of practitioners' preferences for guideline presentations is to be expected, rendering a one-look-suits-all approach unlikely to be effective. Some clinicians, for example, might prefer evidence tables to recommendations or algorithms. Alternative or perhaps multiple formats for guideline recommendations might be in order.
This study had several limitations. First, as with any survey study, there is always the question of whether and how respondents differ from nonrespondents. In our case, although no significant difference was found between the two with respect to age, board certification, and gender, other, more subtle differences may exist. Specifically, the two may differ with respect to their awareness of guidelines; respondents may be more likely than nonrespondents to be aware of them. Nonresponse bias is a persistent limitation of survey studies, although our response rate of 55% is in line with most other published studies involving physicians.27 Second, providers may already practice in accordance with guidelines and therefore not find them to be helpful. This study was not designed to assess conformity with published guidelines. Others have demonstrated that physicians frequently fail to comply with them and may claim to comply even when they do not.2,4 We were interested in discerning awareness of guidelines, the first step in the knowledge to adherence model,28 and whether guidelines are deemed helpful because, according to the Institute of Medicine's definition of CPGs, this is one of their intended purposes.29
In conclusion, CPGs continue to be viewed with some skepticism by providers. Although some clinicians do find them useful and report behavior changes in response to them, many do not. If guidelines are here to stay, then what is needed is a cultural paradigm shift, a movement away from cookbookness, artistry, and expert opinion. Increasing the extent to which guidelines are perceived as helpful may not require changing their content per se. It may be that the presentation of the evidence or the phrasing of the recommendations is the problem. Guidelines may need to be supplemented or even replaced with concise evidence summaries from which clinicians can easily glean information applicable to their patients. To do so, clinicians need to value, process, and ultimately rely on evidence in their day-to-day practice.
This research was supported by a grant from the Group Health Cooperative Foundation to the Consortium for Child Health Research.
We are grateful to Nicholas Christakis, MD, MPH, PhD; Richard Deyo, MD, MPH; Robert Merrill, MD; and Danielle Zerr, MD for their thoughtful readings of early drafts of this manuscript. Bi-Lan Chiong's administrative assistance was invaluable.
- Received July 11, 1997.
- Accepted September 9, 1997.
Reprint requests to (D.A.C.) UWRWJ Clinical Scholars Program, H-220A Health Sciences Center, Box 357183, Seattle, WA 98195-7183.
Dimitri Christakis was a Robert Wood Johnson Clinical Scholar while this research was conducted.
The opinions expressed here are not necessarily those of the Robert Wood Johnson Foundation.
- CPG =
- clinical practice guideline •
- AHCPR =
- Agency for Health Care Policy and Research •
- USPSTF =
- US Preventive Services Task Force •
- AAP =
- American Academy of Pediatrics •
- AMA =
- American Medical Association
- ↵Smith E. Variations in clinical guidelines. Ann Emerg Med. 1993;22:1196–1197. Editorial; Comment
- Young PC
- Oxman AD,
- Thomson MA,
- Davis DA,
- Haynes RB
- Buss PW,
- Evans RJ,
- McCarthy G,
- Scorrer T,
- Kumar V
- ↵Gupta L, Ward JE, Hayward RS. Clinical practice guidelines in general practice: a national survey of recall, attitudes and impact. Med J Aust. 1997;166:69–72. See comments
- Tunis SR,
- Hayward RSA,
- Wilson MC,
- et al.
- ↵Agency for Health Care Policy and Research. Managing Otitis Media With Effusion in Young Children. Rockville, MD: Public Health Service, US Dept of Health and Human Services; 1994; AHCPR publ no 94-0623
- ↵United States Preventive Services Task Force. Guide to Clinical Preventive Services. Alexandria, VA: International Medical Publishing; 1995
- American Academy of Pediatrics, Provisional Committee for Quality Improvement and Subcommittee on Hyperbilirubinemia
- ↵Sebring R. Personal communication; August 1997
- Brennan T
- Hyams AL,
- Shapiro DW,
- Brennan TA
- Daly M
- Kluge EH
- Forrest D,
- Hoskins A,
- Hussey R
- Asch DA,
- Jedrziewski MN,
- Christakis NA
- ↵Field MJ, Lohr KN. Clinical Practice Guidelines. Directions for a new agency. Washington, DC: National Academy Press; 1990
- Copyright © 1998 American Academy of Pediatrics