 |
|
 |
|
 |
 |
|
 |
 |
 |
 |
 |
 |
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
SPECIAL ARTICLE |
* Department of Pediatrics, Emma Children's Hospital, Academic Medical Center, University of Amsterdam, Amsterdam, Netherlands
Department of Pediatrics, Erasmus Medical Center, Sophia Children's Hospital, Rotterdam, Netherlands
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODSRESULTSDISCUSSIONCONCLUSIONSAPPENDIX 1.APPENDIX 2.REFERENCES |
|---|
Methods. We searched Medline, Embase, and relevant Web sites of guideline development programs and national pediatric societies to identify evidence-based pediatric guidelines. A list with titles of identified guidelines was sent to 51 leading pediatricians in the Netherlands, who were asked to select the 5 most urgent topics for guideline development. Three pediatrician reviewers appraised the available guidelines on the 10 most frequently mentioned topics with the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument.
Results. A total of 215 evidence-based pediatric guidelines were identified; of these, 17 guidelines on the 10 most frequently mentioned topics were appraised. The AGREE instrument rates guidelines among 6 domains. For the scope and purpose domain, the mean score was 84% of the maximal mark. For stakeholder involvement, the mean score was 42%, with 12 guidelines (71%) scoring <50%. For rigor of development, the mean score was 54%, with 5 guidelines (29%) scoring <50%. For clarity and presentation, the mean score was 78%, with 4 guidelines (24%) scoring <50%. For applicability and editorial independence, performance was poor, with mean scores of 19% and 40%, respectively. Low scores were partly attributable to poor reporting. After considering all domain scores, the reviewers recommended 14 of 17 guidelines (82%) to be used in local practice.
Conclusions. The current volume of pediatric guidelines categorized as evidence based in popular databases is large. Overall, these guidelines scored well, compared with other studies on guideline quality in fields outside pediatrics, when assessed for quality with the AGREE instrument. This holds especially for guidelines published or endorsed by the American Academy of Pediatrics or registered in the National Guideline Clearinghouse.
Key Words: practice guidelines • evidence-based medicine • quality of health care • pediatric
Abbreviations: AGREE, Appraisal of Guidelines, Research, and Evaluation • AAP, American Academy of Pediatrics • SIGN, Scottish Intercollegiate Guidelines • CMA, Canadian Medical Association • NGC, National Guideline Clearinghouse • ICC, intraclass correlation coefficient
Clinical practice guidelines are systematically developed statements to assist practitioners and patients in making decisions about appropriate health care in specific clinical circumstances.1 For many health care conditions, a gap exists between what medical science has shown to be effective practice and what is actually done.2 The primary goal of practice guidelines in pediatrics is to improve the health of infants and children by ensuring that they receive up-to-date, evidence-based care. Practice guidelines represent one of the various tools that can be used to improve the quality of care.3 Several studies have shown that adherence to evidence-based guidelines leads to improvement in the quality of care provided.4–10
In recent decades, the number of available clinical practice guidelines has grown enormously. It is estimated that 
2500 guidelines are already in existence. This recent increase in the production of clinical practice guidelines has been accompanied by growing concern about the variations in guideline recommendations and quality. In fact, several studies suggested that many existing guidelines are of poor quality.11–15 As the numbers of published guidelines increase, there have been calls for the establishment of internationally recognized standards to improve the development and reporting of clinical guidelines. For this purpose, an international group of researchers from 13 countries, the Appraisal of Guidelines, Research, and Evaluation (AGREE) Collaboration, has developed and validated a generic instrument that can be used to assess the quality of clinical guidelines.16
The number of published pediatric guidelines available to pediatricians is also increasing rapidly. However, their quality has not been assessed systematically. We set out to measure the current volume of potentially high-quality, published, pediatric guidelines, to assess their quality with the AGREE instrument, and to see whether we could adjust them for local use (to avoid potential duplication of effort within our Dutch national guideline development program). We limited our search to evidence-based guidelines, to enhance the yield of high-quality guidelines. We asked the following 2 questions. What evidence-based pediatric guidelines exist currently? What is the quality of these guidelines?
|
METHODS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSAPPENDIX 1.APPENDIX 2.REFERENCES |
|---|
Inclusion of Guidelines
Included were practice guidelines that concerned the management of diseases among children and neonates. Not all agencies that produce guidelines focus solely on the pediatric age group and, to be included in our sample, guidelines were required to make specific recommendations for children and/or neonates. Excluded were guidelines on prevention, screening, prenatal diagnoses, psychiatry, surgery, general practice, and very rare disorders. One of our aims was to appraise critically a sample of the retrieved guidelines, to see whether we could adopt them for use in the Netherlands. We sent the list of evidence-based guidelines that had been identified through the literature search to all directors of pediatric training programs in both academic teaching hospitals (n = 10) and affiliated teaching hospitals in community practice settings (n = 22) and to the chairpersons of all subspecialty working groups of the Dutch Board of Pediatricians in the Netherlands (n = 19), and we asked them to select the 5 most urgent topics for national pediatric guideline development.
Appraisal of Guidelines With the AGREE Instrument
We appraised the evidence-based guidelines published up to 2002 that were identified through the literature search for the 10 topics mentioned most frequently by 51 Dutch leaders in pediatrics, with 3 separate reviewers (N.B., C.R.L., and M.O.) using the AGREE instrument to evaluate the scientific quality of the selected guidelines. We collected all documentation related to each guideline available in the public domain. Before we started to appraise the guidelines, we appraised a practice set to reach consensus about the interpretation of all items. The AGREE instrument is an international, methodologically rigorously developed, validated instrument.16 It contains 6 domains, with a total of 23 items, and allows for the assessment of several components that are integral to guideline development, as follows: (1) scope and purpose (3 items), (2) stakeholder involvement (4 items), (3) rigor of development (7 items), (4) clarity and presentation (4 items), (5) applicability (3 items), and (6) editorial independence (2 items) (Appendix 1). The score for each domain is obtained by summing all of the scores of the individual items in a domain and then standardizing as follows: (obtained score – minimal possible score)/(maximal possible score – minimal possible score). The maximal possible score for each domain would be the number of questions multiplied by the number of reviewers times 4 (ie, the score for "strongly agree"). The minimal possible score for each domain would be the number of questions multiplied by the number of reviewers times 1 (ie, the score for "strongly disagree").
The final item of the AGREE instrument involves a recommendation regarding the use of the guideline in practice, as "strongly recommended," "recommended with provisos or alterations," "would not recommend," or "unsure." For ease of interpretation, we considered "strongly recommended" and "recommended with provisos or alterations" as responses indicating "recommended" and "would not recommend" or "unsure" as responses indicating "not recommended."
Cohen's 
, a widely used measure of agreement, was considered. For this purpose, the AGREE response categories were dichotomized into strongly agree/agree versus strongly disagree/disagree, because we judged an analysis of agreement at this level to be sufficient. However, substantial imbalances in the distribution of the tables' marginal totals were present, making comparison and interpretation of the values according to Cohen's criteria inappropriate.17,18 Intraclass correlation coefficients (ICCs) were considered to assess the inter-rater reliability within each domain. Similarly to values, ICCs are dependent on the variance of the item scores and thus the resulting domain scores. Because of lack of variance of the obtained item scores for some domains in this study, resulting ICCs were low and could be misinterpreted as indicating lack of agreement. For that reason, we report only the observed proportion of overall agreement among the reviewers for each of the 23 items of the AGREE instrument.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSAPPENDIX 1.APPENDIX 2.REFERENCES |
|---|
Priority List
The response to our request to colleagues to provide a list of 5 topics that needed a Dutch national guideline most urgently was 47% (24 of 51 individuals) after 2 mailings. The 10 most frequently mentioned topics and clinical problems are listed in Table 1.
|
|
Stakeholder Involvement
This domain evaluates the degree to which the guideline represents the views of its intended users. Included are questions regarding the composition of the guideline development group (specifically, whether individuals from all relevant professional groups were represented), whether patients' experiences and expectations influenced the development of the guideline, whether the target users of the guideline were well defined, and whether the guideline was pilot-tested among end-users. Overall, the mean score for this domain was 42% (range: 17–69%), with 12 guidelines (71%) scoring <50%. Thirteen guidelines (76%) included individuals from all relevant professional groups in the guideline development stage, but none involved patients in the development or was pilot-tested among end-users.
Rigor of Development
This domain evaluates specifically whether systematic methods were used to search for evidence, whether the criteria for selecting the evidence and the methods used to formulate the recommendations were described clearly, whether there was an explicit link between the recommendations and the supporting evidence, whether health benefits, side effects, and risks were considered during formulation of the recommendations, whether the guideline was reviewed externally by experts before publication, and whether a procedure for updating the guideline was provided. Overall, the mean score for this domain was 54% (range: 2–84%), with 5 guidelines (29%) scoring <50%. Specifically, 14 guidelines (82%) described systematic methods for searching and selecting the evidence, 9 guidelines (53%) considered health benefits, side effects, and risks during formulation of the recommendations, 11 guidelines (65%) described the methods used to formulate the recommendations, 14 guidelines (82%) indicated an explicit link between the supporting evidence and the recommendations, and 11 guidelines (65%) were reviewed externally before publication. Only 3 guidelines (18%) described a procedure for updating the guideline.
Clarity and Presentation
This domain describes the clarity of the guidelines. Specifically, it describes whether the recommendations were specific and unambiguous, whether the different management options were presented clearly, whether key recommendations were easily identifiable, and whether the guideline was supported with tools for application. Overall, the mean score for this domain was 78% (range: 28–97%). Four guidelines (24%) scored <50% for this domain.
Applicability
This domain evaluates the likely organizational, behavioral, and cost implications of applying the guideline. In addition, review criteria that link guideline use to audits and other quality improvement initiatives should be developed. The score on this domain was the lowest of all, with a mean score of 19% (range: 0–59%). Only 2 guidelines (12%) scored 
50%. One guideline provided review criteria for monitoring purposes, and 3 discussed potential organizational barriers. Only 1 guideline discussed cost implications.
Editorial Independence
This domain addresses conflict of interest, specifically whether the guideline was editorially independent from the funding body and whether potential conflicts of interest were reported for the members of the guideline development group. The mean score in this domain was 40% (range: 17–78%). Seven guidelines (41%) scored >50%. In 15 guidelines (88%), potential conflicts of interest on the part of guideline developers were not recorded.
Agreement Among Reviewers
Of the 17 guidelines appraised, the reviewers agreed about the subjective judgments for 88% (15 guidelines) and came to a consensus with respect to an overall recommendation for each guideline. In total, we recommended 14 of 17 guidelines (82%). Table 3 summarizes the observed simple agreement among reviewers for the 23 items of the AGREE instrument. Observed agreement among reviewers was 41% to 60% for 3 items, 61% to 80% for 12 items, and >80% for 8 items.
|
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSAPPENDIX 1.APPENDIX 2.REFERENCES |
|---|
We used the AGREE instrument to assess the quality of current pediatric guidelines. Mean domain scores in this survey were 84% for scope and purpose, 42% for stakeholder involvement, 54% for rigor of development, 78% for clarity of presentation, 19% for applicability, and 40% for editorial independence. Compared with other studies of the quality of guidelines assessed with the AGREE instrument in fields outside pediatrics, these scores are fairly good. For example, the international validation survey of the AGREE instrument (www.openclinical.org/prj_agree.html) showed mean domain scores of 69% for scope and purpose, 36% for stakeholder involvement, 41% for rigor of development, 66% for clarity of presentation, 37% for applicability, and 30% for editorial independence. A Canadian study reviewed the quality of drug therapy guidelines developed or endorsed by Canadian organizations from 1994 to 1998.14 Only 5% of the 217 guidelines reviewed met one half or more of the 20 criteria defining the rigor of development process. Another recent study assessed the quality of clinical practice guidelines in lung cancer with the AGREE instrument.19 Of the 51 relevant guidelines identified, most were of poor overall quality. Only 19 of 51 (37%) of those guidelines were recommended for use in practice.
Application of Pediatric Guidelines
In this study, we recommended 14 of 17 (82%) pediatric, evidence-based, practice guidelines for use in the Netherlands. This high percentage of recommended guidelines is attributable in part to the fact that we searched deliberately for high-quality guidelines. We restricted our searches in Medline and Embase with the term "evidence-based," because searching Medline for practice guidelines (publication type) yielded many consensus statements, position papers, workgroup reports, clinical policies, and standards. Although such documents are potentially useful in practice, their emphasis is usually not on the key clinical management questions and the supporting scientific evidence to answer those questions but rather on opinion and expert advice. We recognize that, because this was a selective search for potentially high-quality guidelines, we might have missed some good-quality guidelines.
All 14 guidelines were recommended with provisos or alterations. Typically, guideline recommendations are based on evidence, which is considered to be global, and other considerations, which are local and may differ among cultures. Each country has its own cultural and legal standards, and values and organizational limitations may affect the local recommendations. It is therefore not surprising that, in light of the same scientific evidence, different guideline developers produce different recommendations for local practice. For the successful implementation of good research evidence into clinical practice, we think that local groups of practitioners should create their own recommendations involving all relevant evidence and all local stakeholders.
Sources of Pediatric Guidelines
The NGC is a freely available database with >1000 high-quality, evidence-based, practice guidelines. Not only American but also foreign guidelines can be registered if they fulfill a set of criteria, eg, a systematic review of the literature should be performed. The SIGN and AAP guidelines are all registered in the NGC. The appraised guidelines indexed in the NGC and produced by the AAP were all recommended. Two of the 3 guidelines indexed in the CMA Clinical Practice Guidelines Infobase were not recommended because they scored low on rigor of development (2% and 11%). The guideline found in Embase was also not recommended; the score on rigor of development was 40% for this guideline.
Agencies that produce good-quality guidelines submit them to the NGC, which applies stringent quality criteria before it indexes a given guideline. To search for high-quality guidelines, therefore, we recommend starting by searching the NGC. However, in searching for a guideline on a specific topic that cannot be found in the NGC, it can be rewarding to search the Internet, because some guideline developers prefer simply to post their guidelines on their Web sites, as opposed to publishing them in journals or having them indexed in the NGC.
Areas for Pediatric Guideline Improvement
On the basis of the results of our survey, we identified various areas in which pediatric guidelines can be improved. First, patient preferences and experiences were not sought. This is especially important for guidelines in which quality of life plays an important role or in which treatment can have significant morbidity or side effects. However, in the present selection of guidelines, this might have been of lesser importance. Furthermore, most guidelines did not provide evidence of pilot testing. This is an important issue to ensure that the guideline can be put into actual clinical use.
Because we were looking for evidence-based guidelines, the AGREE domain scores for rigor of development are notable because they relate directly to how evidence based a guideline is. The fairly low mean score on this domain (54%) is partly attributable to the fact that the rigor of development domain contains not only questions on how evidence based a guideline is but also questions about whether the guideline was reviewed externally by experts before publication and whether a procedure for updating the guideline was provided. Items 8, 9, and 10 (ie, whether systematic methods were used to search for evidence and whether the criteria for selecting the evidence and the methods used to formulate the recommendations were clearly described) are the most important items relating to the issue of how evidence based a guideline is. If we only take these questions into account, the mean score would be 70% for our 14 recommended guidelines. It is generally recommended that guidelines be updated at least every 3 years, because new evidence can change the recommendations.20 However, only 3 guidelines (18%) described a procedure for updating the guideline and 5 guidelines were already outdated by this criterion (ie, >3 years old). For most recommended guidelines, we advised additional literature searches to update the evidence.
Another area in which most guidelines performed poorly was in the domain of applicability. Well-developed guidelines should include at least some consideration of potential barriers to implementation and cost implications, and they should supply monitoring criteria to assess the guideline's impact on practice organization and patient outcomes. Finally, editorial independence was stated rarely. Poor performance in this domain could represent true conflicts of interest between funding sources and guideline development panels or might reflect simply poor reporting on these topics.
Limitations of the AGREE Rating Instrument
In this survey, we used the AGREE instrument to assess the quality of pediatric guidelines. Although this instrument is fairly new, it is one of the few guideline assessment tools with demonstrated validity and reliability.16 A guideline that addresses the issues covered by the AGREE instrument is more likely to be a rigorously developed guideline. However, the AGREE instrument showed its limitations in this particular survey. We found that some of the variability in ratings might be attributable to differences in interpretation of several items. For example, for item 22, which is in the domain of editorial independence, the observed agreement was 53%. This apparently poor agreement probably arises from the fact that some reviewers considered that the criterion was not met unless the statement was explicitly made in the guideline, whereas others interpreted the criterion to be met if the funding agency was a national government. In contrast, for 3 items (items 7, 8, and 17), the observed agreement was >90%. This is probably attributable to the fact that these questions are more straightforward. Another potential limitation of the AGREE instrument concerns the validity of the responses to the question on the overall assessment of the guideline's quality. No clear rules have been established regarding how to weight the different domains. However, in a review of the assessments compared with the domain scores, the responses appear to be valid and to reflect the quality of the guidelines. For each guideline that was recommended by all 3 raters in the present survey, the overall domain scores were 50% for at least 3 domains, with an average of 4 domains. For guidelines that were not recommended, only 1 domain scored >50%. Furthermore, the scores on the domain of rigor of development for recommended guidelines were high. All scores were 50%, with an average of 66%. Conversely, for guidelines that we did not recommend for use in our settings, the average score for this domain was only 18%. Another limitation of the AGREE instrument is that it assesses only the reporting of the different items and not the content validity of the recommendations. To asses the content validity of the recommendations, the rater must have both pediatric subject matter knowledge and skills in evidence-based medicine. When items are not reported specifically, they receive a low score (ie, 1, strongly disagree), although in fact the developers might have met the criterion. It is therefore important that future guidelines report all different items specifically.
Opportunity for International Collaboration on Pediatric Guidelines
The development of some parts of a guideline, specifically the comprehensive literature reviews, is time-consuming work. For some topics, we found several evidence-based guidelines with a substantial overlap in evidence summaries. Collaboration between guideline developers throughout the world could be a way to avoid unnecessary duplication of effort. One current collaboration in this respect is the International Guidelines Network (www.g-i-n.net), which has grown to 46 member organizations from 24 countries. The International Guidelines Network seeks to improve the quality of health care by promoting systematic development of clinical practice guidelines and their application in practice, by supporting international collaboration in guideline development.
Currently, no pediatric organization is a member of the International Guidelines Network. If we had a worldwide collaboration in guideline development, then the work would be made lighter and faster through sharing of agendas for topics for guideline development and state-of-the-art guideline development methods, literature searches, and critical appraisal. When a guideline is out of date, additional literature searches should be performed. Within a framework of guideline development collaboration, local guideline development groups could produce their own recommendations without duplicating the time-consuming literature search and critical appraisal process. When, despite a rigorous search and analysis of the scientific literature, clear evidence for key recommendations is missing, it should be stated clearly on what basis local consensus has been reached. Also, recommendations for future empirical research to fill evidence gaps can be given.
|
CONCLUSIONS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSAPPENDIX 1.APPENDIX 2.REFERENCES |
|---|
|
APPENDIX 1. |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSAPPENDIX 1.APPENDIX 2.REFERENCES |
|---|
|
|
APPENDIX 2. |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSAPPENDIX 1.APPENDIX 2.REFERENCES |
|---|
|
|
ACKNOWLEDGMENTS |
|---|
We thank Maruschka Merkus, PhD, for advice with the statistical analyses.
|
FOOTNOTES |
|---|
Address correspondence to Nicole Boluyt, MD, Center for Pediatric Clinical Epidemiology, Emma Children's Hospital, Room H3-145, Academic Medical Centre, PO Box 22660, 1100 DD Amsterdam, Netherlands. E-mail: n.boluyt{at}amc.uva.nl
No conflict of interest declared.
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSAPPENDIX 1.APPENDIX 2.REFERENCES |
|---|
.
J Clin Epidemiol. 1993;46
:423
–429[CrossRef][ISI][Medline] viewed in terms of sensitivity and specificity.
J Clin Epidemiol. 2000;53
:499
–503[CrossRef][ISI][Medline]This article has been cited by other articles:
 |
|
 |
|
  F E Babl, N Sheriff, M Borland, J Acworth, J Neutze, D Krieser, P Ngo, J Schutz, F Thomson, E Cotterell, et al. Paediatric acute asthma management in Australia and New Zealand: practice patterns in the context of clinical practice guidelines Arch. Dis. Child., April 1, 2008; 93(4): 307 - 312. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Watine, B. Friedberg, E. Nagy, R. Onody, W. Oosterhuis, P. S. Bunting, J.-C. Charet, and A. R. Horvath Conflict between Guideline Methodologic Quality and Recommendation Validity: A Potential Problem for Practitioners Clin. Chem., January 1, 2006; 52(1): 65 - 72. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||
|
||
Home | My Pediatrics | Journal Information | Current Issue | Past Issues | Subscriptions & Services | Contact Us Click here for faster international access © 2005 American Academy of Pediatrics. All rights reserved. |
||
|
||
