PEDIATRICS Vol. 122 No. 1 July 2008, pp. 182-183 (doi:10.1542/peds.2008-1042)
COMMENTARY |
Comparative Quality Measures: Putting Evidence Above Expediency
a Department of Pediatrics, University of Colorado School of Medicine, Aurora, Colorado
Departments of b Clinical Informatics
c Epidemiology and Community Pediatrics, Children's Hospital, Aurora, Colorado
d Department of Preventive Medicine/Biometrics, University of Colorado School of Medicine, Aurora, Colorado
Abbreviations: AHRQ, Agency for Healthcare Research and Quality • PDI, pediatric quality indicator
WITH THE CURRENT enthusiasm to inform consumer choice and improve the quality of hospital care, there has been an explosion in the creation and proposed implementation of a host of measures by national and state organizations. Colorado, our home state, has mandated a statewide comparative hospital quality report card based on "widely endorsed, evidence-based quality measures" but limited to readily available administrative data to avoid additional data-collection requirements.1 The Agency for Healthcare Research and Quality (AHRQ) has catalogued 1395 quality measures, many of which were calculated from readily available but limited hospital discharge data sets, often with some form of risk adjustment based on case-mix methodologies.2 Like Texas, Florida, and Wisconsin, Colorado is considering the AHRQ-sponsored pediatric quality indicators (PDIs) as measures of comparative hospital quality for pediatric care. We hear frequently: "If it is good enough for AHRQ, it is good enough for us," although the AHRQ explicitly cautions that none of their measures have been validated for comparing across institutions. Several recent editorials have raised the warning that focusing on what is readily measurable may be "more of a distraction than a benefit"3 and may actually be "diverting the patient safety revolution."4
Ensuring comparability of measures across institutions that have large differences in patient populations, severity, case mix, administrative/coding practices, clinical practices, and specialty resources is tricky business. The article by Scanlon et al5 in the June 2008 issue of Pediatrics Electronic Pages provides specific examples of how many of the PDI measures may not be internally useful or externally comparable. Extremely broad ranges of incidence rates, high rates of conditions present on admission (a critical issue for tertiary referral centers), inconsistent coding of comorbidities, and very low rates of preventable cases detected by the current indicators all degrade the accuracy and appropriateness of PDIs for cross-institutional comparisons. Updating the PDIs to use a "present-on-admission flag" could address some of the findings by Scanlon et al, but most of the other factors, especially the inability to distinguish between preventable and nonpreventable events, are well-known fundamental limitations of relying on administrative data sets. In addition, the case-mix methods used in an attempt to adjust for the substantial differences in children's hospital and general hospital pediatric populations have never been rigorously validated.
The bottom line is that there simply is no substitute for the detailed, unglamorous "grunt work" of manual chart reviews by knowledgeable domain experts across multiple institutions to validate proposed comparative quality measures. Without these time-consuming efforts to create an objective evidence base, comparative public quality report cards using these measures, although convenient for technical, logistic, and expediency reasons, could fundamentally misinform insurers and consumers and potentially lead to misguided choices and misdirected quality-improvement efforts.
There is a better model. It was recognized very early in the Joint Commission's ORYX process that its value was primarily for internal improvement based on repetitive measurement over time. External comparisons were unreliable and often misleading unless children's hospitals were compared with similar institutions. When the Joint Commission announced its intention to require all accredited hospitals to report a predefined set of "core measures," the pediatric health care community, recognizing the inadequacy of the proposed adult-focused measures, made an important counterproposal. Children's health care organizations, including the National Association of Children's Hospitals and Related Institutions, the Child Health Corporation of America, Medical Management Planning, Inc, and the American Academy of Pediatrics, obtained a waiver and worked together to design and validate the Children's Asthma Care measures, which have been adopted nationally. It is important to note that these measures include critical elements that are missing from most other national measures, including homogeneous denominator populations (inpatient asthma) that obviate the potential bias of generic risk-adjustment, evidence-based process measures that predict favorable clinical outcomes (use of relievers and controllers), and rigorous validation by 26 pilot hospitals before implementation using tedious manual chart-review comparisons similar to those used by Scanlon et al.
Without the knowledge generated by real-world, multiinstitutional validation efforts such as those described by Scanlon et al, it is inappropriate to consider a measure to be evidence based or validated for comparative use. It is unfortunate that many current measures being endorsed by national organizations are handicapped by imprecise definitions and validated only by very limited "expert panel opinion," which is considered by the American Academy of Pediatrics to be the weakest level of evidence.6 If we really seek to improve quality and transparency in pediatric health care through public reporting, all stakeholders should commit not to embracing convenient measures but to creating and validating meaningful, reliable, and comparable ones.
 |
ACKNOWLEDGMENTS |
|---|
Dr Kahn was supported in part by National Institutes of Health grant MO1-RR00069, General Clinical Research Centers Program, National Center for Research Resources, the National Institutes of Health, and the Children's Hospital Research Institute.
|
FOOTNOTES |
|---|
Accepted Apr 10, 2008.
Address correspondence to Michael G. Kahn, MD, PhD, University of Colorado, Department of Pediatrics, c/o Children's Hospital, 13123 East 16th Ave, B400, Aurora, CO 80045. E-mail: michael.kahn{at}uchsc.edu or kahn.michael{at}tchden.org
Financial Disclosure: Dr Kahn is a shareholder in Medidata Solutions, a privately held company that produces clinical trials software for the pharmaceutical industry; Dr Todd has indicated he has no financial relationships relevant to this article to disclose.
Opinions expressed in these commentaries are those of the author and not necessarily those of the American Academy of Pediatrics or its Committees.
|
REFERENCES |
|---|
 TOP REFERENCES |
|---|
1. Colorado Hospital Association. Colorado hospital report card, 2007. Available at: www.cha.com/index.php?option=com_content&task=view&id=940&Itemid=180. Accessed March 24, 2008
2. Agency for Healthcare Research and Quality. National quality measures clearinghouse.2008 . Available at: www.qualitymeasures.ahrq.gov. Accessed March 24, 2008
3. Hayward RA. Performance measurement in search of a path.
N Engl J Med.2007; 356
(9):951
–953
4. Wachter RM. Is the measurement mandate diverting the patient safety revolution? Available at: www.qualitymeasures.ahrq.gov/resources/commentary.aspx?file=Patient_Safety.inc. Accessed March 24, 2008
5. Scanlon MC, Harris JM, Levy F, Sedman A. Evaluation of the Agency for Healthcare Research and Quality pediatric quality indicators. Pediatrics.2008; 121 (6). Available at: www.pediatrics.org/cgi/content/full/121/6/e1723
6. American Academy of Pediatrics, Steering Committee on Quality Improvement and Management. Classifying recommendations for clinical practice guidelines.
Pediatrics.2004; 114
(3):874
–877
PEDIATRICS (ISSN 1098-4275). ©2008 by the American Academy of Pediatrics
CiteULike 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
This article has been cited by other articles:
|
|
 |
|
  M. Kronman, M. Hall, A. D. Slonim, and S. S. Shah Limitations in the Agency for Healthcare Research and Quality Pediatric Quality Indicators Result in Flawed Call for National Benchmarks: In Reply Pediatrics, October 1, 2008; 122(4): 903 - 904. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
