PEDIATRICS Vol. 120 No. 5 November 2007, pp. 1131-1132 (doi:10.1542/peds.2007-2323)
COMMENTARY |
Evaluating Quality Improvement
Department of Medicine, Center for Health Policy Research, Irvine, California
Abbreviations: RCT, randomized, controlled trial • CAASA, California Asthma Among the School-aged
The bloom is off the rose, at least partially. From the time they emerged in the 1940s as a substantial advance over available research designs, randomized, controlled trials (RCTs) quickly became the gold standard for testing medical interventions. However, it has long been acknowledged that for some interventions, particularly programs with multiple components or those intended for broad application or implementation when generalizability is of primary concern, RCTs for a variety of reasons either cannot be performed or are of less value than other research designs. A recent Institute of Medicine roundtable compendium encouraged movement toward increasingly sophisticated research designs other than RCTs to assess quality improvement or patient-safety initiatives.1–3
Rigorous evaluations of programs such as the California Asthma Among the School-aged (CAASA) project face the challenges and tensions inherent in testing the effectiveness of multiple-component interventions that are intended to be applied to highly disparate patient populations, widely varied clinical venues, and divergent geographic locations. In an extremely large, well-designed RCT that addressed effects of region, practice size, patient characteristics, and within-practice clustering, Homer et al4 found no effects of a quality improvement intervention program for children with asthma. The authors of that study specifically pointed to variability of program implementation and intensity and duration of the intervention as contributors to the null results. In pre-post, nonequivalent control-group studies of a national quality improvement project for adult patients with chronic disease, Landon et al found either modest improvements in the process of care, but no improvements in patients’ health outcomes,5 or no effects on quality of care at all.6 The authors also pointed to the variability or lack of intensity of the intervention after the initial sessions as contributing to the absence of an observed intervention effect.
In contrast, as shown in the October Pediatrics Electronic Pages, Fox et al7 performed a pre-post study of the impact of the CAASA project on the quality of care for children with asthma in 7 clinics distributed throughout the state of California. Rather than identifying a control group a priori, the authors defined a control group posthoc from those clinics that did not administer the quality improvement intervention according to study protocol. They further created the equivalent of a "dose-administration" variable, which was defined by the level of intensity of administration of their intervention or level of adherence to their study protocol.
Despite the acknowledged limitations of the authors’ study design, their findings that the CAASA project improved the quality of care for children with asthma are compelling for 3 reasons. First, all of the process and outcome measures of quality improved, often substantially, over baseline. On the basis of other empirical quality-of-care literature, magnitudes of the improvements shown in this study exceeded what would be expected from secular trends or regression to the mean. The uniformity and magnitude of these effects are particularly striking in light of the recent research showing mixed, if not disappointing, results from studies of quality improvement interventions.4–6 Second, although it does not meet standards for asserting causality, the evidence that Fox et al presented documenting a strong association between level of adherence to their study protocol and improvement (adjusted for patient and site characteristics) supports the premise that observed improvements were related to the intervention. Their data are suggestive of a "dose response," with greater improvements in quality related to more intensive administration of the intervention. Finally, and probably most important, is the plausibility of their findings. As a researcher and someone who has participated in quality improvement studies for many years, I am acutely aware of the difficulties encountered when trying to change patients’ and providers’ behavior. That it takes the intensity of a concerted program defined by the complete approach defined by the CAASA project is no surprise, particularly for poor and underserved patient populations, the health problems of which are known to be more difficult to address.
If we are to improve quality of care for the underserved, conducting large, expensive RCTs may not be the optimal strategy. Methodologic approaches such as that defined by Fox et al should be considered for additional testing. If their results are replicable, costs of implementing and evaluating such interventions could be offset by savings in unnecessary emergency department visits and hospitalizations. Evaluations of cost-effectiveness of these more intensive approaches to quality improvement will be needed to determine their feasibility for broader implementation.
 |
FOOTNOTES |
|---|
Accepted Sep 6, 2007.
Address correspondence to Sheldon Greenfield, MD, Center for Health Policy Research, Department of Medicine, 111 Academy, Suite 220, Irvine, CA 92697. E-mail: sgreenfi{at}uci.edu
The author has indicated he has no financial relationships relevant to this article to disclose.
Opinions expressed in these commentaries are those of the authors and not necessarily those of the American Academy of Pediatrics or its Committees.
|
REFERENCES |
|---|
 TOP REFERENCES |
|---|
- Greenfield S, Kravitz RL. Heterogeneity of treatment effects: subgroup analysis. Am J Med. 2007;120(4A) :S1 –S35
- Greenfield S, Kravitz RL. Heterogeneity of treatment effects: subgroup analysis. In: Olsen L, Aisner D, McGinnes JM, eds. The Learning Healthcare System: Workshop Summary. Washington, DC: National Academies Press; 2007:113 –123
- Institute of Medicine. The Learning Healthcare System: Workshop Summary. Washington, DC: National Academies Press; 2007
- Homer CJ, Forbes P, Horvitz L, Peterson LE, Wypij D, Heinrich P.
Arch Pediatr Adolesc Med. 2005;159
:464
–469
[Abstract/Free Full Text] - Landon BE, Hicks LS, O'Malley AJ, et al. Improving the management of chronic disease at community health centers.
N Engl J Med. 2007;356
:921
–934
[Abstract/Free Full Text] - Landon BE, Wilson IB, McInnes K, et al. Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV Study.
Ann Intern Med. 2004;140
:887
–896
[Abstract/Free Full Text] - Fox P, Porter PG, Lob SH, Boer JH, Rocha DA, Adelson JW. Improving asthma-related health outcomes among low-income, multiethnic, school-aged children: results of a demonstration project that combined continuous quality improvement and community health worker strategies. Pediatrics. 2007;120 (4). Available at: www.pediatrics.org/cgi/content/full/120/4/e902
PEDIATRICS (ISSN 1098-4275). ©2007 by the American Academy of Pediatrics
CiteULike 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
