 |
|
 |
|
 |
 |
|
 |
 |
 |
 |
 |
 |
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
a Utah Pediatric Partnership to Improve Healthcare Quality, Salt Lake City, Utah
b Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, Utah
c Utah Valley Pediatrics, American Fork, Utah
d Department of Medicine, University of Utah School of Medicine, Salt Lake City, Utah
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODSRESULTSDISCUSSIONREFERENCES |
|---|
METHODS. After conducting an initial audit of the medical records of twenty 2-year-olds and twenty 4-year-olds for documentation of preventive services on the basis of national standards, practice teams attended a quality improvement workshop. They were presented with evidence to support the value of preventive services and the results of their audits and taught quality improvement methods, eg, rapid cycles of change. Each team developed plans to improve 1 or more services. Brief audits with feedback and monthly conference calls were used to support practices to conduct rapid cycles of change, to discuss barriers and solutions, and to monitor progress. The results of final chart audits of twenty 2-year-olds and 4-year-olds were compared with the initial chart audits. A Preventive Service Score (PSS) was assigned to each practice on the basis of the number of services provided, and initial to final comparisons were made.
RESULTS. Fourteen practices participated. PSSs improved for all practices after the LC. Mean PSS for 2-year-olds increased from 4.0 ± 1.1 to 4.9 ± 1.2 and for 4-year-olds increased from 3.8 ± 1.8 to 5.6 ± 1.9. The proportions of children who received 9 of the 10 individual preventive services also improved significantly.
CONCLUSION. LCs are a potentially effective method of improving the quality of care that is delivered by pediatric practices.
Key Words: quality improvement • preventive services
Abbreviations: NICHQ—National Initiative for Child Healthcare Quality • LC—learning collaborative • QI—quality improvement • AAP—American Academy of Pediatrics • IPS—Intermountain Pediatric Society • UPIQ—Utah Pediatric Partnership to Improve Child Healthcare Quality • VCHIP—Vermont Child Health Improvement Project • PSS—preventive service score • ETS—environmental tobacco smoke
The gap between the quality of health care that patients should receive and the care that they actually do receive is widely recognized.1–3 Increasingly, the causes of this "chasm" are understood as defects in the system in which care is delivered rather than as resulting from a lack of knowledge or desire on the part of individual physicians or other providers.2, 3 To assist providers to develop the knowledge and skills that are necessary to improve quality, national organizations such as the Institute for Healthcare Improvement and the National Initiative for Child Healthcare Quality (NICHQ) have developed approaches called breakthrough series and learning collaboratives (LCs). These improvement partnerships bring health care provider teams together to learn how to overcome within their particular system (eg, a hospital, practice, clinic) the barriers that impede the delivery of high-quality care. Together, they learn to identify an explicit quality goal or aim, to work with their team to develop innovative plans for change within their system, and to use appropriate measurement tools to indicate whether the changes do result in the desired improvement. Two states, Vermont and North Carolina, have developed statewide improvement partnerships of providers and quality experts to help pediatric practices learn about and engage in practice-based quality improvement (QI) activities. Their reports and those of others have demonstrated success in improving the care of children in the areas of preventive services, developmental screening, asthma, and attention/deficit-hyperactivity disorder.4–8
Preventive services, also known as well-child care or health supervision/health promotion, are the most common services delivered by primary care pediatricians. The American Academy of Pediatrics (AAP) has developed a periodicity schedule, Recommendations for Preventive Pediatric Health Care, specifying the content, frequency, and timing of the components of preventive services.9 Most states and many insurers have adopted the schedule, and there is general agreement that the recommendations serve as a standard of preventive care for US children. However, it also is widely known that many children do not receive many of the recommended preventive services.
Responding to a perceived need to improve quality at the practice level, the Utah chapter of the AAP, also known as, the Intermountain Pediatric Society (IPS), developed the Utah Pediatric Partnership to Improve Child Healthcare Quality (UPIQ). UPIQ brought together many of the major stakeholders for child health care quality in Utah, including, in addition to the IPS, the Department of Pediatrics of the University of Utah School of Medicine; the Division of Health Care Financing of the Utah Department of Health (Medicaid); the Division of Family and Community Health Services of the Utah Department of Health; Intermountain Health Care, a large vertically integrated health care corporation in Utah; and Health Insight, a QI organization. After consultation with the Vermont Child Health Improvement Project (VCHIP) and the NICHQ and with initial funding from the Primary Children's Medical Center Foundation, UPIQ conducted an LC that was designed to improve the delivery of preventive services by primary care pediatrics practices. In this article, we describe how the LC functioned and report its effect on the delivery of preventive services. We hypothesized that participating practices would improve their delivery of preventive services after the LC.
|
METHODS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
|
The LC's third phase consisted of monthly conference calls with the practices, facilitated by UPIQ staff, and brief audits of the medical records of five 2-year-olds and five 4-year-olds. Practices were provided with the results of these audits and those of the other 13 practices in the LC. Each practice's results were identifiable only to them, but they were able to compare their results with those of the other participating practices that were identified only by a number. During the conference calls, practice teams discussed with each other and with UPIQ staff their plans, results, problems, and the approaches that they had used to overcome problems. The third phase lasted 6 months. Phase 4 was a final audit of the records of twenty 2-year-olds and twenty 4-year-olds that was conducted by the practice teams and that was identical to that in phase 1.
Analysis
We compared the results of the phase 1 and phase 4 audits for each of the preventive services. For the immunization category, we considered a child to be "fully immunized" when they had received 4 doses of diphtheria-tetanus-acellular pertussis/diphtheria-tetanus-pertussis, 4 Haemophilus influenzae type B, 3 inactivated poliovirus, 3 hepatitis B; 1 measles-mumps-rubella, and 1 varicella zoster virus (4,4,3,3,1,1). Some practices included audits for heptavalent pneumococcal conjugate vaccine and hepatitis A, but we did not include these in the determination of fully immunized or not, because the recommendation for hepatitis A was relatively new and there had been national and state shortages of heptavalent pneumococcal conjugate vaccine. We developed an overall preventive service score (PSS) for the 2 age groups. The PSS was calculated by assigning 1 point for each preventive service provided and summing the results. Results of the phase 1 and phase 4 audits were compared using first the patient and then the practice as the units of analysis. When the patient was the unit of analysis, each patient had a score of 1 when the preventive service was performed and 0 when it was not performed. Patients were clustered within clinics; therefore, patients in the same clinic were more alike than were patients between clinics. To account for this lack of independence in the data, we used a random-effects logistic regression model. The proportion of patients with preventive service performed at phase 1 was compared with the proportion at phase 4, using the significance test for the before versus after main effect term. A matched-pairs approach was not used, because the before and after patients were different patients at the 2 sampled time points. For the PSS outcome, a random-effects linear regression model was used. When the practice or clinic was the unit of analysis, a summary measure approach was used. First, the proportion of patients for which the preventive service was performed, separately for phase 1 and phase 4, was computed. The proportions then were compared using a paired sample t test. The PSS outcome was tested in the same way. In actuality, the patient as the unit of analysis model tested the same hypothesis as the practice as the unit of analysis comparison. However, the paired sample t test provides a P value that is more aligned with the practice-level descriptive statistics displayed in the corresponding table, as each practice's proportion is given equal weight in the significance test, just as in the descriptive statistics. We also used a random-effects logistic regression model to test whether goal setting had an effect on whether a preventive service was performed. In this model, however, main effects were included for both goal setting (yes versus no) and time (phase 1 versus phase 4). A goal setting by time interaction term was used to test whether goal setting influences change. Specifically, it tested whether the change from phase 1 to phase 4 was greater when the service was set as a goal compared with when it was not. All reported P values are for 2-sided comparisons, with a P < .05 considered statistically significant.
The University of Utah Health Sciences Center Institutional Review Board approved the project. Each practice team member provided informed consent for his or her participation. Because no identifiable patient-related data left the practices, patient consent was not required.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
|
|
|
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
Practice-based QI projects such as the one described here assume that clinicians know what it is they want to do but face within their practices barriers that prevent them from doing it consistently. They are designed to help them first to identify these barriers and then to develop and implement quickly plans to overcome the barriers, to use measurement tools to determine whether the plans are working, and to use rapid small cycles of change to make any necessary adjustments. Our results add to a growing body of literature that suggests that when practice teams engage in joint improvement partnerships and receive appropriate support, quality does improve in a measurable way.6–8, 12, 13
We engaged 14 practices in a preventive services LC that included prework, ie, an initial determination by chart audit of how well they were doing with delivering selected preventive services. From the list of preventive services that are recommended by the AAP's periodicity schedule,9 we chose some traditional services, such as being up to date for immunizations at age 2 and assessing car seat use, and some that were of particular local interest, such as assessing exposure to ETS and determining and plotting the BMI at age 4. Our goal was not to convince practices of the value of the selected services but to help practices deliver the ones that they themselves thought were appropriate for their patients. In reality, the services that we selected required little selling to the practices, but the teams varied in their choices of services that were targeted for improvement. For example, 7 of the 14 practices wanted to improve the proportion of 4-year-olds who had a BMI plotted, 7 wanted to improve the proportion of their patients who had a vision screen, and 3 chose to improve their rates of anemia screening. Only 1 team chose to improve the proportion of 4-year-olds in their practice who were fully immunized. This may reflect the results of the initial audits, which revealed that practices were more successful in delivering this service than any of the others.
The second phase of the LC was a 1-day QI workshop. Presentations were made by VCHIP staff (Drs Paula Duncan and Wendy Davis and Kim Paul, an improvement specialist) describing the Model for Improvement and Rapid Cycles of Change using "plan, do, study, act" cycles. The Model for Improvement includes the importance of setting a goal (what are we trying to achieve?), of developing a plan (what innovative ideas for change can we develop?), and the importance of measurement (how will we know that the change resulted in an improvement?). Teams were provided with tools to develop plans and measurement tools. UPIQ staff then worked with each team on the specific details of their plans and measures and established the protocols for follow-up conference calls and audits. Teams had little difficulty determining which services on which to work. However, the importance of implementing changes on a small scale right away and systematically measuring the results were new to most of the practices. Those from larger practices expressed frustration with some of their earlier attempts at change because of a necessity to obtain agreement from everyone in the practice before anything could be changed. UPIQ and VCHIP staff suggested instead that the team start, for example, with 1 physician, on 1 afternoon, with 1 subset of patients and measure what happened. In contrast, 1 solo practitioner whose entire staff, consisting of himself and 3 staff members, attended the workshop, decided on an improvement plan, and, not needing to get approval from anyone else, implemented it within a few days!
The third phase of the LC was a 6-month period of monthly conference calls, audit, and feedback to the practices. Practice teams audited 5 charts in each of the age groups, submitted the results to UPIQ, and then were provided with their results and those of the other participating practices; the latter were presented anonymously. Participation in this phase of the LC varied from month to month and among the practices, but all practices completed some of the audits and participated in some of the conference calls.
We were gratified with the overall improvement in preventive service delivery as reflected by an increased number of patients overall who received appropriate services and by improvement in the proportions of patients who received most of the services in most of the practices. Viewed overall, however, some of the increases, although statistically significant, were modest. Space does not allow presentation of the results for each of the 10 services and the 2 PSSs for all 14 practices, but a few examples demonstrate variations that are obscured by combining the results from all of the practices. In the case of screening for anemia, 1 practice increased the proportion screened from 40% to 100%; another practice, which started at 0, increased to only 10%, whereas another increased from 9% to 55%. The proportion of 4-year-olds who had a BMI plotted appropriately increased from 0% to 95% in 1 practice but from 0 to only 5% in another and, in a third, from an initial rate of 90% to 100%. From the point of view of an organization such as UPIQ, success probably is determined best by an overall determination of how many of the practices improved and by how much, but this approach obscures within practice changes, which are likely to be the most important to the practice teams and will influence how likely they are to extend the approach to other services.
One of the reasons for variability among the practices may have been related to whether the practice chose a particular service as an improvement goal. We anticipated that a practice would be much more likely to improve the delivery of a service when they chose improving it as a specific goal. However, as shown in Table 4, this was not consistently the case. Of the 4 practices that set improving dental screening of 4-year-olds as a goal, 1 went from 100% to 95%, 1 went from 0% to 60%, and 1 went from 100% to 89%, whereas the mean change for the other 11 practices was an improvement from an initial 33% to 65%. Conversely, all 6 practices that targeted screening for ETS exposure improved (mean from 30% to 80%), whereas only 1 of the other 8 improved with respect to this service.
Others have reported positive results from efforts to improve quality at the practice level. Margolis et al6 described a randomized, controlled trial that compared 22 primary care practices in North Carolina that received conventional continuing medical education with 22 that received continuing medical education plus training in process improvement. There was significant and sustained improvement in the intervention practices in the delivery of 4 preventive services to children compared with control practices. In another study, this group reported that delivery of appropriate anticipatory guidance to 1- and 6-month-olds increased significantly in the intervention practices compared with control practices, although the final percentages achieved still were relatively low (24% in the 1-month-olds and 18% in the 6-month-olds).12 Shaw et al7 from the VCHIP described significant and clinically important improvement in the delivery of preventive services by pediatric practices in Vermont. VCHIP staff served as consultants for the UPIQ project, and there are many similarities between the processes that are used by VCHIP and by UPIQ. Some outcomes of care for adults with asthma improved after a breakthrough series asthma care collaborative involving 6 intervention and 3 control sites. In particular, patient self-management practices, which were targeted by the faculty and by the practices, showed clinically important levels of improvement.13 More impressive results with respect to asthma were reported by Cloutier et al.8 Six primary care urban clinics in Connecticut were enrolled in a disease management program that resulted not only in improved adherence to National Asthma Education and Prevention Program guidelines but also in significant reductions in rates of hospitalization and emergency department visits for children with asthma after the intervention.
Despite these reports of success, some have expressed skepticism regarding the benefits of quality collaboratives, citing highly variable results, inconsistent approaches, and generally small sample sizes.14–17 A recent Cochrane Review of QI interventions that were tailored specifically to identify and overcome barriers to change concluded that the effectiveness of such interventions was uncertain and that more rigorous trials were needed.16
Some experts in QI characterize the goals of projects such as the one we conducted as improving the "reliability" of health care.18–20 The reliability level is determined by dividing the number of times that a service is provided correctly by the total number of times that it should have been provided. According to Nolan et al18 at the Institute for Healthcare Improvement, level 1 reliability is achieved when 90% of the time the patient receives the desired care. Stated another way, this is an error rate of 10–1. Some segments of the American economy strive for and achieve rates of 10–6. The approach that we evaluated for this report is unlikely to achieve rates that exceed level 1, according to many experts.18–20 However, the practices that agreed to participate in our LC were not achieving rates of 90% reliability in the delivery of preventive services. Achieving even level 1 reliability would be, we believe, of great value for our patients.
Research to determine the best method of improving health care and evaluating whether a particular QI intervention has been successful and whether its results are generalizable presents numerous challenges.14–21 Krumholz and Herin21 listed 5 challenges that those who conduct QI studies face. First are the issues of sample size and units of analysis. Because the target of most interventions is an individual hospital, practice, or physician, analysis of patient outcomes may be inappropriate because of the effect of clustering. Analyzing at the level of a practice or a hospital results in much smaller sample sizes and loss of power. We analyzed our results using both patients and the practices as the units of analysis, and, although the magnitude and precision of the findings differ, the overall results were similar. A second challenge is that studies that compare an intervention with a control group are subject to "contamination." We used a before-and-after design rather than a randomized cohort approach and, consistent with the "collaborative" model, encouraged the practices to communicate and share ideas, so this consideration may not apply to this type of project. A third challenge is that the ideal time to assess the effect of an intervention is not clear. We assessed at the end of a 6-month intervention, but, clearly, not knowing whether improvements were sustained and whether the practices used the approach to improve in other areas is an important limitation of our study. A fourth challenge, according to Krumholz, is the generalizability of the intervention. We used the Model for Improvement method that has been reported to be successful by others, but the fact that our practices all had volunteered to participate in the LC and are in a particular geographic area are important for others to consider when developing their own QI initiatives. Finally, most QI interventions actually contain multiple strategies, and it is difficult to determine which component is most effective. The LC that we describe here contained at least 4 components: baseline measurement; a 1-day workshop; a 6-month period of conference calls, audit, and feedback; and a final measurement. We did not attempt to determine or compare the value of any of these individual components, and it is potentially important to do so. For example, the 1-day workshop required that a 3-person team from each practice attend, but we acknowledge that the absence of key personnel from a practice is disruptive and expensive. We believe that bringing the practice teams together to hear experts in QI to develop jointly plans for change and appropriate measurement tools was a valuable part of the LC, but if measurement, audit, feedback, and brief conference calls alone were just as effective, then more practices may be willing to participate. Additional research that is devoted to examining which components of a QI project or an LC and doing so in a more rigorous manner should be undertaken.21
In addition to the issues noted above, there are several other important limitations to the study reported here. First, participation in the LC was voluntary; the practice teams that participated may not be representative of pediatric practices in general. Second, there were no control practices. It is possible that the practices that did participate might have improved without participating in the LC. Third, the before-and-after results are based on data from chart audits of a modest number of patients. Fourth, we did not attempt to measure actual child health outcomes, and we cannot assume that even if the children had received the services, their health would be better. Finally, we do not know whether these results will be sustained over time.
Despite the limitations of our study and of those reported by others, our findings are suggestive that motivated practice teams can learn and apply easily basic improvement methods and, when they do so, that the quality of the care that they deliver seems to improve. However, we believe that pediatricians want to improve the reliability of preventive and other services beyond the levels achieved here. Nolan et al18 described the attitudes and innovations that are required to achieve 95% to 98% reliability in health care settings, and interested readers are referred to their white paper, which is downloadable from the Institute for Healthcare Improvement (www.ihi.org).
Although we chose to focus on the preventive services that are listed in the AAP guidelines,9 some have suggested that new approaches to well-child care, emphasizing behavioral and developmental issues, are needed.22 We believe that the approach described here could be used easily to target and improve the quality of care related to these important services. We also believe that participation in LCs or similar improvement partnerships that are led by state AAP chapters or other professional organizations would be a good way for pediatricians to meet the American Board of Pediatrics' new requirements for pediatricians to engage in QI activities to maintain their board certification.
|
ACKNOWLEDGMENTS |
|---|
|
FOOTNOTES |
|---|
Address correspondence to Paul C. Young, MD, Department of Pediatrics, University of Utah School of Medicine, 50 N Medical Dr, Salt Lake City, UT 84132. E-mail: paul.young{at}hsc.utah.edu
The authors have indicated they have no financial relationships relevant to this article to disclose.
This work was presented in part at the Pediatric Academic Societies meeting; May 17, 2005; Washington, DC.
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
This article has been cited by other articles:
 |
|
 |
|
  C. M. Lannon, K. Flower, P. Duncan, K. S. Moore, J. Stuart, and J. Bassewitz The Bright Futures Training Intervention Project: Implementing Systems to Support Preventive and Developmental Services in Practice Pediatrics, July 1, 2008; 122(1): e163 - e171. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. T. Boonyasai, D. M. Windish, C. Chakraborti, L. S. Feldman, H. R. Rubin, and E. B. Bass Effectiveness of Teaching Quality Improvement to Clinicians: A Systematic Review JAMA, September 5, 2007; 298(9): 1023 - 1037. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. E. Mercier, S. E. Barry, K. Paul, T. V. Delaney, J. D. Horbar, R. C. Wasserman, P. Berry, and J. S. Shaw Improving Newborn Preventive Services at the Birth Hospitalization: A Collaborative, Hospital-Based Quality-Improvement Project Pediatrics, September 1, 2007; 120(3): 481 - 488. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||
|
||
Home | My Pediatrics | Journal Information | Current Issue | Past Issues | Subscriptions & Services | Contact Us Click here for faster international access © 2006 American Academy of Pediatrics. All rights reserved. |
||
|
||
