Objective. Immunizations and other cost-effective preventive services remain underused by many children, especially those living in poverty. Given the effectiveness of provider-based tracking systems and the widespread use by managed care organizations of financial incentives to influence physician practice patterns, we designed and tested an intervention combining these strategies. We studied whether a system of semiannual assessment and feedback, coupled with financial incentives, could improve pediatric preventive care in a Medicaid health maintenance organization (HMO).
Methodology. We randomly assigned primary care sites serving children in a Medicaid HMO to one of three groups: a feedback group (where physicians received written feedback about compliance scores), a feedback and incentive group (where physicians received feedback and a financial bonus when compliance criteria were met), and a control group. We evaluated compliance with pediatric preventive care guidelines through semiannual chart audits during the years 1993 to 1995.
Results. Compliance with pediatric preventive care improved dramatically in the study period. Repeated measures ANOVA demonstrated a significant increase in all three study groups throughout the time in total compliance scores (from 56%–73%), as well as scores for immunizations (from 62%–79%) and other preventive care (from 54%–71%). However, no significant differences were observed between either intervention group and the control group, nor were there any interaction (group-by-time) effects.
Conclusions. Feedback to physicians, with or without financial incentives, did not improve pediatric preventive care in this Medicaid HMO during a time of rapid, secular improvements in care. Possible explanations include the context and timing of the intervention, the magnitude of the financial incentives, and lack of physician awareness of the intervention.
- HMO =
- health maintenance organization •
- HMA =
- Healthcare Management Alternatives, Inc •
- FB+I =
- feedback and incentive •
- FBO =
- feedback only
Immunization is a cost-effective means of preventing childhood disease. Despite recent improvements in immunization rates, the nation still has not met the Healthy People 2000 goal of completely immunizing 90% of all children by age 2.1 In 1992 to 1993, 65% of 2-year-old children had completed a full series of vaccinations;2 by 1996 to 1997, that figure had risen to 78%.3 Some of this success can be attributed to the Childhood Immunization Initiative, launched in 1993 by the Centers for Disease Control and Prevention. This coordinated national effort sought to increase basic vaccination levels for 2-year-old children to 90% by 1996, and full vaccination levels to 90% by 2000. Although the 1996 goals of the Childhood Immunization Initiative have been met for the most critical doses of each vaccine, racial and socioeconomic disparities persist. Young children in poverty have levels 4% to 11% lower than other children, and levels are lowest among children of racial/ethnic minorities living in poverty.4 More than 1 million preschoolers remain inadequately immunized.3
Childhood immunization rates also reflect the adequacy of pediatric health care in general. Underimmunized children make fewer preventive health care visits, miss more appointments, and are far less likely to be screened for anemia, lead, or tuberculosis than their fully immunized counterparts.5 ,6 In the inner city, underimmunized children are at especially high risk for anemia.7
Both parental and provider factors contribute to inadequate pediatric preventive care, and many interventions have focused on improving physician attention to preventive care. Provider-based tracking systems with assessment and feedback can be effective in improving immunization rates, even in challenging patient populations of lower economic status.6 ,8
Physician financial incentives also may be a factor in the delivery of pediatric preventive care.9 ,10 In states with higher Medicaid physician fee levels, children are more likely to have a doctor's office as a usual source of care (as opposed to a hospital clinic or other health care facility) and have a high number of preventive visits at office-based sites.11 In addition, the Vaccines for Children program in New York City, which provides dramatically higher physician reimbursement for vaccine administration and free vaccines to providers, significantly improved vaccination rates and other preventive care.12
Although several large insurers such as Aetna/US Health Care and Kaiser Permanente have reported using financial incentives to improve quality, these programs have not been systematically evaluated, and no results have been made available through publication. Given the effectiveness of provider-based tracking systems, and the widespread use by managed care organizations of financial incentives to influence physician practice patterns, we designed and tested an intervention combining these strategies. We studied whether a tracking system with assessment and feedback, coupled with financial incentives, could improve pediatric preventive care in a Medicaid HMO.
This randomized, controlled trial evaluated the impact of feedback, with or without financial incentives, on physician compliance with pediatric preventive care guidelines. The study population included primary care physician practices serving pediatric members of a managed care plan in Philadelphia. We conducted the study between 1993 and 1995 with Healthcare Management Alternatives, Inc (HMA), a Medicaid plan structured like an Independent Practitioner Association, with primary care physicians paid by capitation.
At the start of the study, we identified all primary care physician sites in HMA's network having at least 25 pediatric members up to the seventh birthday. We excluded smaller pediatric sites to increase the likelihood that an adequate number of charts would be available for audit, and to focus the study on those sites with larger pediatric panels, which would be more likely to respond to feedback and/or financial incentives related to compliance with pediatric care guidelines. In total, 53 sites were randomly assigned to one of three study groups: feedback and incentive (FB+I), feedback only (FBO), or control. We stratified the randomization by practice type (solo/group) to ensure adequate representation of each. Four sites—2 in the FBO group and 2 in the control group—discontinued participation in the health plan during the study. Thus, the final sample included 49 sites (19 FB+I, 15 FBO, 15 control).
The health plan's pediatric preventive care guidelines, chart audit tools to assess physician compliance with these guidelines, and audit result feedback reports were used for the study. The study focused on compliance with guidelines for children between the first and seventh birthdays (ages 1 through 6). HMA's preventive care guidelines (Table 1) were adapted from recommendations by the US Preventive Services Task Force and the Philadelphia Department of Health, and were reviewed and approved by the health plan's peer review committees, to increase physician acceptance and buy-in. The guidelines were distributed in brochure form to providers in all three study groups shortly before the study began.
Pediatric preventive care audit tools were developed by HMA to assess compliance with the guidelines. Chart audits were performed for practice sites in all three groups at 6-month intervals: baseline and three follow-up points, for a total of four audits during 18 months. For each audit, trained nurse-abstractors used by the health plan contacted the site, scheduled the on-site record review, and requested a specific random sample of charts for members ages 1 through 6 years, who had been enrolled with the provider for at least 6 months. The chart request lists were generated by the health plan's information systems department. At each audit, the HMA nurse reviewed 15 records, using a standardized tool. The nurse-reviewers were blinded to which sites were in each of the three study groups. Intrarater reliability tests indicated a high degree of reproducibility (r 2 = 0.89).
Audit data were entered into a scoring program developed by HMA, to produce summary reports and compliance scores. For each site, compliance scores were calculated for each of the indicators presented in Table 1, as the number of reviewed charts for which the indicator was met, divided by the number of charts for which the indicator was applicable. For example, if 12 of the 15 reviewed charts were for children >age 2, and 6 of these had blood pressure recorded at least once per year after age 2, the compliance score for blood pressure would be 50% (6/12). Compliance for immunizations was defined as receipt of vaccines on or before the maximum recommended age, with a 2-month grace period. A total compliance score also was calculated as the total number of indicators met across all charts, divided by the total number of applicable indicators across those same charts.
Sites that were randomized to the control group were audited at 6-month intervals, but did not receive feedback reports or financial incentives. Sites in the FBO group received a feedback report after each audit, showing their performance (compliance scores on each indicator and total compliance score) in comparison with the performance of all audited sites. The FBO group letter for the first cycle of feedback also included an explanation that the health plan would now be conducting audits and providing feedback reports on a semiannual basis.
The FB+I sites received semiannual feedback, plus a cover letter with the feedback report notifying them if they had qualified for a bonus based on their performance. At the start of the study, a letter was mailed to all physicians at the FB+I sites notifying them that HMA was starting a new quality-based financial incentive system linked to pediatric preventive care audit scores. The details of the bonus and the rules for qualifying were explained. Eligibility for bonuses was based on the total compliance score. Bonus eligibility required a minimum compliance score of 20% for each indicator. The 3 sites with the highest total compliance scores received a full bonus (20% of the site's total 6-month capitation for pediatric members up to the seventh birthday); the 3 next best scoring sites received a partial bonus (10% of capitation); and the 3 sites showing the most improvement from the last audit, also received this partial bonus, provided that their total compliance score increased by at least 10%. These percentages were based on a previous survey of managers that suggested bonus levels that would garner physicians' attention.13 A minimum bonus amount was established ($500) in consideration of smaller sites.
Although randomization, auditing, and bonus distribution were site-specific, we directed all correspondence to individual physicians in the FB+I and FBO groups to maximize awareness of the study. To determine levels of awareness, we surveyed all intervention group physicians after baseline audits and introductory letters. The survey included a brief description of the program and questions about physician awareness. A second mailing targeted only nonresponders and responders who reported being unaware of the program.
An analytic file was created containing information on characteristics of the sites in each study group (number of providers, specialty, HMA panel size, and so forth) and audit results at baseline and each of the three follow-up points. Eight sites had a missing value for one audit. We used mean values from the respective study group to impute values for these sites. Other estimation techniques, such as regression imputation, did not produce significantly different results. Imputation did not affect our conclusions, but did improve the power of the study.
Statistical testing included χ2 and Student'st tests for descriptive comparisons, and repeated measures ANOVA with the site as the unit of analysis for between-group effect, time effect, and group-by-time interaction.
Bonuses paid out during the course of the study to FB+I sites ranged from $772 to $4682 per site, with an average of $2000. Thirteen of 19 sites received at least one bonus during the study, and 6 sites received two bonuses during the course of the study.
Table 2 presents a comparison of practice type and specialty among the study groups. No significant differences were observed. Table 2 also presents the results of the awareness survey. In 27 of the 34 intervention sites, at least one physician responded to the survey (79% response rate). If at least half of the physicians at a group practice indicated that they were aware of the study, the site was considered aware. Fifteen (56%) of the 27 responding sites were aware after the second mailing. Responders and nonresponders did not differ by practice type (solo or group) or site specialty (pediatrics, family practice, or general practice). FB+I sites had a higher response rate, but were not more aware of the program than FBO sites.
Table 3 presents results of audits 1 (baseline) through 4 for each study group. Figure 1 graphically presents the total compliance score changes, by group, throughout the four audit points. Repeated measures ANOVA demonstrated a significant increase in total compliance scores, as well as scores for immunizations and other preventive care, in all study groups throughout the time period (P < .001). However, no significant differences were observed between either intervention group and the control group, nor were there any interaction (group-by-time) effects.
Other analyses revealed that group practices had higher immunization rates than solo practices, and that pediatricians had higher overall compliance scores than other practitioners (P < .05). We found no significant differences in compliance scores between the intervention sites considered aware and those considered unaware.
This evaluation of the impact of feedback and financial incentives on physician behavior in a Medicaid managed care setting was unprecedented in several ways: 1) we used a randomized clinical trial methodology, whereas most studies have had limited or no comparison groups; 2) we attempted to use financial incentives to improve the quality of care, whereas most incentives seek to limit resource utilization and costs; and 3) we focused on preventive care for an urban poor Medicaid population, whereas the majority of managed care research has taken place in commercial plans. Because nearly half of all Medicaid recipients are enrolled in managed care, this setting is particularly important for disease prevention and health promotion initiatives.
This study failed to demonstrate the effectiveness of feedback and physician financial incentives in improving pediatric preventive care in a Medicaid managed care setting. Our results are consistent with another trial of financial incentives and feedback, which failed to demonstrate improvements in cancer screening in women >50 years of age in the same Medicaid HMO.14 The results conflict with a recently published randomized trial of physician financial incentives in improving influenza immunization rates among the elderly.15 However, the latter trial involved only physicians paid on a fee-for-service basis who were already involved in a Medicare demonstration project to improve influenza vaccination rates. In addition, the latter study demonstrated mixed success, with a nonsignificant 6% improvement in immunization rates throughout 1 year in the incentive group relative to a control group, and a significant 7% change in median practice-specific improvement from baseline.
Our findings should not be interpreted to mean that financial incentives cannot capture physicians' attention toward preventive care. Other factors may have contributed to the lack of effect we observed, and may provide insight into how to structure financial incentives more effectively. These factors include: the magnitude of the financial incentive, the lack of physician awareness, the context and length of the intervention, and the study power.
The Magnitude of Financial Incentive
Previous research suggests that as little as an additional 5% of a physician's capitation income could influence physician behavior.15 The bonus amounts also seemed feasible for a managed care organization to maintain in the long run. The incentive provided 10% to 20% additional capitation for each site's population of pediatric members. However, most physicians participated in many other Medicaid and non-Medicaid health insurance plans. Therefore, the 10% to 20% incentive for HMA members alone provided lower percentage improvements in physicians' overall income. When multiple insurers seek to capture a physician's attention to various forms of guidelines and procedures, it is difficult to detect the impact of any single influence. Another limitation to the impact of the financial incentives may have been that many of the practices were composed of several physicians, who may have been salaried, or less likely to respond to an incentive targeted to the site; however, randomization stratified for practice type (solo/group) and analysis did not reveal any differences in the intervention's impact for either stratum.
Lack of Physician Awareness
Only 56% of the intervention sites reported awareness of the feedback and incentive program, despite repeated mailings directly to all intervention group physicians. Many physicians may not read mailings from the HMOs in which they participate, may not remember which HMO offers which programs and incentives, or may delegate the review of administrative policies to an office manager. Thus, physicians either remain unaffected by policy change or they remember an amalgam of policies and procedures that they apply to all patients, regardless of insurance type. Other methods to reach physicians (eg, in-person detailing, group meetings) may have produced better results. Nonetheless, we found no difference among those aware versus those unaware. Although we did not measure aware physicians' interests in the intervention, it is possible that they did not perceive the potential payoff as worthwhile given the perceived costs associated with increasing preventable care activity. Also, unlike other studies, we did not disseminate information on individual performance, so there was no peer pressure or public accountability to strengthen the financial incentive.6 ,15
The Context and Length of the Intervention
Research on actual clinical practice cannot be highly controlled. During the course of the study, much public attention was focused on improving pediatric preventive care, including the Childhood Immunization Initiative and the Vaccines for Children Program. Because childhood immunization was one measure of quality in the Health Plan Employer Data and Information Set many HMOs conducted quality improvement activities, which likely had spillover effects. These system-wide events could have easily overshadowed the impact of our incentive, during the short time it took place. A longer study may have produced different results (Fig 1).
Given the mean compliance scores of 50% at baseline, we hypothesized that the FBO group could improve by at least 15%, and FB+I group could improve even more. The study had 88% power to detect a 15% improvement in mean compliance scores at the 0.05 significance level. The 15% increase represents a large effect size, given the standard deviation rates of 10%. Other studies suggest that similar interventions produce more modest effects.6 However, from a feasibility standpoint, HMOs are unlikely to commit significant financial resources to interventions which produce minimal clinical improvements.
In this Medicaid HMO setting, feedback to physicians, with or without financial incentives, did not improve pediatric preventive care. This study evaluated systems of feedback and financial incentives linked to audits which were accepted by the HMO's management and peer committees and incorporated into routine operations. Further research is needed to refine the ways in which such positive inducements to physicians can be used to improve pediatric preventive care.
This study was conducted with funding received from the Agency for Health Care Policy and Research (Grant RO1 HS 07634) and from Healthcare Management Alternatives.
We thank Christopher Schmitt and Danita Joell for their administrative and research assistance throughout the study. We also appreciate the assistance provided by Dr Denise Hamilton Ross, HMA's Chief Executive Officer and Medical Director, and the staff of HMA's Quality Management Department. Finally, we thank the Agency for Health Care Policy and Research for financial support.
- Received November 13, 1998.
- Accepted February 10, 1999.
Reprint requests to (A.L.H.) University of Pennsylvania, 423 Guardian Dr, Room 1123, Philadelphia, PA 19104-6021. E-mail:
- ↵US Department of Health and Human Services. Healthy People 2000: National Health Promotion and Disease Prevention Objectives. Boston, MA: US Department of Health and Human Services; 1991
- ↵Adams WG, Geva J, Coffman J, Palfrey S, Bauchner H. Anemia and elevated lead levels in underimmunized inner-city children. Pediatrics. 1998;101(3). URL: http://www.pediatrics.org/cgi/content/full/101/3/e6
- Cohen JW,
- Cunningham PJ
- Hillman AL,
- Pauly MV,
- Kerman K,
- Martinek CR
- Copyright © 1999 American Academy of Pediatrics