PEDIATRICS Vol. 113 No. 1 January 2004, pp. 217-227
Implementing and Using Quality Measures for Children’s Health Care: Perspectives on the State of the Practice
Dale Shaller, MPA
 |
ABSTRACT
|
|---|
Objective. The objective of this study was to identify issues, obstacles, and priorities related to implementing and using child health care quality measures from the perspectives of 4 groups: 1) funders of quality-measurement development and implementation; 2) developers of quality measures; 3) users of quality measures (including Medicaid and the State Children’s Health Insurance Program, employer coalitions, and consumer groups); and 4) health plans and providers (in their role as both subjects and users of quality measures).
Methods. A series of semistructured interviews was conducted with 
40 opinion leaders drawn from these 4 groups. The interviews were conducted by telephone between September and December of 2001. Major topic areas covered in the interviews were similar across the groups. Topic areas included 1) strategic vision and/or objectives for funding, developing, or using quality measures for children’s health care; 2) a brief summary of the specific quality measures funded, developed, or used; 3) issues and challenges facing funders and developers of measures; 4) major successes achieved; 5) obstacles to implementation and use of measures; and 6) priority needs for future funding.
Results. Leaders from all 4 groups acknowledge the importance of developing a robust set of quality measures that can serve multiple objectives and multiple audiences. Standardization of measures is viewed as a critical feature related to all objectives. An assessment of specific quality measures funded, developed, or used by strategic objective shows a high correlation between the uses intended by funders and developers and the actual applications of the various users. The most commonly cited measures across all groups are the Consumer Assessment of Health Plans Survey and Health Plan Employer Data and Information Set, followed by the Child and Adolescent Health Measurement Initiative and special topic studies to support quality-improvement applications (eg, asthma, diabetes, etc).
The major issues and challenges cited in common among funders and developers are 1) the lack of trained capacity in the field to conduct needed research and development, and 2) the difficulty in generating sustained interest and support among funders because of the complexity of quality-measurement issues, competing funding priorities in the face of limited funds available to allocate, and the lack of clear and compelling evidence that quality measurement and improvement actually result in better outcomes for children.
The 3 most common successes cited across all 4 groups are 1) the growing consensus and collaboration among diverse stakeholder groups involved in measurement development and implementation; 2) the increasing collection and use of specific measures; and 3) early documentation of tangible results in terms of improved quality of care. Specific measurement tools cited as successes by funders and developers include the Medicaid Health Plan Employer Data and Information Set, Consumer Assessment of Health Plans Survey, the Child and Adolescent Health Measurement Initiative, and Rand QA Tools.
The most important obstacle reported across all groups is the lack of a strong and compelling "business case" that clearly demonstrates the benefits of quality measurement relative to the costs of implementation. Strongly related to this barrier is the cost of implementing and using measures without a sustainable source of financing as well as the absence of strong public awareness and political support for children’s health care quality measurement. Another major barrier cited is the lack of coordination among funders, which prevents the field from developing a unified approach to addressing the numerous technical, political, and administrative issues also cited at length by the leaders interviewed.
The 5 top needs for future funding identified across all 4 groups follow directly from the major obstacles that they reported: 1) develop the business case for children’s health care quality measurement and improvement based on rigorous cost-benefit analysis and documentation of quantifiable successes; 2) develop new measures to fill the gaps in critical areas (including adolescent health care, behavioral health, and chronic conditions) that can be applied at the hospital and ambulatory care provider levels; 3) invest in building needed research capacity, a trained pool of users of quality measures, and the capacity among providers to understand and use quality-improvement methods and tools; 4) invest in developing an information infrastructure that will support the efficient collection and use of measures for multiple purposes, including clinical practice, quality measurement, and quality improvement; and 5) develop increased public awareness and support for quality measurement based on improved strategies for communicating with consumers, purchasers, providers, and policy makers.
Conclusions. Several implications are suggested by these perspectives for the future direction of quality measurement in children’s health care. First, to meet the funding needs identified, many funders must improve coordination to reduce the noise and fragmentation generated by numerous competing or redundant activities. Improved coordination among funders will help assure maximum impact and the efficient use of scarce resources. Second, the importance attached to standardization of measures by both users and developers may conflict at times with the need for innovation and flexibility. Child health quality leaders will need to manage this tension between standardization and innovation to maintain an appropriate balance between the benefits of both. Finally, many of the obstacles identified are not unique to children’s health care. Child health quality leaders will need to determine to what extent their efforts to overcome these obstacles can be successfully undertaken independently as opposed to in concert with groups concerned about other populations and sectors in the health care system.
Key Words: child health care quality • quality measurement • quality improvement
Abbreviations: SCHIP, State Children’s Health Insurance Program • CAHPS, Consumer Assessment of Health Plans Survey • HEDIS, Health Plan Employer Data and Information Set • CAHMI, Child and Adolescent Health Measurement Initiative • HCUP, Health Care Cost and Utilization Project • MEPS, Medical Expenditure Panel Survey • PHDS, Promoting Healthy Development Survey • YAHCS, Young Adults Health Care Survey • CMS, Centers for Medicare and Medicaid Services
The growing evidence of problems with the quality of health care in the United States has focused increased attention on the relative lack of measures designed specifically to assess the quality of health care for children. As efforts emerge to develop and test new measures in response to this need, it will be important to inform such efforts with the perspectives of the various groups responsible for implementing and using these measures to accomplish both their common and unique objectives.
The purpose of this study was to identify the issues, obstacles, and priorities related to implementing and using child health care quality measures from the perspectives of 4 groups:
- Funders of quality-measurement development and implementation;
- Developers of quality measures;
- Users of quality measures (including Medicaid and the State Children’s Health Insurance Program [SCHIP], employer coalitions, and consumer groups); and
- Health plans and providers (in their role as both subjects and users of quality measures).
The perspectives presented in this article were compiled on the basis of a series of semistructured interviews conducted with representatives of each of these 4 groups. The information gathered through the interview process also suggests priorities for developing new measures and taking other actions needed to advance both the science and practice of using quality measures to assess and improve the quality of health care for children.
|
METHODS
|
|---|
Methods for conducting this interview project consisted of 1) developing the interview guide; 2) identifying specific individuals to interview from the 4 informant groups; 3) scheduling and conducting the interviews; and 4) compiling and synthesizing the information gathered by topic area.
Developing the Interview Guide
A preliminary list of questions was circulated to project sponsors for review and comment. Based on this review, a revised list was compiled. Most questions were common across all groups, whereas some were specific to certain groups. In all cases, the interview guide was intended as a starting point for discussion, as opposed to a rigid script, to allow for probing and elaboration on a case-by-case basis. (See Appendix A for the interview guides by informant group.)
Identifying Individuals to Interview
As with the questions, a preliminary list of individuals to interview was compiled and circulated to project sponsors for review. Individuals were selected on the basis of their role as opinion leaders and their known involvement and experience in the funding, development, and/or use of children’s health care quality measures. Effort was made to assure some representativeness among the groups as follows:
- Funders: a balance of federal agencies and private foundations.
- Developers: a mix of federal agencies, nonprofit organizations, and accrediting groups.
- Users: a mix of users drawn from state Medicaid agencies and SCHIP programs, private employer purchasing coalitions, and child consumer advocacy groups. Within each of these subgroups, an attempt also was made to assure some geographic distribution.
- Plans and providers: a balance of institutional and ambulatory care providers, health plan administrators, and school-based service providers.
Given time and budget parameters for the project, the total number of interviews was limited to 40. A summary of organizations and initiatives included in the interview process is shown in Table 1. (See Appendix B for a list of specific individuals interviewed, organized by informant group.)
Scheduling and Conducting the Interviews
Individuals to be interviewed were sent the interview guide in advance. Interviews were scheduled and conducted over the 3-month period between September 12 and December 11, 2001. All but 1 of the interviews was conducted by telephone. The average interview time was 50 minutes. In many cases, information gathered during the interview was supplemented by reference to written materials posted on the organization’s web site or sent for review as hard copy.
Compilation and Synthesis by Topic Area
At the end of the interview process, responses were compiled and organized for each group by the following topic areas:
- Strategic objectives;
- Quality measures;
- Issues and challenges;
- Successes;
- Obstacles to implementation and use; and
- Future funding needs.
Limitations
Interviews with key informants can yield a rich amount of qualitative information related to personal and organizational views and perspectives that are not available from any other source. An obvious drawback of such interviews is that they do not provide the representativeness and scope of perspectives that can be achieved through structured surveys. However, as with any research, methods should be carefully matched with the research objectives. Given that the primary purpose of this research has been to inform a discussion involving many of the same groups and individuals that were interviewed, these methods seem appropriate for the specific use intended.
|
RESULTS
|
|---|
Strategic Objectives
Understanding the strategic objectives that the various groups are trying to achieve is important for establishing a context for their perspectives on issues, obstacles, and priorities for measurement implementation and use. Indeed, many participants in the interview process commented on the need to have a clearly defined set of objectives and actions as a starting point for guiding the development of quality measures.
Funders
All funders cited improving the quality of care for children as the overarching goal motivating their interest in funding the development and implementation of quality measures. Most of them indicated that quality measures should be designed to support multiple applications including accountability, consumer and purchaser choice, provider quality improvement, policy making, and research. Several funders previously interested only in accountability objectives have expanded their focus recently to include improvement, and 1 funder previously focused on provider improvement has broadened its focus to now include accountability. In sum, there seems to be a convergence among funders, including both federal agencies and private foundations, to embrace multiple strategic objectives for the development and use of quality measure for children’s health care.
Developers
Developers of measures offered a perspective similar to funders, favoring multiple strategic objectives for developing quality measures. This congruence is not surprising, because many of the developers are supported by the funders interviewed. Several developers stated that measures designed for accountability should also support improvement applications, and one expressed concern that measures intended for internal improvement applications may not be sufficiently rigorous for public accountability. Several of them commented on their specific mission to help fill the gaps in quality measures designed for children and adolescents. All developers emphasized the need for standardization to assure comparability and consistency.
Users
The greatest diversity of objectives was reported by users:
- Medicaid and SCHIP users cited a wide range of objectives and audiences for quality measures. All mentioned applications in health plan contracting and compliance with federal reporting requirements. Many states are using measures for public reporting, but the primary audiences are legislators and policy makers as opposed to individual consumers. Several states also commented on their increasing emphasis on quality improvement now that the groundwork for public reporting has been established.
- Employer coalitions voiced a common mission to drive quality improvements in the health care system through information and incentives. The coalitions differed, however, in their emphasis on public reporting for consumer choice versus collaborative measurement with providers to support internal quality improvement. All the coalitions acknowledged the importance of quality measurement for children but noted that a specific focus on improving the quality of health care for children and adolescents has not been singled out as a purchasing priority.
- Consumer groups expressed a primary interest in using quality measures at the system level, to support improvement objectives with providers, and to raise awareness among policy makers. Although consumer groups noted the importance of supporting parent decision-making, they all remarked that available quality measures are not sufficient for consumer choice. Their emphasis currently is on providing consumers with basic information for understanding and using the health care system, especially managed care. One consumer group noted that quality of care is not as high a priority for its advocacy work as issues of access and cost.
Plans and Providers
Virtually all providers stated that their most important objective in using quality measures is to provide data needed for quality-improvement programs. Some providers also cited the need for these same measures to also support accountability objectives, which they view as important for motivating internal improvement activities. Several of them emphasized the importance of having standardized measures for both objectives.
Quality Measures
Because of the limited scope of this interview project, the information gathered from informants regarding the specific quality measures they have funded, developed, or used should not be viewed as comprehensive. (For a comprehensive assessment of available health care quality measures for children, please see Beal et al.1)
Although not a comprehensive list, it is interesting to examine the use of specific measures for the strategic objectives identified by each of the informant groups. As can be seen from Table 2, Consumer Assessment of Health Plans Survey (CAHPS) and Health Plan Employer Data and Information Set (HEDIS) measures for children are clearly the most widely used of the reported measures across multiple objectives. Quality improvement stands out as the objective for which the largest number of measures were reported, followed by public accountability. The fewest measures were reported for the objectives of accreditation and meeting federal reporting requirements. Use of multiple measures for research and evaluation objectives is clearly more important to funders and developers than to the various types of users. However, on the whole, it seems that the number and types of measures reported by actual users are comparable with the number and types of measures reported by funders and developers. This suggests that the application of measures, at least among the groups interviewed, closely resembles the uses intended by the funders and developers.
Issues and Challenges
Questions related to issues and challenges were targeted only to funders and developers. Several common themes were noted, as well as some issues unique to each group.
Common Themes Across Funders and Developers
The key issues and challenges cited in common by funders and developers were:
- Lack of trained capacity: A strong shared concern among funders and developers is the lack of qualified researchers with the necessary breadth and depth of experience to develop and test quality measures specific to child and adolescent populations. Given the relatively low priority attached to children’s health care issues compared with other populations, much of the talent in health services research historically has been attracted in other directions. Although excellent researchers and developers are working on child health quality issues, they are few in number. This lack of trained capacity in the field makes it difficult to undertake the technical and political complexities of child health quality-measures development as well as the many challenging issues related to implementation.
- A complex "sell": The complexity of the various strategies and uses of quality measures make it a difficult topic for foundation boards and other funding decision-makers to understand. Quality measurement is more difficult to sell than programs focused on direct provision of services, and there are always strong competing priorities for funding that make it difficult to allocate resources specifically for this purpose. The difficulty in generating and sustaining interest among funders is also related to the long-term nature of the solution and the lack of compelling short-term evidence to help maintain interest and support. Several developers noted the difficulty in generating interest among some funders at all, particularly in the private sector, given the relative low visibility and priority of children’s health care among private-sector purchasers. This challenge of "selling" children’s health care quality measurement to funders is related to the need to develop the "business case" for these measures, described further below in "Obstacles to Implementation and Use."
Issues Specific to Funders
Other issues and challenges in funding quality measures for children cited specifically by funders include:
- Lack of good information regarding program priorities of other funders, what projects they are funding, and specific opportunities for partnerships.
- The absence of a strong voice on Capitol Hill advocating funding for this purpose, in contrast to support for other special populations and conditions.
Issues Specific to Developers
Issues and challenges mentioned specifically by developers include:
- Difficulty in developing and testing measures through a collaborative stakeholder model, given the organizational and methodological complexity of pediatric measurement. For example, leaders of the Child and Adolescent Health Measurement Initiative (CAHMI) noted that, although it is important to involve as many affected stakeholders as possible in the design and implementation of measures, such collaboration can often slow down the process of decision-making.
- Demand among users to apply measures to public reporting purposes for which they are not intended. For example, managers of the Health Care Cost and Utilization Project (HCUP) database, based on hospital administrative data, have expressed concern that business coalition leaders and reporters want to apply these data to reporting of hospital outcomes, although there are significant methodological issues in doing so.
- Lack of a strong methodological survey research base on which to draw in developing new measures specific to children and adolescents. This concern was expressed especially in relation to the Medical Expenditure Panel Survey (MEPS) database, to which questions related to children’s health care quality are being added.
Successes
A number of common themes emerged across the groups in their description of the major successes they have achieved.
Funders and Developers
Funders and developers alike focused their reports of success on specific quality measures that they were involved in funding and/or developing. The specific quality measures mentioned are grouped together here and presented in order of the frequency with which they were mentioned.
- Medicaid HEDIS: One of the most important early achievements cited by both funders and developers is HEDIS, the data set applied to Medicaid populations. Before Medicaid HEDIS, states faced requirements for implementing quality assurance but lacked access to a specific set of measures from which they could draw. Medicaid HEDIS supplied this tool set, and today surveys of state implementation efforts consistently show that HEDIS measures are the most widely used source of information on quality of care for children, especially in the area of preventive services.
- CAHPS Child Surveys: As with HEDIS, CAHPS applied to children gave states a common tool to use in quality measurement and reporting. More recently, the CAHPS Child Survey with the chronic-condition screener and supplemental questions developed as part of the CAHMI living-with-illness measure, represents an important modification to the use of CAHPS with children. The adoption of the new CAHPS Child Survey into HEDIS may also stimulate increased use of CAHPS among the child commercial population.
- CAHMI measures: In addition to the living-with-illness measure now part of the CAHPS Child Survey, the development and use of the Promoting Healthy Development Survey (PHDS) and Young Adults Health Care Survey (YAHCS) measures were also cited as a major success. These measures are now being used by a growing number of states and have been included also in several national data-collection and -reporting systems such as MEPS and State and Local Area Integrated Telephone Survey.
- Other measures: Other specific measures mentioned as successes include Rand QA Tools, the Vermont Oxford Network (an international database supporting quality improvement in >350 participating neonatal intensive care units), and the development of practice guidelines through the American Academy of Pediatrics as a basis for measurement as part of quality improvement.
In addition to specific measures, funders and developers mentioned other notable achievements including:
- Core of child researchers: There is a small but growing cadre of pediatric researchers that did not exist before, along with a growing understanding in the health services research community that children have unique issues that must be considered in terms of measures development and quality improvement.
- Collaboration between funders and developers: There is a growing collaboration between funders, working together with developers on common measures. Such collaboration is illustrated by various initiatives such as the success of CAHMI, the initial phase of which was jointly funded by Agency for Healthcare Research and Quality and the David and Lucille Packard Foundation; the Commonwealth Fund leveraging ongoing David and Lucille Packard Foundation support for CAHMI, with a specific focus on the implementation of PHDS in state Medicaid programs; Joint Commission on Accreditation of Healthcare Organizations and Centers for Medicare and Medicaid Services (CMS) achieving consensus on conditions on which to focus; and the leaders of National Association of Children’s Hospitals and Related Institutions and Child Health Corporation of America agreeing to collaborate on a common approach to developing information systems to support quality measurement and improvement in children’s hospitals.
Users
Similarities in success reported among the various types of users (Medicaid and SCHIP, employer coalitions, and consumer groups) were also striking. Success stories reported by at least 2 informants include:
- Impact of public reporting: The public reporting of quality measures has led to a growing visibility and public awareness of child health quality issues. Consumer groups believe that this has strengthened their clout. Purchasers laud the increased attention to quality in consumer decision-making.
- Standardization of measures: The emerging consensus and standardization of national quality measures (eg, HEDIS and CAHPS) has led to tools that can be used for comparing and benchmarking performance within and across markets.
- Cooperation of health plans and providers: States and coalitions cited a growing acceptance and cooperation by health plans and providers around the collection and reporting of quality measures. For example, in New York, coordination with plans and providers has become the norm, in contrast to the conflict and resistance engendered by measurement issues in earlier years.
- Use of measures to improve performance: Finally, there is mounting evidence that plans and providers are actually using quality measures to improve rates of preventive services and other aspects of health plan performance. For example, in Maryland, the most recent annual quality of care audit used by state officials to monitor the performance of Medicaid managed care plans showed a significant improvement in 15 of the 16 standards used. Similar progress in performance improvement was reported by other states such as Minnesota, New York, and Texas.
Plans and Providers
The most consistent success story reported by plans and providers is the growing understanding of and commitment to quality measurement and improvement among the nation’s pediatricians, children’s hospitals, and health plans. Beyond this global marker of achievement, other successes mentioned include:
- Use of tools for quality improvement: Consistent with funders, developers, and users, providers perceive notable progress in creating and using practical tools for quality improvement. For example, leaders of National Initiative for Children’s Healthcare Quality and children’s hospitals cite gains in developing information systems that support clinical decision-making and the tracking of improvement plans for specific conditions such as pediatric asthma and attention-deficit disorders.
- Early evidence of impact: Attendant with the use of these tools, providers note some early evidence of tangible results such as documented reductions in hospital intensive care unit medication errors and improvements in patient functional status and quality of life. This evidence can help make the make the business case for quality. For example, researchers at Children’s Hospital of San Diego have shown a 3-to-1 financial return in terms of preventing lost school days and emergency department visits among child asthma patients whose care has been improved.
Obstacles to Implementation and Use
One of the most fruitful and interesting areas of discussion with informants was the identification of obstacles to the implementation and use of children’s health care quality measures. Many common themes were mentioned, as well as several barriers unique to individual groups.
The following is a synthesis of the most common obstacles cited across all the groups interviewed:
- Lack of a business case: As noted in "Issues and Challenges," many opinion leaders agree that a key barrier is the lack of a strong and compelling business case that clearly demonstrates the benefits of quality measurement relative to the costs of implementation. Although there seems to be some early evidence of impact that can help make the business case, most leaders interviewed would agree that the existing evidence is not sufficient, or sufficiently documented, to address the need. Faced with numerous competing priorities for the use of scarce resources, a much greater body of evidence documenting the benefits of investing in quality measurement must be developed to help move the field forward.
- Absence of clout: Related to the lack of a compelling business case for quality measurement is the relative low priority given to health care quality measurement for children compared with other populations. Several leaders noted a continuing division in the field of quality measurement between groups focused on children and adolescents and those focused on the general population or other populations that have more "clout." In contrast to adult measures, the key users of quality measures for children are states and school-based programs, both of which are underfunded and therefore at a disadvantage as vehicles for political leverage and implementation. The absence of a strong lobby among consumer groups, states, and commercial purchasers for children’s quality measures is a major impediment to implementation, particularly given an overall lack of urgency about quality as a problem in general.
- Cost and financing: Another common barrier reported is the lack of funding needed to develop and apply measures at all or to implement them as extensively as desired across plans, provider groups, geographic areas, or special populations. For example, many state users specifically mentioned funding problems in the face of recent state budget cutbacks. These states and other users expressed concern that funding is a chronic, long-term problem that often is exacerbated even further by periodic budget tightening or a slowing economy, such as the one that we are currently experiencing in the aftermath of September 11th.
- Lack of trained capacity: Several states noted that a serious limitation in using child health care quality measures is the difficulty of training and retaining qualified staff in-house. Staffing and training issues are related to funding problems, because greater resources might contribute to increased staff capacity. They are viewed as separate issues, however, in that more measures and reports will not lead to improved decision-making and program management without qualified managers. Providers also noted the lack of trained manpower skilled in quality-measurement and -improvement practices as a major barrier.
- Lack of scientific evidence: In addition to the need for a more compelling and clearly articulated business case for quality, there is a similar but different need for more scientific evidence related to specific measures. For example, one leader commented that the federal policy framework driving quality measurement at the state level (through Medicaid and SCHIP) is not science-based; there is a lack of evidence to support that certain required measures (eg, well-child visits) are actually strongly correlated with improved health outcomes. Others noted the lack of a strong evidence base for pediatric quality measurement in the scientific literature, and 1 observer specifically noted the lack of a basic methodological foundation to assure validity and reliability of survey questions.
- Lack of needed measures: A number of users noted that available measures for children are spotty, which limits the ability to look at the system as a whole. Specific priority needs mentioned that are not currently being met include mental health and substance abuse, adolescent care, prenatal care, and children with special health care needs. Some users also cited the general lack of measures related to health care outcomes as opposed to process and structural measures.
- Data-quality issues: Developers and users of quality measures noted many technical issues related to data quality that represent barriers to implementation. Poor data quality, especially related to fee-for-service providers, was cited often as a barrier to implementing and using measures. Poor data quality leads to difficulty in identifying sample populations. High rates of turnover in managed care also complicate sampling issues. Another data issue cited is the problem of small numbers in some plans or populations to generate adequate sample sizes. Finally, lack of good data complicates good analysis, including methods of risk adjustment.
- Lack of comparative data: Even when sufficient data of adequate quality can be collected, users are hampered in their analysis by the lack of comparable data from other states or national sources to use as benchmarks for their own performance.
- Privacy concerns: Growing concerns about patient privacy protections in general may limit access to data needed for measures. One specific barrier in administering and using surveys is the issue of collecting sensitive behavioral information from or about adolescents and teens.
- Measures of accountability not linked to improvement: An important barrier noted by funders and developers as well as users of measures is that quality measures designed for accountability often are not linked to quality-improvement efforts. Data collected for one purpose is usually not aligned or used for other purposes. Relying on separate data sources contributes to an unnecessary data-collection and -measurement burden, which aggravates cost concerns. In addition, there often are long lag times between the collection and reporting of quality measures and the need to use this information for improvement.
- Lack of standardization: Despite some progress noted above in moving toward uniformity of measures, many informants expressed concern over the lack of standardized data definitions, nomenclature, and methods for measuring quality. Some noted a lack of agreement about the definition of quality. Others commented that although some measures have been standardized, there is no single, overall plan or framework guiding the industry in terms of how quality measures and reporting requirements fit together.
- Provider resistance: Related to many of the barriers cited above, users and providers mentioned provider resistance to measurement and skepticism about its value compared with the cost and burden of implementation.
In addition to the many common barriers cited across informant groups, some obstacles were mentioned that were unique to specific groups.
Obstacles Specific to Developers
Obstacles identified specifically by developers include:
- Lack of coordination among funders: The absence of coordination among funding agencies has led to serious problems with overlaps, inconsistencies, and confusion among measurement developers.
- Methodological issues: Disagreement among experts over the cost benefit of using medical record data versus administrative data, the relevance of survey data versus clinical data, and other methodological issues serve to complicate progress in the development and use of measures.
Obstacles Specific to Users
Obstacles reported by Medicaid and SCHIP programs, employer coalitions, and consumer groups include:
- Changes in requirements: Changes in measurement and reporting requirements imposed by federal agencies or industry accrediting bodies make trending difficult from year to year.
- Politics of collaboration: As noted above in "Issues and Challenges," the collaborative stakeholder model of implementation can be an effective process for achieving consensus in the long term, but it slows down decision-making considerably in the short term.
- Lack of industry leadership: The lack of leadership from within the health care industry to make quality a priority is a major obstacle.
- Salience of measures to consumers: Many available measures of quality (such as preventive measures in HEDIS) are not intuitive or face-valid to consumers. Measures must be made relevant to consumers if we expect them to be used by consumers.
Obstacles Specific to Plans and Providers
The obstacles reported uniquely by plans and providers were:
- Purchaser use of measures: Few state or private purchasers actually use quality measures in payment and reward systems for plans and providers. As a result, plans and providers who actually use quality measures and improve their performance are not rewarded and may even be penalized for their efforts.
- Bias against industrial models of improvement: Some providers expressed concern about what they perceive as a longstanding bias among funders and the academic community for the "scientific" model of quality based on the ideal of controlled clinical trials as opposed to the "industrial" model of quality improvement that is more applied, pragmatic, and relevant to practicing providers.
Funding Needs
The various needs identified by informants for future funding of child health quality measures follow closely from the obstacles identified above. As with the other topic areas covered in these interviews, a number of common themes were mentioned both within and across groups. The following represents a synthesis of the most dominant common themes:
- Build the business case: One of the most important needs identified by all groups is to build the business case for children’s health care quality measurement and improvement. This case needs to be based on a rigorous cost-benefit analysis and careful documentation of success stories that are persuasive in ways that previous efforts have not been. Efforts to construct the business case should also recognize that what matters as a "bottom-line" concern will be different for different audiences. For example, health plans need evidence that prevention can save dollars and attract or retain enrollees, whereas employers may need evidence that improved quality not only saves in the direct costs of care but also can lead to increased employee satisfaction, lower absenteeism, higher productivity, and stronger loyalty to the organization. Efforts to build the business case also must address the need to develop long-term funding models to sustain implementation.
- Develop new measures: In building the business case, it is also clear that new measures need to be developed that meet the needs of specific users in achieving their strategic objectives. The most common areas mentioned in which further measures are needed include:
- Mental health and behavioral health for adolescent and teen populations;
- Children with special health care needs; and
- Pediatric care in hospital settings.
In addition to filling gaps in these areas, leaders also noted a need to develop new generalizable measures for quality beyond survey measures that can be used across conditions. Others called for developing a single set of standardized measures that address the continuum of care across sectors (plans, providers, families, and schools). Developers and users alike noted the importance of involving users (whether purchasers, consumers, providers, or policy makers) directly in determining information needs to be met by new measures.
- Apply measures at the provider level: Leaders in all groups identified the need to develop methods for applying existing and new measures at the provider level, specifically at the level of the medical group and individual practice. Because variation is greatest at the provider level, that is where quality measures and tools for improvement are needed most.
- Build trained capacity: In response to the obstacles cited above related to the lack of trained capacity, many leaders cited a critical need to invest in training to build the research capacity necessary to develop and test new quality measures and applications. Training is also needed to build analytic capacity among program staff at the state level. Finally, training is needed to build capacity among providers for understanding and using quality-improvement tools and methods.
- Integrate collection and use of measures: An important theme emerging from these interviews is the need to develop approaches for linking quality measures more directly to the specific activities that they are designed to support (eg, consumer choice, purchasing, and quality improvement) and to find ways in which the same measures can be used for multiple purposes. For example, survey measures collected for purposes of assessing health plan performance are seldom used to support internal quality-improvement efforts within the plan. Ideally, information systems should be designed that can support clinical practice, measurement, and improvement practices concurrently.
- Build public awareness and support: Another need identified across the groups interviewed related to building awareness and support for children’s health care quality measurement among key stakeholder groups including consumers, purchasers, providers, and policy makers. Although progress has been made as a result of public reporting of measures, a more concerted education campaign or initiative may be needed to generate the level of understanding and support required among the general population as well as specific stakeholder groups.
- Improve reporting methods: Beyond developing measures, leaders agree that it is important to support the use of these measures by improving methods of communicating this information to various audiences. Improvements in presentation and dissemination methods can stimulate greater interest in use as well as help assure appropriate use of the information.
In addition to these common needs identified across the groups, users of measures identified some specific needs including:
- Simplify and focus measures: Numerous suggestions were made that clearly were motivated to simplify and focus quality measurement from the perspective of users including:
- Eliminate redundancy in collecting similar information in slightly different ways for different populations (eg, commercial, Medicaid, and Medicare).
- Focus on a small number of key indicators that can capture the essential attributes of quality and that make sense to patients, are verifiable by outsiders, and are feasible to implement (consistent with criteria for choosing measures developed by the Leapfrog Group).
- Design measures that build on existing information systems (to take advantage of resources already in place).
- Determine performance levels at which it makes sense to stop measuring certain indicators (eg, when mammograms are being performed for an optimal percent of the eligible population).
- Coordinate technical assistance: Users of measures, especially states, have indicated an interest in having access to a "one-stop, full-service" source of technical assistance that could provide a range of tools and approaches to meet particular needs. Because it is unrealistic for a single organization to address all needs, the coordination of efforts between multiple organizations with complementary tools and expertise may be a reasonable way to move toward such integration. Ideally, such assistance would be provided by an independent, objective source of expertise, familiar with the range of measures available but not tied to a specific set of measures.
|
CONCLUSIONS
|
|---|
The perspectives of opinion leaders summarized in this article suggest several important implications for the future direction of quality measurement in children’s health care: - Coordination among funders: To accomplish any of the objectives, many informants agree that steps are needed to coordinate efforts and reduce the noise and fragmentation generated by numerous competing, redundant activities. Improved coordination among funders will help assure maximum impact and the efficient use of scarce resources. Ideally, an overall strategy and action plan would be developed to coordinate funding and research and development activities across federal agencies and private foundations.
- Standardization versus innovation: The importance attached to standardization of measures by both users and developers may conflict at times with the need for innovation and flexibility. Although there are obvious benefits in terms of comparability and consistency of standardized measures, many states prefer to customize their approaches to fit their specific circumstances or at least retain the option to do so. Developers of measures support standardization (especially if it is their measures that are being standardized) but also value the opportunity to create new measures that may perform better on either a scientific or practical basis. Child health quality leaders will need to manage this tension between standardization and innovation to maintain an appropriate balance between the benefits of both.
- Independence versus integration: Many of the obstacles identified are not unique to children’s health care. Barriers related to cost, lack of trained capacity, poor data sources, inadequate infrastructure, provider resistance, and lack of public understanding and support are also encountered in efforts to develop and implement quality measures for other populations. To the extent that these issues are pervasive throughout the field of quality measurement and reporting, child health quality leaders will need to determine to what extent their efforts to overcome these obstacles can be successfully undertaken independently as opposed to in concert with groups concerned about other populations.
|
APPENDIX A: INTERVIEW QUESTIONS BY INFORMANT GROUP
|
|---|
Funders
- What is your overall objective/strategic vision for measures development and implementation?
- What specific measures are you funding and/or have you funded?
- What are the key issues and challenges you face(d) in funding these measures?
- What are the major successes you have achieved?
- What are the key obstacles to implementing and using these measures?
- What are the top 3 needs you would like to see addressed in future funding of child health care quality measures?
Developers
- What is your overall objective/strategic vision for measures development?
- What measures are you developing and/or have you developed? (For each measure, describe the purpose [quality assessment versus quality improvement], data sources required, evidence of validity/reliability, major users and applications, etc.)
- What are your funding sources for these measures?
- What are the key issues and challenges you face(d) in developing these measures?
- What are the major successes you have achieved?
- What are the key obstacles to implementing and using these measures?
- What are the top 3 needs you would like funders to know about?
Users
- What are your strategic objectives for using quality measures? What forces (internal/external) are driving you in this direction?
- What quality measures do you use? (For each measure, describe the application, source of financing, type of reporting, etc.)
- What sources of technical assistance and/or funding assistance have you used?
- What are the key obstacles to implementing and using these measures?
- What have been your major successes?
- What are the top 3 needs you would like funders to know about?
Plans and Providers
- What are your strategic objectives for using quality measures? What forces (internal/external) are driving you in this direction?
- What quality measures do you use? (For each measure, describe the application, source of financing, type of reporting, etc.)
- What sources of technical assistance and/or funding assistance have you used?
- What are the key obstacles to implementing and using these measures?
- What have been your major successes?
- What are the top 3 needs you would like funders to know about?
|
APPENDIX B: LIST OF INDIVIDUAL INTERVIEWEES
|
|---|
Funding Organizations
Rachel Block, Director, Finance, Systems, and Quality Group
Ann Page, RN, MPH, Technical Director, Division of Quality Systems and Management Center for Medicaid and State Operations Centers for Medicare and Medicaid Services
Eugene Lewit, PhD, Senior Program Manager The David and Lucile Packard Foundation
Merle McPherson, MD, Director Division of Services for Children with Special Health Care Needs Health Resources and Services Administration
Ann Monroe, Director Quality Initiative California Health Care Foundation
Michael Rothman, Senior Program Officer The Robert Wood Johnson Foundation
Lisa Simpson, MB, BCh, MPH, Deputy Director
Gregg Meyer, MD, Director Center for Quality Improvement and Patient Safety Agency for Healthcare Research and Quality
Kathryn Taaffe McLearn, PhD, Assistant Vice President The Commonwealth Fund
Measure Developers
Christina Bethell, PhD, MBA, MPH, Senior Vice President, Research and Policy FACCT—The Foundation for Accountability
Charles Darby, MA, Social Science Administrator
Christine Crofton, PhD, Social Science Administrator
Kathy Crosson, PhD Center for Quality Improvement and Patient Safety Agency for Healthcare Research and Quality
Anne Elixhauser, PhD Healthcare Cost and Utilization Project Agency for Healthcare Research and Quality
Steven Hill, PhD, Service Fellow Economist Center for Cost and Financing Studies Agency for Healthcare Research and Quality (MEPS)
Jerod Loeb, PhD, Vice President Research and Performance Measurement
Sharon Sprenger, RHIA, CPHQ, MPA, Project Director Core Measure Identification and Evaluation, Division of Research Joint Commission on Accreditation of HealthCare Organizations
Elizabeth McGlynn, PhD, Director Center for Research on Quality in Health Care Rand Corporation
Greg Pawlsen, MD, Executive Vice President National Committee for Quality Assurance
Tom Reilly, PhD, Manager National Quality Report Agency for Healthcare Research and Quality
Users
Medicaid and SCHIP Programs
Rachel Block and Ann Page CMS Center for Medicaid and State Operations
Debbie Chang, Deputy Secretary for Health Care Financing Maryland Department of Health and Mental Hygiene
Jason Cooke, Director of Medicaid and SCHIP Operations Texas Health and Human Services Commission
Foster Gesten, MD Office of Managed Care New York State Department of Health
Nicolette Highsmith, Director of Policy, Medicaid Managed Care Program
Anna Fallieras, Senior Program Officer, Medicaid Managed Care Program Center for Health Care Strategies
Mary Kennedy, Assistant Commissioner and Medicaid Director
Vicki Kunerth, Director, Performance Measurement and Quality Improvement Minnesota Department of Human Services
Employer Coalitions
Becky Cherney, President and CEO Central Florida Health Care Coalition
Cheryl Damberg, PhD Director, Research and Quality Pacific Business Group on Health
Suzanne Delbanco, Executive Director The Leapfrog Group
Carolyn Pare, Chair Buyers Health Care Action Group
Consumer Groups
Chris Molnar, Director, Managed Care Education Community Service Society of New York
Ron Pollack, Executive Director Families USA
Lois Salisbury, JD, President Children Now
Nora Wells, MSEd, National Coordinator of Research Activities Family Voices
Providers and Plans
Steve Berman, MD, Immediate Past President American Academy of Pediatrics
Karen Brodsky, MHS, Vice President for Program Center for Health Care Strategies
Jack Ebeler, President and CEO
Jonathan Gelfand, Director, Quality Programs Alliance of Community Health Plans
Charles Homer, MD, Executive Director National Initiative for Children’s Healthcare Quality
George Isham, MD, Chief Health Officer HealthPartners
Julia Lear, PhD, Director Center for Health and Health Care in Schools
Blair Sadler, JD, President and CEO
Paul Kurtin, MD, Director, Center for Child Health Outcomes Children’s Hospital of San Diego
Ellen Schwalenstocker, MBA, Director, Child Health Quality National Association of Children’s Hospitals and Related Institutions
Dennis Sexton, President and CEO All Children’s Hospital
Others
Kenneth Kizer, MD, President and CEO
Elaine Power National Quality Forum
|
ACKNOWLEDGMENTS
|
|---|
I gratefully acknowledge the financial support of the David and Lucille Packard Foundation in carrying out this study.
|
FOOTNOTES
|
|---|
Received for publication May 13, 2003; Accepted Oct 2, 2003.
Address correspondence to Dale Shaller, MPA, 6381 Osgood Ave North, Stillwater, MN 55082. E-mail: dshaller{at}prodigy.net
|
REFERENCES
|
|---|
- Beal AC, Co JPT, Dougherty D, et al. Quality measures for children’s health care.
Pediatrics.2004; 113
:199
–209[Abstract/Free Full Text]
PEDIATRICS (ISSN 1098-4275). ©2004 by the American Academy of Pediatrics
This article has been cited by other articles:
|
|
|
|
 
H. Campbell, T. Duke, M. Weber, M. English, S. Carai, G. Tamburlini, and Pediatric Hospital Improvement Group
Global Initiatives for Improving Hospital Care for Children: State of the Art and Future Prospects
Pediatrics,
April 1, 2008;
121(4):
e984 - e992.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
S. E. Krug and M. Gausche-Hill
When There Is "No Room at the Inn": Potential Solution for Overcrowding in a Tertiary Pediatric Center
Arch Pediatr Adolesc Med,
September 1, 2007;
161(9):
911 - 913.
[Full Text]
[PDF]
|
|
|
|
|
|
|
T. W J Schulpen and K. M J Lombarts
Quality improvement of paediatric care in the Netherlands
Arch. Dis. Child.,
July 1, 2007;
92(7):
633 - 636.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
D. Dougherty and L. A. Simpson
Measuring the Quality of Children's Health Care: A Prerequisite to Action
Pediatrics,
January 1, 2004;
113(1/S1):
185 - 198.
[Abstract]
[Full Text]
[PDF]
|
|
|
TOP
ABSTRACT
METHODS
