Risk Adjustment for Pediatric Quality Indicators

ISSUES IN RISK-ADJUSTING PEDIATRIC QUALITY MEASURES

Many characteristics make children special: relatively low mortality and morbidity rates; specialized pediatric services (eg, neonatal intensive care unit [NICU] care); use of providers and institutions outside the traditional health care delivery system (eg, school health clinics); and the need for adult help and supportive environments. These considerations often make risk adjustment more challenging for children than adults. Importantly, these factors also complicate efforts to define pediatric quality measures regardless of risk. Several factors are especially important when designing risk adjustment for pediatric quality measures.

The first factor is the epidemiology of childhood disease and chronic conditions. Except for specialty clinics, the number of children with any particular serious health condition is likely to be small, especially for individual providers. Even for relatively common chronic conditions such as asthma, numbers of patients may be too small for individual providers to support QIs or adjust for risk. Most children are hospitalized relatively rarely; hospital-based quality measures will be less useful than for an adult population. Nonetheless, comparisons between hospitals or hospital units are still useful in children, especially when their care is concentrated (such as comparing NICU outcomes or the results of congenital heart disease surgery). These comparisons require appropriate adjustments.⁶

Second, the meaning and use of diagnoses, drugs, and procedures are sometimes different for children compared with adults. Although diagnoses can serve as important risk adjusters for the high percentage of older adults with multiple conditions, symptoms, developmental milestones, and functional status may be more pertinent for children. The standard diagnosis coding nomenclature in administrative databases, the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM)⁷ does not contain codes for many of these developmental or functional attributes. ICD-9-CM codes for symptoms and signs give little indication of condition severity or chronicity. In addition, the meaning of codes may be different for children and adults (eg, murmur and sleep disturbance). Most studies of coding completeness and accuracy have targeted Medicare populations,^8,9 as have most regulations involving auditing and oversight of coding and medical record documentation. Diagnosis coding and medical record documentation may be even less accurate and clinically meaningful for children than adults.

Researchers recently developed risk-adjusted tools based on pharmacy claims. Patterns of medication use are different in adults and children.^10,11 Many drugs are not tested in children, and as such, pediatric use is less evidence-based. As for adults, some conditions are undercoded (eg, acquired immunodeficiency syndrome and mental health conditions); whether this impact is greater in children for whom socially disadvantageous labels may have long-lasting effects is unknown. On the other hand, parents may seek diagnoses to obtain services (eg, attention-deficit disorder or dyslexia) through special education or other programs.

Third, most health care encounter databases include services covered by one private or public insurer. However, children often use services from diverse settings within the health care delivery system and outside of it. Use of school and public health clinics is generally not included in administrative databases; medical records in private doctors’ offices do not link automatically with those in school or public health settings. Thus, children who use early intervention, public health, and school-based clinics may seem healthier than they actually are simply because existing data sets do not include all health-related information. These potential biases could compromise risk adjustment.

A related concern is that children use both the pediatric care system (including pediatric subspecialists, general pediatricians, and children’s hospitals) and the adult care system. Because the pediatric care system likely serves children with more severe conditions, risk adjustment may be particularly important when comparing pediatric and adult providers.

Finally, family and community factors strongly influence child utilization and health status. Risk therefore should be measured, at least in part, at the family or parental level. Familial attributes may be particularly important for process-of-care measures. Because of the complexity of identifying and quantifying child outcomes, process measures are likely to be used more commonly than outcomes measures as QIs for children. Timely delivery of preventive services is measured easily and is a common QI for children. Although virtually all children should be immunized, characteristics of the family (eg, living with mother and maternal grandmother, cash assistance eligibility, father living in home with child, presence of an additional preschool sibling, and use of Supplemental Nutrition Program for Women, Infants, and Children) are associated with reduced immunization rates.^12,13 As noted above, stratification may be the appropriate first step for analyzing differences in process measures, but few databases contain information on these critical familial and environmental risk factors. One promising possible solution makes use of data on socioeconomic status obtained from the census to adjust performance measures such as physician profiles.¹⁴

The prominent role of the family and environment in which children live is tied to the concerns about equity in the delivery of medical care raised by the Institute of Medicine.¹⁵ Unless these contextual factors are recognized and adjusted for, providers and plans seeing disproportionate numbers of relatively sick and vulnerable children could fare poorly in quality measurement; they may seem as if they are giving substandard care. This could unfairly penalize some providers and make others less inclined to care for disadvantaged children. On the other hand, as noted above, one would not want to hold providers to a lower standard of care for any subpopulations. As such, the decision about risk adjustment depends on both the conceptual model guiding its use and the quality measure’s purpose. Because the requisite data on these risk factors are generally unavailable, comparisons of performance among providers caring for children across the socioeconomic spectrum must proceed cautiously.

RISK-ADJUSTMENT METHODOLOGIES FOR CHILDREN

Here we briefly review existing techniques and examples for adjusting pediatric quality measures. Risk adjustment varies depending on the specific quality measure; we present several available pediatric quality measures only as examples. A more complete review of pediatric quality measurement per se is in Beal et al.¹⁶ We consider risk adjustment in 3 categories: 1) generic risk adjustment using administrative data; 2) risk adjustment for specific conditions or settings using administrative or clinical data; and 3) risk adjustment found in major quality-measurement data sets.

GAPS AND IMPORTANT RESEARCH AREAS FOR PEDIATRIC RISK ADJUSTMENT

How much risk adjustment is adequate to “level the playing field?” The answer to this question is: It depends on the quality measure itself and how the information will be used. Even with the relative sophistication of risk-adjustment methods for NICU, PICU, and congenital heart disease deaths, for example, it is still probably unwise to make final decisions about the quality of NICU, PICU, or heart surgery programs on this basis alone. Unmeasured risk factors may still differ across providers (eg, differences in “do-not-resuscitate” practices, at least partially a reflection of patients’ preferences for life-sustaining care). Risk-adjustment methods that account for the most important risk factors are more likely to facilitate drawing inferences about quality of care than methods that account minimally for pertinent risk factors.

Risk adjustment for pediatric quality measures has received little scrutiny. The areas that have generated more attention (NICU and PICU outcomes) represent acute, high-technology care, rarely needed by children. Nonetheless, these adjustment systems are, in general, well tested and validated. Given the well-developed status of ICU outcomes adjustment methods, large-scale risk-adjusted comparisons of ICU outcomes could be used to motivate improvement in the quality of NICU and PICU care. On the other hand, more work is needed on applying risk adjusters in diverse practice settings, in different sociodemographic populations, and across a variety of quality dimensions. As new quality measures are derived, developers must consider which risk factors are pertinent and produce risk-adjustment strategies appropriate to each particular measure.

Despite the conceptual rationale for risk adjustment specific to individual quality measures, reality intervenes. Developing risk adjusters de novo can be expensive, often requiring extensive clinical and methodological expertise, creation of large databases for empirical testing and validation, or data that are difficult or impossible to collect. It therefore may be reasonable to examine the utility of existing risk adjusters for a particular quality measure, recognizing the conceptual limitations and therefore interpreting results with appropriate caveats. An example involves the risk adjusters developed for predicting future resource consumption and setting capitated payments. Considerable effort has gone into their development and statistical validation; they rely on readily available administrative databases. Testing their validity for risk-adjusting pediatric quality measures seems reasonable, especially for measures relating to resource consumption (eg, hospitalization rates). One attraction of these approaches for pediatrics is their consideration of the totality of all diagnoses: because relatively few children might have one specific diagnosis, these risk adjusters overcome problems of small disease-specific sample sizes. The obvious drawback is their focus on resource consumption.

As noted at the outset, risk adjustment becomes necessary when people with different risk profiles are not assigned randomly to the “unit of observation” (ie, the groups whose performance one wants to compare). Relevant units of observation for children may differ from those for adults. For children we may care more about schools, health department jurisdictions, provider groups, or communities than hospitals. In addition, we need to better understand the impact on health risks and thus risk adjustment of the large numbers of children with no health care or with care only for prevention and/or acute illness or injuries. Given current data and information systems, it may prove impossible to capture pertinent risk factors for children across different units of observation. Therefore, research must consider how unmeasured risk factors affect perceptions of relative quality across diverse units of observation.

Closely related is the need to know more about the role of family and community characteristics as risk adjusters for pediatric quality measures. As noted above, children depend heavily on their families and communities, and these factors therefore are potentially important risk adjusters. Information from the census and other sources can provide important community information. Parental information may be available in the same data sets as child information when parents and children are insured by the same plan. When parents report on quality, parents can answer questions about themselves and their families.

Linking data systems among institutions that could provide information about child health, such as schools, public health departments, and children’s hospitals, would be an enormous undertaking. However, sharing data will be important to obtaining community-level views of child health quality and various risk factors. Defining relevant roles for various organizations (eg, school and public health departments, Medicaid programs, State Children’s Health Insurance Program, pediatric health centers, and children’s hospitals) and the data available from each might be a good starting point. These issues extend beyond the narrow topic of risk adjustment, highlighting barriers to obtaining comprehensive information about child health, let alone the quality of pediatric care. Risk adjustment is limited inherently by inadequate or unavailable data.

A variety of technical issues also need more study. Many efforts to compare provider performance must contend with small sample sizes; individual providers frequently treat too few patients of a given type each year to generate accurate and stable estimates of their long-term average performance with such patients. Better ways to estimate expected outcomes, such as empirical Bayes and hierarchical modeling, need more investigation. Simultaneous concerns about risk adjustment and small sample sizes can trap comparative efforts between “a rock and a hard place.” The main strategy for increasing the accuracy of estimates is including more cases, but this generally increases population heterogeneity and makes risk adjustment both more necessary and more problematic.

Thinking about valid risk adjustment is premature without assuring the validity of the quality measure to be risk adjusted. Risk adjustment aims to control for patient factors outside provider control so that residual differences in performance could, potentially, reflect quality differences. Few measures currently used as ostensible QIs have undergone such scrutiny. For example, risk-adjusted NICU and PICU mortality rates are putative quality measures, but no one really knows whether differences across institutions reflect quality problems that, if resolved, would lower mortality rates. Research about validity of quality measures themselves should be linked directly to risk-adjustment research: they should go hand in hand. Methods must be validated across time and settings of care (eg, academic health centers versus community facilities). Changes in patterns of care (eg, diagnostic testing or therapeutic interventions) and differences across settings could affect the type of data available for risk adjustment and the relationships between the risk factors and outcome of interest. Risk adjusters should generally be updated periodically, requiring an ongoing research commitment and investment.

Finally, we need to better understand how risk-adjusted quality measures will be used and disseminated. Risk adjustment will never be perfect; it is impossible to adjust for all relevant risk factors. Understanding better how to disseminate admittedly imperfect information to clinicians, parents, plans, purchasers, and even older children requires more study. Given the ultimate goal of improving child health care, it is critical that battles over measurement methods not unduly divert resources and attention, as has happened sometimes in adult settings.