Abstract
OBJECTIVES. With >6 million hospital stays, costing almost $50 billion annually, hospitalized children represent an important population for which most inpatient quality indicators are not applicable. Our aim was to develop indicators using inpatient administrative data to assess aspects of the quality of inpatient pediatric care and access to quality outpatient care.
METHODS. We adapted the Agency for Healthcare Research and Quality quality indicators, a publicly available set of measurement tools refined previously by our team, for a pediatric population. We systematically reviewed the literature for evidence regarding coding and construct validity specific to children. We then convened 4 expert panels to review and discuss the evidence and asked them to rate each indicator through a 2-stage modified Delphi process. From the 2000 and 2003 Agency for Healthcare Research and Quality Healthcare Cost and Utilization Project Kids' Inpatient Database, we generated national estimates for provider level indicators and for area level indicators.
RESULTS. Panelists recommended 18 indicators for inclusion in the pediatric quality indicator set based on overall usefulness for quality improvement efforts. The indicators included 13 hospital-level indicators, including 11 based on complications, 1 based on mortality, and 1 based on volume, as well as 5 area-level potentially preventable hospitalization indicators. National rates for all 18 of the indicators varied minimally between years. Rates in high-risk strata are notably higher than in the overall groups: in 2003 the decubitus ulcer pediatric quality indicator rate was 3.12 per 1000, whereas patients with limited mobility experienced a rate of 22.83. Trends in rates by age varied across pediatric quality indicators: short-term complications of diabetes increased with age, whereas admissions for gastroenteritis decreased with age.
CONCLUSIONS. Tracking potentially preventable complications and hospitalizations has the potential to help prioritize quality improvement efforts at both local and national levels, although additional validation research is needed to confirm the accuracy of coding.
With mounting efforts to monitor and improve quality of patient care and safety in the United States, health care quality evaluation for children has recently become a priority.1–4 In 2000, patients under the age of 18 years accounted for 18% of all of the hospitalizations in the United States, and >6-million children were hospitalized at an approximate cost of $46 billion.5
The uniqueness of children as a patient group leads to special requirements for assessing their health care quality.6 The epidemiology of ambulatory pediatric health care is very different from that of adult health care in that most children are relatively healthy, they seldom have multiple concurrent illnesses, and most medical encounters involve preventive care. In inpatient settings, the illnesses that children face, their treatment, and/or their sequelae are often unique to the pediatric population. Children depend on parents and other adults for their care. Demographically, children represent a very diverse group ranging from premature neonates to adolescents and are more likely to live in poverty than adults. Finally, they are in a constant state of physical, emotional, and cognitive development; therefore, issues important for one age group might be of less concern for another.2,7
These factors make it important for pediatric patients to have their own mechanisms for health care evaluation, but few quality measures have been developed specifically for children using routinely available hospital data.2 Also, few tools exist for assessing the quality of pediatric inpatient care at a national level except for those developed or promoted by the National Association of Children's Hospitals and Related Institutions (NACHRI), the Child Health Corporation of America, and other health quality institutions in the Pediatric Data Quality Systems collaboration.8–10
Beginning in 2001, the Agency for Healthcare Research and Quality (AHRQ) quality indicators (QIs) offered some tools to evaluate inpatient health care and access to health care. The original set of inpatient QIs contained 2 pediatric heart surgery indicators, and many of the initially released prevention QIs and patient safety indicators (PSIs) included children in their patient samples. The inpatient QIs and prevention QIs were developed together and include mortality indicators, volume and use indicators, and potentially preventable admission indicators.11 The PSIs provide rates for potentially preventable complications of care.12
The PSIs were used to evaluate pediatric patient safety using 1997 and 2000 Healthcare Cost and Utilization Project (HCUP) data in 2 studies by Miller et al.13,14 They found that potential patient safety events for pediatric patients occurred in significant numbers of pediatric inpatients and that these events were associated with increased length of stay, hospital charges, and risk of mortality. However, an important limitation of the studies by Miller et al13,14 was that they applied the AHRQ QIs by simply limiting the age group and did not tailor the indicators specifically for the pediatric population.
In addition, Sedman et al6 undertook a study that applied the original PSIs to the NACHRI database containing data from 1999 to 2002. They found that, whereas the PSIs could be relevant as screening tools for children's hospitals, the cases identified by the measures were not always preventable events and that certain measures were not accurate for a pediatric population.
With this evidence of the need for pediatric-specific QIs, our team (under contract with AHRQ) drew from the existing AHRQ QIs and other published work to develop a set of pediatric indicators using routinely collected hospital discharge data. In addition, we used these newly developed pediatric QIs (PDIs) to summarize aspects of the current health care quality landscape in the United States for pediatric patients.
METHODS
Indicator Development
The first phase of PDI development, summarized in this article, focused on adapting the existing AHRQ QIs to the pediatric population. A second phase, to be completed in late 2008, focuses on review and refinement of existing indicators from other sponsors, with a particular emphasis on the neonatal period. The AHRQ QIs are based on administrative data, specifically, data contained in the discharge abstract, which includes patient age; gender; International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) procedure and diagnosis codes; and admission, discharge, and procedure dates.
Based on the subset of AHRQ QIs potentially applicable to the pediatric population (ie, all of the indicators that did not focus on diseases specific to adults only, such as postoperative hip fracture), we developed the PDIs using a multifaceted approach similar to that described in previous work.11,12 Figure 1 summarizes the development process to define and assess potential indicators.
Indicator development process.
To redefine the original QIs for pediatrics, we first conducted a PubMed15 search supplemented by hand searching to synthesize the evidence for the candidate indicators and their underlying relationship to quality of care for children. Second, we conducted empirical analyses using the 2000–2003 HCUP nationwide inpatient sample, state inpatient databases, and Kids' Inpatient Database (KID)16 to explore alternative definitions and risk groupings. Third, we applied findings from chart reviews of pediatric hospitalizations flagged by the original AHRQ QIs, including a project led by NACHRI17 and similar efforts by users submitted to the AHRQ QI user support line, to tailor the original definitions more specifically to pediatric care. We also consulted with expert coders from the American Health Information Management Association to ensure proper interpretation of ICD-9-CM codes.
To establish face validity and refine indicator definitions further, we conducted clinical panel review modifying the Rand/University of California Los Angeles appropriateness method as applied previously in the development of the PSIs.12,18 We sought nominations of panelists from 44 professional clinical organizations and hospital associations, resulting in a pool of 125 potential panelists. Twenty-two revised pediatric indicators were reviewed by 1 of 4 clinical panels, each consisting of 8 to 13 physicians and nurse practitioners. We constructed panels for diversity in terms of clinical specialty and geographic location. Although individual panel composition varied based on what measures were being reviewed, the final panels had representatives from pediatric nursing, surgery (including pediatric, subspecialist, and pediatric subspecialist surgeons), ambulatory and/or community pediatricians, family practitioners, and inpatient pediatricians (including intensivists and hospitalists). Panelists received definitions and literature reviews for each indicator and were asked to rate the indicator on 7 dimensions of importance to quality measurement, including overall usefulness for quality improvement and overall usefulness for comparative purposes. All of the panelists received anonymized information about how their scores compared with those submitted by their colleagues and then participated in a facilitated conference call, during which they discussed the strengths and limitations of each indicator and evaluated potential modifications (which were adopted only by consensus). Panelists then confidentially rerated each indicator. Final ratings for usefulness informed decisions about inclusion in the final PDI set. In addition, additional empirical analyses suggested by the panelists further informed indicator specifications.
It should also be emphasized that all of the neonates with a birth weight of <500 g were excluded for the indicators, because mortality rates rise markedly below this threshold, and many of these infants do not receive aggressive treatment. This birth weight is commonly excluded from studies of neonatal outcomes.19–24 Obstetric patients were excluded, because their clinical situations differ from the majority of the patients identified by the PDIs and because they are covered by the obstetric-specific AHRQ QIs.
Indicator Rates
We estimated the national incidence of each PDI by applying the indicators to the 2000 and 2003 HCUP KID. The KID is the largest publicly available US all-payer database, with data from >3000 hospitals in 36 states. From each hospital, 10% of uncomplicated births and 80% of other discharges are sampled to create a data set of manageable size that retains the great majority of high-risk infants and children.16 Based on the KID sampling design, we weighted all of the results to generate estimates for the entire pediatric population of discharge subjects from US nonfederal hospitals.
Although not used in the analyses presented in this article, a de novo risk adjustment system (incorporating age, gender, high-risk strata, and comorbidity clusters based on the AHRQ clinical classification system) was developed for these indicators to allow for comparisons among hospitals, hospital types, areas, and other factors. This was done because no risk adjustment technique existed for pediatric quality measures using this type of data. This system and the adjusted results of our analyses will be discussed in a separate article. Adjusted results for the pediatric heart surgery measures, using the Risk Adjustment for Congenital Heart Surgery,25,26 will also be presented in that article. Rates presented in this article are all observed, unadjusted rates.
RESULTS
Indicator Set
Of the 30 indicators in the AHRQ QI potentially applicable to pediatrics, we eliminated the following 8 indicators based on initial empirical, literature, and user review: pneumonia mortality is extremely rare in children; postoperative pulmonary embolism and/or deep vein thrombosis events are clinically different in children (typically central line thromboses), invalidating the indicator's original quality rationale; and failure to rescue, death in low-mortality diagnosis-related groups, and complications of anesthesia require extensive redefinition and still may not be valid for children, given evidence from user input and chart review findings. Also, the 3 obstetric indicators were felt to be more appropriate for evaluation of all patients. Although, certainly, pregnant adolescents usually face a different and much larger set of issues than their adult counterparts, in terms of the quality issues addressed by these QIs, there is minimal evidence showing a meaningful clinical difference between adolescent and adult obstetric patients. For example, studies have shown that the risks for perineal lacerations in adolescents are similar to or lower than those of women aged 25 to 34 years,27 and primiparity (independent of age) is one of the major risk factors for perineal injury.28
Clinical panels reviewed the remaining 22 indicators and raised major concerns regarding ∼4 indicators (Table 1) that could not be addressed with changes to the definitions, either within the project time frame or at all. Table 2 summarizes the final pediatric-tailored definitions for the 18 indicators recommended for inclusion in the AHRQ PDIs because of acceptable ratings from clinical panels and reasonable evidence from literature findings (eg, studies confirming the significant incidence of the QI events in pediatric populations,6,13,14 studies highlighting ways to reduce these events as an indication of their potential preventability,29–37 etc), empirical analyses, and expert coding consultation. Thirteen of these indicators are intended for use at the hospital level, whereas the other 5 are area-level indicators, intended to reflect potential problems in access to high-quality outpatient care. For several indicators (eg, postoperative sepsis and decubitus ulcer), panelists noted that the indicators are of minimal value when excluding the high-risk populations, as was done in the previous AHRQ QIs. In a pediatric population, unlike in adults, uncomplicated patients are much less likely to develop the types of outcomes detected by these QIs. Panelists suggested that quality improvement interventions are best aimed at populations that are more likely to develop a complication. In addition, focusing only on low-risk populations reduces the outcome rate to a level where meaningful comparisons over time and institutions may be difficult.
Other Indicators Considered for AHRQ PDI
Indicators Selected for Inclusion in AHRQ PDI
The inclusion of high-risk patients required more attention to risk stratification. The clinical risk strata for the accidental puncture and laceration indicator were based on area or organ system of surgery, as well as major diagnostic category groupings. For decubitus ulcer, patients with a diagnosis reflecting paralysis or significantly decreased mobility were placed in the high-risk group. Patients with clinically significant coagulopathies were stratified to the high-risk group in postoperative hemorrhage or hematoma, and for postoperative sepsis and infection because of medical care, patients were grouped into strata according to their immunocompromised status. Postoperative sepsis also stratified patients based on the type of surgery (eg, clean and elective versus potentially contaminated and nonelective).
The panelists' advice to include high-risk populations in children heightened concerns about the appropriate use of the indicators. We asked panelists for 2 overall usefulness ratings, 1 for quality improvement and 1 for comparative reporting. In general, panelists were more conservative in the ratings for comparative reporting.
Assessing National Rates for PDIs
Figure 2 shows 16 of the 18 PDIs. The first 12 indicators shown represent 16520 and 18003 potentially preventable complications occurring in hospitalized patients in 2000 and 2003, respectively. The last 4 area level indicators sum to ∼300000 hospital admissions in each year sensitive to both the quality of and access to outpatient care. Two PDIs, transfusion reaction and foreign body left in, had extremely low event rates of <0.1 per 1000 patients. Another 4 hospital measures (accidental puncture and laceration, iatrogenic pneumothorax-neonate, iatrogenic pneumothorax-nonneonate, and postoperative wound dehiscence) and 2 area indicators (diabetes-short-term complications and urinary tract infection) exhibited event rates under 1 per 1000 eligible admissions for hospital level indicators or geographic population for area level indicators. The remaining indicators had rates ranging from 1.68 to 26.12 per 1000. All of the rates remained relatively constant from 2000 to 2003, with the possible exception of postoperative sepsis, for which the observed rate increase may be attributable to known changes in coding guidance.38 The rate of postoperative respiratory failure also increased, for less clear reasons.
National rates for PDIs based on KID 2000 and 2003.
Rates from 2003 for indicators with stratified risk subgroups are shown in Fig 3. In the postoperative sepsis measure, rates were almost double for the highest-risk clinical strata of patients, those who undergo potentially contaminated nonelective surgery, compared with the overall rate. Rates of ∼20 per 1000 high-risk patients were observed for postoperative hemorrhage and hematoma, decubitus ulcer, and selected infection attributable to medical care.
Rates for PDIs with strata from KID 2003.
The only indicators not shown in the figures are perforated appendix and pediatric heart surgery volume: the perforated appendix measure has a very different and highly selected denominator, that is, drawing from children with appendectomies (versus other area indicators that draw from virtually all admissions), and pediatric heart surgery volume is not identifying an outcome or rate, rather simply the number of surgeries performed by an institution. More than 23000 children experienced a perforated appendix annually. The perforated appendix PDI rate was particularly high at ∼1 in 3 children with appendectomies (0.31 in both years). Pediatric heart surgery volume was of interest because of the previously demonstrated volume-outcome relationship, which allows volume to be used as a potential proxy for quality. Almost 25000 pediatric heart operations are performed annually in the United States, with the range of number of surgeries per institution spanning from <10 per year to >1500 per year.16
Observed rates also varied by age, in fairly predictable ways. In 2003, rates per 1000 increased by age for 3 indicators: decubitus ulcer (0.86 for 29 days to <1 year vs 1.86 for 3–5 years and 4.89 for 13–17 years), foreign body left in during procedure (∼0.03 for younger age groups to 0.06 for teenagers), and diabetes short-term complications (0.23 for 6- to 12-year-olds vs 0.41 for 13- to 17-year-olds). In contrast, rates decreased by age for 9 indicators, with the most pronounced variation seen in pediatric gastroenteritis (from 10.30 in the 61-day to <1-year group to 0.32 in the 13- to 17-year-olds), pediatric asthma (6.24 in 2-year-olds to 0.68 in teens age 13–17 years), and urinary tract infection (with a precipitous drop from 4.10 in the youngest age bracket to 0.75 for 1- to 2-year-olds and then ∼0.3 for children ≥3 years). Other rates that decreased with age were: postoperative hemorrhage or hematoma, postoperative respiratory failure, postoperative sepsis, pediatric heart surgery mortality, and perforated appendix. The rate of postoperative wound dehiscence was high in infants (<1 year of age) at 1.70, dropped to 0.22 in the 3- to 5-year-olds, and then climbed again to 0.49 in the 13- to 17-year-olds. The rate of neonatal iatrogenic pneumothorax varied by birth weight (2.08 for 500–999 g, 0.45 for 1000–1499 g, 0.23 for 1500–1999 g, and 0.08 for 2000–2499 g). The remaining indicators had relatively similar rates across age groups.
DISCUSSION
We developed a set of 18 pediatric indicators based exclusively on routinely collected hospital data that capture a number of important quality and patient safety concerns. In February 2006, these indicators were publicly released as the AHRQ PDIs, establishing an indicator set tailored specifically for pediatric inpatient care. This article provides the first national estimates of the incidence of potential patient safety events based on these indicators.
These indicators can be used by analysts and organizations that may not have the resources to collect detailed clinical data, including individual hospitals or hospital systems, researchers, and state or regional health agencies. In addition, nearly all of the indicators are “cross-cutting,” meaning that they apply to large numbers of hospitalized children, and, as a result, potentially provide tools to screen overall quality of care, even for hospitals that do not treat highly specialized conditions. These indicators may be suitable for use in quality improvement activities. Hospitals may use the indicators to screen for potential quality concerns and to investigate further to improve in-house quality. Potential quality improvement strategies are also readily available in the literature for complications like decubitus ulcer, neonatal pneumothorax, and infection because of medical care (hospital protocols on risk assessment and repositioning for decubitus ulcer29–31,39); antenatal steroids, appropriate surfactant administration, and appropriate ventilation for neonatal pneumothorax32–35; and protocols on line changes and removal for infection because of medical care.36,37 The indicators may also be appropriate for comparative reporting and more useful in quality improvement work with the addition of a novel risk adjustment system for pediatric comorbidities and socioeconomic status, which was not used here but will be presented in a separate article.
Risk adjustment and stratification of patients open the door for a variety of uses. The stratification schemes available for 6 indicators allow hospitals to conduct analyses by risk group to identify more easily potential targets for improvement. Based on previous experience with publicly available QIs, the pediatric indicators are also likely to be used for comparing hospitals, either for research or reporting purposes. Our clinical panels rated most of the indicators as potentially useful for comparative reporting, although some concerns about validity were expressed. Use in a public reporting arena should probably await additional validation work, using medical chart reviews to establish sensitivity and predictive value.
Our research and the resulting indicators have several important limitations. First, indicators based on administrative data rely on the completeness of physician charting, as well as the accuracy and completeness of ICD-9-CM coding. Without adequate adherence to coding standards and consistent use of codes, hospitals may look better or worse because of issues in coding as opposed to actual quality of care. Second, the limited information contained in administrative data precludes risk adjustment using potentially valuable clinical information, such as physiologic or laboratory measurements, or other important considerations, such as social factors and family support. Because this first phase of the PDIs was based on existing AHRQ QIs, the indicator set does not cover some important domains of care, such as mental health, a major source of admissions and cost for older children and adolescents.40
To further understand and address these limitations, future research is essential. Although we have established face validity41 through our panel of experts, more extensive validity testing remains to be done. These validation studies of the indicators are needed to understand the ability of administrative data to capture these important events, to identify interventions that can improve quality of care, and to establish the completeness of the risk adjustment for the PDIs. NACHRI has undertaken a project to investigate the sensitivity and specificity of the measures using chart reviews,42 and final results of these efforts and other potential studies will help inform users about the accuracy of the measures. Also, a future article will address indicators of neonatal care quality, which have been identified in the second phase of PDI development.
The data shown in this article offer a first level of construct validation.41 High-risk groups consistently demonstrate significantly higher rates of each complication than the general population. For example, pediatric patients with limited mobility had rates of decubitus ulcers that were >7 times greater than that in the overall eligible patient population. Patients with coagulopathies had rates of postoperative hemorrhage and hematoma ∼10 times that of the overall eligible population. Patients who underwent nonelective surgeries in potentially contaminated fields had substantially higher rates of postoperative sepsis than the overall group.
The age trends are also as one might predict from a clinical perspective. Admission rates for dehydration, as identified by that PDI, for example, are much higher in infants and younger children than in teens. This finding is congruent with clinicians' practices, because younger patients are at greater risk for this complication when they become ill with gastroenteritis or other infections. The rates for perforated appendix also decline with age. This finding is also consistent with other literature, indicating that older children are better able to describe their symptoms and are more likely to receive timely intervention because of this ability.43,44 Similarly, the rates for iatrogenic pneumothorax-neonate increase in decreasing birth weight groups. Again, this is as one might predict clinically, because the smallest infants are those at greatest risk for this complication (because of greater fragility of their lungs and greater chances of requiring mechanical ventilation). Thus, whereas the administrative data are imperfect, and further validation work is pending, the expected trends and patterns shown by the PDI rates supplement literature evidence and clinical panel review findings supporting the ability of these QIs to identify potentially important quality problems.
These indicators, coupled with the risk adjustment method developed for them, offer a potentially useful set of tools that addresses the quality of health care for children. Although the use of administrative data offers broad national coverage and allows for any interested party to apply the PDIs at low cost, further research to establish the criterion validity of these indicators is essential. Focusing on the issue of preventability, it will be particularly important to establish whether and how these indicators can be used to help facilities and health care professionals improve the care that they provide to their pediatric patients.
Acknowledgments
This project was funded by a contract from the Agency for Healthcare Research and Quality (290-04-0020).
Footnotes
- Accepted March 28, 2008.
- Address correspondence to Corinna A. Haberland, MD, MS, Health Research and Policy, T139, HRP/Redwood Building, Stanford, CA 94305. E-mail: corinnah{at}stanford.edu
Financial Disclosure: All of the authors, except for Ms Ku, are currently contracted, or subcontracted with the Agency for Healthcare Research and Quality for the refinement and ongoing support of the Pediatric Quality Indicators.
What's Known on This Subject
Quality measurement in health care is a rapidly expanding field. However, most inpatient quality measures are not applicable to the pediatric population, despite >6 million hospital stays per year for children.
What This Study Adds
We offer a description of the development and specifications of the Agency for Healthcare Research and Quality pediatric quality indicators, as well as preliminary national rates for these measures.
REFERENCES
- Copyright © 2008 by the American Academy of Pediatrics
In this issue
Jump to section
Related Articles
Cited By...
- Development and Validation of an Agency for Healthcare Research and Quality Indicator for Mortality After Congenital Heart Surgery Harmonized With Risk Adjustment for Congenital Heart Surgery (RACHS-1) Methodology
- Recognizing Differences in Hospital Quality Performance for Pediatric Inpatient Care
- Cost Saving and Quality of Care in a Pediatric Accountable Care Organization
- Primary care quality indicators for children: measuring quality in UK general practice
More in this TOC Section
Similar Articles
Keywords
- adverse events
- health services research
- pediatric
- quality improvement
- quality of care
- NACHRI—National Association of Children's Hospitals and Related Institutions
- AHRQ—Agency for Healthcare Research and Quality
- QI—quality indicator
- PSI—patient safety indicators
- HCUP—Healthcare Cost and Utilization Project
- PDI—pediatric quality indicator
- ICD-9-CM—International Classification of Diseases, Ninth Revision, Clinical Modification
- KID—Kids' Inpatient Database