 |
|
 |
|
 |
 |
|
 |
 |
 |
 |
 |
 |
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ARTICLE |
a Division of Pediatric Critical Care, Medical College of Wisconsin, Milwaukee, Wisconsin
b National Association of Children's Hospitals and Related Institutions, Alexandria, Virginia
c Division of Critical Care Medicine, University of Texas Southwestern Medical Center, Dallas, Texas
d Division of Pediatric Nephrology, Department of Pediatrics, University of Michigan Health System, Ann Arbor, Michigan
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
METHODS. The Agency for Healthcare Research and Quality pediatric quality indicator algorithms were applied to 76 children's hospital's discharge abstract data (1 794 675 discharges) from 2003 to 2005. Rates were calculated for 11 of the pediatric quality indicators from all 3 years of discharge data: accidental puncture or laceration, decubitus ulcer, foreign body left in during a procedure, iatrogenic pneumothorax in neonates at risk, iatrogenic pneumothorax in nonneonates, postoperative hemorrhage or hematoma, postoperative respiratory failure, postoperative sepsis, postoperative wound dehiscence, selected infections caused by medical care, and transfusion reaction. Subsequently, clinicians from 28 children's hospitals reviewed 1703 charts in which complications had been identified. They answered questions as to correctness of secondary diagnoses that were associated with the indicator, whether a complication was already present on admission, and whether that complication was preventable, nonpreventable, or uncertain.
RESULTS. Across 3 years of data the rates of pediatric quality indicators ranged from a low of 0.01/1000 discharges for transfusion reaction to a high of 35/1000 for postoperative respiratory failure, with a median value of 1.85/1000 for the 11 pediatric quality indicators. Indicators were often already present on admission and ranged from 43% for infection caused by medical care to 0% for iatrogenic pneumothorax in neonates, with a median value of 16.9%. Positive predictive value for the subset of pediatric quality indicators occurring after admission was highest for decubitus ulcer (51%) and infection caused by medical care (40%). Because of the very large numbers of cases identified and its low preventability, the indicator postoperative respiratory failure is particularly problematic. The initial definition includes all children on ventilators postoperatively for >4 days with few exclusions. Being on a ventilator for 4 days would be a normal occurrence for many children with extensive surgery; therefore, the majority of the time does not indicate a complication and makes the indicator inappropriate.
CONCLUSIONS. A subset of pediatric quality indicators derived from administrative data are reasonable screening tools to help hospitals prioritize chart review and subsequent improvement projects. However, in their present form, true preventability of these complications is relatively low; therefore, the indicators are not useful for public hospital comparison. Identifying which complications are present on admission versus those that occur within the hospitalization will be essential, along with adequate risk adjustment, for any valid comparison between institutions. Infection caused by medical care and decubitus ulcers are clinically important indicators once the present-on-admission status is determined. These complications cause significant morbidity in hospitalized children, and research has shown a high level of preventability. The pediatric quality indicator software can help children's hospitals objectively review their cases and target improvement activities appropriately. The postoperative-respiratory-failure indicator does not represent a complication in the majority of cases and, therefore, should not be included for hospital screening or public comparison. Chart review should become part of the development process for quality indicators to avoid inappropriate conclusions that misdirect quality-improvement resources.
Key Words: quality • indicator • AHRQ • complications
Abbreviations: AHRQ—Agency for Healthcare Research and Quality • PSI—patient safety indicator • PDI—pediatric quality indicator • POA—present on admission • PPV—positive predictive value
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
In response to the growing demand for quality metrics, a number of agencies and groups have advanced new indicators for measuring quality and safety. These include the Joint Commission's National Patient Safety Goals and the Agency for Healthcare Research and Quality's (AHRQ's) patient safety indicators (PSIs), inpatient quality indicators, and pediatric quality indicators (PDIs).3,4
Central to the use of any of these metrics is the premise that the measures are meaningful for use with the targeted population. That is, a set of measures intended to accurately reflect the quality of a pediatric population should be valid and reliable and, ideally, should guide both improvement and accountability activities.5,6 That a published measure set is a valid assessment of quality care should not be assumed; evaluations of the AHRQ PSIs in the pediatric population revealed potential value in some areas for guiding improvement projects, yet there were significant issues that limited their utility for public reporting.7,8
Subsequent to the release of the AHRQ PSIs and follow-up published reviews,7,8 a pediatric-specific set of measures was released by the AHRQ.9 The PDIs were designed exclusively for pediatric patients and intended to "screen for problems that pediatric patients experience as a result of exposure to the health care system and that may be amenable to prevention by changes at the system or provider level."9 The aim of this project was to assess the validity and reliability of the AHRQ provider-level PDIs in a cohort of children's hospitals by using chart-review methodology.
|
METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
The PDIs use a logistic regression risk-adjustment model that takes into account the age, gender, reason for admission (diagnosis-related group), and comorbidities of individual cases. Regression coefficients used in the models were developed by using only pediatric cases from Health Care Utilization Project state inpatient databases, and there is a different set of coefficients for each individual PDI. Previously, any AHRQ quality indicators that could have been applied to pediatric cases used the same coefficients developed by using an all-age database, so this method is an improvement. There are also some indicator-specific stratifications that are part of the AHRQ PDI software that attempt to further the ability to compare indicator rates across providers9 (Table 1).
|
For each PDI there were 10 chart reviews by each of the 28 hospitals. Hospitals were asked to review the most recent cases first; therefore, case selection was chronological and not random. However, for a number of PDIs, some hospitals did not have 10 cases over 3 years and, therefore, reviewed fewer cases. Area-level PDIs (asthma admission rate, diabetes short-term complication rate, gastroenteritis admission rate, perforated appendix admission rate, and urinary tract infection admission rate) were excluded from this study, as were the 2 cardiac surgery provider-level PDIs (cardiac surgery volume and mortality rates) because the cardiac physicians were performing a separate analysis of those indicators to compare them to physiologic databases. A total of 1703 charts representing 1890 potentially preventable events were reviewed by the participating 28 children's hospitals.
For each PDI-positive screen, the participants at each hospital received patient and encounter identifiers. For each of the reviewed cases, the clinician obtained a core data set including:
For each of the reviewed PDIs, a set of general and PDI-specific questions were applied. These questions were developed by a panel of pediatric experts before the initiation of data collection and included a clinical assessment of preventability. If the complication was deemed "not preventable," the reviewer was asked to list the reason why. There was no formal training in identifying preventability, although discussions took place during teleconferences, and problematic cases were used as examples. After completion of review, the data were submitted back to the National Association of Children's Hospitals and Related Institutions for analysis without inclusion of any protected health information.
Descriptive and comparative analysis was performed by using SPSS Base 12.0 (SPSS Inc, Chicago, IL). Cases that were deemed present on admission (POA) were removed from the analysis of preventability. For example, a child who had a chronic Broviac line in place and was admitted with an infection in the line would have "infection caused by medical care" counted as already present (POA) and would not be counted as occurring within the hospitalization; therefore, such an event was not preventable during the hospitalization. There were also cases for which secondary complication codes were recorded incorrectly, such as when an absorbable suture was purposely left in and then coded as a "foreign body left in during procedure." In both these examples, the statistics were applied after the incorrect cases were removed. Positive predictive values (PPVs) were calculated for each PDI by dividing the true-positive results over the sum of the true-positive and false-positive results.10 In each case, the lower PPV estimates used only those events identified as clearly preventable in the calculation. In the high estimates, those events classified as being of uncertain preventability were included with the clearly preventable in the numerator of the equation.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
|
The AHRQ PDIs are identified on the basis of coded secondary diagnoses or procedures. Therefore, for each reviewed PDI, clinicians were asked to indicate whether the secondary diagnosis or procedure associated with the identified event was correct. The rate of incorrect secondary diagnoses/procedures ranged from 0% for PDI13 (transfusion reaction) to 25.7% for PDI9 (postoperative respiratory failure) (Table 2).
For those cases in which the secondary diagnosis or procedure associated with the identified event was correct, investigators were then required to identify whether the event that indicated a positive PDI screen was present at the time of admission to the hospital. This is critical, because a majority of hospitalized children have chronic disease and, therefore, often have chronic intravenous lines in place that become infected at home or have decubiti that form outside the hospital. These are important clinical problems; however, the PDI system is meant to count problems that occur because of inadequate hospital care. Therefore, conditions that are POA must be removed from the data to give hospitals appropriate information. This POA status ranged from 0% in PDI4 (iatrogenic pneumothorax in neonates at risk) to 43% for PDI12 (infection caused by medical care) (Table 2).
Finally, for those cases in which the event was not POA, investigators were charged with determining whether the potential events identified by the PDI software were preventable. Events were classified as clearly preventable, clearly not preventable, or uncertain. Events identified as clearly preventable ranged from 0% for PDI13 (transfusion reaction) to 51.45% for PDI2 (decubitus ulcer). Events identified as clearly not preventable ranged from 19.31% for PDI12 (selected infections caused by medical care) to 80% for both PDI4 (iatrogenic pneumothorax in neonates at risk) and PDI13 (transfusion reaction).
For each PDI chart review, clinicians were asked to comment on problems with the indicators to determine if the indicator actually captured an event and why. If the clinicians categorized the event as nonpreventable, they were asked why they felt the event occurred. The comments are summarized for each PDI in the paragraphs below. Brief definitions of the denominator and numerator are included. The full definitions are available in the online manual.9 The total number of events in 76 children's hospitals over 3 years is listed after the title of each PDI. The rates per 1000 discharges, rates per year in 76 children's hospitals, and average rate per children's hospital per year are listed in Table 2.
PDI1: Accidental Puncture/Laceration (n = 3168)
The denominator is all medical and surgical patients aged 0 to 17 years; the numerator includes patients who had a secondary diagnosis code for accidental puncture or laceration during a procedure.
Seven percent of the children had the problem before they were admitted to the hospital (ie, POA). This is common in children's hospitals, to which many patients are referred for complications that occur in other institutions. Clinician reviewers felt that these events were not clearly preventable 68% of the time. Of 295 events, 20 occurred during lysis of adhesions, for which operative notes described significant scarring present in children who had multiple surgeries before this particular event. Thirty-eight percent of the time, the laceration occurred during gastrointestinal procedures and 29% of the time during cardiovascular or thoracic surgery. Comments often included documentation that these were children who had congenital abnormalities such as gastroschisis, omphalocele, diaphragmatic hernias, cloacal defects, and cardiac defects who were coming back into the hospital for 1 of multiple procedures that had previously involved significant scarring. Although the laceration or puncture was not a therapeutic part of the surgery, clinicians felt that they may have been unavoidable for the surgeon to do what was necessary. Incidents that were clearly listed as preventable were often complications of line or device placements that subsequently punctured vessels, lungs, or the gastrointestinal tract.
PDI2: Decubitus Ulcer (n = 1688)
The denominator is all medical and surgical patients aged 0 to 17 years with a length of stay >5 days, and the numerator is any patient with decubiti noted on his or her record. In 40% of the cases, children were identified as having the decubiti before they were admitted; therefore, the event was not preventable within the hospitalization. For events not determined to be POA, clinician reviewers listed the decubiti as not clearly preventable during the current hospitalization 49% of the time. A number of cases were questioned by the clinicians because of confusion at what level skin disruption actually should be defined as a decubitus. For example, a child with myelogenous leukemia came in with cutaneous involvement and subsequently had skin breakdown that was coded as decubitus in the chart but was not considered preventable by the clinician reviewer. Also, there is confusion between skin irritation or breakdown occurring because of irritation from friction or tape and actual decubiti, which clinicians usually associate with pressure injury. Therefore, especially in small infants, there is significant variation in coding. A number of decubiti occurred on the occiput of infants after cardiac surgery or who were on extracorporeal membrane oxygenation, but the majority occurred on the sacrum or heels. Although this study did not include what percentage of children needed skin grafts and extensive therapy to recover, we know from other chart reviews that this is the case for many children who suffer from decubiti.
PDI3: Foreign Body Left in During Procedure (n = 153)
The denominator is all medical and surgical patients aged <17 years, and the numerator is those individuals who are coded to have a foreign body accidentally left in during a procedure. The indicator was POA 19% of the time, often a result of transfers because of complications at other hospitals.
This is a rare event (0.089/1000 patients) for which reviewers felt the trigger code was incorrect 22% of the time and was felt to be preventable only 44% of the time. A majority of these events do not represent actual foreign bodies (such a sponges "forgotten" by the surgeon); instead, they represent catheters, screws, drains, etc that "broke off" during procedures or drains or sutures that were purposely left in the body. Of 72 cases, 5 actually involved retained sponges, several of which were found at sponge count and removed before the end of anesthesia; therefore, they were not truly left in the body postoperatively, although they prolonged the operative time. Over a 3-year period, reviews indicated that only 3 cases occurred in which a physician truly forgot a foreign body in the patient. These were often noted to include errors in the sponge count and were later found incidentally on radiograph. Of more importance, these data highlight the fact that we must have a way of collecting and analyzing data concerning equipment that breaks off during normal usage so that the manufacturers can be made aware of potentially resolvable defects.
PDI4: Iatrogenic Pneumothorax in Neonates at Risk (n = 65)
The denominator is all neonates with a birth weight of 
2500 g; infants who weighed <500 g were excluded. Also, children who had a code indicating thoracic or cardiac surgery were excluded. The numerator is those infants coded with the diagnosis of iatrogenic pneumothorax. There were no cases POA. Clinician reviewers felt that pneumothorax in premature infants was not clearly preventable 90% of the time in the charts reviewed. The exclusions should leave primarily premature infants who sustain barotrauma from high ventilator pressures. However, 3 of 11 pneumothoraxes that were reviewed occurred after patent ductus arteriosus repair; several others procedures included sternotomy for placement of pacemakers, and, therefore, should have been excluded because entering the chest was an expected part of the surgery.
PDI5: Iatrogenic Pneumothorax in Nonneonates (n = 646)
The denominator is all medical and surgical admissions of children aged 0 to 17 years excluding neonates, and the numerator is any child with iatrogenic pneumothorax excluding those with any thoracic surgery.
Eleven percent of the pneumothoraxes were POA, often because of complications that occurred in an outside hospital or during transport or because of trauma and, therefore, were not preventable within the hospitalization. Of the remaining events, 60% were not considered preventable because they were expected during procedures that the software had not flagged for exclusions (eg, tracheal reconstruction). Several cases were flagged because chest tubes had been removed and, subsequently, a pneumothorax recurred and was coded as iatrogenic, although the first occurrence was associated with cardiothoracic surgery. Thirty percent of the pneumothoraxes, the majority of which were associated with central line placement, were considered preventable.
PDI8: Postoperative Hemorrhage or Hematoma (n = 483)
The denominator is restricted to patients aged 0 to 17 years who had elective (ie, scheduled) surgery; the numerator includes patients who had a hemorrhage documented with either a procedure to control hemorrhage or drain a hematoma. Of these events, 2.8% were POA. Clinician reviewers felt that 87% were not clearly preventable. More than half of the reviewed cases were of cardiac surgeries, most of which involved cardiopulmonary bypass, heparinization, and subsequent bleeding. Clinicians often stated that they were uncertain whether these hemorrhages/hematomas were preventable, noting the difficulty of the surgeries and the fact that bleeding was a known complication. Other cases that were coded as being uncertain or not preventable included resection of neoplastic tumors that were very vascular or reperformance of surgeries in children who had had severe congenital defects (eg, diaphragmatic hernia) and now had significant scarring when revisions were performed. Clinicians listed preventable hemorrhage when a procedure was not frequently associated with severe bleeding, such as cleft palate repair, tonsillectomy, or hernia repair, or when placement of lines caused significant hemorrhage or hematoma.
PDI9: Postoperative Respiratory Failure (n = 6098)
The denominator is all surgical patients aged 0 to 17 years except for children with cardiac surgery and those who had a primary diagnosis of respiratory or circulatory disease. The numerator included patients who had elective (scheduled) surgery and had mechanical ventilation for >96 hours postoperatively or were required to be intubated 1 or 2 days after the index procedure for respiratory failure. Of these cases, 16.9% were POA, often transfers from other institutions.
This PDI had a very high rate of 35 cases per thousand. Clinicians felt that these events were not clearly preventable 86% of the time, many because children were expected to be on ventilators for 
4 days if they had major thoracic (such as tracheal reconstruction), neurosurgical, abdominal, or spinal surgeries. Twenty eight of the 290 reviewed cases had dorsal spinal fusions with planned intubation of >4days. A number of the neurosurgical cases involved removal of extensive tumors from patients who postoperatively had decreased consciousness or increased intracranial pressure and remained intubated for that reason. Even when clinicians thought the prolonged stay on a ventilator was preventable, it often involved trying to decide if the child had fluid overload postoperatively, requiring prolonged intubation, or was getting too much sedation (a difficult call when a child is in significant pain). In general, clinicians felt that this indicator, as it is presently defined, is not helpful as far as identifying a preventable complication in the pediatric setting.
PDI10: Postoperative Sepsis (n = 4367)
The denominator is all surgical patients aged 0 to 17 years (excluding neonates) with a hospital stay of >4 days who did not have sepsis or infection coded as a principal diagnosis when they were admitted and subsequently had a code for sepsis in the secondary diagnosis field. The numerator was any patient with 1 of 20 different codes that indicated sepsis, such as streptococcal septicemia or systemic inflammatory response syndrome due to infectious process with organ dysfunction. Forty percent of the patients had infection of some type POA, and of the remaining events, clinicians thought that the sepsis was not preventable 74% of the time. Many were patients who were admitted for a procedure but had long-term severe disease (ie, malignancy, chronic peritoneal or hemodialysis, chronic tracheostomies, post liver or kidney transplant), had a procedure and in the process were found to have infection—many fungal infections. There was confusion when there was questionable bloodstream infection in a line postoperatively but the child was not truly septic but simply receiving a "septic workup." Despite the false-positive results and the confusion about line infections being defined as sepsis, clinicians felt that the infection was directly related to the procedure and could have been prevented 25% of the time. The clear-cut cases usually involved children who came in relatively well and subsequently developed wound or line infection and became septic. Considering there were 4367 cases identified in our 3-year data set, preventing 25% of them (even taking into account the percentage with incorrect triggers and POA) would have saved 177 children per year from a significant complication.
PD 11: Postoperative Wound Dehiscence (n = 102)
The denominator is all abdominopelvic surgical patients aged 0 to 17 years except for those with lengths of stay of <2 days. There were also exclusions for infants with gastroschisis or omphalocele, with which there are often staged procedures for closure, and patients with immunodeficiencies. The numerator is patients with the code for reclosure of disruption of abdominal wall.
In the 3-year period there were 102 cases, and 10% were POA. Clinician reviewers thought that 66% of the remaining events were not clearly preventable. There were a number of patients with diaphragmatic hernia in which the wound was left open purposefully and was closed in stages when there was decreased swelling. Several clinicians noted that dehiscence occurred when children has severe crying and coughing, sometime occurring after extubation, and concluded that better sedation and pain management might have prevented the dehiscence.
PD 12 Selected Infection Caused by Medical Care (n = 8922)
The eligible patients included all surgical and medical patients aged 0 to 17 years who were admitted without infection as their principal diagnosis. The numerator is the number of patients who during their hospitalization had an infection that was coded after an infusion, injection, or transfusion or caused by a vascular device, implant, or graft. Forty-three percent were POA, often in children who had chronic lines or ports in place and were infected at home or in other facilities. These represent an enormous problem, although they were not strictly preventable during their hospitalization.
The majority of the bloodstream infections occurred secondary to an intravenous line. Forty percent were felt to be clearly preventable by the reviewing clinicians; this represents 
605 cases per year of children who have a preventable infection, even taking into account the percentage with incorrect triggers and POA status.
PDI13 Transfusion Reaction (n = 15)
The denominator was all medical and surgical patients aged 0 to 17 years excluding neonates. The numerator is any patient with a code for ABO incompatibility reaction, Rh incompatibility reaction, or mismatched blood. Twenty-nine percent were POA. This is extremely rare (15 cases in 3 years in 76 hospitals), and none of the 7 cases reviewed were considered preventable; they were reactions to correctly typed blood known to occur even with the best typing available because of untypeable antigens or antibodies. The reactions were usually transient fevers or rashes.
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Children have unique characteristics that require pediatric-specific patient safety and quality indicators.16,17 To that end, a number of general and specific indicators have been suggested for use in children.7,9,18 Study of the ARHQ PSIs suggests that there is potential utility in using some of these indicators to guide safety efforts.7,13 However, chart review of the PSIs also revealed significant limitations that suggested limits to their appropriateness for public reporting.8
Recognizing the limitations of the PSI for a pediatric population, the AHRQ developed and released provider- and area-level PDIs. In keeping with other AHRQ quality indicators, the PDIs use administrative discharge data to identify events, which may represent preventable patient harm. Conceptually, the PDIs could help direct organizations in their improvement efforts. In addition, government and private pay-for-performance initiatives might leverage the AHRQ PDIs as accountability measures. Thus, we chose to perform a chart review of those admissions to a cohort of pediatric hospitals that were identified as having a positive screen using the provider-level PDIs.
Our analyses revealed several significant findings. First, the study confirmed the wide spectrum in the prevalence of PDI events with half of the PDIs being very low-volume or rare events. Low volume is not a problem by itself if the event is clinically very significant and highly preventable, but this is not the case. Transfusion reactions occurred at a rate of 0.01 per 1000 hospital discharges, and none were found to be preventable. In contrast, the PDI screen identified postoperative sepsis and postoperative respiratory failure as more frequent events, with rates of 29.41 and 35.04 per 1000 discharges, respectively. This would suggest that preventable postoperative complications of respiratory failure and sepsis are not uncommon and would merit a commitment of resources to decrease their incidence. However, both of these PDIs were judged to have low preventability, especially postoperative respiratory failure, and so this might actually misdirect a lot of hospitals’ quality-improvement activities.
At the time of this study, the AHRQ PDI methodology did not account for the presence of the secondary diagnoses at the time of hospital admission. That is, a patient may be admitted to a tertiary hospital with a diagnosis that triggers the PDI screen, which may have been acquired elsewhere. However, the lack of robust methods for handling POA cases creates an opportunity for incorrect attribution of harm to a hospital. This can best be appreciated with PDI2 (decubitus ulcer). The chart review revealed that nearly 60% of the ulcers identified with the software were present at the time of admission to the hospital identified as having the potential harm event. To be clear, these decubitus ulcers are very real and are sources of significant harm and cost to patients. However, the incorrect attribution of harm to a hospital that may have merely "inherited" this diagnosis may result in significant harm to the hospital, too. The plan for the Center for Medicare and Medicaid Services to no longer reimburse hospitals for decubitus ulcers acquired in hospitals beginning in October 2008 increases the stakes for correct assignment of these harm events.2,19 Medicare has implemented a new billing form, the UB-04, to replace the previous UB-92 form; with this change, a POA indicator has been added.20 Updated versions of the PDI will take advantage of these new fields and include improved ways of identifying conditions that are POA. Ideally, through improved documentation and thoughtful assessment of patient safety and quality screening tools, this issue will no longer limit the meaningfulness of tools such as the PDIs.
Perhaps the most important finding of this study relates to issues of preventability of events. Harm to patients is clearly undesirable. However, not all complications sustained by patients are preventable by using currently available medical science. Because the primary goal of using patient safety and quality indicators should be to improve care and reduce harm, a focus on nonpreventable events may result in suboptimal use of resources for improvement. In our study, preventability of identified events was classified into 1 of 3 categories: clearly preventable, clearly nonpreventable, and uncertain as to the preventability. Two of the PDIs (iatrogenic pneumothorax in neonates at risk and postoperative respiratory failure) had greater than half the identified events categorized as clearly nonpreventable. Our findings were then leveraged to calculate the PPV for each measure. Defined as true-positive results over the sum of true-positive and false-positive results, this epidemiologic concept brings about a clinically relevant question: if a test is positive, how likely is it that a patient has the disease? In the case of the PDIs, the question becomes: if a PDI screen is positive, how likely is it that the patient has a preventable event? As was true throughout our evaluation of the PDIs, a large variation in the PPV was identified.
The variation in PPV was aggravated, in part, by the percentage of uncertain preventable events for any given PDI. A low and high range of PPVs was calculated for each of the PDIs by excluding and including events classified as uncertain preventability in the true-positive portion of the equation. If events classified as uncertain preventability are excluded from the true-positive portion of the PPV calculation, then only 1 of the PDIs, decubitus ulcer, had a PPV of >50%. Or, for every 2 positive screens, only 1 indicated a preventable harm event to a patient. If the events classified as uncertain preventability are included in the true-positive portion of the PPV calculation, then 4 of the 11 studied PDIs have a PPV of >75%. In the case of PDI12 (selected infections during medical care), the PPV was >80%, which suggests that this may be a useful indicator to screen for preventable harm in patients.
There are some potential limitations to this study. First, there was no formal assessment of interrater reliability between the clinicians at the various centers who were performing the evaluations.
A second potential limitation lies in the lack of a strict definition for preventability between the reviewers. In part, this lack of standard definition may explain the theme of being unable to determine preventability. However, the patient-safety literature reflects similar methods for identifying preventability, with the exception of using >1 reviewer to examine each chart.15,21,22 Despite the potential for either overassignment or underassignment of a given event as preventable on the basis of a single reviewer, the recurrent nature of assignment to an unpreventable category for some of the PDIs would suggest that the problem is related to PDIs falsely identifying harm events as preventable.
A third potential limitation is the study design and the decision to not evaluate charts that were not screened as positive for the PDIs. If studied, a determination of the true-negative and false-negative rates could be made. Instead, this limitation provides grounds for future work.
From the perspective of a patient or family member, this study could be viewed as largely irrelevant. If a child suffers harm during medical care, whether at a specific hospital or before admission or whether preventable or not, then the harm is a real entity to be dealt with by the patient, family, and caregivers. However, from the perspective of payers, regulators, hospitals, and physicians, the findings of this study are important. As with our review of the AHRQ PSIs in a pediatric population, infections caused by medical care remain a significant problem and, once POA is accounted for, could be publicly reported and compared between hospitals.
With this in mind, the low preventability coupled with the POA issue suggests that at the present time, AHRQ PDIs are unsuitable for public reporting. A focus on these indicators as a tool for accountability may result in penalizing hospitals for issues that are beyond their control and the opportunity cost of misallocating resources to improving the same issues.
Measurement of care to improve quality is an important and laudable endeavor. The AHRQ has used expert opinion to drive the evolution of the process. However, the experts on the AHRQ panel lacked a clear consensus as to what measures were accurate enough for public reporting. Our study reinforces that lack of consensus and confirms that chart review would be very useful during the development process and in the future should be a mandatory component in measure development before the release of new measures.
|
CONCLUSIONS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Chart review should become an essential and mandatory component of the development of administrative-based quality indicators along with expert opinion. Until this is done, broad use of the indicators for public reporting and pay for performance may serve as its own source of preventable harm.
|
ACKNOWLEDGMENTS |
|---|
|
FOOTNOTES |
|---|
Address correspondence to Aileen Sedman, MD, FAAP, University of Michigan Health System, Ann Arbor, MI 48109-0825. E-mail: asedman{at}umich.edu
Financial Disclosure: Dr Sedman is a paid medical advisor to the National Association of Children's Hospitals; the other authors have indicated they have no financial relationships relevant to this article to disclose.
| What's Known on This Subject PDIs were created by the AHRQ in 2006 on the basis of expert opinion. The purpose was to define potentially preventable complications in hospitalized children, such as nosocomial infections, iatrogenic pneumothorax, hemorrhage, and decubiti.
|
| What This Study Adds This study is the first to use the AHRQ quality indicator software and apply it to 76 children's hospitals to calculate rates of the events. Chart review was conducted by pediatric clinicians to calculate the accuracy of the indicators as to whether the complication occurred before hospitalization and the rate of preventable occurrences.
|
|
REFERENCES |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||
|
||
Home | My Pediatrics | Journal Information | Current Issue | Past Issues | Subscriptions & Services | Contact Us Click here for faster international access © 2008 American Academy of Pediatrics. All rights reserved. |
||
|
||
