Published online May 1, 2008
PEDIATRICS
Vol. 121
No. 5
May 2008, pp.
e1201-e1207
(doi:10.1542/peds.2007-2609)
Reevaluating the Safety Profile of Pediatrics: A Comparison of Computerized Adverse Drug Event Surveillance and Voluntary Reporting in the Pediatric Environment
Jeffrey Ferranti, MD, MSa,b,
Monica M. Horvath, PhDb,
Heidi Cozart, RPhb,
Julie Whitehurst, PharmD, MPHb and
Julie Eckstrand, RPhb
a Division of Neonatology, Department of Pediatrics, Duke University School of Medicine, Durham, North Carolina
b Duke Health Technology Solutions, Duke University Health System, Durham, North Carolina
 |
ABSTRACT
|
|---|
OBJECTIVES. Children are at exceptionally high risk for adverse drug events. At Duke University Hospital, computerized adverse drug event surveillance and voluntary safety reporting systems work synergistically to identify adverse drug events. Here we identify the most deleterious drug classes to pediatric inpatients and determine which detection methodology provides the greatest opportunity to reduce harm.
PATIENTS AND METHODS. We evaluated all of the medication-related events detected by our computerized surveillance and safety reporting systems over a 1-year period for Duke University Hospital pediatric inpatients. Events from both systems were scored for severity and assigned a drug event category. Surveillance events were additionally scored for causality.
RESULTS. A total of 849 medication-related reports were entered into the safety reporting system, and 93 caused patient harm, resulting in an adverse drug event rate of 1.8 events per 1000 pediatric patient-days. Seventy eight of the 1537 medication-related events detected by surveillance resulted in patient harm, giving a rate of 1.6 events per 1000 patient-days. The most common events identified by the safety reporting system were failures in the medication use process (26.9%), drug omissions (16.1%), and dose- or rate-related events (12.9%). The most frequent adverse drug event surveillance categories were nephrotoxins (20.7%), narcotics and benzodiazepines (19.3%), and hypoglycemia (11.5%). Most voluntarily reported events originated in ICUs (72.0%), whereas surveillance events were split evenly across intensive and general care. There was little overlap between methodologies.
CONCLUSIONS. The epidemiology of pediatric adverse drug events is best addressed by using voluntary reporting in tandem with other strategies, such as computerized surveillance and targeted chart review. Although voluntary reporting excels at identifying administration errors, surveillance excels at detecting adverse drug events caused by high-risk medications and identifies evolving conditions that may provoke imminent patient harm. Surveillance underperformed in pediatrics when compared with adult detection rates, suggesting that tailored rules may be necessary for a robust pediatric adverse drug event surveillance system.
Key Words: adverse drug event pediatrics voluntary reporting ADE surveillance
Abbreviations: ADE—adverse drug event ADE-S—adverse drug event surveillance DUH—Duke University Hospital SRS—safety reporting system DUHS—Duke University Health System IV—intravenous
Trecent Institute of Medicine report, Preventing Medication Errors, estimated that >1.5 million hospitalized patients experience preventable adverse drug events (ADEs) each year in the United States and recommends that patient safety efforts shift focus from error reduction to reducing patient harm.1–4 Although much is known about the detection and prevention of ADEs in adult population,5–8 there is little guidance on how to best find these events in specialized populations, such as children. Children are unique patients, because their immature renal and hepatic systems and fragile physiology place them at particularly high risk for harm when a medication error occurs.9 Even worse, their likelihood of experiencing medication errors is 3 times higher than that of adults, because drug doses must be hand calculated based on age, weight, and, in some cases, gestational age.10 As a result, these patients are at exceptionally high risk for harm, because nearly one quarter of all ADEs are caused by medication errors.5
Given these challenges, the Institute of Medicine and the Institute for Healthcare Improvement have released guidelines urging providers to monitor the rate of ADEs in all care settings.1,3 To date, clinicians have used 3 detection methods. The most common, manual chart review, produces highly valuable information but is extremely time and resource intensive.11–13 The second method, voluntary reporting, captures ADEs as observed by health care providers. Although voluntary reporting is relatively easy to implement, it suffers from inconsistent reporting rates, user noncompliance, variable definitions, and concerns of punitive action.12,14,15 Computerized ADE surveillance is the newest of these strategies. It identifies potential ADEs, as well as evolving unsafe conditions, by applying a series of logic-based rules and creating computerized alerts, or "triggers." These rules run daily on a hospital's clinical information system and evaluate medication orders, laboratory results, and diagnoses.7,12,16 The details of our ADE surveillance (ADE-S) system have been published previously.17,18 Because many of the surveillance rules suffer from high false-positive rates, triggers must be reviewed by specially trained clinicians to ensure that they represent true ADEs.12 Although technically challenging to implement, ADE-S can provide a quantitative measure of ADE rates throughout a health system.
Careful planning must underlie the choice of any one of these detection methodologies in pediatrics, because their ADE risk is so high.9,10 Although voluntary reporting systems are routinely used, it is essential to recognize they only provide 1 piece of the puzzle. To balance the strengths and weaknesses of the different discovery strategies, Duke University Hospital (DUH) has implemented a 2-pronged approach for continuous ADE detection: a Web-enabled, voluntary, safety reporting system (SRS) and a computerized ADE-S system. This article compares these methods in pediatric inpatients to do the following: (1) determine the rates of ADEs according to each discovery method, (2) identify the primary drug classes involved in pediatric events, (3) outline areas where one detection methodology may be superior to the other, and (4) suggest areas for future enhancement.
 |
METHODS
|
|---|
Study Setting and Design
Duke Children's Hospital is a tertiary care facility located within DUH. We conducted a retrospective review of all medication-related events detected by computerized surveillance (ADE-S) and voluntary safety reporting (SRS). This review encompassed all of the pediatric patients at DUH receiving service on our 7 inpatient units (3 ICUs, 2 general pediatric wards, and 2 transitional care units). The study period was from December 1, 2004, to January 31, 2006. All of the pediatric units used paper medication orders, which were reviewed by pediatric clinical pharmacists. Events occurring outside of the patient's hospitalization at DUH were excluded from analysis. Twenty-nine days of ADE-S events were also excluded where staffing issues prevented complete review of all of the triggers. This study was approved by the Duke University Health System (DUHS) Institutional Review Board.
Voluntary SRS
The DUHS voluntary SRS accepts reports from all DUH employees via an internally developed Web application. Employees are encouraged to report every witnessed safety incident, as well as situations with potential to cause harm if corrective action is not taken. On the SRS Web portal, employees must assign their safety report to 1 of the following categories: medication or intravenous (IV) related, blood transfusion, surgical or invasive, fall, treatment or testing, equipment, dissatisfied patient, or "other." Event reporting at DUH is integral to the culture of safety and is nonpunitive, although reporters may elect to remain anonymous. The medication reports are analyzed, summarized, and communicated to hospital leadership by a team of medication safety pharmacists and analysts.
After the initial report, a team of 4 medication safety pharmacists investigates each medication/IV-related report, scores them for severity (Table 1), and categorizes the events according to system failure, preventability, and attributable cause.14 As recommended by the Institute of Medicine and Nebeker et al,19 ADEs are defined as "an injury resulting from the use of a drug"; however, we expanded the definition to include errors of omission where the initial drug order never reached the patient and harm resulted.20 All of the voluntarily reported events scored with a severity of
3 are deemed ADEs. Interrater reliability statistics are not currently calculated among DUH medication safety pharmacists for event scoring and categorization. However, after the initial investigation is complete, the report is submitted to a multidisciplinary peer-review team, which validates the scoring process and affirms the status of a report as an ADE.
Computerized ADE-S
The DUHS ADE-S system was developed in-house and deployed on November 1, 2004.17,18 It has since been operationalized and continuously evaluates all of the inpatient medication and laboratory information against a set of 57 clinical rules warning of potential ADEs or evolving unsafe conditions.17 These rules span 3 main categories: abnormal laboratory results, drug dispenses for known antidotes, and drug-laboratory combinations. When a rule condition is met, a trigger is fired and sent to 2 dedicated clinical pharmacists. They perform focused chart review of the daily trigger list to assess whether an ADE has occurred and whether an intervention in care is necessary. The pharmacists assign a causality score using the algorithm of Naranjo et al21 and a severity score using the SRS 7-point scale (Table 1).18 ADE-S uses the same ADE definition as the SRS system, although omissions cannot be detected by the current rule set. All of the events scored with a causality of
5 and a severity of
3 are considered true ADEs. Interrater reliability (
statistic) exceeded 0.88 between the clinical pharmacists.17,18
Data Analysis
Three clinical pharmacists (
0.88) manually examined all of the ADEs detected by the SRS and ADE-S systems and assigned event categories. SRS report severity was not rescored by ADE-S pharmacists. Positive predictive values for each system were calculated as the fraction of voluntary reports (SRS) or triggers (ADE-S) that were deemed as ADEs subsequent to pharmacist evaluation. For the ADE-S system, any ADE found by a trigger was counted in the calculation even if that ADE was not what the trigger was originally designed to detect. ADE rates were calculated as either the number of ADEs detected per 1000 patient-days or the number of ADEs detected per 100 admissions. Pearson
2 or Fisher's exact tests were used to test for significant differences in categorical data, as appropriate. SEs are provided when mean values are reported. Given the descriptive nature of this study and the relatively few tests completed, we followed a recommendation to omit multiple testing correction in our analysis, given concern of inflating type II error.22 We do, however, report which tests were still significant after Bonferroni adjustment for reader interest.
 |
RESULTS
|
|---|
During the study period (4711 admissions; 51046 patient service days), a total of 849 voluntary reports (SRS) were submitted on pediatric inpatients, giving a daily average of 2.4 ± 0.1 reports. Table 2 further summarizes these data. After severity scoring, 93 (11.0%) of these were deemed ADEs, resulting in an SRS incidence rate of 1.8 ADEs per 1000 patient-days. For the ADE-S system, a total of 1537 triggers fired (daily average: 4.0 ± 3.0). After review by clinical pharmacists, 78 ADEs were found (5.1%), resulting in an overall rate of 1.6 ADEs per 1000 patient-days. The difference in ADE discovery rates was not statistically significant between the 2 systems (Pearson's
2 test, P = .48). The remaining events not deemed as ADEs by each system were false-positives (ie, no medication involved), did not cause harm, or did not have objective evidence of being an event.
Table 3 directly compares the 2 systems by event category. Because surveillance is based on a series of predefined trigger rules, it was not possible to directly pair all of the SRS events with a trigger category. Where possible, we directly compared the efficacy of each methodology. The paired pediatric events fell into 6 broad categories: nephrotoxins, narcotics and benzodiazepines, Clostridium difficile infections, hypoglycemia, hyperkalemia, and anticoagulants. ADE-S was statistically superior to voluntary reporting in detecting narcotic and benzodiazepine events (P = .02) and C difficile infections (P = .003). Surveillance was also 3 times as effective in detecting insulin, hyperkalemia, nephrotoxic, and anticoagulation-related events, although our sample size was too small to have statistical significance (P = .08, .13, .07, and .17, respectively). Aminoglycosides, commonly used in the treatment of Gram-negative bacteria infections, accounted for 92.8% of the nephrotoxic events, with the primary medication involved being gentamicin (57.0%). This drug class is well known to be nephrotoxic and is administered based on body weight. The 15 narcotic and benzodiazepine events predominantly involved fentanyl (42.9%) and morphine (35.7%).
Table 4 outlines SRS events that did not have analogous triggers in the ADE-S system. A total of 24.7% of SRS reports (n = 23) involved an error in the medication administration process, such as incorrect drug dose or rate (n = 12), miscellaneous IV errors (n = 3), incorrect drug route (n = 2), incorrect drug given (n = 2), drug ordering (n = 3), and drug transcription (n = 1). These findings are consistent with other studies.15 Another 26.9% of SRS ADEs involved IV infiltrates (n = 25). Drug omission represented 16.1% of reported events (n = 15), which is significant, because ADE-S is unable to detect errors of omission, meaning we are 100% reliant on voluntary reporting to measure the frequency of this class of events. For example, ADE-S did not detect an event reported by the attending nurse into SRS where a 14-year-old patient did not receive the fivefold increase in a hydromorphone patient-controlled analgesia dose. This error was not recognized until 4 hours later, when the patient-controlled analgesia cassette was replaced, at which time the patient's pain score was recorded at a level of 9 of 10.
Table 5 presents a comparison of detection methodologies by patient location and level of care. The majority of voluntarily reported events were entered in our ICUs (73.1%, 2.3 ADEs per 1000 patient-days), whereas only 26.9% of events were entered on general pediatric wards (1.2 ADEs per 1000 patient-days). Most NICU events were attributed to issues with replacement fluids (38.7%; n = 12) and total parenteral nutrition or lipid preparations (29%; n = 9), whereas the PICU events were more evenly distributed across numerous drug categories: replacement preparations (19.2%; n = 5), sympathomimetic agents (15.4%; n = 4), and total parenteral nutrition or lipid preparations (11.5%; n = 3). In contrast to SRS, the ADE-S system seemed to perform better on our general pediatric wards and in the pediatric bone marrow transplant unit. The bone marrow transplant unit had 3.2 events per 1000 patient-days (18 total), compared with 1.6 per 1000 patient-days by SRS reporting (10 total). Two general care units collectively had 1.8 events per 1000 patients (36 total) according to the ADE-S system.
After examining each system separately, we assessed areas in which the 2 systems overlapped in both reports and ADEs (Table 6). A total of 12 events (1.4% of SRS reports and 0.8% of ADE-S triggers) were detected by both systems and represented a variety of drug and event categories. Of these, 9 SRS reports were categorized as ADEs given their severity score of
3. However, only 4 of these 9 were considered ADEs when the matching trigger was examined by ADE-S pharmacists. The 5 remaining ADE-S triggers not scored as ADEs in SRS indicated high serum drug levels, but there was no evidence of adverse patient effects. These discrepancies in scoring illustrate the need to deploy standardized definitions and to streamline event evaluation processes across all detection methods.
 |
DISCUSSION
|
|---|
In this study we examined the rates of ADEs in pediatric populations at a large academic medical center. We compared the effectiveness of 2 complementary detection methodologies in an effort to better understand the strengths and weaknesses of each. The SRS system had a better positive predictive value (93 ADEs of 849 reports [11.0%]) than did ADE-S (78 ADEs of 1537 reports [5.1%]; P < .001). This is an expected result, because someone must first recognize a problem in the medication use process for an SRS event to be reported. In the ADE-S system, a trigger may fire because of a high laboratory result value, but in practice that value may not represent any abnormality in the patient. Although the SRS system has a better true-positive rate, ADE detection by SRS will always be hampered by selective reporting. This is a known deficiency of any voluntary reporting system.15 The ADE-S system, on the other hand, can detect events consistently and reliably but only if the appropriate rules have been developed and the underlying electronic data used by the rules remain intact. Both detection strategies have equivalent efficacy for finding ADEs in pediatric patients, although the profile of events detected is quite different between the systems. Only 4 ADEs overlap between the 2 systems, which is only 4.3% of SRS events and 5.1% of ADE-S events.
Although ADE-S is extremely good at consistently detecting known areas of risk, such as nephrotoxicity and narcotic-related events, SRS provides greater insight into system failures, such as drug omission, administration errors, and lapses in clinical monitoring. Because such events are diffuse in nature, expanding the trigger tool to capture them is quite difficult. These types of errors are not easily detected by automated techniques and highlight the need for a synergistic approach balancing the strengths of each method. The SRS system also found more events in ICUs than ADE-S. ICU patients are, by definition, more ill than those in general care, which could contribute to the SRS event density. This trend has been reported in adult populations.6 In addition, many of the SRS reports at our organization were submitted by ICU nurses, who typically have fewer patients when compared with those in general care units. From a voluntary reporting standpoint, this may confer the advantage of expanded knowledge of the patient and his or her care plan.
Errors of omission, which constitute 16.1% of all SRS reports, can only be detected by voluntary reporting unless a more robust clinical information system is developed. Similarly, allergic reactions are more easily detected using voluntary reporting. Although it is possible to detect these events using the computerized surveillance trigger tool by surveying for corticosteroids and Benadryl use,7,12,17 these rules were inactivated shortly after implementation because of their associated inefficiencies and high false-positive rates in the adult population.12,17
Several studies have evaluated the clinical merits of these systems in adults. Bates et al5 identified ADEs by using 3 methods: voluntary reporting, direct questioning of health care practitioners, and chart review. Over 6 months at 2 large, tertiary care hospitals, their incidence rates for ADEs and potential ADEs were reported as 6.5 and 5.5 per 100 nonobstetrical admissions, respectively. Raschke et al23 used a logic-based trigger system to target 37 potential ADEs, which yielded a rate of 6.4 adult events per 100 admissions. As for our own surveillance system at DUH, it detects adult ADEs at a rate of 7.5 events per 1000 patient-days (4.4 ADEs per 100 admissions), which was considerably greater than that seen in pediatric patients (1.6 events per 1000 patient-days or 1.8 events per 100 admissions). In addition, in our study, computerized surveillance did not outperform voluntary reporting in ADE detection, although it has been shown to do so in adult populations.12 ADE-S was designed on adult parameters, and it has been suggested that pediatric trigger rules should be calibrated to understand pediatric drug usage and embrace the dynamic changes that occur during childhood.23 New rules should be designed that take into account both the critical laboratory values and therapeutic drug dose ranges specific to children of different age ranges. Based on this study, we expect that new trigger rules should include a special focus on the treatment of allergies and the use of antibiotics. Sharek et al24 recently described a manual trigger tool for use in the NICU. It will be interesting to see whether these rules can be applied in our automated surveillance system. We are currently piloting a project to analyze the mostly commonly used critical laboratory values and therapeutic drug dosages in pediatrics at DUH to design more sensitive ADE triggers.
The value of ADE-S is clearly demonstrated in its ability to detect narcotic and C difficile events, because it is 3 to 4 times more effective than SRS based on examination of rates. This finding is consistent with the conclusions of other reviews, which hold that methods of voluntary reporting will be overall less sensitive than those that use triggers.25 ADE-S is, therefore, a uniquely effective method for monitoring medication safety in pediatrics.
 |
CONCLUSIONS
|
|---|
Voluntary reporting and ADE-S are complimentary methodologies, and the detection of ADEs in pediatrics can be maximized by using both. When used in isolation, each presents only a partial picture. Although voluntary reporting is useful as a qualitative tool, ADE-S results that are validated by focused chart review can provide quantitative metrics that act as a longitudinal scorecard to assess overall organizational health. Surveillance has the potential to play a pivotal role in pediatrics. Regardless of the strategy deployed, a standardized methodology for evaluating, categorizing, and scoring events must be in place so that data can be compared in a consistent manner. Surveillance rules created for adult populations must be recalibrated for pediatric populations given the unique challenges that these specialized patients create for ADE detection.
 |
ACKNOWLEDGMENTS
|
|---|
This study was supported by grant 5UC1HS014882-03 from the Agency for Healthcare Research and Quality, National Institutes of Health.
 |
FOOTNOTES
|
|---|
Accepted Nov 5, 2007.
Address correspondence to Monica M. Horvath, PhD, Duke Health Technology Solutions, DUMC 2718, 2424 Erwin Rd, Durham, NC 27710. E-mail: monica.horvath{at}duke.edu
The authors have indicated they have no financial relationships relevant to this article to disclose.
| What's Known on This Subject
There is much literature describing both strategies for adult ADE detection and comparative analyses that dissect each methods' strengths and weaknesses. There is considerably less information available for pediatric populations.
|
| What This Study Adds
We describe a pediatric ADE detection strategy at a highly computerized medical center. Voluntary reporting and computerized surveillance synergistically provide both qualitative feedback and quantitative metrics to benchmark safety. We detail the scenarios in which each method had maximal efficacy.
|
 |
REFERENCES
|
|---|
- Institute of Medicine.
To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 1999
- Institute of Medicine.
Patient Safety: Achieving a New Standard of Care. Washington, DC: National Academy Press; 2004
- Institute of Medicine.
Preventing Medication Errors: Quality Chasm Series. Washington, DC: National Academy Press; 2006
- Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools.
Qual Saf Health Care. 2003;12
(suppl 2):ii39
–ii45[Abstract/Free Full Text]
- Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events: implications for prevention. ADE Prevention Study Group.
JAMA. 1995;274
(1):29
–34[Abstract]
- Bates DW, Miller EB, Cullen DJ, et al. Patient risk factors for adverse drug events in hospitalized patients. ADE Prevention Study Group.
Arch Intern Med. 1999;159
(21):2553
–2560[Abstract/Free Full Text]
- Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance of adverse drug events in hospital patients.
JAMA. 1991;266
(20):2847
–2851[Abstract]
- Kilbridge PM, Classen DC. Automated surveillance for adverse events in hospitalized patients: back to the future.
Qual Saf Health Care. 2006;15
(3):148
–149[Free Full Text]
- Goldmann D, Kaushal R. Time to tackle the tough issues in patient safety.
Pediatrics. 2002;110
(4):823
–826[Free Full Text]
- Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients.
JAMA. 2001;285
(16):2114
–2120[Abstract/Free Full Text]
- Bates DW, O'Neil A, Boyle D, et al. Potential identifiability and preventability of adverse events using information systems.
J Am Med Informatics Assoc. 1994;1
(5):404
–411[Abstract/Free Full Text]
- Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report.
J Am Med Inform Assoc. 1998;5
(3):305
–314[Abstract/Free Full Text]
- Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients: results of the Harvard Medical Practice Study II.
N Engl J Med. 1991;324
(6):377
–384[Abstract]
- Leape L, Lawthers, AG, Brennan, TA, Johnson, WG. Preventing medical injury.
Qual Rev Bull. 1993;19
(5):144
–149
- Suresh G, Horbar JD, Plsek P, et al. Voluntary anonymous reporting of medical errors for neonatal intensive care.
Pediatrics. 2004;113
(6):1609
–1618[Abstract/Free Full Text]
- Bates DW, Evans RS, Murff H, Stetson PD, Pizziferri L, Hripcsak G. Detecting adverse events using information technology.
J Am Med Inform Assoc. 2003;10
(2):115
–128[Abstract/Free Full Text]
- Kilbridge P, Alexander, L, Ahmad A. Implementation of a system for computerized adverse drug even surveillance and intervention at an academic medical center.
JCOM. 2006;12
(2):94
–100
- Kilbridge PM, Campbell UC, Cozart HB, Mojarrad MG. Automated surveillance for adverse drug events at a community hospital and an academic medical center.
J Am Med Inform Assoc. 2006;13
(4):372
–377[Abstract/Free Full Text]
- Nebeker JR, Barach P, Samore MH. Clarifying adverse drug events: a clinician's guide to terminology, documentation, and reporting.
Ann Intern Med. 2004;140
(10):795
–801[Abstract/Free Full Text]
- Gurwitz JH, Field TS, Avorn J, et al. Incidence and preventability of adverse drug events in nursing homes.
Am J Med. 2000;109
(2):87
–94[CrossRef][ISI][Medline]
- Naranjo CA, Busto U, Sellers EM, et al. A method for estimating the probability of adverse drug reactions.
Clin Pharmacol Ther. 1981;30
(2):239
–245[ISI][Medline]
- Rothman KJ. No adjustments are needed for multiple comparisons.
Epidemiology. 1990;1
(1):43
–46[Medline]
- Raschke RA, Gollihare B, Wunderlich TA, et al. A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital [published correction appears in JAMA. 1999;281(5):420].
JAMA. 1998;280
(15):1317
–1320[Abstract/Free Full Text]
- Sharek PJ, Horbar JD, Mason W, et al. Adverse events in the neonatal intensive care unit: development, testing, and findings of an NICU-focused trigger tool to identify harm in North American NICUs.
Pediatrics. 2006;118
(4):1332
–1340[Abstract/Free Full Text]
- Sharek PJ, Classen D. The incidence of adverse events and medical error in pediatrics.
Pediatr Clin North Am. 2006;53
(6):1067
–1077[CrossRef][ISI][Medline]
PEDIATRICS (ISSN 1098-4275). ©2008 by the American Academy of Pediatrics