ARTICLE |
a Departments of Pediatrics
f Psychiatry, University of Massachusetts Medical School/UMass Memorial Medical Center, Worcester, Massachusetts
b Department of Pediatrics, Boston University School of Medicine/Boston Medical Center, Boston, Massachusetts
c Departments of Medicine, Brigham and Women's Hospital and Children's Hospital Boston, Boston, Massachusetts
d Norwood Caritas Hospital, Norwood, Massachusetts
e Department of Pediatrics, Tufts University School of Medicine/New England Medical Center, Boston, Massachusetts
| ABSTRACT |
|---|
|
|
|---|
METHODS. Using interrupted time-series regression analysis, we reviewed all charts, orders, and incident reports for 40 admissions per month to the NICU, PICU, and inpatient pediatric wards for 7 months before and 9 months after implementation of commercial computerized physician order entry in a general hospital. Nurse data extractors, who were unaware of study objectives, used an established error surveillance method to detect possible errors. Two physicians who were unaware of when the possible error occurred rated each possible error.
RESULTS. In 627 pediatric admissions, with 12 672 medication orders written over 3234 patient-days, 156 medication errors were detected, including 70 nonintercepted serious medication errors (22/1000 patient-days). Twenty-three errors resulted in patient injury (7/1000 patient-days). In time-series analysis, there was a 7% decrease in level of the rates of nonintercepted serious medication errors. There was no change in the rate of injuries as a result of error after computerized physician order entry implementation.
CONCLUSIONS. The rate of nonintercepted serious medication errors in this pediatric population was reduced by 7% after the introduction of a commercial computerized physician order entry system, much less than previously reported for adults, and there was no change in the rate of injuries as a result of error. Several human-machine interface problems, particularly surrounding selection and dosing of pediatric medications, were identified. Additional refinements could lead to greater effects on error rates.
Key Words: patient safety medication errors medical errors computer order entry quality improvement
Abbreviations: CPOE—computerized physician order entry CI—confidence interval IRR—incidence rate ratio
Medication errors affect 1 in 10 pediatric hospital admissions and injure thousands of children annually.1–4 The Institute of Medicine and the American Academy of Pediatrics advocate for implementation of computerized physician order entry (CPOE) as 1 promising intervention to prevent pediatric medication errors.5,6 In hospitalized adults in a single hospital, CPOE was shown to reduce nonintercepted serious inpatient medication errors (errors that had the potential to injure the patient and were not caught by hospital staff) by 55%.7 Because of these and other positive findings, more than half of the nation's hospitals are planning to implement CPOE in the next several years.8
Among hospitalized children, initial research on CPOE was promising, but recent reports have been less positive (Table 1). An analysis of pediatric incident reports showed a 40% reduction in reported medication errors,9 and a study of a pediatric critical care unit found a 41% reduction in potentially dangerous errors after implementation of CPOE10; however, McPhillips et al,11 in a review of outpatient pediatric pharmacy administrative data, found no difference in rates of potential overdosing or underdosing errors between clinics that used basic CPOE and those that did not use CPOE.11 A recent evaluation of the potential benefit of CPOE over pharmacist review for ordering errors demonstrated that CPOE could reduce potentially dangerous prescribing errors but would have no effect on administration errors, which have a high risk of patient injury.12 A study of hospital mortality data reported increased mortality among PICU transfers after implementation of CPOE.13 The ecologic nature of this study and seasonal imbalance between before and after periods may have confounded the results, or, alternatively, mortality may have been related to the method of implementation of the CPOE system.13,14 Several other studies reported errors that were caused by health information technology, including CPOE.15,16 Overall, controversy remains in the current literature about the value of CPOE for pediatric patients.
|
| METHODS |
|---|
|
|
|---|
Study Site and CPOE System
The study site and CPOE system have been described in detail elsewhere.17 Boston Medical Center is an urban hospital with 4 PICU beds, 15 NICU beds, and 40 surgical and medical pediatric ward beds. The Sunrise Clinical Manager CPOE System, by Eclipsys, was implemented on the pediatric inpatient wards in April 2002 and in the PICU and NICU by June 2002. For each medication, the user has the option to select a pediatric version or an unspecified version (eg, the users could choose "pediatric ceftriaxone" or "ceftriaxone"). For pediatric medications, after the order is signed, the system uses a weight-based dosage calculator to automatically check medication dosages on the basis of the child's weight, generating wrong-dosage alerts. When writing medication orders, before signing them, the user may opt to use a pediatric weight-based dosage calculator to assist in calculating medication dosages. Other features include drug–drug interaction alerts and allergy alerts. Pediatric order sets are available for nonnarcotic pain control (eg, sucrose, acetaminophen, fluoromethane spray) and newborn care (eg, hepatitis vaccine, vitamin K injection), for which the user may select all of a list of medications or select individual medications from the list. All CPOE users attended a 2-hour training session before being given a password. Nursing medication administration records were, at the time of the study, paper based. Before implementation of this system, only the hospital pharmacy was automated with a system that had drug–drug interaction and drug allergy checking.
Inclusion Criteria
Patients who were admitted to the pediatric inpatient wards, NICU, or PICU between September 2001 and March 2002 (pre-CPOE) and between September 2002 and May 2003 (post-CPOE) were included in the sampling frame. Identical months were included in the pre-CPOE and post-CPOE sampling frame to control for seasonal effects on errors rates with 2 additional months (April and May) in the post-CPOE period. Forty patients per month were randomly selected from all pediatric admissions for study inclusion. When after at least 3 attempts we were unable to obtain the selected chart, another admission was selected and matched to the patient with the missing medical chart on unit of admission, date of admission, length of stay, and patient age.
Comprehensive Error Surveillance Methods
Error surveillance methods that were used in this study are described in detail elsewhere17 and followed previously well-established methods.18–21 All components of the inpatient record, including all medication orders, medication administration records, progress notes, nursing notes, and the discharge summary, were reviewed for possible medication errors and possible adverse drug events by trained pediatric nurses who were unaware of the study objectives. Nurses then presented a description of possible medication errors to 2 pediatricians who were unaware of whether the possible errors occurred before or after CPOE.
An adverse drug event was defined as an injury as a result of medication use; a preventable adverse drug event was an adverse drug event that was caused by an error. A medication error was defined as an error in drug ordering, transcribing, dispensing, administering, or monitoring. Because the study focused on CPOE- related errors, ordering errors were further categorized into wrong or incomplete orders, which CPOE should prevent; computer-related errors; and other ordering errors. Wrong or incomplete orders were orders that did not contain all of the normal parts of a medication order, such as excluding dosage or medication name. Computer errors were errors that were highly unlikely to occur in a paper-based system and had a clear computer-based mechanism.17 A serious medication error was defined as a medication error that caused harm or had substantial potential to cause harm and included preventable adverse drug events, nonintercepted serious medication errors, and intercepted serious medication errors. Intercepted errors were defined as errors that were caught by hospital staff before reaching the patient.
Hospital incident reports for all included patients were obtained. Physicians reviewed and rated these incidents using the same physician review process as for chart review. One incident report described an adverse drug event already detected in chart review, and that incident report was not included in the analysis to avoid counting any single adverse event more than once.
Pediatrician Reviews
Pediatricians (Drs Walsh, Landrigan, and Schainker) who were trained in error identification and analysis made judgments about whether a possible error should be classified as (1) an adverse drug event (injury), (2) a serious medication error without injury, (3) an error with little or no potential for harm, or (4) neither an error nor an adverse drug event (exclusion). For all serious medication errors or adverse drug events, physicians were asked to rate severity as (1) fatal, (2) life-threatening, (3) serious, or (4) significant. The preventability of all errors was rated as (1) intercepted error, (2) definitely preventable, (3) probably preventable, (4) probably not preventable, or (5) definitely not preventable. Preventability was defined as able to be avoided. We intentionally did not constrain reviewers into defining "preventable" in a precise way; rather, it was an implicit judgment based on their expertise. If a patient had been administered a medication to which he was known to be allergic and developed an allergic reaction, then this would be considered definitely preventable. If a patient had developed an allergic reaction to a medication to which he was previously not known to be allergic, then this would be considered definitely not preventable.
Interrater reliability scores for pediatrician judgments during review were calculated using
scores. Interrater reliability for judgments about the classification of the possible error was 0.7 (95% confidence interval [CI]: 0.68–0.84), about the severity of the error was 0.4 (95% CI: 0.26–0.57), and about the preventability of the error was 0.8 (95% CI: 0.67–0.82).
Analysis
The primary study outcome was the rate of nonintercepted serious medication errors per 1000 patient-days. The number of patient-days was the sum of the lengths of stay for each included patient. The number of nonintercepted serious medication errors was divided by the number of patient-days and multiplied by 1000 to calculate the rate of nonintercepted serious medication errors per 1000 patient-days. We used linear, interrupted time-series analysis to estimate sudden changes in levels or trends in the time series of the study outcome. Regression models included a constant term, a term for linear time trend, and terms to estimate changes in the level or trend of nonintercepted serious medication errors that coincided with the introduction of CPOE. We controlled for autocorrelation by assuming a first-order autoregressive process (correlation between 2 consecutive observations), and we used residual analysis to test the adequacy of the resulting models. We determined the statistical significance of regression coefficients by means of 2-tailed t tests. The time-series regression models were performed using Proc Autoreg in SAS 9.1.3 (SAS Institute, Cary, NC).
| RESULTS |
|---|
|
|
|---|
|
In the pre-CPOE and post-CPOE study periods, we were unable to obtain 22 (8%) and 32 (9%) medical charts, respectively (P = .95), and replaced these patients with missing medical charts with admissions matched by unit of stay, length of stay, and age. There was no statistical difference in length of stay or unit of stay (both of which have been previously shown to be associated with rates of errors2) between missing charts and those included in the study. Nine charts before CPOE and 20 charts after CPOE were incomplete (3.3% before and 5.7% after; P = .2; details available on request). An additional 5 charts before CPOE and 8 charts after CPOE were excluded because many parts were missing at the time of review, making adequate review impossible. After CPOE, 7 charts contained a total of 24 (0.3%) handwritten orders.
Nonintercepted Serious Medication Errors
Overall rates of medication errors, serious medication errors, and nonintercepted serious medication errors (those that were not caught by hospital staff) for the total period of study (pre-CPOE and post-CPOE periods combined) were 48.2 per 1000 patient-days, 32.5 per 1000 patient-days, and 21.6 per 1000 patient-days, respectively (Table 3), and on univariate analysis were unchanged after CPOE implementation. Overall rates of nonintercepted serious medication errors for the inpatient wards, NICU, and PICU were 29.7 per 1000 patient-days, 12.8 per 1000 patient-days, and 36.7 per 1000 patient-days, respectively. There was no statistically significant change in rates of nonintercepted serious medication errors on the inpatient wards (29.7 vs 30.3; incidence rate ratio [IRR]: 1.02), NICU (12.8 vs 14.7; IRR: 1.15), or PICU (36.7 vs 34.7; IRR: 0.95) after implementation of CPOE on univariate analysis.
|
|
Types of Nonintercepted Serious Errors
Although the CPOE system under study did contain automated pediatric weight–based dosage checking for all pediatric medications, the rate of dosing errors, the most common form of pediatric medication error, did not change in time-series analysis (Fig 2). An example of a dosing error is a 10-year-old child with sickle cell and acute chest who is ordered an underdose ceftriaxone on admission (30 mg/kg per day). After 3 days, the error is noted and the dosage is increased. In univariate analysis, the rate of nonintercepted serious dosing errors was 8 per 1000 patient-days (n = 11) before CPOE and 10 per 1000 patient-days after CPOE (n = 19) (IRR: 1.25; Fig 3). Of the 19 nonintercepted serious dosing errors after CPOE, only 2 generated computer alerts. Both alerts were overridden by the ordering physician without a change in the order. The second most common nonintercepted serious medication error was missed or extra dose administration errors, which accounted for 6 errors per 1000 patient-days before CPOE and 4 errors per 1000 patient-days after CPOE. Three serious nonintercepted medication errors were computer related (1.6 per 1000 patient-days).17 An example of a computer-related error is a 5-year-old boy for whom both Tylenol infant drops and suspension each at maximum dosages are ordered because the computer will not allow 1 order with a choice of either formulation. Although 1 might expect fewer computer-related errors as resident physicians became more familiar with the CPOE system that they were using, there was no improvement in the rate of computer-related errors during the period of this study.
|
|
| DISCUSSION |
|---|
|
|
|---|
These findings are substantially different from those in adult inpatients, for whom the introduction of CPOE was followed by a 55% reduction in nonintercepted serious medication errors.7 We used the same comprehensive error surveillance methods used in the study by Bates et al7 in adult inpatients and the same outcome measure, and we had similar pre-CPOE serious error rates to paper-based systems previously studied.2,7 We believe that there are at least 3 possible explanations for the differences in rates between these studies. First, we studied a commercially available system, whereas Bates et al studied a "homegrown" system developed specifically for use within the hospital of study. It is possible that the homegrown system studied previously was better tailored to meet the needs of the institution where it was designed and implemented than the version of the commercial system studied here. Second, the CPOE system that was evaluated in this study was not optimally designed to prevent common pediatric medication errors, such as using weight-based dosing calculation to prevent dosage errors, but was better designed to prevent common adult errors, such as overdoses based on adult maximum dosages or drug interactions. Our finding that the rate of dosing errors, the most common type of pediatric medication error, did not decrease after CPOE supports this second explanation. Last, the smaller impact of CPOE on pediatric errors in this study may result from differences in local implementation of this particular CPOE system.
We also found a smaller effect than early studies of pediatric inpatients using incident report review and medication order review that showed a 40% reduction in incident reports of errors after CPOE9 and a 41% reduction in potentially dangerous ordering errors, respectively.10 We used comprehensive error surveillance methods, which have been shown to be more sensitive for detecting ordering errors than incident reports19 and more intensive than simple medication order review. Using these methods, we found rates of serious medication errors similar to those in previous research.2 In examining ordering errors alone, to compare with the previous study of order-writing errors, we found no statistically significant change in the incidence of serious ordering errors after CPOE compared with before CPOE. Similar to a study of outpatient pediatric pharmacy data that showed no difference in potential dosing errors between clinics that were and were not using CPOE,11 we found no effect on rates of pediatric nonintercepted serious dosing errors.
There was a decrease in nonintercepted serious medication error rates from September to March in both the pre-CPOE and post-CPOE periods: from 38 per 1000 patient-days in September 2001 and also in September 2002 to 12 per 1000 patient-days in March 2002 and 8 per 1000 patient-days in March 2003. This three- to fourfold decline in nonintercepted serious errors is larger than the 55% decline in serious error rates originally reported with CPOE use7; however, the rate of errors increased in May 2003 to 23 nonintercepted serious errors per 1000 patient-days; still a 40% decline from the rate in September. This highlights the importance of time-series research in accounting for relatively large seasonal trends in the evaluation of error prevention interventions. A simple pre–post study sampling a pre–data point in the fall and a post–data point in the spring could attribute a large seasonal change to the intervention of study.
Although this study used comprehensive active error surveillance and time-series analysis, there are a number of limitations to consider. First, findings may not be easily generalizable to other CPOE systems or hospitals. The CPOE system of study, the Eclipsys system, is a commercially available system that is the leading vendor of CPOE.22 As more hospitals use commercially available systems that are not specially created or adapted for the institution, studies of commercial systems are of particular importance.
Second, although this study used an interrupted time-series design, considered 1 of the best quasi-experimental methods, a randomized clinical trial was not possible within our single hospital because CPOE was used in particular units on the basis of different medication ordering characteristics, such as pediatric inpatient wards versus pediatric emergency department, which would make selection of control subjects from other units impossible. In interrupted time-series design, several time periods are assessed, improving control for possible background changes in the outcome of study. The hospital did not implement any other major systemic change at the time of implementation of CPOE. We did not include a case-mix measure in our analyses to account for disease severity. Although severity of disease would affect rates of medication errors, a change in severity of disease from the pre-CPOE period to the post-CPOE period would be gradual, changing the background rate of errors, which we control for using time-series methods. Only in the case that severity of disease changed abruptly at the same time that CPOE was implemented would it confound the results of the study.
Other limitations include the possibility that the effect of CPOE was delayed even further than 1 year after CPOE or that the time period that was permitted for implementation and learning of the system (3–6 months) did not allow trainees enough time to learn the system adequately. The number of NICU admissions increased after CPOE implementation, although not statistically significantly. The rate of NICU errors was smaller than in other units, so this increase would have biased the study toward inflating the effect of CPOE. Finally, the comprehensive error surveillance method that we used is an insensitive method for detecting certain administration errors that are not recorded in the chart19; however, because CPOE would be expected to have only a limited effect on administration errors, missing some administration errors would bias this study toward overestimating the effect of CPOE on overall error rates.
The move from a paper-based health care system to a technology-based system holds promise of improving care in many domains beyond patient safety, such as facilitation of physician communication and provision of discharge instructions. It is possible that CPOE systems will require redesign and adaptation to meet the needs of children better. Changes in CPOE to facilitate weight-based dosing better are needed to prevent pediatric dosing errors, because dosing errors are the most common form of pediatric medication error.
Our experiences in working with this system in the past 5 years suggest that the CPOE system that we assessed might be improved to prevent dosing errors through a few simple changes. At the time of study, the automated weight-based dosage checking was used whenever a pediatric medication was ordered; however, the user needed to select the pediatric version of the medication (eg, "pediatric ceftriaxone") rather than the unspecified version (eg, "ceftriaxone"). If the user selected the unspecified version rather than the pediatric form, then ceftriaxone could still be ordered for the child, but the automated weight-based dosage checking could not be used. Since the completion of this study and after recognition of this issue as a problem, the system has been improved to perform weight-based dosage checking for all medications ordered for all patients. This is an example of how important it is for hospitals to monitor, continually modify, and improve CPOE systems on the basis of data derived from their own institution.
| CONCLUSIONS |
|---|
|
|
|---|
| ACKNOWLEDGMENTS |
|---|
| FOOTNOTES |
|---|
Address correspondence to Kathleen E. Walsh, MD, Department of Pediatrics, Benedict Second Floor, UMass Medical Center, 55 North Lake St, Worcester, MA 01655. E-mail: walshk02{at}ummhc.org
The authors have indicated they have no financial relationships relevant to this article to disclose.
| REFERENCES |
|---|
|
|
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||