The Effect of Point-of-Care Personal Digital Assistant Use on Resident Documentation Discrepancies
,||,#
* Children’s Health Services Research, Indiana University School of Medicine, Indianapolis, Indiana
Robert Wood Johnson Clinical Scholars Program
Department of Pediatrics
|| Division of Biomedical Informatics
¶ School of Medicine
# Child Health Institute, University of Washington, Seattle, Washington
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODSRESULTSDISCUSSIONREFERENCES |
|---|
Background. We recently found documentation discrepancies in 60% of resident daily-progress notes with respect to patient weight, medications, or vascular lines. To what extent information systems can decrease such discrepancies is unknown.
Objective. To determine whether a point-of-care personal digital assistant (PDA)-based patient record and charting system could reduce the number of resident progress-note documentation discrepancies in a neonatal intensive care unit (NICU).
Design/Methods. We conducted a before-and-after trial in an academic NICU. Our intervention was a PDA-based patient record and charting system used by all NICU resident physicians over the study period. We analyzed all resident daily-progress notes from 40 randomly selected days over 4 months in both the baseline and intervention periods. Using predefined reference standards, we determined the accuracy of recorded information for patient weights, medications, and vascular lines. Logistic and Poisson regression were used in analyses to control for potential confounding factors.
Results. A total of 339 progress notes in the baseline period and 432 progress notes in the intervention period were reviewed. When controlling for covariates in the regression, there were significantly fewer documentation discrepancies of patient weights in notes written by using the PDA system (14.4%–4.4% of notes; odds ratio [OR]: 0.29; 95% confidence interval [CI]: 0.15–0.56). When using the PDA system, there were no significant changes in the numbers of notes with documentation discrepancies of medications (27.7%–17.1% of notes; OR: 0.63; 95% CI: 0.35–1.13) or vascular lines (33.6%–36.1% of notes; OR: 1.11; 95% CI: 0.66–1.87).
Conclusions. The use of our PDA-based point-of-care patient record and charting system showed a modest benefit in reducing the number of documentation discrepancies in resident daily-progress notes. Further study of PDAs in information systems is warranted before they are widely adopted.
Key Words: resident • documentation • PDA • NICU
Abbreviations: IOM, Institute of Medicine • PDA, personal digital assistant • NICU, neonatal intensive care unit • OR, odds ratio
The 1999 report by the Institute of Medicine (IOM), To Err Is Human: Building a Safer Health System, made medical error prevention an important goal of many organizations by reporting that 44 000 to 98 000 patients die each year due to medical errors.1 More than 50% of these errors were potentially preventable.2,3 Many believe that technology can be used to reduce these errors. An IOM report, Crossing the Quality Chasm: A New Health System for the 21st Century, makes this explicitly clear.4 Their 2001 report stated that safety and quality cannot improve without the use of information technology such as electronic medical records.4 Some believe that a general-purpose electronic medical record might be able to provide benefits.5–7 Focused studies have shown that specific aspects of an electronic medical record (specifically, computerized physician order entry, pharmacy systems, and clinical reminder systems) can reduce error rates and improve compliance with guidelines.8–11 This has not gone unnoticed. Investment in health care information technology rose from $6.5 billion in 1990 to more than $20 billion in 2000.12 Independent consultants, however, estimate that 25% to 50% of these costs are paying for redundant work processes.12 Some believe that new information technology is as likely to decrease efficiency as it is to increase it.13 Clearly, new systems need to be studied before being set in place.
One area of errors that remains relatively unexplored is that of documentation errors. As information is transcribed over and over, it is at higher risk for becoming corrupted. We recently found documentation discrepancies in >60% of resident daily-progress notes relating to patient weights, medications, or vascular lines.14 Those patients who were sickest and had the most information to be maintained were most susceptible to corruption of their documented data.14 Because information recorded in the chart can be central to decision-making processes, erroneous information has the potential to compromise patient safety. The limitations of paper charting often require a number of repeated transcriptions before information finally reaches the chart, potentially leading to corruption of the data. We hypothesized that a point-of-care personal digital assistant (PDA)-based patient record and charting system would reduce the number of transcriptions of data and the prevalence of documentation discrepancies.
|
METHODS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
Design/Setting
This was a before-and-after trial conducted in an academic tertiary neonatal intensive care unit (NICU). Resident physicians with training in pediatrics, family practice, or obstetrics and gynecology, who were responsible for writing daily-progress notes on their patients, took part in this study.
The study protocol was approved by the University of Washington Institutional Review Board.
Definitions
To quantify the existence of and changes in documentation discrepancies, we used methods that we developed previously in an earlier study.14 These methods are described briefly below.
Resident Progress Notes
The required resident progress note is intended to summarize the patient’s changes and care plan each day. Standard of care at the study institution mandates that resident progress notes be written daily on all patients. Note templates are generated by a database storing some basic demographic and clinical data, printed, and then filled in by hand.
Documentation Discrepancies
We defined a documentation discrepancy as an occurrence when information on the progress note did not match a predefined reference standard. As in our previous study,14 we focused on 3 pieces of information: patient weights, vascular lines, and medications, all of which were recorded on the daily-progress note. We defined the reference standard for weight to be the nursing flowsheet; the reference standard for line access was a combination of the nursing flowsheet and the nursing daily-assessment sheet; the reference standard for medications was the pharmacy medication-administration record.
Vascular lines were defined as an intravenous line, peripheral or central venous catheter, umbilical vein or artery catheter, or arterial line.
Medications given in hyperalimentation were excluded. Vitamins and dietary supplements were not considered medications, because they were not consistently listed in the medication section of notes. Drip medications were counted as medications. As-needed medications that the patient did not receive that day were ignored. If, however, a patient was given an as-needed medication on a given day, then we counted that as a medication the patient was currently prescribed. Topical as-needed medications were ignored. To be correct, a medication in the progress note only had to have the correct name. Dose and frequency were not used in this analysis.
Lines and medications were considered in or prescribed if they had begun before 8:00 AM. Any that were discontinued during the day could either be documented or not documented without penalty. In each case, residents were not penalized if it was possible that their documentation was correct at any time of the day.14
We again divided medication and vascular-line documentation discrepancies into 2 groups:1 Errors of omission occurred when a resident did not record on the progress note a medication that appeared on the pharmacy medication-administration record or a line that the patient had in place;2 errors of commission occurred when a resident did record on the progress note a medication that was not on the pharmacy medication-administration record or a line that the patient did not have in place. Errors of omission could not occur when the patient was prescribed no medications or had no lines, so these patients were excluded when calculating percentages of notes with this type of discrepancy. All notes were included when calculating errors of commission.
Because the daily weights were sometimes recorded after notes had been written, we accepted as correct any weight that exactly matched, to the number of decimal places written, the weight recorded on the day of the note or the previous day. Again, in each case, the progress note was deemed accurate if the information was correct at any time of the day.
Intervention
We designed a point-of-care PDA-based client/server patient record and charting system for use in the NICU.15 This system was tested and piloted over the course of 2 years before being implemented. In November, 2001, we trained the resident physicians in the use of the system and began using it on November 16, 2001. For the next 4 months, residents used the PDA-based system exclusively for documenting and charting the care of patients. At the end of the 4-month intervention, the PDA system was removed, and the NICU returned to the previous method of documentation.
Residents would enter data into the PDAs over the course of the day, eliminating the need for pen-and-paper recording (Fig 1). Information entered into the PDA was automatically moved into appropriate databases on a personal computer, which allowed the system to generate printed progress notes, admission notes, summary sheets, and overnight signout. All progress notes were automatically populated with information in the PDA concerning medications, vascular lines, and patient weights.15 All residents were trained in the use of the system as they started their NICU rotations, and instruction manuals were distributed detailing the functions and use of the system.
|
Progress-Note Abstraction
Our intervention period covered the 4 months that the residents used the PDA system beginning on November 16, 2001; our baseline period covered 4 months exactly 1 year earlier. Using prior methods,14 we examined resident progress notes written on 40 random days in each 4-month period. We reviewed each eligible note for errors in documentation of weight, lines, and medications. Thirty percent of the notes were reviewed twice to validate the reliability of collected information. All questions about errors were resolved by consensus of 2 authors (A.E.C. and E.O.).
In the 30% of charts reviewed independently by 2 people, the data collector and physician reviewer had excellent agreement for notes with errors in the documentation of weight (98%; 
= .84), lines (97%; = .92), and medications (94%; = .81).
Analysis
We recorded documentation discrepancies both by the numbers of notes and numbers of discrepancies per note. We used 
2 analysis to look for changes in the existence of documentation discrepancies in resident progress notes. We also performed regression analyses of discrepancy rates, adjusting for a number of covariates using Logistic and Poisson regression with robust error estimates clustered by patient to control for the nonindependence of notes, because each patient could have had >1 note included in our study.16 Because note writing is often a team process, we could not accurately identify or control for which resident wrote which part of each note. We calculated odds ratios (ORs) to determine, after controlling for covariates, whether the PDA system caused any significant changes in the existence of a documentation discrepancy on each note. We calculated incident rate ratios to determine how the system affected the number of each type of discrepancy on each note. We used covariates that we had investigated previously and thought might increase the likelihood of incurring a documentation error.14 These covariates included the total number of medications per patient, the number of lines per patient, day of the note (weekend or weekday), the corrected gestational age (estimated gestational age at birth plus day of life), and number of days the patient had been hospitalized. Calculations were performed by using the STATA statistical package (STATA Corporation, College Station, TX).
Using standard power calculations for detecting a difference in proportions in 2 independent groups, we determined that to detect a 10% absolute reduction in the rate of documentation discrepancies in any of our 3 areas with 80% power and an 
of .05, each group would need 321 notes for review.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
During the 4-month baseline period, there were 83 admissions compared with 113 admissions during the 4-month intervention period (Table 1). The 40 randomly selected days included 339 progress notes in the baseline period and 432 progress notes in the intervention period. No progress notes were missing from the charts in either the baseline or intervention period. In general, patients in the intervention period were slightly older on average, although the average length of stay was similar in both groups.
|
In an unadjusted analysis, there were reductions in documentation discrepancies of weight (13.3% of notes before and 4.4% after) and medications (27.7% of notes before and 17.1% after) with the use of the PDA system (Fig 2). There were increases, however, in the number of documentation discrepancies of vascular lines (33.6% before and 36.1% after). Overall, there was a decrease in the number of documentation discrepancies (61.7% of notes before and 51.2% after).
|
Results were more mixed when looking at specific discrepancies of omission and commission. There was a minimal increase in the rate of medication documentation omissions and a minimal decrease in vascular-line documentation commissions (Fig 3). There was a larger reduction in the number of documentation medication commissions (18.6% of notes before and 8.6% after). There was also a larger increase in the number of vascular-line documentation omissions (13.9% of notes before and 21.4% after).
|
When controlling for covariates in the regression, however, many of these changes proved to be insignificant. There were significantly fewer documentation discrepancies of patient weights in notes written by using the PDA system (14.4%–4.4% of notes; OR: 0.29; 95% confidence interval: 0.15–0.56). The documentation of medications and vascular lines was not significantly affected by the PDA system (Table 2). When the types of medication and vascular-line documentation discrepancies were examined separately, none were affected significantly by the use of the PDA system.
|
Results from the Poisson regression showed that none of the measured rates of medication and vascular-line documentation discrepancies were significantly affected by the use of the PDA system.
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
Our PDA-based point-of-care patient record and charting system showed mixed results with respect to reducing the number of documentation discrepancies in resident daily-progress notes. Significant improvements were made in the accuracy of the documentation of patient weights. No clear improvements existed in the documentation of medications and vascular lines when controlling for other factors that could influence the numbers of discrepancies, such as numbers of medications and lines as well as hospital day. Some may argue that weight is the more important of our measures, because medication dosing depends heavily on these data and dosage errors have been found to be the most common ones in NICU.17 Overall, however, the lack of clear improvement in documentation across the board would lead us to conclude that the PDA system failed to provide a clear benefit.
This study has limitations that warrant consideration. The design was a before-and-after trial, not a randomized, controlled trial. We found that there was no way to randomize by patient, resident, or month without compromising the integrity of patient data. We took every step to reduce possible bias, however. By using the same months for both the baseline and intervention periods, we hoped to account for secular trends, seasonal variation, and stage of resident training. Furthermore, no other systematic change in documentation was made during the intervening time period. Published firm trials such as ours have frequently used similar designs.18–21
This study also occurred in a single teaching institution. It therefore can be generalized only conservatively. This is also a study of our PDA system; other, more-robust systems might have more of an effect. However, we used a multistage development process that intimately involved end users15 and had every potential to succeed. As with our previous study,14 there may be questions about our choice of reference standards for weight, medications, and vascular lines. As mentioned previously, we do not know the extent to which documentation discrepancies such as these are associated with untoward outcomes. We only studied resident progress notes, and many would argue their relative importance in the care of patients. Many residents rely primarily on their notes, however, and would disagree. Finally, we only looked at a process measure, but we believe that the accuracy of documented information is important and worthy of study.
We hypothesized that allowing residents to enter information at the point of care would reduce the number of transcriptions and lead to fewer discrepancies in documentation. Although the accuracy of weight documentation did improve, the documentation of other areas did not. This may be due to limitations of the technology. We found it encouraging that all the residents adopted the system quickly, with everyone complying immediately after its institution. Even so, there were significant user issues that should be noted in the future planning of such systems. These issues included problems with ease of data entry, size of the PDA screen, and other hardware, software, and user issues.22 Residents found that entering numbers such as weights were relatively simple on the PDAs. Keeping track of other, more-text-intensive information was not. Whether these limitations or other factors resulted in the lack of improvement in the rates of discrepancies is not known.
These findings have implications beyond this trial, however. Millions of dollars are spent every year on new and, for the most part, untested technologies. Many groups such as the IOM believe that improved information technology is a key component of improved care. Although PDAs are becoming more popular in medicine, they have not been clearly shown to be beneficial. A recent survey found that 30% of family practice resident programs are requiring residents to use PDAs for information management.23 Although some of these have been studied in a limited setting,24,25 there is little evidence to support their widespread adoption. Even so, many are moving forward and transitioning to systems such as these. At Long Island Jewish Hospital (New Hyde Park, NY), all 80 first-year residents are now keeping records on PDAs, reportedly to reduce medical errors.26 Without rigorous study, however, there is no way to know how the PDAs are affecting error rates.
Information technology has often not lived up to its promise. According to the Standish Group in Cape Cod, Massachusetts, >30% of computer systems built internally by corporations for their employees are either canceled or rejected after completion.13 Before we blindly transition to new and perhaps exciting technologies, well-designed trials are needed to determine whether there are actual benefits or even harms.27 Additional study of PDAs in information systems is certainly warranted before they are widely adopted.
|
ACKNOWLEDGMENTS |
|---|
Support for Dr Carroll was provided by the Robert Wood Johnson Foundation.
We thank the University of Washington, Division of Neonatal Medicine, for patience and support. We also thank Frederick P. Rivara, MD, MPH, for helpful suggestions in the preparation of this manuscript.
|
FOOTNOTES |
|---|
Received for publication Mar 4, 2003; Accepted May 29, 2003.
Address correspondence to Aaron E. Carroll, MD, MS, Riley Research, Rm 330, Indiana University School of Medicine, 699 West Dr, Indianapolis, IN 46202. E-mail: aaecarro{at}iupui.edu
The views expressed within this article are those of the authors and do not necessarily represent the views of the Robert Wood Johnson Foundation or the University of Washington.
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
- Kohn LT, Corrigan J, Donaldson MS. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000
- Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med.1991; 324 :377 –384[Abstract]
- Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care.2000; 38 :261 –271[CrossRef][Web of Science][Medline]
- Institute of Medicine, Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001
- Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology.
J Am Med Inform Assoc.2001; 8
:299
–308
[Abstract/Free Full Text] - Leape LL. Error in medicine.
JAMA.1994; 272
:1851
–1857
[Abstract/Free Full Text] - Jerant AF, Hill DB. Does the use of electronic medical records improve surrogate patient outcomes in outpatient settings? J Fam Pract.2000; 49 :349 –357[Web of Science][Medline]
- Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention.
J Am Med Inform Assoc.1999; 6
:313
–321
[Abstract/Free Full Text] - Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors.
JAMA.1998; 280
:1311
–1316
[Abstract/Free Full Text] - Borel JM, Rascati KL. Effect of an automated, nursing unit-based drug-dispensing device on medication errors. Am J Health Syst Pharm.1995; 52 :1875 –1879
- Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting.
J Am Med Inform Assoc.1996; 3
:399
–409
[Abstract/Free Full Text] - Dorenfest S. The decade of the ’90s. Poor use of IT investment contributes to the growing healthcare crisis. Healthc Inform.2000; 17 :64 –68[Medline]
- Gibbs WW. Taking computers to task. Sci Am.1997; 277(7) :82 –89
- Carroll AE, Tarczy-Hornoch P, O’Reilly E, Christakis DA. Resident documentation discrepancies in a neonatal intensive care unit.
Pediatrics.2003; 111
:976
–980
[Abstract/Free Full Text] - Carroll AE, Saluja S, Tarczy-Hornoch P. Development of a Personal Digital Assistant (PDA) based client/server NICU patient data and charting system. Proc AMIA Symp.2001 :100 –104
- STATA Corporation. STATA Reference Manual. Release 7, Version 7.0. College Station, TX: Stata Press; 2001
- Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients.
JAMA.2001; 285
:2114
–2120
[Abstract/Free Full Text] - Benjamin EM, Schneider MS, Hinchey KT. Implementing practice guidelines for diabetes care using problem-based learning. A prospective controlled trial using firm systems.
Diabetes Care.1999; 22
:1672
–1678
[Abstract/Free Full Text] - Goldberg HI, Neighbor WE, Cheadle AD, Ramsey SD, Diehr P, Gore E. A controlled time-series trial of clinical reminders: using computerized firm systems to make quality improvement research a routine part of mainstream practice. Health Serv Res.2000; 34 :1519 –1534[Web of Science][Medline]
- Goldberg HI, Neighbor WE, Hirsch IB, Cheadle AD, Ramsey SD, Gore E. Evidence-based management: using serial firm trials to improve diabetes care quality. Jt Comm J Qual Improv.2002; 28 :155 –166[Medline]
- Hirsch IB, Goldberg HI, Ellsworth A, et al. A multifaceted intervention in support of diabetes treatment guidelines: a cont trial. Diabetes Res Clin Pract.2002; 58 :27 –36[CrossRef][Web of Science][Medline]
- Carroll AE, Saluja S, Tarczy-Hornoch P. The implementation of a Personal Digital Assistant (PDA) based patient record and charting system: lessons learned. Proc AMIA Symp.2002 :111 –115
- Criswell DF, Parchman ML. Handheld computer use in U.S. family practice residency programs.
J Am Med Inform Assoc.2002; 9
:80
–86
[Abstract/Free Full Text] - Garvin R, Otto F, McRae D. Using handheld computers to document family practice resident procedure experience. Fam Med.2000; 32 :115 –118[Web of Science][Medline]
- Rao G. Introduction of handheld computing to a family practice residency program. J Am Board Fam Pract.2002; 15 :118 –122[Abstract]
- Talan J. PDAs, the doctor’s latest bedside helper. Newsday. July 23,2002 :D03
- Tierney WM, Overhage JM, McDonald CJ. A plea for controlled trials in medical informatics.
J Am Med Inform Assoc.1994; 1
:353
–355
[Free Full Text]
PEDIATRICS (ISSN 1098-4275). ©2004 by the American Academy of Pediatrics
CiteULike 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
This article has been cited by other articles:
|
|
 |
|
  D. M. Hilty, S. Benjamin, G. Briscoe, D. J. Hales, R. J. Boland, J. S. Luo, C. H. Chan, R. S. Kennedy, H. Karlinsky, D. B. Gordon, et al. APA Summit on Medical Student Education Task Force on Informatics and Technology: Steps to Enhance the Use of Technology in Education Through Faculty Development, Funding and Change Management Acad Psychiatry, December 1, 2006; 30(6): 444 - 450. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. M. Hilty, D. J. Hales, G. Briscoe, S. Benjamin, R. J. Boland, J. S. Luo, C. H. Chan, R. S. Kennedy, H. Karlinsky, D. B. Gordon, et al. APA Summit on Medical Student Education Task Force on Informatics and Technology: Learning About Computers and Applying Computer Technology to Education and Practice Acad Psychiatry, February 1, 2006; 30(1): 29 - 35. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
