Published online January 4, 2006
PEDIATRICS Vol. 117 No. 1 January 2006, pp. 15-21 (doi:10.1542/peds.2004-2741)
This Article
Right arrow Abstract Freely available
Right arrow P3Rs: Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when P3Rs are posted
Right arrow Alert me if a correction is posted
Services
Right arrow E-mail this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to My File Cabinet
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via CrossRef
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Roukema, J.
Right arrow Articles by Moll, H. A.
Right arrow Search for Related Content
PubMed
Right arrow Articles by Roukema, J.
Right arrow Articles by Moll, H. A.
Related Collections
Right arrow Office Practice

Paper Versus Computer: Feasibility of an Electronic Medical Record in General Pediatrics

Jolt Roukema, MDa, Renske K. Los, MScb, Sacha E. Bleeker, MD, PhDa, Astrid M. van Ginneken, MD, PhDb, Johan van der Lei, MD, PhDb and Henriette A. Moll, MD, PhDa

a Department of Pediatrics, Sophia Children's Hospital
b Department of Medical Informatics, Erasmus Medical Center, Rotterdam, Rotterdam, Netherlands


   ABSTRACT
TOP
 ABSTRACT
METHODS
 RESULTS
 DISCUSSION
 CONCLUSIONS
 REFERENCES
 
BACKGROUND. Implementation of electronic medical record systems promises significant advances in patient care, because such systems enhance readability, availability, and data quality. Structured data entry (SDE) applications can prompt for completeness, provide greater accuracy and better ordering for searching and retrieval, and permit validity checks for data quality monitoring, research, and especially decision support. A generic SDE application (OpenSDE) to support documentation of patient history and physical examination findings was developed and tailored for the domain of general pediatrics.

OBJECTIVE. To evaluate OpenSDE for its completeness, uniformity of reporting, and usability in general pediatrics.

METHODS. Four (trainee) pediatricians documented data for 8 first-visit patients in the traditional, paper-based, medical record and immediately thereafter in OpenSDE (electronic record). The 32 paper records obtained served as the common data source for data entry in OpenSDE by the other 3 physicians (transcribed record). Data entered by 2 experienced users, with all patient information present in the paper record, served as the control record. Data entry times were recorded, and a questionnaire was used to assess users' experiences with OpenSDE.

RESULTS. Clinicians documented 44% of all available patient information identically in the paper and electronic records. Twenty-five percent of all patient information was documented only in the paper record, and 31% was present only in the electronic record. Differences were found in patient history and physical examination documentation in the electronic record; more information was missing for patient history (38%) than for physical examination (15%). Furthermore, physical examination contained more additional information (39%) than did patient history (21%). The interobserver agreement of documentation of patient information from the same data source was fair to moderate, with {kappa} values of 0.39 for patient history and 0.40 for physical examination. Data entry times in OpenSDE decreased from 25 minutes to <15 minutes, indicating a learning effect. The questionnaire revealed a positive attitude toward the use of OpenSDE in daily practice.

CONCLUSION. OpenSDE seems to be a promising application for the support of physician data entry in general pediatrics.

Key Words: electronic medical record • structured data entry • pediatrics • evaluation

Abbreviations: EMR—electronic medical record • SDE—structured data entry • CI—confidence interval

Several weaknesses of paper-based medical records have been identified, such as illegible handwriting, ambiguous and incomplete data, data fragmentation, and poor availability.1 In addition, paper records often become bulky with time, which leads to lack of overview. Because paper records still represent the usual medium for collecting and recording patient data, these weaknesses could impede the continuity and quality of care.

Implementation of electronic medical record (EMR) systems promises significant advances in the quality of patient care, because such systems may enhance readability, availability, and data quality.2 In an EMR system, structured data are preferable to free text, because most benefits of EMRs rely on structured coded data. Structured data entry (SDE) applications can prompt for completeness, provide better ordering for searching and retrieval, and permit validity checks for data quality, research, and especially decision support.1–3

Despite potential benefits, user acceptance will be the major barrier in implementation of EMR systems, because clinicians will face a change in their practice habits. The advantages of coded data must outweigh the disadvantages of capturing such data for SDE to become successful in clinical practice.4 Functionality and the user interface will therefore be crucial for successful implementation.5,6

A generic SDE application for documentation of narrative data (OpenSDE) was developed and then tailored for the recording of patient history and physical examination results in the domain of general pediatrics, a broad specialty (S.E.B., G. Derksen-Lubsen, MD, PhD, A.M.v.G., J.v.d.L., and H.A.M., unpublished data, 2004). The purpose of this study was to analyze the pediatric OpenSDE for its completeness, uniformity of reporting, and applicability in pediatric outpatient care.


   METHODS
TOP
 ABSTRACT
 METHODS
RESULTS
 DISCUSSION
 CONCLUSIONS
 REFERENCES
 
Materials
OpenSDE was developed in the Department of Medical Informatics, Erasmus Medical Center, to support the structured recording of patient data in any medical domain.7 To record patient history and physical examination findings, OpenSDE uses a tree of medical concepts that represent the available descriptive options. Each node in the tree is described in additional detail by the nodes of its branches. A sequence of nodes represents a clinical "finding" (eg, a cardiac murmur is described by the sequence of physical examination/chest/heart/auscultation/murmur) (Fig 1). The tree of general pediatrics consists of 8648 nodes, with a maximal depth of 9 ramifications. Patient history contains 6312 nodes and physical examination 2336. The principle of OpenSDE is that a generic application is used for different domain-specific tree structures (eg, a pediatric cardiologist uses a tree that is more detailed with respect to cardiac history and physical examination findings than does a general pediatrician).


Figure 1
View larger version (36K):
[in this window]
[in a new window]
  		FIGURE 1 OpenSDE screen capture. The lower left side of the interface shows the tree of medical concepts and descriptive options. The right side of the screen displays an entry form representing the descriptive options for the selected concept in the tree ("cardiac murmur"). The upper left side presents an overview of entered data.

 
When recording patient data, OpenSDE displays the tree of clinical findings on the left and the descriptors (eg, the branching nodes) of the selected clinical finding on an entry form on the right side of the screen (Fig 1). A comment can be added to any finding or description, thereby facilitating the use of free text. The user can also define custom entry forms that may contain a user-defined selection of nodes in the tree for a specific medical problem or task.

Before this study, data from 100 existing, paper-based, pediatric medical records were entered into OpenSDE by experienced users, for evaluation of the ordering and coverage of a tree that was developed previously for the domain of general pediatrics (S.E.B., G. Derksen-Lubsen, A.M.v.G., J.v.d.L., and H.A.M., unpublished data, 2004). Adjustments were made to improve the usability of the application in practice, namely, rearrangement and completion of the tree structure and addition of predefined, visible (rather than on-demand), entry fields for free-text comments.

Procedures
Two pediatricians and 2 senior residents in pediatrics voluntarily followed a short, standardized, education course on the use of OpenSDE. They had never worked with OpenSDE before. In the outpatient department of the Sophia Children's Hospital, they all documented data for 8 first-contact patients (eg, patients referred to the pediatrician for the first time by their general practitioners) in the regular, paper-based, pediatric medical record, the "paper record." Immediately after departure of the patient, the same patient information was entered into OpenSDE, the "electronic record." The electronic record was made after departure of the patient to avoid a "checklist effect," resulting in incomparable data.8 The participants were aware that the electronic records were not to be used for patient care, because the paper record was still the standard.

Methods for comparing paper records with electronic records are not readily available.9 We did not endeavor to compare the paper and electronic records with a patient-based standard. Therefore, the paper record was transcribed into a control record containing all written information. Two experienced OpenSDE users (J.R. and A.M.v.G.) determined independently which clinical findings documented in the paper record could have been entered into OpenSDE as structured data by the participants. Findings in the paper record that could not be structured were entered as free text. The 2 entries were compared to reach consensus on an optimal electronic record, which contained all information present in the paper record and served as the "control record."

In addition, each (anonymous) handwritten paper record was entered into OpenSDE by the other 3 physicians, yielding the "transcribed record." For each record (paper, electronic, and transcribed), we assessed the number of entered details and free-text annotations and the data entry time. We corrected the data entry time for the number of entered details. Institutional review board review was not required.

The participants received a questionnaire based on the validated Questionnaire on User Interface Satisfaction, a general user evaluation instrument for interactive computer systems, to obtain their subjective opinions regarding the use of and experiences with OpenSDE. Four major topics (content, layout, system, and general response) were rated on a 6-point scale.10 The collected data were used to analyze (1) similarities and differences regarding which patient data were documented in the primary paper record versus those entered into OpenSDE, comparing the electronic and control records in each case; (2) interobserver agreement in describing patient information from a paper record in an EMR, comparing the transcribed and control records in each case for the presence of information; and (3) users' experiences with OpenSDE and data entry times during the research period (learning effect).


   RESULTS
TOP
 ABSTRACT
 METHODS
 RESULTS
DISCUSSION
 CONCLUSIONS
 REFERENCES
 
Records Collected
Thirty-two paper records and a total of 155 OpenSDE records were obtained, including 32 electronic records prepared by primary physicians, 91 transcribed records (5 missing), and 32 control records.

Similarities and Differences Between Electronic and Control Records
On average, a total of 212 findings were entered for each case. Table 1 shows the average proportions of all entered patient information present in the electronic record, the control record, or both.


View this table:
[in this window]
[in a new window]
  		TABLE 1 Average Number of Documented Findings in Each Case (n = 32)

 
In the electronic record, compared with the control record, patient history lacked more information (38%) than did physical examination (15%). In addition, physical examination in the electronic record contained more additional information (39%), compared with the control record, than did patient history (21%).

Uniformity of Data Entry From a Common Data Source
On average, 3 transcribed records were prepared from each paper record by different physicians. On average, 135 findings (95% confidence interval [CI]: 56–214 findings) were recorded in the transcribed records. Table 2 shows to what extent physicians entered the information available in the paper record and entered additional information and how much information was missing for patient history and physical examination. In the transcribed records, 67% (patient history) and 84% (physical examination) of the information was documented by at least 2 participants uniformly.


View this table:
[in this window]
[in a new window]
  		TABLE 2 Uniformity of Reporting (Transcribed Records)

 
Eighty-one percent of the additional information (ie, information present in the transcribed record but absent in the control record) in patient history and 72% of that in physical examination was entered by only 1 participant. All transcribed records were missing information and contained extra information; the amounts of missing and extra information differed.

On average, 39% of the missing information in patient history was missing in 1 of the 3 transcribed records, whereas 66% of that in physical examination was missing in 1 of the 3 transcribed records. However, 29% of the missing information in patient history and 14% of the missing information in physical examination was missing in all 3 transcribed records. The average interobserver agreement statistics ({kappa} values) were 0.39 (95% CI: 0.25–0.53) for patient history and 0.40 (95% CI: 0.28–0.51) for physical examination.

Use of Free Text
The use of free text is shown in Table 3. Free text was divided into free text that could have been entered into OpenSDE in a structured way (incorrect use of free text) and free text that could not have been entered into OpenSDE because the tree did not contain the nodes to express the findings involved (correct use of free text). Each transcribed record contained an average of 18 free-text entries, and a mean of 212 findings were recorded. A total of 27.7% of the free-text entries were incorrect.


View this table:
[in this window]
[in a new window]
  		TABLE 3 Use of Free Text

 
Users' Experiences With OpenSDE and Data Entry Times
Four major topics were evaluated, ie, content, layout, system, and general response. All participants returned the questionnaire. The content for patient history and physical examination was found to be complete or very complete. Arrangement, legibility of characters, and navigation through the tree were all judged as good or very good. The system used during the research period was found to be reliable and easy to learn and use, and the participating physicians shared the opinion that data entry could be performed quickly enough.

Data entry times were recorded for both the electronic and transcribed records and were corrected for the number of entered findings. The mean data entry time for the first 4 electronic records was 24:22 minutes (range: 11:17–50:54 minutes). The second 4 records took an average of 21:50 minutes (range: 8:57–54:34 minutes).

The mean data entry time for the first 8 transcribed records was 26:42 minutes (range: 10:32–50:50 minutes). The last 8 transcribed records took an average of 17:46 minutes per case (range: 9:43–30:29 minutes). The average time to document patient data in the paper-based record was ~9 minutes.


   DISCUSSION
TOP
 ABSTRACT
 METHODS
 RESULTS
 DISCUSSION
CONCLUSIONS
 REFERENCES
 
The first objective of this study was to compare current documentation in the control record with electronic documentation of the same information in OpenSDE. The electronic record contained more patient information than the control record. In the electronic record, physical examination contained more additional information than patient history. In patient history, however, more information was missing than in physical examination. This could be explained to a great extent by the nature of information that is recorded in patient history and physical examination. Patient history contains the patient's narrative, whereas generally physical examination consists of objective measures and observations, registered in a systematic way. SDE might therefore be more suitable for documentation of physical examination findings (ie, objective data) than for patient history findings (ie, subjective data). Walsh11 stated that, because narratives are essential to each patient's episode of illness, computers should enable clinicians to record these narratives easily. For efficient recording of physician-gathered information, one could consider using a less-detailed tree structure for recording patient history than for physical examination in OpenSDE.12

The presence of 31% additional information (in the electronic record versus the control record) is probably attributable to the fact that primary physicians made the electronic record immediately after having seen the patient. The nature of SDE might be inviting for recording more detail.8 The tree structure seems to serve as a reminder about available descriptive options, and the entry forms provide additional detail. For example, when the remark "abdomen no abnormalities" or "heart sounds normal" was made in the paper record, this was often specified more completely in the electronic record (eg, instead of "heart sounds normal," "normal rhythm, first heart sound normal, second heart sound normal, cardiac murmur absent" was recorded). These findings may be of higher "quality" but also of less importance. Furthermore, the experimental setting itself, and the fact that physicians were not distracted during documentation in the EMR, could have promoted more detailed data entry, leading to information bias.

The 25% of the patient data that were found only in the control record mostly involved concepts that required relatively much navigation to enter in OpenSDE but were easy to document on paper. For example, medical history was often presented in detail in the paper record, whereas less-detailed free text was often used in the electronic record, although structured options for description of medical history were readily available.

The second objective was to assess the uniformity of reporting in the transcribed records. The inter-rater agreement statistics ({kappa} values) were 0.39 (95% CI: 0.25–0.53) for patient history and 0.40 (95% CI: 0.28–0.51) for physical examination. Although this indicates a fair to moderate strength of agreement, it also points out that recording data with SDE (as implemented currently), even in an artificial environment, does not necessarily improve uniformity in coverage.

In patient history, more information was missing in 2 or 3 transcribed records of 1 case than in physical examination. Most of the missing information concerned detailed, clinically less relevant information (eg, number of days attending day care or occupation of the parents). This detailed information required relatively much navigation and was therefore difficult to find and to document in OpenSDE. The information absent in the transcribed records could have been forgotten or neglected. An explanation could also be that free text in the paper records was found illegible by the participants, resulting in less complete documentation in the transcribed records. Fifty-three of the 91 transcribed records contained an average of 2 findings (1.5% of 135 findings) that conflicted with the control record. It is unclear whether these discrepancies were attributable to misinterpretation or erroneous transcription.

Twenty-eight percent of the free-text entries could have been entered in a structured way. These entries concerned mostly the use of free text on a level high in the tree where participants were able to document their findings quickly and easily, at the cost of structured entry of the same information. For example, 1 physician documented the presence, duration, and setting of wheezing as free text, whereas the same information could have been entered in a structured manner (Fig 2). Seventy-two percent of the free-text entries were correct (ie, no structured options were available). These entries concerned mostly narrative information, such as a description of eating habits for a child with constipation.


Figure 2
View larger version (49K):
[in this window]
[in a new window]
  		FIGURE 2 Screen capture of the entry form for "wheezing," showing an example of free-text documentation (upper left) and structured documentation (lower right) of the same information. The 2 contain overlapping and nonoverlapping information. Free text that is also present in the structured information field represents incorrect use.

 
Already SDE applications have proved to be valuable in concise subject areas such as sonography, radiology, and endoscopy.13–17 Rosenbloom et al17 reported that documents generated with SDE contained 64% more concepts, compared with documents generated with a standard dictation/transcription model. Kuhn et al16 reported that, for the description of a technical examination (eg, upper abdominal sonography), there is evidence of superiority of a structured approach over free-text dictation. In broad specialties, such as internal medicine or general pediatrics, the drawbacks of capturing a patients' narrative in coded data may not be outweighed readily by eventual advantages. Because SDE by nature limits individual expression, free text should not be replaced fully by SDE.3,12,18

A learning effect in data entry times could be identified with OpenSDE. Physician data entry has proved to be a major barrier to the implementation of EMRs.12,18,19 For SDE to be successful in clinical practice, the time spent documenting patient data in an EMR must be minimized. However, SDE may require extra effort.4 Initially, data entry times were increased because of unfamiliarity with OpenSDE. Because the average time for documenting patient data in the paper record was 9 minutes, it was still faster than with OpenSDE. However, the latter contained more information, on average, and not all functions in OpenSDE to facilitate data entry were used in this study (eg, templates and customized entry forms).

For EMRs to be accepted in clinical practice they must meet the demands of their users, namely, physicians.4–6 The results of the Questionnaire on User Interface Satisfaction revealed a positive attitude toward the use of OpenSDE. All participating physicians shared the opinion that data entry could be performed quickly enough. The general response was that OpenSDE could be very useful in clinical practice.

Currently, OpenSDE functionality is being implemented in our hospital information system, after a pilot trial in the pediatric emergency department. Explicit information was extracted from OpenSDE to generate reports.


   CONCLUSIONS
TOP
 ABSTRACT
 METHODS
 RESULTS
 DISCUSSION
 CONCLUSIONS
REFERENCES
 
The electronic records contained ~65% of all patient information that was present in the control record. Physical examination was found to be more complete and contained more additional information than patient history, which indicates that SDE is more suitable for documentation of objective data. The participating physicians had a positive attitude toward the use of a SDE application, a requirement for successful implementation.


   ACKNOWLEDGMENTS
 
We gratefully acknowledge M. de Wilde for technical support and C. de Bruin, MD, C. S. van het Zandt, MD, M. Twilt, MD, A. L. M. Boehmer, MD, and T. Hendriks, MD, for their participation in this feasibility study.

The generic OpenSDE application, as well as the tree structure for general pediatrics (Dutch language), is available at the OpenSDE Web site (http://www2.eur.nl/fgg/mi/openSDE).


   FOOTNOTES
 
Accepted Mar 8, 2005.

Address correspondence to Jolt Roukema, MD, Department of General Pediatrics, Room SP 1545, Sophia Children's Hospital, Erasmus Medical Center, PO Box 2060, 3000 CB Rotterdam, Netherlands. E-mail: j.roukema{at}erasmusmc.nl

The authors have indicated they have no financial relationships relevant to this article to disclose.


   REFERENCES
TOP
 ABSTRACT
 METHODS
 RESULTS
 DISCUSSION
 CONCLUSIONS
 REFERENCES
 

  1. Dick RS, Steen EB. The Computer-Based Patient Record: An Essential Technology for Health Care. Revised ed. Washington, DC: National Academy Press; 1997
  2. Adams WG, Mann AM, Bauchner H. Use of an electronic medical record improves the quality of urban pediatric primary care. Pediatrics. 2003;111 :626 –632[Abstract/Free Full Text]
  3. van Ginneken AM. The computerized patient record: balancing effort and benefit. Int J Med Inform. 2002;65 :97 –119[CrossRef][Medline]
  4. Powsner SM, Wyatt JC, Wright P. Opportunities for and challenges of computerisation. Lancet. 1998;352 :1617 –1622[CrossRef][ISI][Medline]
  5. Dansky KH, Gamm LD, Vasey JJ, Barsukiewicz CK. Electronic medical records: are physicians ready? J Healthc Manag. 1999;44 :440 –445[ISI][Medline]
  6. Sittig DF, Kuperman GJ, Fiskio J. Evaluating physician satisfaction regarding user interactions with an electronic medical record system. In: Proceedings of the American Medical Informatics Association Symposium 1999. Bethesda, MD: American Medical Informatics Association; 1999:400 –404
  7. Los RK, van Ginneken AM, de Wilde M, van der Lei J. OpenSDE: row modeling applied to generic structured data entry. J Am Med Inform Assoc. 2004;11 :162 –165[Abstract/Free Full Text]
  8. Wyatt J. Quantitative evaluation of clinical software, exemplified by decision support systems. Int J Med Inform. 1997;47 :165 –173[Medline]
  9. Stausberg J, Koch D, Ingenerf J, Betzler M. Comparing paper-based with electronic patient records: lessons learned during a study on diagnosis and procedure codes. J Am Med Inform Assoc. 2003;10 :470 –477[Abstract/Free Full Text]
  10. Chin JP, Diehl VA, Norman KL. Development of an instrument measuring user satisfaction of the human-computer interface. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery; 1998:213 –218
  11. Walsh SH. The clinician's perspective on electronic health records and how they can affect patient care. BMJ. 2004;328 :1184 –1187[Free Full Text]
  12. McDonald CJ. The barriers to electronic medical record systems and how to overcome them. J Am Med Inform Assoc. 1997;4 :213 –221[Abstract/Free Full Text]
  13. Gouveia-Oliveira A, Salgado NC, Azevedo AP, et al. A unified approach to the design of clinical reporting systems. Methods Inf Med. 1994;33 :479 –487[Medline]
  14. Bell DS, Greenes RA. Evaluation of UltraSTAR: performance of a collaborative structured data entry system. In: Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care. Bethesda, MD: American Medical Informatics Association; 1994:216 –222
  15. van der Meijden MJ, Tange HJ, Boiten J, Troost J, Hasman A. An experimental electronic patient record for stroke patients, part 2: system description. Int J Med Inform. 2000;58 –59:127–140
  16. Kuhn K, Gaus W, Wechsler JG, et al. Structured reporting of medical findings: evaluation of a system in gastroenterology. Methods Inf Med. 1992;31 :268 –274[Medline]
  17. Rosenbloom ST, Kiepek W, Belletti J, et al. Generating complex clinical documents using structured entry and reporting. Medinfo. 2004;11 :683 –687[Medline]
  18. Gilbert JA. Physician data entry: providing options is essential. Health Data Manag. 1998;6 :84 –86,88,90–92[Medline]
  19. Kaplan B. Reducing barriers to physician data entry for computer-based patient records. Top Health Inf Manage. 1994;15 :24 –34[Medline]
PEDIATRICS (ISSN 1098-4275). ©2006 by the American Academy of Pediatrics




This Article
Right arrow Abstract Freely available
Right arrow P3Rs: Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when P3Rs are posted
Right arrow Alert me if a correction is posted
Services
Right arrow E-mail this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to My File Cabinet
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via CrossRef
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Roukema, J.
Right arrow Articles by Moll, H. A.
Right arrow Search for Related Content
PubMed
Right arrow Articles by Roukema, J.
Right arrow Articles by Moll, H. A.
Related Collections
Right arrow Office Practice