Abstract
OBJECTIVE: The goal was to examine the current abilities and future potential of electronic health record (EHR) systems to measure childhood developmental screening and follow-up rates in primary care settings.
METHODS: A group of pediatric clinicians and health informatics experts was convened to develop quality indicators reflecting different aspects of the developmental screening process. These indicators included the administration of a standardized, validated instrument to screen children for developmental delays, the documentation of abnormal screening results, and the provision of follow-up care. Six integrated provider systems across the United States, with fully implemented EHR systems, were evaluated to determine the feasibility of implementing these measures within each system. Barriers related to measure implementation were identified.
RESULTS: The EHR systems of all 6 health care organizations could implement measures examining developmental screening rates and could identify and track children with abnormal screening results. However, most of the systems did not have the ability to capture data for more-complex EHR-based measures. In particular, data elements based on workflow actions could not be captured with current EHR system designs.
CONCLUSIONS: This study identified 2 main barriers to the implementation of developmental quality measures: concerns about data reliability and the tracking of care coordination within patient records. Potential solutions to these problems, including terminology standardization, patient portal use, and use of a single developmental screening instrument, are discussed.
Fifteen percent to 18% of US health care organizations report having adopted an electronic health record (EHR) system.1,2 However, these systems have been designed primarily to serve adult patients. Child health providers report having less access to EHR systems, especially those with features such as electronic prescribing and electronic order entry.3 Where it is used, however, EHR-supported data collection in well-child care can significantly improve the quality of care.4
The adoption of EHR systems could significantly improve care quality by facilitating both the monitoring of developmental administration and the coordination of care in response to abnormal screening results. Developmental delays are common among children, with a prevalence in the US pediatric population of 12% to 16%.5 Early intervention in developmental delays leads to better long-term outcomes, which indicates that screening for developmental delays is a critical element of well-child care.6–8 Guidelines currently recommend screening at 9, 18, and 30 months.9 Yet despite the substantial benefits and modest time cost of standardized screening, developmental assessments are not performed routinely,10 and as many as 40% to 50% of US children receive inadequate screening.11,12
EHRs allow for advanced monitoring capabilities that can facilitate quality assessment and higher-quality care. The screening guidelines proposed by the American Academy of Pediatrics (AAP) provide a performance standard that can be translated easily into a quality measurement. Screening results and necessary follow-up care can then be systematically and efficiently identified and monitored more easily than allowed by paper-based systems. With paper-based systems, for example, patients with abnormal screening results may be difficult to identify. In contrast, EHR functions such as a problem list (a central list of diagnoses for each patient record) can indicate abnormal screening results, allowing providers to initiate follow-up care such as referrals to appropriate specialists. EHRs can therefore enable the creation of an automated clinical decision support system that could improve care for children with developmental concerns. Clinical reminders, for example, can alert practitioners to actions needed to address missing screenings or abnormal findings. Such alerts have been shown to be effective at improving preventive care13,14 despite the existence of “alert fatigue,” in which providers begin to ignore programmed reminders.15 EHRs could streamline the administration of screenings and can be modified to capture and to analyze new data elements in response to the latest information regarding clinical best practices.
Despite these potential benefits, little is known about whether EHR systems can be used to accomplish these tasks. This article reports on an assessment of the feasibility of implementing EHR-based measures of developmental care and identifies issues that may arise when quality-of-care measures are integrated into EHR systems.
METHODS
This study was conducted as part of a larger project to develop EHR-based quality measures for child health and developmental services. Experts in child development, quality measurement, and EHR systems, and representatives from the American Academy of Family Physicians, the AAP, and the Center for Health Care Strategies met in April 2007 to develop measurement concepts for developmental screening. Three quality indicators were chosen from a larger set of 18 to serve as the basis for evaluating the feasibility of implementing developmental service measures within EHRs. These measures assess different aspects of the developmental screening process: that is, administration of a standardized, validated, screening instrument; identification of abnormal screening results; and provision of follow-up care. Certain indicators were selected on the basis of their varying degrees of EHR dependence, to highlight common themes and limitations across the organizations. Further information on indicator criteria is provided in Fig 1.
EHR developmental screening indicators evaluated.
Monthly telephone interviews were conducted with representatives from 6 large EHR health organizations that were “early adopters” of EHRs: Park Nicollet Health Services (Minnesota), Kaiser Permanente Northwest (Oregon), the Nemours Foundation (Delaware), the Billings Clinic (Minnesota), the Geisinger Health System (Pennsylvania), and the Boston Medical Center (Massachusetts). These representatives participated in the refinement of measures and commented on the current availability and location of key data elements for each quality indicator. Interviewees also commented on the specific challenges presented by their EHR systems. A case study of one organization (Park Nicollet) was conducted to examine more fully the implementation barriers the organization encountered.
Each organization also completed a worksheet identifying the accessibility and format of the data elements necessary for the implementation of developmental screening indicators. Three levels of data availability were specified. In decreasing ease of data extraction for large-scale analysis, these levels were data stored in a “data warehouse,” data documented in the EHR without being extracted to a data warehouse, and data not currently documented in any part of the EHR system. Patient care information located in a data warehouse was abstracted as structured data elements and was stored separately from the patient record. A summary of the data elements available from the 6 organizations is provided in Tables 1 to 3.
Numbers of Organizations Capturing Data Elements Required for Measure 1 (Screening Occurrence)
The EHR systems were assessed on the basis of their performance on 3 indicators (Fig 1). The first indicator was whether developmental screenings were conducted during well-child visits. It measured the proportion of children who had undergone a standardized, validated, developmental screening assessment at their 9-, 18-, or 30-month well-child visit within the past 12 months. This measure was a translation of the current AAP guidelines for developmental screening.10 These guidelines were developed by the AAP without respect to EHR capabilities, and compliance with them also can be examined by using traditional techniques, such as analyses of billing data and patient charts. An EHR system fully satisfied this measure if the system could record whether a well-child visit occurred, whether a screening occurred, and the age of eligible patients.
The second indicator was whether an EHR system could identify children who required monitoring for developmental delays. The indicator was specified as the proportion of children with abnormal screening results documented on their problem list in the previous 12 months. This measure tested whether an EHR system was capitalizing on its potential to support clinical decision-making. A system fully satisfied this measure if it could identify an abnormal screening result, update the patient problem list, and supply the data elements required for the first measure.
The third indicator was whether the primary care provider (PCP) reviewed specialist notes for patients who received medical specialist care for developmental concerns. Because early identification and treatment considerably benefit children with developmental delays, the occurrence of follow-up care is an important attribute of quality care. Care coordination by the PCP is key to timely treatment, particularly because developmental delays often necessitate services from a variety of medical and nonmedical providers. This measure focused on an important aspect of care coordination–whether information transfer between primary care and medical specialist settings occurred after referral visits. A system fully satisfied this measure if it could identify medical specialist visits (both in and out of network) and communications between medical specialists and PCPs.
RESULTS
Performance on Indicators
All 6 EHR systems were able to identify whether screening was performed during a visit (Table 1). Four systems captured this information as structured data; the other 2 systems, which captured the data in EHRs as free text, could be modified to translate this information into structured data.
Although all of the systems could identify abnormal screening results, the methods for reporting this information differed. Most systems used billing codes, procedure codes, or a combination of the 2 to identify abnormal screening results. One system used a free-text field in which the screening score and abnormal indication were entered manually by the PCP. Three of the organizations reported that they could not capture abnormal screening results as structured data. Instead, the information was captured by using a combination of free-text fields, scanned information, and manual entry on a patient problem list. Although these methods offer advantages over the use of hard-copy records, they still impede quality measurement by requiring significant processing before the information can be analyzed.
Five of the 6 organizations could identify abnormal screening results on a patient's problem list (Table 2). The sixth organization reported that abnormal screening results could be flagged only in the patient record, rather than in the problem list. Although this method does not fully satisfy the measure specifications, it does suggest that automated identification of screening results ultimately could occur in all systems.
Numbers of Organizations Capturing Data Elements Required for Measure 2 (Abnormal Screening Result)
No organization could fully implement the specialist follow-up measure, principally because of the lack of an automated process for documenting physician communications and other actions. Three organizations capture almost all of the necessary data elements as structured data, and 2 organizations could abstract many of these elements from patient records. The remaining organization could not capture any data regarding specialist care (Table 3). This lack of automated data collection hinders assessment of this quality measure, because this information often is too complex to gather manually. For example, although it would be too resource-intensive to determine manually whether PCPs read specialist notes, EHR systems could be designed to time-stamp and monitor which parts of a record are reviewed by a provider (thereby producing the provider's “data trail”). Although this measure was satisfied only by tracking of communication with medical specialists, 1 organization reported that its EHR system could track referrals to some nonmedical services but could not identify communications with medical specialists. This emphasizes the breadth and depth of information that EHR systems can be designed to capture.
Numbers of Organizations Capturing Data Elements Required for Measure 3 (Specialist Follow-Up Care)
Other Challenges
Case study interviews identified additional barriers to full implementation of EHR-based monitoring of developmental screenings. The first was data reliability, that is, accurate consistent identification of developmental screening results. Although most organizations indicated that they could abstract the necessary data elements automatically, they currently use manually entered diagnostic and procedural codes (eg, International Classification of Diseases, Ninth Revision [ICD-9], and Current Procedural Terminology [CPT] codes) to identify whether a screening was performed and whether it yielded abnormal results. This can make it difficult to determine whether the absence of manually entered codes in the EHR actually means that a screening did not occur, and it can cause diagnostic codes to be inconsistent among patients. This is complicated by the fact that some systems pull billing information directly into the data warehouse, whereas others rely on information from the patient record. Moreover, manual data entry can cause unreliable coding.
Another common system limitation was the inability of EHR systems to track patient care coordination. Most systems would require significant modifications to identify instances when a PCP reviewed communications from a specialist or modified a patient record. For all organizations, referrals to social services or preschools were stored only as free-text clinician notes or scanned information and thus were not accessible as structured data. In addition, some systems were able to track only information provided through in-network specialist communications. Interestingly, although 1 organization could not abstract specialist information from in-network communications, it was able to track communications from out-of-network specialists.
DISCUSSION
This study of early-adopting health care organizations shows that these organizations have developed sophisticated EHR systems that can facilitate quality measurement and developmental screening. Two of the 3 measures developed could be implemented with few or no modifications to existing systems, which emphasizes the flexibility and potential of these systems.
The organizations also exhibited considerable variation in what data they captured and how they stored data, indicating that the information needed for quality indicators can be successfully generated by a variety of EHR systems. This is strong evidence that the implementation of EHR-based measures for developmental screening is feasible. An important next step is to establish uniform measurement specifications for each screening indicator. This would allow developmental screening indicators to include a high level of specificity, taking advantage of new EHR-specific features. Standardization of terminology would allow for generalizability across different systems as EHR use in primary care settings increases.
The study also identifies some common challenges that should be addressed. One of the biggest challenges is the inability to identify screenings and abnormal findings consistently across eligible patient encounters. This is largely attributable to the inconsistent use of procedure and billing codes within and across patient records. EHR systems could be designed to address this problem by automatically validating these codes against other information in the patient record. With time, broader system checks could be developed to specifically address these concerns. The use of consistent coding terminology, such as the freely available Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT) terminology infrastructure, could also help achieve organizational consistency.
The conversion of a paper-based screening tool into an electronic format integrated with the EHR system could be a more comprehensive, valid, and reliable solution. This option would require the investment of considerable time and effort to develop and to test entry formats and scoring algorithms, because electronic data capture methods would need to be validated against existing paper-based tests.
Alternatively, settling on a common developmental screening instrument might encourage greater consistency. Nine general developmental screening tools have been identified by the AAP as acceptable screening measures. However, these tools vary greatly in format, sensitivity, specificity, and modality.7 Moreover, since none of these tools are in the public domain, providers incur costs for their use. Consequently, there is no consensus regarding which screening tool to use in pediatric primary care settings.16,17 Collaborative efforts among providers or even EHR vendors could increase awareness of the benefits of EHR-based screening while decreasing the costs associated with translating a measure into an electronic format.
Personal health records and “patient portals,” which allow Internet-based health applications such as questionnaires to be linked to a patient's health record, could also facilitate screening and the tracking of screening administrations. Families could complete Internet-based screening questionnaires before the well-child visit or by using a handheld device in the waiting area. The responses could be scored automatically by the system and placed in the child's record to be reviewed during the visit. Such a process could increase screening rates significantly without negatively affecting visit workflow or duration. Studies have shown that, for some patient populations, electronic questionnaires are preferred over their paper-based counterparts18,19 and reduce the amount of missing data.20 Studies have suggested that the validity of parent-completed developmental screening measures is similar to that of questionnaires administered by clinicians9,12,17 and that online data collection from parents is effective for certain child conditions, such as attention-deficit/hyperactive disorder.21,22
Another challenge was that no organization was able to track medical specialist follow-up care adequately. This is consistent with the general difficulty EHR systems have in tracking a provider's actions. Given that creation of audit trails is commonplace in other types of systems, however, it should be possible to use a provider's data trail to identify what was reviewed, added, or modified in a patient file. Such information could offer measures of quality and care coordination that cannot be replicated in a paper-based system.
Although the specialist follow-up measure focused on referrals to medical specialists, the multidisciplinary nature of developmental care presents an added challenge that should be considered. More than many other types of care, developmental care requires the ability to monitor sources outside the traditional medical system, such as preschool programs and other social service providers.7 Although 1 EHR system captured information about nonmedical referrals as free-text information, the systems are not presently designed to collect this information as structured data. It may be possible to use natural-language software to analyze and to code the free-text data, as has been done to measure the quality of smoking cessation programs23 and diabetes care.24 In addition, nonmedical providers may be willing to collaborate with health care organizations to develop joint systems to capture care information. Finally, it may be helpful for EHR systems to capture information about instances where referrals do not succeed. For instance, if a child is unable to see a specialist because of state eligibility requirements or administrative processing delays, then it would be important for that to be considered when assessing the quality of care received from the PCP.
This study has several limitations. Information about the EHR systems was gathered through interviews rather than through direct inspection. Because the EHR systems were not tested directly, some information about data elements that could have been used in the measures and technical concerns associated with implementation might have been overlooked. In addition, even though this early adopter group used a wide range of EHR designs, all of the health care organizations were integrated delivery systems with advanced EHR systems. Therefore, they do not offer a comprehensive depiction of the difficulty of implementing these measures for other US health care providers. The measures used in the study were based on the assumption that developmental screening results have the sensitivity and specificity necessary to identify the full spectrum of children at risk for developmental delays. For example, although in practice PCPs may interpret borderline scores by using additional data, such as family history, the measures used in the study did not account for this type of information. Finally, we focused on a limited set of measures in assessing the feasibility of implementing these measures. As EHR technology evolves, however, the number and scope of EHR-based quality measures available are likely to grow. For example, future measures could examine whether children received additional follow-up care after referrals or could draw on important socioecological information in the EHR, such as the presence of maternal depression or having a sibling with developmental delays.
Despite these limitations, this article is the first to discuss the role of EHRs in measuring the quality of child developmental services. Furthermore, it shows how these systems can be used to monitor developmental screening and highlights the potential of EHR systems to adapt, to modify, and to create new data elements. It is important to anticipate and to preempt technological problems and deficiencies in data collection, extraction, and implementation; doing so should lead to greater usability and generalizability of EHR-based developmental measures.
This type of data collection can provide important data on developmental screening rates without the unreliability inherent in the parent and provider sources that are currently being used. Such information could help to increase screening rates and to improve child health outcomes, while enabling more-streamlined and more-accurate quality measurements.
Acknowledgments
This project was supported by funding from the Commonwealth Fund (grant number 20060455) and the Nemours Foundation.
Special thanks go to Kris Benson, MD, at Park Nicollet for her assistance.
Footnotes
- Accepted May 29, 2009.
- Address correspondence to Kitty S. Chan, PhD, Department of Health Policy and Management, Bloomberg School of Public Health, Johns Hopkins University, 624 N. Broadway, Room 633, Baltimore, MD 21205. E-mail: kchan{at}jhsph.edu
Financial Disclosure: The authors have indicated they have no financial relationships relevant to this article to disclose.
What's Known on This Subject:
EHR systems are becoming increasingly common in the United States and provide new opportunities to improve quality of care. Although developmental delays are prevalent among children and can be identified through developmental screening, many US children receive inadequate screening.
What This Study Adds:
Little is known about whether EHR systems can monitor developmental screening successfully. This study assesses the feasibility of implementing EHR-based measures of developmental quality of care and identifies issues that may arise when such measures are integrated into EHR systems.
REFERENCES
- Copyright © 2009 by the American Academy of Pediatrics
In this issue
Jump to section
Related Articles
Cited By...
- What to expect from electronic patient record system implementation: lessons learned from published evidence
- Completeness and accuracy of anthropometric measurements in electronic medical records for children attending primary care
- Variation in Outcomes of Quality Measurement by Data Source
- Variations in Measurement of Sexual Activity Based on EHR Definitions
- Feasibility of Evaluating the CHIPRA Care Quality Measures in Electronic Health Record Data