BACKGROUND AND OBJECTIVES: Seattle Children’s Hospital was one of the early adopters of computerized physician order entry. As part of our 2003 go-live, order sets were created opportunistically by using an ad hoc development process. A pilot study revealed that this ad hoc development process resulted in order sets that were neither internally nor externally consistent. We sought to update order sets by using software development techniques, to try and improve consistency and also to review clinical content so that they could be updated to current evidence and consensus-based best practice. We also sought to identify and categorize errors found in the original order sets.
METHODS: This is a case study of a new order set development process that: (1) assigned order set ownership; (2) created and applied standards for how orders should appear and be organized within order sets; (3) supported multidisciplinary review and update; and (4) enforced submitting completed specifications before order set build. We extracted order sets into Microsoft Word specifications, updated content by using the Track Changes function, and then updated our Clinical Information System. Changes were reviewed and organized according to themes.
RESULTS: We created standard order formats for 98 orders; 191 order sets were standardized. Multidisciplinary review identified medication issues in 37% of order sets (used in 47.6% of inpatient admissions).
CONCLUSIONS: This case study demonstrates that it is not sufficient to simply implement computerized physician order entry. Clinical decision supports should be subject to rigorous development processes to ensure both clinical appropriateness and correctness.
- CDS —
- clinical decision support
- CE —
- Clinical Effectiveness
- CIS —
- Clinical Information Systems
- CPI —
- Continuous Performance Improvement
- CPOE —
- computerized physician order entry
- SCH —
- Seattle Children’s Hospital
Clinical decision support (CDS) enhances health-related decision-making by providing evidence-based clinical knowledge when it is needed.1 One type of CDS is computerized physician order entry (CPOE). CPOE has been recognized as enhancing the efficiency, safety, and quality of medical work.2 A key factor contributing to the success of CPOE is order sets. An order set is a group of orders used to manage a disease state or procedure.3
Seattle Children’s Hospital (SCH) is a 250-bed tertiary pediatric teaching hospital in Seattle, Washington. SCH was an early adopter of this functionality,4 implementing inpatient CPOE with order sets in our Clinical Information System (CIS) in November 2003 (Cerner PowerChart, Cerner Corporation, North Kansas City, MO). A total of 230 order sets were developed and implemented initially, following the process in Table 1, and we now have >400 order sets.
In 2008, the Clinical Effectiveness (CE) team was formed. The CE team was charged with applying the principles of continuous performance improvement (CPI) (based on the Toyota Production System)5 to the advancement of clinical care. The CE team sought to create evidence-based clinical recommendations, to hardwire these recommendations into CIS, and to use CIS reports to monitor adherence and clinical outcomes. With these 3 key components, CE facilitates iterative clinical improvement. As a first step, the CE team engaged clinicians, informaticians, and local leaders in evidence-based medicine and information services to examine existing work processes during a 3-day rapid process design workshop. Analysis of the order set development process and individual order sets revealed the following factors.
Order Sets Lacked Clinical Owners
This finding had subtle implications. Clinicians could request contradicting order set changes, resulting in either rework or delays while awaiting clarifications. Departments differed in how well they maintained order sets, with order set work prioritized according to departmental activity instead of hospital strategic initiatives. A culture grew wherein clinicians felt they had to practice within the confines of existing CIS content, not appreciating its customizability.
Lack of Leadership Support to Maintain Standards
SCH experienced rapid growth in CIS clinical decision supports such as order sets. Consequently, governance broke down. Order set standards, often not documented, were lost as analysts transitioned to other projects. Eventually, order sets were specified by clinicians largely in the absence of standards, resulting in poor internal and external consistency (Table 2).
Ad Hoc Multidisciplinary Review
Instead of a formal review process, it was assumed that clinicians would solicit input from stakeholders before requesting new order sets or updates. The only check was medication verification by a pharmacist, but this step was not documented and did not always occur. This resulted in order sets that may not reflect or respect current practice.
Order Sets Not Maintained
There was no process for scheduled maintenance of order sets. Changes to the formulary, clinical guidelines, or hospital policies would occur without order set updates. As a result, many order sets contained outdated orders or inappropriate orders given newer hospital policies.
Requested Order Sets Not Sufficiently Developed for Building
Order sets were requested through an online submission process. However, often new requests had poor specificity; for example, “We need an order set to manage anticoagulation.” Refining requests into something buildable resulted in new order sets requiring months or even years to develop.
After the workshop, we created a new process (Fig 1) to address the following needs: to determine order set ownership; to create and apply standards for order appearance and sequence; to support multidisciplinary team review, update, and sign off on order set content; and to have complete, reviewed, and approved order set specifications before submission of a build request. In addition, we added a step for informatician review of order sets in the production domain.
We piloted our new process to update 20 order sets for use with our new observation unit. We found a surprising number of order set updates were needed.
From these findings, and the Institute for Safe Medication Practices’ recommendation to periodically review order sets,6 we planned to update order sets to conform to current practice, policies, formulary, and available evidence and consensus-based clinical practice guidelines. We also sought to identify and categorize types of errors corrected to try to prevent those types of errors in the future.
The core project team consisted of 2 physicians, a senior clinical informatics analyst, and a project manager. The project was jointly sponsored by the vice president for CPI and the pediatrician-in-chief/chief medical information officer, who served as a point of escalation for project and clinical issues. This team had the ability to engage resources from pharmacy, laboratory, nutrition, nursing, CE, information services, CPI, individual service lines, clinical departments, and order set users.
Initial order set usage analysis led to inclusion of all order sets used in the previous 18 months. Order sets were then categorized according to department and function. Order sets were excluded that were billing-only, medication-only, or had a very narrow departmental purpose (eg, radiology contrast orders).
We requested that individual physicians be order set owners. When unable to assign ownership, department heads assigned owners.
Standards for Order Specification and Sequence Created
We created an order set standardization guide (Guide) modeled after software development reference manuals. It included standards for how 98 orders would be formatted in order sets and in which sequence the orders would appear within order sets.
The Guide also contained detailed templates for emergency department, admission, and postoperative order sets. These templates were organized to include an order set name, last modified date, clinical owner, target age group, and links to related policies. Following this general information, orders from the Guide appeared within the following sections: admission information, nursing, nutrition, continuous infusions, medications, laboratory, radiology, diagnostic tests, consults/therapies, and discharge information. Finally, templates included a review checklist (as discussed in the next section).
The risk of amplifying errors through standardization was recognized and mitigated by using robust peer review of the new clinical content standards by multiple physicians and clinical nurse specialists. Disputes regarding clinical standards were resolved by the pediatrician-in-chief.
Multidisciplinary Approval Documented
We created a review checklist to ensure that all order sets were approved by the physician owner and appropriate clinical nurse specialist. Our pain service reviewed order sets with opioids, and our infectious disease service reviewed order sets with antibiotics. Laboratory, radiology, and nutrition reviewed orders pertinent to their domains. In addition, pharmacy reviewed all order sets containing medications. Orders in the Guide were preapproved for use in future order sets. The order set specification (Microsoft Word document) was sent by e-mail to all identified reviewers, with comments reconciled electronically until everyone was satisfied. Specifications were updated to match exactly what was built in CIS. Owners were notified of any significant changes resulting from order set build.
In the pilot, a single informatician reviewed all order sets in production to ensure that there were no errors which could harm patients. However, even with only 20 order sets, order sets would start to blur together, where a detail-oriented reviewer with fresh eyes could accomplish this task more quickly and accurately. We changed this process so that order sets were reviewed in the testing domain, defects were recorded, and corrections were made before building in the production domain; we typically used at least 2 different physicians to review each order set.
We defined the order set standardization and update process reflecting the aforementioned steps (Fig 2).
Methods of Evaluation
We planned to estimate the impact of this project by examining the number of hospital encounters in which updated order sets were used. We also wanted to examine changes made as a result of this process. We extracted existing CIS order sets into Microsoft Word order set specifications, and recorded all changes by using the Track Changes function. We examined all the changes made during the review process through qualitative analysis of themes, assuming that the final work products represented the gold standard.
After analysis, 235 order sets were included in the project. This work was performed from July 2010 to February 2011 (Fig 3) following the new standardization and update process. Eighteen order sets were removed because they were obsolete, and 39 were postponed because the divisions were actively updating clinical guidelines related to the order sets. A total of 178 order sets went through the standardization process (Fig 4). New clinical content standards were created (Table 3).
There were a wide range of errors corrected (Table 4). Multidisciplinary review identified medication issues in 37% of order sets (used for 47.6% of inpatient admissions). Some orders for epinephrine had the wrong route (eg, should be intramuscular). There were nonformulary medications or unapproved routes for medication specified. There were a number of errors in which the dose of perioperative antibiotics was inconsistent with the formulary and the perioperative antibiotic policy. Sometimes the errors were more subtle: the order set would apply to all age ranges (including neonatal), but the dosing provided would only be for large children (either in dose or frequency). There were orders for high-risk medications that did not have a duration specified, so medications that should have been available only for procedural sedation in the ICU were ordered and on the patient’s medication administration record and available for administration even after the procedure was completed. There were also errors in laboratory orders, duplicate orders within order sets due to build issues, and incorrect nursing orders that were identified.
There were several key successes of our project. We developed a reliable method to create, update, review, and approve order sets. We engaged our clinical community and put the ownership of order set content into the hands of the clinical experts. We created a Guide to ensure that the standards developed are documented, maintained, and, when indicated, updated to reflect current institutional and patient care standards. Our project identified and corrected orders that were unclear or posed potential patient safety issues.
We faced several challenges throughout the course of the project. Early on, we had difficulty engaging our clinical community. Busy clinicians would take a long time to review the order set specifications and would not return them in a timely manner. Hospital leadership had to step in by having multiple divisions review and approve their order sets on an institution-wide patient safety day. Eventually, the culture did change from clinicians having the view that CIS was not changeable in a timely manner, to feeling like their divisions owned the content and could update order sets as needed.
We communicated with subject matter experts and reviewers by sending out a Microsoft Word document for review and reconciling comments offline. We found this method did not scale well to updating so many order sets. Reconciling comments was tedious, time-consuming, and created version control issues. Furthermore, e-mail use by reviewers was inconsistent and sporadic. It was best when we could bring the multidisciplinary group of stakeholders together in a room to discuss changes; in the long run, this was much simpler than having complex discussions over e-mail. Migrating toward a single master version of each order set helped to reduce confusion and errors.
The team made several adjustments to the management of project resources during the project. Despite the fact that they do not order, nurse and pharmacist review were instrumental in ensuring that order sets respected existing clinical workflows; the project sponsors ensured that these reviews occurred for every order set. Also, the team was relocated to work at a single location 2 days per week for 3 months, improving communication efficiency. In this workspace, posters of order sets in different phases of development cued analysts and informaticians regarding work to be done. Co-locating the team had the unexpected benefit of improving camaraderie as well as expediting communication. It was pleasant to work closely together and to interact in an environment without outside interruptions. It provided a supportive environment with resultant improved morale, collaboration, and productivity gains.
We found that from a resource perspective, the CIS analysts were less busy at the beginning of the project but were extremely busy (up to 5 analysts) at the end of the project. An unexpected benefit of creating clinical standards for orders was that through repetition, the nonclinical analysts found it easier to build clinically correct content because the orders to build were consistent.
We found that having a program manager/administrative assistant role was invaluable in helping to track project status and scope changes. Some divisions took this opportunity to overhaul their ordering approach, choosing to reorganize clinical content between order sets to improve the overall ordering experience; other hospital groups such as pain management asked us to eliminate all acetaminophen combinations from our order sets. Incorporating these changes related to other hospital initiatives expanded our timeline from 7 to 10 months. Although these changes to project approach and scope affected the timeline, they resulted in a higher quality product.
We have continued to refine our processes. We now use simplified document management with 1 master copy available online instead of many copies distributed by e-mail. We have 4 in-person meetings when creating CDS: a design and an approval meeting for the creation and sign-off of the Microsoft Word document specifications, and BUILD and PROD reviews of the actual order set in a testing and in the production systems. Getting everyone into a room has been more efficient than having to support document version control and reconciliation, as has having quality, safety, and usability reviews occurring in the build and production domains. As we have learned, fresh eyes can catch subtle errors.
Despite the increasing use of CPOE, little evidence exists supporting the need for systematic approaches to order set creation through standard and rigorous development, review, and implementation processes sustained over time. Vendors and organizations have not determined best practices for customizations, resulting in systems with poor usability and unintended consequences of use.7
The major patient safety research relating to implementation of CPOE shows varied results in patient mortality,8–10 which may be attributed to the approach these institutions applied to order set development and implementation. To our knowledge, this is the first article that discusses order set standardization as an approach to computerized CDS that may improve patient safety.
SCH is a unique site to plan a project of this nature because the hospital culture tolerates continual measurement and incremental improvement.11 Where there were difficulties, any team member could “stop the line,” forcing us to review, discuss, and incrementally change processes until common understanding and work habits were well established.
It is difficult to predict the generalizability of our methods. SCH has used CPI methods for many years to improve patient care. All employees receive basic training, with leaders receiving advanced training. This underlying philosophy enabled our project to identify barriers to success and rapidly improve our processes over the course of the project. In addition, this culture of change likely led to the ease at which we were able to implement new order set standards without much resistance from clinicians. Institutions not well versed in similar process improvement methods may be challenged to apply this type of rapid change.
Sustaining our project’s gains received much focus, especially by our project sponsors. Most of our order sets were developed with our institution’s initial implementation of CPOE in 2003. Many order sets were not reviewed again until this project. There is risk of this happening again, lacking a formal/regular review process. Some order sets will be updated as new clinical pathways are developed. Pathway maintenance includes comprehensive literature surveillance and regular order set review.
We are working on incorporating order set review into our hospital processes for formulary and policy change review. Failure to fully integrate order set review into the many avenues for change at the institution will ultimately result in the inability of our project to maintain its gains over time.
Careful design, review, and maintenance of electronic order sets are vital to providing effective CDS to promote safe, evidence-based patient care. However, it is not merely sufficient to create CDS; it must be maintained to conform to the latest patient care evidence, medication formulary standards, and information services standards. Often, insufficient time is allotted to maintenance, resulting in outdated or inconsistent CDS that may contribute to patient harm. Our article implies that it is not enough for electronic health records to implement CPOE, but that a certain level of quality in usability and rigorous content development practices is needed, which may explain some of the conflicting results in the literature regarding CPOE and patient safety outcomes.
Providing a consistent user experience through a standard order set format improves usability, while ensuring that ordering clinicians with varying levels of training and experience place orders adhering to institutional standards.12 This foundational work not only supports consistency for ordering clinicians but also to the downstream consumers of these orders (eg, nursing, pharmacy, laboratory), reducing rework and improving efficiency of patient care (eg, decreased pharmacy or nurse calls for unclear orders, decreased reordering). Future studies are needed to better quantify these less visible benefits.
Tracking order set revisions and approvals is invaluable in reconciling discrepancies among and between order sets. Over time, this is critical in addressing questions related to order set content and may prove to be a routine tool used in reviewing patient safety events.
Multidisciplinary review and consensus is frequently discussed as an important piece of quality patient care but may be overlooked in order set development and implementation. We found this incredibly valuable to the quality of our order sets. Multidisciplinary review leads to invigorating discussions among colleagues within and across almost every discipline in the institution. In particular, including institutional leaders intimately familiar with hospital policy and procedure in multidisciplinary review ensures order sets reflect hospital policies and procedures. It is often challenging to engage multidisciplinary stakeholders who are frequently overburdened with administrative and clinical responsibilities. However, these individuals and the groups they represent are instrumental in establishing and maintaining a robust order set catalog to provide clear, up-to-date CDS.
Finally, verifying that medications are dosed correctly, matching the hospital drug formulary, is essential in order set development and maintenance. Many medications used in pediatric institutions have little or no available information on pediatric-specific dosing from the drug manufacturers. Engaging pharmacy partners and using an institution’s pharmacy resources (eg, on-line drug formulary, medication code sheets) will ensure that medication orders are safe and evidence based.
As the government encourages hospitals to rapidly implement electronic health records, it is equally important to ensure that CDS and other customized clinical content is carefully designed and subject to rigorous development process, as CDS have broad implications for care delivery.
We thank SCH administration for funding this strategic initiative. We also acknowledge and thank Tony Payauys, PMP, for his work developing the methodology for project execution.
- Accepted December 20, 2012.
- Address correspondence to Michael G. Leu, MD, MS, MHS, Clinical Effectiveness, M1-7, Seattle Children's Hospital, 4800 Sand Point Way NE, Seattle, WA 98105. E-mail:
Drs Leu and Morelli, Ms. Chung, and Ms. Radford have had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
This work was presented at a Cerner webinar on August 28, 2012.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
- ↵Osheroff JA, Pifer EA, Teich JM, Sittig DF, Jenders RA. Improving outcomes with clinical decision support: an implementer’s guide. First Edition. HIMSS. 2005. Available at: www.himss.org/content/cdsw/2005/introduction_2005.pdf. Accessed March 5, 2012
- Ash JS,
- Stavri PZ,
- Kuperman GJ
- Teufel RJ II,
- Kazley AS,
- Basco WT Jr
- Liker JK
- ↵Institute for Safe Medication Practices. ISMP’s Guidelines for Standard Order Sets. ISMP Web site. Available at: www.ismp.org/Tools/guidelines/StandardOrderSets.pdf. Accessed March 15, 2012
- ↵McDonnell C, Werner K, Wendel L. Electronic Health Record Usability: Vendor Practices and Perspectives. Rockville, MD: Agency for Healthcare Research and Quality; May 2010. AHRQ Publication No. 09(10)-0091-3-EF
- Han YY,
- Carcillo JA,
- Venkataraman ST,
- et al
- Del Beccaro MA,
- Jeffries HE,
- Eisenberg MA,
- Harry ED
- Longhurst CA,
- Parast L,
- Sandborg CI,
- et al
- Wellman J,
- Jeffries H,
- Hagan P
- Copyright © 2013 by the American Academy of Pediatrics