May 2016, VOLUME137 /ISSUE 5

Implementation of Electronic Clinical Decision Support for Pediatric Appendicitis

  1. Anupam B. Kharbanda, MD, MSca,
  2. Manu Madhok, MD, MPHa,
  3. Ernest Krause, BAb,
  4. Gabriela Vazquez-Benitez, PhDc,
  5. Elyse O. Kharbanda, MD, MPHc,
  6. William Mize, MDd, and
  7. David Schmeling, MDe
  1. aDepartments of Pediatric Emergency Medicine,
  2. bResearch and Sponsored Programs,
  3. dRadiology, and
  4. eSurgery, Children’s Hospitals and Clinics of Minnesota, Minneapolis, Minnesota; and
  5. cHealthPartners Institute, Minneapolis, Minnesota.
  1. Dr A.B. Kharbanda conceptualized and designed the study, conducted data analyses, analyzed and interpreted the data, drafted the initial manuscript, revised the manuscript critically for important intellectual content, and approved the final manuscript as submitted; Drs Madhok, Mize, and Schmeling and Mr Krause were involved in the design of the study, participated in the interpretation of the data, reviewed and revised the manuscript, and approved the final manuscript as drafted; and Drs Vazquez-Benitez and E.O. Kharbanda analyzed and interpreted the data, reviewed and revised the manuscript critically for important intellectual content, and approved the final manuscript as submitted.


BACKGROUND AND OBJECTIVE: Computed tomography (CT) and ultrasound (US) are commonly used in patients with acute abdominal pain. We sought to standardize care and reduce CT use while maintaining patient safety through implementation of a multicomponent electronic clinical decision support tool for pediatric patients with possible appendicitis.

METHODS: We conducted a quasi-experimental study of children 3 to 18 years old who presented with possible appendicitis to the pediatric emergency department (ED) between January 2011 and December 2013. Outcomes were use of CT and US. Balancing measures included missed appendicitis, ED revisits within 30 days, appendiceal perforation, and ED length of stay.

RESULTS: Of 2803 patients with acute abdominal pain over the 3-year study period, 794 (28%) had appendicitis and 207 (26.1% of those with appendicitis) had a perforation. CT use during the 10-month preimplementation period was 38.8% and declined to 17.7% by the end of the study (54% relative decrease). For CT, segmented regression analysis revealed that there was a significant change in trend from the preimplementation period to implementation (monthly decrease –3.5%; 95% confidence interval: –5.9% to –0.9%; P = .007). US use was 45.7% preimplementation and 59.7% during implementation. However, there was no significant change in US or total imaging trends. There were also no statistically significant differences in rates of missed appendicitis, ED revisits within 30 days, appendiceal perforation, or ED length of stay between time periods.

CONCLUSIONS: Our electronic clinical decision support tool was associated with a decrease in CT use while maintaining safety and high quality care for patients with possible appendicitis.

  • Abbreviations:
    best-practice advisory
    clinical decision support
    clinical practice guideline
    computed tomography
    emergency department
    electronic health record
    information technology
    right lower quadrant
  • Each year, >70 000 pediatric patients are diagnosed with appendicitis, the most common surgical emergency in children.1 Despite its high incidence, up to 40% of children with appendicitis are initially misdiagnosed.2 There is also considerable variability in the emergency department (ED) evaluation of children with possible appendicitis, specifically in the use of diagnostic imaging.36 In a survey of 40 Children’s Hospitals, among children with appendicitis, computed tomography (CT) use ranged from 21% to 49%, and ultrasound (US) use ranged from 2% to 26%.6

    CT has been promoted as a method to improve diagnostic accuracy in patients with possible appendicitis. Unfortunately, recent increases in CT use nationwide5,7,8 have not led to meaningful decreases in rates of appendiceal perforation.1,3,912 In addition, potential overuse of CT is problematic given the risks of radiation exposure. It is estimated that 1 radiation-induced malignancy may result from every 700 abdominal CT scans obtained in young males, and the risks may be even higher in young females.1315 Although, there are no guidelines endorsed by national physician organizations for the management of children with possible appendicitis, there is increasing consensus on the need to limit unnecessary CTs.2,16,17

    Clinical practice guidelines (CPGs) to standardize care and reduce reliance on abdominal CT have been developed and implemented in pediatric and general EDs.1620 Previous studies have demonstrated that it is possible, within a single academic ED, to safely limit the use of advanced diagnostic imaging in patients with suspected appendicitis.21,22 To impact care over the long term, a CPG must be well integrated into the clinical workflow and provide information appropriate to the clinical scenario.23,24 Clinical decision support (CDS) delivered through the electronic health record (EHR) provides a potentially innovative and scalable method for integrating CPGs into routine care within large health systems. Use of EHR-linked CDS can help limit the need for nurse or provider training and ensures that targeted guidelines are displayed at the point of care.25 Studies in adult EDs have demonstrated that EHR-linked CDS improved adherence to guidelines and outcomes for a variety of emergent conditions.2628

    Despite the need to standardize care and limit radiation exposure in children with possible appendicitis, we are not aware of any published EHR-linked tools to guide ED clinicians in the evaluation and management of this patient population. In the current study, we developed, implemented, and evaluated the safety and effectiveness of a novel EHR-linked CDS tool for patients with suspected appendicitis within a large Midwestern pediatric hospital system. Our primary goal was to safely reduce CT use in this population, without increasing total imaging or rates for adverse events.


    Study Design and Setting

    Using a quasi-experimental design, we conducted a quality improvement study of patients 3 to 18 years old with possible appendicitis who presented to 2 urban, tertiary care pediatric EDs from January 1, 2011 through December 31, 2013. The 2 sites collectively treat 95 000 pediatric patients annually. Both EDs are part of a single pediatric health system and are staffed by the same pediatric emergency medicine, pediatric radiology, and pediatric surgical groups. ED staff include 47 pediatric emergency medicine physicians, 8 nurse practitioners, and 9 pediatric emergency medicine fellows. US and CT were available 24 hours a day, 7 days a week, with all imaging studies interpreted by an attending pediatric radiologist. The study was approved with a waiver of informed consent by the Children’s Hospitals and Clinics of Minnesota Institutional Review Board.

    Intervention: Guideline Development

    As the evaluation of children with possible appendicitis is variable, we designed an evidenced-based CPG for these patients. The guideline was based on review of the literature and was developed by a multidisciplinary team that included: 1 pediatric surgeon, 3 pediatric emergency physicians, 1 pediatric radiologist, and members of the hospital information technology (IT) team. The guideline team met from January through August 2011 and agreed to the guideline as well as the process for integrating the guideline into the clinical workflow, as described in detail below. The guideline was shared with additional clinicians from the ED, radiology, and surgery and with ED nursing leadership for comment. Once consensus was obtained, medical directors from the respective departments reviewed and approved the guideline.

    The electronic CDS tool included 3 components: a standardized abdominal pain order-set, a Web-based risk stratification tool, and a “time of ordering alert.” The order-set specified options for pain medications and laboratory tests (eg, white blood cell [WBC] count) and contained a hyperlink to the Web-based risk stratification tool. The stratification tool provided information on classifying patients with acute abdominal pain into the following risk categories: high (>90%), medium, or low risk (<5%) for appendicitis. Criteria to risk stratify used a combination of the pediatric appendicitis score29 for identifying patients at medium or high risk and the “low risk rule”20 to identify patients at low risk for appendicitis (Supplemental Fig 3). Next steps in management were provided, based on risk group. For high risk patients, surgical consultation was recommended before diagnostic imaging; imaging was ordered at the discretion of the surgical attending physician. Low risk patients were recommended for discharge without imaging and with outpatient or ED follow-up in 12 to 24 hours. A focused abdominal US was recommended for medium risk patients; CT was to be considered if the US was equivocal, unavailable, or at the request of the surgical attending physician. The risk stratification tool was provided as guidance, to be used in conjunction with clinical judgment.

    The third component of the care pathway was a best-practice advisory (BPA), which appeared when a clinician attempted to order a CT or US. This BPA provided guidance on risk stratification and recommendations for diagnostic imaging (as outlined above).

    Implementation of the Guideline

    To promote awareness, agreement, and adoption of our guideline, we introduced the standardized approach and EHR-linked tools to clinicians in surgery, radiology, and emergency medicine via brief (20 minute) group and individual training sessions over a 4-week time period (October 2011). During the implementation period, group quarterly e-mails were sent to ED providers describing the guidelines and risk stratification approach. The order-set, BPA, and associated care pathway were turned on the week of November 8th, 2011. The start date of the intervention was based on the date when the tools were turned on in the EHR. We monitored performance of the guideline on a monthly basis; summary reports were provided to department medical directors after the first intervention year (December 2012) to reinforce institutional support and encourage continued adherence with the guideline.

    Identifying the Population of Interest for Outcome Assessment

    We identified our population of interest using a combination of automated and manual processes. Our goal was to include pediatric patients who presented with right-sided or diffuse abdominal pain for whom the clinician obtained laboratory testing, a diagnostic imaging study, or a surgical consult. To do this systematically, we first developed an automated monthly query of the EHR (Cerner, Kansas City, KS) to identify patients 3 to 18 years old with: (1) chief complaint of abdominal pain at ED triage; or (2) an ICD-9 code for appendicitis or perforated appendicitis. For the resulting cohort, we then used automated methods to exclude patients with ICD-9 code(s) for selected chronic medical conditions (eg, active cancer, immunodeficiency, diabetes, metabolic disease, cystic fibrosis, sickle cell anemia, and inflammatory bowel disease) where the WBC count may be attenuated. Exclusion criteria were defined a priori to identify a cohort similar to that evaluated, by our group and others, in previous studies of appendicitis risk.

    Medical records for the remaining cohort of patients were then manually screened by trained research assistants to identify additional exclusion criteria: (1) acute abdominal pain of >96 hours duration; (2) no documented history of right lower quadrant (RLQ) or diffuse abdominal pain; (3) pain that was preceded by trauma; (4) transfer from outside facility with US or CT already completed; (5) pregnancy; and (6) history of abdominal surgery (Fig 1). All patient records excluded during this process were reviewed by the study coordinator and study principal investigator for accuracy and were re-coded if necessary.


    Effectiveness outcomes were: use of CT, US, and total imaging. All US and CT reports were manually reviewed. Safety outcomes were: missed appendicitis, negative appendectomies, appendiceal perforation, and ED length of stay. Medical records were reviewed for 30 days after the index ED visit. Missed appendicitis was defined as a case in which appendicitis was not diagnosed at the index ED visit, but the child subsequently returned to the ED within 30 days and underwent an appendectomy. Negative appendectomies and appendiceal perforation were identified through manual review of operative and pathology reports.

    Additional covariates abstracted from the EHR included history of anorexia, nausea, vomiting, and migration of pain to the RLQ, physical exam findings, and highest temperature during ED visit. Laboratory values collected included WBC, C-reactive protein, neutrophil, and band percentage. Two trained research assistants conducted full chart abstractions for the variables above using a standardized abstraction tool. In addition, 10% of charts were abstracted by an independent reviewer to ensure accuracy. Agreement for all variables of interest was very good (κ > 0.75). Discrepancies were resolved by the principal investigator of the study.

    Data Analysis

    Demographic and clinical findings of study subjects in the preimplementation and implementation periods were described using frequency distributions and medians. Missing data were excluded from analysis. We compared the preimplementation and implementation cohorts using the χ2 test for categorical variables and the Mann–Whitney test for continuous variables.

    Segmented regression analysis was used to evaluate the effectiveness of the electronic CDS tool, including a 10-month preimplementation period and a 26-month implementation period. Segmented regression analysis can detect whether an intervention effect is significantly greater than the secular trend. Percent change over time in the rate of the outcome during the implementation period was compared with the percent change over time of the preimplementation period. Data were aggregated by month; data from the first week of the implementation period were excluded. We used a Poisson regression model and scaled for observed overdispersion. Model parameters included intercept, time trend before implementation, implementation, and time trend after implementation. A second model was implemented incorporating age and gender as covariates. Statistical analyses were performed in SAS/STAT 9.3 (SAS Institute, Cary, NC), and plots were generated by using R Package Stats (; version 3.1.1).


    Study Population

    Over the 3-year study period, a total of 2852 patients with acute abdominal pain were analyzed, including 767 patients preimplementation and 2085 postimplementation (Fig 1). After excluding 49 subjects from the week of CDS implementation, our final intervention cohort included 2803 subjects. The number of eligible patients per month ranged from 49 to 99. In Table 1, we present the demographic and clinical characteristics of our cohort. Patients during both periods did not differ significantly by demographic characteristics or rate of appendicitis. The cohorts did differ by presence of nausea, emesis, and migration of pain. In addition, WBC counts differed slightly between time periods. Over the 36-month study period, 794 (28%) patients were diagnosed with appendicitis, of whom 207 (26.1%) had a perforated appendix. The rate of appendicitis was not significantly different between time periods (27% vs 29%, P = .38).

    TABLE 1

    Overall Patient Demographics and Characteristics

    Primary Outcomes

    Rates of CT, US, and total imaging are presented in Table 2 and Fig 2 A, B, and C. The rate of CT use during the preimplementation period was 38.8%, with no significant trend toward increasing or decreasing use. During the implementation period, CT use declined each month by 2.5% (95% confidence interval [CI]: –3.3% to –1.7%) with a significant change in trend from the preimplementation period (–3.5 [95% CI: –5.9% to –0.9%, P = .007]). This decline resulted in a 54% relative decrease in CT use from the preimplementation period to the end of the study (38.8% to 17.7% end of study).

    TABLE 2

    Association of Electronic CDS With Primary Outcomes: Interrupted Time-Series

    FIGURE 2

    Use of CT (A), US (B), and imaging (C). Vertical green dashed lines represent guideline introduction; vertical red lines represent when electronic decision support was activated. aIndicates when summary reports were distributed; dashed lines in blue represent the 95% confidence band. Labels for alternative months were eliminated for better display.

    Increasing use of US preceded the implementation of the CDS tool. Preimplementation, the percent increase of US per month was 2.7% (95% CI: –0.1 to 5.7]; during implementation, the upward trend in US use persisted, with a 1.4% percent increase per month (95% CI: 0.8% to 2.1%). There was no significant change in US trend from the preimplementation to implementation period (Table 2). Preimplementation, 72.4% of subjects had any imaging (US or CT). Imaging rates were increasing in this period at a monthly rate of 1.8% (95% CI: 0 to 3.7). Total imaging continued to increase during the implementation period at a monthly rate of 0.5% (95% CI: 0 to 0.9). There was no significant change in the total imaging trend over time. Models adjusted for age and gender were evaluated, but findings for US, CT, and imaging use were unchanged, so only results from the unadjusted models are presented.

    Balancing Measures

    We noted no significant changes from the preimplementation to implementation periods in rates of appendiceal perforation, return visits to the ED within 30 days, or ED length of stay (Table 3). Finally, for rare safety outcomes, we were not able to use a time series approach. However, we noted no significant differences in the rates of negative appendectomies by time period (1.9% during preimplementation vs 2.5% during implementation) or missed appendicitis (0.7% during preimplementation vs 0.3% during implementation).

    TABLE 3

    Association of Electronic CDS With Safety Outcomes: Interrupted Time-Series


    In this quality improvement study, we have demonstrated that an EHR-linked CDS for children with possible appendicitis can reduce CT use at 2 busy pediatric EDs within a large pediatric health system. Over the course of our study, we noted a consistent drop in CT use with a relative aggregate drop of 54%. In addition, there was no significant change in overall imaging use or our balancing measures that evaluated the safety of our intervention. Our study adds to the growing body of literature demonstrating the effectiveness, feasibility, and safety of leveraging the EHR to improve care.

    The ED offers an ideal venue for standardizing care through CDS and delivering targeted clinical recommendations via the EHR.30 As ED providers are charged with assessing and treating acutely ill patients both accurately and quickly, point-of-care CDS may improve decision-making in this setting. For example, a recent study by Dean et al26 demonstrated that an EHR-linked CDS could improve adherence with pneumonia management guidelines. Our intervention consisted of several relatively simple tools: (1) an order-set that predefined a standardized approach to children with possible appendicitis; (2) a time-of-ordering alert reinforcing the consensus clinical practice guideline; and (3) a method to risk stratify children. In contrast to previous interventions requiring monthly reminders,16,17 after the initial brief educational efforts conducted around the time of implementation, limited guidance or reinforcement was provided.

    Multiple previous studies from academic centers have evaluated the benefits of standardizing care for children with acute abdominal pain through the use of CPGs that incorporate risk stratification.16,17,19,22 The methods for assigning risk for appendicitis differed across these studies. For example, Santillanes et al17 based their guideline on the work of Garcia Pena et al,31 Fleischman et al22 developed a CPG based on the work of Samuels29 and Kharbanda et al,20 whereas Russell et al21 developed a CPG that emphasized early surgical consultation and the use of ultrasound. Regardless, these studies have shown a reduction in CT use, with effect sizes of around 50%.21,22 Most previous studies have had limited follow-up periods, and the long-term effectiveness of these CPGs is not known. In addition, despite the demonstrated effectiveness of CPGs in these previous studies, barriers to widespread adherence to guidelines have been well described, including lack of awareness or relevant information not being presented at the point of care.32,33 For this reason, in the current project, we developed an EHR-linked CDS, providing clinicians with contextually appropriate information integrated into the ED workflow.

    We were reassured by the sustained decrease in CT use over the course of our study. In addition, this decline was not associated with an increase in adverse events including appendiceal perforation, missed appendicitis, negative appendectomy, or repeat ED visits. These findings are consistent with previous studies demonstrating stable negative appendectomy rates (range, 1% to 5.2%) and missed appendicitis rates (2%)17,22 during periods of decreased CT use. We believe our results will facilitate adoption of similar EHR-linked tools for treatment of pediatric acute abdominal pain across EDs.

    Our total imaging rates, although not statistically different between time periods, remained high during the course of the study. This is an area of on-going work at our institution, because simply replacing CT with US should not be the goal of a CDS tool. Future modifications to our CDS will aim to reduce US use and improve the quality of US interpretation. Areas to address include: targeting populations for US based on BMI, reducing equivocal readings on US through templated interpretation, and using inpatient observation for select populations. In addition, when considering the potential for widespread dissemination of this and other similar appendicitis CPGs, it will be important to evaluate the short- and long-term economic impacts of decreasing CT use and exposures to ionizing radiation while potentially increasing patient time in the ED. Finally, our efforts should provide confidence to other institutions attempting to implement clinical guidelines. Our experience would indicate that key elements for successful implementation include: (1) creating a collaborative guideline committee to ensure widespread acceptance; (2) obtaining support of leadership, especially in IT; and (3) integrating the CPG into the clinical workflow.

    Several additional limitations should be noted. First, despite the use of an interrupted times series for analysis, it is possible that trends in CT use preimplementation and postimplementation were because of other temporal or secular changes in clinical practice. Although a cluster randomized trial was considered, it would have required additional clinical sites and was not feasible within the scope of the current study. Second, we did not track the use of the CDS tools. We expected that, over time, clinicians would learn to anticipate the CDS prompts, and thus they would not always open the CDS risk stratification tool. Yet, a mechanism to track use and provide targeted feedback at the individual provider level should be considered as part of future efforts, because audit/feedback has been shown to improve compliance with CPGs.34 Third, we were not able to conduct telephone follow-up interviews to ensure that patients discharged from the hospital did not have an appendectomy at an outside institution. However, as the largest provider of acute care for children in our region, we are reassured that children in our cohort with an initial ED visit would return for additional care within our network. Finally, we did note clinical differences between the preimplementation and implementation time periods. We were reassured that the rate of appendicitis was similar, but other differences were likely due to the retrospective nature of our data abstraction. In future evaluations, it would be preferable to develop fully automated methods to identify the population of interest.


    We have demonstrated that a multicomponent CDS, linked within the EHR, providing guidance for the management of patients with possible appendicitis can reduce the use of CT while maintaining safety and high quality care for patients.


    We thank David Gustafson, MD, for his assistance in developing the CDS tool with our IT department. In addition, we thank Sam Kim, Meghan Udoeyop, and Kristin Frenn for their diligence and attention to detail in conducting the chart reviews.


      • Accepted October 16, 2015.
    • Address correspondence to Anupam B. Kharbanda, MD, MSc, Department of Pediatric Emergency Medicine, 2525 Chicago Ave South, MDB 17-104 Minneapolis, MN 55404. E-mail: anupam.kharbanda{at}
    • This work was presented at the annual meeting of the Pediatric Academic Societies, May 5th, 2013; Washington, DC; and at Children’s Hospital Research Day, December 8th, 2014, Minneapolis, MN.

    • FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

    • FUNDING: This work was supported by Children’s Hospital’s internal grant program.

    • POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.