OBJECTIVES: Standard metrics for evaluating rapid response systems (RRSs) include cardiac and respiratory arrest rates. These events are rare in children; therefore, years of data are needed to evaluate the impact of RRSs with sufficient statistical power. We aimed to develop a valid, pragmatic measure for evaluating and optimizing RRSs over shorter periods of time.
METHODS: We reviewed 724 medical emergency team and 56 code-blue team activations in a children’s hospital between February 2010 and February 2011. We defined events resulting in ICU transfer and noninvasive ventilation, intubation, or vasopressor infusion within 12 hours as “critical deterioration.” By using in-hospital mortality as the gold standard, we evaluated the test characteristics and validity of this proximate outcome metric compared with a national benchmark for cardiac and respiratory arrest rates, the Child Health Corporation of America Codes Outside the ICU Whole System Measure.
RESULTS: Critical deterioration (1.52 per 1000 non-ICU patient-days) was more than eightfold more common than the Child Health Corporation of America measure of cardiac and respiratory arrests (0.18 per 1000 non-ICU patient-days) and was associated with >13-fold increased risk of in-hospital death. The critical deterioration metric demonstrated both criterion and construct validity.
CONCLUSIONS: The critical deterioration rate is a valid, pragmatic proximate outcome associated with in-hospital mortality. It has great potential for complementing existing patient safety measures for evaluating RRS performance.
- critical illness
- early warning score
- hospital rapid response team
- medical emergency team
- physiologic monitoring
- ARC —
- acute respiratory compromise
- CBT —
- code-blue team
- CHCA —
- Child Health Corporation of America
- CHOP —
- The Children’s Hospital of Philadelphia
- CI —
- confidence interval
- CPA —
- cardiopulmonary arrest
- DNAR —
- do not attempt resuscitation
- MET —
- medical emergency team
- ROC —
- receiver operating characteristic
- RRS —
- rapid response system
What's Known on This Subject:
The availability of rapid response systems to assist deteriorating patients is the standard of care in children’s hospitals. Metrics for evaluating their effectiveness include cardiac and respiratory arrest rates, rare events that require years of data to show significant improvements.
What This Study Adds:
A proximate outcome for in-hospital mortality among patients receiving rapid response system assistance was developed. This “critical deterioration” metric was eightfold more common than arrests and demonstrated criterion and construct validity, facilitating meaningful evaluation over shorter periods of time.
In recent years, rapid response systems (RRSs) have been implemented at hospitals around the world in attempts to reduce rates of cardiac arrest, respiratory arrest, and mortality outside the ICU. RRSs have 2 clinical components, an identification arm and a response arm. The identification arm consists of predictive tools to identify patients at risk for deterioration over time and detective tools to identify actively deteriorating patients who need immediate assistance.1 The response arm consists of medical emergency teams (METs) that clinicians caring for patients outside the ICU can summon to the bedside of patients exhibiting signs of deterioration. In addition to their clinical components, RRSs also include an administrative arm that oversees RRS operations and a process improvement arm that evaluates RRS outcomes. Due in large part to the Institute for Healthcare Improvement’s 5 Million Lives Campaign with >4000 participating hospitals,2–4 presence of an RRS is now the standard of care.
Despite their intuitive appeal and wide dissemination, the success of pediatric RRSs in reducing respiratory arrest, cardiac arrest, and death (rare events that are critically important to prevent) has been highly variable among hospitals.5–14 Because death and cardiac arrest happen infrequently in hospitalized children, finding statistically significant changes in their rates often requires accumulating years of data. Thus, rapid-cycle evaluation and optimization projects undertaken by the RRS’s process improvement arm often are underpowered by using traditional measures. Nevertheless, a commonly used dashboard metric for evaluating within-hospital patient safety success and comparing performance between hospitals is the quarterly rate of exceedingly rare cardiac and respiratory arrest events outside the ICU.15 Small, statistically insignificant differences in this rate may be used to inappropriately draw positive or negative conclusions about performance.
In this study, we aimed to develop a pragmatic, valid, proximate outcome associated with in-hospital mortality that occurs frequently enough to provide the statistical power needed to evaluate RRS performance over a period of months, not years.
We performed this retrospective cohort study at The Children’s Hospital of Philadelphia (CHOP), an urban, tertiary care pediatric hospital with 473 beds, of which 75 are dedicated to neonatal, 55 to pediatric, and 26 to cardiac intensive care. On February 8, 2010, the hospital implemented an RRS available to all of the non-ICU medical and surgical units except the cardiology step-down, tracheostomy-ventilator, and obstetric units. The RRS consisted of (1) an identification component, including an early warning score with corresponding care guidelines, and (2) a response component with a 30-minute response MET available 24 hours per day, 7 days per week. Informal ICU “curbside” consultation was eliminated. An immediate-response code-blue team (CBT) remained in place. Additional information about the RRS’s characteristics is available in Supplemental Tables 3–5.
We reviewed each MET and CBT activation in the 1-year period after implementation (February 9, 2010 to February 8, 2011). MET activations were logged on paper forms by the responding nurse and subsequently entered into a database by a research assistant; CBT activations were verified by reviewing CBT messages sent via the pager system and logged directly into a database by the pediatric ICU clinical nurse specialist. We abstracted patient and event characteristics, the outcome of the activation, and discharge disposition from patient charts.
For children who were transferred to the ICU, we reviewed flowsheets for 12 hours after transfer and recorded the time from ICU arrival to life-sustaining interventions, including initiation of continuous or bilevel positive airway pressure, tracheal intubation, and vasopressor infusion administration (ie, dobutamine, dopamine, epinephrine, isoproterenol, milrinone, or norepinephrine). Five of the authors, including experts in pediatric hospital medicine (C.P.B. and R.K.), an international authority in RRSs and cardiopulmonary resuscitation (V.M.N.), an expert in pediatric critical care nursing (K.E.R.), and the medical director of the CHOP pediatric ICU (M.A.P.), selected these items by consensus. We classified events requiring any of these interventions in the first 12 hours after ICU transfer as “critical deterioration.” We hypothesized that critical deterioration would be a useful proximate outcome for in-hospital mortality.
We also identified all respiratory and cardiac arrests occurring on non-ICU wards during the study period and classified them according to the Child Health Corporation of America (CHCA) Codes Outside the ICU measure.15 CHCA is a business alliance of 43 not-for-profit tertiary care children’s hospitals in the United States. Codes Outside the ICU is 1 of several CHCA Whole System Measures designed to help hospital leaders evaluate their health system’s overall performance on the core dimensions of quality and value. The measures are developed by using a consensus process with a panel of physicians and quality experts from CHCA hospitals. In the CHCA Codes Outside the ICU measure, qualifying events include cardiopulmonary arrest (CPA) and acute respiratory compromise (ARC) events occurring on non-ICU units. CPA events require either pulselessness or a pulse with inadequate perfusion necessitating chest compressions and/or electric shock. ARC events require respiratory insufficiency with bag-valve-mask or invasive airway interventions, followed by transfer to a higher level of care for sustained airway support. The number of combined CPA and ARC events per 1000 non-ICU patient-days is shared among all participating CHCA hospitals on a quarterly basis for internal performance improvement and external comparative benchmarking.
Evaluation of Critical Deterioration as a Proximate Outcome for In-Hospital Mortality
In evaluating the critical deterioration metric as a proximate outcome associated with in-hospital mortality, we first excluded events with do-not-attempt-resuscitation (DNAR) orders at the time of activation. We then analyzed the time to life-sustaining ICU interventions like a diagnostic test for in-hospital mortality, generating a receiver operating characteristic (ROC) curve. We calculated the in-hospital mortality rate, sensitivity, specificity, and positive and negative predictive values for potential time cut points of 1 and 4 hours, and compared those to our critical deterioration metric by using the 12-hour cut point. We compared the test characteristics for each cut point with those of the CHCA Whole System Measure for Codes Outside the ICU among CHOP patients. We selected a time cut point to use for further analysis by identifying a point with high sensitivity and relative risk, without a clinically unacceptable loss of specificity. For admissions with multiple ICU transfers, we used the transfer with the shortest time to life-sustaining intervention for the analysis, because that event likely represented the most severe CBT/MET activation during the admission.
We then evaluated the criterion and construct validity of the proposed proximate outcome. Criterion validity is defined as the correlation of a metric with some other measures of the trait or disorder under study, usually a gold standard.16 As a measure of criterion validity, we evaluated the association between the time to life-sustaining ICU interventions and the gold standard of death by calculating the relative risks of death at potential time cut points of 1 and 4 hours and compared those with our hypothesized critical deterioration metric by using the 12-hour cut point.
Construct validity is defined as the ability of a metric to measure the hypothetical construct it is intended to measure.16 In this study, we used the CHCA measure as the construct and assessed construct validity in 2 ways. First, we determined the proportion of patients meeting the CHCA Whole System Measure for Codes Outside the ICU who also met the criteria for the time to life-sustaining ICU intervention metric at the selected cut point. Second, among patients who were transferred to the ICU and received life-sustaining interventions, we compared the median time to life-sustaining interventions between the patients who met CHCA Whole System Measure for Codes Outside the ICU and those who did not by using the Wilcoxon rank-sum test.
We managed the data by using Microsoft Access 2003 (Microsoft Corp, Redmond, WA), and analyzed it by using Stata 11.1 (StataCorp, College Station, TX). Because a de-identified quality improvement data set was used for the analysis, this project was granted an exemption by CHOP’s institutional review board.
Between February 9, 2010 and February 8, 2011, there were 780 combined MET and CBT activations for 525 patients and 596 admissions over a total of 79 428 non-ICU patient-days and 28 015 all-hospital admissions (9.8 activations per 1000 non-ICU patient-days, 27.8 activations per 1000 hospital admissions, Fig 1). Fifty-six calls (7.2%) resulted in immediate-response CBT activations, and 724 calls (92.8%) resulted in 30-minute response MET activations. Twelve of the CBT activations fulfilled CHCA ARC criteria, and 2 fulfilled CPA criteria. The CHCA Codes Outside the ICU rate at CHOP was 0.18 codes per 1000 non-ICU patient-days (95% confidence interval [CI]: 0.10–0.30 codes per 1000 non-ICU patient-days). The patient and event characteristics are presented in Table 1.
Of the 724 MET activations, 272 (37.6%) resulted in transfer to the ICU, compared with 44 of 56 (78.6%) CBT responses (P < .001). Of the total 780 activations, 121 (15.5%) met our definition of critical deterioration (requiring life-sustaining ICU interventions in the first 12 hours after ward-to-ICU transfer). The rate of critical deterioration was 1.52 per 1000 non-ICU patient-days, more than eightfold more common than CHOP’s CHCA Codes Outside the ICU rate.
Evaluation of Critical Deterioration as a Proximate Outcome for In-Hospital Mortality
Among the 596 admissions comprising the 780 CBT and MET activations, 3 patients (with a combined total of 5 events) were still admitted at the time of analysis, leaving 593 admissions with 775 events with hospital discharge outcome data. We excluded 13 activations for patients with DNAR status among 10 admissions (4 of whom remained eligible for earlier events, before DNAR orders were in place), leaving 762 events among 587 admissions and 519 patients for analysis. Twenty-five of the 587 admissions ended in death, for an overall in-hospital mortality rate of 4.3%. One of the 14 CHCA-qualifying events occurred in a patient with an active DNAR order, leaving 13 CHCA-qualifying events among 12 admissions for analysis. Of those 12 admissions, 5 ended in death, for an in-hospital mortality rate of 41.7%.
Of the 587 admissions with MET or CBT activations, 114 (19.4%) fulfilled critical deterioration criteria. Of those 114 admissions, 19 ended in death, for an in-hospital mortality rate of 16.7%. Of the remaining 473 admissions not requiring life-sustaining ICU interventions in the first 12 hours after ward-to-ICU transfer, 6 ended in death, for an in-hospital mortality rate of 1.3%.
The area under the ROC curve for the time to life-sustaining ICU interventions was 0.82 (95% CI: 0.73–0.91) (Fig 2). The in-hospital mortality rate, test characteristics, and relative risk of death for cut points of 1 and 4 hours compared with the 12-hour critical deterioration cut point are shown in Table 2. The sensitivity of the critical deterioration metric was 76.0% (95% CI: 54.9–90.6), the specificity was 83.1% (95% CI: 79.7–86.1), the positive predictive value was 16.7% (95% CI: 10.3–24.8), and the negative predictive value was 98.7% (95% CI: 97.3–99.5). Compared with life-sustaining ICU intervention cut points of 1 and 4 hours, the 12-hour critical deterioration metric offered higher sensitivity while maintaining specificity >80%. Compared with the ICU transfer cut point (which considers a positive test to be any ICU transfer regardless of whether interventions were required), the 12-hour cut point offers higher specificity and positive predictive value at an only slightly lower sensitivity. The choice of the 12-hour cut point also is supported graphically by the ROC curve, which shows a sharp rise in sensitivity from 1 to 12 hours with minimal loss of specificity, followed by a flattening of the curve between 12 hours and the ICU transfer cut point, representing a substantial loss of specificity with minimal gain in sensitivity. In evaluating the criterion validity of the critical deterioration metric, we found that the relative risk of death was 13.1 (95% CI: 5.4–32.1).
In comparison, the sensitivity of the CHCA measure was 20.0% (95% CI: 6.8–40.7), the specificity was 98.8% (95% CI: 97.4–99.5), the positive predictive value was 41.7% (95% CI: 15.2–72.3), and the negative predictive value was 96.5% (95% CI: 94.7–97.9). In evaluating its criterion validity, we found that the relative risk of death was 12.0 (95% CI: 5.4–26.6).
In our first method of evaluating the construct validity of the critical deterioration metric, we found that 11 of 12 admissions (91.6%) with a CHCA Code Outside the ICU also fulfilled critical deterioration criteria. The 1 patient who did not meet critical deterioration criteria qualified as a CHCA CPA because staff administered chest compressions briefly while the patient was being treated in the ward, but the patient was hemodynamically stable on CBT arrival, was not transferred to the ICU, and likely had not actually experienced a cardiac arrest.
In our second method of evaluating the construct validity, we found that among the 11 admissions with a CHCA code blue who fulfilled critical deterioration criteria, the median time to intervention was 0 minutes (most interventions were started en route to the ICU or immediately on arrival), and the interquartile range was 0 to 15 minutes. Of the 103 admissions not meeting CHCA code-blue criteria but meeting critical deterioration criteria, the median time to intervention was 45 minutes, and the interquartile range was 15 to 170 minutes (P < .001).
To improve the process improvement component of RRSs, we set out to develop a pragmatic measure for their ongoing evaluation and optimization. This study is the first to directly compare the performance of a national benchmark metric to a proximate, correlated outcome and the gold standard of death before hospital discharge. We developed the critical deterioration metric as a useful outcome upstream of in-hospital mortality and demonstrated its criterion and construct validity by evaluating its association with in-hospital mortality and a commonly used dashboard metric, the rate of CHCA Codes Outside the ICU. Critical deterioration occurred >8 times more frequently than the CHCA metric and was associated with a >13-fold increased risk of death among patients who received treatment from the MET and the CBT. Its sensitivity was nearly 4 times that of the CHCA metric, with only a modest loss of specificity.
Cardiac and respiratory arrests defined by using CHCA criteria for CPA and ARC were rare, with only 14 events occurring on non-ICU inpatient units in 1 year, for a rate of 0.18 codes per 1000 non-ICU patient-days. This rate is consistent with post-RRS implementation combined code rates in children’s hospitals.6,9,10 Although these events are important to measure because they represent critically ill children who required emergent resuscitation while hospitalized on non-ICU wards, they are so rare that accruing a sufficient number to detect statistically significant changes can take years. For example, to detect a 50% reduction in code rate from 0.20 to 0.10 codes per 1000 patient-days with 80% power and an α of .05, a study would require >500 000 patient-days. Because critical deterioration is approximately eightfold more common than CHCA code events, detecting a 50% reduction from 1.50 to 0.75 per 1000 patient-days would require far fewer patient-days (∼70 000). Therefore, it can be used to more efficiently evaluate rapid-cycle RRS optimization steps in a single hospital or to compare the performance of multiple hospitals over a period of months rather than years.
When faced with the challenge of measuring reductions in rare events, an alternative strategy is analyzing the number of days between events. This strategy is recommended by the Institute for Healthcare Improvement17 for the measurement of rare events, including cardiac and respiratory arrest. The days between events are convenient to measure and easy to plot on g-type statistical control charts.18 Yet they have an important limitation. Comparing the number of days between events ignores the denominator: the person-time at risk for the outcome (such as non-ICU patient-days). The developer of g-charts acknowledges this limitation and advises that it is “fairly safe to ignore [variation in the denominator] if it does not vary from its average by more than around 10%.”18 The census of children’s hospitals varies widely (>10%) by season19 and among hospitals,20 making the days between metrics inadequate for evaluating within-hospital change or across-hospital comparison.
At our hospital, the combined MET/CBT activation rate was high, at 27.8 calls per 1000 hospital admissions. Previously reported pediatric MET call rates range from 2.8 to 44.0 calls per 1000 admissions across 6 studies, with a median of 9.6 calls per 1000 admissions.5,6,8–12 In adult hospitals, high MET utilization “dose” (>25 calls per 1000 admissions) is associated with improved patient outcomes.21 Further optimization of RRSs requires finding the optimal MET utilization dose beyond which there is marginal improvement in critical deterioration rates.
There were a few limitations to this project. First, we developed the critical deterioration metric by using data from a single center. Multicenter validation is needed. Second, the positive predictive value of the critical deterioration metric was lower (16.7%; 95% CI: 10.3–24.8) than the CHCA metric (41.7%; 95% CI: 15.2–72.3). Although this difference may be a limitation, owing to the low number of patients with CPA and ARC, the positive predictive value of the CHCA measure has a 95% CI that is so wide that its precision is questionable. Third, the expected 30-minute MET response time is longer than the 5- to 15-minute expected response times in other studies.5,6,10 This difference in response time may have resulted in selection bias, identifying a slightly different population than that served by RRSs at other hospitals. Fourth, this metric depends in part on the quality of care provided in the ICU. Variations in ICU care within or between centers may change the critical deterioration rate just as they may change the in-hospital mortality rate or other metrics, without directly reflecting differences in RRS effectiveness. Fifth, because we demonstrated the association between the critical deterioration metric and in-hospital mortality but did not demonstrate that changes to the critical deterioration rate resulting from an intervention (such as RRS implementation) predict changes in the in-hospital mortality rate, it does not yet fulfill formal criteria for a surrogate endpoint.22,23 Future studies should seek to establish that relationship.
We developed the critical deterioration metric, a rate defined as the number of patients transferred to the ICU and requiring life-sustaining interventions within 12 hours per 1000 non-ICU patient-days. We showed that it is a pragmatic, valid proximate outcome for in-hospital mortality with great potential for complementing existing patient safety metrics to evaluate RRS performance.
- Accepted November 25, 2011.
- Address correspondence to Christopher P. Bonafide, MD, MSCE, The Children’s Hospital of Philadelphia, 34th St and Civic Center Blvd, Suite 12NW80, Philadelphia, PA 19104. E-mail:
Dr Bonafide had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis; Drs Bonafide, Priestley, Nadkarni, Keren, and Ms Roberts contributed to study concept and design, and to analysis and interpretation of data; Dr Bonafide, Ms Roberts, Ms Tibbetts, and Ms Huang contributed to the acquisition of data; Dr Bonafide and Ms Roberts contributed to drafting of the article; all authors contributed to critical revision of the article for important intellectual content; Dr Bonafide performed the statistical analysis; Drs Bonafide and Keren obtained funding for the article; and Drs Nadkarni and Keren perfomed study supervision.
FINANCIAL DISCLOSURE: Drs Bonafide and Keren receive funding from the Pennsylvania Health Research Formula Fund Award to perform research on the effectiveness of pediatric rapid response systems; the other authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
- ↵Institute for Healthcare Improvement. Overview of the Institute for Healthcare Improvement Five Million Lives Campaign. Available at: www.ihi.org/IHI/Programs/Campaign/Campaign.htm?TabId=1. Accessed December 22, 2008
- Institute for Healthcare Improvement. About IHI. Available at: www.ihi.org/ihi/about. Accessed July 18, 2010
- ↵Institute for Healthcare Improvement. Deploy rapid response teams. Available at: www.ihi.org/IHI/Programs/Campaign/RapidResponseTeams.htm. Accessed July 18, 2010
- Kotsakis A,
- Lobos A-T,
- Parshuram C,
- et al.,
- Ontario Pediatric Critical Care Response Team Collaborative
- Brilli RJ,
- Gibson R,
- Luria JW,
- et al
- Tibballs J,
- Kinney S,
- Duke T,
- Oakley E,
- Hennessy M
- Hanson CC,
- Randolph GD,
- Erickson JA,
- et al
- ↵Child Health Corporation of America. Available at: www.chca.com. Accessed August 1, 2011
- Evans R,
- Elwyn G,
- Edwards A
- ↵Benneyan J. Measuring rare events and time-between measures. 2011. Available at: www.ihi.org/knowledge/Pages/Presentations/MeasuringRareEventsandTimeBetweenMeasures.aspx. Accessed August 11, 2011
- ↵Fieldston ES, Ragavan M, Jayaraman B, Allebach K, Pati S, Metlay JP. Scheduled admissions and high occupancy at a children's hospital. J Hosp Med. 2011;6(2):81–87
- Fieldston ES,
- Hall M,
- Sills MR,
- et al
- Copyright © 2012 by the American Academy of Pediatrics