OBJECTIVE: Pediatric early warning scores (PEWS) are being advocated for use in the emergency department (ED). The goal of this study was to compare the validity of different PEWS in a pediatric ED.
METHODS: Ten different PEWS were evaluated in a large prospective cohort. We included children aged <16 years who had presented to the ED of a university hospital in The Netherlands (2009−2012). The validity of the PEWS for predicting ICU admission or hospitalization was expressed by the area under the receiver operating characteristic (ROC) curves.
RESULTS: These PEWS were validated in 17 943 children. Two percent of these children were admitted to the ICU, and 16% were hospitalized. The areas under the ROC curves for predicting ICU admission, ranging from 0.60 (95% confidence interval [CI]: 0.57−0.62) to 0.82 (95% CI: 0.79–0.85), were moderate to good. The area under the ROC curves for predicting hospitalization was poor to moderate (range: 0.56 [95% CI: 0.55–0.58] to 0.68 [95% CI: 0.66–0.69]). The sensitivity and specificity derived from the ROC curves ranged widely for both ICU admission (sensitivity: 61.3%–94.4%; specificity: 25.2%–86.7%) and hospital admission (sensitivity: 36.4%–85.7%; specificity: 27.1%–90.5%). None of the PEWS had a high sensitivity as well as a high specificity.
CONCLUSIONS: PEWS can be used to detect children presenting to the ED who are in need of an ICU admission. Scoring systems, wherein the parameters are summed to a numeric value, were better able to identify patients at risk than triggering systems, which need 1 positive parameter.
- ATS —
- Australasian Triage Scale
- CI —
- confidence interval
- ED —
- emergency department
- ESI —
- Emergency Severity Index
- IQR —
- interquartile range
- MTS —
- Manchester Triage System
- PedCTAS —
- pediatric Canadian Triage and Acuity Scale
- PEWS —
- pediatric early warning scores
- ROC —
- receiver operating characteristic
What’s Known on This Subject:
Pediatric early warning scores (PEWS) for hospital inpatients have been developed to identify patients at risk for deterioration. Beyond triage, similar systems that identify ill patients and predict requirements for a higher level of care are needed in the emergency department.
What This Study Adds:
The validity of the different PEWS in pediatric emergency care patients has never been evaluated. This study showed that PEWS are capable of detecting children in need of ICU admission.
Pediatric early warning scores (PEWS) are physiology-based scoring systems developed to identify patients admitted to inpatient pediatric wards at risk for clinical deterioration.1 A recent publication showed that early warning scores are needed to quickly identify critically ill patients in the emergency departments (EDs) so that treatment can be started without delay.2 Moreover, the use of the same system in the ED and inpatient wards allows continuity for patient assessment.
According to an adult study performed in the United Kingdom, early warning scores are used in the majority of EDs, although the evidence for this claim is lacking.2 To date, there are few data on the use of PEWS in children presenting to the ED.3,4 Bradman and Maconochie3 validated only 1 of the several PEWS that are currently in use. Egdell et al4 conducted a pilot study to validate a designed for initial assessment at the ED and showed that the system was able to identify children requiring ICU admission.
The goal of the current study was to compare the performance of different PEWS to predict ICU admission or hospitalization in a large population of children visiting the pediatric ED.
Different versions of PEWS were evaluated in a large prospective cohort of children presenting to the ED. The different PEWS were based on patients’ age and vital sign values (heart rate, respiratory rate, oxygen saturation, blood pressure, temperature, and level of consciousness) prospectively collected during the triage assessment.
The current study used data collected for an ongoing study on the validity of the Manchester Triage System (MTS) in pediatric patients.5,6 The medical ethics committee of Erasmus MC approved the study, and the requirement for informed consent was waived.
Setting and Selection of Participants
Data collection included all children aged <16 years who presented to the ED of the Erasmus MC−Sophia Children’s Hospital, Rotterdam, Netherlands, between August 2009 and June 2012. The Erasmus MC−Sophia Children’s Hospital is a large inner-city university hospital with a pediatric ED that is open 24 hours a day. The ED receives ∼8000 children annually from a catchment area with a multisocioeconomic and multiethnic population of 2 million inhabitants.
Pediatric Early Warning Scores
A PubMed search was performed in June 2012 using the terms “pediatric early warning,” “paediatric early warning,” “track and trigger,” “trigger criteria,” “calling criteria,” “medical emergency team,” “pediatric alert criteria,” or “paediatric alert criteria.” Studies were limited to children aged 0 to 18 years and a publication date within the past 10 years. Subsequently, the titles, abstract, and full text articles were screened, and the reference lists of systematic reviews and studies on the use of PEWS in the ED were scanned to complete the search. The PEWS were included if the scores were newly developed for children presenting to the ED or admitted to an inhospital pediatric ward or if the original scores were adjusted.
The PubMed search retrieved a total of 75 articles. After exclusion of studies not addressing PEWS (n = 45), original research on PEWS (n = 8), or children (n = 6), 16 studies remained. Eight studies described newly developed or derived PEWS and the remaining 8 studies validated these PEWS. Four studies were included after screening the reference lists, resulting in a total of 12 PEWS, of which 11 were developed for inpatient use7–17 and 1 for use in the ED.4
The PEWS can be differentiated into scoring systems and triggering systems.1 A scoring system contains different parameters (eg, heart rates or respiratory rates). If these parameters show an increased deviation from normal values, the given scores are greater. The scores for all the different parameters are cumulated to 1 numeric value, which, depending on the cutoff level, determines a patient’s risk for clinical deterioration. In a triggering system, the patient is considered at risk if 1 of the parameters is positive.
Six PEWS were considered as scoring systems4,7–11 and 6 as triggering systems.12–17 Most PEWS were developed for inhospital patients and therefore not all parameters were available at triage assessment. Parameters that contain diagnostics, therapeutic interventions, or suspected diagnoses were removed from the scoring and triggering systems. Only therapeutic interventions such as oxygen therapy and bolus fluids remained in the model because these parameters are surrogate markers of low saturation and severe dehydration, which are features scored by triage nurses. The PEWS of Hunt et al17 and Sharek et al16 are not useful for triage assessment in the ED because continuous monitoring of vital signs is needed to assess acute change in vital signs. Therefore, 10 PEWS remained for analysis.
ED nurses specialized in both pediatric and emergency care collected standardized data on the different parameters of the PEWS during triage assessment and recorded this information on structured electronic or paper (2006−2009) ED forms. Heart rates, oxygen saturation, and blood pressure were measured by using electronic devices. Respiratory rates were measured by counting respiratory movements for 30 seconds. The measurement of vital signs was left to the discretion of the nurse. The database was checked for outliers (values >3 times the interquartile range above the 75th percentile and <3 times the interquartile range below the 25th percentile18). Patient characteristics and data on follow-up were extracted from the electronic hospital system and merged in SPSS version 20.0 (IBM SPSS Statistics, IBM Corporation, Armonk, NY) for analysis.
To impute missing vital signs values, we used a multiple imputation model, including age, gender, vital signs values, hospitalization, ICU admission, MTS category, and presenting problem. This method means that missing data are replaced by a value that is drawn from an estimate of the distribution of the variable to create a complete database.19 This process was executed 10 times to generate 10 complete databases. Statistical analyses on each database were performed and pooled for a final result.
A numeric score was calculated for the different scoring systems and a binary score for the triggering systems. The validity of the PEWS was expressed by the areas under the receiver operating characteristic (ROC) curves, sensitivity, specificity, and positive likelihood ratios and negative likelihood ratios for ICU admission and admission to the hospital. To calculate sensitivity, specificity, and likelihood ratios, the numeric scores of the scoring systems had to be dichotomized at the most optimal cutoff level of the ROC curves.
Sensitivity, specificity, positive likelihood ratios, negative likelihood ratios, and the 95% confidence intervals (CIs) were calculated with the VassarStats Web site (http://vassarstats.net/clin1.html). Statistical analyses were performed by using SPSS and R package version 2.13.1 (R Foundation for Statistical Computing, Vienna, Austria) using the Design, Hmisc (AregImpute) function.
In total, 18 073 children presented to the ED during the study period. Data were not available for 130 children. Therefore, 17 943 children remained for analysis, of whom 16% (n = 2828) were admitted to the hospital and 2% (n = 373) were admitted to an ICU or died in the ED. Patients’ characteristics are shown in Table 2.
Ninety-six percent of patients (n = 17 289) had at least 1 vital sign measured. Heart rate was measured in 9062 (51%) children; respiratory rates in 6671 (37%); blood pressure in 3632 (20%); oxygen saturation in 4901 (27%); temperature in 10 050 (56%); and level of consciousness in 16 319 (91%). The absence of vital signs was more frequent in patients allocated to lower MTS urgency categories and in patients presenting with traumatic problems than in those presenting with medical problems.
Performances of PEWS in the Total Population
The ROC curves of the PEWS are shown in Fig 1. The discriminative ability to predict ICU admission and admission to the hospital was higher when scoring systems were used than when triggering systems were used (Table 3). Moreover, PEWS were better suited to predict ICU admission than admission to the hospital, because the areas under the ROC curves decreased significantly when admission to the hospital was used as the outcome measure.
For all PEWS, the optimal cutoff level to calculate sensitivity and specificity for both ICU admission and admission to hospital was set at 1, except for the PEWS of Duncan et al10 and Parshuram et al,11 for which the cutoff levels were set at 3 for ICU admission and 2 for admission to the hospital (Table 3). The sensitivity and specificity at different cutoff levels of the scoring systems are shown in Appendix 2.
The sensitivity and specificity of the PEWS at the optimal cutoff levels varied widely. When ICU admission was used, the sensitivity of the different PEWS ranged from 61.3% to 94.4% and the specificity ranged from 25.2% to 86.7%. These findings resulted in a positive likelihood ratio between 1.3 and 4.6 and a negative likelihood ratio between 0.22 and 0.45.
When hospitalization was used, the sensitivity ranged from 36.4% to 85.7% and the specificity ranged from 27.1% to 90.5%. None of the PEWS showed both a high sensitivity and a high specificity. Sensitivity, specificity, positive likelihood ratios, and negative likelihood ratios of the individual PEWS are shown in Table 3.
Twelve different PEWS were described in the literature, of which 10 were potentially suited for use in the ED. The discriminative ability of the PEWS (area under the ROC curve) were moderate to good for ICU admission (range: 0.60−0.82) and poor to moderate for admission to the hospital (range: 0.56−0.68). Moreover, scoring systems with parameters leading to a numeric value were better able to identify patients at risk than triggering systems, which need 1 positive parameter. The c-statistics of the different scoring systems, however, were not statistically different. The choice of best PEWS in the ED should depend on other factors such as ease of use.
The scoring systems of Egdell et al4 and Duncan et al10 contain more parameters than the scores of Monaghan,7 Akre et al,8 Skaletzky et al,9 and Parshuram et al11 and thus are more time-consuming at initial assessment. Moreover, the PEWS of Duncan et al and Parshuram et al included blood pressure, which is difficult to obtain in a standardized manner in a busy ED. For this reason, the applicability of scoring systems should be evaluated for the individual setting before implementation.
However, scoring systems with more parameters provide a wider range of sum scores and can therefore differentiate patients into >2 risk groups. This categorization can be important when PEWS are not only used to identify patients in need of ICU admission but also patients in need of admission to a pediatric ward. The PEWS of Duncan et al10 and its bedside version from Parshuram et al11 are the only scores with different optimal cutoff levels for hospitalization and ICU admission, and they are therefore best suited to allocate patients to >2 risk groups.
Thresholds for abnormal vital signs influence the validity of the PEWS, because PEWS that only differ according to vital sign thresholds showed different c-statistics. This finding suggests that the PEWS could be optimized by choosing the optimal cutoff levels for vital sign values. At present, most PEWS use cutoff levels based on the Advanced Pediatric Life Support program.20,21 However, recent publications suggest that reference ranges for vital signs should be updated with new thresholds.22–24
At present, conventional triage systems such as the MTS,25,26 the Emergency Severity Index (ESI),27 the pediatric Canadian Triage and Acuity Scale (PedCTAS),28 and the Australasian triage Scale (ATS)29 are used in the ED to allocate the patient’s acuity. In the MTS, PedCTAS, and ATS, trained triage nurses had to recognize patient’s signs and symptoms to allocate acuity.25,26,28,29 In the ESI, the urgency categories are based on the need of life-saving interventions and resource use.27 In all triage systems, vital signs are included to allocate urgency. However, the use of these vital signs differed from the use in PEWS scoring systems, because they are dichotomized into normal and abnormal for the ATS, PedCTAS , and ESI, and in the MTS, they were included as discriminators such as “shock,” “abnormal pulse,” and “increased work of breathing”; thus, values for abnormality in children were not provided. In South Africa, an early warning score was included to allocate patients to the lowest urgency levels. This triage strategy is inexpensive and can be executed by an inexperienced staff.30
Although PEWS can identify patients at risk in the ED for ICU admission and, to a lesser extent, identify patients at risk for hospitalization, we do not advise using warning scores as triage tools to prioritize patients.31 At present, there is no evidence that PEWS are better than conventional triage systems. To prove that PEWS as triage tools are better than conventional triage systems or that PEWS have added value to conventional triage systems, a direct comparison study should be conducted in which patient outcomes and costs are included.
Currently, PEWS in the ED should be an adjunct of conventional triage. They can be used as a tool to indicate ICU admission or as a monitoring tool to identify patient deterioration, due to their ability to continue a patient’s assessment when admitted to the hospital.2,32
The main limitation of the current study is that the different PEWS were not implemented in the ED itself and therefore were not evaluated in practice. Conversely, because the PEWS have not been implemented, clinicians did not know the PEWS scores when examining the patients. The decision to admit patients to the ICU or pediatric ward was not influenced by the outcome of the PEWS and therefore could not bias our results.
Second, ICU admission and admission to the hospital were chosen as a proxy for acuity because a golden standard for acuity does not exist. Worldwide, hospitalization and ICU admission have been used extensively as a proxy for severity of illness in the ED.33–37 Also, it is a limitation that vital signs were not measured in all patients. We resolved this problem by using a multiple imputation model that can be used when the outcome measure (ICU admission) and predictor (presence of vital signs) on X and Y are correlated.38
Lastly, the study population comprises children from 1 hospital, which could influence the generalizability of the results. However, the population included a varied case-mix of ∼18 000 children, selected from a multicultural, inner-city ED population, and the result are therefore likely to be generalizable to other pediatric ED populations.
PEWS are capable of identifying children in need of ICU admission. Scoring systems, with parameters leading to a numeric value, were better able to identify patients at risk than triggering systems, which need 1 positive parameter.
- Accepted July 11, 2013.
- Address correspondence to Henriëtte A. Moll, MD, PhD, Department of Pediatrics, Room Sp 1540, Erasmus MC–Sophia Children's Hospital, University Medical Centre Rotterdam, PO Box 2060, 3000 CB Rotterdam, Netherlands. E-mail:
Dr Seiger conceptualized and designed the study, carried out the initial analyses, and drafted the initial manuscript; Dr Maconochie conceptualized and designed the study, and reviewed and revised the manuscript; Dr Oostenbrink designed the data collection instruments, supervised data collection, and reviewed and revised the manuscript; Dr Moll conceptualized and designed the study, drafted the article, and analyzed the data; all authors approved the final manuscript as submitted. Dr Moll is guarantor.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by an unrestricted grant from Europe Container Terminals B.V.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Griffiths JR,
- Kidney EM
- Egdell P,
- Finlay L,
- Pedley DK
- ↵van Veen M, Steyerberg EW, Ruige M, et al. Manchester triage system in paediatric emergency care: prospective observational study. BMJ. 2008;337:a1501
- van Veen M,
- Steyerberg EW,
- Van’t Klooster M,
- et al
- Skaletzky SM,
- Raszynski A,
- Totapally BR
- Tibballs J,
- Kinney S,
- Duke T,
- Oakley E,
- Hennessy M
- Edwards ED,
- Powell CV,
- Mason BW,
- Oliver A
- Brilli RJ,
- Gibson R,
- Luria JW,
- et al
- ↵Hoaglin D, Mosteller F, Turkey J, eds. Understanding Robust and Exploratory Data Analysis. Wiley Classics Library Edition. New York, NY: Wiley; 2000
- ↵Mackway-Jones K. Advanced Paediatric Life Support: The Practical Approach. 4th ed. London, Malden, Mass: BMJ, Blackwell; 2005
- ↵Biarent D. European Paediatric Life Support Course. 2nd ed. London, UK: Resuscitation Council (UK); 2006
- Nijman RG,
- Thompson M,
- van Veen M,
- Perera R,
- Moll HA,
- Oostenbrink R
- Thompson M,
- Harnden A,
- Perera R,
- et al
- Mackway-Jones K,
- Marsden J,
- Windle J
- ↵Mackway-Jones K, Marsden J, Windle J. Emergency Triage. 2nd ed. London, UK: BMJ Publishing Group; 2006
- Warren DW,
- Jarvis A,
- LeBlanc L,
- Gravel J,
- CTAS National Working Group,
- Canadian Association of Emergency Physicians,
- National Emergency Nurses Affiliation,
- Association des Médecins d’Urgence du Québec,
- Canadian Paediatric Society,
- Society of Rural Physicians of Canada
- ↵Guidelines on the Implementation of the Australasian Triage Scale in Emergency Departments. Australasian College for Emergency Medicine. Available at: www.acem.org.au/media/policies_and_guidelines/G24_Implementation__ATS.pdf. Accessed July 30, 2013
- Gottschalk SB,
- Wood D,
- DeVries S,
- Wallis LA,
- Bruijns S,
- Cape Triage Group,
- Proposal from the Cape Triage Group
- Burch VC,
- Tarr G,
- Morroni C
- Rees JE,
- Mann C
- Martins HM,
- Cuña LM,
- Freitas P
- Copyright © 2013 by the American Academy of Pediatrics