BACKGROUND: The Respiratory Distress Assessment Instrument (RDAI) and Respiratory Assessment Change Score (RACS) are frequently used in bronchiolitis clinical trials, but evidence is limited on their measurement properties. We investigated their validity, reliability, and responsiveness.
METHODS: We included data from up to 1765 infants with bronchiolitis enrolled in 2 studies conducted in pediatric emergency departments. We assessed RDAI construct validity by testing hypotheses of associations with physiologic measures (respiratory rate, oxygen saturation) and with constructs related to hospitalization, using correlation coefficients, and multivariable analysis. RDAI/RACS responsiveness was evaluated by using anchors of change based on these constructs; measures of responsiveness included the area under the curve. RDAI test-retest agreement and interrater reliability were evaluated by using limits of agreement and intraclass correlation coefficients.
RESULTS: Baseline RDAI scores were weakly correlated with respiratory rate (r = 0.38, P < .001), and scores increased in lower oxygen saturation categories (P < .001). Higher RDAI scores were associated with hospitalization (odds ratio: 1.36; 95% confidence interval: 1.26–1.47); scores differed between participants who were discharged, admitted, or stayed in the emergency department (P < .001). Our hypotheses were met, but the magnitude of associations was below our predefined thresholds. RDAI test-retest limits of agreement were −3.80 to 3.64 (20% of the range), whereas interrater reliability was good (intraclass correlation coefficient = 0.93). Formulated hypotheses for responsiveness were confirmed, with moderate responsiveness (area under the curve: RDAI, 0.64–0.70; RACS, 0.72).
CONCLUSIONS: RDAI has poor to moderate construct validity, with good discriminative properties but considerable test-retest measurement error. The RDAI and RACS are responsive measures of respiratory distress in bronchiolitis but do not encompass all determinants of disease severity.
What’s Known on This Subject:
The Respiratory Distress Assessment Instrument (RDAI) and the Respiratory Assessment Change Score (RACS) are the most frequently used measurement instruments in bronchiolitis clinical trials. Evidence is scarce regarding their measurement properties and their suitability for use as evaluative instruments in clinical trials.
What This Study Adds:
The RDAI is an incomplete measure of respiratory distress in bronchiolitis, with poor to moderate construct validity. It has adequate discriminative properties but considerable test-retest measurement error. The RDAI and RACS were moderately responsive, but methodologic issues limit the interpretation of this finding.
Acute viral bronchiolitis is the most common lower respiratory tract infection in infants and carries substantial clinical and financial burden.1,2 There is wide practice variation in its management, with heterogeneous evidence for many therapeutic approaches.3–7 Systematic reviews have highlighted various shortcomings in randomized controlled trials (RCTs) in this field.8–10 One of the major issues is the heterogeneous choice of outcome measures. There has been inconsistency in selected measurement instruments, whose measurement properties have often not been adequately studied.8,11
Respiratory status is an important dimension and determinant of severity in bronchiolitis. The Respiratory Distress Assessment Instrument (RDAI) and Respiratory Assessment Change Score (RACS) are often used to measure this domain in bronchiolitis.8 Lowell et al12 first described them in an RCT of epinephrine in wheezing infants. The RDAI includes items on retractions and wheezing, whereas the RACS is a change score based on the RDAI and respiratory rate. Evidence is limited regarding RDAI and RACS measurement properties and their suitability for use as evaluative instruments in clinical trials.13–15 Previous RCTs have reported some data on which reliability and validity can be assessed, whereas the first formal validation study is recent.16 The aim of this study was to assess and compare the measurement properties of RDAI and RACS, that is, validity, reliability, and responsiveness.
We used data from 2 related studies conducted simultaneously in 8 Canadian pediatric emergency departments (EDs) during 3 bronchiolitis seasons (2004–2007): a 2 × 2 factorial RCT (Canadian Epinephrine/Steroid Trial, CanBEST; N = 800)17 and a prospective cohort study (N = 1554 infants, 584 of whom also participated in CanBEST).18 Both studies included infants aged <12 months with acute bronchiolitis (first episode of wheezing) and excluded those with previous asthma, wheezing, or use of bronchodilators. Additional exclusion criteria in CanBEST were as follows: prematurity with corrected age <6 weeks, chronic cardiopulmonary disease or immunodeficiency, recent corticosteroid use or exposure to varicella, very mild or severe distress (pulse rate >200 beats per minute, respiratory rate >80 breaths per minute, or RDAI score <4 or >15), or lethargy.
Participants in CanBEST were randomly assigned to receive oral dexamethasone or placebo and nebulized epinephrine or placebo, both administered in the ED (Fig 1). During the first 90 minutes, only supplemental oxygen or acetaminophen was allowed. Other participants in the cohort study were given standard treatment as decided by the attending physicians. In both studies, written informed consent was obtained from the parents or guardians of the infants, and both were approved by ethics committees at each site and by Health Canada.
Instruments and Outcome Measures
We assessed the RDAI as described by Lowell et al12 and a modification of the RACS as reported by Schuh et al19 (Tables 1 and 2). The following measurements were performed at baseline for both studies and every 30 minutes until admission/discharge or 240 minutes for CanBEST: RDAI, respiratory rate, heart rate, oxygen saturation (Sao2), and activity status (Fig 1). Fever was also assessed at baseline. Research nurses performed all measurements after formal training and using written instructions. Sao2 was measured by using pulse oximeters available locally. In both studies, the attending physician independently determined whether to admit or discharge the infant; RDAI was not used clinically at any site. In CanBEST, physicians and nurses were blinded to treatment interventions, and by protocol any decisions regarding admission, discharge, or continued stay in the ED were to be made only after the study interventions (ie, after 90 minutes).
We used the Consensus-based Standards for the selection of health Measurement Instruments initiative (COSMIN) definitions of measurement domains and properties regarding validity, reliability, and responsiveness.20
Construct Validity of the RDAI
There is no “gold standard” to assess bronchiolitis severity or respiratory distress. We assessed construct validity of the RDAI by formulating hypotheses about the direction and magnitude of the association of RDAI scores with both physiologic measures (respiratory rate, Sao2) and clinical decision-making constructs (decision to admit/discharge and time to admission/discharge).21 We studied both convergent and discriminative validity.
We hypothesized that baseline RDAI scores and respiratory rate would have a strong positive correlation (Pearson’s r ≥ 0.7). We used multiple linear regression analysis to explore possible confounding of this association, by activity status and fever (data from both studies) and age and weight (data from CanBEST). We further hypothesized a negative association between RDAI and Sao2, which we expected to be weaker and nonlinear (Spearman’s r ≤ −0.5), and we compared RDAI scores between 3 categories of Sao2 (<92%, 92%–95%, >95%; data from both studies).
We hypothesized that a higher RDAI score would increase the risk of admission (expected odds ratio [OR] for admission: ≥1.5 for RDAI scores above the median). For this analysis, we used the last RDAI score assessed or registered before the time of admission/discharge (data from CanBEST). Multiple logistic regression analysis was used to evaluate whether that association was confounded by center, treatment group, age, and Sao2. Furthermore, we expected participants who stayed in the ED longer to have intermediate scores compared with those who were admitted or discharged sooner (data from CanBEST after the main trial interventions).
Reliability of the RDAI
For RDAI test-retest reliability, we considered that the group of CanBEST participants who had received both placebo interventions was stable between the 90- and 120-minute measurements. The same research nurse assessed the same child unblinded to the previous assessment. In a convenience sample of participants from each study, 2 nurses performed baseline RDAI measurements independently to obtain interrater assessments.
For both test-retest and interrater conditions, we distinguished measures of measurement error from reliability measures.22 To evaluate measurement error we calculated the standard error of measurement (SEM) and, the smallest detectable change (SDC), and we obtained a Bland-Altman plot and the 95% limits of agreement (LoA; formulas are shown in Supplemental Information). The Bland-Altman plot shows the mean differences between the test and retest scores (expressed in the unit of the scale) along the range of the scale. The LoA show the scores between which 95% of these differences lie. We assessed reliability by calculating the intraclass correlation coefficient (ICC) in a 2-way random-effects model, including patient, observer, and residual variance components (formula shown in Supplemental Information).
Responsiveness of the RDAI/RACS
As with validity, we studied the responsiveness of the RDAI and RACS through testing hypotheses concerning the expected associations of change scores.21 There is no clear criterion for change in bronchiolitis and none of the studies included explicit assessments of change. We based our hypotheses on physiologic and clinical constructs of change using a priori–defined criteria to identify groups of participants who improved versus those who had remained stable or deteriorated, irrespective of interventions. Criteria were based on respiratory rate and Sao2, and we used combined group data from CanBEST (Table 3).
We used different measures of responsiveness assessing statistical change or clinically important change, focusing on comparing the improved group with the stable/deteriorated group.23 These included the following: testing differences in RDAI change scores and/or RACS within and between groups, and calculating standardized/Cohen’s effect size (ES) and the responsiveness ratio (formulas shown in Supplemental Information). We hypothesized that patients who had improved would have larger change scores and ESs than patients who had not improved. We also used the area under the curve (AUC) of the receiver operating characteristic curve of improved versus stable groups (AUC >0.70 considered appropriate).
For all analyses, we excluded participants with nonvalid or missing data, with no imputation. Statistical significance was set at P < .05, and we calculated 95% confidence intervals (CIs) when applicable. We used SPSS version 19 (IBM SPSS Statistics, IBM Corporation, Armonk, NY).
Figure 2 shows data sources and participants included in the analysis of each measurement property. The baseline characteristics of participants and selected outcomes from both studies are presented in Table 4. Participants in CanBEST were older than those in the cohort study, whereas baseline severity was greater in the latter study.
Construct Validity of the RDAI
We found a weak positive correlation between RDAI score and respiratory rate at baseline with data from both studies (Pearson’s r = 0.38; 95% CI: 0.35 to 0.45; P < .001; N = 1765). Correlations for retractions and wheezing subscores were r = 0.41 and r = 0.17, respectively. By using simple linear regression, the coefficient estimate was a 1.55 (95% CI: 1.38 to 1.73) increase in respiratory rate (breaths per minute) per increase in RDAI unit (P < .001). The estimate was comparable when adjusting for fever and activity status (adjusted estimate: 1.52). When restricting the analysis to CanBEST data, the correlation was weaker (Pearson’s r = 0.22; unadjusted linear regression estimate: 0.98; n = 800). The association was not confounded by age, weight, fever, or activity status (adjusted estimate: 0.92).
There was a weak negative correlation between baseline RDAI scores and Sao2 levels (Spearman’s r = −0.24; P < .001; n = 1761). Correlations for retractions and wheezing were r = −0.25 and r = −0.14, respectively. RDAI scores increased in lower Sao2 categories (Fig 3). The median (interquartile range) RDAI scores were 10 (8–12), 8 (6–10), and 7 (5–10) for Sao2 <92%, 92%–95%, and >95%, respectively (Kruskal-Wallis test, P < .001).
We found an association between the decision to admit or discharge and the last RDAI score of CanBEST participants. The preceding RDAI score was higher in admitted patients than in those who were discharged (mean difference: 2.28; 95% CI: 1.75 to 2.81; t test, P < .001; n = 798). A higher RDAI score was associated with higher risk of admission (OR:1.36; 95% CI: 1.26 to 1.47) per increase in RDAI unit; and 2.54 (95% CI: 1.65 to 3.92) when the RDAI was > 8). Adjusted analyses for center, treatment group, age, and Sao2 revealed no relevant changes in these associations.
In addition, we found that RDAI scores measured after CanBEST interventions differed between the groups of participants who were discharged (median [interquartile range]: 5 [2–6]), hospitalized (8 [5–10]), or who stayed in the ED (6 [4–8]) (n = 695; Kruskal-Wallis test, P < .001) (Fig 4). Differences between discharged participants and the 2 latter groups were statistically significant (P = .01 and P < .01, respectively; Bonferroni post hoc test). Patients with a higher RDAI score at 90 minutes had higher risk of ED stay >240 minutes (OR: 1.33; 95% CI: 1.24 to 1.43 per increase in RDAI unit). Overall, although results were in accordance with our validity hypotheses, the magnitude of the associations was mostly below our predefined thresholds.
Reliability of the RDAI
Test-retest assessments were available from 79 CanBEST participants. The mean difference between the 2 repeated assessments was 0.08 (95% CI: −0.35 to 0.5; P = .72). The SEM was 1.34 and the SDC was 3.72 RDAI units. The test-retest 95% LoA were −3.8 to 3.64 RDAI units (95% CI: −4.53 to −3.07, to 2.91 to 4.37). This finding means that if a child is assessed twice, the second score could be between 3.64 points lower and 3.8 points higher than the first score, just because of measurement error. The magnitudes of differences between repeated measurements remained the same over the whole range of mean values as shown in the Bland-Altman plot (Fig 5). The test-retest ICC was 0.80 (95% CI: 0.70 to 0.87).
Interrater assessments were performed in 107 participants. There was no significant difference between 2 repeated assessments (mean difference: −0.06; 95% CI: −0.28 to 0.15; P = .54). The SEM was 0.78, and the interrater LoA were −2.1 to 2.22 RDAI units (95% CI: −2.46 to −1.74, to 1.86 to 2.58]. The ICC was 0.93 (95% CI: 0.9 to 0.94).
Responsiveness of the RDAI and RACS
Measures of responsiveness for RDAI and RACS based on the different constructs of change are presented in Table 5. By using both anchors, the mean RDAI scores decreased in both improved and stable groups (paired t test, P < .001; for all within-group comparisons), with larger mean changes in scores of the improved group (unpaired t test, P < .001; for all between-group comparisons). These results were in accordance with our predefined hypotheses. Between-group differences in mean RDAI change scores ranged from −1.31 (95% CI: −1.85 to −0.77) for the 25% respiratory rate reduction criterion to −2.03 (95% CI: −2.9 to −1.16) for the probability of admission criterion. Standardized ESs for the improved group ranged from 1.43 to 1.71, whereas responsiveness ratios ranged from 1.54 to 1.61, and AUCs from 0.64 to 0.7. The RACS was larger in the improved group (between-group difference: −2.81; 95% CI: −3.92 to −1.7), with a responsiveness ratio of 1.96 and an AUC of 0.72.
This study of measurement properties of RDAI and RACS in acute bronchiolitis identifies strengths and limitations of their use as outcome measures. The RDAI was evaluated in 3 systematic reviews of measurement properties of asthma or wheezing severity scales in children.13–15 Limited data on its reliability and responsiveness were provided in the original description of the scale and in later reports of RCTs.12,19,24 However, none of these were adequately designed measurement studies, and no formal assessment of validity was found. Destino et al16 recently reported the first validation study on RDAI in bronchiolitis, showing poor construct validity, interrater reliability, and responsiveness. Findings on validity were fairly consistent with our results; differences in setting, raters, and methods may explain why results on reliability and responsiveness were distinct.
Our results show that the RDAI has poor to moderate construct validity. The RDAI was developed ad hoc with no elaboration on the underlying conceptual model, item selection, scoring, or weighting. Although in the original report only the RACS was used as an outcome measure, later trials used RDAI scores separately for single or repeated assessments.12 In our conceptual framework, respiratory distress was putatively reflected by RDAI items (ie, reflective model) and contributed to the multidimensional construct of bronchiolitis.21 We found poor convergent validity with respiratory rate, but RDAI scores discriminated well between clinically meaningful Sao2 subgroups. Measurement properties from other respiratory scales or their individual items, which often include respiratory rate or Sao2, are seldom available.13–15 When they are reported, there is substantial heterogeneity in correlations with Sao2, ranging from poor to moderate. Thus, our predefined cutoffs may have been too strict. Most, but not all, studies are consistent with our findings of weaker correlations between Sao2 and auscultatory items when compared with work of breathing.13–15,25
These results reflect the pathophysiology and clinical correlates of respiratory distress in bronchiolitis. It is known that as disease progresses and severity increases, so do the disturbances in ventilation and ventilation-perfusion matching.26,27 Many patients have effective compensatory mechanisms for these disturbances, although others do not. However, clinical signs of respiratory distress may not capture hypoxemia/hypercapnia balance equally. Furthermore, the correlation between Sao2 (reflecting oxygenation), and respiratory rate (also dependent on respiratory drive and ventilation) varies across conditions.25 Therefore, the RDAI likely does not represent all dimensions of respiratory distress in bronchiolitis, and a combination of parameters may be more relevant for the measurement of respiratory distress, as seen in formally developed scales.28,29 However, most other scales were not developed specifically for bronchiolitis, and their measurement properties cannot be transferred between different respiratory conditions without further validation.
We found that the RDAI had reasonable predictive validity based on its association with hospitalization and length of stay in the ED. Our findings are consistent with those of Corneli et al,30 who identified RDAI score, Sao2, and respiratory rate as predictors of hospitalization in bronchiolitis. On the contrary, Destino et al16 found that RDAI sum scores did not discriminate well between admitted and discharged patients, but the item on retractions did. Two large prognostic studies have also identified retractions as predictors of severe disease in ED and hospitalized patients.31,32 Decisions regarding hospitalization and length of stay in the ED are multifactorial. Nonrespiratory severity parameters (eg, feeding), prognostic factors (eg, age), social issues, clinical judgment, available resources, and local practices influence decision-making.30 Furthermore, there are limits to the validity of static measurements of respiratory distress in a highly dynamic condition. From an outcome measure perspective, the RDAI does not encompass all determinants of bronchiolitis severity.
Interrater reliability measured by the ICC was good, both at the group and individual level, as was interrater measurement error. These findings mean that RDAI scores can adequately discriminate participants assessed by different raters at the same time point in both clinical and research settings. Data from previous RCT reports also showed good interrater reliability, but Destino et al16 found a strikingly low ICC. Differences may relate to training, familiarity with the instrument, raters, and population heterogeneity. On the other hand, we found considerable test-retest measurement error at the individual level, because a patient should change at least close to 4 points (approximately one-fifth of the scale) before a change is detectable beyond measurement error. Thus, in clinical practice, changes in individual patients should be interpreted with caution. For the RACS, we must also consider measurement error for respiratory rate.33,34 The SDC is paramount to interpretability parameters such as the minimal important change (MIC), because a large SDC relative to the MIC means that observed change may be caused by measurement error rather than change per se.35 At the individual level, taking repeated measurements and averaging the value would reduce the measurement error with a factor √k (k is the number of measurements). Although reassessment is a key component when evaluating children with respiratory distress, many repeated measurements might not be practical in clinical practice. At the group level, the SDC of a mean change is equal to SDC/√n, which reduces its impact.21 Because the ICC was high, the RDAI is reliable for use in studies. Overall, these results suggest that the RDAI has adequate discriminative properties, but test-retest measurement error should be minimized.
The RDAI was responsive according to our predefined hypotheses based on 2 distinct constructs of change. Previous data on RDAI responsiveness are scarce.13–15 Hardly any intervention can be considered clearly effective in bronchiolitis in the ED setting, and thus none is a reasonable gold standard to assess change. Destino et al16 reported a mild correlation between the change in the RACS and the Children’s Hospital of Wisconsin Respiratory Score, but data on responsiveness of this latter scale are also missing. We anchored our constructs of change on physiologic change and change in clinical status likely to be relevant for decisions regarding patient disposition at the ED. Measures of responsiveness that took into account both improved and stable groups (responsiveness ratio and AUC) were comparable between anchors for the RDAI. The AUC value was close to the frequently used cutoff of acceptability (0.7) for both the RDAI and RACS, with the RACS being slightly more responsive. These data suggest that the RDAI and RACS are moderately responsive, but any comparison with other respiratory scales is limited.
Our study has limitations related to design constraints of both included studies. First, less heterogeneity of RDAI scores in the selected sample of CanBEST participants may explain why we found a weaker correlation with respiratory rate and lower test-retest ICC scores. Further validation is needed when considering children with very mild or severe disease, who were excluded in CanBEST. Second, our results are applicable to infants with a first episode of wheezing and no relevant comorbidities and should be interpreted with caution when defining bronchiolitis differently in other populations.36 Third, concurrent factors that affect decisions of hospitalization were not collected, and the exact timing of this decision was not known. Although, in ideal conditions, managing physicians would be blinded to RDAI/RACS scores, blinding to their individual items is not expected. Finally, defining stability and change can be problematic and time-dependent due to the dynamic nature of bronchiolitis. When assessing responsiveness by using data collected at different time points (mostly between 90 and 240 minutes), we observed significant improvements in RDAI scores in groups that we considered a priori to be stable. This finding is likely a limitation of our anchors and may also reflect the effect of supportive measures and the nebulized “placebo.” These limitations should be considered when calculating the MIC of the RDAI, which will be the focus of future work.
In conclusion, we found the RDAI to be an incomplete measure of respiratory distress in bronchiolitis, with poor to moderate construct validity and adequate interrater reliability. The RDAI had considerable test-retest measurement error, and although both the RDAI and RACS were moderately responsive, methodologic issues may limit the interpretation of this finding. Finally, the RDAI does not encompass all determinants of bronchiolitis severity.
We thank the children and families who participated in CanBEST and the prognostic cohort study, the site research coordinators, the coordinating site staff, the data manager, all of the research nurses and participating site physicians and nurses, and Pediatric Emergency Research Canada members.
- Accepted March 18, 2015.
- Address correspondence to Ricardo M. Fernandes, MD, Clinical Pharmacology Unit, Instituto de Medicina Molecular, University of Lisbon, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal. E-mail:
Dr Fernandes conceptualized and designed the study, carried out all analyses, and drafted the initial manuscript; Dr Plint was the principal investigator for both original studies on which this study is based; she reviewed the study protocol and critically reviewed the manuscript; Drs Terwee and Sampaio critically reviewed the manuscript; Dr Klassen reviewed the study protocol and critically reviewed the manuscript; Dr Offringa conceptualized and designed the study and critically reviewed the manuscript; Dr van der Lee, conceptualized and designed the study, supervised all analyses, and critically reviewed the manuscript; and all authors approved the final manuscript as submitted.
This work was presented at the annual international congress of the European Respiratory Society; September 6–10, 2014; Munich, Germany.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Funding for the data collection for this study was provided by a grant from the Canadian Institutes of Health Research and the Ontario Thoracic Society. Dr Fernandes was supported by the Programme for Advanced Medical Education, sponsored by Fundação Calouste Gulbenkian, Fundação Champalimaud, Ministério da Saúde and Fundação para a Ciência e Tecnologia, Portugal. Dr Plint is supported by a University of Ottawa Research Chair in Pediatric Emergency Medicine.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Zorc JJ,
- Hall CB
- Hartling L, Fernandes RM, Bialy L, et al. Steroids and bronchodilators for acute bronchiolitis in the first two years of life: systematic review and meta-analysis. BMJ. 2011;342:d1714
- ↵Florin TA, Byczkowski T, Ruddy RM, Zorc JJ, Test M, Shah SS. Variation in the management of infants hospitalized for bronchiolitis persists after the 2006 American Academy of Pediatrics bronchiolitis guidelines. J Pediatr. 2014;165(4):786–792
- Lowell DI,
- Lister G,
- Von Koss H,
- McCarthy P
- Destino L,
- Weisgerber MC,
- Soung P,
- et al
- Plint AC,
- Johnson DW,
- Correll R,
- et al
- ↵De Vet H, Terwee C, Mokkink L, Knol D. Measurement in Medicine: A Practical Guide. Cambridge, United Kingdom: Cambridge University Press; 2011
- ↵Javouhey E, Pouyau R, Massenavette B. Pathophysiology of Acute Respiratory Failure in Children with Bronchiolitis and Effect of CPAP. In: Esquinas AM, ed. Noninvasive Ventilation in High-Risk Infections and Mass Casualty Events: Vienna, Austria: Springer; 2014:233–249
- ↵Thorburn K, McNamara P. Bronchiolitis. In: Wheeler DS, Wong HR, Shanley TP, eds. Pediatric Critical Care Medicine. London, United Kingdom: Springer; 2014:75–85
- Simoes EA,
- Roark R,
- Berman S,
- Esler LL,
- Murphy J
- Rusconi F,
- Castagneto M,
- Gagliardi L,
- et al
- Copyright © 2015 by the American Academy of Pediatrics