Validation of New Quality Measures for Transitions Between Sites of Care
BACKGROUND AND OBJECTIVE: Assessing and improving the quality of transitions to home from the emergency department (ED) or hospital is critical for patient safety. Our objective was to validate 8 newly developed caregiver-reported measures of transition quality.
METHODS: This prospective observational study included 1086 caregiver survey respondents whose children had an ED visit (n = 523) or hospitalization (n = 563) at Seattle Children’s Hospital in 2014. Caregivers were contacted to complete 2 surveys. The first survey included the newly developed transition quality measures and multiple validation measures including modified versions of Child Hospital Consumer Assessments of Healthcare Providers and Systems (HCAHPS) composites, assessing communication and discharge planning. The second survey (administered 30 days later) included questions about follow-up appointments and ED return visits and readmissions. Using multivariate regression, we examined associations between the newly developed transition quality measures and each validation measure.
RESULTS: All transition quality measures were significantly associated with ≥1 validation measures. The hospital-to-home transition measure assessing whether discharge instructions were easy to understand, were useful, and contained necessary follow-up information had the largest association with the Child HCAHPS nurse–parent and doctor–parent communication composites (β = 55.6; 95% confidence interval, 43 to 68.3; and β = 48.3; 95% confidence interval, 36.3 to 60.3, respectively, scaled to reflect change associated with a 0 to 100 change in the transition measure score).
CONCLUSIONS: Newly developed quality measures for pediatric ED- and hospital-to-home transitions were significantly and positively associated with previously validated measures of caregiver experience. These new measures may be useful for assessing and improving on the quality of ED- and hospital-to-home transitions.
- aOR —
- adjusted odds ratio
- CI —
- confidence interval
- COE4CCN —
- Center of Excellence on Quality of Care Measures for Children With Complex Needs
- ED —
- emergency department
- HCAHPS —
- Hospital Consumer Assessments of Healthcare Providers and Systems
- PMCA —
- Pediatric Medical Complexity Algorithm
- QI —
- quality improvement
What’s Known on This Subject:
Transitions to home from the emergency department or hospital may pose safety risks for pediatric patients and their families. There is a dearth of rigorously developed and validated quality measures to assess these care transitions.
What This Study Adds:
We demonstrate the validity of 8 newly developed caregiver-reported emergency department-to-home and hospital-to-home transition quality measures. These new measures may be useful for assessing and tracking the quality of pediatric transitions between sites of care.
Transitions to home from the emergency department (ED) or hospital may pose safety risks for pediatric patients and their families. Among adults, adverse events after ED- or hospital-to-home transitions are often related to insufficient discharge preparation and communication failures between inpatient and outpatient providers.1–4 Within pediatrics, a study of primary care providers demonstrated a need to improve the quality of hospital discharge summaries,5 and a study of caregivers revealed that 20% experienced postdischarge problems, half of which were medication related.6 Pediatric research also has demonstrated that adequate caregiver preparation before discharge is associated with reduced readmission rates.7 Despite this demonstrated need to improve the quality of transitional care, few valid quality measures focus on transitions.
With funding from the Agency for Healthcare Research and Quality and the Centers for Medicare and Medicaid Services, the Center of Excellence on Quality of Care Measures for Children With Complex Needs (COE4CCN) developed measures to assess the quality of ED- and hospital-to-home transitions.8,9 The COE4CCN developed transition measures that were best assessed by caregiver report (because some aspects of care are best measured by those who experience it) and other measures that were best assessed through medical records data. In this article, we report on the validation of the caregiver-reported measures.
This study sought to assess the convergent and predictive validity of 8 newly developed caregiver-reported transition care quality measures: 2 focused on ED-to-home transitions and 6 focused on hospital-to-home transitions.
Development and Operationalization of Transition Quality Measures
Details about the development of the ED- and hospital-to-home transition quality measures, survey instruments, and scoring are described elsewhere.8,10 Briefly, we identified topics for scientific literature review based on a conceptual framework, drafted quality measures based on available evidence, and convened a multistakeholder Delphi panel to prioritize draft measures. The Delphi panel endorsed 2 caregiver-reported ED-to-home transition measures and 6 caregiver-reported hospital-to-home transition measures (Table 1). Two separate surveys were then developed in parallel to assess the endorsed measures.8 Survey items specified the eligible population for each measure (denominator) and whether the indicated process of care was received among those eligible (numerator). Cognitive interviews were conducted in English and Spanish to improve the understandability and consistency of the interpretation of survey items in these languages. Transition quality measures were scored either on a 0 to 100 scale such that higher scores indicated better care, or as a percentage of respondents who reported that they received the indicated care.
Survey Design, Sample, and Administration
The 2 surveys implemented for this study included the transition quality measure survey items, items to assess the validation measures, and items to collect child and caregiver characteristics. The ED-to-home transition survey included 29 items, and the hospital-to-home transition survey included 41 items (see Supplemental Materials).
Eligible survey respondents were defined as English- or Spanish-speaking caregivers (≥18 years old) of patients aged 2 months to 17 years who had an ED visit or hospitalization for any reason at Seattle Children’s Hospital between June 30, 2014 and September 1, 2014. All eligible caregivers of patients with a hospitalization during the study period were invited to complete the hospital-to-home transition survey. An equal number of caregivers of patients who had an ED visit during the study time frame were randomly sampled and invited to complete the ED-to-home transition survey. Eligible caregivers were randomly assigned so that 50% received a mailed survey and 50% were called to complete a telephone interview. All caregivers were contacted within 42 days of their child’s discharge from the ED or inpatient settings and had 42 days to complete the survey. All survey respondents were also invited to complete a second survey 30 days after their response to the initial survey, which included additional validation measures (see online Supplemental Materials).
Validation Measures: Initial Survey
The initial survey included 5 validation measures to assess the convergent validity of the quality measures. Two were communication composites (Nurse–Parent and Doctor–Parent Communication) modified from the Child Hospital Consumer Assessment of Healthcare Providers and Systems (Child HCAHPS) Survey, which included questions about listening carefully to the caregiver (ED survey only), communicating in a way that was easy for the caregiver to understand, and treating the caregiver with courtesy and respect.11 Responses were scored on a 0 to 100 scale for each question within the composite (Never = 0, Sometimes = 0, Usually = 0, Always = 100, consistent with the top-box scoring method used in Child HCAHPS), where higher scores reflected better communication. All caregivers who answered ≥1 of the questions within the composite received a composite score calculated as the mean of the nonmissing responses12,13 (missingness by question ranged from 0% to 1.5%). These 2 communication composites were used as validation measures for both ED- and hospital-to-home transition measures.
A third validation measure (inpatient survey only) was a modified version of the Child HCAHPS Preparing You and Your Child to Leave the Hospital discharge planning composite, which included items about assessing caregiver readiness for discharge, explaining postdischarge care and expectations, and symptoms to watch for after discharge. This composite was scored similarly to the communication composites.
The remaining 2 validation measures (inpatient survey only) assessed parent perceptions of discharge and were associated with decreased readmissions in a previous study.7 The measures were caregiver perceptions of their child’s readiness for discharge and caregiver understanding of how to manage their child’s care after discharge, and they were scored dichotomously (consistent with previous studies using these measures7) as: strongly agree = 1, all other responses = 0.
Validation Measures: Second Survey
The second survey included 3 additional validation measures to assess predictive validity: whether all needed follow-up visits were scheduled or completed, whether the child had a hospital readmission since discharge, and whether the child had a return ED visit since discharge.
Using linear and logistic regression models, we assessed the convergent validity of the newly developed transition quality measures by examining the association between the transition quality measures and the modified Child HCAHPS and parent perceptions of discharge measures. We also used logistic regression models to assess predictive validity by examining the association between the quality measures and successful scheduling or completion of needed follow-up appointments, ED return visits, or hospital readmissions. Only a priori hypothesized associations between the transition and validation measures were tested (Table 2). Each quality measure was evaluated separately, and associations were summarized via adjusted regression coefficients and odds ratios.
We adjusted for child age, sex, medical complexity, caregiver education, and preferred language for medical communication. These characteristics were selected because they were associated with both the transition quality and validation measures. Medical complexity was assessed with the Pediatric Medical Complexity Algorithm (PMCA), which uses International Classification of Diseases, Ninth Revision, Clinical Modification codes to classify children into 3 levels: complex chronic disease, noncomplex chronic disease, and no chronic disease.14 This study was approved by the institutional review boards at RAND Corporation and Seattle Children’s Hospital.
Of the 1419 eligible caregivers invited, 523 completed the initial ED-to-home transition survey (37% response rate). Of the 1413 eligible caregivers invited, 563 completed the initial hospital-to-home transition survey (40% response rate). Characteristics of the survey respondents and their children are shown in Table 3. Among respondents to the initial surveys, 67% and 70% also responded to the second survey for ED-to-home and hospital-to-home transitions, respectively.
ED-to-Home Transition Measures: Convergent Validity
Higher scores on the 2 ED-to-home transition measures were significantly associated with better Nurse–Parent Communication composite scores (Table 4). Adjusted regression β-coefficients in Table 4 were scaled to reflect the change in the validation metric associated with a 0 to 100 change in the transition measure score. For example, for the ED-to-home transition measure assessing whether the caregiver understood how to use medical equipment at home and whom to call with questions, a score of 100 versus a score of 0 on this measure was associated with a 33.5-point higher Nurse–Parent Communication composite score.
ED-to-Home Transition Measures: Predictive Validity
Associations between ED-to-home transition measures and ED return visits and hospital admissions were not examined because of the very low occurrence of these events in the eligible study sample (2 for 1 measure, 5 for the second measure).
Hospital-to-Home Transition Measures: Convergent Validity
Higher scores on each of the hospital-to-home transition measures were significantly associated with ≥1 of the modified Child HCAHPS validation measures (Table 4). Adjusted odds ratios (aORs) in Table 5 were scaled to reflect the change in the validation metric associated with a 0 to 100 change in the transition measure score. Higher hospital-to-home transition quality measure scores were also significantly and positively associated with the parent perceptions of discharge survey items (Table 5).
Hospital-to-Home Transition Measures: Predictive Validity
The hospital-to-home transition measure that assessed whether the caregiver received assistance with scheduling follow-up appointments was significantly associated with a higher odds of successful scheduling or completion of all needed follow-up appointments (aOR = 3.2; 95% confidence interval [CI], 1.1 to 8.8). Scores on the hospital-to-home transition measures were not associated with caregiver-reported ED return visits or readmissions (results not shown; overall, 14.3% [N = 56] of respondents to the hospital-to-home transition survey had an ED return visit and 12.7% [N = 50] had a hospital readmission).
In this study examining the convergent and predictive validity of new caregiver-reported quality measures for pediatric ED- and hospital-to-home transitions, we found that all measures were significantly associated with ≥1 of the validation measures assessed, including modified versions of previously validated communication and discharge planning composite scores, parent perceptions of discharge survey items, and scheduling or completion of needed follow-up visits. These results are consistent with our a priori hypotheses that higher-quality transitions between sites of care would be associated with higher scores on the caregiver-reported validation measures and a higher likelihood of scheduling or completing needed follow-up visits. No associations were found between the new hospital-to-home transition measures and ED return visits or readmissions.
The development and validation of these new transition quality measures provide an important resource for providers to identify processes of care where quality improvement (QI) efforts can be focused. Although previously developed quality measures, such as Child HCAHPS, provide meaningful assessments of caregiver experience and readiness to leave the hospital, they do not identify specific processes of care that could be improved. The convergent validity of our newly developed measures with these previously developed measures supports their scientific soundness as quality measures and indicates that the processes of care assessed by these measures can be targeted for QI efforts.
In terms of predictive validity, we found that caregivers who received assistance with scheduling follow-up appointments were more likely to adhere to the follow-up care plan. This finding also illustrates the utility of these measures regarding specific actions a hospital can take to improve the quality of care transitions. This result is consistent with findings in previous work that have shown significant associations between providing adult patients with appropriate assistance and information at discharge and higher adherence to follow-up appointments.15–18
Unlike previous adult studies,15,19–22 we did not find a significant association between the new measures and health care utilization. Such utilization outcomes are uncommon in pediatrics and may not be the best metrics for transition quality assessment, particularly in this population.23–25 To better understand transition quality we need to consider aspects of transitional care that can be targeted for QI intervention. These 8 newly validated quality measures may help fill this gap. Although establishing the validity of these measures is important, testing is needed to understand their responsiveness to QI interventions. Responsiveness to QI efforts would strengthen the utility of these measures and potentially allow them to be used to measure performance improvement over time.
Our study has some limitations. First, the majority of the validation measures came from the initial surveys, and therefore associations cannot be interpreted as causal. However, 3 validation measures were obtained from a second survey, 1 month after the initial survey response. It is important to note that the initial survey informed the caregiver that he or she would receive a follow-up survey that would ask about completing follow-up visits. Thus, attendance at follow-up appointments may be higher in this study than in the general population. Second, we were unable to examine associations between ED-to-home transition measures and ED return visits or hospital readmissions because of the small number of patients having 1 of these events. Future studies should include adequate sample sizes to examine these key validation measures. In addition, because ED return visits and hospital readmissions were assessed via caregiver report, there was potential for recall bias. Third, although our response rate of 37% to 40% for the initial survey is comparable to that of similar surveys, such as Adult and Child HCAHPS (which range from 17% to 42%26,27), the results may be subject to nonresponse bias. Lastly, our sample of caregivers may not be a representative sample because this study was conducted in a single academic institution. Future validation work across geographically and structurally diverse hospitals should be performed. Despite this latter limitation, given our results, it is reasonable to use these measures to guide QI efforts related to transitional care for children seen in the ED or hospital settings.
These results demonstrate the validity of 8 new caregiver-reported quality measures for pediatric ED- and hospital-to-home transitions. These measures can be used to examine whether specific evidence-based transition care processes were completed by ED or hospital providers from the perspective of caregivers.
- Accepted February 22, 2017.
- Address correspondence to Layla Parast, PhD, RAND Corporation, 1776 Main St, Santa Monica, CA 90401. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
POTENTIAL CONFLICTS OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
FUNDING: Done under funding from a cooperative agreement with the Agency for Healthcare Research and Quality and Centers for Medicare and Medicaid Services, grant U18HS020506, part of the Children’s Health Insurance Program Reauthorization Act Pediatric Quality Measures Program.
- Forster AJ,
- Rose NG,
- van Walraven C,
- Stiell I
- Heath J,
- Dancel R,
- Stephens JR
- Berry JG,
- Ziniel SI,
- Freeman L, et al
- Desai AD,
- Burkhart Q,
- Parast L, et al
- Leyenaar JK,
- Desai AD,
- Burkhart Q, et al
- Seattle Children's Research Institute.
- Agency for Healthcare Quality and Research
- Paddison C,
- Elliott M,
- Parker R, et al
- Simon TD,
- Cawthon ML,
- Stanford S, et al
- Zorc JJ,
- Scarfone RJ,
- Li Y, et al
- Bardach NS,
- Vittinghoff E,
- Asteria-Peñaloza R, et al
- Press MJ,
- Scanlon DP,
- Ryan AM, et al
- Toomey SL,
- Zaslavsky AM,
- Elliott MN, et al
- Copyright © 2017 by the American Academy of Pediatrics