August 2016, VOLUME138 /ISSUE 2

Quality Measures to Assess Care Transitions for Hospitalized Children

  1. JoAnna K. Leyenaar, MD, MPHa,
  2. Arti D. Desai, MD, MSPHb,c,
  3. Q. Burkhart, MSd,
  4. Layla Parast, PhDd,
  5. Carol P. Roth, RN, MPHd,
  6. Julie McGalliard, BAc,
  7. Jordan Marmet, MDe,
  8. Tamara D. Simon, MD, MSPHb,c,
  9. Carolyn Allshousef,
  10. Maria T. Britto, MD, MPHg,
  11. Courtney A. Gidengil, MD, MPHh,i,j,
  12. Marc N. Elliott, PhDd,
  13. Elizabeth A. McGlynn, PhDk, and
  14. Rita Mangione-Smith, MD, MPHb,c
  1. aDepartment of Pediatrics, Tufts Medical Center, Boston, Massachusetts;
  2. bDepartment of Pediatrics, University of Washington, Seattle, Washington;
  3. cSeattle Children’s Research Institute, Seattle, Washington;
  4. dRAND Corporation, Santa Monica, California;
  5. eDepartment of Pediatrics, University of Minnesota, Minneapolis, Minnesota;
  6. fFamily Voices of Minnesota, Stillwater, Minnesota;
  7. gCincinnati Children’s Hospital Medical Center, Cincinnati, Ohio;
  8. hRAND Corporation, Boston, Massachusetts;
  9. iDivision of Infectious Diseases, Boston Children’s Hospital, Boston, Massachusetts;
  10. jHarvard Medical School, Boston, Massachusetts; and
  11. kKaiser Permanente Center for Effectiveness and Safety Research, Pasadena, California
  1. Dr Leyenaar designed the study, assisted with interpretation of results, and drafted the initial manuscript; Drs Desai and Simon, Ms Allshouse, and Drs Britto, Gidengil, McGlynn, and Mangione-Smith conceptualized and designed the study, assisted with interpretation of results, and reviewed and revised the manuscript; Mr Burkhart and Drs Parast and Elliott carried out the statistical analyses, assisted with interpretation of results, and reviewed and revised the manuscript; Ms Roth and Ms McGalliard designed the data collection instruments, coordinated and supervised data collection, and critically reviewed the manuscript; Dr Marmet supervised data collection in the community hospitals and critically reviewed the manuscript; and all authors approved the final manuscript as submitted.


BACKGROUND: Transitions between sites of care are inherent to all hospitalizations, yet we lack pediatric-specific transitions-of-care quality measures. We describe the development and validation of new transitions-of-care quality measures obtained from medical record data.

METHODS: After an evidence review, a multistakeholder panel prioritized quality measures by using the RAND/University of California, Los Angeles modified Delphi method. Three measures were endorsed, operationalized, and field-tested at 3 children’s hospitals and 2 community hospitals: quality of hospital-to-home transition record content, timeliness of discharge communication between inpatient and outpatient providers, and ICU-to-floor transition note quality. Summary scores were calculated on a scale from 0 to 100; higher scores indicated better quality. We examined between-hospital variation in scores, associations of hospital-to-home transition quality scores with readmission and emergency department return visit rates, and associations of ICU-to-floor transition quality scores with ICU readmission and length of stay.

RESULTS: A total of 927 charts from 5 hospitals were reviewed. Mean quality scores were 65.5 (SD 18.1) for the hospital-to-home transition record measure, 33.3 (SD 47.1) for the discharge communication measure, and 64.9 (SD 47.1) for the ICU-to-floor transition measure. The mean adjusted hospital-to-home transition summary score was 61.2 (SD 17.1), with significant variation in scores between hospitals (P < .001). Hospital-to-home transition quality scores were not associated with readmissions or emergency department return visits. ICU-to-floor transition note quality scores were not associated with ICU readmissions or hospital length of stay.

CONCLUSIONS: These quality measures were feasible to implement in diverse settings and varied across hospitals. The development of these measures is an important step toward standardized evaluation of the quality of pediatric transitional care.

  • Abbreviations:
    American Academy of Pediatrics
    The Center of Excellence on Quality of Care Measures for Children With Complex Needs
    emergency department
    length of stay
    observed difficulty of delivery
    primary care provider
    Pediatric Medical Complexity Algorithm
    University of California, Los Angeles
  • What’s Known on This Subject:

    Transitions between sites of care, both within the hospital and from the hospital to home, are at-risk times for hospitalized children. However, we lack standardized methods to evaluate these transitions of care in pediatric populations.

    What This Study Adds:

    Three new pediatric quality measures were developed and field tested at children’s hospitals and community hospitals, where scores varied significantly across sites. These quality measures may be used to standardize evaluations of pediatric transitions of care for hospitalized children.

    Hospital care in the inpatient setting accounts for >40% of all pediatric health care expenditures in the United States, representing ∼2 million pediatric hospitalizations annually.1 Inherent to each of these hospitalizations are transitions between sites of care, which may include transitions within the hospital, such as between the ICU and inpatient unit, and from the hospital to home or another health care facility at the time of discharge. Several studies have documented risks associated with these transitions, including medication errors and communication failures, all of which can adversely affect patients and their families.27 As a result, the Centers for Medicare & Medicaid Services and the Agency for Healthcare Research and Quality have selected transitions between sites of care as a priority area for the development of pediatric quality measures.

    The Children’s Health Insurance Program Reauthorization Act of 2009 Pediatric Quality Measures Program provided funding to 7 Centers of Excellence to develop and validate pediatric-specific quality measures.8 The Center of Excellence on Quality of Care Measures for Children With Complex Needs (COE4CCN) was tasked with development and validation of quality measures for pediatric transitions of care, applicable to all children regardless of medical complexity. The objective of this article is to describe the development and validation of these new pediatric transitions of care quality measures, obtained from medical record data.


    Development and validation of these measures involved review of the pediatric and adult transitions of care literature, with a focus on studies examining relationships between transitional care processes and improved health and health care outcomes5; development of draft quality measures based on this evidence review, with evaluation of these measures by a multistakeholder Delphi panel; operationalization of measures into detailed measure specifications that could be applied to medical record data; medical record abstractions at 5 hospitals; calculation and interpretation of measure scores; and validation of medical record measures via outcomes available in hospital administrative data. These steps are detailed below. All study procedures were approved by the participating institutions’ institutional review boards.

    Measure Development

    Detailed methods regarding our evidence review process and multistakeholder Delphi panel are provided in “Methods 1” of the Supplemental Information. In brief, after targeted literature review, COE4CCN research staff drafted a set of 5 potential medical record–based quality measures to assess pediatric transitions of care (Table 1). These measures were evaluated by a panel of 9 experts nominated by professional bodies that conduct activities related to pediatric transitions of care, with participants representing emergency medicine, rehabilitation medicine, hospital medicine, general pediatrics, case management, complex care, state Medicaid agencies, and the family’s perspective. By using the RAND/University of California, Los Angeles (UCLA) modified Delphi method, panelists independently scored each potential measure on a 9-point Likert scale, rating their face validity and feasibility.9 After the initial independent scoring, panelists received a synopsis of responses, including the score distribution and their own score for each measure. Panelists met as a group to discuss controversial measures, after which they independently rescored the face validity and feasibility of each measure.9 Four of the 5 measures were endorsed by the panel (Table 1).

    TABLE 1

    Draft Transitions of Care Quality Measures Evaluated by the Multistakeholder Delphi Panel

    Measure Operationalization and Field Testing

    These measures were transformed into a detailed data abstraction tool to ensure efficient and reliable data collection, and the feasibility of data collection was assessed with a random sample of medical records from the 3 children’s hospitals. At this stage, 1 of the 4 measures endorsed by the panel, documentation of a transition needs assessment, could not be reliably operationalized from medical records because of the very broad scope of the measure content (Table 1). The specifications for determining measure eligibility and scoring are detailed in the Supplemental Information; the data abstraction tool can be accessed online.10

    Eligible patients were 2 months to 18 years of age and hospitalized during January to December 2013 at the children’s hospitals and January 2012 to December 2013 at the community hospitals. The 2-year time period was used for the community hospitals because of the lower pediatric patient volumes at these sites. The children’s hospitals were in different geographic regions of the country; the 2 community hospitals were located in the same state but were operationally independent. Both community hospitals had pediatric inpatient units with care provided by pediatric hospitalists but did not have PICUs. Thus the ICU-to-floor transition measure was not assessed at these sites. Patients who did not experience either an ICU-to-floor transition or a hospital-to-home transition, and patients who died in the hospital, were excluded. Patients were randomly selected with a target of 200 cases per children’s hospital and 150 cases per community hospital to ensure adequate precision in measure assessment within each hospital. For patients with >1 admission during the study period, only the first hospitalization was included. At the children’s hospitals, we used a stratified random sampling approach to include 25% who had an ICU stay to allow testing of the ICU-to-floor transition quality measure. Only patients who had been discharged to home were eligible for the hospital-to-home transition record quality measure. Eligibility for the timely discharge communication between providers quality measure required patients to have been discharged to home and have a primary care provider (PCP) who was not the hospital provider. Patients who experienced a transfer from the ICU to the inpatient floor were eligible for the ICU-to-floor transition note quality measure; for patients with >1 transfer, only the first transfer note was examined.

    Variables manually extracted from the electronic medical records at all hospitals included content for scoring of the transition measures (Table 1; Supplemental Information), date of birth, dates and times of hospital admission and discharge, admission and transfer locations of care (ICU or floor), and discharge destination (home, transfer to another health care facility, left against medical advice, deceased). These data were merged with children’s hospital administrative data extracted from the Pediatric Health Information System, a comparative database used by 45 children’s hospitals nationally to document clinical and resource utilization,11 and from administrative records at community hospitals. These data included patients’ demographic characteristics, length of stay (LOS) in the hospital, 7- and 30-day all-cause readmission, and 7- and 30-day return visits to the emergency department (ED). At the children’s hospitals patient medical complexity was also assessed with the previously validated Pediatric Medical Complexity Algorithm (PMCA) applied to Pediatric Health Information System data.12 PMCA uses International Classification of Diseases, Ninth Revision, Clinical Modification codes to classify children into 1 of 3 disease groups: complex chronic disease, noncomplex chronic disease, and no chronic disease. These data were not available from the community hospitals’ administrative records.

    After a 2-day training session, registered nurses from the research staff implemented the data abstraction tool at the 5 participating hospitals. Two nurses at each hospital each abstracted half of that hospital’s medical record sample, with each chart abstraction taking ∼15 minutes. At 2 of the children’s hospitals, a randomly selected subsample of each nurse abstractor’s medical records was reabstracted by the other nurse to assess interrater reliability. Prevalence and rater bias–adjusted κ statistics were calculated to examine reliability in assessing patient eligibility for each measure and measure scoring.13

    Analytic Approach

    We examined differences in demographic and hospitalization characteristics by hospital type by using 2-sample t tests and Fisher’s exact tests, given that the populations of children cared for at freestanding children’s hospitals may differ from those receiving care at community hospitals.

    Detailed measure specifications were used to calculate quality measure scores (Supplemental Information). For the individual-level binary measures (eg, timely discharge communication between providers), scores were 0 if absent (poor quality) and 100 if present (good quality). Binary subcomponents for multiple-component measures (eg, presence of a discharge medication list) were also scored in this manner and then summarized to produce a mean composite score for the measure on a 0 to 100 scale (eg, overall score for the hospital-to-home transition record quality measure). Measure subcomponents that were not applicable to a particular patient, such as inclusion of immunizations given during the hospitalization in the transition record for patients who received none, were not scored or included in the composite scores for such patients. Hospital-level scores, summarizing both binary and multicomponent measures, ranged from 0 to 100, with higher scores indicating better quality.

    Because it is more challenging to achieve high scores on some measure components than others, and because not all patients were eligible for all measure components, we adjusted the hospital-to-home transition record quality measure scores and the hospital-to-home transition measure summary scores to account for the observed level of difficulty associated with achieving a high score on a given subcomponent. For patients who were eligible for only a subset of the components, this observed difficulty of delivery (ODD) adjustment accounted for the difference between the average overall pass rates for the patient’s subset of measures and the full set, adjusting upward if they were not eligible for an easy-to-pass item and vice versa.14,15 We performed this adjustment by subtracting each measure component for a given patient from its mean and averaging the “centered” measures applicable to a given patient; this patient-level average deviation from the mean was then added to the grand mean across all measure components to obtain a score on the original 0 to 100 scale.

    We used 1-way analysis of variance to test the statistical significance of hospital-level variation in the hospital-to-home transition summary measure and the ICU-to-floor transition quality measure against the null hypothesis that all hospitals have the same mean scores.

    Validation Measures

    To evaluate the validity of these quality measures, we assessed the associations of the summary score and the 2 hospital-to-home transition quality measures with having ≥1 readmissions to the hospital within 7 and 30 days of discharge and with having ≥1 ED return visits within 7 and 30 days of discharge. For the ICU-to-floor transition note quality measure, we assessed associations between measure scores and having ≥1 ICU readmissions, and LOS, in days, truncated at the 99th percentile. Adjusted associations were calculated via logistic or linear regression according to the outcome, adjusted for sex, race or ethnicity, insurance type, and hospital. Medical complexity was also included in the validation analysis for the ICU-to-floor transition note quality measure; this variable was not included in the validation analyses for the hospital-to-home transition quality measures because it was not available from the community hospitals’ administrative data. However, recognizing that medical complexity is a potential confounder of the relationship between these quality measure scores and the validation outcomes, we conducted a sensitivity analysis in which we added PMCA classification to the models by using eligible observations from the children’s hospitals (n = 621, or 67% of the total field test sample).


    A total of 927 charts were reviewed: 624 at 3 children’s hospitals and 303 at 2 community hospitals. Children receiving care at the children’s hospitals and community hospitals had similar age and sex distributions (Table 2). A greater proportion of children receiving care at the community hospitals were white and privately insured compared with those from the children’s hospitals. More than half of the patients in the children’s hospital sample had complex chronic medical conditions. Mean LOS was greater at the children’s hospitals than community hospitals, with higher rates of both 7-day and 30-day readmissions relative to the community hospitals.

    TABLE 2

    Demographic and Hospitalization Characteristics of Participants at Children’s Hospitals and Community Hospitals

    Field testing of the medical record abstraction tool demonstrated almost-perfect interrater reliability on 2 levels: determination of eligibility for the quality measures (κ = 0.83–0.94) and scores on the quality measures (κ = 0.89–1.0; κ for the ICU-to-floor transition note quality measure could not be calculated because of the small sample size).13,16 A total of 924 patients from our sample were discharged to home and eligible for the hospital-to-home transition record quality measure. For this measure, scores for each component ranged from a low mean score of 3.4 (SD 18.0) for inclusion of pending test results to a high mean score of 96.3 (SD 18.8) for inclusion of discharge medications (Table 3). The mean ODD-adjusted composite score for the hospital-to-home transition record quality measure was 65.5 (SD 18.1). The mean score for the timely discharge communication between providers quality measure was 33.3 (SD 47.1), with 4 hospitals having mean scores <30 and 1 having a mean score of 98.0. The ODD-adjusted mean hospital-to-home transition summary score (combined score for hospital-to-home transition record quality measure and timely discharge communication between providers quality measures) was 61.2 (SD 17.1). A total of 126 patients in our sample experienced ICU-to-floor transitions; the mean score for the ICU-to-floor transition note quality measure was 64.9 (SD 47.1). Among the 119 patients who were eligible for all 3 quality measures, only 3 patients (2.5%) had a score of 100 on all 3 measures.

    TABLE 3

    Transition Quality Measure Scores at Children’s Hospitals and Community Hospitals

    The mean ODD-adjusted hospital-to-home transition quality summary scores, combining the 2 hospital-to-home transition measures, varied significantly across the 5 hospitals in our sample (Table 4). For the ICU-to-floor transition note quality measure, scores at 1 hospital differed significantly from scores at the other 2 children’s hospitals. We also observed variation in scores on these quality measures within hospitals. For example, Hospital B had the highest mean hospital-to-home transition summary score and the lowest mean ICU-to-floor transition note quality score.

    TABLE 4

    Variation in Hospital-to-Home Transition Summary Score Means and ICU-to-Floor Transition Note Quality Score Means Across Hospitals

    Results of the validation analyses are shown in Table 5. There were no significant associations of any of the hospital-to-home transition quality measures with hospital readmissions or return ED visits. Results were unchanged in the sensitivity analyses that added medical complexity to the models. Similarly, there were no significant associations of ICU-to-floor transition note quality measure scores with ICU readmissions or LOS.

    TABLE 5

    Validation Analysis for Transition Quality Measure Scores


    Transitions in sites of care are inherent to all hospitalizations, and the risks associated with poor transitions are well documented among adult populations.35,17,18 The development of these new pediatric-specific transition of care quality measures, designed for evaluation with medical record data available at both children’s and community hospitals, is an important step toward a standardized assessment of care transition quality for hospitalized children.

    Two of the quality measures endorsed by our multistakeholder panel and operationalized in this research assess hospital-to-home transitions, and the third evaluates the quality of transitions between the PICU and floor. Several of the measure subcomponents included in the hospital-to-home transition record quality measure align with the priorities for pediatric discharge summary content identified by hospitalists and PCPs in a previous national survey, including discharge diagnoses, discharge medications, dates of admission and discharge, immunizations given during hospitalization, follow-up appointment, and pending laboratory and test results. Consistent with results published by Coghlin et al,7 we observed low scores on transition record subcomponents related to immunizations given during the hospitalization, pending test results, and a hospital contact number for postdischarge problems. We also observed particularly low scores on the quality measure evaluating discharge communication between providers. Although it is possible that this measure underestimated the true frequency of communication because of incomplete documentation, our findings align with several previous studies in both adult and pediatric populations illustrating suboptimal rates of communication between inpatient and outpatient health care providers at the time of hospital discharge.2,4,7,19,20

    Several studies among adults have shown improved patient outcomes and reduced health care utilization when individualized hospital-to-home transition records and improved provider communication were included in bundled hospital-to-home transition interventions.2124 In our validation analyses, we observed no associations between hospital-to-home transition quality measure scores and hospital readmissions or return ED visit rates, perhaps because these outcomes are uncommon in pediatric populations or because our sample size was insufficient to detect associations between these quality measures and rare outcomes. Subsequent research examining associations between these measures and other established quality indicators such as postdischarge physical functioning, patients’ and families’ experiences of care, survey-based pediatric transitions of care measures, and frequency of medical errors and preventable adverse events may provide valuable data to inform transitional care priorities for pediatric populations.2529

    Although the importance of high-quality hospital-to-home transitions is supported by a growing body of research and national health care policy, the quality of transitions between hospital units has received less attention. We observed substantial variation across the children’s hospitals in ICU-to-floor transition note quality measure scores, with scores ranging from a low of 48.6 to a high of 71.5. Comprehensive transition documentation for transfers between clinical units may be challenged by differing clinical priorities and expectations, yet poor-quality handoffs can result in transfer delays, negative patient and family experiences, and increased risk of adverse events.28,30 Transitions between the ICU and general inpatient floor may put children at particular risk given the complexity of problems necessitating ICU care, although we did not observe associations between scores on this quality measure and ICU readmissions or hospital LOS. Building on the growing body of literature illustrating associations between standardized handoffs and decreased risk of medical errors, more studies are needed to understand how transitional care interventions affect pediatric outcomes for transfers between clinical units including the ED, ICU, operating room, and general pediatric floor.29,3133

    This quality measure development process had some limitations. First, pediatric evidence to guide measure development was sparse; the majority of studies illustrating associations between transition care processes and health care outcomes were conducted in adults.5 Therefore, these measures should be reviewed and revised as pediatric evidence continues to emerge. Second, although medical record–based measures are advantageous given the comprehensiveness of clinical data available relative to administrative data, it is possible that these measures may underestimate transition care quality if documentation is incomplete. Conversely, failure of documentation may itself indicate poor quality. Third, although our field test was conducted across 5 hospitals, including both children’s hospitals and community hospitals, performance of these measures may not be generalizable to all settings where children receive hospital-based care.


    These newly developed quality measures were feasible to implement in both children’s and community hospitals and demonstrated variation in care between settings. The development of these measures is an important first step toward standardized evaluations and improvement of transition care quality for hospitalized children.


    We acknowledge the following members of our multistakeholder Delphi panel for their insight and constructive feedback: Julie Beckett (Family Voices); Gregory Conners, MD, MPH, MBA (American Academy of Pediatrics [AAP] Committee on Pediatric Emergency Medicine); David Cooperberg, MD (AAP Section on Hospital Medicine); John Gordon, MD (Children’s Hospital Association); Mary Keen, MD (AAP Home Care Committee); Nancy Skinner, RN-BC, CCM (Case Management Society of America); and Christopher Stille, MD, MPH (Academic Pediatric Association).


      • Accepted May 18, 2016.
    • Address correspondence to JoAnna K. Leyenaar, MD, MPH, Division of Pediatric Hospital Medicine, Department of Pediatrics, Tufts University School of Medicine, 800 Washington St, Boston, MA 02111. E-mail: jleyenaar{at}
    • FINANCIAL DISCLOSURE: Mr Burkhart, Dr Parast, Ms Roth, and Drs Gidengil and Elliott are employed by RAND; the other authors have indicated they have no financial relationships relevant to this article to disclose.

    • FUNDING: Performed under funding from a cooperative agreement with the Agency for Healthcare Research and Quality and Centers for Medicare & Medicaid Services, grant U18HS020506, part of the Children’s Health Insurance Program Reauthorization Act Pediatric Quality Measures Program. Dr Leyenaar was supported by grant K08HS024133 from the Agency for Healthcare Research and Quality.

    • POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

    • COMPANION PAPER: A companion to this article can be found online at