OBJECTIVES: A well-recognized gap exists in assessing and improving the quality of care for medically complex patients. Our objective was to examine evidence for primary care based on the patient-centered medical home model and to identify valid and meaningful quality measures for use in complex pediatric patients.
METHODS: We conducted literature searches on Medline and the National Quality Measures Clearinghouse for existing measures, as well as evidence to inform the development of new quality measures. We used a 3-step process to select relevant sources from published literature: (1) the titles were screened by 2 independent reviewers; (2) the abstracts were reviewed for quality-of-care contents or constructs; and (3) full-text articles were obtained and reviewed for measure specification. All materials were reviewed for the Oxford Centre For Evidence-Based Medicine level of evidence and for relevance to primary care of complex pediatric patients. A national expert panel was convened to evaluate and rate the measures by using the Rand/University of California Los Angeles Appropriateness Method.
RESULTS: We presented 74 quality measures to the expert panel for review and discussion. The panel rated and accepted 35 measures as valid and feasible for assessing primary care quality in complex pediatric patients. The final set of quality measures was grouped in the following domains: primary care–general (14), patient/family-centered care (8), chronic care (2), coordination of care (9), and transition of care (2).
CONCLUSIONS: By using the patient-centered medical home framework of accessible, continuous, family-centered, coordinated, and culturally effective care, a national expert panel selected 35 primary care quality measures for complex pediatric patients.
- AAP —
- American Academy of Pediatrics
- IOM —
- Institute of Medicine
- PCMH —
- patient-centered medical home
- UCLA —
- University of California Los Angeles
What’s Known on This Subject:
There are known gaps in quality measures for children. More clinical effectiveness research is needed. The patient-centered medical home may serve as a model to guide the development of quality measures, particularly for children with complex medical conditions.
What This Study Adds:
This study combined systematic literature review and the Rand/University of California Los Angeles appropriateness method to develop quality measures for children with complex medical conditions. These are valid and feasible quality measures based on the patient-centered medical home framework that may be used to assess care.
A well-recognized gap exists in assessing and improving the quality of care for US children.1–5 Research findings across various diseases, patient populations, and clinical settings suggest that the quality chasm is pervasive1–3,5; moreover, concerns regarding the ineffectiveness of traditional models of care are also mounting, particularly for patients with complex medical problems and/or multiple chronic conditions.6–10 The Agency for Healthcare Research and Quality has defined complex patients as persons with 2 or more active chronic conditions.11 The complexity arises in that 1 condition may influence the treatment and/or outcomes of another condition (eg, drug interactions). Furthermore, the persistent burden of a particular condition may lead to the neglect of another condition and/or the patient as a whole person.8,12 Medically complex patients can also affect families, financially and otherwise.13–15
Children with multiple conditions and/or special health care needs are often managed by several providers, including generalists, specialists, and other health professionals. The chronic and complex nature of these conditions, coupled with fragmented interaction and communication across multiple providers, can lead to inconsistent and poorly managed care.
The “patient-centered medical home” (PCMH), defined by the American Academy of Pediatrics (AAP) and other primary care professional organizations as accessible, continuous, comprehensive, family-centered, coordinated, compassionate, culturally effective care, has been adopted as a promising care model for all patients; it is a particularly excellent care model for children with complex conditions and special health care needs.16 The PCMH is a primary care setting that facilitates partnerships between patients (patients’ families) and their physicians. For the purpose of this study, we restrict the PCMH to primary care physician–led medical homes. Specialist- or subspecialist-led medical homes may operate under different requirements or constraints.
To provide high-quality care, attention needs to be focused on well-designed, evidence-based processes of care.6,10,17,18 This is particularly challenging for chronic care because it involves multiple disciplines and requires effective collaboration among providers and patients. Quality measures can be a useful mechanism for the translation and application of evidence-based practices19–22; they provide a standardized evidence-based metric that can be widely implemented in clinical settings. Quality measures can also be used to align payment strategies and motivate improvements in care.10,23
The goal of this study was to assess, through expert consensus, recommended primary care processes for complex pediatric patients by using the PCMH approach as a helpful first step toward establishing a candidate set of quality measures. To accomplish this, we first conducted a systematic review of the literature to identify existing quality measures applicable to the primary care of complex pediatric patients. Then we reviewed established guidelines/recommendations and clinical evidence from the literature to develop new measures to fill existing gaps. Last, we convened a national panel of experts by using the Rand/University of California Los Angeles (UCLA) Appropriateness Method to assess the validity and feasibility of these measures.22
Review of Literature
We conducted a literature review of published evidence (up to the publication date of December 2009) to identify factors linking processes of care to pediatric patient outcomes. First, we searched Medline by using the following search terms, alone and in conjunction with one another: primary care, patient-centered care, family-centered care, medical home, chronic care model, disease management, care coordination, coordination of care, continuity of care, comprehensive care, chronic conditions, complex conditions, developmental conditions, and quality measures/indicators. Citations were retained if they were available in English; relevant for infants and children younger than 18; and were reviews, practice guidelines, meta-analyses, randomized controlled trials, cohort studies, case-control studies, quality-measure reports, or expert consensus, opinion, or “first principles” research. Figure 1 presents a schematic of the article-selection process. Then, we searched the National Quality Measures Clearinghouse (http://www.qualitymeasures.ahrq.gov), including the Healthcare Effectiveness Data and Information Set, for existing quality measures pertaining to the previously mentioned search term constructs.
We reviewed all citations by using the following steps: (1) the titles of the articles were reviewed for relevance; (2) the abstracts of the articles passing step 1 were then reviewed for contents; and (3) articles passing step 2 were obtained and reviewed in full. Articles were excluded if they did not meet the retention criteria described previously. Quality measures that were redundant (ie, with similar constructs and processes) or inappropriate for primary care settings were removed in step 3. We also performed “reference mining” of the full articles by examining the cited references and identifying additional sources. Every article was reviewed by 2 independent reviewers to identify established guidelines/recommendations and quality measures. Disagreements were resolved by discussion until consensus was reached.
Data Extraction and Synthesis
We reviewed all retrieved material for level of evidence and relevance. We identified only a handful of sources with clinical evidence and practice guidelines directly applicable to children with complex medical needs; much of the published guidelines and existing measures were related to single conditions. Thus, we used the PCMH as a framework to guide the identification and inclusion of quality criteria that conceptually will improve the care of children with complex or multiple chronic conditions. We gave priority to studies with the strongest designs based on the Oxford Centre for Evidence-Based Medicine Levels of Evidence.24 At the end of literature review, we identified 36 existing quality measures appropriate for the primary care of complex pediatric patients.
By combining clinical evidence/practice guidelines obtained in the literature review with parent focus group data, we also constructed 38 new quality measures. These 38 measures do not overlap with the existing 36 measures already identified from literature review. Patient and family inputs on care were represented as focus group data aimed at addressing the patient-centeredness component of the PCMH; this was collected through 2 semistructured, hour-long focus groups by using a convenience sample of parents of chronically ill children at Children’s Hospital Los Angeles (n = 7 per group). The first author facilitated both focus group discussions. A total of 74 quality measures, representing 58 unique published sources, were grouped into the following 5 categories: primary care–general, 28 measures from 18 sources; patient/family-centered care, 16 measures from 12 sources; chronic care, 9 measures from 8 sources; coordination of care, 16 measures from 12 sources; and transition of care, 5 measures from 5 sources. Each measure was assigned a content type (process, structure, or outcome), and an anticipated data source (medical record/chart, practice-based survey, or patient survey).
We convened a national expert panel to interpret the evidence detailed by the literature review and to assess the validity and feasibility of the proposed set of measures. Nine expert panelists were selected through official nominations from the AAP, Academic Pediatric Association, and the National Association of Children’s Hospitals and Related Institutions (Table 1). We solicited nominations from these professional organizations because of their strong interest and work in the areas of primary care, chronic care, and PCMH. We used a modified version of the Rand/UCLA Appropriateness Method to assess the opinions of the expert panelists.22 The approach consisted of 2 rounds of anonymous ratings for validity and feasibility and a face-to-face panel meeting between rounds (held on July 23, 2010, in Los Angeles). Quality measures developed by using the Rand/UCLA Appropriateness Method have been shown to be reproducible, and to have content, construct, and predictive validity.25–27
We began by sending each panelist the 74 proposed quality measures and relevant references/data. We asked the panelists to rate the validity and feasibility of each measure on a scale of 1 to 9, where 1 was “definitely not valid” (and similarly for feasibility, “definitely not feasible”) and 9 was “definitely valid” (and similarly for feasibility, “definitely feasible”). A measure was considered valid if (1) there is adequate scientific evidence or professional consensus to support a link between what it measures and improved patient outcome; (2) a provider performing the actions described by the measure would be considered to deliver better care than a provider who did not perform those actions; and (3) the provider or the practice could “control” the process specified by the measure. A measure was considered feasible if it could be assessed by using data sources currently available in most pediatric primary care settings; these include paper medical charts/records, administrative/billing data, laboratory and pharmacy data, or electronic medical record.
The panelists submitted their first round of ratings before the face-to-face meeting, along with their comments and any additional references or sources they chose to provide. We incorporated additional sources suggested by the panelists to include unpublished or not yet published material. We then prepared summaries of the first-round ratings for distribution at the panel meeting. The summary was individualized for each panelist and included the following: (1) the actual rating submitted by the individual; (2) a histogram of anonymous ratings by all panelists; (3) the median rating, as a measure of central tendency; (4) the mean absolute deviation from the median, as a measure of dispersion; (5) a statistical classification of “agreement” or “disagreement” among the panelists; and (6) whether each measure was accepted, rejected, or of indeterminate disposition after the first round of ratings. Agreement among the panelists was defined as the inability to reject the hypothesis that 80% of the hypothetical population of repeated ratings would be within the same 3-point region (ie, 1–3, 4–6, or 7–9) as the median, and disagreement otherwise.22 A measure was accepted if the median validity was in the 7 to 9 range, the median feasibility was in the 4 to 9 range, and there was agreement for the measure (for example, “Children should be screened by the primary care team for developmental delays [at the 9-, 18-, and either 24- or 30-month visits] by using validated instruments” was accepted with a median validity of 8, median feasibility of 8, and in “agreement” with all 9 validity ratings in the 7–9 region). A measure was rejected if the median validity or feasibility was in the 1 to 3 range and there was agreement. Measures that were neither accepted nor rejected after the first round of ratings were classified as indeterminate (for example, “The primary care team [including physicians, nurses, and so forth] meets at least once a week to discuss the range of treatment and care choices for their patients,” which had a median validity of 6, did not meet the criteria either for acceptance or rejection, and was classified as intermediate).
During the panel meeting, measures of indeterminate disposition after the first round were discussed first. Then, each panelist was asked to bring to attention any measure he or she would like the group to discuss regardless of its disposition. Some measures were reworded or clarified by the panel, and 2 new measures were suggested based on the additional sources provided by panelists. After the panel meeting discussions, each panelist was asked to re-rate all measures for validity and feasibility; the median, mean absolute deviation, and agreement/disagreement were subsequently recalculated based on these second-round ratings. A measure was accepted to the final set if its median validity was in the 7 to 9 range, its median feasibility was in the 4 to 9 range, and there was agreement after the second round of ratings. Any measure not accepted after the second round of ratings, ie, those that were rejected or of indeterminate status, was dropped from further consideration.
The literature review identified 165 articles for a full-text review, of which 37 articles were removed based on the inclusion/exclusion criteria. We presented 74 quality measures to the expert panel for review and discussion. The expert panel rated and accepted a final set of 35 measures, representing 32 unique sources, as valid and feasible for assessing primary care quality for complex pediatric patients. We grouped the measures into the following 5 domains: primary care–general (n = 14), patient/family-centered care (n = 8), chronic care (n = 2), coordination of care (n = 9), and transition of care (n = 2). In the final set, 11% of the quality measures were structure measures; the remaining 89% were process measures. Nearly half of the measures (49%) would require medical record or chart abstraction, 23% would require an assessment of the clinic operation/resources by using practice-based survey, and the remaining 29% would be based on patient survey. A summary of the final set of quality measures is presented in Table 2. The final list of 130 references obtained from literature review (128) and expert panel (2) is presented in the Appendix.
In their report Envisioning the National Health Care Quality Report,28 the Institute of Medicine (IOM) committee emphasized 4 components of health care quality: safety, effectiveness, patient-centeredness, and timeliness. Among our final set of 35 quality measures, 4 focused on the safety aspect of primary care, tracking and following up on laboratory findings and diagnoses as well as ensuring proper hand-offs between providers. Among the remaining 31 measures, 14 focused on the effectiveness of care, ensuring the delivery of appropriate evidence-based best practices. Another 8 measures focused on the timeliness of care, emphasizing coordination of care (the IOM Future Directions for the National Healthcare Quality and Disparities Reports also identified care coordination as a systemwide priority area along with infrastructure) and the provision of services in a timely manner. The rest addressed patient- or family-centeredness in care, encouraging shared decision-making and partnership. The IOM committee highlighted equity as a crosscutting issue, but we were unable to identify measures specifically designed to assess equity; this remains a gap to be addressed.
A review article published by Beal et al29 reported that there were gaps in quality measures to comprehensively assess health care quality for children. This study is a step toward filling that gap in pediatric health care quality assessment. By using a modified Rand/UCLA Appropriateness Method, we developed a set of 35 quality measures for the assessment of primary care in complex pediatric patients based on the medical home goal of delivering accessible, continuous, comprehensive, family-centered, coordinated, compassionate, culturally effective care. Because we used stringent criteria for panel agreement, coupled with practice guidelines for primary care that relied heavily on recommendations rather than level I evidence, only 35 of the proposed 74 measures were accepted by the panel.
We organized the final set of quality measures into 5 domains of care: primary care–general; patient/family-centered care; chronic care; coordination of care; and transition of care (to adult providers). General primary care is of enormous importance to complex pediatric patients and their families. The primary care team is often responsible for guiding patients/parents and advocating for them. There is also some evidence linking primary care to improved outcomes and reduced costs.30
There is broad consensus on the emerging need to provide patient- and family-centered care; for example, researchers have noted with concern the rarity with which patient and family perspectives are considered valid evidence to inform quality measures or guidelines,31 and the AAP has explicitly urged pediatricians to establish partnerships with families of medically complex children in planning and executing care.32 As shown in Table 2, in the patient/family-centered domain, the expert panel selected several quality measures that empower patients and families with shared decision-making (measures 2 and 4) and inputs into the functioning of the medical home (measures 3 and 5); moreover, the panel also placed emphasis on cultural sensitivity and the provision of language services (measures 7 and 8).
Chronic diseases represent perhaps the largest burden on our heath care system today.33 Unlike care for acute illnesses, which is usually contained in a finite number of visits, medical care for chronic illnesses is a long-term process that requires multifaceted care. Wagner and colleagues34–36 commented that the needs of patients with chronic illnesses are unlikely to be met by an acute care system. In the chronic care domain of Table 2, the expert panel placed importance on defining a management plan that includes regular assessments and interactions between patients and their primary care team (measure 1).
Coordination of care is also an important component of care for clinically complex patients. A recent Commonwealth Fund report suggested a “roadmap” for coordination of care, advocating that it should be proactive, planned, comprehensive, support/rely on team care, and jointly developed (with patient/family) to ensure effective communication and collaboration.37 The expert panel selected several quality measures that place the responsibility of the care coordination on the provider-patient partnership (measures 3–6). For chronically ill children, another facet of care involves the effective transition to adult care providers. The panel selected measures that promote early initiation of care transition and placed the responsibility of identifying future adult providers on the primary care team.
The expert panel focused on several key areas when evaluating quality measures. First, they supported the importance of preventive care for all complex pediatric patients; this includes a comprehensive range of services, from immunizations to timely developmental assessments, as well as dental health screening and obesity prevention. Second, they emphasized that primary care should be delivered by a team of providers, and this is reflected by the consistent use of the term “primary care team.” Last, patient- and family-centeredness is viewed as an integral part of the care delivery process. The panel endorsed making medical records available and accessible to families, building a culturally sensitive environment, and forming a patient-provider partnership of shared decision-making.
The expert panel ratings were also consistent and in support of 4 of the 6 Maternal and Child Health Bureau’s core outcomes for children with special health care needs: the quality measures reflected the importance of family sharing in medical decision-making (core outcome 1), children with special health care needs receiving coordinated care in the medical home setting (core outcome 2), children receiving early screening for special health care needs (core outcome 4), and transition planning to adult health care (core outcome 6). The remaining 2 core outcomes, related to insurance coverage and community-based services, were outside the scope of quality of care.
Our study has several limitations. We defined complex pediatric patients based on the Agency for Healthcare Research and Quality framework of patients with 2 or more chronic conditions; this is not an all-inclusive approach and our results cannot be generalized to all children with medically complex issues. In addition, there may be socioeconomic factors in providing high-quality primary care that we did not consider. Our expert panel nomination process resulted in an underrepresentation of safety-net providers. The issue of comanagement of complex patients between primary care providers and subspecialists was also not addressed. Last, although there was evidence to support the value of quality measures in assessing the care delivered for 1 specific condition (eg, asthma management), the evidence to support using the same approach for complex patients with multiple (potentially interacting) conditions is much weaker.
Our study is an effort to address a noted gap in pediatric quality-of-care assessment. We hope this work will prompt further development in quality measurement for medically complex patients, particularly in the pediatric population. The findings from this study contribute to an ongoing effort to achieve quality-of-care goals nationally.
- Accepted November 9, 2011.
- Address correspondence to Alex Chen, MD, MS, Children’s Hospital Los Angeles, MS #30, 4650 Sunset Blvd, Los Angeles, CA 90027. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Dr Chen was supported by the Agency for Healthcare Research and Quality grant 5K02HS018087 during the study period.
COMPANION PAPER: A companion to this article can be found on page 574, and online at www.pediatrics.org/cgi/doi/10.1542/peds.2011-3776.
- Reed MC, St. Peter RF. Satisfaction and Quality: Patient and Physician Perspectives. Washington, DC: Center for Studying Health System Change; 1997
- Institute of Medicine
- Klerman LV,
- Perloff JD
- Newacheck PW,
- Stein RE,
- Walker DK,
- Gortmaker SL,
- Kuhlthau K,
- Perrin JM
- ↵AHRQ special emphasis notice. Available at: http://grants.nih.gov/grants/guide/notice-files/NOT-HS-08-004.html. Accessed October 6, 2008
- American Academy of Pediatrics Committee on Children with Disabilities
- ↵Hobbs N, Perrin JM, Ireys HT. Chronically Ill Children and Their Families. San Francisco, CA: Jossey-Bass Inc, Publishers; 1985
- Medical Home Initiatives for Children With Special Needs Project Advisory Committee. American Academy of Pediatrics
- Dougherty D,
- Simpson LA
- Schuster MA,
- Asch SM,
- McGlynn EA,
- Kerr EA,
- Hardy AM,
- Gifford DS
- Brook RH
- Christianson JB,
- Knutson DJ,
- Mazze RS
- ↵Center for Evidence Based Medicine. Available at: www.cebm.net/index.aspx?o=1025. Accessed May 18, 2010
- Hurtado MP,
- Swift EK,
- Jorrigan JM
- Beal AC,
- Co JP,
- Dougherty D,
- et al
- Palfrey JS,
- Sofis LA,
- Davidson EJ,
- Liu J,
- Freeman L,
- Ganz ML,
- Pediatric Alliance for Coordinated Care
- Bodenheimer T,
- Chen E,
- Bennett HD
- Antonelli R,
- McAllister J,
- Popp J
- Copyright © 2012 by the American Academy of Pediatrics