BACKGROUND AND OBJECTIVES: Defining and measuring health for children with medical complexity (CMC) is poorly understood. We engaged a diverse national sample of stakeholder experts to generate and then synthesize a comprehensive list of health outcomes for CMC.
METHODS: With national snowball sampling of CMC caregiver, advocate, provider, researcher, and policy or health systems experts, we identified 182 invitees for group concept mapping (GCM), a rigorous mixed-methods approach. Respondents (n = 125) first completed Internet-based idea generation by providing unlimited short, free-text responses to the focus prompt, “A healthy life for a child or youth with medical complexity includes: ___.” The resulting 707 statements were reduced to 77 unique ideas. Participants sorted the ideas into clusters based on conceptual similarity and rated items on perceived importance and measurement feasibility. Responses were analyzed and mapped via GCM software.
RESULTS: The cluster map best fitting the data had 10 outcome domains: (1) basic needs, (2) inclusive education, (3) child social integration, (4) current child health-related quality of life, (5) long-term child and family self-sufficiency, (6) family social integration, (7) community system supports, (8) health care system supports, (9) a high-quality patient-centered medical home, and (10) family-centered care. Seventeen outcomes representing 8 of the 10 domains were rated as both important and feasible to measure (“go zone”).
CONCLUSIONS: GCM identified a rich set of CMC outcome domains. Go-zone items provide an opportunity to test and implement measures that align with a broad view of health for CMC and potentially all children.
- CMC —
- children with medical complexity
- CS Global —
- Concept System Global Max
- GCM —
- group concept mapping
What’s Known on This Subject:
Despite significant research on children with medical complexity (CMC) and interest in adopting a population health framework, a conceptualization of their health and potential health outcomes is underdescribed.
What This Study Adds:
Group concept mapping with experts on CMC resulted in a more comprehensive list of 10 outcome domains, leading to a population health framework for CMC.
Children with medical complexity (CMC) generate 40% of child Medicaid expenditures despite being only 3% of the pediatric population.1,2 Although the number of clinical programs for CMC has increased rapidly to meet care coordination needs of CMC and their families, there is no consensus on which CMC health outcomes are feasible to measure and important to pursue.3 To date, the CMC research and clinical fields have suffered from an absence of theoretically grounded conceptualization of shared health outcomes for this diverse population as well as a lack of understanding about which outcomes are measurable in clinical settings. The lack of conceptually driven health measures for CMC has contributed to an ad hoc reliance on hospital-related metrics and a relative lack of attention to nonhospital-related measures and needs for CMC, such as long-term wellness and social and/or developmental measures.3
While previously developing candidate domains of health outcomes for CMC from a systematic literature review (E.S.B., R.C., and B.B.N.; unpublished observations) and in-depth interviews with stakeholder experts on CMC,3 we noted that CMC experts tended to prioritize family functioning and wellbeing as well as child mental health, wellbeing, and social integration more than was reflected in existing literature (E.S.B., R.C., and B.B.N.; unpublished observations).3 Instead, researchers in the literature strongly emphasized traditional measures of health care use, which most experts also acknowledged as important (E.S.B., R.C., and B.B.N.; unpublished observations).3 We decided to engage a much larger, national group of stakeholder experts to attempt to develop a consensus set of core outcome domains that could be used to guide the selection of specific CMC health outcomes that are foreseeably measurable in real-world settings.
Group concept mapping (GCM) is a rigorous, structured, mixed-methods approach designed specifically for such a purpose.4 This participatory research methodology combines qualitative and quantitative data to produce a conceptual model representing how a group views a particular topic.4 GCM has been used by the Robert Wood Johnson Foundation to define its “culture of health.”5 It has also been successfully applied by researchers across disciplines to gather and synthesize collective input in a dynamic and systematic fashion to achieve a structured conceptualization of an idea or issue.4
Our objective was to use GCM to (1) generate a framework depicting how CMC health is conceptualized by a broadly representative and diverse group of experts and (2) develop a set of candidate outcomes prioritized by importance and measurement feasibility. A comprehensive and meaningful set of CMC health outcomes could be used to guide health systems, policy makers, and community leaders in more effectively monitoring health and delivering needed services to CMC.
GCM was conducted as the final phase of a larger project in which we combined a systematic literature review (phase 1; E.S.B., R.C., and B.B.N.; unpublished observations) and an in-depth qualitative interview analysis (phase 2)3 with GCM (phase 3). GCM was used to generate and then synthesize a comprehensive list of potential health outcomes for CMC from a diverse sample of experts.
Through snowball sampling starting with our previous in-depth interview participants (who themselves represented CMC caregivers, advocates, providers, researchers, and policy or health systems experts), we identified a list of diverse, nationally representative CMC experts. Inclusion criteria were having been nominated as a CMC expert, being competent to consent for study participation, and being at least 18 years of age. Potential participants received e-mail invitations to participate in the Web-based concept mapping activity. In the invitations, we explained that participation would take ∼1 hour divided over 2 phases: (1) the idea generation activity and (2) the sorting and rating activity occurring several weeks later. Of the 182 potential participants contacted, 125 individuals across the United States and Canada agreed to participate (69% response rate).
The GCM process involves 4 steps: (1) preparation, (2) idea generation (brainstorming), (3) sorting and rating generated ideas, and (4) empirically creating idea maps.4 We used Concept System Global Max (CS Global) software, which provided a Web-based platform for the concept mapping method (https://www.conceptsystemsglobal.com/). CS Global allowed us to collect, analyze, and represent data from diverse stakeholders at geographically disparate locations at the timing of participants’ convenience. Data collection and analytic procedures undertaken during the GCM phase were based on previous GCM studies4 and were performed under the guidance of Concept Systems consultants. Figure 1 diagrams an overview of this process.
GCM Step 1: Preparation
During multiple team meetings, the research team developed a focus prompt for the brainstorming activity. The focus prompt was intended to elicit a wide array of short, free-text responses regarding potential health outcomes for CMC. Using an iterative consensus-seeking approach, the research team developed 3 potential brainstorming prompts and then pilot tested them with CMC experts who were independent of the research team. Responses to the pilot prompts were reviewed. The team then selected the following prompt: “A healthy life for a child or youth with medical complexity includes: ____.” During GCM preparation, the research team also identified the dimensions on which participants would rate each of the brainstormed statements during the GCM rating phase. After group discussion, the research team established the dimensions as (1) feasibility of measurement and (2) perceived importance, each to be rated on a 6-point Likert scale. The team prepared written instructions and webinars to teach GCM participants how to complete the brainstorming and sorting and rating activities.
GCM Step 2: Idea Generation (Brainstorming)
For the idea generation (ie, brainstorming) step of GCM, each respondent was asked to provide, via the CS Global Web portal, an unlimited number of short, free-text responses to the focus prompt, “A healthy life for a child or youth with medical complexity includes: ____.” Because our purpose was to create as many unique statements as possible without identifying the most popular ones, generated statements were visible to all participants in real time, which reduced redundancy by allowing participants to view what others had already submitted. Participant responses were not linked to names, contact information, or any other personal identifiers.
During the brainstorming step, participants generated 707 statements that completed the focus prompt provided. These statements included 879 individual ideas (ie, some statements had multiple ideas). Because many of these ideas were highly similar, the research team performed coding and editing selection using an iterative consensus-seeking approach to aggregate brainstormed statements into a final list of 77 unique statements with only 1 idea in each statement. The final list was reviewed by research team members to ensure that all 879 individual ideas were represented.
GCM Step 3: Sorting and Rating
After the completion of the brainstorming activity, participants were then recontacted via e-mail and invited to participate in the sorting and rating activity. During this phase, participants were first asked demographic questions, including their geographic location, sex, and self-identified primary and (if any) secondary role related to CMC (choices were CMC parent and/or caregiver, advocate, provider, researcher, policy expert, or health systems expert). On the basis of how participants answered these questions, we categorized participants into 4 mutually exclusive subgroups for analysis: (1) CMC parents and/or caregivers and advocates, (2) CMC providers, (3) CMC researchers, and (4) CMC policy or health systems experts.
For the sorting component of the activity, participants were asked to enter the CS Global Web portal and sort the reduced list of 77 unique ideas. Participants were instructed to group statements according to how similar in meaning or theme they were to one another without a restriction on the number of groups. We explicitly asked participants not to create groups according to priority or value, such as “important” or “hard to measure,” but rather to group statements that they viewed as conceptually similar. Participants sorted the 77 unique ideas into an average of 8 groups.
For the rating component of the sorting and rating activity, participants were asked to rate each of the collectively generated 77 statements in which participants describe outcomes for CMC based on (1) the importance of that outcome and (2) participants’ perceived feasibility of measurement. The following instructions were given: “By measurable, we mean that you can envision this outcome being widely measured across a health system serving children with medical complexity within the next 10 years. By important, we mean how important the outcome is in comparison to all the other outcomes using the full range of the scale.” Responses were recorded on a 6-point Likert scale, with most important and easiest to measure being scaled highest, respectively.
GCM Step 4: Creation of Maps
Once sorting and rating were complete, we used the CS Global software to perform multidimensional scaling and hierarchical cluster analyses, creating visual three-dimensional concept maps that represented the ideas of the participants (ie, cluster representation).4 Generation of the concept maps involved the creation of a point map, a cluster map, and a go-zone plot. The point map acted as the foundation for all other maps, with each statement represented as 1 point on the map. Distances between all points were determined by the frequency with which any 2 statements were sorted into the same group by participants.4
After the point map was created, we then created cluster maps with CS Global by bounding points on the point map into polygons to create nonoverlapping clusters. For the clusters, we empirically grouped the original statements into domains using dissimilarity thresholds calculated from pairwise distances between all points (ie, shorter distances equal less dissimilarity). Multiple maps were automatically generated by using different thresholds, resulting in different numbers of clusters. The research team was then tasked with determining the optimal number of clusters for the final cluster map on the basis of which map was used to qualitatively best identify conceptually meaningful and distinct domains. GCM methodology researchers advise that there is no single or “correct” number of clusters that should be represented on a cluster map but rather that the researchers must determine the appropriate number of clusters to aid in the optimal interpretation of their concept map.4 Regardless of cluster number, the underlying point map remains constant. After analyses of the upper and lower limits of 5 and 20 clusters, respectively, for cluster concept mapping analysis, the research team determined that 10 clusters best fit the data (Fig 1). The research team then created domain names for each cluster that best represented the group of statements encompassed in each cluster.
We also represented our rating data with what is called a go-zone map.4 The go-zone map was used to compare statements across our 2 rating criteria: perceived importance and feasibility of measurement. A go-zone map is a bivariate plot with a point for each statement based on its average participant rating for both variables.4 The map is divided into quadrants, with the horizontal line showing the mean of the rating values for the rating criteria on the x-axis (importance) and the vertical line showing the mean of the rating values for the rating criteria on the y-axis (measurability).4 The top-right quadrant shows statements that were rated as above average for both criteria, termed the go zone. The go zone represents CMC outcomes that have both above-average importance and above-average measurement feasibility according to the stakeholder groups.
Finally, the GCM creation process was repeated for each of the 4 participant subgroups individually to compare and contrast the ideas of participant subgroups.
The University of California, Los Angeles Institutional Review Board approved the study protocol.
In total, 110 participants completed brainstorming and 62 participants completed the sorting activity, with 125 individuals participating in total. The highest percentage of participants (32%) identified their role as a CMC researcher, 26% were CMC providers, 21% were CMC caregivers or advocates, and 21% were policy or health systems experts. Participants were generally diverse in terms of geography, race and/or ethnicity, and sex; participants represented 23 different states and the District of Columbia, and 1 participant resided outside of the United States.
As described above, the brainstorming activity resulted in 707 statements, which included 879 individual ideas. Statements were synthesized and reduced by the research teams to 77 unique ideas (Fig 1). Participants then sorted and rated the 77 unique statements, and these results were used to generate a cluster map. The cluster map that best fit the data had 10 clusters (Fig 2). The team assigned a domain name to each of the 10 mapped clusters. The 10 domains for CMC health outcomes were (1) basic needs, (2) inclusive education, (3) child social integration, (4) current child health-related quality of life, (5) long-term child and family self-sufficiency, (6) family social integration, (7) community system supports, (8) health care system supports, (9) a high-quality patient-centered medical home, and (10) family-centered care. Domain names are described and sample prompts are provided in Table 1.
Importance and Measurability
Among the 10 outcomes for CMC deemed most important to measure, the perceived measurability of these outcomes varied (Table 2). Nearly all domains had at least 1 outcome represented. In the Supplemental Information, we provide a comprehensive list of the domain descriptions and statements listed in order of importance as identified by participants.
Across domains, a go-zone chart revealed 17 outcomes to be both important and feasible to measure (Fig 3). These 17 outcomes included items from 8 of the 10 domains. Example go-zone outcomes included “minimizing medical errors,” “timely access to medical equipment and supplies,” “absence of pain,” “living in a home with a family,” and “an individualized education plan that optimizes learning and development.” Pattern match testing groups visually revealed the perceived importance and feasibility of each domain, with domains being analyzed as a unit (Supplemental Fig 4). The domains basic needs and health care system supports were ranked high in importance and measurability, whereas the domain child social integration was ranked lowest. Subanalyses of pattern match testing by each stakeholder group type revealed consistency in the go-zone items across stakeholder groups (results not shown).
GCM revealed outcomes for CMC that are largely missing from existing work. Furthermore, with it, we efficiently incorporated input from a diverse group of stakeholders (all with expertise on CMC) to combine their perspectives into a shared view of health for CMC. This approach yielded a rich set of domains from which to develop a more strategic approach to examining and measuring health outcomes for CMC and reshaping what we mean by health for this population.
In our preceding studies, we found that researchers in existing CMC literature do not consistently address several of the domains that are ranked as important and feasible, such as basic needs (E.S.B., R.C., and B.B.N.; unpublished observations).3 This represents a large and disturbing gap in our field. Consistent with our 1-on-1 interviews with experts on CMC, these outcomes extend beyond traditional conceptualizations of health, which are often focused on health care use measures that are directly fiscally relevant to health systems.3 With GCM, we elucidate core domains that extend beyond this to give a fuller view of health for CMC, 1 that perhaps more accurately represents both the day-to-day and the long-term challenges and opportunities that CMC and their families face. Our results reveal that more attention to psychiatric, social, and developmental needs may be needed. It will be up to health systems, and the policy makers whose rules govern them, to make these domains fiscally relevant and redesign their care delivery structures and processes accordingly.
To more comprehensively measure health for CMC, the implementation of additional measurement tools beyond what are commonly used in the CMC field is needed. We conducted an informal review of the biomedical literature to explore existing measurement tools for outcomes in the go zone (ie, ranked important and feasible to measure). We found that the richest existing measurement tools were primarily within research surveys such as the National Survey of Children with Special Health Care Needs.6 Moreover, for several go-zone items, tools that might be adaptable to clinical settings do not seem to exist at all. To better align with the outcomes identified by this group conceptualization process, a shift toward a more clinically appropriate measurement framework is needed. To achieve this, health system leaders and policy makers will need to be engaged alongside researchers in the process of setting measurement tool development agendas. The field will actively need to balance perceived importance and measurability. Purposeful alignment with other sectors, such as education and social services, may expedite the adaptation and implementation of a more comprehensive set of measures; for example, educational attainment (as a proposed inclusive education measure) and food security (as a proposed basic needs measure) were developed in nonmedical sectors. In fact, in this study, basic needs was rated as the outcome that was most important to measure. Seizing opportunities to align with existing measures in nonmedical domains and examine correlations with CMC may prove to be strategic. It remains an open question as to whether clinical settings are the best setting in which to collect these data. For instance, if data-sharing systems across sectors were enhanced, then one could potentially imagine multiple sources of data collection that could be used to increase the efficiency of outcome measurement.
As the field of pediatric complex care evolves, the eventual goal would be to incorporate measures of health for CMC that align with the broader view of health that emerged from this study and others like it. Ultimately, measures aligning with a broader view of health can become measures of health system quality as monitored and regulated by health systems. Additionally, policy makers and medical educators alike may draw from this conceptualization of health to cultivate communities and workforces of providers who are able to optimize health for CMC. Doing so, however, will require a collaborative multisector financing approach that currently eludes most health systems. Throughout this process, attentiveness to issues of equity (related not only to disability but also to factors such as race, ethnicity, and socioeconomic status) is crucial.
Several limitations of our approach exist. First, the inductive content analysis inherent to GCM is subject to investigator biases. Second, although the participant sample was large and diverse, the study sample may lack some important perspectives, including those of the CMC themselves. Finally, the feasibility of measuring many of the identified concepts is not yet clear, and perceived measurability is fairly subjective. Participants may have had differing ability to assess feasibility, and factors that influence measurability can change over time. Additionally, in our rating question on the perceived feasibility of measurement for each outcome, there may have been conflation between feasibility in developing a measurement tool versus actual logistic feasibility of measurement in a clinical setting. Nevertheless, we believe we provide valuable guidance on priorities for both developing and implementing the measurement of a broader set of key CMC health outcome measures.
The extent to which the health outcomes identified though the GCM approach extend beyond CMC to other pediatric populations is unclear. For CMC, the focal point for the domain of access to a high-quality patient-centered medical home may be centered on access to a subspecialist as a medical home provider. How would outcomes and prioritization differ for the larger group of children with special health care needs, for all pediatric populations, or infants versus school-aged children versus adolescents? Because CMC are an extreme example of a pediatric population with health needs (and the high family burden, reward, and involvement that result from these needs), lessons learned from CMC may elucidate important outcomes that are more easily obscured in other pediatric populations. Pediatrics might benefit from judiciously applying lessons from CMC to other populations, as was done historically in the conceptualization of the pediatric medical home from CMC and subsequent extension to the larger pediatric population.7
Inclusive, systematic research approaches such as GCM may lead to a better, more comprehensive approach for identifying common health outcomes for CMC. The conceptualization achieved through GCM can be used to push the field closer to a more comprehensive population health framework for CMC. This framework can be used to inform policies and programs to better serve CMC and their families. Furthermore, lessons learned from CMC may have implications for the conceptualization and measurement of health for other special needs populations and potentially for all children.
We thank Dr Edward Schor for his contributions.
- Accepted May 17, 2018.
- Address correspondence to Elizabeth S. Barnert, MD, MPH, MS, Department of Pediatrics, UCLA, 10833 Le Conte Ave, 12-467 MDCC, Los Angeles, CA 90095. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by a grant from the Lucile Packard Foundation for Children’s Health and by the University of California, Los Angeles Children’s Discovery and Innovation Institute.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
COMPANION PAPER: A companion to this article can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2018-1925.
COMPANION PAPER: A companion to this article can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2018-1925.
- Cohen E,
- Kuo DZ,
- Agrawal R, et al
- Berry J,
- Agrawal R,
- Cohen E,
- Kuo K
- National Survey of Children with Special Health Care Needs
- Copyright © 2018 by the American Academy of Pediatrics