Pediatric Medical Complexity Algorithm: A New Method to Stratify Children by Medical Complexity
OBJECTIVES: The goal of this study was to develop an algorithm based on International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), codes for classifying children with chronic disease (CD) according to level of medical complexity and to assess the algorithm’s sensitivity and specificity.
METHODS: A retrospective observational study was conducted among 700 children insured by Washington State Medicaid with ≥1 Seattle Children’s Hospital emergency department and/or inpatient encounter in 2010. The gold standard population included 350 children with complex chronic disease (C-CD), 100 with noncomplex chronic disease (NC-CD), and 250 without CD. An existing ICD-9-CM–based algorithm called the Chronic Disability Payment System was modified to develop a new algorithm called the Pediatric Medical Complexity Algorithm (PMCA). The sensitivity and specificity of PMCA were assessed.
RESULTS: Using hospital discharge data, PMCA’s sensitivity for correctly classifying children was 84% for C-CD, 41% for NC-CD, and 96% for those without CD. Using Medicaid claims data, PMCA’s sensitivity was 89% for C-CD, 45% for NC-CD, and 80% for those without CD. Specificity was 90% to 92% in hospital discharge data and 85% to 91% in Medicaid claims data for all 3 groups.
CONCLUSIONS: PMCA identified children with C-CD (who have accessed tertiary hospital care) with good sensitivity and good to excellent specificity when applied to hospital discharge or Medicaid claims data. PMCA may be useful for targeting resources such as care coordination to children with C-CD.
- administrative data
- claims data
- medical complexity
- C-CD —
- complex chronic disease
- CD —
- chronic disease
- CDPS —
- Chronic Illness and Disability Payment System
- COE4CCN —
- Center of Excellence on Quality of Care Measures for Children with Complex Needs
- CRG —
- Clinical Risk Group
- ED —
- emergency department
- ICD-9-CM —
- International Classification of Diseases, Ninth Revision, Clinical Modification
- NC-CD —
- noncomplex chronic disease
- PMCA —
- Pediatric Medical Complexity Algorithm
- PQMP —
- Pediatric Quality Measures Program
- SCH —
- Seattle Children’s Hospital
- WA-Medicaid —
- Washington State Medicaid
What’s Known on This Subject:
Quality measures developed by the Pediatric Quality Measures Program are required to assess disparities in performance according to special health care need status. Methods are needed to identify children according to level of medical complexity in administrative data.
What This Study Adds:
The Pediatric Medical Complexity Algorithm is a new, publicly available algorithm that identifies the small proportion of children with complex chronic disease in Medicaid claims and hospital discharge data with good sensitivity and good to excellent specificity.
In March 2011, the Centers for Medicare & Medicaid Services and the Agency for Healthcare Research and Quality partnered to fund 7 Centers of Excellence that constitute the Pediatric Quality Measures Program (PQMP) mandated by the 2009 Child Health Insurance Program Reauthorization Act. 1 The charge to the PQMP is to develop new quality of care measures and/or enhance existing measures for children’s health care across the age spectrum. 2 , 3 The Center of Excellence on Quality of Care Measures for Children with Complex Needs (COE4CCN) was charged with identifying and/or developing a valid method to assess disparities in care according to level of medical complexity for children with special health care needs.
COE4CCN initially considered several methods to classify children according to level of medical complexity but chose to focus on algorithms that use International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), codes. Although limited by reliance on billing data, 4 approaches based on ICD-9-CM codes are relatively inexpensive to use compared with survey methods, can be used on existing administrative data with relative efficiency, and are favored by states and insurance plans. Although several proprietary ICD-9-CM–based algorithms might be used for these purposes, the mandate was to identify or develop a tool that was open access and publicly available for widespread use in quality measurement. After reviewing and evaluating several existing algorithms, 5 – 14 we chose to modify the Chronic Disability Payment System (CDPS) algorithm. 12 The objectives of the present study were to develop the modified version of CDPS and assess its sensitivity and specificity for correctly identifying children with varying levels of medical complexity.
This development and validation research was undertaken in 5 stages: (1) development of consensus definitions for 3 levels of medical complexity; (2) selection of an existing ICD-9-CM algorithm; (3) modification of the algorithm to conform to the consensus medical complexity definitions; (4) selection of a gold standard population of children through medical record review; and (5) evaluation of the modified algorithm’s sensitivity and specificity when applied to the gold standard population. All study procedures were reviewed and approved by the institutional review boards of Seattle Children’s Research Institute and Washington State.
Developing Consensus Definitions
To develop an algorithm that would facilitate assessment of disparities according to special health care need status, COE4CCN developed consensus definitions for 3 levels of medical complexity: children with complex chronic disease (C-CD), children with noncomplex chronic disease (NC-CD), and children without chronic disease (CD) (Table 1). The COE4CCN Medical Complexity Working Group developed the first draft of these consensus definitions after review and discussion of 2 previously published care coordination conceptual frameworks and their accompanying definitions for levels of medical complexity. 15 , 16 The working group is composed of individuals with expertise in inpatient and outpatient management of children with C-CD and/or NC-CD. The entire COE4CCN then provided review and feedback on the draft consensus definitions. COE4CCN includes 43 representatives from 2 state Medicaid agencies, Family Voices, pediatric nursing, hospital medicine, and outpatient primary care, as well as pediatric health services research. The final consensus definitions incorporated the center-wide feedback.
After evaluating several existing algorithms, 5 – 14 we chose to modify CDPS, 12 which is a diagnosis-based risk adjustment model that uses ICD-9-CM codes to assess risk of incurring high costs. 17 CDPS was selected because it has the most comprehensive listing of ICD-9-CM codes among the algorithms evaluated, is publicly available, and, in contrast to most other algorithms, includes important codes for mental health conditions.
We developed and evaluated a novel algorithm for this study, the Pediatric Medical Complexity Algorithm (PMCA). PMCA represents a modification of CDPS that conforms to the COE4CCN consensus definitions for medical complexity. CDPS modification involved removing several types of ICD-9-CM codes, including those consistent with adult illness (eg, myocardial infarction), related to childbirth, consistent with acute illness (eg, acute otitis media), and representing pediatric chronic conditions that are most often mild in severity (eg, eczema, myopia, iron deficiency anemia). Excluding codes for conditions with a substantial proportion of mild disease (eg, eczema) from PMCA reduces the potential for overestimation of disease burden in the population (ie, false-positive findings). However, this process may result in children with more severe forms of such diseases not being captured or correctly classified (ie, false-negative findings).
Further modifications included the addition of “flags” to each retained ICD-9-CM code by 2 authors (T.D.S. and R.M.-S.). Conditions associated with deteriorating health and an increased risk of shorter life expectancy in adulthood (defined as death in the fourth to fifth decade [eg, cystic fibrosis, complex congenital heart disease, malignancy]) were flagged as progressive; when consensus was not immediately reached, life expectancy data for the condition were reviewed, and a final decision was made. Body system flags were also assigned to permit body system counts and subsequent classification to NC-CD (1 body system) or C-CD (≥2 body systems). A full list of the ICD-9-CM codes included in PMCA, and their progressive and body system flags, is provided in Supplemental Table 5. SAS programming code (SAS Institute, Inc, Cary, NC) was subsequently developed for PMCA to categorize children into the 3 levels of medical complexity based on adjudicated claims.
To capture data on children with C-CD based on technology dependence for >6 months (Table 1), we adapted a previously developed set of technology assistance ICD-9-CM codes 18 and tested PMCA’s sensitivity and specificity with and without these additional codes.
Identifying a Gold Standard Population: Classifying Children by Medical Complexity By Using Medical Record Review
Children 0 to 18 years old, insured by Washington State Medicaid (WA-Medicaid), and seen at Seattle Children’s Hospital (SCH) for ≥1 emergency department (ED) visit and/or inpatient stay in 2010 were potentially eligible for the study. To oversample children in the C-CD group, these children were categorized into 1 of 9 mutually exclusive risk groups by using 3M Clinical Risk Group (CRG) software (St Paul, MN) 19 applied to 4 years (2007–2010) of SCH ED, inpatient, and day surgery administrative data. After CRG categorization, a sample of 1000 children was randomly selected, with oversampling (n = 500) for children with lifelong chronic conditions (CRG groups 5b, 6, 7, 8, and 9).
A trained nurse researcher (J.P.) blinded to CRG categorization made assignments into 1 of 3 levels of medical complexity (Table 1) by reviewing all available SCH electronic medical records. When level assignment was unclear, cases were reviewed by a panel of physicians (T.D.S., A.Y.C., M.H., and R.M.-S.) also blinded to CRG categorization, and assignments were made by consensus. Among the sample of 1000 randomly selected children, medical records were reviewed until the target gold standard population of 700 children was assembled. The target population included 350 children with C-CD, 100 with NC-CD, and 250 without CD. These sample sizes were determined a priori to allow for stable estimates of PMCA’s sensitivity and specificity for correctly classifying patients into the 3 levels of complexity.
Almost all (699 of 700) of the gold standard population children were successfully matched in the WA-Medicaid claims database. Twenty individuals >18 years old were excluded because they were not eligible for WA-Medicaid for substantial portions of the study period and had incomplete claims data. One child having only secondary Medicaid coverage was also excluded. The final WA-Medicaid study sample numbered 678, whereas all 700 children were included in the SCH study sample.
To determine the representativeness of the gold standard population, characteristics for the 678 study children were compared with the overall WA-Medicaid–insured child population from 2009 to 2011.
Three versions of the PMCA SAS code were developed to characterize the timing and frequency of coded conditions from administrative data: the least, more, and most conservative versions described in Table 2. Children in the sample had up to 3 years of data available for analysis in both the SCH and WA-Medicaid claims databases, 1 year before and 1 year after the year of their hospitalization or ED visit (ie, January 1, 2009–December 31, 2011). All children were included regardless of how much data they had available to contribute to the analysis. All children in both the SCH and WA-Medicaid samples had at least 1 claim in 2010 that represented the ED and/or inpatient encounter making them eligible for gold standard population selection.
We determined PMCA’s sensitivity and specificity for correctly classifying children into the 3 levels of complexity by using SCH discharge and WA-Medicaid claims data. SCH data included administrative claims from inpatient, ED, and day surgery encounters. WA-Medicaid data included all inpatient and outpatient claims provided to the state. We also evaluated the performance of 3 different versions of the PMCA SAS code described in Table 2.
After applying the PMCA SAS code to SCH discharge and WA-Medicaid claims data, we examined cases misclassified by PMCA to determine if there were patterns that might inform future modifications to the algorithm.
Characteristics of the gold standard population were compared with the overall WA-Medicaid–insured population of children (Table 3). The study sample was younger and more racially and ethnically diverse. A disproportionate number of study children had fee-for-service coverage (47% vs 20%) and prolonged Medicaid eligibility (56% vs 46% with 36 months).
Using WA-Medicaid data, 536 (79%) of the study sample had 3 years of claims data to contribute to the analyses, 122 (18%) had 2 years, and only 20 (3%) had just 1 year. In contrast, using SCH data, 152 (22%) of the sample had 3 years, 252 (36%) had 2 years, and 296 (42%) had 1 year of claims data to contribute to analysis.
Optimal performance conditions for PMCA in hospital discharge and Medicaid claims data are shown in Table 4. Using up to 3 years of hospital discharge data and the least conservative version of the PMCA code (Table 2), algorithm sensitivities for correctly classifying children were 84% for children with C-CD, 41% for children with NC-CD, and 96% for children without CD. Using up to 3 years of WA-Medicaid claims data and the more conservative version of the PMCA code, sensitivities were 89% for children with C-CD, 45% for children with NC-CD, and 80% for children without CD. Specificity was good to excellent in both hospital (90%–92%) and Medicaid (85%–91%) data for all 3 levels of complexity according to the least and more conservative PMCA code versions, respectively. The most conservative version of the PMCA code resulted in the highest rate of misclassification. Addition of the technology assistance codes did not improve PMCA’s sensitivity or specificity regardless of which version of the code was used (data not shown).
We identified 3 patterns of misclassification in which PMCA (applied to either hospital discharge or Medicaid claims data) categorized children in the gold standard population as having no CD who were determined to have NC-CD according to medical record review. The first pattern involved children with mild episodic conditions that were counted by the medical record reviewer as chronic but are not included in PMCA (eg, eczema, headache). The second pattern involved children with conditions that had largely resolved (eg, epilepsy but seizure-free for 2 years). For these cases, there was evidence for the condition in the medical record, but no health care utilization/claims for it during the study period (January 1, 2009–December 31, 2011). The third pattern involved children with mental health conditions or developmental delay, conditions that are commonly undercoded in claims data. 3 , 20
Children determined to have NC-CD according to medical record review who were misclassified by PMCA as C-CD in both hospital and Medicaid data often had chronic disease in 2 systems with resolved disease in at least 1 system (eg, asthma, repaired ventricular septal defect). In such cases, the resolved condition was not counted by the medical record reviewer but did result in claims that were detected by using PMCA.
To enable assessment of disparities in care according to special health care need status, we developed a novel algorithm by modifying an existing ICD-9-CM–based algorithm (CDPS) to align with the COE4CCN consensus definitions for 3 levels of medical complexity. PMCA exhibited good sensitivity for correctly categorizing children with C-CD, excellent sensitivity for correctly categorizing children without CD, but poor sensitivity for correctly categorizing children with NC-CD. For optimal identification of these 3 groups of children using to Medicaid claims data, we recommend using the more conservative version of the PMCA code and up to 3 years of claims data when available. For hospital discharge data (limited to ED, inpatient, and day surgery claims), we recommend using the least conservative version of the PMCA code and up to 3 years of data when available. (The PMCA SAS code and documentation are available from the authors upon request.)
In hospital discharge data, ICD-9-CM codes for chronic disease are relatively infrequent because they are assigned only at the time of an ED, inpatient, or day surgery encounter. As a result, the least conservative version of PMCA, in which only 1 use of an included ICD-9-CM code is required during a 3-year time period (Table 2), resulted in the best performance. In contrast, Medicaid claims data capture far more health care utilization; therefore, the more conservative version of PMCA, in which 2 uses of an included ICD-9-CM code are required during a 3-year time period, resulted in the best performance.
Children with NC-CD are the most difficult to correctly identify by using administrative data. These conditions may be episodic for a given child and can encompass a wide range of severity, from mild to severe. This variation results in highly fluctuating health care utilization over time within the NC-CD group. With hospital discharge data, PMCA misclassified children with NC-CD into both the without-CD and C-CD groups. Children with NC-CD who were misclassified according to PMCA as without CD in hospital data often had episodic disease, developmental concerns, or a resolving problem that required outpatient care. Although electronic medical records used for this study included outpatient, inpatient, and ED encounters, the hospital-based discharge data consisted of ED, inpatient, and day surgery claims. SCH outpatient claims were not available, thus constraining PMCA’s ability to correctly detect conditions being managed exclusively in that setting. Due to the relative infrequency of encounters in the SCH discharge data, we elected to use the least conservative version of PMCA (Table 2). However, this approach unfortunately increased the likelihood that false-positive findings would be observed in the C-CD group because only 1 claim for each of 2 separate body systems is required during the measurement period for classification into this group. This method potentially results in children who have a history of chronic disease in 2 body systems but resolved disease in at least 1 (NC-CD) being incorrectly classified according to PMCA as having C-CD.
In Medicaid claims data, PMCA more commonly misclassified children with NC-CD as having C-CD. Because our medical record review focused on visits occurring in the SCH outpatient and inpatient settings, substantial portions of a child’s care occurring in settings outside of SCH were not available to the reviewer; it is possible, therefore, that relevant conditions were not captured when categorizing children in the gold standard population. In contrast, when PMCA was applied to Medicaid claims data, health care utilization both within and outside of SCH were available to inform categorization. Replication of this study in a system in which all medical records data are available for review could potentially address this limitation.
We purposefully oversampled children with C-CD for the gold standard population (350 of 700); this population therefore differs from the overall state Medicaid population. As expected with large numbers of children with C-CD, the study sample had more fee-for-service coverage and prolonged Medicaid eligibility. In addition, the sample was younger and more racially and ethnically diverse than the state Medicaid population. Overall, 6% of children insured by WA-Medicaid from 2009 to 2011 were classified with C-CD, 16% with NC-CD, and 74% without CD.
As the Patient Protection and Affordable Care Act is implemented, Medicaid and the health care system increasingly need strategies to allocate resources. Children with C-CD are most likely to benefit from care coordination and other resources, and accurate identification of this group is critical. 21 These children may suffer the worst quality of care for many of the measures under development by the PQMP. Use of PMCA will allow us to address the legislative mandate to assess disparities by using special health care need status and further test the hypothesis that children with C-CD experience poorer quality of care than either children with NC-CD or healthy children.
The present study had several additional limitations. Approaches relying on secondary data containing ICD-9-CM codes are only as good as the completeness of the original sources. Therefore, children who do not interact with the health care system will lack the necessary information to define their level of medical complexity, and children enrolled in ≥2 health plans will have incomplete encounter information in both data sets. Furthermore, encounters for conditions that do not result in a claim will not be captured in administrative data. Any manual assessment of medical records data are subject to human error; however, the reviewer for the present study is experienced in this type of review 22 and brought cases in which she was unsure to the physician panel for evaluation. Assignment of the body system and progressive flags to ICD-9-CM codes in PMCA was based on the clinical experience of 2 authors; others may thus disagree with our designations of whether a condition is progressive or with body system assignments for conditions. Because this study targeted identification of children with C-CD, our gold standard population was drawn from a tertiary care hospital and was not representative of WA-Medicaid–insured children. Further validation work in other populations of children, including health systems in which most children primarily access outpatient care, is needed. We also anticipate the need for further changes to PMCA to ensure compatibility with future widespread adoption of International Classification of Diseases, 10th Revision, Clinical Modification, codes.
Despite its limitations and potential future revisions, PMCA is a new, publicly available algorithm that identified children with C-CD (who have accessed tertiary hospital care) with good sensitivity and good to excellent specificity when applied to either hospital discharge or Medicaid claims data. As health care reform is implemented, use of PMCA will be critical to target resources and services such as care coordination to children with the most needs.
The COE4CCN Medical Complexity Working Group also included Kim Arthur, Penny Knapp, Michael Murphy, Justine Nelson, Jeff Tenney, and Chuan Zhou. We appreciate feedback on the manuscript provided by Doug Opel.
- Accepted February 27, 2014.
- Address correspondence to Tamara D. Simon, MD, MSPH, Seattle Children's Research Institute, Building 1, M/S JMB-9, 1900 Ninth Ave, Seattle, WA 98101. E-mail:
Dr Simon helped design the study, modified the Chronic Illness and Disability Payment System algorithm to develop the Pediatric Medical Complexity Algorithm, reviewed cases in developing the gold standard population, and drafted the initial manuscript; Dr Cawthon helped design the study, supervised the analyses in Medicaid data, reviewed and revised the manuscript, and takes responsibility for the integrity of the Medicaid data and accuracy of the Medicaid data analysis; Ms Stanford conducted the analyses in hospital discharge data, critically reviewed the manuscript, and takes responsibility for the integrity of the hospital data and accuracy of the hospital data analysis; Ms Popalisky performed chart review to identify the gold standard population, discussed cases with the physician panel, and critically reviewed the manuscript; Ms Lyons and Mr Woodcox conducted the analyses in Medicaid claims data and critically reviewed the manuscript; Drs Hood and Chen reviewed cases in developing the gold standard population and critically reviewed the manuscript; and Dr Mangione-Smith conceptualized and designed the study, modified the Chronic Illness and Disability Payment System algorithm to develop the Pediatric Medical Complexity Algorithm, reviewed cases in developing the gold standard population, and reviewed and revised the manuscript. All authors approved the final manuscript as submitted.
The contents of this article are solely the responsibility of the authors and do not necessarily represent the official view of the sponsors. None of the sponsors participated in design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This study was done under funding from a cooperative agreement with the Agency for Healthcare Research and Quality and Centers for Medicare and Medicaid Services, grant #U18HS020506, part of the Children's Health Insurance Program Reauthorization Act Pediatric Quality Measures Program. Dr Simon is also supported by award K23NS062900 from the National Institute of Neurological Disorders and Stroke. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Zima BT ,
- Murphy JM ,
- Scholle SH ,
- et al
- Feudtner C ,
- Christakis DA ,
- Connell FA
- Feudtner C ,
- Silveira MJ ,
- Christakis DA
- Feudtner C ,
- Christakis DA ,
- Zimmerman FJ ,
- Muldoon JH ,
- Neff JM ,
- Koepsell TD
Kuhlthau KA, Beal AC, Ferris TG, Perrin JM. Comparing a diagnosis list with a survey method to identify children with chronic conditions in an urban health center. Ambul Pediatr. 2002;2(1):58–62
University of California, San Diego. Chronic Illness and Disability Payment System. Available at: http://cdps.ucsd.edu/index.html. Accessed April 15, 2013
Antonelli RC, McAllister JW, Popp J. Making Care Coordination a Critical Component of the Pediatric Health System: A Multidisciplinary Framework. New York, NY: Commonwealth Fund; 2009
McDonald KM, Schultz E, Albin L, et al. Care Coordination Atlas Version 3 (Prepared by Stanford University under subcontract to Battelle on Contract N. 290-04-0020). AHRQ Publication No. 11-0023. Rockville, MD: Agency for Healthcare Research and Quality. November 2010
- Cohen E ,
- Kuo DZ ,
- Agrawal R ,
- et al
- Copyright © 2014 by the American Academy of Pediatrics