Objective. To determine whether a reduced item set can identify children who have chronic conditions with a level of at least 90% accuracy compared with the complete Questionnaire for Identifying Children With Chronic Conditions (QuICCC).
Background. The QuICCC was developed to operationalize a conceptually based, noncategorical definition of chronic conditions developed by Stein et al. It contains 39 item sequences administered to a parent that assess 3 types of consequences: functional limitations; reliance on compensatory mechanisms or assistance; and service use or need above usual for age. The QuICCC has been validated and widely adopted as a means of identifying children without using a diagnosis checklist, but there is considerable interest in shortening it.
Design/Methods. Through secondary analyses of 3 data sets (Ns = 1265, 1388, and 4831), we identified a short list of items that identified >90% of children who were identified by the 39-item QuICCC. We administered these 16 items to 2 new samples of parents. In Study 1 we administered the 16 items in the shortened version first, followed by the other 23 items, and compared the results on the short and reordered long versions. In Study 2, the 39- and 16-item versions were each administered, one in person and the other by phone, in random order to the same respondent within a 2-week period. These data were analyzed to compare the short and longer versions at the 2 time points and within the single, longer 39-item format (simultaneous data).
Results. In Study 1 (N = 630) only 4 children were missed by the 16-item version who were identified by the longer version (sensitivity 98.6%; specificity 100%; positive predictive value 100%; negative predictive value 98.8% κ 0.987). In Study 2 (N = 552), no children were missed by the 16-item subset of the 39 items when looking at the simultaneous data. When the two forms were administered 2 weeks apart, the 16-item version had a sensitivity of 87%, specificity of 90%, positive predictive value of 93%, negative predictive value of 82%, and κ of 0.78 compared with the longer QuICCC. These results correspond exactly to the data obtained in a 2-week test-retest study for the QuICCC itself. The new form (the QuICCC-R) takes <2 minutes to administer on average (range 1–4 minutes) compared with 7 to 8 minutes for the full QuICCC.
Conclusions. The results met our criteria for agreement, and we conclude that the QuICCC-R is a satisfactory alternative for screening populations. However, the full QuICCC has other applications beyond screening that may not apply to the QuICCC-R, the shorter version.
- NACHRI =
- National Association of Children's Hospitals and Related Institution •
- QuICCC =
- Questionnaire for Identifying Children With Chronic Conditions •
- QuICCC-R =
- Questionnaire for Identifying Children With Chronic Conditions-Revised
Current concerns about the use of health care resources have heightened interest in the identification of children with chronic health conditions. However, few mechanisms exist for accomplishing this task. In part, this void is a result of inconsistencies in definitional issues, but there has been increasing convergence around a unified conceptualization.1–3 As a result, the identification of children with chronic conditions has moved from a checklist of diagnoses to an approach that classifies children noncategorically.1–3 Until recently, there have been no validated techniques for implementing such an approach. To date there are only 2 published validated tools for the identification of children with chronic conditions. One of these is the system developed by National Association of Children's Hospitals and Related Institutions (NACHRI). The NACHRI system is based on use of billing codes and depends on a complicated computerized algorithm.4 It is not yet available and will be a commercial product.
The second mechanism is an instrument we developed to identify children with chronic conditions based on a parental interview, the Questionnaire for Identifying Children With Chronic Conditions (QuICCC).5* This questionnaire is easily administered by a lay interviewer and contains 39-item sequences that parents are able to answer comfortably about their children's functional limitations, reliance on assistive devices or other compensatory mechanisms, and service use or need. The measure is based on a specific conceptual definition of chronic conditions3 and has been demonstrated to have considerable validity for both epidemiologic purposes and other applications.6 The QuICCC also has been endorsed for use in implementing the definition of children with special health care needs adopted by the Maternal and Child Health Bureau.2
Although the QuICCC is a successful operationalization of a noncategorical definition, its administration takes an average of 7 to 8 minutes for all children in a household.5 Because this time may be excessive for some purposes, there is interest in reducing the number of questions used or the length of time needed for administration if this can be done without seriously compromising the properties of the measure.
We undertook a series of studies to test the feasibility of reducing the length of the QuICCC. The purpose of this report is to describe the steps taken to shorten the QuICCC through analyses of existing data sets, and to present the findings from 2 subsequent studies designed to compare the long and short versions using new samples.
There were 2 steps to the process of developing a shorter version of the QuICCC. The first involved secondary analysis of 3 data sets containing the original QuICCC items. The second involved field testing the shorter version and comparing results obtained with it to those obtained by administering the full instrument. Both field tests and the secondary analyses were reviewed and approved by the Institutional Review Board of Albert Einstein College of Medicine.
Using 1 data set at a time, we employed a list-wise deletion analysis. In this method, children identified by the most frequently subscribed item were ascertained and the children selected by that item were removed from the sample. Then, the next most frequent item that identified the remaining children was found and these children were removed. The process was continued until virtually all of the children identified by the full QuICCC were selected. These procedures were repeated on each of the 3 data sets identified below. All 3 data sets had been collected as part of our work to validate the QuICCC.
Bronx Sample: The first sample was obtained by a random digit-dial telephone survey. Eligible respondents were English- and Spanish-speaking primary caretakers of children from birth to 18 years old living in the Bronx, New York. A total of 3092 households were randomly contacted. Of these, ∼10% refused to participate or terminated the interview at some point. Of the remaining 2781 households, 662 or 25% had children under 18 years old. We obtained data on a total of 1265 children who lived in these 662 households. In this random digital-dial survey, the respondent was asked to list all children under 18 years old living in the household and the QuICCC was then administered. QuICCC information was collected on each child in the household.
National Sample: The second data set replicated the survey content and design of the Bronx study described above on a national sample. However, only English-speaking households with children under the age of 18 were eligible to be in the sample. A multistage sampling design was used, calculated from the distribution of households with children by region of the country based on the 1990 Census Population and Housing Summary. Telephone numbers were randomly generated to represent this distribution. The sample was drawn in replications of 400 numbers. All chosen numbers were dialed up to 4 times before a substitute number was used. A total of 7998 numbers were dialed. Three thousand six hundred thirty-nine of the numbers were successfully contacted. Of those who agreed to participate, 80% did not have children under the age of 18, leaving a final sample size of 730 households. Complete data were collected on 712 households, representing 1388 children. The data have been analyzed previously by us for other purposes.7–8
Arizona Sample: The Arizona Department of Health Services Office for Children with Special Health Care Needs conducted a 2-part Arizona Child Health Screen and Family Needs Survey that included the QuICCC. The study was done in collaboration with our research group. Data collection across Arizona was conducted by the Arizona Office for Children with Special Health Care Needs in 1995, and data were gathered using household surveys from a statewide random sample that included 5160 children 0 to 21 years old. QuICCC data were collected on each individual child. For consistency and comparability with other data sets, we limited our analyses to the 4831 children <18 years old.
The subset of items identified by secondary analyses was reformulated into a shortened version of the QuICCC named the QuICCC-Revised (QuICCC-R). The QuICCC-R was then field tested in 2 ways against the 39-item QuICCC. In the first field test, the entire QuICCC was administered to parents, but the content was reordered so that the shorter list of QuICCC-R items was administered first, followed by the remaining items that were being proposed for omission. Administering the items in a new sequence allowed us to assess whether the initial success of the shorter item set was dependent on the previous order or context of items within the original longer instrument. It enabled us to ascertain whether anyone missed by the QuICCC-R was identified by the remaining items.
The second field test was conducted by administering both the original QuICCC and the proposed QuICCC-R to the same respondent using a format parallel to a test-retest model. The order of the instruments was randomly alternated, with the first administered in person and the second administered by telephone within a 2-week period. This design allowed us to analyze the data in 2 ways: 1) to compare the children identified by shorter item subset when it was embedded within the original 39-item form with those identified by the full item set (simultaneous data); and 2) to compare information from the same respondent obtained on the 2 different instruments albeit at different times. All field testing were done by trained lay interviewers. We set a κ of 0.90 or above as a criterion for success.
For both field tests, we drew the sample from parents bringing their children for medical care at an urban hospital center and its affiliated ambulatory network medical center sites. These sites included the inpatient and ambulatory service sites (both specialty and primary care). Parents were approached and asked to participate in an interview about child health. We aimed to collect data from a sample of convenience of at least 400 parents for each field test. Consenting parents were asked to identify all the children in their household <18 years old; the child with the most recent birthday was selected to be the subject of the study.
We aimed to include in the 2 samples children with acute illnesses (short- and long-term), because these children pose an increased risk of misidentification (ie, overidentification as a child with a chronic condition). We also oversampled children with chronic illnesses to assure that we would obtain a sufficient number of identified children for purposes of statistical analysis and validation. We expected that between one third and one half of the children would be determined to have a chronic condition using the full 39-item QuICCC.
Parents were recruited systematically on site in each of the settings by trained lay interviewers. Eligible parents or guardians were invited to participate in the study and were interviewed in person. In each study, parents were interviewed for ∼15 minutes. In addition to one of the versions of the QuICCC, parents were asked some questions about their demographic characteristics.
For the first study, we approached 1153 parents. Of these, 219 were excluded because they did not communicate well enough in English to be interviewed and 214 (23%) parents refused to participate in the study. A total of 720 parents were interviewed.
For Study 2, a total of 1026 parents were approached: 185 were excluded because they did not speak English and 12 because they had no home telephone for the follow-up; 277 parents (33%) refused to participate, leaving a sample of 552. There were no significant differences in the characteristics of those with (n = 444) and without (n = 108) successful follow-up. There were no differences in follow-up rates based on the version of the survey that was given first, with 80% and 81% of the parents in the QuICCC and QuICCC-R groups, respectively, completing the second version of the survey. There also were no differences in follow-up based on whether the child was identified as having or not having a chronic condition at the time of the first survey administered (79% vs 82%, respectively). In terms of sociodemographic differences between the groups with and without follow-up, parents who completed the follow-up were more likely to speak a language other than English in their homes. However, differences in race, education, and receipt of public assistance were not statistically significant.
Data were analyzed using the Statistical Package for the Social Sciences (SPSS Inc, Chicago, IL) for Windows 9.0.10Identifications using the QuICCC-R and QuICCC were compared and sensitivity, specificity, positive and negative predictive values, and κ statistics were calculated.
Secondary analysis determined that a common set of 16 items identified >95% of the children who had been identified by the full 39-item set in each of the 3 data sets. The results of these analyses are found in Table 1. This shortened 16-item version is called the QuICCC-R. A summary of QuICCC items is shown in Table 2.
A total of 720 resequenced questionnaires were administered and 327 (45%) children were identified by the 16-item sequence, compared with 331 (45.9%) identified by the full 39-item resequenced QuICCC. All but 4 children who were identified by the full 39-item survey also were identified by the 16 items in the QuICCC-R subset. The sensitivity (ie, percentage of positive cases correctly identified) of the 16-item subset against the full reordered QuICCC survey was 98.8%, and the κ statistic specifying the level of agreement between the 2 forms was extremely high at 0.99 (P < .0001; Fig 1).
Study 2—Separate Administrations of Original QuICCC and Shortened QuICCC-R
To reproduce the analysis done in Study 1, the full 39-item QuICCC (given at either administration in this study in the original order) also was compared with its own corresponding 16-item subset (simultaneous data). In this analysis, no child identified as having a condition by the full QuICCC was missed by the smaller item subset.
We then compared how children were classified by the 2 alternating versions of the survey when they were administered at 2 separate times. Of the 444 cases with 2 administrations, data from 2 respondents had to be excluded because of incomplete data on the long form. Therefore, all additional analyses discussed below were based on a sample size of 442 respondents.
There was 88% agreement in classification of children between the 2 forms (Fig 2). On the full QuICCC, 261 of 442 (59.0%) children were identified as having a chronic condition and 181 did not have a condition. The sensitivity (percentage of positive cases correctly identified) of the shortened QuICCC-R compared with the original QuICCC was 87.1 (n = 227), and its specificity (percentage of negative cases correctly identified) was 89.0 (n = 161). The κ statistic specifying the level of agreement between the 2 forms was 0.75, and it was significantly different from chance (P < .0001). These values are comparable to those obtained when the 2-week test–retest reliability of the original QuICCC was examined.5 In that study, κ was 0.73 and the percent agreement between the 2 administrations also was 88%.
To understand the discrepancies we examined possible sources of error. Each QuICCC item asks about a specific consequence the child might experience, and most items contain additional probes about that consequence's relationship to a condition and its expected duration. Therefore, there are 2 potential sources of discrepancies. We found that discrepancies between the 2 administrations of the survey were equally divided between parents who changed their answers about whether the child had a particular consequence from yes to no (or vice versa) on the first part of the question and those who said yes to the initial portion of the question sequence both times they were interviewed, but changed their answers on either of the probes that asked about relationship to a condition or duration. A majority of the discrepancies occurred on the most commonly subscribed items.
We also examined respondent characteristics to ascertain whether these characteristics were associated with discrepancies in the classification of the same children using the 2 different forms. Ethnic background and respondent education were unrelated to the likelihood of discrepant classifications. The only significant factor was whether the respondent spoke a language in addition to or other than English in the home. The odds ratio showed that the children of these respondents were more than twice as likely (odds ratio: 2.13) to have discrepant classifications on the 2 surveys, both of which were administered in English. When we examined κ among the 253 respondents speaking only English at home, it increased to 0.82.
The results clearly demonstrate that it is possible to shorten the QuICCC with only minimal loss of capacity to identify children. It is not surprising that the shorter QuICCC-R identifies most children, as procedures used in the current study in many ways reproduce those used in developing the original QuICCC.4 Most of the children were initially identified with just a few items. However, on discovering children who also seemed to fit the definition based on clinical application of the conceptual definition, the developers of the QuICCC attempted to learn what consequences these children were experiencing that they had failed to inventory. This, in turn, led to the development of additional questions and to the full 39-item sequence in the QuICCC. It is only the full QuICCC that we believe optimizes the identification of virtually all children with infrequent consequences.
The high level of agreement between the original QuICCC and the shorter QuICCC-R is encouraging, and undoubtedly the number of children likely to be missed by the QuICCC-R is small. However, the potential clinical and policy importance of the children who might be omitted should not be overlooked, and these omissions may be critical, depending on the purpose for which the QuICCC is being applied as the mechanism of identification. For example, if positive QuICCC screening is used as a criterion for program or services eligibility, the consequences for individual children who are excluded might be quite serious. On the other hand, the loss of a few children's data from a large quality assurance survey could be expected to have little consequence for the outcomes of that evaluation.
Therefore, we believe that the purpose or goal of the identification process should dictate the choice of efficiency (short form) or completeness (long form). Our great fear, however, is that the availability of the shorter tool may at times have the effect of obliterating this choice.
Although the QuICCC was originally intended only for group identification, some investigators have suggested that the QuICCC may have applications in assessing severity of illness or in risk adjustment and capitation. These additional applications of the measure are in an exploratory phase. Nevertheless, it is quite likely that the accuracy of these potential applications of the QuICCC would be lower with a shortened version. This is because the 39-item QuICCC not only identifies children with a chronic condition, but also classifies children by the number and types of consequences that they experience (ie, none, 1, 2, or 3 of the following domains: functional limitations, compensatory dependency, or service use/need).6–8 We have previously reported that there are differences in patterns of child and parental adjustment for children with different types of consequences.7,9 This suggests that the types of consequences are important dimensions. However, our preliminary data also suggest that the QuICCC-R does not always produce the same classification for a child in terms of the number of types of consequences that the child is classified as having by the full QuICCC. The implications of this type of misclassification await additional study, both in terms of the mental health correlates of the conditions and in terms of the potential applications for risk adjustment and classification of illness severity. Thus, the briefer QuICCC-R may not serve the user who wishes to classify children into different consequence domains. Nevertheless, the QuICCC-R seems to be well suited for use in larger-scale efforts to identify groups of children with chronic conditions.
Supported in part by a Data Utilization and Enhancement Grant (MCJ-36D257) from the Maternal Child Health Bureau (Title V, Social Security Act), Health Resources and Service Administration, Department of Health and Human Services, Bethesda, MD (R.E.K.S.).
We thank the Arizona State Department of Health's Office of Children with Special Health Care Needs for sharing its data with us.
- Received September 1, 2000.
- Accepted January 8, 2001.
- Address correspondence to Ruth E. K. Stein, MD, Department of Pediatrics, Montefiore Medical Center, 111 East 210th St, Bronx, NY 10467. E-mail:
This study was presented in part at the Annual Meeting of the Pediatric Academic Societies, Boston, Massachusetts, May 13, 2000.
* The QuICCC is available from the authors.
- Perrin EC,
- Newacheck P,
- Pless IB,
- et al.
- McPherson M,
- Arange P,
- Fox H,
- et al.
- Stein REK,
- Westbrook LE,
- Bauman LJ
- Westbrook LE,
- Silver EJ,
- Stein REK
- Copyright © 2001 American Academy of Pediatrics