PEDIATRICS Vol. 105 No. 3 Supplement March 2000, pp. 697-705

Evaluation of New York State's Child Health Plus: Methods

Peter G. Szilagyi, MD, MPH^*, Laura Pollard Shone, MSW^*, Jane L. Holl, MD, MPH^¶, Lance E. Rodewald, MD^{*, #}, Jacqueline Jennings, PA^*, Jack Zwanziger, PhD, Dana B. Mukamel, PhD, Sarah Trafton, JD, Andrew W. Dick, PhD, Richard Barth^*, and Richard F. Raubertas, PhD

From the Departments of ^* Pediatrics,  Community and Preventive Medicine, and  Biostatistics, University of Rochester School of Medicine and Dentistry, Rochester, New York; ^¶ Children's Memorial Hospital, Department of Pediatrics and Institute for Health Services Research and Policy Studies, Northwestern University School of Medicine, Chicago, Illinois; and ^# National Immunization Program, Centers for Disease Control and Prevention, Atlanta, Georgia.

	ABSTRACT

Top Abstract References

Background.  The State Children's Health Insurance Program (SCHIP) is the largest public investment in child health care in 30 years, targeting 11 million uninsured children, yet little is known about the impact of health insurance on uninsured children. In 1991 New York State implemented Child Health Plus (CHPlus), a health insurance program that was a prototype for SCHIP. A study was designed to measure the association between CHPlus and access to care, utilization of services, and quality of care.

Methods.  The setting was a 6-county region in upstate New York (population 1 million) around and including the city of Rochester. A before-and-during design was used to compare children's health care for the year before they enrolled in CHPlus versus the first year during CHPlus, for 1828 children (ages 0-6.99 years at enrollment) who enrolled between November 1, 1991 and August 1, 1993. An additional study involved 187 children 2 to 12.99 years old who had asthma. Parents were interviewed to assess demographic characteristics, sources of health care, experience with CHPlus, and impact of CHPlus on their children's quality of care and health status. Medical charts were reviewed to measure utilization and quality of care, for 1730 children 0 to 6.99 years and 169 children who had asthma. Charts were reviewed at all primary care offices and at the 12 emergency departments and 6 public health department clinics in the region. CHPlus claims files were analyzed to determine costs during CHPlus and to impute costs before CHPlus from utilization data.

Analyses.  Logistic regression and Poisson regression were used to compare the means of dependent measures with and without CHPlus coverage, while controlling for age, prior insurance type, and gap in insurance coverage before CHPlus.

Conclusions.  This study developed and implemented methods to evaluate the association between enrollment in a health insurance program and children's health care. These methods may also be useful for evaluations of SCHIP.  Key words:  Child Health Plus, uninsured, underinsured, health insurance, SCHIP, asthma, utilization, quality of care, health status.

Nearly 14% of children in the United States (between 9 and 11 million children) are uninsured.^1-4 Even more children are underinsuredthat is, residing in a household that spends >10% of income on health care^5,6 or insured by indemnity plans that do not cover primary or preventive care.^7,8 Studies have documented that uninsured children are more likely than insured children to: 1) lack access to health care^9-13; 2) utilize fewer needed health services,^10,1113-18; 3) have poorer quality of care^19,20; 4) have worse reported health status^11,14,21; and in some cases 5) have potentially avoidable hospitalizations.²²

These studies compared uninsured children with insured children. The implication is that providing health insurance to uninsured children would result in outcomes equivalent to those of insured children. However, uninsured and insured children vary widely in important characteristicssome have been measured (eg, income, race, employment),^3,8,11 and some are difficult to measure (eg, intrinsic propensity to seek health care). Therefore, it may not be appropriate to generalize the experience of insured children with those who are uninsured in an attempt to estimate the impact of health insurance for uninsured children.

Despite considerable evidence that uninsured children fare poorly, few studies have evaluated the benefit of providing health insurance to previously uninsured or underinsured children.^23,24 Studies evaluating the impact of Medicaid expansion programs or health insurance for low-income families have focused largely on obstetric care.^25-27 One recent pediatric study found limited but improved access to care among children enrolled in a Medicaid expansion program.²⁸

The goal of this study was to measure the effects of providing children with health insurance on the utilization and costs of health care, the quality of care received, and health outcomes for children. The study utilized a natural experimentthe voluntary enrollment of uninsured and underinsured children into a statewide health insurance planto examine changes in children's health care after their enrollment. This article describes the study design and methods.

This study evaluated a population of children similar to that targeted by the children's health insurance legislation (the State Children's Health Insurance Program [SCHIP]) signed by President Clinton in August 1997.²⁹ SCHIP is designed to provide health insurance for children <19 years old whose families have incomes below 200% of the federal poverty level, who cannot afford private health insurance, but who do not qualify for Medicaid.²⁹ The law allocates $24 billion over 5 years (or $40 billion over 10 years), mostly as block grants to states. A major policy question is the impact of SCHIP on children's health and health care. Because the current study involved a similar population of children and a similar health insurance program to SCHIP, its findings may be predictive of the effects of SCHIP.

	CHILD HEALTH PLUS (CHPLUS) IN NEW YORK STATE

In 1991, New York State introduced Child Health Plus (CHPlus), a health insurance plan for children in low-income families who were not eligible for Medicaid.³⁰ Covered services included: preventive and illness visits, screening tests, immunizations, specialty care, emergency department (ED) care, ambulatory surgery, referrals to specialists, and prescription drugs. Hospitalization services were not covered. At the time of this study, 14 insurance plans administered CHPlus in the state (3 indemnity, 3 Independent Practice Association (IPA)-model, and 8 staff-model managed care plans); they offered similar benefits, although copayments varied for prescriptions and ED visits. Children were eligible if they were state residents, <13 years old, not enrolled in Medicaid, and lacked coverage equivalent to that of CHPlus. Children were allowed to have dual coverage for hospitalizations (although few had such coverage). CHPlus was financed by the Statewide Bad Debt and Charity Pool and from insurance premiums that varied by family income. Families whose gross incomes fell below 160% of poverty level (86% of enrollees) were eligible for full premium subsidy; those whose incomes were between 160% and 222% (13% of enrollees) paid $25 per child up to a family maximum of $100 per year. A few families whose incomes were above 222% of the poverty level purchased CHPlus for $498 to $798 per child per year, depending on the insurance plan. Some families (8%) purchased hospitalization coverage from other plans. Statewide enrollment rose from 10 000 in 1992, to 40 000 in 1993, to 67 000 in 1994, to 90 000 in 1995, to 100 000 in 1996, to 150 000 in 1997, and to >200 000 in 1998.³¹

	STUDY SETTING

The geographic setting for this study was the 6-county region in upstate New York served by Blue Cross/Blue Shield of the Greater Rochester Area. In this region, CHPlus was administered by a single managed care organization, a freestanding IPA under a contractual agreement with Blue Cross/Blue Shield, that served 20 000 Medicaid managed care recipients and nearly 6000 CHPlus recipients at the time of the study. The practitioner panel, including 1700 primary and specialty care practitioners, was essentially the panel for the dominant IPA in the area.

This geographic region has 1 urban/suburban county (Monroe County, including the city of Rochester), designated as a Standard Metropolitan Statistical Area (SMSA) county,³² and 5 contiguous rural (non-SMSA³²) counties. The total population is >1 million, including 288 000 children <19 years old; 69% of children reside in Monroe County. Many of the social and demographic characteristics of this region are similar to those of the rest of New York State³³ (Table 1), although New York City has a larger proportion of impoverished children. The proportion of children in this region who were uninsured was estimated to be 4% to 6% at the time of the study,^33,34 lower than the 12% estimated for New York State.^34,35

The health care settings for this study included 164 primary care practices (>400 primary care practitioners), the 12 EDs, and the 6 public health department clinics in the region.

	OBJECTIVES

The objectives of the study were to describe demographic and health characteristics of children enrolled in CHPlus, and examine the association between CHPlus health insurance and utilization of health services (primary care, specialty, emergency department, hospital), access to care, quality of care, health status, health care and health status for children who have asthma, and costs of health care. The study was designed to assess changes in health care for children after their enrollment in CHPlus. The study did not attempt to estimate the impact of CHPlus on all uninsured children or on the entire population eligible for CHPlus. Voluntary enrollment into CHPlus resulted in only a portion of eligible children enrolling, just as voluntary enrollment in SCHIP will lead to only a portion of eligible children enrolling in SCHIP.³⁶ While precise figures for the 6-county region were not available, we estimated that about 50% of children eligible for CHPlus in upstate New York were actually enrolled.^34,35

	STUDY DESIGN AND OVERVIEW OF METHODS

To estimate changes associated with enrolling in CHPlus, detailed information was needed for time periods both before and during enrollment in CHPlus. A before-and-during study design was used to obtain information about a 24-month period for each childthe year immediately before enrollment in CHPlus and the first year during enrollment in CHPlus. The study compared the data between the 2 years.

To perform a comprehensive evaluation of CHPlus, multiple data sources were employed, including parent interviews, medical chart reviews at primary care offices, medical chart reviews at EDs and public health clinics, and CHPlus claims files (available only for the period during CHPlus). These sources are described below.

Subject Selection

Children enrolled in the 6-county area were the target population. Subjects were eligible if they met the following selection criteria: 1) initial CHPlus enrollment between November 1, 1991 and August 1, 1993; 2) minimum of 9 continuous months of CHPlus enrollment; and 3) 0 to 6.99 years old at enrollment in CHPlus or 2 to 12.99 years old if the child had asthma.

Children who were enrolled for <9 months (35% of enrollees) were excluded because the length of their enrollment was considered to be insufficient to assess the impact of CHPlus. In addition, many of the children enrolled for the 1- to 2-month presumptive period were found to be ineligible for CHPlus after verification of application documents, and subsequently they were disenrolled from CHPlus.

The main study involved children 0 to 6.99 years old. Younger children were selected because they are more vulnerable to adverse outcomes and because key outcome measures (such as preventive visits or immunizations) occur more frequently for younger children than for older children. A substudy focused on children who had asthma; this disease was used as a tracer condition for chronic medical conditions. Children who had asthma were identified during the interview. We included children who had asthma who enrolled in CHPlus between the ages of 2.0 and 12.99 years; the upper age cut-off was the maximum age for CHPlus eligibility; the lower age cut-off (2.0 years) was used because establishing the diagnosis of asthma is frequently difficult in younger children.

Patient Contact

In the greater Rochester area, enrollment in CHPlus began in September 1991. Because this study did not begin until November 1992, it was not possible to interview all parents at the time of CHPlus enrollment. The parents of children enrolled between November 1991 and October 1992 were contacted 12 months after enrollment, and the parents of all children who enrolled between November 1992 and August 1993 were contacted at enrollment and again 12 months later (the 2 interviews were identical). Most of the measures analyzed are from the interview performed 12 months after CHPlus enrollment. The interview at enrollment was used primarily to identify sources of care before CHPlus and reasons for enrolling in the insurance program and to check for accuracy of parent recall on the second interview.

Case Identification, Contact, and Consent

Demographic information on new enrollees was provided monthly to the research team by the insurer; parents were then contacted. The process of contacting, tracking, and interviewing families was ongoing, with new families added to the study each month.

An introductory packet was mailed to families of all children eligible for the study informing them of the study and requesting permission for their participation in a telephone interview and for release of medical records and insurance claims for review. In February 1993, a method of obtaining consent by telephone was added at the request of respondents to improve the response rate.

Results of Patient Contact

We attempted to contact 3203 families, representing 3754 children 0 to 6.99 years old (younger children) and 1769 children 7 years and over (older children). Younger children were eligible for the main study and older children were included if results of screening for asthma were positive. Contact was successful for 2606 (81% of 3203) families; representing 3136 (84% of 3754) younger children, and 1472 (83% of 1769) older children. Of those with whom contact was successful, 188 (6% of 3136) younger children and 100 (7% of 1472) older children were ineligible because parents reported that they terminated CHPlus enrollment after <9 months. Parents of 715 (23% of 3136) younger children and 313 (21% of 1472) older children refused to participate. Parents of 2232 (71% of 3136) younger children and 1058 (72% of 1472) older children agreed to participate in the study.

Parent interviews 12 months after enrollment in CHPlus were completed for 1828 younger children. Screening interviews for 1058 older children yielded 82 older children who had asthma. Children 2 to 7 years old were also screened for asthma, yielding 97 younger children who had asthma. Thus, the total number of cases, combining younger and older children, was 1910. Medical chart reviews (below) were completed for 1802 (94%) cases. Results are presented for younger children in all articles in this supplement, except when otherwise noted by the author. Results for children with asthma are discussed solely in the article on asthma.³⁷

The number of subjects in key study groups are shown in Table 2: 66% of children in the study resided in Monroe County; 36% resided in the 5 rural counties. This distribution was virtually identical to the distributions of CHPlus enrollees and of the general child population in the 6 counties.

Parent Interviews

The parent interviews assessed demographic characteristics, sources of health care before and after enrollment in CHPlus, experience with CHPlus, health status, and perceived impact of CHPlus on quality of care and on health status of children. In addition, the interviews served to identify children who had asthma; these children received an extended interview. A detailed description of the questions relating to asthma is included in the accompanying article on asthma.³⁷

A protocol was followed to ensure uniform efforts to contact parents. Telephone calls were made by a staff of 6 interviewers between 9:00 AM and 9:30 PM, Sunday through Friday. At least 10 attempts were made and were distributed over all available calling hours to ensure that contact was attempted at a variety of times and days. Telephone contact was considered unsuccessful if 120 days elapsed without successful contact. The telephone directory, directory assistance, insurer records, and physician records were used to obtain updated telephone numbers. When needed, interviews were conducted in Spanish (the major second language in this region). Families with no telephone were contacted by mail.

The interview consisted of structured questions (the interview instrument is available from the authors on request). Many of the questions on access, utilization, and health status were from prior National Health Interview Surveys (NHIS) and the 1988 NHIS Child Health Supplement.¹¹ The average interview lasted 22 minutes. Table 2 shows the number of completed interviews by insurance status before CHPlus, and Table 3 shows the variables obtained by parent interview.

Medical Chart Reviews

Medical charts were reviewed at primary care offices to measure utilization and quality of care for general pediatric and for asthma care. It was assumed that parents could recall sites of care during the past 2 years, but were less accurate in documenting specific visits or services delivered. Such visit-specific data have been found to be obtained more accurately from medical charts than from interviews.^38-40

Letters were sent to all CHPlus primary care practitioners requesting their cooperation with the medical chart review. All CHPlus practitioners (400 physicians at 164 sites) agreed to participate, and all but 2 non-CHPlus physicians (who cared for 2 subjects before CHPlus) agreed to participate.

Medical chart reviews were conducted at primary care sites for the 12 months before and the first 12 months during CHPlus enrollment. Initial record searches were targeted at sites of care identified by parents. The majority of children (80%) received care from the same site both before and during CHPlus, while 20% changed physicians at the time of CHPlus enrollment. Record transfers noted in the primary care chart were tracked to facilitate review of records at prior sources of care.

A child's chart was considered complete if a medical chart was reviewed from every source of primary care listed by the parent during the interview or obtained from another medical chart. Overall, 1802 children (94%) had a complete medical chart review (Table 2). Chart reviews were incomplete for 108 of 1910 children (6%), for the following reasons: 1 physician retired and charts could not be located; 2 non-CHPlus physicians refused; 1 pre-CHPlus physician was out of the area; and in the remainder of cases the physicians identified by the family had no record of ever seeing the patient.

Ancillary Medical Chart Reviews at EDs and Public Health Clinics

The ED databases at the 12 hospitals in the region were matched by name and birthdate against the file of study participants. Medical charts were reviewed at the 12 EDs for all ED visits during the appropriate 2-year study period.

A similar process was performed at the 6 county public health department clinics, which provide immunizations to small numbers of children and do not provide comprehensive primary care. The medical records at each public health department clinic were reviewed for children residing in that particular county and for any children noted in the primary care charts to have visited a health department clinic. All health department clinics were reviewed a second time for children who had incomplete immunization records to ensure that a complete immunization profile was obtained. Altogether, ED medical records were reviewed for 577 children (947 ED visits) and health department immunization clinic records were reviewed for 179 children (199 health department visits).

Analysis of Hospitalizations

Hospitalizations were determined by both the ED chart review and by parent interview. For each ED visit, the disposition was determined and admissions from the ED were recorded. In this region, 68% of all hospitalized children are admitted via the ED (personal communication, Rochester Healthcare Information Group, June 1998). The majority of direct admissions (that do not involve the ED) are either for chronic conditions such as malignancies, or for elective surgical procedures. Most admissions for acute medical conditions are via the ED; for example, 96% of hospitalizations for childhood asthma are from the ED (personal communication, Rochester Healthcare Information Group, June 1998). Although not all hospitalizations are included by this ED chart review method, it is unlikely that there would be a systematic difference in identification of hospital admissions between the year before CHPlus enrollment and the year during CHPlus. Analysis of claims for hospitalizations was not possible because CHPlus did not include hospital coverage at the time of this study.

As a second source of information on hospitalizations, parent interviews were used to estimate whether the child was hospitalized during the year before or after enrollment in CHPlus. It was assumed that hospitalization of a child was such an unusual and sentinel event that parents would be likely to recall such an event. However, because the accuracy of such recall is unknown, we used the ED chart reviews as the major source, with parent interviews as a second source to help validate the ED chart reviews. As shown in the article on utilization,⁴¹ findings on hospitalizations from the 2 data sources were similar.

Analysis of CHPlus Claims Files

The CHPlus claims files were analyzed to determine costs during CHPlus for each of the main categories of services shown in Table 3. Insurance claims files were not available for the period before CHPlus because children were either uninsured or covered by a variety of insurance plans. Costs for the year before CHPlus were imputed from costs during CHPlus using utilization data (see accompanying article⁴²).

Measures

Table 3 shows the key study measures and the methods used to determine each measure.

Demographic Characteristics Family composition, race, ethnicity, income, parent education, and parental employment status were determined by interview.

Prior Health Insurance Coverage and Parent Experience With CHPlus Prior health insurance coverage (most recent before CHPlus) was categorized as uninsured, underinsured, Medicaid, or fully insured. Underinsured included insurance plans that did not cover most preventive services (eg, traditional indemnity plans). For certain services such as immunizations for which uninsured and underinsured states were identical, these 2 groups were combined.⁴³ Fully insured included insurance plans that fully covered most preventive services (eg, health maintenance organizations). The gap in insurance coverage immediately before CHPlus was categorized into 3 groups: no gap (ie, insured right up until enrollment in CHPlus), 1- to 5-month gap, and 6+-month gap. Parents' experience and satisfaction with CHPlus were assessed, as well as additional services that were desired but were not included in the CHPlus benefit package.

Health Status (Overall and Asthma-Specific) The interviewers assessed parents' perceptions of the child's overall health status and change in health status over 2 years. For children who had asthma, questions about asthma severity were asked, and an overall asthma severity score was calculated.³⁷

Access to Care The parent interview determined the presence of a routine source for preventive and sick care (a medical home) and any switching of primary care practitioners attributable to CHPlus. Measures of accessibility included the presence of 24-hour provider coverage, travel time in minutes to the primary care provider, and whether the preventive care and sick care sites were the same (a measure of continuity).

Utilization of Health Services Medical chart reviews from primary care offices were used to determine the number and type of primary care visits. Chart reviews at the 12 EDs were used to measure the number of ED visits and to classify the ED visits as discretionary or nondiscretionary. The ED visit was classified as nondiscretionary if the child was admitted to a hospital or if a procedure or laboratory study was performed; other ED visits were classified as discretionary. For each ED visit, a measure of continuity of care was obtained by determining whether a visit to the child's primary care provider was made within 1 month of the ED visit. Utilization of specialty care was assessed by parent interviews because primary care charts frequently lacked such information. The number of specialists seen was determined (rather than the number of visits to specialists), and the types of specialists were noted. Hospital utilization was determined from ED chart reviews and parent interviews as described above.

Quality of Care Parents were asked to rate the overall quality of medical care before and during CHPlus and to assess the impact of CHPlus on the quality of care their child received. For children who had asthma, parents were asked to assess the impact of CHPlus on quality of their child's asthma care, and on asthma severity.³⁷

Costs Costs were determined from the CHPlus claims files for the key categories of services and imputed for the year before CHPlus based on utilization.⁴² Costs included the payment by the insurer to providers, copayment by the families, and (whenever applicable) payments made by other insurers. All costs were converted to 1994 dollars by using the medical care component of the Consumer Price Index for upstate New York.

Statistical Methods

The goal of the analyses was to estimate changes in health care utilization, quality of care, and health status associated with enrollment in CHPlus. Outcome measures were analyzed using logistic regression for dichotomous outcomes and Poisson regression for outcomes that were counts (eg, number of primary care visits during a study period). Each child in the study typically contributed 2 observations to the analysis, 1 for the year before enrollment in CHPlus (pre-CHPlus period) and 1 for the year after enrollment (CHPlus period). To allow for possible correlations between observations for the same child and among observations for children from the same family, the regression models were estimated using generalized estimating equations.⁴⁹ An exchangeable working correlation structure was used for observations from the same family. Tests of statistical significance were based on standard errors calculated from an empirical (sandwich) variance estimator. The explanatory variables in the regression models were: type of period covered by the observation (pre-CHPlus or CHPlus); age (in years) at the start of the period; type of health insurance coverage before CHPlus (4 levels, described above); gap in insurance coverage immediately before CHPlus enrollment (3 levels, described above); and interactions of period type with each of the other factors. The interactions were included to test whether changes associated with CHPlus differed across the levels of the other factors.

In most analyses, age was treated as a categorical variable, with 1 level per year of life. This avoided the assumption of a linear relationship between age and the outcome variable. However, in the analyses of data for children who had asthma this approach was not feasible because the sample size was smaller (187 children) and the range of ages was wider (2-12 years). Therefore, age was represented by linear and quadratic terms in regression models for asthma-related outcomes. The regression models were used to estimate means of the outcome measures with and without CHPlus while controlling for the other factors in the models. The means were standardized to the distribution of age and insurance history for the children 1 to 5.99 years old in our dataset, using the models. These distributions are summarized in Table 4. The difference between means with and without CHPlus was the estimate of the CHPlus effect for that variable. Means were calculated in the units of the outcome variable so that they are interpretable, but tests of the statistical significance of the CHPlus effect were conducted in the units of the underlying regression model (logits for logistic regression, logs for Poisson regression).

For some dichotomous outcomes, nearly all observations had the same value (eg, few children were hospitalized during the study interval, and nearly all had a source of primary care). This led to convergence problems for the logistic regression models. In these cases the models were simplified by leaving out interaction terms. When this was not sufficient to allow convergence, the data were analyzed by McNemar's test (with exact, small-sample P values), treating the pre-CHPlus and CHPlus observations for each child as paired data, and unstandardized means are reported. These cases are noted in the results of the relevant articles.

Insurance history was not applicable as an explanatory variable for children who enrolled in CHPlus as infants (age at enrollment equal to 0). Therefore, data for infants were analyzed separately. The regression models had period type (pre-CHPlus or CHPlus) as the only explanatory variable; they essentially compared children whose CHPlus period started in infancy to children whose pre-CHPlus period started in infancy. Generalized estimating equations were again used to fit the models, using an exchangeable working correlation structure for observations for children from the same family.

Methodologic Limitations and Sources of Bias

This was an observational study, and the estimated CHPlus effects (differences between CHPlus and pre-CHPlus periods) may have been influenced by factors that could not be controlled for in the analyses. Factors with the potential to affect estimated CHPlus effects include secular trends in outcome measures, nonresponse bias, and parental recall errors.

Because CHPlus periods in the analyses were on average 1 year later than pre-CHPlus periods, estimated CHPlus effects were partially confounded with any secular trends in outcome measures. However, there are arguments that such trends did not affect the findings. First, the range of CHPlus enrollment dates included in this study was <2 years, which minimizes the potential for major secular changes in outcome measures. Second, annual reports on CHPlus published by the New York State Department of Health³¹ do not show substantial changes in demographic characteristics or in prior insurance experience of children who enroll, suggesting that the study period was not unusual.

Because we wanted to minimize the potential confounding of CHPlus effects with secular trends, only a 1-year period after CHPlus enrollment for each child was studied. Thus, we were not able to evaluate longer-term effects of CHPlus.

Interviews were completed for 62% of the eligible children for whom a telephone or written contact was successful. The interview response rate was similar to or better than the rate in many large-scale surveys of difficult-to-reach populations^2850-52 or insurance plan enrollees.^53,54 Furthermore, young children (0-6.99 years) who had completed interviews (N = 1828) were similar (P > .05) to the eligible CHPlus population in the region in terms of county of residence, gender, and age distribution within study criteria. These were the demographic characteristics for which information was available about the entire eligible CHPlus population from intake interviews at enrollment. Medical chart reviews were completed for 95% of younger children (1730 of 1828 children) and for 90% of the children who had asthma (169 out of 187) who had completed parent interviews; therefore, missing or incomplete chart reviews are unlikely to be a significant source of bias in this study.

Interview procedures required parents to recall their child's health status and health care events up to 2 years into the past. The interviewers attempted to minimize recall errors by using cues to help parents recall dates or child ages. The longer recall for the pre-CHPlus year may have led to reduced reliability of data for that period, and some events or services during the pre-CHPlus year may have been attributed to the CHPlus year, and vice versa. The extent and direction of any recall bias are unknown.

Altogether, parents of 882 subjects were interviewed both at enrollment and 12 months after enrollment in CHPlus. A comparison of responses on the 2 interviews to questions about the pre-CHPlus period allowed estimation of their reliability; agreement was measured by the  statistic. The  statistic ranged from .85 for race, .77 for Hispanic ethnicity, .66 for prior insurance, .53 for gap in insurance coverage, .45 for use of emergency care, and .64 for hospitalization, down to .37 for use of preventive care, .30 for sick care, and .19 for specialty care. The latter 3 items indicate poor reliability.

Specialty utilization had to be obtained by parent interview because specialty visits are often not noted in primary care office charts and because it was possible that chart recordings of specialty visits may have varied according to the child's health insurance status. All of the other utilization measures were obtained by medical chart reviews, rather than relying on parent recall. Although medical charts may contain missing or even incorrect information, it is unlikely that data in medical charts were biased toward or against showing a CHPlus effect. It is possible, however, that if a child received care from multiple sources, the sources used during the pre-CHPlus year were less likely to be remembered by parents than sources used during the CHPlus year. This could lead to underestimation of health care utilization during pre-CHPlus periods, and a larger estimated CHPlus effect.

Despite the large size of this study, the sample size was limited for children who had asthma, for specific subgroups (such as children eligible for screening), and for rare outcomes (such as hospitalizations). Because all available subjects were included in this study, it was not feasible to increase the sample size within this 6-county region.

	DISCUSSIONMETHODOLOGIC ISSUES IN EVALUATING SCHIP

Because little is known about the impact of providing health insurance to uninsured or underinsured children and because SCHIP will likely provide coverage to several million children who currently lack health insurance,^36,55 it is critical to measure the impact of SCHIP. The strengths and limitations of this study can provide guidance for the design of future local, state, or national evaluations of SCHIP, and for studies of other health insurance programs.

Any evaluation of the impact of a health insurance program such as CHPlus must take into account the baseline health status and health care utilization of enrollees. Unfortunately, it is difficult to assess accurately the baseline characteristics of enrolled populations because of lack of data. When a population is largely uninsured or underinsured at baseline, insurance claims are not helpful. Thus, information must be obtained through interviews or medical record reviews, both of which have limitations and are costly. Although interviews are perhaps simpler to perform, they have a limited ability to measure utilization and quality of care accurately. For these reasons, we included both interviews and chart reviews, as well as claims analyses, in this study.

Because SCHIP will be implemented in natural experiment settings, just as in the case of CHPlus, it will be a challenge to differentiate the impact of SCHIP from other factors such as secular trends in outcome measures. Studies that use a before-after design, particularly longer-term studies, will need to address this issue. In our study, we selected a short study period to minimize the effect of potential secular trends; however, this limited our ability to assess the long-term effects of CHPlus. In other settings, it will be useful to include comparison groups,²³ particularly if secular trends in the comparison groups can be assessed.

The goal of this study was to evaluate the association between health insurance and health care for children who voluntarily enrolled in CHPlus. This study cannot address the potential impact of health insurance on the entire population of uninsured or on all underinsured children eligible for CHPlus. Because of self-selection bias, children who enroll may have different baseline characteristics from those who do not enroll, and therefore the impact of health insurance may be different for those who enroll than it would be for those who do not enroll. Studies of SCHIP enrollees will also have this limitation because SCHIP is voluntary and because it is estimated that <50% of eligible children will enroll in SCHIP,³⁶ similar to the proportion of eligible children enrolled in CHPlus.³⁴ As discussed in an accompanying article,³⁵ a higher proportion of children in this study had a medical home than did uninsured children across the United States.¹⁸ To the extent that having a medical home is one of the critical means by which health insurance affects health care, the results found in this study may underestimate the impact of health insurance programs like SCHIP on uninsured children. In areas in which a lower proportion of SCHIP enrollees have a medical home before enrollment, it is possible that the impact of SCHIP will be even larger than the estimated effects of CHPlus noted in this study.

Although the ultimate goal of health insurance is to improve health status, it is well known that child health status is greatly affected by factors unrelated to health insurance.⁵⁶ Thus, it is more difficult to demonstrate improved health status attributable to the provision of health insurance than to demonstrate improved utilization or quality of care. Because other studies have shown that improved receipt of health care leads to better outcomes for children,⁵⁷ (Torres NP, Wolf WJ. Improving the health of uninsured children: quality of care in the States' Children's Health Insurance Proposals (CHIP). 1998. Unpublished paper) evaluations of SCHIP should focus on the receipt of care (utilization and quality of care) as critical determinants of the success of SCHIP programs, rather than focusing primarily on the health status of children.

In addition to measuring the impact of SCHIP on the enrolled population, it will also be important to measure its impact on the entire population of uninsured or eligible children.¹⁸ However, as already noted, findings from such population-based studies will be affected by the proportion of eligible children who enroll, the degree of self-selection bias for enrolled children, and the impact of SCHIP on the children who enroll. Different study methodologies will be needed to address each of these issues, with the current study methodology designed to address the association between health insurance and outcomes for children who enroll.

The impact of SCHIP may depend on the type of SCHIP program. Three general types of SCHIP programs are being developed by individual states⁵⁵: Medicaid expansion programs, state programs (like CHPlus) based on commercial insurance plans,⁵⁵ (Torres NP, Wolf WJ. Improving the health of uninsured children: quality of care in the States' Children's Health Insurance Proposals (CHIP). 1998. Unpublished paper) and combinations of these 2 approaches. At least half of the states are developing programs similar to CHPlus. These natural experiments will provide a unique opportunity to understand the impact of SCHIP on the health care of enrolled children.

	ACKNOWLEDGMENTS

This study was supported by Grant 92-5391 from The David and Lucile Packard Foundation, the Monroe Plan for Medical Care, and the Daisy Marquis Jones Foundation.

The study was approved by the Research Subjects Review Board of the University of Rochester and by Blue Cross/Blue Shield of the Greater Rochester Area.

We appreciate the many individuals who contributed to the data collection and data management.

	FOOTNOTES

Received for publication Oct 25, 1999; accepted Dec 6, 1999.

Address correspondence to Peter G. Szilagyi, MD, MPH, Department of Pediatrics, University of Rochester School of Medicine, Box 632, Strong Memorial Hospital, 601 Elmwood Ave, Rochester, NY 14642. E-mail: peter_szilagyi{at}urmc.rochester.edu

	ABBREVIATIONS

SCHIP, State Children's Health Insurance Program; CHPlus, Child Health Plus; ED, emergency department; IPA, Independent Practice Association; SMSA, Standard Metropolitan Statistical Area; NHIS, National Health Interview Survey..

	REFERENCES

Top Abstract References

1. Lewit EM, Baker LS Health insurance coverage. Future Child. 1995; 5:192-204 [Medline]
2. Fronstein P Children without health insurance coverage: an analysis of the increase in uninsured children between 1992 and 1993 . Inquiry. 1995; 32:353-359
3. US Bureau of the Census. Current Population Survey. Washington DC: US Bureau of the Census; 1994
4. US General Accounting Office. Health Insurance for Children. Many Remain Uninsured Despite Medicaid Expansion. Washington, DC: US General Accounting Office; 1995. Publication No. GAO/HEHS-95-175
5. Pepper Commission. Bipartisan Commission on Comprehensive Health Care: A Call for Action. Washington, DC: Government Printing Office; 1990. No. 101-114
6. Short PF, Banthin JS New estimates of the underinsured younger than 65 years. JAMA 1995; 274:1302-1306 [Medline]
7. Farley PJ Who are the uninsured? Milbank Memorial Fund Q Health Soc 1985; 63:476-503
8. Monheit AC, Cunningham PJ. Children Without Health Insurance. Washington, DC: US Department of Health and Human Services; 1992. Publication No. 93-0025
9. Cunningham PJ, Hahn BA The changing American family: implications for children's health insurance coverage and the use of ambulatory care services. Future Child. 1994; 4:24-42 [Medline]
10. Kogan MD, Alexander GR, Teitelbaum MA, Jack BW, Kotelchuck M, Pappas G The effects of gaps in health insurance on continuity of a regular source of care among preschool-aged children in the United States. JAMA. 1995; 274:1429-1435 [Medline]
11. Holl JL, Szilagyi PG, Rodewald LE, Byrd RS, Weitzman ML Profile of uninsured children in the United States. Arch Pediatr Adolesc Med. 1995; 149:398-406 [Medline]
12. Weissman JS, Stern R, Fielding SL, Epstein AM Delayed access to health care: risk factors, reasons, and consequences. Ann Intern Med. 1991; 114:325-331 [Medline]
13. Newacheck PW, Stoddard JJ, Hughes DC, Pearl M. Children's access to health care: the role of social and economic factors. In: Stein REK, ed. Health Care for Children. New York, NY: United Hospital Fund; 1997
14. Patrick DL, Madden CW, Diehr P, Martin DP, Cheadle A, Skillman SM Health status and use of services among families with and without health insurance. Med Care. 1992; 30:941-949 [Medline]
15. Stoddard JJ, St Peter RF, Newacheck PW Health insurance status and ambulatory care for children. N Engl J Med. 1994; 330:1421-1425 [Abstract/Full Text]
16. Himmelstein DU, Woolhandler S Care denied: US residents who are unable to obtain needed medical services. Am J Public Health. 1995; 85:341-344 [Abstract]
17. Margolis LH, Kotelchuck M, Chang HY Factors associated with early maternal postpartum discharge from the hospital. Arch Pediatr Adolesc Med. 1997; 151:466-472 [Medline]
18. Newacheck PW, Stoddard JJ, Hughes DC, Pearl M Health insurance and access to primary care for children. N Engl J Med. 1998; 338:513-519 [Abstract/Full Text]
19. Rodewald LE, Shiuh T, Zill E, Dietz V, Szilagyi PG Health insurance and underimmunization: lessons from the 1991 National Health Interview Survey. Pediatr Res. 1995; 37:144
20. Zimmerman R, Janoski J Immunization barriers in Minnesota private practices: the influence of economics and training in vaccine timing. Fam Pract Res J. 1993; 13:213-214 [Medline]
21. Braveman P, Oliva G, Grisham M, Reiter R, Egerter S Adverse outcomes and lack of health insurance among newborns in an eight-county area of California, 1982 to 1986. N Engl J Med. 1989; 321:508-513 [Abstract]
22. Weissman JS, Gatsonis C, Epstein AM Rates of avoidable hospitalization by insurance status in Massachusetts and Maryland. JAMA. 1992; 268:2388-2394 [Medline]
23. Lave JR, Keane CR, Lin CJ, Ricci EM, Amersbach G, LaValle CP Impact of a children's health insurance program on newly enrolled children. JAMA. 1998; 279:1820-1825 [Medline]
24. Martin DP, Diehr P, Cheadle A, Madden CW, Patrick DL, Skillman SM Health care utilization for the "newly insured": results from the Washington Basic Health Plan. Inquiry. 1997; 34:129-142 [Medline]
25. Haas JS, Udvarhelyi S, Epstein AM The effect of health coverage for uninsured pregnant women on maternal health and the use of cesarean section. JAMA. 1993; 270:61-64 [Medline]
26. Hill IT The role of Medicaid and other government programs in providing medical care for children and pregnant women. Future Child. 1992; 2:134-153
27. Piper J, Ray W, Griffin M Effects of medical eligibility expansion on prenatal care and pregnancy outcome in Tennessee. JAMA. 1990; 264:2219-2223 [Medline]
28. Rosenbach ML, Irvin C, Coulam RF Access for low-income children: is health insurance enough? Pediatrics 1999; 103:1167-1174 [Abstract/Full Text]
29. Balanced Budget Act of 1997, Public Law 105-33, Subtitle JState Children's Health Insurance Program
30. New York Public Health Law Article 25, Title I-A, Sections 2510-2511, Child Health Insurance Plan
31. Child Health Plus Health Plan for Kids. Annual Reports (1992, 1993, 1994, 1995, 1996, 1997, 1998). New York State Department of Health, and www.state.ny.renewdoc/final/healthcare-46.htm
32. New York Department of Labor. New York State Standard Metropolitan Statistical Areas (New Area Definitions). New York, NY: New York Department of Labor; 1997
33. New York State Department of Health, Bureau of Child and Adolescent Health. Maternal, Child and Adolescent Health Profile1991. New York, NY: New York State Department of Health, Bureau of Child and Adolescent Health; 1994
34. Szilagyi PG, Zwanziger J, Rodewald LE, et al. Evaluation of Child Health Plus in New York State. Final Report to the New York State Department of Health. Albany, NY: Department of Pediatrics, University of Rochester; 1996
35. Holl JL, Dick A, Shone LP, A profile of the population enrolled in New York State's Child Health Plus. Pediatrics. 2000; 105:706-710 [Medline]
36. Congressional Budget Office. Budgetary Implications of the Balanced Budget Act of 1997: Table 11. Washington, DC: Congressional Budget Office; 1997
37. Szilagyi, PG, Holl JL, Rodewald LE, et al Evaluation of New York State's Child Health Plus: children who have asthma. Pediatrics. 2000; 105:719-727 [Medline]
38. Long T, Rogers KD A limitation in the use of office records for health care evaluation. Pediatrics 1975; 56:295-296 [Medline]
39. Brown JB, Adams ME Patients as reliable reporters of medical care process. Recall of ambulatory encounter events. Med Care 1992; 30:400-411 [Medline]
40. Goldstein KP, Kviz FJ, Daum RS Accuracy of immunization histories provided by adults accompanying preschool children to a pediatric emergency department. JAMA 1993; 270:2190-2194 [Medline]
41. Holl JL, Szilagyi PG, Rodewald LE, Evaluation of New York State's Child Health Plus: access, utilization, quality of health care, and health status. Pediatrics. 2000; 105:711-718 [Medline]
42. Zwanziger J, Mukamel DB, Szilagyi PG, Evaluation of New York State's Child Health Plus: how much does providing health insurance to uninsured children increase health care costs? Pediatrics. 2000; 105:728-732 [Medline]
43. Rodewald LE, Szilagyi PG, Holl J, Zwanziger J, Mukamel D, Dick A, et al Health insurance for lower income working families: effect on the delivery of immunizations to preschool children Arch Pediatr Adolesc Med. 1997; 151:798-803 [Medline]
44. American Academy of Pediatrics. Guidelines for Health Supervision II. Elk Grove Village, IL: American Academy of Pediatrics; 1988
45. American Academy of Pediatrics, Committee on Infectious Diseases. Red Book: Report of the Committee on Infectious Diseases. Elk Grove Village, IL: American Academy of Pediatrics; 1994
46. US Department of Health and Human Services. Healthy People 2000. National Health Promotion and Disease Objectives. Washington, DC: US Department of Health and Human Services; 1990
47. Campbell JR, Schaffer SJ, Szilagyi PG, O'Connor KG, Briss P, Weitzman M Blood lead screening practices among US pediatricians. Pediatrics. 1996; 98:372-377 [Abstract]
48. National Asthma Education Program. Executive Summary: Guidelines for the Diagnosis and Management of Asthma. Bethesda, MD: National Institutes of Health; 1991. NIH Publication No. 91-3042A
49. Liang KY, Zeger SL Longitudinal data analysis using generalized linear models. Biometrika 1986; 73:13-22
50. Sisk JE, Gorman SA, Reisinger AL, Glied SA, DuMouchel WH, Hynes MM Evaluation of Medicaid managed care. Satisfaction, access, and use. JAMA 1996; 276:50-55 [Medline]
51. Gold M, Wooldridge J. Plan-Based Surveys of Satisfaction With Access and Quality of Care: Review and Critique. Washington, DC: US Agency for Health Care Policy and Research; 1994:1-32
52. Weiss B, Senf J Patient satisfaction survey instrument for use in health maintenance organizations. Med Care 1990; 28:434-445 [Medline]
53. Davis K, Collins KS, Schoen C, Morris C Choice matters: enrollees' views of their health plans. Health Aff (Millwood) 1995; 14:99-112 [Medline]
54. Misha SI, Dooley D, Catalano R, Serxner S Telephone health surveys: potential bias from non-completion. Am J Public Health 1993; 83:94-99 [Abstract]
55. American Academy of Pediatrics, Division of State Government and Chapter Affairs. Comparison of Major State Options for State Children's Health Insurance Program. Elk Grove Village, IL: American Academy of Pediatrics; 1998
56. Szilagyi PG, Schor EL The health of children. Health Serv Res 1998; 33:1001-1039 [Medline]
57. Starfield B. The Effectiveness of Medical Care: Validating Clinical Wisdom. Baltimore, MD: Johns Hopkins University Press; 1985