Assessing the Impact of Pediatric-Based Developmental Services on Infants, Families, and Clinicians: Challenges to Evaluating the Healthy Steps Program
- Bernard Guyer, MD, MPH*,
- Nancy Hughart, RN, MPH*,
- Donna Strobino, PhD*,
- Alison Jones, PhD‡,
- Daniel Scharfstein, ScD§,
- the Healthy Steps Evaluation Team∥
Background. Begun in 1996, the Healthy Steps for Young Children Program (HS) is a new model of pediatric practice that incorporates child development specialists and enhanced developmental services for families of young children. HS is for all families, not just those at high-risk. It is expected to strengthen parents' knowledge, attitudes, and behaviors in ways that promote child health and development, and in turn, to lead to improved child outcomes, such as improved language development, increased utilization of well child care, and decreased problem behaviors, hospitalizations, and injuries. The HS evaluation is designed to assess whether HS is successful in achieving the desired outcomes, measure the program's costs, and determine the relation of the program's costs to its outcomes.
Objective. This article is the first report of the HS evaluation. It describes the evaluation design and characteristics of the HS sites and sample for the evaluation.
Methods. The evaluation is following a cohort of children from birth to age 3 at 15 evaluation sites across the country. The sites represent a range of organizational practice settings that include group practices, hospital-based clinics, and health maintenance organization pediatric clinics. The evaluation design relies on 2 comparison strategies. At 6 randomization design sites, 400 children were randomized to the intervention or control group. At 9 quasi-experimental design sites, a comparison location with a similar organizational setting and patient profile has been selected and up to 200 children are being followed at each of these sites.
At each site, 2 developmental specialists (or their full-time equivalents) work as a team with 4 to 8 pediatricians and pediatric nurse practitioners. The specialist conducts office visits (jointly or sequentially with the pediatric clinician) and home visits, assesses children's developmental progress, provides referrals and follow-up to resources in the community, organizes and conducts parent discussion groups, coordinates early reading activities, and maintains a telephone information line for questions about child development and behavior.
The evaluation relies on many data sources including self-administered provider surveys, key informant interviews, forms completed by parents at office visits, telephone interviews with parents, medical record reviews, data from each site on program costs and health services use, and an ongoing log of family contacts maintained by each developmental specialist.
Analyses for this article are based on enrollment data for the Healthy Steps sample and national data on 1997 US live births. The χ2 goodness-of-fit test was used to evaluate whether the distribution of selected demographic variables, insurance, and infant's birth weight for the Healthy Steps sample was similar to the distributions for US births in 1997. In addition, comparisons were made between intervention and comparison families at the randomization and quasi-experimental evaluation sites. The χ2 test of independence was used to evaluate differences in variables across groups.
Results. Throughout a 26-month period, 5565 children enrolled in the evaluation, 2963 (53.2%) children in the intervention group and 2602 (46.8%) in the comparison group. More than 10% of mothers in the Healthy Steps sample are teenagers; 18% have 11 years of education or less; 27% have completed college; 18% are black or African-American; slightly >20% are of Hispanic origin; 36% are single; and close to one-third used Medicaid for their prenatal care. Approximately 7% of infants were low birth weight.
When compared with national birth data for the United States as a whole, the Healthy Steps sample seems similarly diverse. However, with the exception of maternal age, the distribution of variables was significantly different from the distribution for US births.
There are no differences between intervention and comparison families at randomization sites on any of the maternal characteristics, insurance status, or infant's birth weight. However, there are a number of differences between the intervention and the comparison groups at the quasi-experimental sites and between the randomization and quasi-experimental sites. At quasi-experimental sites, mothers in the comparison group were more likely than were mothers in the intervention group to be 29 years old or younger, to have fewer years of education, to be black or African-American, to report Hispanic origin, and to be single. Mothers at the quasi-experimental sites were slightly older than were mothers at randomization sites. They also were more likely to be married, to have <11 years of education (but also to be college graduates), to be of Hispanic origin, and to report private insurance or self-pay rather than Medicaid as their source of payment for prenatal care.
Conclusion. The Healthy Steps sample is economically and ethnically diverse, reflecting the diversity of the nation as a whole. There seem to be differences between randomization and quasi-experimental sites as well as between intervention and comparison groups at the quasi-experimental design sites. As important, randomization at the sites that selected to use this approach seems to have been effective in equalizing the characteristics of families in the intervention and comparison groups.
The differences between the intervention and comparison families at the quasi-experimental sites indicate the need to take account of these differences in analyzing program effects. In addition, to account for within-site correlation of outcomes, between-site variability of the effects of HS, and between-provider type variability of the effects of HS, random effects models will be used in the data analyses. These models are also referred to as hierarchical linear models or multilevel models.
The HS evaluation is carefully designed to address the complexities of a program with multiple objectives, multiple components, and a wide range of expectations. The evaluation will provide information for practicing clinicians on the effectiveness of HS in improving care for families. It will provide policy makers with empirical evidence to inform the national debate on whether pediatric practices are an appropriate venue for helping parents promote their children's development. In addition, it will assess whether potential outcomes such as increased satisfaction with care and decreased hospitalizations make HS a valuable intervention.
- early child development
- HS =
- Healthy Steps for Young Children Program •
- HSS =
- Healthy Steps Specialist
Research on early brain development emphasizes the importance of the family environment in maximizing learning potential in early infancy.1 At the same time, the dynamics of US families are changing with increasing numbers of mothers of young children in the workforce, and families having fewer children and less experience with raising children. There is growing demand among parents for advice on their children's development. In a recent survey, 79% of parents reported that they wanted more information in at least 1 of 6 areas of child rearing, and 53% wanted more information in at least 3 areas.2 Given the routine use of pediatric care for very young children, the pediatric setting represents a convenient site to provide services for helping parents develop their children's learning potential. However, pediatricians, particularly with the pressures of managed care to see increasingly more patients, may not have adequate time or contact with parents to address developmental concerns.
The Healthy Steps for Young Children Program (HS) responds to these important realities of modern American life. The program is intended to enhance the developmental potential of young children (birth to 3 years old) and to strengthen the involvement of mothers and fathers in their children's early development. To achieve these goals, HS is testing a new model of pediatric practice that emphasizes the child's health as a whole, including behavioral, emotional, and intellectual development, as well as physical well-being.3 A significant component of this model is the inclusion in pediatric care of a new health care professional trained in child development, the Healthy Steps Specialist (HSS).4 ,5
Although HS shares attributes with previous child development programs, it differs from most of these efforts. In contrast with other programs such as the Infant Health and Development Study,6 the Brookline Early Education Project,7 Early Head Start,8 and Parents-as-Teachers,9 ,10 HS focuses on all families, not just those at high-risk. HS serves families across a wide range of socioeconomic status, and its services are located in pediatric practices.
Healthy Steps most closely resembles 2 pilot programs that delivered child development services in pediatric practices. The Pediatric Pathways to Success program at Boston Medical Center, the model for HS, was designed to enhance pediatric primary care by providing child development information and parent support to high-risk and vulnerable infants and their families.11 Zero-to-Three's Developmental Specialist in Pediatric Practice Project introduced part-time child developmental specialists into 2 pediatric practices serving middle-class as well as higher-risk families.12
The scientific community often laments that clinical interventions were not submitted to rigorous, experimental evaluation at the outset and subsequently become impossible to properly assess vis-à-vis outcomes. HS provides a unique opportunity because plans for a comprehensive evaluation were integrated into the early planning of the program. This article is the first report of the HS evaluation. It describes the evaluation design and characteristics of the HS sites and sample for the evaluation.
The Healthy Steps Intervention
The Healthy Steps program was developed by a multidisciplinary team from Boston University in collaboration with professionals from the Commonwealth Fund.4 The foundations of HS can be located in Starting Points: Meeting the Needs of Our Youngest Children 1 and in Bright Futures; Guidelines for Health Supervision of Infants, Children, and Adolescents.13 Fundamental to HS is the belief that the health care system is the best place to locate this type of early intervention on behalf of children. Nationally, 88% of children have a regular source of pediatric care, and pediatric care has an established base of funding in the health care sector.2
The principal components of HS are shown in Table 1. At each site, 2 HSSs (or their full-time equivalents) work as a team with 4 to 8 pediatricians and pediatric nurse practitioners; each HSS serves ∼100 children and their families. The HSS may be an early childhood educator, a nurse, nurse practitioner, social worker or other professional who has expertise in early childhood development. Trained in HS by the Boston University team, the specialist conducts office visits (jointly or sequentially with the pediatric clinician) and home visits, assesses children's developmental progress, provides referrals and follow-up to resources in the community, organizes and conducts parent discussion groups, coordinates early reading activities, and maintains a telephone information line for questions about child development and behavior. During extended well child visits, the clinician and the HSS use “teachable moments”14 to discuss with parents how their children learn, respond to stress, and develop. They also evaluate early signs of developmental delay or behavioral problems and identify health behaviors that may place a child and his family at risk.4
HS is grounded in the assumption that educating and supporting parents benefits families and children. Figure 1 provides a conceptual framework for the expected relationships between program activities and child and parent outcomes. It is assumed that: 1) the major impact of the HS services will be on parents by increasing their knowledge of child development, sense of competence, and satisfaction with parenting; 2) the HS services directly or indirectly will enhance parent behaviors that support and nurture child development; and 3) improvements in child outcomes will result indirectly from the impact of the HS services on the parent.
The evaluation has 3 overall goals: 1) to assess whether the Healthy Steps Program is successful in reorienting pediatric practice to emphasize child development issues; in increasing parents' knowledge about early nurturing of infants; in involving parents more in their children's development; and in promoting practices that improve the health, safety, and health care utilization of their children; 2) to measure the costs of the program; and 3) to determine the relation of the program's costs to its outcomes.
An outcome evaluation will assess the impact of HS on parents, children, and providers. It will determine if HS has an effect on parents' knowledge, beliefs, and practices regarding parenting, including their understanding of early child development, parental adoption of health and safety activities, and parental satisfaction with pediatric care. Parameters of children's health, behavior, and language development, and providers' knowledge, beliefs, and practices are other outcomes to be evaluated. The outcome evaluation relies on many data sources including self-administered provider surveys, brief standardized forms completed by parents at office visits, telephone interviews with parents, and medical record reviews. (Selected evaluation outcomes and measures are listed in Appendix A, see page 8).
A process evaluation will document the context of HS, examining the implementation process and the extent to which services are consistent with the program plan. It uses key informant interviews, self-administered provider surveys, site documents, and an ongoing log of the HSS's activities to document the Healthy Steps intervention.
In a cost-effectiveness evaluation, readily monetized benefits such as reductions in health services utilization will be used in conjunction with data on program costs to produce an estimate of net program costs. These cost data, in turn, will be used together with data on outcomes to evaluate the cost-effectiveness of HS (relative to currently existing nonmonetizable pediatric practice) in producing improvements in such measures as utilization of well child care and parent satisfaction. Differences in use of health services between intervention and comparison groups will be measured in terms of marginal resource costs to determine the extent to which HS produces cost savings among participants.
The evaluation also will assess the potential for institutionalization of HS. The overall goal of assessing the potential for adoption of HS into pediatric practice is to look beyond the immediate limits of HS to its implications for the future of infant and child health care.
The evaluation design relies on 2 kinds of comparison strategies in which 6 evaluation sites use a randomized controlled trial, and 9 sites, a quasi-experimental nonequivalent control group design. At the 6 randomization sites, families who are provided services at the same location were randomized at the time of the infant's birth into intervention and control groups. This strategy was restricted to sites that could provide adequate numbers of children for both the intervention and the control groups, and for which procedures could be adopted to reduce the potential for contamination of the control group. At the remaining 9 quasi-experimental sites, a comparison location was selected in a comparable community. Families in the comparison group are still considered to be “Healthy Steps” families, receiving all aspects of excellent pediatric services, but not the services of the HSSs or the specific HS components.
Between September 1996 and July 1997, the Commonwealth Fund selected 15 evaluation sites. The goal of site selection was to represent a variety of major organizational types of pediatric practices including private group practices, health maintenance organizations, and hospital-based clinics, all which now participate in managed care arrangements. Table 2provides a description of the 15 sites. Like pediatric care in the United States in general, the majority of the practices were group-based practices. For budgetary and logistic reasons, the target number of newborns at each site and its comparison was limited to 400 (200 intervention; 200 controls). At least half of the sample is expected to remain in the study for 3 years.
Enrollment and Eligibility
Consecutive enrollment of newborns began in September 1996 at 1 pilot site. The initiation of subsequent sites was staggered throughout a 1-year period to allow time to set up enrollment and other evaluation procedures. It took, nevertheless, 2 years to complete enrollment of the entire sample for the evaluation because the length of time required to obtain an adequate sample was more than a year at some sites. Enrollment took place either in the hospital after the child's birth or at the first pediatric office visit. In cases of multiple births, 1 child was randomly selected for the evaluation. To be eligible for HS, the newborn had to be <4 weeks old at the time of enrollment (from birth to 28 days of life, inclusive) and a patient at the Healthy Steps site. Children were excluded from eligibility only if: 1) their parents expected to move from the area or change site of care within 6 months; 2) their mothers (or fathers if they were the custodial parents) did not speak English or Spanish fluently; 3) they were to be adopted or placed in foster care; or 4) they were too ill to make an office visit within the first 28 days of life. A brief standardized form, available in Spanish and English, was self-administered by the parent(s) or administered by an interviewer at the time of enrollment. This form provided data on the demographic characteristics of the newborn and his/her parents. Enrollment of families ended in November 1998. Children in the evaluation are being followed through November 2001. Figure 2is a graphic representation of the progression of the children through the evaluation.
Analyses for this article are based on enrollment data for the Healthy Steps sample and national data on 1997 US live births.15 After entry and verification, data describing the Healthy Steps sample were summarized and compared with data for the United States. The χ2 goodness-of-fit test was used to evaluate whether the distribution of these variables for the Healthy Steps sample was similar to the distributions for US births in 1997. In addition, comparisons were made between intervention and comparison families at the randomization and quasi-experimental evaluation sites. These 2 types of sites were separated for the analysis to examine the full range of potential differences across randomization and quasi-experimental sites and intervention and comparison families. Moreover, the degree of selection bias associated with inclusion in the intervention or control group should be minimal in the randomization sites. The χ2 test of independence was used to evaluate differences in variables across groups.
Approval of the design protocol was obtained from the Johns Hopkins University Committee on Human Research and the institutional review boards of the parent organization for each site.
Size of HS Sample
Of 6279 eligible children, 5565 enrolled in the evaluation, 2963 (53.2%) children in the intervention group and 2602 (46.8%) in the comparison group (Table 3). The parents of 5563 children completed enrollment questionnaires.
When compared with the national birth data for the United States as a whole, the Healthy Steps sample seems similarly diverse with respect to mother's age, race, ethnic origin, education, marital status, live birth order, insurance during pregnancy, and infant's birth weight (Table 4). However, with the exception of maternal age, the distribution of the variables was significantly different from the distribution for US births.
More than 10% of the Healthy Steps mothers are teenagers whereas >30% are aged 30 or older. Mothers in the Healthy Steps sample are somewhat better educated than mothers nationally. Eighteen percent have 11 years or less of education whereas 27% have completed college. Also, 18% of Healthy Steps mothers are black or African-American compared with 15% of mothers nationally. Although the percentage of mothers reporting “other' race is high in the Healthy Steps sample, it primarily consists of women who reported being Hispanic, the majority of whom are likely to be white. Approximately 20% of Healthy Steps mothers are of Hispanic origin versus 18% of mothers nationally. Thirty-six percent of the HS mothers are single compared with 32% nationally. Approximately half of the newborns were first-born, 46% in the Healthy Steps sample versus 41% nationally. Close to one-third of mothers in both groups were insured by Medicaid during their pregnancy. Approximately 7% of Healthy Steps infants were low birth weight, weighing <2500 g at birth, compared with nearly 8% of infants nationally.
There are no differences between intervention and comparison families at randomization sites on any of the maternal characteristics, insurance status, or infant's birth weight (Table 5). At quasi-experimental sites, a greater percentage of the mothers in the comparison group than the intervention group were aged 29 years or younger. Also, a greater percentage of these comparison group mothers had 11 or less years of education and a smaller percentage are college graduates. Mothers in the comparison group were more likely to be black or African-American and to be of Hispanic origin. They also were less likely to be married. There were no differences between intervention and comparison groups at the quasi-experimental sites in the percentage of mothers whose source of payment for prenatal care was Medicaid or in the percentage of infants who were low birth weight or first born.
There are a number of differences between the randomization and quasi-experimental sites (Table 5). The mothers at the quasi-experimental sites were slightly older and more likely to be married than were mothers at the randomization sites. They were more likely to have 11 years of education or less, but they were also more likely to have graduated from college. Moreover, they were less likely to report being white and more likely to report ”other' racial groups. Families at the quasi-experimental sites also were less likely to be on Medicaid than were families at randomization sites. Finally, there was a lower percentage of low birth weight infants at the quasi-experimental sites.
The Healthy Steps Sites and Sample
The Healthy Steps population is economically and ethnically diverse, reflecting the diversity of the nation as a whole. There is a wide range of demographic characteristics of the sample with regard to mother's age, education, race, ethnicity, marital status, and health insurance coverage.
There seem to be differences between randomization and quasi-experimental sites as well as between intervention and comparison groups at the quasi-experimental design sites. As important, randomization at the sites that selected to use this approach seems to have been effective in equalizing the characteristics of families in the intervention and comparison groups.
The differences between the intervention and comparison groups at the quasi-experimental design sites indicate the need to take account of these differences in analyses of the effect of the Healthy Steps Program. The differences between the randomization and quasi-experimental design sites seem to be greater than the differences between the quasi-experimental intervention sites and their matched comparison sites. This result is not surprising because the comparison sites were selected to minimize differences with the intervention sites. The large differences between randomization and quasi-experimental sites are also not surprising because at the very least these sites differ in terms of the size of the practice; the randomization sites generally had to serve larger numbers of newborns to obtain adequate numbers for assignment to both the intervention and comparison groups.
Implementation of Healthy Steps
HS has a set of written protocols that specify the core content of HS. Although the HSS and the pediatric team at each site were trained and have access to a manual and a common set of materials, there will undoubtedly be variability in program implementation. To assess the effect of HS in the real world, where delivery of clinical services cannot be controlled, all children enrolled in HS will be considered in the evaluation even if some were known to have received less than the full intervention or to have dropped out. Evaluation data on the services to families will be used to interpret differences (or the lack thereof) in outcomes between families in the intervention and comparison groups.
Comparing Quasi-experimental and Randomization Designs
The evaluation includes both randomization and quasi-experimental designs. Relying on a quasi-experimental design allowed for selection of sites more representative of the variety of pediatric care arrangements in the United States. At the quasi-experimental design sites, families receive care at separate locations and are treated by different clinicians. Thus, the quasi-experimental design provides an opportunity to examine the effects of site characteristics on implementation. However, no matter how careful the selection process, it can provide no guarantee that there are not differences between the intervention and comparison groups that may influence the observed effects of HS; that is, the study participants in the intervention practice may be healthier than those in the control practice and may perform better even if there were no positive benefit of HS. Not unexpectedly, our baseline findings suggest some important differences in the demographic characteristics of families in the quasi-experimental intervention and comparison groups. The analysis will adjust for measurable differences in the composition of the 2 groups.
The randomization sites, on the other hand, are less subject to this potential bias. Randomization increases the probability that the characteristics of families in the intervention and control groups and the conditions to which they are exposed are similar on observed and unobserved factors, other than the intervention, and that differences between them are because of random errors. Our baseline findings suggest no differences in these groups on the demographic characteristics of families. Because the same clinicians treat the intervention and control families, clinician-related factors are not likely to influence the effects, except where clinicians do not honor the requirements of the study design to not provide HS services to control families. If the effects of HS at randomization sites are in the same direction as those observed at the quasi-experimental sites, then considerable strength will be added to the conclusions of the evaluation.
Range of HS Pediatric Sites
In addition to combining randomization and quasi-experimental designs, the evaluation includes 3 organizational types (hospital-based clinics, group practices, and health maintenance organization pediatric practices). To account for within-site correlation of outcomes, between-site variability of the effects of HS, and between-provider type variability of the effects of HS, random effects models16 ,17will be used in the data analyses. These models also are referred to as hierarchical linear models18 or multi-level models.19 The estimation techniques allow simultaneous estimation of site-specific, provider type-specific and overall estimates of the effects of HS. The method also provides estimates of the between-site and between-provider type variability.
Calculating Sample Size for the Evaluation
Sample size analyses focused both on ideal sample sizes and also on questions that could be answered with the available samples of families. These estimates were difficult because literature on home visiting and other intervention strategies used in HS provided little guidance on the potential magnitude of the effect of specific HS program components.
Assessing the cost-effectiveness of HS also is not straightforward. Adjustments must be made to the annual expenditure data provided by the sites to move from site-reported expenditures to economic costs. The main problem is variability across sites that reflects: 1) local variations in input prices and structural characteristics that are unique to each site; 2) differences in training and experience of HSSs across and within sites; and 3) reporting errors. In addition to these adjustments, wages at each site must be adjusted of any variation attributable to variations in labor market conditions and local cost of living.20 There are also some adjustments for variation in HSS training and experience. Data on children's use of services will be used in conjunction with data on the amount reimbursed for these services to determine if or not there are any cost offsets that result from substitution of HS services for medical care services. Costs will be adjusted for the percentage of time HSSs spend on activities other than providing HS (eg, time spent on evaluation-related activities). Finally, cost estimates will be subjected to sensitivity analysis to provide information on the stability of these estimates when the assumptions that underlie them are varied.
Measuring Program Effects
Many Healthy Steps sites, because they were selected as best pediatric practices, already have high baseline levels for key outcomes like vaccination status. With less room for improvement, changes may be marginal and less easily measured. In addition, with the increasing national focus on early child brain development, pediatric practices, in general, may be expected to change their services throughout time. Improvements in child development services, to the extent that both randomization and quasi-experimental sites adopt them, will add to this ceiling effect. Survey data from sites and parents will provide information on these confounding factors so that they may be accounted for in the analysis.
The HS evaluation will document the implementation of the program, assess its outcomes, and determine if the costs are less than, equal to, or greater than the benefits. It is carefully designed to address the complexities of a program with multiple objectives, multiple components, and a wide range of expectations. The evaluation will provide information for practicing clinicians on the effectiveness of HS in improving care for families. It will provide policy makers with empirical evidence to inform the national debate on whether pediatric practices are an appropriate venue for helping parents promote their children's development. In future years Healthy Steps' sites and families, through their contributions to the evaluation, will provide valuable information on the extent to which HS works for them, information that will benefit efforts to improve pediatric services for families and children in the nation as a whole.
We wish to thank Margaret E. Mahoney, Chairman of the Healthy Steps National Advisory Committee, for her vision and leadership and committee members for their guidance, advice, and support. They include: Howard Berman, T. Berry Brazelton, Peter Budetti, Katie Couric, Barbara DeBuono, Fernando Guerra, Robert Haggerty, Myron Hofer, Judith B. Igoe, Nancy L. Kassebaum, David M. Lawrence, Charlene Rydell, Ruth J. Simmons, and Edward Zigler. We wish to thank the Healthy Steps National Program Team including Barry Zuckerman, Margot Kaplan-Sanoff, and Andrea Bernard at Boston University School of Medicine; Kathryn Taaffe McLearn and Michele Yellowitz at the Commonwealth Fund; and Michael Barth and Juliet Fried at ICF Kaiser International. We are grateful to Karen Davis, President of the Commonwealth Fund, for her leadership and continued support. We thank other members of the Johns Hopkins University Evaluation Team for their contributions to the development of the proposal and the implementation of the evaluation. We are particularly grateful to the parents and providers of the Healthy Steps evaluation.
The Healthy Steps Evaluation Team members include: Mary Benedict; Diane Burkom; Margaret Caughy, ScD; Janice Genevro, PhD, MSW; Holly Grason, MA; Ashrafal Huq, MBBS, MPH; William Hou, MSc; Cynthia Minkovitz, MD, MPP; and Eleanor Szanton, PhD.
- Received July 23, 1999.
- Accepted October 12, 1999.
Reprint requests to (B.G.) Department of Population and Family Health Sciences, Johns Hopkins University School of Hygiene and Public Health, 624 N Broadway, Baltimore MD 21205. E-mail:
‖ The Healthy Steps Evaluation Team members are listed in the “Acknowledgments” section.
The Healthy Steps for Young Children Program is a program of the Commonwealth Fund, local funders, and health care providers across the nation. It is cosponsored by the American Academy of Pediatrics. Funding for the Healthy Steps National Evaluation is being provided by the Commonwealth Fund and local funders. The views presented here are those of the authors and not necessarily those of the Commonwealth Fund, its directors, officers, or staff.
- Copyright © 2000 American Academy of Pediatrics