BACKGROUND: Although several systematic reviews have concluded that home visiting has strong evidence of effectiveness, individual evaluations have produced inconsistent results. We used a component-based, domain-specific approach to determine which characteristics most strongly predict outcomes.
METHODS: Medline and PsycINFO searches were used to identify evaluations of universal and selective home visiting programs implemented in the United States. Coders trained to the study criterion coded characteristics of research design, program content, and service delivery. We conducted random-effects, inverse-variance–weighted linear regressions by using program characteristics to predict effect sizes on 6 outcome domains (birth outcomes, parenting behavior and skills, maternal life course, child cognitive outcomes, child physical health, and child maltreatment).
RESULTS: Aggregated to a single effect size per study (k = 51), the mean effect size was 0.20 (95% confidence interval: 0.14 to 0.27), with a range of –0.68 to 3.95. Mean effect sizes were significant and positive for 3 of the 6 outcome domains (maternal life course outcomes, child cognitive outcomes, and parent behaviors and skills), with heterogeneity of effect sizes in all 6 outcome domains. Research design characteristics generally did not predict effect sizes. No consistent pattern of effective components emerged across all outcome domains.
CONCLUSIONS: Home visiting programs demonstrated small but significant overall effects, with wide variability in the size of domain-specific effects and in the components that significantly predicted domain-specific effects. Communities may need complementary or alternative strategies to home visiting programs to ensure widespread impact on these 6 important public health outcomes.
- birth outcomes
- child cognitive development
- child maltreatment
- child physical health
- early childhood
- home visiting
- maternal life course
- parenting behavior
- CI —
- confidence interval
- HomVEE —
- Home Visiting Evidence of Effectiveness
Early childhood marks a period of rapid growth and development that lays the foundation for future health and success in school and life.1 Because parents play a critical role in shaping children’s early development, interventions that reach families in these early years have great potential for producing long-term benefits.2 Prenatal and early-childhood home visiting is a widely endorsed method for delivering a vast array of preventive and early intervention services to families in need of support. By engaging families in home visiting programs during the prenatal or early-childhood period, providers seek to improve children’s long-term developmental trajectories by fostering improved parenting knowledge and skills, social support, coping and problem-solving skills, and access to community and health services.3
Despite national and international endorsement of home visiting as a strategy to prevent child maltreatment and promote enhanced functioning and well-being for children and families,4–8 previous meta-analyses and literature reviews of home visiting programs across a wide range of outcomes suggest mixed, modest findings depending on the programs and outcomes examined.6,9–12 A recent review funded by the US Department of Health and Human Services, the Home Visiting Evidence of Effectiveness (HomVEE) review, identified 13 models that met the department’s criteria for effectiveness.13 Across and even within these “evidence-based” models, the findings have been inconsistent, leaving gaps in knowledge about the effectiveness of home visiting across various outcome domains. The mixed findings may be due to program design, the match between program components and expected outcomes, or the quality of implementation of the program or the evaluation. Alternatively, the differences in effects might simply be explained by the variation in the way home visiting programs are comprised and delivered.
Best-practice recommendations concerning home visiting have generally either taken the form of suggesting wholesale adoption of models that have been shown to be effective (eg, HomVEE [homevee.acf.hhs.gov], Promising Practices Network [promisingpractices.net]) or have been based on clinical impression about particular approaches (eg, recommendations for a particular schedule of home visits). Although model ratings are important for guiding practitioners in adopting a program model, any particular program may not include the most effective combination of components to produce maximum results for a given population or community. In addition, as the Maternal, Infant, and Early Childhood Home Visiting Program14 impels increased focus on outcomes, a pressing question is how to best build the effectiveness of a program model or enhance models that may already be in operation; that is, what elements (eg, content, service delivery methods) in home visiting programs seem most important for program success?
Although 2 systematic reviews conducted before 2002 examined the relationship between parent and child outcomes and a small subset of program components,12,15 no reviews have fully disassembled home visiting programs into individual components or included studies conducted during the last decade. Therefore, a component analysis applying meta-analytic techniques was used to synthesize the results of published evaluations of home visiting programs to determine which individual home visiting program components have the most power to predict key parent and child outcomes.
In September 2010, the PsycINFO and Medline databases were searched for literature published between 1979 and 2010 with evaluations of home visiting programs. Studies were limited to those published in English as a journal article, book, or book chapter, although programs could be implemented in any language. Details of the search strategy are outlined in the Appendix. The initial search was designed to be very broadly inclusive of home visiting programs.
The original literature search resulted in 3252 unduplicated studies. Of these, 49 were literature reviews and meta-analyses, from which we identified additional relevant publications. A secondary search was conducted on author names that appeared at least twice in the original search results. In addition, unduplicated studies from HomVEE were examined. These follow-up strategies yielded an additional 1875 records, providing 5127 total abstracts for possible inclusion.
Inclusion criteria were selected to define the scope of the meta-analysis as evaluations of universal and selective (ie, for at-risk families) programs that used home visiting as a primary delivery strategy for pregnant women and families with children from birth through age 3 years in the United States. Programs that conducted only 1 or 2 home visits were excluded as dissimilar to the rest of the field. Home visiting programs targeting families for existing identified problems (eg, family preservation programs or programs that provided services to families with a substantiated child maltreatment case) were excluded. Similarly, criteria were selected to ensure that evaluation results could be generalized to a broad population of typically developing children and parents. Thus, we excluded programs that targeted parents or children because of developmental disabilities, chronic illness, feeding disorders, or bereavement because the programs provide specialized components not found in the general field of home visiting.
Figure 1 presents the PRISMA flow diagram for study inclusion. Abstracts identified in the literature search were screened by 2 project staff members to determine eligibility. A study was excluded at this point only if both staff members agreed that it met none of the inclusion criteria; 525 documents were retrieved and reviewed in full text. To allow for calculation of comparable effect sizes, studies that used a single-case evaluation method, lacked a control or comparison group, or did not contain enough statistical information to calculate effect sizes were excluded. The resulting 126 studies were coded for meta-analysis; a subsample of the 51 articles including the 6 outcome measures (maternal life course, birth outcomes, parent behaviors and skills, child cognitive outcomes, child physical health, and child maltreatment) selected for this study were analyzed.
Coding forms adapted from Kaminski et al16 captured information about the document, author(s), home visiting program, participants, evaluation design, outcome measures, and statistical results. Table 1 lists and describes the variables coded for these analyses. Full coding forms are available from the first author. When an article referred to a secondary study or article providing additional program information, that secondary document was obtained, and the information was coded. Before coding independently, data abstractors were trained to criteria of coding 3 consecutive articles with >90% accuracy.
Effect sizes analogous to Cohen’s d statistic17 were calculated from means and standard deviations whenever possible or from other reporting methods, including categorical data, correlations, and odds ratios, by using Comprehensive Meta-Analysis 2 software (Biostat, Inc, Englewood, NJ).18 Effect sizes were calculated based on unadjusted data if available or adjusted data if not. Once effect sizes were calculated, they were exported into SPSS version 20 (IBM SPSS Statistics, IBM Corporation, Armonk, NY) for analyses by using macros for multivariate analyses of effect sizes.19,20 We applied Hedges’ small sample correction to all effect sizes before analysis and weighted each by the inverse of the variance.21
Within and across articles, some samples were represented multiple times (eg, the same sample assessed at different time points, assessed with different measures, or reported in different articles). Including all published reports of those samples would have allowed a small number of frequently published programs to bias the results. Thus, for each analysis, we selected or aggregated effect sizes such that each sample (eg, a program implemented in a particular location) only provided a single effect size for that analysis. Data on birth outcomes at any time point in a study were included. For all other outcomes, immediate posttest assessments were preferred. If immediate posttest data were not available for a particular sample, we included assessments that occurred during the intervention but after two-thirds of the intervention was delivered. Follow-up data were excluded due to a lack of comparability in the length of follow-up periods. When “total” scores and “subscale” scores from particular measures were reported, preference was given to the total score if it fell within a single outcome category. When a single study included ≥3 study arms, the effect size most closely attributable to the effect of only the home visiting program (eg, treatment versus no-treatment comparison, or treatment plus enhancement versus enhancement only) was selected.
We first examined overall program effects on the 6 outcome categories by aggregating to a single effect size per study sample. We calculated overall weighted mean effect size, 95% confidence interval (CI), and Q and I2 statistics.22 Adhering to the analytic strategies set forth by Kaminski et al,16 we next investigated outcome-specific mean effect sizes by aggregating to a single effect size per study sample for each outcome category, as well as CIs and Q and I2 statistics. We used inverse-variance–weighted analyses of variance to examine the impact of 4 indicators of methodologic rigor (random assignment, assessment of initial equivalence, using a pure no-treatment comparison group, and testing the effect of the home visiting program as a stand-alone intervention versus as part of a larger package of interventions) and timing of the outcome measure (before versus at the end of treatment) on effect sizes for each outcome category. Finally, we used inverse-variance–weighted linear regression to test the impact of program components on effect sizes, with the goal of determining the predictors of strongest program effects. Only components theoretically expected to contribute to particular outcomes were tested for those outcomes. As the intent of the analyses was to model variability among studies, all reported results were obtained via random-effects models.
The overall weighted effect size of the final set of 51 studies was 0.20 (95% CI: 0.14 to 0.27). The 251 effect sizes ranged from –0.68 to 3.95. The Q test of homogeneity of effect sizes was significant (P < .001), with an I2 value of 65%. Table 2 shows the number of studies and summary statistics according to outcome category. Three outcome categories (maternal life course, child cognitive outcomes, and parent behaviors and skills) resulted in significant, positive average effect sizes. Average effects sizes were not significantly different from zero for birth outcomes, child physical health, and child maltreatment. Between 52% and 86% of the heterogeneity observed for each outcome was attributable to true variance rather than to chance, suggesting the need to further examine the nature of the heterogeneity.
In the inverse-variance–weighted analysis of variances, only 1 research design variable was a significant predictor of any outcome: effect sizes of maternal life outcomes were higher among studies reporting outcomes during treatment (mean effect size: 0.23 [95% CI: 0.13 to 0.33]) than studies reporting outcomes immediately posttest (mean effect size: 0.02 [95% CI: –0.11 to 0.15]). Measurement timing was therefore included as a covariate in the regression analysis of maternal life outcomes.
Results of the inverse-variance–weighted linear regressions assessing relationships between program components and effect sizes are presented in Table 3. Controlling for timing of assessment, no components significantly predicted maternal life outcomes. Effect sizes based on birth outcomes were significantly larger for programs using nonprofessional home visitors, programs that matched clients and home visitors on race and/or ethnicity, and programs that included problem solving. Effect sizes for the parent behaviors and skills outcome were significantly larger for programs that taught parents developmental norms and appropriate expectations, discipline and behavior management techniques, responsive and sensitive parenting practices, and programs that addressed parental substance use. Children’s cognitive outcomes were better in programs that taught parents responsive and sensitive parenting practices and programs reporting that they required parents to role-play or practice skills during home visits. Using professional home visitors was a significant predictor of better child physical health outcomes, as was teaching discipline and behavior management techniques. However, providing parents with a support group was associated with smaller effect sizes on child physical health. Better child maltreatment outcomes were associated with teaching parents how to select alternative caregivers for children and problem solving.
To ensure that these results were not unduly influenced by effect sizes based on results reported in studies as adjusted statistics, we removed those effect sizes and re-examined regression analyses with significant components. Of the 14 components reported earlier as significant, 3 could not be analyzed without the adjusted effect sizes due to low frequency (the 2 components significant for child maltreatment outcomes and the relationship between child physical health outcomes and teaching discipline and behavior management techniques). Ten of the other 11 components maintained statistical significance in these sensitivity analyses. The effect of teaching parents problem-solving strategies on birth outcomes was no longer significant and thus may be a less robust finding than other component effects.
The overall effect size of home visiting programs (aggregated across the 6 selected outcome domains) was significant and equivalent to approximately one-fifth of a standard deviation favoring the intervention group. Translated to an odds ratio, such an effect is equivalent to the comparison group being ∼1.5 times more likely to have poorer outcomes. Consistent with results of previous meta-analyses of home visiting programs,6,9,12,15 parents and children participating in home visiting programs achieved more positive outcomes overall than parents and children in control/comparison groups. However, outcome-specific mean effect sizes revealed significant but small effects only on maternal life course, child cognitive outcomes, and parent behaviors and skills. In contrast, home visiting programs did not produce significant average effects on 3 frequent program targets (birth outcomes, child physical health, and child maltreatment), suggesting that programs were, on average, not effective in addressing these outcomes. The nonsignificant effect sizes, combined with the relatively small significant effect sizes, suggest that communities may need complementary or alternative strategies to home visiting programs to have a greater impact on these important public health outcomes.
Although surveillance bias (ie, program involvement increases the likelihood of detecting maltreatment) may partially explain the lack of a significant effect size on child maltreatment outcomes measured through child protective services data, previous studies have found surveillance bias effects to attenuate but not eliminate group differences where they exist.23,24 In addition, the present analyses included self-reports of abusive parenting practices in addition to child protective services reports. Thus, the presence of a surveillance bias would likely not fully explain the lack of statistical significance.
Research design variables were generally not significantly predictive of effect sizes, whereas many program components were. Similar to other systematic reviews, no clear and consistent pattern of effective home visiting program components emerged across outcome domains.12 Only 3 components were predictors of larger effects on >1 outcome; 1 of those components was only robust for 1 outcome in the sensitivity analyses. All other significant components were only predictive of effect sizes for a single outcome domain. These results suggest that the “home visiting” label represents a diversity of approaches with differing effectiveness, and that attention to specific program content and delivery characteristics is critical to the effectiveness of these programs.
The components that emerged as significant for >1 outcome (teaching sensitive and responsive parenting, teaching discipline and behavior management techniques, and teaching problem-solving) make intuitive sense; teaching new parenting skills and behaviors was associated with greater effects on parenting behaviors, which may also translate into more positive impacts on other, sometimes more distal, outcomes, such as child cognitive development, child physical health, and child maltreatment. Using professional home visitors was unexpectedly associated with smaller program effects on birth outcomes but larger effects on child physical health outcomes. The inconsistency in these results may be due to the professional background or type of professional providing the services, as different professionals may be more or less effective with different health outcomes. Alternatively, the inconsistent results might be due to other differences not analyzed here between programs using professional and nonprofessional home visitors. Programs that enroll participants prenatally and use professional home visitors may want to look for ways to boost their effectiveness, specifically on birth outcomes.
It is important to note that not all components were tested for each outcome, either because the components were not theoretically linked to the outcome or due to limited variability of the component among studies reporting a particular outcome. In addition, nonsignificant components may be contributing to program outcomes (eg, as precursors to or in combination with other components) in interactive ways that cannot be tested by using these analytic methods. The presence of a significant component thus indicates a robust effect, but the absence of significance for a component does not necessarily imply a lack of impact. We can only conclude that the nonsignificant components did not by themselves distinguish more successful programs from less successful programs on that outcome and are thus components that are unlikely to be sufficient to produce outcomes they did not significantly predict.
Our results for the impact of different components must be taken as correlational and not as an experimental manipulation. Our results are also based on published studies and are dependent on the completeness of reporting of components within each study. Many theoretically interesting and relevant program characteristics (eg, program dosage, sample demographic characteristics, fidelity of implementation, staff training, home visitor caseload, study or program attrition) could not be tested due to insufficient numbers of studies reporting those characteristics. For example, the timing of enrollment in home visiting programs during pregnancy might be associated with a program’s ability to promote positive birth outcomes; variability in gestation at enrollment could explain the lack of significance with birth outcomes. However, this relationship could not be tested due to insufficient reporting on initiation of services.
The present meta-analysis marks a distinct departure from the common practice of recommending the wholesale adoption of evidence-based programs. Although model ratings are important for guiding practitioners in adopting a packaged program model, any particular program may not include the most effective combination of components to produce maximum results. Instead of considering each program as a black box, the coding scheme used in the present study allowed the authors to disassemble home visiting programs and examine the impact of specific components. The results suggest that certain existing components are more likely to be associated with positive effects on specific outcomes. Although careful evaluation of modifications or adaptations to existing programs would be critical, changes to include more of the significant components identified are likely to produce programs that are more potent with respect to these parent and child outcomes. For other outcomes, components that significantly predict positive outcomes remain to be identified. Our findings point to new program and research opportunities within the home visiting field, whether through the development or selection of a home visiting program, or for improving programs already labeled efficacious or effective.
- Accepted August 26, 2013.
- Address correspondence to Jill H. Filene, MPH, James Bell Associates, 3033 Wilson Blvd, Suite 650, Arlington, VA 22201. E-mail:
Ms Cachat collected data and critically reviewed the manuscript; Ms Filene conceptualized and designed the study, designed the data collection instruments, and drafted the initial manuscript; Dr Kaminski conceptualized and designed the study, designed the data collection instruments, conducted the analyses, and drafted the initial manuscript; and Dr Valle conceptualized and designed the study, designed the data collection instruments, and reviewed and revised the manuscript. All authors approved the final manuscript as submitted.
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This study was supported by The Pew Center on the States. The views expressed are those of the authors and do not necessarily reflect the views of the Pew Center on the States or The Pew Charitable Trusts.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Shonkoff J,
- Phillips D
- Guterman N
- Krugman RD
- American Academy of Pediatrics. Council on Child and Adolescent Health
- National Governors Association, Center for Best Practices. The benefits and financing of home visiting programs. Available at: www.nga.org/Files/pdf/BENEFITSFINANCINGHOME.pdf. Accessed August 23, 2008
- Butchart A,
- Harvey AP,
- Mian M,
- Furniss T
- ↵Gomby DS. Home Visitation in 2005: Outcomes for Children and Parents. Invest in Kids Working Paper No. 7. Washington, DC: Committee for Economic Development; 2005
- Layzer JI,
- Goodson BD,
- Bernstein L,
- Price C
- ↵Avellar S, Paulsell D, Sama-Miller E, Del Grosso P. Home visiting evidence of effectiveness review: executive summary. Office of Planning, Research and Evaluation, Administration for Children and Families, US Department of Health and Human Services. Washington, DC. Available at: http://homvee.acf.hhs.gov/HomVEE_Executive_Summary_2012.pdf. Accessed November 22, 2012
- ↵Health Resources Service Administration. Maternal, Infant, and Early Childhood Home Visiting Program. Available at: http://mchb.hrsa.gov/programs/homevisiting. Accessed December 5, 2012
- Hedges LV,
- Olkin I
- Bornstein M,
- Hedges L,
- Higgins J,
- Rothstein H
- Lipsey MW,
- Wilson D
- ↵Wilson DB. Meta-analysis macros for SAS, SPSS, and Stata. Available at: http://mason.gmu.edu/∼dwilsonb/ma.html. Accessed January 23, 2012
- Higgins JP,
- Thompson SG,
- Deeks JJ,
- Altman DG
- Olds D,
- Henderson CR Jr,
- Kitzman H,
- Cole R
- Chaffin M,
- Bard D
- Copyright © 2013 by the American Academy of Pediatrics