Published online April 1, 2005
PEDIATRICS
Vol. 115
No. 4
April 2005, pp.
1113
(doi:10.1542/peds.2004-2652)
Can the Results Be Believed?
James Glauber, MD, MPH, FAAP
Neighborhood Health Plan
Boston, MA 02210
To the Editor.—
I applaud the rigorous research of Olds et al.1 This is a critical topic, and his research has added critically needed information on strategies to improve outcomes for low-income mothers and children.
However, I must raise questions about the discussion section of the article. Olds et al concluded that home-visiting interventions have been efficacious across a variety of outcomes based on finding a P value <.05 in several of the outcome measures. However, this is based on using a significance level of P < .05. This cutoff for concluding that a finding is statistically significant with 95% confidence is appropriate for 1 comparison. The Olds et al research involves multiple comparisons. In the nurse-visiting study, there are 50 comparisons, with 12 reported as having significance levels of P 
.05. In the paraprofessional study, there are 38 comparisons, with 7 having a significance level of P .05. With multiple comparisons, especially when the primary and secondary outcomes have not been stated a priori, the appropriate cutoff for significance is .05/n, where n = the number of comparisons.
In the first study, therefore, the P level for significance should be .001. By this standard, none of the reported comparisons would reach statistical significance in either study.
This is not merely an academic argument. The discussion section in the Olds et al article would lead the casual reader to conclude that the interventions have been proven efficacious across a variety of outcomes. The natural extension of this argument is to determine if these intensive interventions can yield "corresponding effects in practice." Significant efforts and resources would need to be devoted to implement comparable programs in practice.
As medical director for a Medicaid managed care plan, I would be enthusiastic to explore models that have been proven effective in reducing morbidity or improving outcomes for vulnerable mothers and children. However, the methodologic limitations of the Olds et al study leave me with skepticism whether the conclusions are truly supported by the data.
REFERENCE
- Olds DL, Robinson J, Pettitt L, et al. Effects of home visits by paraprofessionals and by nurses: age-four follow-up of a randomized trial.
Pediatrics. 2004;114
:1560
–1568[Abstract/Free Full Text]
PEDIATRICS (ISSN 1098-4275). ©2005 by the American Academy of Pediatrics
