The Review Process Fails to Require Appropriate Statistical Analysis of a Group-Randomized Trial
Peter J. Hannan, MStatSimone A. French, PhD
John H. Himes, PhD, MPH
Jayne A. Fulkerson, PhD
Mary Story, PhD
Division of Epidemiology
School of Public Health
University of Minnesota
Minneapolis, MN 55454-1015
To the Editor.—
We are surprised that the journal’s review process of the school-randomized trial reported by MacKelvie et al1 did not insist on an analysis appropriate to the group-randomized design or at least require stronger justification of the assumptions involved in ignoring the randomization design in the analysis.
Randomizing intact social groups is a common approach outside the clinic because it is often easier, and possibly only feasible, to intervene with a whole class, troop, church, or community rather than to work with individuals. Members within intact social groups tend to be more like each other than they are like members in other groups, making for some redundancy of information and increased variance compared with the same number of subjects individually randomized. As a general rule, group-randomized trials that are analyzed by using methods appropriate for individual-level trials will overestimate the significance of the effects.
Twenty-five years ago, Cornfield2 warned clearly that "randomization by cluster accompanied by an analysis appropriate to randomization by individual is an exercise in self-deception and should be discouraged." Methodological reviews (eg, Donner et al,3 Simpson et al,4 and Smith et al5) show that Cornfield’s message is not well heeded and point to neglect in the review process for insisting on appropriate attention to the analytic issues incurred by the choice of a group-randomized trial. When randomization is by group but analyzed by individual, chance differences between the schools randomized to the control versus the intervention condition are confounded with the intervention effect. If these chance differences are not separated out as a school-random effect, 2 consequences occur: 1) the realized difference is attributed to the intervention, and 2) the variation against which to assess the intervention effect is underestimated,6 making for an overly sensitive analysis that increases the probability of false-positive inference. The reviewers allowed a curious statement: "...we analyzed and adjusted for these differences by subject so as not to bias results by comparing school means."1(p.e450) Adjustment for individual covariates is possible in a group-randomized trial.7 In the limitations the authors say: "Randomization by school, as opposed to individual children, introduced bias and influenced the generalizability of results. However, this design was most feasible...."1(p.e451) in avoiding contamination, etc. Yes, there are good reasons for randomizing intact social groups, but the statistical and inferential implications cannot then be ignored.
Because Pediatrics is a prominent vehicle for disseminating important pediatric research, we urge that the editorial and review process incorporate more careful statistical review of submissions in which the study design is group-randomized.
REFERENCES
- MacKelvie KJ, Khan KM, Petit MA, Janssen PA, McKay HA. A school-based exercise intervention elicits substantial bone health benefits: a 2-year randomized controlled trial in girls. Pediatrics.2003 :112(6) . Available at: www.pediatrics.org/cgi/content/full/112/6/e447
- Cornfield J. Randomization by group: a formal analysis.
Am J Epidemiol.1978
:108; 100
–102
[Free Full Text] - Donner A, Brown KS, Brasher P. A methodological review of non-therapeutic intervention trials employing cluster randomization, 1979–1989.
Int J Epidemiol.1990
:19; 795
–800
[Abstract/Free Full Text] - Simpson JM, Klar N, Donner A. Accounting for cluster randomization: a review of primary prevention trials, 1990 through 1993.
Am J Public Health.1995
:85; 1378
–1382
[Abstract/Free Full Text] - Smith PJ, Moffatt MEK, Gelskey SC, Hudson S, Kaita K. Are community health interventions evaluated appropriately? A review of six journals. J Clin Epidemiol.1997 :50; 137 –146[CrossRef][Web of Science][Medline]
- Zucker DM. An analysis of variance pitfall: the fixed effects analysis in a nested design. Educ Psychol Meas.1990 :50; 731 –738[Abstract]
- Murray DM. The Design and Analysis of Group-Randomized Trials. New York, NY: Oxford University Press; 1998:187–190
Kerry MacKelvie O’Brien, PhD
Endocrinology and Diabetes Unit
BC Children’s Hospital
Food, Nutrition and Health,
University of British Columbia,
Vancouver, BC, Canada V6H 3V4
Heather McKay, PhD
Department of Orthopaedics/Family Practice
Division of Orthopaedic Engineering Research
University of British Columbia
Vancouver, BC, Canada V5Z 1L8
Rachel M. Altman, PhD
Department of Statistics and Actuarial Science
Simon Fraser University
Burnaby, BC, Canada V5A 1S6
Patricia A. Janssen, PhD
Department of Health Care and Epidemiology
University of British Columbia
St Paul’s Hospital,
Vancouver, BC, Canada V6Z 1Y6
Karim Khan, MD, PhD
Department of Family Practice
University of British Columbia
Vancouver, BC, Canada V6T 1V6
In Reply.—
We thank Mr Hannan et al for their comments with respect to our school-based intervention.1 We appreciate the opportunity to discuss the cluster randomization model and address some of the challenges confronting those who conduct "real-world" pediatric research trials. Hannan et al highlight the need for appropriate methods of statistical analysis when cluster randomization is used in school or community trials. School-based studies have a nested structure whereby schools reside within a community, classes within schools, and students within classes.
In the context of our intervention study, randomization of schools was the only means to deliver an exercise program to children without contaminating the control group. Because the intervention frequently took place outside of closed classrooms (ie, on the playground), changes in physical activity patterns within schools would easily be observed by controls if children were randomized at an individual level within classes or if classes (within schools) were randomized as clusters. Hannan et al raise the issue that responses observed on students within the same class or within the same school may be correlated. This correlation, if it exists, should then be accounted for in the data analysis.
One method of analyzing clustered data is to take the school as the unit of analysis and to compare the means of the responses observed at each school. Because these derived data (school means) can be treated as independent observations, this approach allows us to circumvent the correlation issue altogether. However, we believe that this method is inappropriate for 2 reasons. First, although teachers delivered the intervention to all students in their class, we assessed only those children who provided consent. Thus, the numbers of participants ranged from 1 to 12 per school at the end of the 20-month trial. The wide range in number of participants per school implies that the variability associated with the school means may differ substantially among schools. Standard statistical techniques rely on the assumption that observations (in this case, school means) have a common variance and hence are not applicable here. Second, the biological differences in the timing and magnitude of linear growth, which is highly individualized, represents the greatest source of variability in pediatric bone studies.2,3 Adjustment for confounders on an individual level, which is not possible when school is the unit of analysis, is therefore desirable. Simpson et al4 recommend using the individual as the unit of analysis while accounting for relationships between responses of individuals in the same cluster.
In our original analysis, we did not specify either class or school as clusters. With respect to schools, we agree with Hannan et al that "chance" differences may exist among schools and that this warrants both biological and statistical consideration. However, our target schools in the Richmond school district are located in close geographical proximity and are fairly homogenous in their racial and socioeconomic mix. We had no reason to believe that between-school differences would be greater than within-school differences. With respect to classes, we would expect differences in the way that the physical activity intervention is delivered by classroom teachers and therefore that responses observed on children in the same class may be correlated. However, students received the intervention from different classroom teachers as they progressed through grade levels, and classes do not remain as intact groups over school years (thus, "class" is not a well defined cluster in our study). To compensate for this limitation of the study, we conducted classroom visits and teacher training to standardize the intervention across classes and schools and to equalize teacher influence across cases as far as possible. Formally, we may assess the effect of school on the change in bone mineral content at the femoral neck and the lumbar spine using a linear mixed model, with the school effect designated as random. We incorporated baseline bone mineral content, height, height change, maturity, and physical activity outside of school as covariates. In both analyses, the variability across schools (femoral neck: SD = 0.00001; lumbar spine: SD = 0.13) was far less than the variability among students within each school (femoral neck: SD = 0.20; lumbar spine: SD = 2.70). Furthermore, the estimated effect of the intervention and its standard error were similar to those in our original analysis (when school was not designated as random), which also suggests a negligible school effect.
We welcome the debate on the utility of schools as the unit of randomization and analysis in our work with children and look forward to ongoing discussions of these methods through our work and that of our colleagues.
REFERENCES
- MacKelvie KJ, Khan KM, Petit MA, Janssen PA, McKay HA. A school-based exercise intervention elicits substantial bone health benefits: a 2-year randomized controlled trial in girls. Pediatrics.2003 :112 (6). Available at: www.pediatrics.org/cgi/content/full/112/6/e447
- Bailey DA, McKay HA, Mirwald RL, Cracker PRE, Faulkner RA. The University of Saskatchewan Bone Mineral Accrual Study: a six year longitudinal study of the relationship of physical activity to bone mineral accrual in growing children. J Bone Miner Res.1999 :14; 1672 –1679[CrossRef][Web of Science][Medline]
- McKay HA, Bailey DA, Mirwald RL, Davison KS, Faulkner RA. Peak bone mineral accrual and age of menarche in adolescent girls: a six-year longitudinal study. J Pediatr.1998 :133; 682 –687[CrossRef][Web of Science][Medline]
- Simpson JM, Klar N, Donner A. Accounting for cluster randomization: a review of primary prevention trials, 1990 through 1993. Am J Public Health.1995 :85; 1378 –1382
PEDIATRICS (ISSN 1098-4275). ©2004 by the American Academy of Pediatrics
CiteULike 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
