Comparison of Recruitment Strategy Outcomes in the National Children’s Study
Abstract
BACKGROUND AND OBJECTIVES: In 2000, the US Congress authorized the National Institutes of Health to conduct a prospective national longitudinal study of environmental influences on children’s health and development from birth through 21 years. Several recruitment methodologies were piloted to determine the optimal strategy for a main National Children’s Study.
METHODS: After an initial pilot recruitment that used a household enumeration strategy performed poorly, the National Children’s Study Vanguard Study developed and evaluated the feasibility, acceptability, and cost of 4 alternate strategies to recruit a large prospective national probability sample of pregnant women and their newborn children. We compare household-based recruitment, provider-based recruitment, direct outreach, and provider-based sampling (PBS) strategies with respect to overall recruitment success, efficiency, cost, and fulfillment of scientific requirements.
RESULTS: Although all 5 strategies achieved similar enrollment rates (63%–81%) among eligible women, PBS achieved the highest recruitment success as measured by the ratio of observed-to-expected newborn enrollees per year of 0.99, exceeding those of the other strategies (range: 0.35–0.48). Because PBS could reach the enrollment target through sampling of high volume obstetric provider offices and birth hospitals, it achieved the lowest ratio of women screened to women enrolled and was also the least costly strategy. With the exception of direct outreach, all strategies enrolled a cohort of women whose demographics were similar to county natality data.
CONCLUSIONS: PBS demonstrated the optimal combination of recruitment success, efficiency, cost, and population representativeness and serves as a model for the assembly of future prospective probability-based birth cohorts.
- ARS —
- alternate recruitment strategy
- DO —
- direct outreach
- ECHO —
- Environmental Influences on Child Health Outcomes
- EHBR —
- enhanced household-based recruitment
- IVS —
- Initial Vanguard Study
- NCS —
- National Children’s Study
- NICHD —
- Eunice Kennedy Shriver National Institute of Child Health and Human Development
- O/E —
- observed-to-expected
- PBR —
- provider-based recruitment
- PBS —
- provider-based sampling
- PSU —
- primary sampling unit
- SSU —
- secondary sampling unit
What’s Known on This Subject:
Recruitment of a nationally representative birth cohort sample is critical to defining the influence of environmental factors on children’s health and development. No studies in the United States have compared recruitment methodologies with respect to efficiency and cost.
What This Study Adds:
The National Children’s Study Vanguard Study compared 5 different recruitment strategies. A strategy that recruits pregnant women from a sample of provider offices and birth hospitals achieved high efficiency at low cost and serves as a model for future research.
Numerous birth cohort studies have been conducted around the world with a common goal of contributing to our understanding of human life course development in multiple dimensions. Although many cohort studies have started in early childhood,1 more recent cohorts have started in prenatal or perinatal settings. The Danish National Birth Cohort Study,2 the Generation R Study,3 the China-Anhui Birth Cohort Study,4 the Japan Environment and Children’s Study,5 and the Pregnancy and Infant Development Study in the Netherlands6 recruited women at their first pregnancy visit with participating practitioners. The Norwegian Mother and Child Cohort Study7 recruited pregnant women at the time of routine ultrasound examinations at hospitals or maternity units. The Elfe Child Cohort Study in France recruited women at birth in maternity units.8 The UK Life Study included a pregnancy sample recruited at visits to maternity care units and a probability sample of live births from the birth register.9 Only the Elfe Child Cohort Study and the birth sample of the UK Life Study used a probability sample design.
In the United States, Congress passed the Children’s Health Act of 2000 (Pub L No. 106–310) that authorized the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) “to conduct a national longitudinal study of environmental influences (including physical, chemical, biological, and psychosocial) on children’s health and development” by studying a large prospective cohort of children from birth to adulthood.10 After extensive discussion, the NICHD designed the National Children’s Study (NCS) as a probability-sample birth cohort that aimed to enroll a large and nationally representative sample of women of child-bearing age and then to enroll their newborns for follow-up through age 21.11,12
The original design developed for the NCS employed a national, multistage area probability sample. With a goal of recruiting 100 000 newborns, 110 primary sampling units (PSUs) were chosen with a target of sampling 250 births per PSU per year for 4 years.13 A PSU was almost always chosen to be a single county but could include up to 4 geographically contiguous counties. Within each PSU, secondary sampling units (SSUs) were defined as groups of census blocks or geographically defined neighborhoods. A number of SSUs in each PSU were then chosen to recruit the expected number of births while ensuring proportional representation of geographic, demographic, and socioeconomic subpopulations. The final sample was to consist of women age 18 to 49, residing in sampled SSUs at the time of delivery, and their neonates, born during a 4-year recruitment period.
In January 2009, the NCS opened a pilot Initial Vanguard Study (IVS) protocol in 7 PSUs.14 IVS recruitment used a household enumeration approach to identify, screen, and enroll women eligible for participation. Toward the end of 2009, it became clear that this recruitment methodology was not fiscally sustainable and that enrollment was substantially lower than had been targeted.15 Perception of these difficulties led to vigorous discussion about more efficient alternative study methodologies.16–19
In response, NICHD developed 3 alternate recruitment strategies (ARSs) for testing: enhanced household-based recruitment (EHBR), direct outreach (DO) recruitment, and provider-based recruitment (PBR).20 The EHBR strategy was similar to the IVS protocol but formalized community outreach and engagement activities that each study center tailored to the characteristics of its community in the target SSUs.21 The PBR protocol abandoned household-based canvassing in favor of partnering with obstetric and other health care providers in the community to identify, screen, and recruit women residing in the target SSUs to provide a sample.22 The DO strategy used passive recruitment methods, such as study mailings, to conduct pregnancy screenings and allowed participants to choose 1 of 2 initial intensity levels of data collection.23 This strategy expanded the geographic eligibility area by adding SSUs contiguous to the target SSUs. Because women self-selected to participate and SSUs were added ad hoc, enrollees did not represent an equal probability sample and, as such, the sample was not expected to be representative of births in the PSU as were the IVS, EHBR, and PBR samples.
Funded but as yet inactive study centers submitted competitive letters of intent to participate in 1 or more of these methodologies. The NIH, in a nonrandomized process, chose 10 study centers to conduct each of these 3 ARSs. Recruitment in these 3 protocols rolled out in the 30 study centers between November 2010 and February 2011. Compared with the EHBR, which devoted comprehensive efforts to identify age- and address-eligible women for screening, the PBR and the DO employed less costly measures. Beyond this difference, all 3 ARSs tested the same recruitment criteria to screen and enroll women who were pregnant or were likely to become pregnant and to follow enrolled women at periodic intervals to ensure the collection of preconception and early pregnancy data before birth.
When preliminary analysis showed that the PBR strategy achieved the greatest efficiency of operations, a provider-based sampling (PBS) strategy was designed as a fourth ARS.24 In contrast to the first 4 strategies that relied on geographic area sampling, the PBS sampled prenatal care provider locations and birthing hospitals that were used by residents in the PSU and then further sampled women at these provider locations for recruitment. The PBS, therefore, consisted of 2 recruitment subcohorts, a prenatal cohort in which pregnant women were sampled and recruited from prenatal care provider offices at their first prenatal visit and a hospital cohort in which women and newborns were sampled and recruited at the time of delivery from hospitals. The hospital cohort included women who had not accessed prenatal care. Because PBS preferentially sampled providers who cared for greater numbers of women residing in the PSU, this methodology was not expected to produce a precise equal probability sample. Three study centers initiated the PBS strategy in November 2012 and completed enrollment of neonates in March 2014. Figure 1 shows the chronology of the recruitment strategy phases along with the number of study centers involved in each phase.
Timeline of the NCS Vanguard Study phases.
In December 2014, the NIH terminated the NCS Vanguard Study upon the advice of an expert review group, and the plan to enroll 100 000 newborn infants was abandoned.25 The UK Life Study, that planned to follow 80 000 neonates throughout their lives, also ended just months after its official launch because of initial poor recruitment results.26 These experiences underline the major challenges inherent to the successful design and implementation of national and comprehensive longitudinal birth cohort studies.
We now aim to compare the recruitment outcomes of the 5 piloted strategies to the extent possible given the differences in time, location, and operational limitations specific to each strategy. The lessons learned will benefit researchers designing birth cohort studies that enroll preconceptional and/or pregnant women.
Methods
Although recruitment methodologies varied among the strategies, the main steps involved in recruiting and consenting women and neonates are common to all 5 strategies. These can be summarized as: (1) identifying women to screen, (2) administering eligibility screening, (3) consenting eligible women, and (4) enrolling neonates born to consented women. Table 1 summarizes the recruitment and eligibility criteria under each recruitment strategy. PBS eligibility criteria differed from the first 4 strategies in that PBS excluded preconceptional women and expanded geographic eligibility to the whole county.
NCS Vanguard Study Participant Recruitment and Eligibility Criteria by Recruitment Strategy
We compared recruitment outcomes by using the following 5 metrics: (1) recruitment success, (2) recruitment efficiency, (3) recruitment costs, (4) collection of early pregnancy data, and (5) sample representativeness. Where indicated, we normalized these metrics to account for different durations of study recruitment at study centers and numbers of study centers in each recruitment strategy.
Recruitment Success
We computed enrollment rates for women simply as the number of women who consented to participate in the study divided by the number eligible for enrollment. We also computed enrollment rates for neonates as the percentage of enrolled pregnant women who were retained to the time of birth. Reasons for failure to retain included miscarriages and stillbirths, as well as withdrawals and loss of contact.
We measured overall recruitment success by using an observed-to-expected (O/E) ratio of neonates enrolled because children are the target population for this planned longitudinal study. On the basis of the sample design, study centers were assigned an enrollment target of 250 neonates per year (except for 1 small PSU that was assigned a target of 150 neonates per year). For the observed number of neonates, only those born to women enrolled during the active recruitment period were counted, starting with the date at which the first consent was obtained at each study center. The observed number of neonates was then normalized to 1 year and divided by the expected annual recruitment number to compute the O/E ratio.
Recruitment Efficiency
We assessed recruitment efficacy by using participant eligibility and pregnancy yield rates. We calculated the participant eligibility rate as the ratio of the sum of pregnant and high-trying preconceptional women to the total number of women screened. We determined the pregnancy yield rate as the ratio of the number of pregnancies to the number of enrolled pregnant and preconceptional women. These rates provide different insights into the strategy-specific efficiencies of identifying the target population of women and of enrolling newborns.
Recruitment Costs
We reported the number of women who required screening to enroll 1 eligible woman as a relative index of recruitment costs across the 5 strategies. Because study centers were required to report staff hours devoted to direct recruitment activities as part of field data transmission, we also calculated the field staff hours expended to enroll 1 woman as a more direct measure of recruitment costs across the 3 ARS and PBS strategies.
Collection of Early Pregnancy Data
To collect early maternal exposure data, all strategies sought to identify a cohort of preconceptional women, except for the PBS strategy, whose design abandoned this cohort. All 5 strategies sought to collect exposure data in the first trimester of pregnancy. We report on the proportion of neonates born to enrolled preconceptional women as well as on the distribution of enrolled mothers by the trimester of pregnancy at the time of enrollment. We also compared the mean gestational age at the time of the first pregnancy data collection on the basis of the final best estimate of the expected date of delivery.
Sample Representativeness
Finally, to measure how well the recruited sample represented the population of births in each PSU, we compared the demographic characteristics of NCS mothers with those of all mothers age 18 and older residing in that PSU by using the Centers for Disease Control National Center for Health Statistics 2010 natality data.27 The sample and population percentage distributions were computed for each PSU and then as the simple average of percentage distributions across the PSUs in each recruitment strategy. Comparisons were made by mother’s age, race and/or ethnicity, education, and marital status. For each recruitment strategy, the goodness of fit between the NCS distributions on demographic characteristics was compared with the population distributions by using the likelihood ratio χ2 test statistics and the Rao-Scott correction to account for the NCS clustered sample design.
Results
For each of the 5 recruitment strategies, Table 2 summarizes the numbers of participants involved at each step of the recruitment process. The durations of active recruitment (defined as the identification, screening, and enrollment of women) and passive recruitment (limited mostly to follow-up of preconceptional women to identify new pregnancies) differed among the strategies. The IVS enjoyed the longest potential active and passive recruitment periods of 18 months each for a total of 36 months. The PBS strategy operated over the shortest active recruitment period of 7 months.
NCS Vanguard Study Recruitment Summary by Recruitment Strategy, 2009 to 2014
In total, 89 886 women were identified for screening with 40% derived from the 7 IVS centers and 31% derived from the 10 EHBR centers. Of all identified women, 82% were screened for study eligibility and 12% were determined to be eligible. Of the 11 165 eligible women, 71% consented to participate in the study. Among enrolled women who were or became pregnant during the recruitment follow-up period, 87% enrolled their newborns, resulting in 5608 study children.
Figure 2 depicts the number of births per month by recruitment strategy and by mothers’ pregnancy status at enrollment. After active recruitment ceased in September 2010, the IVS enrolled fewer neonates who increasingly represented births to women who had enrolled preconceptionally. The ARS engaged in active recruitment over a shorter period. Although approximately a quarter of the neonates enrolled in the EHBR and DO strategies were born to enrolled preconceptional women, the PBR enrolled few neonates because preconceptional women represent a small proportion of visits to a typical obstetric practice. In PBS, neonates were enrolled from delivery hospitals (hospital cohort) at the time of maternal consent, followed by neonates born to women in the prenatal cohort who were enrolled during their first prenatal visit.
Number of enrolled births per month by recruitment strategy and enrollee type, 2009 to 2014.
Measures of Recruitment Success
The enrollment rates of women and neonates by recruitment strategy are summarized in Table 3. The wide range of both rates across study centers in each recruitment strategy suggests center-to-center differences in operational effectiveness.
Enrollment Rate Among Eligible Women and Newborn Enrollment Rate by Recruitment Strategy
Table 4 describes the O/E ratio of neonatal enrollees together with the range across all study centers in each recruitment strategy. Notably, only the PBS strategy achieved an O/E ratio close to 1.
O/E Ratio of Newborn Enrollees per Year by Recruitment Strategy
Measures of Recruitment Efficiency
The participant eligibility rates of pregnant plus preconceptional women for the household-based and DO strategies were <15% (Table 5). The ability to sample women in obstetric provider settings accounts for the greater participant eligibility rate at 72% in the PBR.
Participant Eligibility Rate and Pregnancy Yield Rate by Recruitment Strategy
The pregnancy yield rate among all enrolled women was lowest in the EHBR and DO strategies (∼70%), also shown in Table 5. The rate was higher among the IVS centers (80%) because of a longer follow-up period in which 56% of preconceptional women became pregnant. The highest pregnancy yield rates occurred in the PBR and PBS strategies in which sampled women were predominantly (PBR) or exclusively (PBS) pregnant at the initial screening. An important common finding in all 3 ARSs was that ∼40% of preconceptional women conceived during the 16 month follow-up period.
Measures of Recruitment Costs
PBS was the most cost-efficient strategy because it achieved the lowest ratio of screened women to enrolled mothers and used the fewest field staff hours (Table 6).
Comparison of Resources Required per Enrolled Woman by Recruitment Strategy
Measures on Collection of Early Pregnancy Data
Figure 3 summarizes the percentage distribution of enrolled mothers by pregnancy status and by recruitment strategy. Although the design of the PBS strategy precluded enrollment of preconceptional women, the 62% proportion of women enrolled in the prenatal cohort by the end of the first trimester rivalled that of the IVS strategy and substantially exceeded the 30% to 42% proportions in the other ARS. The mean gestational age of enrolled mothers at the time of first pregnancy data collection was 20, 23, 24, 24, and 16 weeks for the IVS, EHBR, DO, PBR, and PBS (prenatal cohort) strategies, respectively.
Percentage distribution of enrolled mothers by pregnancy status at the time of enrollment according to recruitment strategy. In IVS, during the initial 6-month period of recruitment, only women <20 weeks pregnant were enrolled. Also, the longer follow-up period for the IVS participants resulted in a higher proportion of preconception women being enrolled compared with strategies with shorter recruitment periods.
Measures of Sample Representativeness
Table 7 compares the race and/or ethnicity, age, education, and marital status of enrolled women to the general population of mothers of newborns. All 5 strategies resulted in recruitment of a cohort of women that was statistically similar to the demographic characteristics of the total PSU population of mothers of newborns except for the DO strategy. The DO enrollees were disproportionally non-Hispanic white (χ2 = 8.05, P = .045) and mothers with college or higher education (χ2 = 18.99, P = .0003) compared with the general population.
Distribution of NCS Mothers’ Characteristics Compared With Population1 Distribution by Recruitment Strategy
Discussion
Prospective cohort studies are considered the most desirable research method for investigating the effects of exposures on health at the population level.28 Particularly in recent decades during which certain childhood diseases and developmental problems have increased without clear causes, the US Congress judged it imperative that large, prospective research on child health be conducted to find answers to these questions.10 In designing the NCS study, the generalizability of the study results and the ability to capture exposures early in pregnancy were the major driving forces that led to the choice of a national, multistage area probability sample involving household enumeration.11,29
The originally designed household-based recruitment method, piloted in the IVS, was highly resource-intensive and enrolled fewer participants than had been expected. Baker et al14 described the challenges in implementing this recruitment methodology and demonstrated that although cooperation rates were respectable at each step of the recruitment process, the multistep process resulted in enrollment that was far below what was expected.
Participant enrollment rates across the recruitment strategies tested are not completely comparable because these methods recruited women at different pregnancy stages and over different durations of recruitment. The DO and PBR exhibited higher enrollment rates than the other strategies. This is expected because DO participants were volunteers who wanted to participate in the study and PBR participants may have been influenced to participate by some providers who actively endorsed the study in their referral.22 One important result to be noted across all the strategies, however, is that large variations in the recruitment rates occurred across the study sites. This variation may be because of characteristics of the target populations and other community and contextual factors but also because of the capabilities of individual study centers in operationalizing the recruitment procedures. Variation could also have resulted because no ARSs were randomly assigned to study centers. Nonetheless, none of the 3 ARSs significantly improve recruitment results compared with the IVS.21–23
Across study centers hosting the same recruitment strategy, the range of recruitment successes illustrates the challenges of operating a center-based network. The marked variability in the enrollment rates and the O/E ratios across study centers in each strategy may have resulted more from operational choices and the influence of local environments than from intrinsic differences in operational efficiencies. For instance, some EHBR study centers chose to conduct enumeration, pregnancy screening, and recruitment throughout all areas of the PSU at the same time, whereas other centers may have accomplished these tasks sequentially by geographic areas. The diversity of the PSUs in the PBR (population, population distribution, and rural or urban classification) made it more challenging for some study centers to negotiate relationships with the necessary complement of provider practices. In addition, there may have been local cultural differences in provider receptivity to study activities both across and within study centers. To the extent that study centers had not achieved a steady state of operations at the end of active recruitment, the O/E ratios in Table 4 may not represent the true enrollment potential of the ARS.
By moving away from preconceptional enrollment and adopting a PBS scheme to enroll pregnant women at provider locations, the PBS was equipped to achieve the expected O/E ratio. Both PBR and PBS performed substantially better than the other 3 strategies on the indirect measure of cost, and PBS performed best on the direct measure of cost. This was not unexpected because both methodologies allowed for recruitment at the sites of obstetric care so that study centers could focus resource utilization on a finite number of sites with enriched participant eligibility and pregnancy yield rates rather than canvassing more broadly to identify potentially eligible participants at lower yields. Not surprisingly, the EHBR was the most resource-intensive of the 4 strategies subsequent to the IVS.
Focusing on data early in the formative stages of life, the IVS was able to enroll the highest percentage of preconceptional and first-pregnancy-trimester women because of its initial eligibility criteria and longer follow-up period. Enrolling preconceptional women offers a potential for collecting data that may impact the fetus during the critical period of organogenesis. However, Stanford et al30 found that the majority of preconceptional women who subsequently delivered a neonate who enrolled in the NCS did not have an exposure assessment within 30 days of conception. Another analysis concluded that their findings “suggest that geographic-based sampling of non-pregnant women for prospective enrollment into a birth cohort study may not be a feasible strategy.”31
Because the PBS design for the prenatal cohort required sampling of women at their first prenatal visit, the majority of the women in the prenatal cohort were enrolled during the first trimester. As a result, data collection among PBS prenatal cohort women began at an earlier mean gestational age when compared with the other 4 strategies.
Lessons learned in the NCS Vanguard Study are important for designing sampling and recruitment methods of future birth cohort studies. The geographic cluster sample, as originally designed, has the benefits of the availability of county-level and census block-level population data to support sampling, estimation, and linking with extant data that are available at such geographic levels. A sample of all births within the sampled segments has the desirable properties of geographic and socioeconomic representativeness. However, using this method to screen women for pregnancy and then following them to birth was prohibitively expensive. The NCS tested 4 other approaches to identify women for screening. Comparative data on participation rate, operational challenges, and relative cost are now available for researchers exploring these options and point to PBS as the most effective and cost-effective strategy.
The benefits of the PBS strategy are in its efficiency in identifying and selecting the target women and the flexibility in manipulating the sampling rate to acquire the desired sample size. There are several potential enhancements that can be further explored with this scheme: for instance, using hospitals as the PSUs and only including provider locations that refer women to sampled hospitals as SSUs. This has the advantage of reducing the number of unique hospitals that need to be recruited and facilitating the identification of eligible prenatal care provider locations. The deficiency caused by the absence of a prospective preconceptional cohort might plausibly be corrected by focusing on recruiting future siblings of enrolled children. Restricting sampling and enrollment of women to the first prenatal visit has demonstrated that early pregnancy exposure profiling is possible. The PBS strategy offers the promise of meeting major goals of a long-term birth cohort study in the most efficient manner as is recommended in a recent discussion article by Duncan et al.32
After halting NCS, NIH launched the Environmental Influences on Child Health Outcomes (ECHO) program.33 Because ECHO was designed to capitalize on existing cohorts with a plan to extend data collection through follow-up visits and/or expand participant populations by recruiting new participants, many of the lessons learned from NCS on sampling strategies will not apply. However, other lessons learned from NCS on recruitment facilitation and retention of participants, especially in and across varying parts of the United States, as well as on data collection methods, can enhance the success of ECHO.
Because ECHO cohort is not a probability-based sample, it remains to be seen whether ECHO will find answers to important questions about the effects of environmental exposures on child health and development that are generalizable to all US children. Should a longitudinal study of a large probability-based birth cohort recruited early in gestation be necessary, the lessons learned from the NCS Vanguard Study will be essential to specify the recruitment process and key operational characteristics of any such study to achieve the degree of efficacy and cost-efficiency necessary for success.
Acknowledgments
This article, a “primary NCS publication,” was developed by a writing team assembled by the NCS Publications Committee for the purpose of timely sharing of centrally collected NCS data. The analysis was conducted as part of the National Children’s Study, supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development, and funded, through its appropriation, by the Office of the Director of the National Institutes of Health, with support through the National Institute of Child Health and Human Development contracts HHSN275201000013C and HHSN275201000126U. The views that are expressed in this article are the responsibilities of the authors and do not represent the views of the National Children’s Study, the National Institutes of Health, or the US Department of Health and Human Services. The authors gratefully acknowledge their indebtedness to the many families, health care providers, and investigators who participated in the NCS.
Footnotes
- Accepted May 17, 2017.
- Address correspondence to Mark L. Hudak, MD, Department of Pediatrics, University of Florida College of Medicine – Jacksonville, 653-1 West 8th St, Jacksonville, FL 32209. E-mail: mark.hudak{at}jax.ufl.edu
This trial has been registered at www.clinicaltrials.gov (identifier NCT00852904).
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported in part by NICHD contracts. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
References
- Copyright © 2017 by the American Academy of Pediatrics
