a Society, Human Development and Health
c Biostatistics, Harvard School of Public Health, Boston, Massachusetts
b National Center for Children and Families, Teachers College, and College of Physicians and Surgeons, Columbia University, New York, New York
d Psychiatry and Behavioral Sciences
e Pediatrics, University of Washington, Seattle, Washington
f Department of Psychiatry, University of Texas Southwestern Medical Center, Dallas, Texas
g Department of Pediatrics, University of Pennsylvania School of Medicine, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
h Department of Pediatrics, University of Miami School of Medicine, Miami, Florida
i Department of Neonatology, Beth Israel Deaconess Medical Center, Boston, Massachusetts
j Children's Hospital Boston, Boston, Massachusetts
k Department of Pediatrics, University of Arkansas for Medical Sciences, Little Rock, Arkansas
| ABSTRACT |
|---|
|
|
|---|
METHODS. This was a prospective follow-up of the Infant Health and Development Program at 8 sites heterogeneous for sociodemographic characteristics. Originally 985 children were randomized to the INT (n = 377) or FUO (n = 608) groups within 2 birth weight strata: heavier low birth weight (HLBW; 20012499 g) and lighter low birth weight (LLBW;
2000 g). Primary outcome measures were the Peabody Picture Vocabulary Test (PPVT-III), reading and mathematics subscales of the Woodcock-Johnson Tests of Achievement, youth self-report on the Total Behavior Problem Index, and high-risk behaviors on the Youth Risk Behavior Surveillance System (YRBSS). Secondary outcomes included Weschler full-scale IQ, caregiver report on the Total Behavior Problem Index, and caregiver and youth self-reported physical health using the Medical Outcome Study measure. Assessors were masked as to study status.
RESULTS. We assessed 636 youths at 18 years (64.6% of the 985, 72% of whom had not died or refused at prior assessments). After adjusting for cohort attrition, differences favoring the INT group were seen on the Woodcock-Johnson Tests of Achievement in math (5.1 points), YRBSS (0.7 points), and the PPVT-III (3.8 points) in the HLBW youth. In the LLBW youth, the Woodcock-Johnson Tests of Achievement in reading was higher in the FUO than INT group (4.2).
CONCLUSIONS. The findings in the HLBW INT group provide support for preschool education to make long-term changes in a diverse group of children who are at developmental risk. The lack of observable benefit in the LLBW group raises questions about the biological and educational factors that foster or inhibit sustained effects of early educational intervention.
Key Words: early intervention long-term results low birth weight randomized controlled trial
Abbreviations: IHDPInfant Health and Development Program LLBWlighter low birth weight HLBWheavier low birth weight INTintervention FUOfollow-up only YRBSSYouth Risk Behavior Surveillance System BPIBehavior Problem Index WASIWeschler Abbreviated Scale of Intelligence PPVT-IIIPeabody Picture Vocabulary Test-Version III HSGhigh school graduate CIconfidence interval LBWlow birth weight
Intensive early educational interventions have been documented to improve the cognitive outcomes1 and, in some cases, reduce antisocial behavior early in the school experience.2 In several single-site, randomized studies of center-based interventions, longer-term follow-up into adolescence and early adulthood has revealed persistence of early-intervention effects seen as greater school achievement,3,4 less grade retention and special education,3,5 and more high school completion6 and participation in college.4 In addition to better educational outcomes, early-intervention programs have also resulted in better social outcomes such as less adolescent parenting4 and less delinquency and fewer arrests.68 Parallel effects have been seen in follow-up of at least 1 home-visiting program9 and a nonrandomized program conducted in several centers in 1 city, the Chicago Child-Parent Center Program.10 However, all of these studies involve healthy, low-income, poor children generally from a single site. Thus, inferences about the generalizability of these results to groups more heterogeneous for socioeconomic status and health are limited.
The Infant Health and Development Program (IHDP) was a multisite, randomized, controlled trial of an educational intervention until 3 years of age for low birth weight (LBW) preterm infants. Infants were sampled from 2 birth weight strata (
2000 g and 20012500 g) to assure an oversample of those at higher developmental risk (the lighter LBW [LLBW] group) and continuity with studies of poor, normal birth weight children (the heavier LBW [HLBW] group) at 8 sites, and as a result, the sample for this study is quite heterogeneous for socioeconomic status and health status, particularly neurologic factors that might influence school outcomes. The intervention consisted of home visits every week for the first year and every other week in the second and third year, daily center-based education beginning at 12 months corrected for duration of gestation continuing until 36 months of age, with the addition of parent support groups. All of the children received intensive pediatric follow-up care. The program has been described in detail elsewhere.1113
The results of the trial through 8 years of age have been published previously1417 and will be summarized briefly here. The assessment of outcomes was conducted in 3 prior phases. At 36 months of age corrected for duration of gestation, at the end of phase I (the trial phase), the IQs of those who received the intervention (INT) were substantially higher than those who received follow-up services only (FUO): a 14-point difference for the HLBW group and 7 points for the LLBW group. Likewise, lower behavior problem scores were seen in the INT group. By age 5 (phase II, the preschool period) and age 8 (phase III, the early school period), no differences in behavior scores were observed. At both ages 5 and 8, the INT in the HLBW group still experienced a significant advantage in IQ scores (4 points higher), as well as mathematics achievement testing (4 points higher). No differences in IQ or achievement between INT and FUO groups were seen for the children in the LLBW group.
The IHDP is based on the design and curriculum of the Abecedarian Project,18 and the results in IQ scores and achievement testing parallel results seen in the Abecedarian Project18 at 8 years of age. More recent data from that project reveal persisting differences in IQ and achievement up to 21 years of age,19 prompting us to extend the follow-up of the IHDP to 18 years with a fourth wave of data collection (phase IV). The purpose of this effort is to ascertain whether such persistent differences would be observed in the IHDP, especially in the HLBW group. In line with other studies of normal birth weight children, we hypothesized that we would see persistent IQ and achievement differences and less risky behavior favoring the INT group.
| METHODS |
|---|
|
|
|---|
2500 g, a gestational age of
37 weeks, and reside in a catchment area defined by distance from the early educational center. Infants were enrolled between October 1984 and August 1985 in 2 birth weight strata:
2000 g, and 20012500 g. Within birth weight strata, infants were randomly assigned to the INT group (n = 377) or FUO group (n = 608). Although the sites were heterogeneous for a variety of characteristics, the randomization procedure resulted in comparability between the INT and FUO groups at study entry.14 Infants in both arms of the study received periodic medical, developmental, and social service assessments with referral for community services when indicated. Those in the INT group received an educational program delivered through home visits (weekly during the first year and every other week during the second and third years of life), a daily center-based program beginning at 12 months corrected for duration of gestation, and parent support groups coinciding with the start of the center-based program.11 These parent support groups met every other month. At the end of the INT period at 3 years of age, the children received whatever community education programs were available at the site. Although the availability of such services varied greatly among the sites, there were no differences between the INT and FUO groups in receipt of such services.15 As noted above, major assessment points for the intervention occurred at 3, 5, and 8 years of age. Completion rates at these ages were 92%, 82%, and 89%, respectively. The lower rate at age 5 reflects the lack of funding for staff to assess children who were not close to any study site.
Data Collection at 18 Years of Age (Phase IV)
Measures
The phase IV assessment at 18 years of age was designed to obtain information in the areas of academic achievement, behavior, cognitive abilities, and physical health. To the extent possible, instruments were selected to provide continuity with previous assessment periods and to have known references or to have been used in large population studies.
Academic Achievement
Academic achievement was assessed using the Woodcock-Johnson Tests of Achievement-Revised.20 These tests yielded letter-word identification, passage comprehension, calculation, and applied problems subtests of the standard scores. Two scores were calculated, the Woodcock-Johnson Tests of Achievement in reading and in mathematics, referenced to a mean of 100 and an SD of 15, with higher scores indicative of better achievement.
In addition, the youth's mother or other caregiver was asked about his/her progress in school. Specifically, these questions deal with completion of school, grade repetition, classification for special education, and school dropout.
Behavior
Two aspects of behavior were assessed: risk behaviors and more general behavioral problems. Risk behaviors were assessed using youth self-report on questions taken from the Youth Risk Behavior Surveillance System (YRBSS).21 Items from this survey were selected to characterize the youth along dimensions affected by early intervention in early studies: conduct problems, suicidal ideation/attempts, smoking, alcohol and marijuana use, and risky sexual activity (see the Appendix for the scoring algorithm). More general behavioral problems were ascertained using the Behavior Problem Index (BPI),22 completed by both the youth and the primary caregiver (the latter for comparison with earlier assessments). For both the YRBSS and the BPI, higher scores indicate more problems.
Finally, we asked the caregiver and the youth specific questions about youth involvement with the justice system. These questions included items on trouble with the police, arrests, and time in jail. A positive event was coded if either reported it.
Cognitive Development
Two measures of cognitive development were administered: the Weschler Abbreviated Scale of Intelligence (WASI)23 and the Peabody Picture Vocabulary Test-Version III (PPVT-III).24 The latter, a measure of receptive vocabulary, has been used in this sample at each assessment point since 3 years of age and was designated the primary outcome because of this continuity over time. Both tests are referenced to a mean of 100 and an SD of 15. The WASI provides 4 subscales (vocabulary, block design, similarities, and matrix reasoning) and can generate a verbal IQ and performance IQ, as well as a full IQ score, with only the last being considered here.
Physical Health
Physical health was characterized by youth self-report and caregiver report on the 36-Physical Component Summary measure.25 This scale reflects limitations in activities of daily living because of health, as well as positive health. It is referenced to a mean score of 50 and an SD of 10, with higher scores indicative of better health.
Of these measures, math and reading achievement tests, the risk behavior scale, youth self-report on the BPI, and the PPVT-III were considered the primary outcomes for the analysis. However, the results for the WASI, caregiver-reported behavior problem score, and self-report and caregiver report on physical health are provided for continuity with earlier reports.
Baseline Measures
As in previous analyses of the IHDP, a standard set of prerandomization variables was used to compare participants and nonparticipants in this assessment and to adjust the outcomes for differences across sites and birth weight strata. These variables included birth weight in grams, maternal age in years, race/ethnicity (black, Hispanic and white/other), gender of child (male/female), maternal educational attainment at the time of the birth of the child (less than high school graduate [HSG], HSG, and more than HSG), and Neonatal Health Index,26 a measure of neonatal length of stay in the hospital relative to birth weight, which serves as an indication of the severity of the neonatal course. This index is referenced to a mean of 100 and SD of 16. Baseline variables also included study site: Arkansas (Little Rock, AK), Einstein (South Bronx, NY), Harvard (Boston, MA), Miami (Miami, FL), Pennsylvania (Philadelphia, PA), Texas (Dallas, TX), Washington (Seattle, WA), and Yale (New Haven, CT). Gestational age at birth was also available.
Assessments at Age 8
In addition, we compared participants and nonparticipants for the cognitive, achievement, and behavioral assessments done at 8 years of age. These measures included the Wechsler full-scale IQ score,27 achievement scores in math and reading,20 the PPVT-Revised,28 and behavior problem scores.29
Analysis
Participants and nonparticipants at the 18-year assessment were compared on the prerandomization variables and 8-year outcomes noted above. A subject was considered a participant if any of the outcomes of interest were observed at the 18-year visit and as a nonparticipant otherwise. The comparison of the differences between participants and nonparticipants in the INT and FUO groups were evaluated by using the t statistic for continuous measures and
2 statistics for categorical measures.
To compare the INT and FUO groups on the outcomes of interest at age 18, separate linear (analysis of covariance) models were developed for each outcome variable within each of 2 birth weight strata using the S-plus software described by Venables and Ripley.30 The same set of 7 baseline variables measured at enrollment as noted above was used in each model of the comparison of the INT versus FUO. Multivariate models were used to estimate the differences between the INT and FUO groups to adjust for differences in sample sizes among the sites. 14 The linear models were used for estimation of population marginal mean values and differences (INT versus FUO) for each outcome. The estimated marginal means were obtained by substituting into the linear model the average values of the covariates for a particular birth weight stratum. Nominal P values derived from the linear models were used for primary outcome comparisons.
We evaluated the robustness of these conventional estimates of the difference between the INT and FUO groups by comparing them with the results of the analysis of covariance where each of the respondents was weighted by the inverse of the outcome-dependent estimated propensity score31 to assess the effect of attrition. We assumed that the propensity score (which, in our analysis, is the outcome-dependent probability of observing an outcome of interest at the 18-year visit) depended on the values of the full-scale IQ and child behavior checklist at the 8-year visit and baseline characteristics of a subject. Thus, for example, subjects from Harvard with low IQs at age 8 were less likely to participate at age 18, and the average participant with these characteristics would be assigned higher weights (more influence) in the propensity-adjusted analyses of the difference between the INT and FUO groups using this approach. Under the modeling assumptions, this method removes nonresponse bias.
Human Subjects
The study was approved by the institutional review boards of all participating institutions. In addition, certification of confidentiality was obtained for this phase. Written informed consent was obtained separately for all of the participants: study subject and caregiver.
| RESULTS |
|---|
|
|
|---|
|
|
2000 g in comparisons adjusted for the baseline variables only (Table 3 line "a" for each outcome), although a difference in reading achievement scores favoring the FUO group achieved P = .08. When the results were also adjusted for attrition (Table 3, line "b"), the difference in reading favoring the FUO group was strengthened.
|
In contrast, among those >2000 g, there were statistically significant elevations in PPVT-III and math achievement scores favoring the INT group in the results adjusted for the baseline covariates. The INT group experienced 5- and 6-point higher reading and math achievement scores, respectively, with the latter at P = .01. No statistically significant difference was seen in self-reported behavior problems or the risk behavior scale from the YRBSS in these analyses. When the comparisons were adjusted for attrition, the difference in reading achievement diminished but that for the math achievement remained statistically significant. A difference was seen favoring the INT group on the YRBSS score as having fewer risky behaviors (P = .05). When the INT and FUO groups were compared on the measures of secondary outcomes, no statistically significant differences were seen, although the difference favoring the INT group on the full-scale IQ score among the HLBW group (Table 4) had a P value of .07.
|
4 points, but this difference has remained stable and consistent over time. Likewise, the differences in PPVT scores favoring the INT have persisted after age 3. Changes in the achievement scores in this group were generally parallel while exhibiting the expected decline in achievement scores with aging into the late teen years. The figure also reveals that results from the current phase suggesting that small differences favoring the FUO group in PPVT, IQ, and math achievement scores also emerged early and persisted.
|
We also explored differences in specific behavioral and school outcomes. Among the INT LLBW group, 16% had been arrested compared with 20% of the FUO group (with P = .11, adjusted for baseline covariates and attrition). The comparable figures for the HLBW group was 21% and 26%, respectively (P = .56). Approximately 12% to 13% of both birth weight groups had already been in jail with no differences between the INT and FUO groups. Among the LLBW group, 10% of the INT group and 8% of the FUO group were classified as having dropped out of school (P = .40); in the HLBW group, it was 11% and 12%, respectively (P = .58). With regard to ever being classified for special education, 30% of INT group and 26% of the FUO group in the LLBW stratum had been so classified (P = .39) and 17% and 24% of the HLBW stratum (P = .12).
| DISCUSSION |
|---|
|
|
|---|
2000 g. These results parallel those seen at 8 years of age.16 As expected, IQ and achievement scores were lower and behavior problem scores higher than expected population references, reflective of patterns seen among those who were LBW and/or premature at birth.32
Our rates of juvenile arrest at age 18 are comparable with those the seen in the Chicago study,10 but the difference between the INT and FUO groups is smaller, 4% to 5% as compared with 8% in the Chicago study, where the difference did achieve statistical significance. Likewise, our incarceration rates are similar to those reported in the Abecedarian intervention group4 but somewhat lower than the rates in the control group. The differences in incarceration rates also fail to achieve statistical significance. Comparisons with the Perry Preschool Project cannot be made, because they report the percentage with
5 arrests at rates substantially higher than arrest records in other studies.
Comparison of our educational outcomes with other studies is complicated by the fact that our sample has not yet established a posthigh school educational pattern in terms of enrollment in higher education as reported in Abecedarian,4 and even final high school graduation rates are unknown. Our dropout rate is substantially lower than the
50% seen in the Chicago study,10 as is the difference between intervention and control groups. Our rates of special education and differences between INT and FUO in the HLBW group are similar to those in the Chicago study, where the difference was highly significant.
In this analysis, we have not examined outcomes by prerandomization variables other than birth weight. This decision reflects the results at age 8 where no difference in effect size is seen at different levels of maternal education in contrast to a hypothesized greater effect of the intervention for children from more disadvantaged backgrounds.16
The persistence of an intervention effect in the HLBW group provides substantial reinforcement to the emerging literature on longer-term effects of early educational intervention.49 First, the IHDP was implemented in 8 sites and is modeled on the Abecedarian Project,4 arguing for replicability of the results. Second, the IHDP has been characterized as methodologically rigorous with strict randomization and masked assessments at major outcome points at all of the follow-up phases and has avoided some of the concerns associated with other projects.8 In contrast to previous studies, which involved largely poor, healthy children, the IHDP expands the results of early educational intervention in 2 ways. First, the sample is heterogeneous with regard to health, especially neurodevelopmental status, by virtue of it being a sample of preterm, LBW children. Although not all LBW children sustain neurologic injury,32 being LBW confers increased risk of neurodevelopmental disability that may make academic achievement difficult. For example, in unpublished analyses, children with cerebral palsy in the IHDP experienced no advantage in cognitive function at age 3 from the intervention. Second, the IHDP was more heterogeneous with regard to socioeconomic status than previous studies. We have documented that children of mothers with some college or more education did not benefit from the intervention immediately at the end of the trial.33 Thus, although the effects sizes are modest, they reinforce the existing literature because they reflect a more heterogeneous study population biased against finding a difference. It should be noted that the pattern of persistence seen in Fig 1, including diminishing performance on achievement testing with age, closely parallels results for the Abecedarian Project.18
As in the 8-year assessment, questions might be raised about the significance of a 4-point difference in achievement or the PPVT-III. However, it should be noted that the difference in mean scores also indicates a shift in the entire distribution in favor of the INT group. In the past, we have argued that such a shift would decrease the percentage of children classified as intellectually deficient.16 Others have also argued the importance of such differences and the potential to alter the distribution of those who will do poorly in school.34
The lack of persistence of an effect in the LLBW children is also relevant to this emerging literature in that it raises questions about greater attention to the factors that foster or inhibit sustained effects. The LLBW groups did initially benefit from the intervention as indicated by a 6.6-point difference in IQ scores favoring the INT group at 3 years of age.14 Moreover, the lack of sustained effect was not because of the experience of very premature or very small infants, because the same effect size is seen among those
1500 g and even
1000 g in birth weight.35 However, differences between the INT and FUO groups among the LLBW children disappear by age 5 years.15 One analysis suggests that the intensity of participation may be a factor in that sustained effects at age 8 were seen among those attending >400 days of center-based education (out of a maximum of 500) during the intervention period.36 In addition, we speculate that earlier entry into special education may also play a role and needs additional investigation.
Several limitations affect the interpretation of our study. The first is the lower response rate than previous rounds of assessment in the IHDP, particularly with loss among more disadvantaged children and those study participants with lower previous cognitive development. Our adjustment of the results for these factors suggests that the losses to the cohort do not significantly affect the results, but differences may have occurred in unmeasured factors.
Second, the age of assessment, 18 years, may not be optimal in identifying economically important outcomes, such as higher educational attainment, definitive occupational status, and eventual income. In the Perry Preschool Project, economically significant differences are seen at 27 and now 40 years of age.6,8 Even at 21 years of age, the Abecedarian Project4 did not detect significant differences in employment but, in results parallel to ours, did find differences in risk behaviors. They did, however, find an intervention advantage in the proportion enrolled in a 4-year college or university.
Third, because our study sample is not all disadvantaged, the power to detect significant differences in some areas is limited. As noted earlier, arrest rates in the Perry Preschool Project8 are substantially higher than other longer-term follow-up studies of early education. With all rates of adverse outcomes, such as arrest or special education classification, differences favoring the INT group, especially the HLBW youth, are either similar or about half the effect size. Thus, we would have needed a substantially larger sample, in the order of the 1500 in the Chicago study,10 to achieve statistical significance. This finding also suggests caution in extrapolating cost-effectiveness arguments of early educational intervention based on reductions in costs among populations with less risk of poor educational attainment and criminal behavior.
Despite these limitations, this phase of the IHDP provides important support for the efficacy of early educational interventions in the longer-term outcomes of children. It also presents significant challenges in identifying those who would most benefit and those who may need continuing support to achieve academically. However, the results provide support for extending such educational opportunities to a broader spectrum of children than included in previous studies.
| APPENDIX |
|---|
|
|
|---|
15% of the group fell into the highest-risk group, the majority (>60%) in the low-risk group, and the remainder in the moderate-risk group. We summed these area-dependent scores to produce an overall score of risky behavior. This approach parallels that of Flisher et al,37 in a report of behavior in a community sample of children and adolescents.1 Of the 636 subjects, data for this analysis was missing on 5%. The resulting score had a mean of 2.4 in our study population, with an SD of 2.6 and a range of 011. Initial bivariate comparisons of INT and FUO groups were done, therefore, using nonparametric techniques. Adjusting for baseline covariates and attrition, however, relied on parametric statistics as described in "Methods."
| ACKNOWLEDGMENTS |
|---|
For previous participants in the IHDP, see refs 1416.
| FOOTNOTES |
|---|
Address correspondence to Marie C. McCormick, MD, ScD, Department of Society, Human Development and Health, Harvard School of Public Health, 677 Huntington Ave, Boston, MA 02115. E-mail: mmccormi{at}hsph.harvard.edu
The authors have indicated they have no financial relationships relevant to this article to disclose.
| REFERENCES |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
A.-C. Roch-Levecq, B. L. Brody, R. G. Thomas, and S. I. Brown Ametropia, Preschoolers' Cognitive Abilities, and Effects of Spectacle Correction Arch Ophthalmol, February 1, 2008; 126(2): 252 - 258. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. M. Cote, M. Boivin, D. S. Nagin, C. Japel, Q. Xu, M. Zoccolillo, M. Junger, and R. E. Tremblay The Role of Maternal Education and Nonmaternal Care Services in the Prevention of Children's Physical Aggression Problems Arch Gen Psychiatry, November 1, 2007; 64(11): 1305 - 1312. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Melchior, T. E. Moffitt, B. J. Milne, R. Poulton, and A. Caspi Why Do Children from Socioeconomically Disadvantaged Families Suffer from Poor Health When They Reach Adulthood? A Life-Course Study Am. J. Epidemiol., October 15, 2007; 166(8): 966 - 974. [Abstract] [Full Text] [PDF] |
||||
![]() |
C. S. Minkovitz, D. Strobino, K. B. Mistry, D. O. Scharfstein, H. Grason, W. Hou, N. Ialongo, and B. Guyer Healthy Steps for Young Children: Sustained Results at 5.5 Years Pediatrics, September 1, 2007; 120(3): e658 - e668. [Abstract] [Full Text] [PDF] |
||||
![]() |
H. Hix-Small, K. Marks, J. Squires, and R. Nickel Impact of Implementing Developmental Screening at 12 and 24 Months in a Pediatric Practice Pediatrics, August 1, 2007; 120(2): 381 - 389. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. M. Black, H. Dubowitz, A. Krishnakumar, and R. H. Starr Jr Early Intervention and Recovery Among Children With Failure to Thrive: Follow-up at Age 8 Pediatrics, July 1, 2007; 120(1): 59 - 69. [Abstract] [Full Text] [PDF] |
||||
![]() |
H. G. Herrod Do First Years Really Last a Lifetime? Clinical Pediatrics, April 1, 2007; 46(3): 199 - 205. [Abstract] [PDF] |
||||
![]() |
Meetings and Activities Clinical Pediatrics, April 1, 2007; 46(3): 292 - 292. [Abstract] [PDF] |
||||
![]() |
P. H. Casey, L. Whiteside-Mansell, K. Barrett, R. H. Bradley, and R. Gargus Impact of Prenatal and/or Postnatal Growth Problems in Low Birth Weight Preterm Infants on School-Age Outcomes: An 8-Year Longitudinal Evaluation Pediatrics, September 1, 2006; 118(3): 1078 - 1086. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Olds Progress in Improving the Development of Low Birth Weight Newborns Pediatrics, March 1, 2006; 117(3): 940 - 941. [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||