PEDIATRICS Vol. 118 No. 6 December 2006, pp. e1805-e1811 (doi:10.1542/peds.2006-0210)
ARTICLE |
Pediatric Residency Duty Hours Before and After Limitations
a American Academy of Pediatrics, Elk Grove Village, Illinois
b Office of Pediatric Medical Education, University of Connecticut, Hartford, Connecticut
c Department of Pediatrics, Division of Hematology-Oncology, Texas Children's Hospital, Houston, Texas
d Department of Pediatrics, University of California, Davis, Sacramento, California
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
OBJECTIVES. The goals were to examine pediatric resident and program director experiences implementing the Accreditation Council for Graduate Medical Education work hour limits and to compare duty hours, moonlighting, and fatigue before and after the limits became effective.
METHODS. National random samples of 500 pediatric residents who graduated in 2002 and in 2004 were surveyed to compare resident duty hours and fatigue before and after the Accreditation Council for Graduate Medical Education limits were implemented. In addition, all US pediatric residency program directors were surveyed at the end of the 2003/2004 academic year, to provide a complementary retrospective examination of limit implementation.
RESULTS. Totals of 65%, 61%, and 83% of 2002 residents, 2004 residents, and program directors, respectively, responded. The proportion of residents who reported working >80 hours per week declined from 49% for NICU/PICU rotations before the limits to 18% after limit implementation. Resident well-being was the factor identified most often by both residents and program directors as being improved since the limitations. Multivariate modeling also showed reductions in the proportions of residents who reported falling asleep while driving from work or making errors in patient care because of fatigue. Overall, 89% of pediatric residents and program directors reported that the current system is effective in ensuring appropriate working hours.
CONCLUSIONS. Since the Accreditation Council for Graduate Medical Education duty hour limits went into effect, pediatric residents report working fewer hours and making fewer patient care errors because of fatigue. Although room for additional improvement remains, the experiences of residents and program directors suggest that implementation of the Accreditation Council for Graduate Medical Education limits in pediatric residency programs is improving resident well-being.
Key Words: duty hour limits • residency training • resident fatigue
Abbreviations: ACGME—Accreditation Council for Graduate Medical Education • OR—odds ratio • CI—confidence interval • AAP—American Academy of Pediatrics
On July 1, 2003, new Accreditation Council for Graduate Medical Education (ACGME) resident duty hour standards went into effect, introducing a major change in graduate medical education.1 Concern for residents' well-being and errors in patient care associated with resident fatigue were the primary reasons for controlling resident duty hours.2,3
Many questioned whether the new rules would leave residents unprepared for future clinical responsibilities.4–6 Others questioned whether patient safety problems associated with reduced continuity of care would outweigh the potential safety benefits associated with reduced fatigue.4,5,7,8 It was also suggested that residents might increase their external moonlighting, resulting in unchanged fatigue and associated medical errors.9
Several more-recent studies yielded findings supporting elements of the ACGME limits. Interns working extended shifts were more likely to be involved in motor vehicle crashes,10 to have attentional failures at work,11 and to make serious medical errors.12 Not all studies showed, however, that system changes designed to reduce sleep deprivation and fatigue resulted in improved patient safety.13,14 In 2 recent retrospective survey studies, program directors and residents reported that duty hour reforms have had an overall negative impact on resident training and continuity of patient care.15,16
Ensuring that residents are educated to provide optimal care remains a high priority for the pediatric community,17 and the effect of duty hour limitations within pediatrics has yet to be examined systematically. The goals of our study were to determine whether pediatric training programs have been able to implement the new limitations and to examine the perceived impact of the limitations on key aspects of residency training, including resident well-being, quality of resident education, and quality of patient care. Specifically, we compared program director and resident experiences with duty hour limitations. We also compared duty hours, moonlighting, and fatigue consequences before and after duty hour limitations.
|
METHODS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Data Sources
Two primary data sources (an American Academy of Pediatrics [AAP] survey of pediatric program directors and the annual AAP survey of graduating residents) were used to gather information about pediatric resident duty hours and moonlighting. All surveys were reviewed and approved by the AAP institutional review board.
Pediatric Program Director Survey
In cooperation with the Association of Pediatric Program Directors, we surveyed all of their members (194 individuals) at the end of the 2003/2004 residency year. The 3-page survey was mailed to program directors up to 3 times, depending on whether they responded, and no remuneration was offered. The survey focused on program directors' experiences implementing the ACGME duty hour limitations. Program directors were asked about actions taken to implement the limitations, current levels of duty hours and moonlighting in their program, and their views regarding the effects of the limitations.
Graduating Resident Survey
Each year, the AAP surveys a national random sample of 500 residents, of the 
2600 categorical pediatric residents graduating from ACGME-accredited residency programs, regarding their training and job search experiences.18 Between May and August, surveys are mailed to residents up to 4 times, depending on whether they respond. The US Postal Service is relied on to forward the surveys to nonrespondents for the last 2 mailings, which occur after the July 1 end-of-year turnover. No remuneration is provided. Each year, the survey includes a core set of training and job search questions and a subset of theme-based questions. In 2002 and 2004, the theme-based questions focused on resident duty hours, moonlighting behavior, and problems associated with fatigue, such as falling asleep in conferences, falling asleep while driving, and errors in patient care. Questions were developed in consultation with both residents and program directors. The survey procedures used in 2002 and 2004 were identical, except that the total survey length was increased from 4 pages to 6 pages in 2004. New questions asked in 2004 included several addressing residents' experiences with implementation of the duty hour limitations within their program. All variables collected in this study came from self-reports of residents or program directors. No independent measures of work hours, fatigue, or resident preparation were collected as part of this study.
Statistical Analyses
By design, there were many identical questions within the 3 surveys. For the questions that did not overlap, such as questions to program directors about how they chose to implement duty hour changes, simple descriptive statistics such as frequencies and means were used to present results and 
2 tests were used to compare the experiences of program directors from larger (
15 residents per year) and smaller programs.
Several questions appeared on both the 2004 graduating resident survey and the 2004 program director survey. Without any correction, resident data would be weighted toward the experiences of larger programs that have more residents, whereas the program director data would weight all programs equally. To adjust for this, we aggregated the resident data to the program level so that, for both the resident and program director data sets, there was a single case representing every program. For continuous variables, a mean of the values provided by the residents was used as the program-aggregated value. For dichotomous categorical variables, the value was set to 1 for a program if the majority of residents responded affirmatively to the question. Independent group t tests and 2 tests were used to test for differences between the program directors and the residents using the aggregated program-level data. A mixed-design analysis of variance was also conducted to examine duty hours and days off according to type of rotation and type of survey (program director versus resident).
Finally, many questions were asked on both the 2002 and 2004 graduating resident survey. This before-and-after comparison did not include any responses from 2003 graduates who were in the transition year when the limitations had been announced but had not gone into effect. Comparisons between the 2002 and 2004 data were made at the individual resident level. Independent group t tests and 2 tests were used to test for differences across these survey years that bracketed the effective date of July 1, 2003.
In addition, simultaneous multivariate logistic regression models were developed for 3 resident outcomes, namely, falling asleep during an educational conference, falling asleep while driving from work, and making an error in patient care that resulted in additional tests or procedures, prolonged hospitalization, or injury to the patient. The stem of this question asked, "Has fatigue from work caused you to do any of the following in the past year?" The models were created to examine whether reductions across survey years held when potentially important differences in resident characteristics between the year cohorts were controlled statistically. The resident-characteristic covariates entered into the models included gender, age, marital status, having children, residency program size, and having moonlighted in the past year. It is possible, for example, that residents who had greater family responsibilities or who moonlighted more had greater likelihood of fatigue.
A P value of 
.05 was used for all inferential tests. The number of cases for each analysis varied slightly, depending on the number of missing values.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Response Rates
A total of 161 program directors (83%) responded to the program director survey. Using American Medical Association Freida-Online data on program size, we found mean program sizes to be similar for respondents and the full target group (P = .217).
For the AAP survey of graduating residents, 323 (65%) of the residents surveyed responded in 2002 and 303 (61%) responded in 2004. Using the AAP membership database, we found that the distributions of gender (P = .682) and age (P = .984) were similar for respondents and the full target samples in 2002. In 2004, the age distributions were similar (P = .380) but there was a larger proportion of women among respondents than in the full target sample (69% vs 65%; P = .019).
Program Director Reactions (2004 Program Director Survey)
Of the program directors who responded to the survey in 2004, 42% reported that their overall reaction to the new ACGME limits on resident duty hours was positive or very positive, 34% reported that they were neutral, and 25% reported negative reactions. Program directors from larger (15 residents per year) and smaller programs were equally likely to report negative reactions (P = .913). Just more than one half (56%) of program directors indicated that they had minimal or no difficulty in adapting their programs to comply with ACGME limits. Program directors from larger programs were more likely to report minimal or no difficulty (64%) than were program directors from smaller programs (51%), but this difference was not significant (P = .103). Only 6% of all program directors reported considerable difficulty. The most frequent changes that directors made were asking attending physicians to work longer hours (45%), implementing night float (42%), and hiring nonphysician providers (38%). Modifying night float (27%) and hiring more attending physicians (19%) were reported less commonly.
Approximately one fourth (24%) of all program directors thought that resident sleep deprivation probably or definitely was a problem in their program, and 8% reported that a resident had asked to leave work because of fatigue during the past year. When asked about possible future negative impact of the limitations on duty hours, the majority of program directors reported that they were concerned or very concerned about the preparation of residents for practice reality (64%), the quality of resident education (56%), and staffing to meet service needs (51%).
Program Director and Resident Comparisons (2004 Program Director and 2004 Resident Surveys)
Residents and program directors both provided estimates of the average number of hours that residents worked per week (Fig 1) and the average number of complete days off per month (Fig 2) in 2004. Both groups reported greater work hours for ward (mean: 73.6 hours; SD: 6.7 hours) and NICU/PICU (mean: 75.9 hours; SD: 6.8 hours) rotations than for the emergency department rotation (mean: 56 hours; SD: 10.5 hours; P < .001). Residents and program directors also reported fewer days off per month for ward (mean: 4.5 days; SD: 1.5 days) and NICU/PICU (mean: 4.5 days; SD: 1.9 days) rotations than for the emergency department rotation (mean: 7.6 days; SD: 3.6 days; P < .001). Program directors' estimates of work hours (mean: 67.3 hours; SD: 5.5 hours) were slightly lower than residents' estimates (mean: 70.1 hours; SD: 5.6 hours; P < 001).
|
|
Residents and program directors also assessed how things had changed in 6 key areas since the ACGME limits on duty hours became effective (Fig 3). The 2 groups agreed that resident well-being was the area that had improved most. A total of 57% of residents, compared with 43% of program directors, reported that they thought resident well-being had improved, and <5% of each group thought resident well-being had worsened. Continuity of patient care was the area that had worsened most. For 5 of the 6 areas examined, residents were significantly more positive than program directors in their assessments of changes. For the quality of patient care, the direction of change was opposite for resident and program director assessments. Residents thought that the quality of patient care had improved slightly, whereas program directors thought that it was slightly worse.
|
The overwhelming majority of residents (89%) and program directors (89%) reported that the requirements limiting duty hours were effective or very effective in ensuring appropriate working hours for residents at their hospital (Table 1). There were, however, some topic areas where resident and program director responses were discordant. For example, 74% of program directors, compared with only 26% of residents, reported that any educational information or formal presentations on the effects of sleep deprivation were provided to residents. Residents were also less likely to report that they had faculty-appointed or resident-selected representatives to address resident work hours at their hospital. Program directors (68%) were more likely than residents (33%) to report that program director approval was needed for moonlighting. Program directors were much less likely than residents to report that federal legislation should regulate resident duty hours (6% vs 70%) or moonlighting (14% vs 38%).
|
Before-and-After Comparisons of Resident Responses (2002 and 2004 Resident Surveys)
In a separate evaluation, we compared resident responses from 2004 graduates with those from 2002 graduates. These groups were very similar to each other with respect to several demographic characteristics, including female gender (2002: 69%; 2004: 69%; P = .882), mean age (2002: 31.2 years; 2004: 31.3 years; P = .491), under-represented minority status (2002: 15%; 2004: 13%; P = .553), international medical graduate status (2002: 15%; 2004: 14%; P = .913), being married (2002: 68%; 2004: 71%; P = .447), and having children (2002: 33%; 2004: 35%; P = .617).
Changes were apparent in the number of work hours and days off that residents reported (Table 2). The proportions of residents who reported that they worked >80 hours per week decreased from 49% to 18% for NICU/PICU rotations and from 33% to 14% for ward rotations between 2002 and 2004. Similarly, significant decreases were found in the number of residents who reported that they received <4 complete days off per month. Moonlighting reported by residents did not increase. In fact, more residents reported that their program had restrictions on moonlighting and the number of residents who moonlighted regularly (in 6 of the past 12 months) decreased significantly in 2004.
|
Improvements across years were also apparent for the fatigue consequence measures. The proportion of residents who reported that they fell asleep during an educational conference was reduced from 81% to 73% (P = .018), the proportion of residents who reported that they fell asleep while driving from work was reduced from 32% to 20% (P < .001), and the proportion of residents who reported that they made an error in patient care because of fatigue that resulted in additional tests or procedures, prolonged hospitalization, or injury to the patient was reduced from 17% to 8% (P < .001). These reductions in fatigue consequences held within multivariate analyses. With control for several resident characteristics and moonlighting behavior, the odds were still significantly reduced across survey years for falling asleep during an educational conference (odds ratio [OR]: 0.77; 95% confidence interval [CI]: 0.62–0.94), for falling asleep while driving from work (OR: 0.74; 95% CI: 0.61–0.90), and for making an error in patient care (OR: 0.66; 95% CI: 0.50–0.86).
Residents in 2004 were also much more likely than residents in 2002 to conclude that the current system is effective or very effective in ensuring appropriate working hours for residents at their hospital (87% vs 46%; P < .001) (Table 2). No concomitant decrease was apparent in residents' self-perceived level of preparation for postresidency activities. Actually, more residents in 2004 than in 2002 thought that their preparation for primary care pediatric practice was very good or excellent (67% vs 58%; P = .028).
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
The ACGME duty hour limitations seem to have had many intended effects on pediatric residency programs. Residents reported working substantially fewer hours, reported falling asleep while driving from work less often, and reported making fewer patient errors. Residents and program directors agreed that resident well-being had improved19 and that the new system is more effective in ensuring ACGME-mandated working hours for pediatric residents.
Despite the improvements, the absolute levels of problems associated with resident fatigue remain concerning. Nearly three fourths of residents reported falling asleep in an educational conference and 1 of every 5 residents reported falling asleep while driving because of work-related fatigue. Furthermore, the number of fatigue events occurring is potentially even greater, because some residents might have had repeat occurrences. It would be beneficial to monitor resident well-being and resident education in other countries, where residents commonly work much less than 80 hours. It is too early to know what number of work hours is optimal for residency education.
We considered possible unintended consequences of the limitations on duty hours but found no evidence of increased moonlighting among residents. Our results were mixed concerning possible decreases in the quality of resident education. Consistent with surveys in other specialties,15,16 the retrospective examination of program directors (and to a lesser degree of residents) suggested that there might have been a decrease in the quality of resident education. In contrast, residents' self-perceived preparation ratings for both primary care practice and subspecialty fellowship training were actually higher for residents who graduated after, rather than before, the duty hour limits became effective. These results highlight the importance of having valid systems for measuring and evaluating clinical competence for all residents.
Both residents and program directors thought that the continuity of patient care had worsened some, but residents, in contrast to program directors, thought that the improvements associated with reduced fatigue outweighed the lesser continuity of care, producing better quality of patient care overall. It will be important to monitor patient outcomes and patient satisfaction now that the limitations are part of the fabric of residency training.
Responses of graduating residents and program directors converged generally, with several key exceptions. First, the institutional requirements of the Residency Review Committee for Pediatrics state that all programs must have a written policy on moonlighting and that the policy must require a written statement of permission to moonlight from the program director. However, 67% of residents and 32% of program directors reported that these requirements were not followed. Second, the Residency Review Committee requirements indicate that committees established to monitor all aspects of residency education should have residents nominated by their peers as voting members. Therefore, the fact that 10% of residents reported a lack of representation is at odds with policy. Third, the Residency Review Committee requirements state that faculty members and residents must be educated to recognize the signs of fatigue, but three fourths of residents and one fourth of program directors reported that no educational information on fatigue was presented to residents. These results suggest an overall need for improved communication of residency requirements. Improved communication is needed to ensure that program directors understand the requirements and that program directors work actively to increase resident awareness.
Despite consensus among residents about the improvement in their well-being, 7 of 10 thought that resident duty hours should be further regulated by the federal government. This may suggest that residents do not think that the improved levels with the limitations are satisfactory. It is also possible that residents are skeptical regarding whether the early improvements will be maintained without federal regulation. This is one area where residents and program directors are not in agreement, and it remains to be seen whether the positive outcomes of the ACGME limits will be sufficient to dissuade lawmakers from instituting federal legislation regarding duty hours.
Another potential concern highlighted in the program directors' responses is the carryover impact of the duty hour limitations on attending physicians. Nearly one half (45%) of program directors reported that attending physicians at their institution were being asked to work longer hours. It will be important also to consider the fatigue and well-being of attending physicians, as well as medical students, in future discussions of resident work hours.
There are several limitations to our study. First, all data were self-reported, and we have no independent indicators of resident fatigue levels or resident preparation for postresidency activities. Moreover, many questions asked residents to report on their experiences over the past year or on their impressions of change across their entire residency, and these questions were especially vulnerable to self-report recollection errors. Second, we had no way to assess the severity of the patient care errors reported by residents. Third, some nonresponse bias was found in 2004, with women being slightly more likely to respond to the survey. This tendency is consistent with previous AAP surveys, for which the nonresponse bias was examined and found to be minimal.20 Finally, our cross-sectional study design could not rule out alternative historical explanations for the observed changes, beside the ACGME duty hour limits.
|
CONCLUSIONS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
On the basis of resident and program director self-reports, many improvements have occurred since the ACGME duty hour limits went into effect. Although room for additional improvement remains, the experiences of residents and program directors suggest that the ACGME limits are being implemented effectively in pediatric residency programs and resident well-being has improved.
|
FOOTNOTES |
|---|
Accepted Jun 20, 2006.
Address correspondence to William L. Cull, PhD, Division of Health Services Research, American Academy of Pediatrics, 141 Northwest Point Blvd, Elk Grove Village, IL 60007. E-mail: wcull{at}aap.org
The authors have indicated they have no financial relationships relevant to this article to disclose.
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
- Philibert I, Friedmann P, Williams WT, Accreditation Council for Graduate Medical Education Work Group on Resident Duty Hours. New requirements for resident duty hours.
JAMA. 2002;288
:1112
–1114
[Free Full Text] - Accreditation Council for Graduate Medical Education. Statement of Justification/Impact for the Final Approval of Common Standards Related to Resident Duty Hours. Chicago, IL: Accreditation Council for Graduate Medical Education; 2003. Available at: www.acgme.org/dutyhours/impactstatement.pdf. Accessed October 19, 2006
- Steinbrook R. The debate over residents' work hours.
N Engl J Med. 2002;347
:1296
–1302
[Free Full Text] - Charap M. Reducing resident work hours: unproven assumptions and unforeseen outcomes.
Ann Intern Med. 2004;140
:814
–815
[Free Full Text] - Holzman I, Barnett S. The Bell Commission: ethical implications for the training of physicians. Mount Sinai J Med. 2000;67 :136 –139[Medline]
- Sectish T, Zalneraitis E, Carraccio C, et al. The state of pediatrics residency training: a period of transformation of graduate medical education.
Pediatrics. 2004;114
:832
–841
[Abstract/Free Full Text] - Drazen JM, Epstein AM. Rethinking medical training: the critical work ahead.
N Engl J Med. 2002;347
:1271
–1272
[Free Full Text] - Fletcher K, Saint S, Mangrulkar R. Balancing continuity of care with residents' limited work hours: defining the implications. Acad Med. 2005;80 :39 –43[CrossRef][Web of Science][Medline]
- Weinstein DF. Duty hours for resident physicians: tough choices for teaching hospitals.
N Engl J Med. 2002;347
:1275
–1278
[Free Full Text] - Barger LK, Cade BE, Ayas NT, et al. Extended work shifts and the risk of motor vehicle crashes among interns.
N Engl J Med. 2005;352
:125
–134
[Abstract/Free Full Text] - Lockley S, Cronin J, Evans E, et al. Effect of reducing interns' weekly work hours on sleep and attentional failures.
N Engl J Med. 2004;351
:1829
–1837
[Abstract/Free Full Text] - Landrigan C, Rothschild J, Cronin J, et al. Effect of reducing interns' work hours on serious medical errors in intensive care units.
N Engl J Med. 2004;351
:1838
–1848
[Abstract/Free Full Text] - Fletcher K, Davis S, Underwood W, et al. Systematic review: effects of resident work hours on patient safety.
Ann Intern Med. 2004;141
:851
–857
[Abstract/Free Full Text] - Veasey S, Rosen R, Barzansky B, Rosen I, Owens J. Sleep loss and fatigue in residency training: a reappraisal.
JAMA. 2002;288
:1116
–1124
[Abstract/Free Full Text] - Reiter ER, Wong DR. Impact of duty hour limits on resident training in otolaryngology. Laryngoscope. 2005;115 :773 –779[CrossRef][Web of Science][Medline]
- Cohen-Gadol AA, Piepgras DG, Krishnamurthy S, Fessler RD. Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery. 2005;56 :398 –403[CrossRef][Web of Science][Medline]
- Task Force on the Future of Pediatric Education. The Future of Pediatric Education II: organizing pediatric education to meet the needs of infants, children, adolescents, and young adults in the 21st century: a collaborative project of the pediatric community.
Pediatrics. 2000;105
:163
–212
[Free Full Text] - Cull WL, Yudkowsky BK, Shipman SA, Pan RJ. Pediatric training and job market trends: results from the American Academy of Pediatrics third-year resident survey (1997–2002).
Pediatrics. 2003;112
:787
–792
[Abstract/Free Full Text] - Fletcher K, Underwood W, Davis S, et al. Effects of work hour reduction on residents' lives: a systematic review.
JAMA. 2005;294
:1088
–1100
[Abstract/Free Full Text] - Cull WL, O'Connor KG, Sharp S, Tang SF. Response rates and response bias for 50 surveys of pediatricians. Health Serv Res. 2005;40 :213 –226[CrossRef][Web of Science][Medline]
PEDIATRICS (ISSN 1098-4275). ©2006 by the American Academy of Pediatrics
CiteULike 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
This article has been cited by other articles:
|
|
 |
|
  T. K. Nuckols, J. Bhattacharya, D. M. Wolman, C. Ulmer, and J. J. Escarce Cost Implications of Reduced Work Hours and Workloads for Resident Physicians N. Engl. J. Med., May 21, 2009; 360(21): 2202 - 2215. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 N. F. Schor Crossroads: Two Points of View: RESIDENT WORK HOURS: ISOLATING ONE LEDGER COLUMN AND MISSING THE POINT Neurology, July 29, 2008; 71(5): 374 - 374. [Full Text] [PDF]
|
|
|
|
 |
|
 S. Parthasarathy, K. Hettiger, R. Budhiraja, and B. Sullivan Sleep and Well-Being of ICU Housestaff Chest, June 1, 2007; 131(6): 1685 - 1693. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 AAP Division of Health Services Research and AAP Division of Graduate Medical Education and Ped Resident work-hour limits paying dividends, study shows AAP News, January 1, 2007; 28(1): 33 - 33. [Full Text]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
