When Children Die: A Seminar Series for Pediatric Residents
From the Department of Pediatrics, Steele Memorial Children’s Research Center, University of Arizona, Tucson, Arizona
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
Objective. Our goals were to assist residents in acquiring skills needed to care for children at the end of life and to increase their comfort level regarding critical aspects of caring for dying children.
Methods. We designed and implemented a 6-part seminar series for pediatric residents in their postgraduate year-2 at the University of Arizona Pediatric Residency Program. The series consisted of small group sessions regarding medical and legal issues facing physicians at the time of a pediatric patient’s death, symptom management in dying children, approaches to limitation of care and discussion of impending death of a child, cultural and developmental factors impacting on the care of children at the end of life, parent and sibling bereavement after the death of a child, and physician’s personal responses to the death of pediatric patients. The sessions were conducted on weekday evenings away from patient care settings and included input from community-based individuals as well as academic pediatric subspecialists. To promote active discussion, size was limited to 12 participants. Pretests and posttests were used to evaluate the success of this program. Residents’ level of comfort with important issues in end-of-life care was measured using a 5-point Likert scale.
Results. Although most of the residents taking part in the seminar series had previously participated in discussions of limitation of care of a child and/or had performed advanced life support measures on a child, few had any previous training in end-of-life care. Responses of the 8 participants who completed both the pretests and posttests were evaluated for assessment of changes in comfort level in areas pertinent to end-of-life care. We found that on completion of this program, residents were significantly more comfortable with logistic issues and symptom management, discussing death and limitation of medical care with families, discussing end-of-life care with colleagues and families, handling conflicts within the health care team, working with families of varying ethnic backgrounds, guiding developmentally appropriate discussions of death, identifying and seeking out advice from a role model regarding end-of-life care, and coping with their own responses to a child’s death. Costs of this seminar series were minimal.
Conclusions. An inexpensive, discussion-based seminar series successfully provided pediatric residents with basic information regarding end-of-life care and significantly increased their confidence as clinicians caring for seriously ill and dying children.
Key Words: medical education • pediatrics • palliative care • internship and residency • communication
Abbreviations: PGY, postgraduate year
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
During their residency training, pediatric residents are frequently involved in the care of seriously ill and dying children. They may be present when the diagnosis of a terminal illness is made or may be at the bedside when a child dies. Although general pediatricians in practice care for <3 children who die each year,1 data from the 1980s showed that during their first 2
years of training, pediatric residents were involved in the care of an average of 35 children who died.2 Based on the higher acuity of patients currently in inpatient settings, it is likely that residents now take care of an even greater number of children at the end of life. However, a recent study revealed that only 13% of the residents responding to a survey at Cornell Medical Center believed that they were adequately trained to deal with their dying patients.3 A distinct minority (8%) of these residents had received instruction relating to end-of-life care during their residency training. Although many medical schools have incorporated sessions regarding death, dying, and end-of-life care into their curricula, residents often do not receive training specifically designed to help them care for dying children once they begin their postgraduate years. A recent longitudinal study of pediatric residents’ attitudes showed that the house officers felt unprepared to deal with end-of-life issues as they began their training and believed that their education during the first 2 years of residency did not prepare them to deal with death and dying.4 By the third year of training, the residents involved in this study generally felt more comfortable with death and dying, but the authors suggest that this may be primarily attributable to increased exposure to end-of-life issues affecting patients and, importantly, affecting their own personal lives. Interestingly, the residents who felt unprepared to deal with dying patients expressed a desire for education regarding death and dying, but educational interventions such as spending time with chaplains, social workers, or mental health professionals were not considered useful by the trainees.4 Sahler et al1 have suggested that education about end-of-life care for children does not require a separate curriculum for trainees and instead should be taught at critical moments throughout training, and that ideally educational efforts in this area should be ongoing. However, it is clear that at this time pediatric residents continue to feel at a loss when confronting issues arising during end-of-life care for children. Ninety-five percent of residents and fellows responding to a recent survey believed that they could benefit from instruction in dealing with death and dying.3 It is our experience that bedside teaching about end-of-life issues can be highly variable, depending a great deal on the relationship between attending and trainee and on the perceived time constraints affecting all parties involved. In addition, as residents spend less time in intensive care and inpatient settings and more time in the ambulatory care environment, their opportunities for bedside teaching diminish. Thus, the need for an end-of-life care curriculum specifically designed for pediatric residents clearly exists.
End-of-life care curricula appropriate for medical students and residents have been developed by a variety of groups,5,6 but we were able to find only 1 report in the past 20 years regarding an educational intervention specifically designed for pediatric residents involved in the care of seriously ill and dying children.7 We now report our experience with a 6-part seminar series for second-year pediatric residents at the University of Arizona. Our objectives were to provide residents with factual information relating to the care of dying children and to help participants gain confidence in their ability to deal with the many issues that arise when children die.
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
The seminar series was designed for trainees who were close to the completion of their postgraduate year (PGY)-2 at the University of Arizona Pediatric Residency Program. Second-year residents were the target group for this intervention because they were expected to have participated in the care of a greater number of critically ill and dying patients than first-year residents. In addition, the more outpatient-oriented PGY-2 curriculum in our institution allows more time for processing and integrating of information and experiences than does the PGY-1 year. The series consisted of 6evening sessions, the content of which is described below and summarized in Table 1. Although attendance was voluntary, the residents who chose to take part were asked to commit to attending as many sessions as possible to allow for continuity and the development of a cohesive group dynamic. When feasible, call schedules were arranged so that service obligations would not prohibit continuing attendance. Twelve PGY-2 trainees were eligible to participate. The size of the group was limited to 12 residents; third-year residents were allowed to attend in place of any second-year resident who did not participate because of previous commitments or personal preference. Nine PGY-2 residents participated during the first year of this program and the remaining residents attending were PGY-3 residents. The pediatric chief residents and 2 pediatric faculty members planned and facilitated the seminar series.
|
Participants filled out pretests and posttests designed to determine the previous experience of participants in the care of children at the end of life and to measure level of comfort in the following 5 areas related to end-of-life care. These included discussion of end-of-life care, pursuit of information and advice regarding end-of-life care, symptom management in children at the end of life, management of issues arising at the time of the death of a child, and personal response to caring for children at the end of life. A 5-point Likert scale was used to assess residents’ level of comfort, with 1 being most comfortable and 5 being least comfortable. Responses to questions in each of the 5 areas were combined to form an overall score for each resident in each area. The scores in each area on pretests and posttests were compared using a Wilcoxon rank sum test because responses were not normally distributed. To ensure confidentiality, questionnaires were identified only by number. At the first session, participants were given binders containing reading material relevant to each session. The expense of this program was limited to photocopying costs. All guests kindly donated their time.
The first session introduced factual and logistic information pertinent to the physician’s role at the time of the death of a child. Using a case-based approach, the process of declaring death was discussed. Residents were encouraged to ask questions about such issues as documentation of brain death, postmortem care of the child’s body, logistics of arranging organ donation, and the completion of necessary paperwork. Participants were encouraged to ask questions relating to deaths of patients they had experienced during residency. They were then presented with a case involving a child dying at home, a scenario with which they were less familiar, but which they considered relevant to their future after residency.
The second session dealt with the medical management of dying children and focused on symptom management and comfort measures. A former pediatric intensive care nurse, the medical director of a community hospice, and a pediatric pulmonologist with training in complementary and alternative medicine facilitated the discussion. Cases were used to elicit discussion of treatment of pain, anxiety, seizures, and bowel and bladder problems. Nutrition and fluid management were also discussed.
The third evening seminar consisted of a roundtable discussion with a panel of local physicians considered by their peers to excel in the care of dying children. The panel consisted of a pediatrician in private practice, a pediatric oncologist, a pediatric intensivist, and a pediatric pulmonologist. Cases were again used to provide a framework for panelists and residents to interact regarding the delivery of bad news, discussion of limitation of care, and personal interactions with parents at the time of a child’s death and thereafter. Panelists discussed approaches to working through conflicts between family members and health care personnel as well as conflicts among family members themselves. Ample time for questions was allotted.
At the end of the roundtable session, residents were asked to talk during the week with family members and/or friends regarding the death of someone important in their own lives, paying specific attention to religious, ethnic, and cultural beliefs and practices relevant to the end of life. The next week’s session was attended by a faculty member with an advanced degree in cultural anthropology. Residents related their own experiences with death and dying and discussed the ways in which their backgrounds have shaped their interactions with patients from differing religious and cultural groups.
The father of a child who had died several years earlier after having been treated for cancer attended the fourth session. Also present were 2 social workers involved in community-based programs for bereaved families. The personal view of the parent who had been through the death of a child was complemented by the perspective of trained professionals able to describe the wide range of responses of families to serious illness and death of children. Developmental concepts of death were discussed in this session.
The final session was attended by the residents and their spouses or significant others. The discussion was facilitated by a psychologist with a personal history of multiply relapsed Hodgkins disease and a professional interest in working with physicians struggling with grief and loss. Housestaff and their partners were encouraged to explore the impact children’s deaths have on their own professional and personal lives.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
Although only 2 of the 10 residents who completed the pretest had any previous formal training regarding death and dying among pediatric patients, all of the residents had been involved in the care of a dying patient. Just 1 of the participants had any formal training regarding grief and bereavement before our seminar series, and 1 participant had previously had formal training regarding approaches to death in different cultures. Seven of the 10 residents who completed the pretest had participated in discussions of limitation of care of a child, and 8 of these 10 had performed advanced life support measures on a child. However, only half of the residents reported having had the opportunity to discuss approaches to terminal care of a child with an attending physician before the seminar series. Eight residents completed both the pretest and posttest, and their responses were considered evaluable. The remaining participants completed 1 but not both of the tests, and therefore changes in their levels of comfort with the topics of interest could not be evaluated.
Discussion of End-of-Life Care
On the pretest, evaluable responders expressed discomfort about discussing end-of-life care with families of pediatric patients. Specifically, they expressed discomfort about initiating conversations regarding impending death and limitation of medical care and were uncomfortable about discussing options for terminal care with families. In particular, they were uncomfortable about their ability to discuss death with families from a variety of ethnic and cultural backgrounds and about their ability to guide parents in developmentally appropriate discussions of death with children. They were also somewhat uncomfortable discussing end-of-life care with colleagues. Overall, participants were significantly more comfortable about discussing end-of-life care after the seminar series than they had been previously (Table 2). Pretest and posttest responses to specific questions regarding discussion of end-of-life care are shown in Table 3.
|
|
Seeking Information and Advice Regarding End-of-Life Care
At the start of the seminar series, residents were more confident in their ability to identify resources to assist them in caring for dying children than they were regarding other aspects of end-of-life care. They were somewhat uncomfortable about seeking help from the medical literature and from community resources regarding end-of-life care, but after the seminar series felt more confident about these pursuits. The single area in which residents felt comfortable at the start of the series was in identifying and seeking advice from a professional role model regarding management concerns. They expressed an even greater level of comfort regarding role models on conclusion of the series. Results are shown in Table 4. Overall, a trend toward a greater level of comfort with information gathering regarding death and dying was observed, but statistical significance was not achieved (Table 2).
|
Symptom Management in Children at the End of Life
Pretest responses revealed that residents were uncomfortable handling medical issues arising during end-of-life care of children. Specifically, their responses revealed discomfort regarding pain management, fluids/nutrition issues, and bowel and bladder care. They were particularly uncomfortable with management of seizures and anxiety. After the seminar series, residents were more comfortable with management in all of these categories, and overall a significant difference in comfort level with symptom control was observed (Tables 2 and 5).
|
Issues Arising at the Time of the Death of a Child
Before the series, residents expressed discomfort about pronouncing death and completing a legal death certificate. They were also uncomfortable addressing organ donation and autopsy with families of dying children as shown in Table 6. Posttest results show that they were significantly more comfortable about performing these tasks at the end of the series than they had been at the outset.
|
Personal Response to Caring for Children at the End of Life
Participants initially felt unprepared to handle conflict between members of the care team involved at the end of a child’s life, but indicated that they were more comfortable doing so after the seminar series. Improvement was also seen in residents’ assessment of their comfort in interacting with families after the death of a child. In addition, participants were more confident in their ability to cope with their own responses to the death of a pediatric patient at the end of the seminar series. Table 7 summarizes these findings. In addition, results shown in Table 2 indicate that, overall, residents’ answers to questions regarding their personal responses to patient deaths were significantly different on the posttest compared with answers on the pretest.
|
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
Our goals in this seminar series were to assist residents in acquiring skills needed to care for children at the end of life and to increase their comfort level in caring for dying children. The latter was particularly important to us because several authors have observed that physicians in training often avoid or withdraw from situations in which they feel uncomfortable.8,9 We chose to assess comfort with aspects of end-of-life care in our pretests and posttests because measurement of this endpoint enabled us to both evaluate our success in achieving our goals and obviated the need to have residents assess their own performance in situations with which their experience is variable. We believed that skill in performance of tasks involved in end-of-life care could not be adequately assessed by a written examination.
To achieve our goals, we created a seminar series with its own curriculum to provide participants with ample opportunities to discuss topics important for those who care for terminally ill children. The series was not designed to replace bedside teaching, but because demands on both attendings and residents often limit in-hospital time available to discuss end-of-life issues, we scheduled sessions that were separate from the regular working day. Residents were enthusiastic about having a specific time to focus on issues relevant to death and dying. One resident wrote, "A separate curriculum allows time for reflection... although bedside talks can be valuable, they typically are not well-suited for a lengthy and in-depth discussion. Furthermore, residents often have additional concerns when they are working (other patients, orders to write, labs to check) and may not feel they have the time to analyze a particular situation fully at the bedside." Data from the pretest support our approach. Residents felt fairly comfortable seeking role models in end-of-life care, indicating that they had had positive experiences at the bedside. Yet they were uncomfortable with many issues relating to the care of dying children, which suggests that their bedside experiences alone were not enough.
We sought to make participants as relaxed as possible as they grappled with difficult issues, and therefore held the seminars in the evenings away from patient care settings to create a less stressful environment. Our program, in contrast to the whole day event described in 1983 by Berman and Villarreal,7 spanned a period of 6 weeks to give residents the time to consider the material presented and follow-up with the group over time. Efforts were made to tailor the series to the culture of our own resident group. For example, other authors have asserted that role-playing exercises should be included in end-of-life curricula.3 However, residents in our program have reacted negatively to role-playing in the past, and many agree with one resident who commented, "Role playing can be stressful for the participants and feels very artificial." We therefore relied heavily on case-based discussions, and residents were remarkably active in their participation. Some of the residents felt that more factual material could have been conveyed with greater efficiency in a lecture format, but the majority preferred the more interactive approach. Sessions tended to run considerably over the time allotted given this format, but residents commented in their evaluations that they appreciated the flexibility that the loose structure afforded.
We chose to begin the series with very factual information because we believed that medical trainees would be accustomed to absorbing facts and mastering a given knowledge base. In addition, we were aware that our residents’ experience with death of pediatric patients was variable, and therefore began the series with sessions that would require less reflection on personal experiences. Residents responded favorably to the concrete nature of the sessions on declaring/documenting death and on medical management of dying children, as reflected in the comment by 1 resident that "a few simple tips gave me much more comfort." Another participant stated, "Many of these practical tools were not taught in any other setting during our residency."
Less factual and more personal matters were broached in a setting in which familiar and well-respected community and faculty attendings participated in a discussion regarding communication with families and staff about the death of a child. We purposely included a general pediatrician with a broad based practice in the community because the majority of our residents choose to practice general pediatrics rather than pursue subspecialties. Residents commented on the importance of the panelists’ experience, the diversity of their perspectives, and their candor in the discussions. Panelists were thoughtful in their responses to questions from the housestaff, and a common theme was that none of the panelists had any standard approaches or easy answers. One resident wrote that it was good to hear about the panelists’ experiences "and to know that even after a long time this will still be a struggle." S/he also commented "before the series I somehow thought there would be a process that I could follow to make the discussions easier... I learned that flexibility and experience can’t be forced." Although the panelists emphasized the challenges inherent in end-of-life care, residents clearly benefited from this session, as data in Tables 3 and 7 demonstrate.
To bring additional perspectives to bear, nonphysician guests participated in sessions regarding family needs, cultural influences at the time of death, and bereavement. Parent volunteers have played important roles in programs designed to teach residents communication skills,10 and in previous studies residents have indicated that discussions with families of patients who had died might be helpful to them during their training.3 To allow for greater candor in the discussion between residents and a family member of a deceased child, we contacted a family whose child was treated in another city, by a health care team outside of the systems familiar to our residents. Social workers from a community-based program that provides services to families before and after the death of a child reflected on the experience of the parent present, and shared their own observations regarding experiences of families from a wide variety of backgrounds. Literature regarding understanding of death at various ages and developmental stages was provided by the social workers in an effort to avoid covering this information in a lecture format. Analysis of pretest and posttest results showed that there was a marked difference in residents’ level of comfort in guiding developmentally appropriate discussions of death with children and families after this educational intervention, and participants expressed a desire for greater time spent on this topic. Future seminar series will therefore include a full session focused in this area.
A considerable change in residents’ level of comfort with discussing death and dying with families of diverse ethnic backgrounds was observed based on test results, but the session dealing with this topic was not as well received as were other sessions. We had specifically avoided a lecture-type approach to cultural issues affecting end-of-life care, but several residents felt that a lecture may have been more helpful. Some appreciated the discussion and believed it would help them avoid the tendency to "assume our views are shared by all," but most believed that this session could have been improved or combined with other sessions.
In contrast, residents believed that the final session aimed at exploring the impact of pediatric patients’ deaths on their personal lives was "especially useful" and a "necessary debriefing." The decision to include significant others at the last meeting in the series was made by the participants, and interestingly, the partners of the residents were quite outspoken during the evening. Posttest results demonstrate a change in residents’ perceptions of their ability to cope with their own responses to the death of a child, but the degree of this change may have been underestimated in our study because most of the participants completed the posttests at the beginning rather than at the end of the final session. Residents subsequently commented that the discussion was "a great combination of the patient and the provider’s perspective on life-threatening illness" and that having participants’ partners present "enabled them to learn what we go through ... and opened up avenues to help them help us cope."
|
CONCLUSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
Based on our experience with this seminar series for pediatric residents, we conclude that this low-cost program was relatively easy to implement and had a positive impact on our residents’ confidence in providing end-of-life care for children. The time required of the series leaders was not trivial; however, the coordinating group was highly motivated and committed to filling what was perceived to be an important gap in resident education. The organizers of the series met monthly for 4 months before the start of the series and divided tasks according to interest and expertise. Chief residents were critical in ensuring that the series was practical and relevant for residents and were able to dedicate focused time to this project.
We realize that self-assessment of comfort level after an intervention such as this carries with it an inherent bias toward improvement. If the improvements seen on posttest scores are in fact attributable to this type of bias, we would anticipate that over time practice patterns of these residents will be no different from those of trainees to whom the course was not available. Longitudinal analysis of practice patterns will be useful in this regard. We also acknowledge that the small number of residents involved in the program and the small number of evaluable responses to the pretest and posttests limit this study. To address this limitation, we plan to continue to offer a seminar series based on this model on a yearly basis, and plan to follow changes in a larger number of residents’ attitudes and practice over time.
|
ACKNOWLEDGMENTS |
|---|
This work was supported by funds from the Vernon and Virginia Furrow Innovation in Medical Education Grant, University of Arizona College of Medicine.
We thank the housestaff who participated in the seminar series and residency director Leslie Barton. We also thank our guests Ross Robb, Julee Gell, Laura Haggerty, Wayne Morgan, Andreas Theodorou, Luke Whitesell, Mary Cochran, Larry Lincoln, John Marx, Anne Wright, and Daniel Shapiro.
|
FOOTNOTES |
|---|
Received for publication Oct 12, 2001; Accepted Feb 8, 2002.
Reprint requests to (R.B.) Pediatric Hematology/Oncology, Steele Memorial Children’s Research Center, University of Arizona, 1501 N Campbell Ave, Tucson, AZ 85724. Email: bagatell{at}peds.arizona.edu
|
REFERENCES |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
- Sahler OJZ, Frager G, Levetown M, Cohn FG, Lipson MA. Medical education about end-of-life care in the pediatric setting: principles, challenges, and opportunities.
Pediatrics.2000; 105
:575
–584
[Abstract/Free Full Text] - Sack W, Fritz G, Krener P, Springer L. Death and the pediatric house officer revisited.
Pediatrics.1984; 73
:676
–681
[Abstract/Free Full Text] - Khaneja S, Milrod B. Educational needs among pediatricians regarding caring for terminally ill children.
Arch Pediatr Adolesc Med.1998; 152
:909
–914
[Abstract/Free Full Text] - Vazirani RM, Slavin S, Feldman JD. Longitudinal study of pediatric house officers’ attitudes toward death and dying. Crit Care Med.2000; 28 :3740 –3745[CrossRef][Web of Science][Medline]
- Weissman D, Ambuel B, Hallenbeck J. Improving End-of-Life Care: A Resource Guide for Physician Education. 3rd ed. Milwaukee, WI: Medical College of Wisconsin Palliative Care Program; 1999
- Bishop M, Gores F, Stempel J, et al. A collaborative end-of-life care curriculum.
Am J Hosp Palliat Care.2000; 17
:137
–140
[Abstract/Free Full Text] - Berman S, Villarreal S. Use of a seminar as an aid in helping interns care for dying children and their families. Clin Pediatr.1983; 22 :175 –179
- Harper M, Wisian N. Care of bereaved parents: a study of patient satisfaction. J Reprod Med.1994; 39 :80 –86[Web of Science][Medline]
- Charlton R. Medical education—addressing the needs of the dying child. Palliat Med.1996; 10 :240 –246[Web of Science][Medline]
- Morgan ER, Winter RJ. Teaching communication skills.
Arch Pediatr Adolesc Med.1996; 150
:638
–642
[Abstract/Free Full Text]
PEDIATRICS (ISSN 1098-4275). ©2002 by the American Academy of Pediatrics
CiteULike 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
This article has been cited by other articles:
|
|
 |
|
  R. D. Boss, N. Hutton, P. K. Donohue, and R. M. Arnold Neonatologist Training to Guide Family Decision Making for Critically Ill Infants Arch Pediatr Adolesc Med, September 1, 2009; 163(9): 783 - 788. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 K. L. Bright, M. Belew Huff, and K. Hollon A Broken Heart--The Physician's Role: Bereaved Parents' Perceptions of Interactions with Physicians Clinical Pediatrics, May 1, 2009; 48(4): 376 - 382. [Abstract] [PDF]
|
|
|
|
 |
|
 M. E. McCabe, E. A. Hunt, and J. R. Serwint Pediatric Residents' Clinical and Educational Experiences With End-of-Life Care Pediatrics, April 1, 2008; 121(4): e731 - e737. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. C. Kolarik, G. Walker, and R. M. Arnold Pediatric resident education in palliative care: a needs assessment. Pediatrics, June 1, 2006; 117(6): 1949 - 1954. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. A. Contro, J. Larson, S. Scofield, B. Sourkes, and H. J. Cohen Hospital Staff and Family Perspectives Regarding Quality of Pediatric Palliative Care Pediatrics, November 1, 2004; 114(5): 1248 - 1252. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
