PEDIATRICS Vol. 124 No. 4 October 2009, pp. e588-e595 (doi:10.1542/peds.2008-3529)
ARTICLE |
Pediatric Pain After Ambulatory Surgery: Where's the Medication?
a Department of Anesthesiology and Perioperative Care, University of California, Irvine, California
b Departments of Pediatric Psychology
c Pediatrics, Children's Hospital of Orange County, Orange, California
| ABSTRACT |
|---|
|
|
|---|
OBJECTIVE: The purpose of this controlled study was to provide a description of children's postoperative pain, including pain intensity and analgesic consumption.
METHODS: Participants included 261 children, 2 to 12 years of age, undergoing routine tonsillectomy and adenoidectomy surgery. Baseline and demographic data were collected before surgery, and a standardized approach to anesthesia and surgical procedures was used. Pain and analgesic consumption were recorded for 2 weeks at home.
RESULTS: On the first day at home, although parents rated 86% of children as experiencing significant overall pain, 24% of children received 0 or just 1 medication dose throughout the entire day. On day 3 after surgery, although 67% of children were rated by parents as experiencing significant overall pain, 41% received 0 or 1 medication dose throughout the entire day.
CONCLUSIONS: We conclude that a large proportion of children receive little analgesic medication after surgery and research efforts should be directed to the discrepancy between high ratings of postoperative pain provided by parents and the low dosing of analgesics they use for their children.
Key Words: postoperative pain pediatric pain surgery analgesia pain management
Abbreviations: GCRC—General Clinical Research Center VAS—visual analog scale PPPM—Parents' Postoperative Pain Measure STAI—State-Trait Anxiety Inventory MBSS—Miller Behavioral Style Scale EASI—Emotionality, Activity, Sociability, and Impulsivity Temperament Survey mYPAS—Modified Yale Preoperative Anxiety Scale
Each year, >5 million children undergo surgery in the United States, and it is estimated that up to 75% of patients experience significant postoperative pain.1 Tonsillectomy and adenoidectomy is one of the most common pediatric surgical procedures and is well known to be associated with significant postoperative pain.2 Indeed, researchers outside the United States suggested high levels of pain after tonsillectomy and adenoidectomy, as well as undertreatment of postoperative pain by parents.3–11 The majority of those studies, however, had methodologic limitations such as small sample sizes, limited use of validated pain measures, and lack of a standardized approach to anesthesia and surgery.
To date, very few researchers in the United States have examined children's postoperative pain management in the home systematically. Therefore, the purpose of this controlled study was to provide a descriptive picture of children's pain after tonsillectomy and adenoidectomy surgery in the United States, using a large sample size, validated outcome measures, and standardized anesthetic and surgical techniques.
The findings of this study are of particular importance to pediatricians because pediatricians, rather than surgeons or anesthesiologists, handle the majority of postoperative follow-up care at home. Poor postoperative pain management among children has been linked to increased utilization of unscheduled medical visits, the majority of which are addressed by family physicians or pediatricians, rather than surgeons or physicians in the emergency department.9
| METHODS |
|---|
|
|
|---|
Participants
Participants were 261 consecutive outpatients 2 to 12 years of age (mean± SD: 6.21 ± 2.39 years), with American Society of Anesthesiology physical status I or II, who were scheduled to undergo routine tonsillectomy and adenoidectomy under general anesthesia. Exclusion criteria included American Society of Anesthesiology classification III or IV, any indication for postoperative admission, developmental delay, chronic illness, premature birth, history of affective disorder, use of psychotropic medications, and BMI of >25 kg/m2. The exclusion criteria were used to standardize the study population. Specifically, children with physical or emotional impairments react to the stressors of surgery and hospitalization differently than do children without such impairments. An institutional review board approved the study, all parents provided written informed consent, and children provided assent as appropriate. Baseline demographic characteristics are presented in Table 1.
|
The data collected for this study were part of a National Institute of Child Health and Human Development-supported trial examining correlates of perioperative distress in children. The main results of the grant-funded study were reported elsewhere12; however, all data reported here were not presented previously.
Measures
Nurse Pain Report: Visual Analog Scale for Pain
The visual analog scale (VAS) is a widely used observational measure of pain.13,14 Nurses completed the VAS at 5 time points in the hospital (admission to the General Clinical Research Center [GCRC] hospital floor, hour 6, hour 12, hour 18, and hour 24). The test-retest reliability of the observational VAS is high when results are measured on adjacent days.15,16
Parent Pain Report: Parents' Postoperative Pain Measure and Numeric Rating Scale
The Parents' Postoperative Pain Measure (PPPM) is a 15-item, observational checklist with which parents rate behavioral changes that correspond to pain severity.17 The PPPM, with the use of a cutoff score of 6, has excellent specificity and sensitivity in identifying children in significant pain, and the measure has good internal consistency.17 The PPPM is recommended as a well-established, observational measure for parents to assess children's postoperative pain in the home, and it has demonstrated high interrater reliability and good construct validity.18
Parents also provided a global rating of their child's pain by using a daily numerical rating scale. Specifically, at 4 time points at home (day 2, day 3, week 1, and week 2), parents completed a 0-to-10 numerical rating scale of their child's pain over the previous 24 hours. The numerical rating scale had anchors at both ends of the continuum, with 0 labeled "no pain" and 10 labeled "worst pain ever."
Child Preoperative Distress: Modified Yale Preoperative Anxiety Scale
The Yale Preoperative Anxiety Scale is a structured observational measure of preoperative anxiety in children.19,20 This 27-item measure is composed of 5 domains of behavior that reflect anxiety in young children. The Modified Yale Preoperative Anxiety Scale (mYPAS) was developed and validated by our study group and has been demonstrated to have good to excellent interobserver and intraobserver reliability and criterion validity.19,20
Child Temperament: Emotionality, Activity, Sociability, and Impulsivity Temperament Survey
The Emotionality, Activity, Sociability, and Impulsivity Temperament Survey (EASI) is a 20-item, standardized measure of behavioral aspects of temperament in children, including emotionality, activity, sociability, and impulsivity.21 Higher scores indicate higher baseline emotionality, activity, sociability, or impulsivity. The EASI has demonstrated good test-retest reliability and validity.22
Parental Anxiety: State-Trait Anxiety Inventory
The State-Trait Anxiety Inventory (STAI) is a widely used measure of both state (situational) and trait (baseline) anxiety.23 The STAI is well validated and has good reliability.23 Parents completed the trait subscale at study recruitment and completed the state subscale during preoperative holding and again immediately after separation for surgery.
Parent Coping Style: Miller Behavioral Style Scale
The Miller Behavioral Style Scale (MBSS) is a self-report assessment of parental coping style that uses 4 scenarios depicting stressful situations.24 This instrument was developed for patients undergoing medical procedures and identifies 4 coping styles, namely, information-seeking, information-avoiding, distraction, and nondistraction. The MBSS has very good reliability and validity.24
Study Protocol
Children and their parents were recruited 7 to 10 days before scheduled surgery, during a preadmission visit. Parental consent and child assent were obtained at that time, and demographic data were collected. Baseline measures also were completed, including the EASI, MBSS, and STAI. On the day of surgery, children's anxiety was measured with the mYPAS at 4 time points, namely, in the preoperative holding area, at separation from parents, at entrance to the induction area, and at introduction of the anesthesia mask. A standardized anesthesia protocol was used for each child, in which anesthesia was induced by using oxygen/nitrous oxide and sevoflurane administered through a scented mask. Isoflurane/nitrous oxide/oxygen, 0.1 mg/kg vecuronium, and 0.1 mg/kg morphine sulfate, administered intravenously, were used to maintain anesthesia. An antiemetic agent (ondansetron, 0.1 mg/kg) was administered to all children.
After surgery, children were monitored in the postanesthesia care unit, where analgesic administration and consumption were recorded. After discharge from the postanesthesia care unit, all children were admitted to the Yale Children's GCRC for a 24-hour observation period. Pain management was standardized, and children with Bieri Faces Scale ratings of
3 were given 10 mg/kg acetaminophen plus 1 mg/kg codeine at 4-hour intervals. After 24 hours in the GCRC, all children were discharged from the hospital.
After the children's discharge from the GCRC, parents were given packets that included validated assessments of postoperative pain and recovery. For pain management, parents were instructed to administer 10 mg/kg acetaminophen plus 1 mg/kg codeine every 4 to 6 hours if children scored
3 on the Bieri Faces Scale. Parents also completed PPPM and VAS ratings of children's pain daily, which were recorded as part of the study on day 2 (first day at home), day 3, week 1, and week 2. Parents were asked to assess the overall pain of their child in the previous 24 hours. Parents were contacted at the aforementioned time points by pediatric nurses from the GCRC, for assessment and recording of pain ratings and analgesic requirements and consumption.
Data Analysis Strategy
All analyses were performed with SPSS 17.0 (SPSS Inc, Chicago, IL). Descriptive statistics were used to provide an illustration of the patterns of pain and medication use in the hospital and at home. To determine any potential predictors of pain severity and medication use, bivariate correlations between these 2 outcome variables and baseline parent and child variables were conducted. One-way analysis of variance was used to explore group differences in baseline characteristics and demographic features according to analgesia received and pain scores. A value of P < .05 was considered statistically significant. On the basis of literature findings, we defined PPPM scores of
6 as indicating children's clinically significant pain.17 In addition, scores of
30 on the nurse-completed VAS and
3 on the parent numerical pain rating were considered to indicate significant pain, because these are the most widely accepted pain levels used in the literature. Because very few parents recorded Bieri scores at the defined intervals, we present only PPPM and VAS ratings.
| RESULTS |
|---|
|
|
|---|
Descriptive Data
A total of 261 children and their parents were enrolled in this cohort study. Means and SDs for demographic and baseline characteristics are reported in Table 1. Although results indicated some significant correlations between aspects of child temperament, including activity (r = –0.16; P = .02) and sociability (r = –0.24; P < .01), and parent ratings of children's pain on the PPPM on days 2 and 3 after surgery, no overall patterns of significant relationships between the 2 outcome variables of interest in the present study (pain severity and analgesic use) and baseline and demographic variables emerged.
Pain in the Hospital
As can be seen in Fig 1A, a large number of nurses in the GCRC rated the children as experiencing significant pain at the 5 time points during the 24-hour hospitalization. For example, the median pain rating at admission was 27.7 (range: 0–100), and 41% of children were in significant pain. At the time of discharge from the hospital, 27% of children were still rated as experiencing pain levels of >30, and the median pain rating was 21.3 (range: 0–100) (Fig 1A).
|
Pain at Home
Examination of the PPPM scores illustrated that 77% and 67% of children were rated by parents as experiencing significant pain on day 2 (mean score: 8.5 [range: 0–15]) and day 3 (mean score: 7.2 [range: 0–15]), respectively. Even 1 week after surgery, 49% of children were rated by their parents as experiencing significant pain (mean score: 5.8 [range: 0–15]). By week 2 after surgery, much improvement in pain had occurred; however, 15 children (7.5%) continued to be rated with the PPPM as experiencing significant pain (mean score: 1.4 [range: 0–15]). Pain ratings are illustrated in Fig 1B. The distributions of PPPM scores for each time point at home (day 2, day 3, week 1, and week 2) are presented in Figs 2 to 5.
|
|
|
|
Results of the numerical rating scale completed by the parents were similar, indicating that 86% and 67% of children were rated as experiencing significant pain on day 2 (mean score: 4.9 [range: 0–10]) and day 3 (mean score: 3.8 [range: 0–10]), respectively. Consistent with the PPPM results, 48% of children were rated by their parents as experiencing significant pain 1 week after surgery (mean score: 2.8 [range: 0–10]). In fact, 23% of children were rated as experiencing pain severity of
5 at week 1. By week 2, 16 children (8.1%) continued to be rated as experiencing significant pain. None of the baseline or demographic parent or child variables was significantly associated with pain severity.
Analgesic Use
All except 2 of the children surveyed received analgesic medication during their 24-hour hospital stay. In contrast, on the first day at home, despite the fact that parents rated 86% of children as experiencing significant overall pain, 24% of children received 0 or just 1 medication dose throughout the entire day. On day 3 after surgery, although 67% of children were rated by parents as experiencing significant overall pain, 41% received 0 or just 1 medication dose throughout the entire day. Overall, 23% of all children received
3 doses of analgesic medications throughout the entire 2 weeks at home, and the median number of total analgesic doses administered at home over the 2-week period was 6 (range: 0–14). Indeed, 71% of the children received fewer than one half of the 16 possible doses of analgesics assessed at home. The numbers of medication doses administered on days 2 and 3 at home are presented in Fig 6. The total numbers of doses children received during the 2 weeks at home are presented in Table 2.
|
|
For determination of any potential predictors of analgesic administration by parents, children were grouped according to whether they received pain medication on day 3 (0 or 1 versus
2 doses received). Day 3 was chosen because of the large proportion (67%) of children rated by parents as experiencing significant pain and the large proportion (41%) of children who received only 1 or 0 medication doses throughout the entire day. Analysis of variance was then used to examine any mean differences in demographic and baseline variables, including child and parent ages, child preoperative distress (mYPAS), parent education, parent state and trait anxiety (STAI), parent coping (MBSS), and child temperament (EASI). There were no differences between the groups with respect to any of the variables, with the exception of EASI sociability scores, which were slightly lower for the children who received fewer analgesics (scores of 17.6 vs 18.8; F1,158 = 5.6; P = .02). In fact, neither pain severity on the PPPM on day 3 (F1,121 = 0.64; P = .43) nor average pain severity on the PPPM over the 2 weeks at home (F1,99 = 0.02; P = .89) predicted medication received on day 3. Finally, we divided children into 3 categories on the basis of PPPM ratings, to determine whether pain levels predicted medication doses. Specifically, children were categorized into low (PPPM ratings of 0–5), moderate (PPPM ratings of 6–10), and severe (PPPM ratings of 11–15) pain groups, and 1-way analyses of variance were performed to examine group differences. There were no significant differences in medication received by the 3 groups on day 2 (F2,156 = 0.84; P = .43), day 3 (F2,147 = 0.05; P = .95), week 1 (F2,153 = 1.40; P = .25), or week 2 (F2,148 = 0.21; P = .81).
| DISCUSSION |
|---|
|
|
|---|
Under the conditions of this study, we have demonstrated that children who undergo tonsillectomy and adenoidectomy in the United States are in significant pain and receive just a few doses of analgesic drugs after surgery. To provide an accurate description of pain and analgesic use, a large sample of children was recruited and intraoperative procedures were standardized, to control for variations in the perioperative environment that might affect postoperative outcomes. In addition, well-validated measures of pain were used and parents were trained to complete the pain rating scales, to increase the validity of the assessments. Our findings regarding postoperative pain at home were consistent with existing literature findings.3,4,7,9 Specifically, children reported moderate/severe pain that lasted up to 2 weeks. Children's pain peaked at day 2, but most children continued to report significant pain through day 3 and close to one half of the children continued to experience significant pain 1 week after surgery.
Examination of postoperative analgesic consumption indicated that children received few doses of pain medication throughout the 2 weeks at home. In addition, medication administration was unrelated to pain severity. Many reasons for the low rates of analgesic administration by parents have been suggested, including fears of medication dependence, lack of understanding of the presentation of pain in young children, and beliefs that analgesics should be used only as a last resort.3–5,25,26 Investigators also found that personality characteristics and sociodemographic factors may affect parents' adherence to pain management recommendations. For example, our group found that parents with less education and more-reactive children gave fewer analgesics, whereas parents rated higher in conscientiousness and those with impulsive children were more likely to provide analgesics to their children postoperatively.27
What is clear from the literature is that there is a wide gap in the translation of scientific knowledge to children's pain management at home. Although there have been significant improvements in pain management in the hospital,28 these improvements have not translated to the home. In fact, our team noted recently that many children continue to experience pain despite the availability of evidence-based treatments and practice guidelines.29 There is a pressing need for researchers to focus attention on the translation of knowledge to target children's pain management at home.
Although our study has scientific merit, it is not without limitations. One limitation is that children's pain was assessed through nurse and parent reports. Although this practice is common, some research suggested that parents and nurses may underestimate children's pain.30,31 This suggests that even more children should have been classified as being in significant pain after surgery. In addition, we did not assess factors that might contribute to the low rate of administration of analgesics at home. Future research should focus on determining the specific reasons why parents may not administer needed analgesia, including parental attitudes regarding analgesic medications, personality characteristics, lack of understanding of the nature of pain and analgesia, and challenges in medicating children postoperatively. In particular, swallowing medications may be particularly difficult after tonsillectomy and adenoidectomy, which may contribute to children's unwillingness to take analgesics or parents' unwillingness to administer analgesics. Therefore, these findings may not be generalizable to patterns of analgesia administration after other types of surgery. Finally, children in this study were hospitalized for 24 hours after surgery, which is not standard practice and which might have resulted in lower pain levels at the time of arrival home. Again, this emphasizes the large number of children who are rated as experiencing significant pain at home after surgery.
Because pediatricians, general practitioners, and family physicians frequently need to address postoperative pain management for children,9,32 this study is of great significance. In fact, it has been documented that up to 60% of parents seek consultation with their family doctors after tonsillectomy because of postoperative complications and concerns, including pain severity and duration.9,32 Therefore, pain contributes to increased health care costs by increasing the need for unscheduled postoperative medical visits.
| CONCLUSIONS |
|---|
|
|
|---|
Optimal pain management after surgery is achieved by providing round-the-clock analgesic treatment to prevent pain recurrence.3 Despite clinical care guidelines, many children continue to experience unnecessary postoperative pain. We conclude that a large proportion of children receive little analgesic medication after surgery and research efforts should be directed to the discrepancy between high ratings of postoperative pain provided by parents and the low dosing of analgesics they use for their children.
| ACKNOWLEDGMENTS |
|---|
This work was supported by the National Institute of Child Health and Human Development (grant R01HD37007-01).
| FOOTNOTES |
|---|
Accepted May 29, 2009.
Address correspondence to Michelle A. Fortier, PhD, Children's Hospital of Orange County, 505 S Main St, Suite 940, Orange, CA 92868. E-mail: mfortier{at}choc.org
Financial Disclosure: The authors have indicated they have no financial relationships relevant to this article to disclose.
| What's Known on This Subject: Research performed outside the United States suggests that children experience significant pain after tonsillectomy and adenoidectomy and are undermedicated by parents; however, that body of literature suffers from significant methodologic limitations.
|
| What This Study Adds: This study provides a description of children's pain after tonsillectomy and adenoidectomy in the United States, with a large sample size, validated outcome measures, and standardized anesthetic and surgical techniques.
|
| REFERENCES |
|---|
|
|
|---|
- Owen H, McMillan V, Rogowski D. Postoperative pain therapy: a survey of patients' expectations and their experiences. Pain. 1990;41 (3):303 –307[CrossRef][Web of Science][Medline]
- Husband AD, Davis A. Pain after tonsillectomy. Clin Otolaryngol Allied Sci. 1996;21 (2):99 –101[CrossRef][Medline]
- Finley GA, McGrath PJ, Forward SP, McNeill G, Fitzgerald P. Parents' management of children's pain following "minor" surgery. Pain. 1996;64 (1):83 –87[CrossRef][Web of Science][Medline]
- Gedaly-Duff V, Ziebarth D. Mothers' management of adenoid-tonsillectomy pain in 4- to 8-year-olds: a preliminary study. Pain. 1994;57 (3):293 –299[CrossRef][Web of Science][Medline]
- Huth MM, Broome ME. A snapshot of children's postoperative tonsillectomy outcomes at home. J Spec Pediatr Nurs. 2007;12 (3):186 –195[CrossRef][Web of Science][Medline]
- Nikanne E, Kokki H, Tuovinen K. Postoperative pain after adenoidectomy in children.
Br J Anaesth. 1999;82
(6):886
–889
[Abstract/Free Full Text] - Swallow J, Briggs M, Semple P. Pain at home: children's experience of tonsillectomy.
J Child Health Care. 2000;4
(3):93
–98
[Abstract/Free Full Text] - Unsworth V, Franck LS, Choonara I. Parental assessment and management of children's postoperative pain: a randomized clinical trial.
J Child Health Care. 2007;11
(3):186
–194
[Abstract/Free Full Text] - Warnock FF, Lander J. Pain progression, intensity and outcomes following tonsillectomy. Pain. 1998;75 (1):37 –45[CrossRef][Web of Science][Medline]
- Wiggins SA, Foster RL. Pain after tonsillectomy and adenoidectomy: "ouch it did hurt bad." Pain Manag Nurs. 2007;8 (4):156–165
- Wilson ME, Helgadottir HL. Patterns of pain and analgesic use in 3- to 7-year-old children after tonsillectomy. Pain Manag Nurs. 2006;7 (4):159 –166[CrossRef][Web of Science][Medline]
- Kain ZN, Mayes LC, Caldwell-Andrews AA, Karas DE, McClain BC. Preoperative anxiety, postoperative pain, and behavioral recovery in young children undergoing surgery.
Pediatrics. 2006;118
(2):651
–658
[Abstract/Free Full Text] - Huskisson EC, Jones J, Scott PJ. Application of visual-analogue scales to the measurement of functional capacity. Rheumatol Rehabil. 1976;15 (3):185 –187[Medline]
- Huskisson EC, Melzack R. Visual Analogue Scales, Pain Measurement and Assessment. New York, NY: Raven Press; 1983
- Cella DF, Perry SW. Reliability and concurrent validity of three visual analogue mood scales. Psychol Rep. 1986;59 (2):827 –833[Web of Science][Medline]
- Folstein MF, Lauria R. Reliability, validity and clinical application of the visual analogue mood scale. Psychol Med. 1973;3 (4):479 –486[Web of Science][Medline]
- Chambers CT, Reid GJ, McGrath PJ, Finley GA. Development and preliminary validation of a postoperative pain measure for parents. Pain. 1996;68 (2–3):307 –313[CrossRef][Web of Science][Medline]
- von Baeyer CL, Spagrud LJ. Systematic review of observational (behavioral) measures of pain for children and adolescents aged 3 to 18 years. Pain. 2007;127 (1–2):140 –150[CrossRef][Web of Science][Medline]
- Kain ZN, Mayes LC, Cicchetti DV, Bagnall AL, Finley JD, Hofstadter MB. The Yale Preoperative Anxiety Scale: how does it compare with a "gold standard"? Anesth Analg. 1997;85 (4):783 –788[Abstract]
- Kain ZN, Mayes LC, Cicchetti DV, et al. Measurement tool for preoperative anxiety in young children: the Yale Preoperative Anxiety Scale. Child Neuropsychol. 1995;1 (3):203 –210[CrossRef][Web of Science]
- Buss A, Plomin R. A Temperament Theory of Personality Development. New York, NY: Wiley-Interscience; 1975
- Buss AH, Plomin R. Theory and Measurement of EAS, Temperament: Early Developing Personality Traits. Hillsdale, NJ: Lawrence Erlbaum Associates; 1984
- Spielberger CD: Manual for the State-Trait Anxiety Inventory (STAI): Form Y. Palo Alto, CA: Consulting Psychologists Press; 1983
- Miller S. Monitoring and blunting: validation of a questionnaire to assess styles of information seeking under threat. J Pers Soc Psychol. 1987;52 (2):345 –353[CrossRef][Web of Science][Medline]
- Forward S, Brown TL, McGrath PJ. Mothers' attitudes and behavior toward medicating children's pain. Pain. 1996;67 (2–3):469 –474[CrossRef][Web of Science][Medline]
- Hamers JP, Abu-Saad HH. Children's pain at home following (adeno)tonsillectomy. Eur J Pain. 2002;6 (3):213 –219[CrossRef][Web of Science][Medline]
- Zisk RY, Grey M, MacLaren JE, Kain ZN. Exploring sociodemographic and personality characteristic predictors of parental pain perceptions.
Anesth Analg. 2007;104
(4):790
–798
[Abstract/Free Full Text] - Howard RF. Current status of pain management in children.
JAMA. 2003;290
(18):2464
–2469
[Abstract/Free Full Text] - MacLaren J, Kain ZN. Research to practice in pediatric pain: what are we missing?
Pediatrics. 2008;122
(2):443
–444
[Free Full Text] - Singer AJ, Gulla J, Thode HC Jr. Parents and practitioners are poor judges of young children's pain severity. Acad Emerg Med. 2002;9 (6):609 –612[CrossRef][Web of Science][Medline]
- Zhou H, Roberts P, Horgan L. Association between self-report pain ratings of child and parent, child and nurse and parent and nurse dyads: meta-analysis. J Adv Nurs. 2008;63 (4):334 –342[CrossRef][Web of Science][Medline]
- Lee WC, Sharp JF. Complications of paediatric tonsillectomy post-discharge. J Laryngol Otol. 1996;110 (2):136 –140[Web of Science][Medline]
PEDIATRICS (ISSN 1098-4275). ©2009 by the American Academy of Pediatrics
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||










