BACKGROUND AND OBJECTIVES: Inadequate pain treatment leaves hospitalized children vulnerable to immediate and long-term sequelae. A multidimensional knowledge translation intervention (ie, the Evidence-based Practice for Improving Quality [EPIQ]) improved pain assessment, management, and intensity outcomes in 16 units at 8 Canadian pediatric hospitals. The sustained effectiveness of EPIQ over time is unknown, however. The goals of this study were to determine the following: (1) sustainability of the impact of EPIQ on pain assessment, management, and intensity outcomes 12, 24, and 36 months after EPIQ; (2) effectiveness of a pain practice change booster (Booster) intervention to sustain EPIQ outcomes over time; and (3) influence of context on sustainability.
METHODS: A prospective, repeated measures, cluster randomized controlled trial was undertaken in the 16 EPIQ units, 12 months after EPIQ completion, to determine the effectiveness of a practice change booster (Booster) to sustain EPIQ outcomes. Generalized estimating equation models examined outcomes controlling for child and unit contextual factors.
RESULTS: Outcomes achieved during EPIQ were sustained in the use of any pain assessment measure (P = .01) and a validated pain assessment measure in the EPIQ units (P = .02) up to 36 months after EPIQ. Statistically significant improvements in pain management practices persisted in EPIQ units; results varied across time. There were no significant differences in outcomes after implementation of the Booster between the Booster and Nonbooster groups.
CONCLUSIONS: Improved pain assessment and management practices were sustained after EPIQ; however, the Booster did not seem to provide additional impact.
- EPIQ —
- Evidence-based Practice for Improving Quality
- GEE —
- generalized estimating equation
- KT —
- knowledge translation
What’s Known on This Subject:
Inadequate treatment of procedural pain in hospitalized children, despite evidence of effective pain relief, is a major concern. A novel knowledge translation intervention, Evidence-based Practice for Improving Quality (EPIQ), improved pain practice and clinical outcomes in 8 Canadian pediatric hospitals.
What This Study Adds:
Effectiveness of a knowledge translation booster intervention was investigated for sustaining improvement in pediatric pain practices after the EPIQ intervention. Sustained improvements were attributed to EPIQ as no additional differences existed between the booster and standard care groups over time.
Knowledge translation (KT) interventions can change practice and improve health outcomes.1,2 However, researchers typically focus primarily on initial intervention effectiveness, while sustainability receives little attention. Sustaining outcomes beyond their initial adoption is crucial for achieving target outcomes.3 However, sustainability is one of the most challenging issues in implementation science, requiring focused study to understand how and under what conditions interventions are sustained.3 In a systematic review of intervention sustainability, Wiltsey Stirman et al4 identified 125 studies; few used rigorous evaluation methods, and only 22% examined the sustainability of intervention outcomes.
The effectiveness of a multidimensional KT intervention, Evidence-based Practice for Improving Quality (EPIQ),5 to improve pain assessment and management and clinical outcomes in hospitalized children was evaluated in 32 patient care units at 8 Canadian tertiary pediatric hospitals (16 EPIQ intervention units and 16 standard care units). EPIQ was informed by the Promoting Action on Research Implementation in Health Services framework, which postulates that evidence, context, and facilitation support outcome improvement.6 The EPIQ intervention involved facilitation of evidence-based practice change by unit pain champions, using quality improvement cycles and select KT strategies tailored to the unit context. Pain assessment and management and pain intensity were improved in EPIQ units compared with standard care units.7,8 However, we hypothesized that these improvements may not be sustained over time. In the present study, we evaluated the effectiveness of a pain practice change booster (Booster) intervention that involved facilitator-led teleconference sessions with pain champions on the EPIQ units to implement KT strategies to sustain or enhance EPIQ outcomes.
Study Design and Eligibility Criteria
The study aim was to determine the following: (1) sustainability of the impact of EPIQ on pain practice (assessment and management) and clinical (pain intensity) outcomes compared with baseline at 12, 24, and 36 months post-EPIQ; (2) effectiveness of a Booster intervention to sustain EPIQ outcomes; and (3) influence of child and unit contextual factors on sustainability.
A prospective, repeated measures, cluster randomized controlled trial, conducted in 16 hospital units that used EPIQ, was introduced 12 months after completion of EPIQ. This randomized controlled trial was approved by the research ethics boards of the participating hospitals and affiliated universities.
The 16 EPIQ units were randomized, using centrally controlled cluster randomization, to the Booster (n = 8) or Nonbooster (n = 8) group (Figs 1 and 2). Randomization was conducted by a biostatistician blinded to unit identity and restricted (1 Booster and 1 Nonbooster unit per hospital) by using a list of random numbers. Blinding of the study intervention was not possible; however, staff, co-investigators, and analysts were blinded to the randomization process.
The Booster intervention was implemented in 4 medical, 3 critical care, and 1 surgical unit. Pain champions, trained during EPIQ, participated in the Booster intervention. Champions implemented validated KT strategies to maintain or improve pain outcomes. Standardized Booster sessions were conducted via teleconference, once every 4 months, over a 24-month period (6 Boosters) (Fig 2), with 3 to 6 pain champions. Sixty-minute Booster sessions were delivered by 2 cofacilitators trained by the principal investigator. In the first session, unit champions established a commitment to change plan, including specific pain assessment and management practice change aim(s) and KT strategies for 4 months. In each successive Booster session, the previous commitment to change plan was reviewed and revised, if necessary, for implementation over the next 4 months. Contextual factors were assessed, and strategies to overcome barriers were considered.
The Nonbooster units did not receive the Booster intervention and were instructed to continue with their own best efforts for sustaining improvements attained through EPIQ.
Sustainability and Booster effectiveness were assessed by using pain practice (assessment and management) and clinical (pain intensity) outcomes (Fig 1). Pain practice outcomes were measured at baseline and at 12, 24, and 36 months after EPIQ completion. They included the proportion of: (1) children per unit whose pain was assessed and whether a validated pain measure was used; and (2) painful procedures conducted without pain management interventions and with pharmacological, physical, or psychological interventions. Data were collected from patient charts for the previous 24 hours, using the Canadian Pediatric Pain Research database, which has been validated in previous studies.7,8 Pain intensity was measured at 12, 24, and 36 months post-EPIQ, during routinely scheduled painful procedures for diagnostic purposes, using 1 of 4 validated, age-appropriate pain measures: the Premature Infant Pain Profile,9 the Faces, Legs, Arms, Cry, Consolability scale,10 the Faces Pain Scale–Revised,11 and the Numerical Rating Scale.12 Ratings were standardized and reported on a 0- to 10-point scale.
Data on child demographic characteristics were collected at all time points. Unit characteristics (eg, unit type, average occupied beds, average patient stay, total patient days, pain service availability) were assessed at baseline only.
Sample Size, Data Collection, and Analyses
The 16 EPIQ intervention units comprised the sample for the Booster randomized controlled trial. Sample size for the original EPIQ study was calculated based on pain practice and clinical outcomes at baseline, and it incorporated a design effect to account for patient clustering within units,13,14 assuming a conservative intraunit correlation of 0.2.15,16 Whether a pain assessment or management strategy was implemented in the preceding 24 hours was determined on 30 patients per unit at 12, 24, and 36 months post-EPIQ (n = 1440). This sample size provided a power of 80% to detect a 15% difference in pain assessment and a 20% difference in pain management at all other times compared with baseline.
Pain intensity was assessed in 20 eligible children per unit at 12, 24, and 36 months post-EPIQ (n = 960) to provide a power of 90% to detect one-half of an SD difference between groups. Data were used as a continuous variable and summarized into 4 categories, based on the National Institutes of Health guidelines: no pain (0 of 10); mild pain (1–3 of 10); moderate pain (4–6 of 10); and severe pain (7–10 of 10).17
All data were analyzed using SAS version 9.3 (SAS Institute, Inc, Cary, NC). Demographic data were summarized using frequency counts and proportions for categorical data, and means and SDs for continuous data. Descriptive statistics described unit and patient variables.
Sustainability of EPIQ Outcomes
Generalized estimating equation (GEE) modeling was used to examine differences in pain assessment and management and pain intensity outcomes over time, controlling for child and unit contextual factors. Logit GEE was used for analyses of binary pain practice outcomes (eg, use of validated pain assessment tool); linear GEE was used for analyses of pain intensity.
Effectiveness of the Booster Intervention
Pain practice and clinical outcomes were compared between the Booster and Nonbooster groups by using logit-based GEE, incorporating the associated baseline measurement as a covariate. Intervention fidelity was assessed according to self-report to determine adherence to the intervention and delivery of all Booster session components.
Influence of Child and Unit Characteristics
Among patients with clinically significant levels of pain during painful procedures (ie, a score >4 of 10), logit-based GEE was used to examine the relative change in odds of patients experiencing severe versus moderate pain as a function of patient- and unit-level characteristics. These models included patient- and unit-level characteristics, as well as time period and intervention status.
Child and Unit Characteristics
There were no significant differences in child age or sex between the Booster and Nonbooster units. There were statistically significant differences in primary diagnoses, with acute diagnoses between 89% at baseline and 83% at 36 months (P = .001) and the proportion of children undergoing surgery (3% at baseline vs 9% at 36 months; P < .001). More children underwent surgery in the Nonbooster group compared with the Booster group at 12 months (P = .004), although the numbers were small. There was a significantly higher proportion of children in surgical units and a lower proportion of medical patients in the Nonbooster group compared with the Booster group at all times (P < .001) (Table 1).
Sustainability of EPIQ Outcomes
There were statistically significant differences over time in the proportion of children whose pain was assessed using any pain measure (68% at baseline to 81% at 36 months; P = .01) and a validated pain measure (26% at baseline to 48% by 36 months; P = .02). At 36 months, a higher proportion of children had any pain assessment (P = .08) and pain assessment using a validated measure compared with baseline (P = .02).
There was a statistically significant improvement in analgesic use linked to a painful procedure at 36 months compared with baseline (30% vs 26%; P = .04). Statistically significant differences were found in physical (P = .04) and psychological (P = .03) pain management strategies over time. The proportion of physical strategies (15% at baseline vs 22% at 36 months; P = .04) and psychological strategies (16% at baseline vs 20% at 36 months; P = .03) significantly improved.
Standardized pain intensity scores were not significantly different between groups or across time. When categorized into clinically relevant pain intensity categories, there were no differences across time (Table 2). However, when adjusting for individual and unit factors, children were less likely to have severe pain intensity at 12 months compared with 36 months after EPIQ (P = .004) (Table 3).
Effectiveness of the Booster
There were no significant differences between the Booster and Nonbooster groups for pain assessment and management (Table 4, Figs 3 and 4) and no significant group-by-time interactions for pain intensity (Table 3) after adjusting for child- and unit-level factors. There also was no significant difference in the odds of having severe pain in the Booster units versus Nonbooster units over time (P = .30) or overall (P = .18) (Table 2).
Influence of Child and Unit Characteristics on Pain Intensity
All age groups were significantly more likely to experience severe pain compared with adolescents. Infants in the NICUs were 2 times less likely to experience severe pain than children in the PICUs. Children on units with higher average bed occupancies and lengths of stay were more likely to report severe pain (Table 3). There was no significant differences in the odds of having severe pain for children in the Booster units versus Nonbooster units over time, after controlling for child and unit factors (adjusted odds ratio, 1.20 [95% confidence interval, 0.96–1.52]; P = .11).
Minimal changes in unit champions occurred during the study (50% lost 1 member, and 25% lost >1 member). All Booster session components were delivered as planned, based on self-reports of champions and facilitators. Of the 8 Booster units, 4 focused on pain assessment practice change aims, 2 on pain management, and 2 on both factors. Unit and organizational challenges reported during the Booster interventions included: (1) competing priorities: hospital-wide introduction of electronic documentation/patient charting (n = 3 units); location moves and accreditation (n = 2 units); (2) changes in structure: integration (n = 2 units) or division (n = 2 units) of units; changes in unit (n = 2 units) and hospital (n = 2 units) leadership; and (3) staff turnover and shortages were common across units.
Sustaining the Effect of EPIQ Over Time
Sustainability of newly adopted evidence-based practices is considered more effective than continued introduction of new interventions.18 EPIQ, a quality improvement, KT intervention to improve pain management and reduce pain intensity for hospitalized children, was effective in improving pain outcomes.7,8 Improvements achieved during EPIQ were sustained up to 36 months compared with baseline. Health care professionals continued with improved pain practices up to 3 years after the original intervention, although not at levels achieved immediately post-EPIQ.7,8 Using the definition of sustainability by Stirman et al,4 EPIQ intervention outcomes7,8 were only partially sustained, as they did not remain at or above results reported post-EPIQ. The Booster intervention, introduced 12 months post-EPIQ, did not enhance or sustain pain outcomes achieved during EPIQ; however, determining whether these results occurred due to the ineffectiveness of the Booster intervention or its implementation is difficult. Boosters have successfully reinforced the effects of interventions aimed at changing practices or behaviors19–21; the effect of Boosters in sustaining improved pain practices and reducing pain intensity for hospitalized children is unknown. Furthermore, the lingering effect of EPIQ on outcomes cannot be differentiated from Booster implementation and intervention effectiveness. This issue epitomizes key challenges in KT research in which little is known regarding understanding the complex process of sustaining outcome improvement, and the major focus is on evaluating intervention effectiveness outcomes compared with implementation outcomes.22
Booster Implementation and Effectiveness
Aspects of Booster implementation (eg, feasibility, fidelity, cost) may have negatively contributed to Booster effectiveness. Booster requirements (eg, time commitment, expectations) may have been challenging for pain champions. Booster delivery via teleconference might not have been effective due to duration or lack of personal contact. Unknown unit-specific issues (eg, competing priorities, stakeholder support) could also have impeded positive outcomes.
The Booster intervention may have reflected or perpetuated the complexity of the change process. The number of sessions (n = 6) was chosen arbitrarily, and it may explain the decrease in pain management strategies used after the initial introduction of the Booster into the setting (after 3 Booster sessions) and the increase by 36 months (after 6 Booster sessions). Time for adaptation and tailoring of an intervention to a new setting may be necessary to ensure the longevity of the intervention beyond its initial introduction. Although initial success is often celebrated, monitoring is rare and evaporation of improvement is common. Future research must investigate sustainability mechanisms (intervention introduction, adaptation, and integration), because these factors may be a precondition for achieving sustainable clinical effectiveness outcomes.22
Evidence, context, and facilitation are key interacting elements in the success of quality improvement, according to the Promoting Action on Research Implementation in Health Services framework,6 which guided the EPIQ intervention. These elements provide insight into the partial sustainability of improved pain outcomes achieved during EPIQ and Booster effectiveness outcomes. The sustainability of increased use of evidence-based pain assessment practices across units may be attributed to the practice change focus of the majority of units during EPIQ (11 of 16 [68%])7 and Booster (6 of 8 [75%]) studies on changing pain practices. This concentrated focus on pain assessment, rather than on pain management, may have contributed to the lack of sustained pain intensity outcomes. Pain management consists of a more complex and varied set of decision-making processes compared with pain assessment, which is often dictated by pain assessment measure and frequency. Evidence-based physical and psychological pain management strategies, while improved over time, were only implemented 20% to 30% of the time by 36 months post-EPIQ. This magnitude of improvement may not have been sufficient to sustain changes over time. These findings are comparable to the study by Ali et al23 on pediatric pain management in emergency departments, in which nonpharmacologic strategies were used <16% of the time. The relationship between health care professionals’ pain practices and clinical patient outcomes needs to be further investigated.
The influence of context (eg, organizational structure) on innovation sustainability has received minimal attention. Rogers24 identified common contributors to sustainability, including leadership, stakeholder support, dedicated champions, and innovation alignment with the mission, vision, and organizational goals. Squires et al25 reported that leadership, culture, evaluation, formal and informal interactions, organizational slack (space), and unit specialty predicted nurses’ research use. In the EPIQ study, culture, social capital, informal interactions, and resources moderated the relationship between instrumental research use and pain assessment.26 Unit and organizational factors that negatively influenced practice change in the Booster study, as reported by the pain champions (eg, electronic charting, leadership changes), were consistent with these researchers’ findings.
Pain champions may also have contributed to outcome sustainability. Facilitation is considered “high” on the knowledge-to-practice continuum, when appropriate supports exist and when task and enabling skills are based on needs.6, 27 The EPIQ and Booster interventions supported individualized practice change goals and implementation of select, tailored KT strategies to achieve these goals; however, the champions’ ability to promote change and organizational support varied widely.
Influence of Child and Unit Characteristics on Sustainability of Outcomes
Child and unit factors may have influenced the sustainability of improved pain outcomes. The impact of these factors on the clinical outcome (ie, pain intensity) is similar to the EPIQ study.7 All age groups were more likely to experience severe pain than adolescents, as were children in units with higher bed occupancy. Over time, infants in NICUs were less likely to have severe pain than those in PICUs. There is need to improve pain management in younger hospitalized children, especially in ICU settings and units with high bed occupancy. Child and unit factors (eg, the proportion of children who had surgery) did not affect pain intensity outcomes in the Booster units compared with Nonbooster units.
All study units participated in the study for 36 months, which is consistent with current recommendations for measuring intervention sustainability.4 The Booster sessions were conducted with high fidelity, and pain assessment and management outcomes were partially sustained over time. The small sample size (16 units) limited generalizability. A more comprehensive approach to measuring implementation (eg, acceptability, feasibility, costs), in addition to intervention fidelity, could have further explained the impact of the Booster intervention on sustainability. Sustainability of pain practice changes may have been influenced by unit context, including unit priorities or the availability of resources and leadership support. Further analysis of factors implicated in successful implementation may have identified areas that affected outcomes.22
Implications for Research
Researchers should explore a wider range of contextual variables related to implementation success and sustainability, as outlined in the Consolidated Framework for Implementation Research28 and recommended by Proctor et al.3 Future efforts should focus on implementation strategies that target factors related to intervention uptake, adaptation, and/or sustainability. Research on advancing development of sustainability measures, applicable in a broad range of health care settings, is required.
Understanding factors that contributed to “evaporation” in pain practice gains could inform the redesign and evaluation of a less resource-intensive Booster intervention. Although there is a need to better comprehend the determinants of sustainability, there is also a need to design simpler, more accessible ways to achieve and sustain practice change, as EPIQ has been described as resource and time-intensive.7,8
Sustainability of practice change is an important consideration in intervention and implementation research. However, sustainability is complex and not well understood. Expected benefits of sustainability, ongoing systems to monitor progress, adaptability of processes, leadership, financial and human resources, and stakeholder support need to be further examined.3,29 Conceptual clarity on the influence of contextual factors may unveil mechanisms related to interventions and outcomes (eg, moderators or mediators), which will inform implementation strategies that facilitate sustainability.
Dr Stevens was supported by the Signy Hilder Eaton Chair in Pediatric Nursing Research at The Hospital for Sick Children from 2000–2015. Dr Holsti (from the CIHR Team in Children’s Pain) is supported by a Canadian Institutes of Health Research Canada Research Chair in Neonatal Health and Development.
Members of the CIHR Team in Children’s Pain (Listed Alphabetically): Fiona Campbell, MD, FRCA, The Hospital for Sick Children and University of Toronto, Toronto, Ontario, Canada; Christine Chambers, PhD, RPsych, IWK Health Centre and Dalhousie University, Halifax, Nova Scotia, Canada; Janice Cohen, PhD, CPsych, Children’s Hospital of Eastern Ontario, Ottawa, Ontario, Canada; Greta Cummings, RN, PhD, FCAHS, FAAN, University of Alberta, Edmonton, Alberta, Canada; Carole A. Estabrooks, RN, PhD, FCAHS, FAAN, University of Alberta, Edmonton, Alberta, Canada; G. Allen Finley, MD, FRCPC, FAAP, IWK Health Centre and Dalhousie University, Halifax, Nova Scotia, Canada; Denise Harrison, RN, PhD, Children’s Hospital of Eastern Ontario and University of Ottawa, Ottawa, Ontario, Canada; Liisa Holsti, OT, PhD, Children’s and Women’s Health Centre of BC and University of British Columbia, Vancouver, British Columbia, Canada; Céleste Johnston, RN, DEd, FCAHS, McGill University, Montréal, Québec, Canada; Margot Latimer, RN, PhD, IWK Health Centre and Dalhousie University, Halifax, Nova Scotia, Canada; Shoo Lee, MD, PhD, FRCPC, Mount Sinai Hospital, The Hospital for Sick Children and University of Toronto, Toronto, Ontario, Canada; Sylvie LeMay, RN, PhD, CHU Ste-Justine and Université de Montréal, Montréal, Québec, Canada; Patrick McGrath, PhD, FRSC, IWK Health Centre and Dalhousie University, Halifax, Nova Scotia, Canada; Judy Rashotte, RN, PhD, Children’s Hospital of Eastern Ontario, Ottawa, Ontario, Canada; Christina Rosmus, RN, MSc, Montreal Children’s Hospital and McGill University Health Centre, Montréal, Québec, Canada; Doris Sawatzky-Dickson, RN, MN, Children’s Hospital of Winnipeg Health Sciences Centre, Winnipeg, Manitoba, Canada; Shannon Scott, RN, PhD, Stollery Children’s Hospital and University of Alberta, Edmonton, Alberta, Canada; Souraya Sidani, RN, PhD, Ryerson University, Toronto, Ontario, Canada; Janet Squires, RN, PhD, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; Jennifer Stinson, RN, PhD, The Hospital for Sick Children and University of Toronto, Toronto, Ontario, Canada; Anna Taddio, PhD, The Hospital for Sick Children and University of Toronto, Toronto, Ontario, Canada; J. Charles Victor, MSc, PStat, Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, Ontario, Canada; Fay Warnock, RN, PhD, Children’s and Women’s Health Centre of BC and University of British Columbia, Vancouver, British Columbia, Canada; and Andrew R. Willan, PhD, The Hospital for Sick Children and Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada.
- Accepted July 5, 2016.
- Address correspondence to Bonnie J. Stevens, RN, PhD, FCAHS, FAAN, The Hospital for Sick Children, 686 Bay St, Room 06.9712, Toronto, Ontario, Canada, M5G 1X8. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: All phases of this study were supported by the Canadian Institutes of Health Research (CTP-79854, MOP-86605, and MOP-111025).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Lee SK,
- Aziz K,
- Singhal N, et al
- Donner A,
- Klar N
- Cosby RH,
- Howard M,
- Kaczorowski J,
- Willan AR,
- Sellors JW
- Taljaard M,
- Donner A,
- Villar J, et al; World Health Organization 2005 Global Survey on Maternal and Perinatal Health Research Group
- Rogers EM
- Stevens B,
- Estabrooks CA,
- Yamada J, et al
- Davies B,
- Edwards N
- Copyright © 2016 by the American Academy of Pediatrics