Abstract
OBJECTIVES: This first-ever country-level study assesses the implementation of the Helping Babies Breathe (HBB) program in 15 of Tanzania’s mainland regions by measuring coverage, adoption and retention of provider skills, acceptability among providers, and barriers and challenges to at-scale implementation.
METHODS: Longitudinal facility-level follow-up visits assessed provider resuscitation knowledge and skills in using objective structured clinical examinations and readiness of facilities to resuscitate newborns, in terms of birth attendants trained and essential equipment available and functional. Focus group discussions were held with providers to determine the acceptability, challenges, and barriers to implementation of the HBB program.
RESULTS: Immediately after HBB training, 87.1% of providers passed the objective structured clinical examination. This number dropped to 79.4% at 4 to 6 weeks and 55.8% at 4 to 6 months (P < .001). Noting this fall-off in skills, the program implemented structured on-the-job training and supportive supervisory visits, which were associated with an improvement in skill retention. At long-term follow-up, >90% of facilities had bag-mask devices available to all beds in the labor and delivery ward, and 96% were functional. Overall, providers were highly satisfied with the HBB program but thought that the 1-day training used in Tanzania was too short, so they would welcome additional training and follow-up visits to reinforce skills.
CONCLUSIONS: The HBB program in Tanzania has gained acceptability and shown success in equipping providers with neonatal resuscitation knowledge, skills, and supplies. However, assessing the program’s impact on neonatal mortality has proven challenging.
- CIFF —
- Children’s Investment Fund Foundation
- HBB —
- Helping Babies Breathe
- MOHSW —
- Ministry of Health and Social Welfare
- OR —
- odds ratio
- OSCE —
- objective structured clinical examination
What's Known on This Subject:
A pilot study of the Helping Babies Breathe program, a newborn resuscitation training program designed for birth attendants, conducted in referral-level facilities in Tanzania showed a 47% reduction in all-cause newborn mortality occurring in the first 24 hours of life.
What This Study Adds:
We present results from the 3-year national rollout of the Helping Babies Breathe program in Tanzania. The program was successful in equipping providers with life-saving newborn resuscitation skills and equipment; however, assessing impact on mortality requires greater efforts.
Neonatal mortality accounts for almost half of all deaths among children <5 years old, and nearly all these deaths occur in low- and middle-income countries.1 Birth asphyxia, the inability to start or maintain normal breathing at birth, is a leading cause of neonatal mortality in these settings, where many births occur without the attendance of a provider capable of resuscitating an asphyxiated newborn.2,3 Therefore, programs to prevent asphyxia-related neonatal mortality, such as Helping Babies Breathe (HBB), have focused on increasing birth attendants’ skills in newborn resuscitation.
The American Academy of Pediatrics and partners launched the HBB program in 2009, and it is now being used in >77 countries worldwide.4 An important pilot HBB study, conducted in 8 referral hospitals in Tanzania, showed a remarkable 47% reduction in all-cause newborn mortality within the first 24 hours of life.5 Based on these encouraging findings, the Tanzania Ministry of Health and Social Welfare (MOHSW) prioritized national expansion of HBB. The Children’s Investment Fund Foundation (CIFF) supported the MOHSW and implementing partner, Jhpiego, in the rollout of HBB to 15 of Tanzania’s mainland regions.6,7 Additionally, CIFF contracted a Harvard-based evaluation team to independently evaluate the impact and effectiveness of the HBB implementation.
This article presents findings from this first large-scale implementation of HBB in Tanzania. The goals of this study were to evaluate program coverage, determine the impact of HBB training on provider skills over time through testing on newborn mannequins, understand provider-level acceptability and perceptions of HBB in Tanzania, and identify the barriers and challenges to implementing HBB at scale.
Methods
HBB Program Implementation
The details of program implementation have been discussed elsewhere,6,8 and a summary is provided here. The CIFF-supported HBB program was rolled out in 15 of Tanzania’s 25 mainland regions from May 2013 through December 2014. The HBB course was adapted from the American Academy of Pediatrics’ HBB curriculum.9
National and district trainers were chosen from a pool of HBB master trainers who were trained in 2009 by the MOHSW. All trainers received refresher trainings and then began a regional rollout of HBB courses to health care providers (Fig 1). Master trainers from the MOHSW and implementing partner traveled to each region and assisted regional trainers in conducting the local trainings. The local trainings were held at or near regional and district hospitals, with providers from lower-level facilities joining providers at these facilities for the trainings. Health care providers from labor and delivery wards and obstetric theaters were eligible to attend the HBB course. The maximum number of eligible providers from each facility level was 20 providers per hospital, 8 providers per health center, and 3 providers per dispensary. A trainer-to-trainee ratio of 1:6 was maintained for trainings. After the training, all health facilities were provided with HBB equipment and training materials, as well as instructions on how to properly use and clean them. Providers who attended the course were expected to return to their facilities and use the training materials to provide on-the-job training to colleagues who could not attend the course, although these colleagues would not be considered formally HBB trained.
Schematic overview of the HBB training program and its evaluation via a region-by-region approach across 3 illustrative regions in Tanzania.
An objective structured clinical examination (OSCE), previously validated and reported by our group, was used longitudinally to assess provider-level adoption and retention of HBB knowledge and skills.6 The OSCE was administered by a national or district trainer to each provider individually. During the OSCE, providers used a mannequin to show skills in 13 tasks relating to newborn resuscitation. Scores could range from 0 to 23, with passing scores starting at 16. Trained providers were first assessed immediately upon completion of the HBB course and again at 4 to 6 weeks after training. Any observed skills gaps were addressed after each participant’s scoring.
Interim Interventions
In February 2014, partners attended the HBB program’s first annual program review, during which there was noted concern over a drop in OSCE scores between immediately after training and the 4- to 6-week follow-up visits. Additionally, when trained providers returned to their facilities, there was limited self-initiated practice. Therefore, program partners developed specific interim interventions to address these concerns. Beginning in May 2014, Jhpiego equipped providers with a structured on-the-job training tool, which guided group practice of HBB skills by using a newborn mannequin. Supportive supervision visits were also instituted between local HBB trainers and providers at neighboring facilities who were HBB trained.
External Evaluation Strategies
CIFF contracted an external evaluation team from Harvard Medical School and Massachusetts General Hospital to conduct an independent and longitudinal evaluation of the HBB training program in Tanzania. The goals of the evaluation included measuring changes in knowledge and skills of providers at targeted health facilities and providing the implementing partner with timely, actionable feedback on program impact. To achieve these goals, the evaluation team conducted unannounced facility-level follow-up visits at 4 to 6 weeks and 4 to 6 months after training to assess provider skills and perceptions, document available equipment, review records, assess program impact on asphyxia-related neonatal mortality, identify barriers and facilitators to scale-up, and undertake a costing analysis by using validated, field-tested tools (Table 1). We attempted facility-level collection of neonatal outcome data from the primary source registers among a sample of facilities, but as other country-level studies have found, historical data were not reliable. Some of these results are reported elsewhere, and the current analysis will focus on findings from the OSCEs, facility checklists, and focus group discussions.7,10
Methods Used to Evaluate the Impact of the HBB Program in Tanzania
Trained evaluation team members conducted OSCEs on all HBB-trained staff present at a facility during the 4- to 6-week and 4- to 6-month facility follow-up visits. The sampling method was a stratified, purposive sample of facilities within all 15 districts receiving the CIFF-supported HBB program. Facilities were chosen to ensure that regional and district hospitals (n = 33), health centers (n = 35), and dispensaries (n = 163) were all represented.
Data Analysis
Data analyses included descriptive and inferential analyses, including frequencies, means, and univariate and multivariate logistic regressions. The mean OSCE scores at the 4- to 6-week follow-up visits conducted by the implementing partner and by the evaluation team were compared via paired t tests and a significance level of P < .05. All quantitative statistical analyses were done in Stata version 12 (Stata Corp, College Station, TX).
For analysis of the focus group discussions, an inductive thematic approach was applied to the written discussion notes.11 Investigators used NVivo 10 software (QSR International, Victoria, Australia) for data management and theme generation, along with regular peer consultations to increase interpretive rigor.
Ethical Considerations
This study was approved by the institutional review board at Massachusetts General Hospital (Boston, MA), the National Institute for Medical Research (Dar es Salaam, Tanzania), and the Ministry of Health and Social Welfare (Dar es Salaam, Tanzania). Informed verbal consent was obtained from all study participants.
Results
Participants
There were 13 169 providers who attended the HBB training. Two (0.02%) of these providers were removed from the analysis because they had incomplete OSCE data in which ≥1 of the OSCE questions had a missing value. Of the 13 167 providers included in the analysis, the majority were nurses or nurse-midwives (51.5%) and providers working at dispensaries (54.9%) (Table 2).
Health Cadre and Facility Level of Providers Attending HBB Courses
Program Coverage
At the 4- to 6-week follow-up visits, the evaluation team found that the percentage of birth attendants who received formal HBB training varied by region from 50.8% to 88.0% (Table 3). By the 4- to 6-month follow-up visits, all regions except Tanga and Dar es Salaam saw a decline in the percentage of formally HBB-trained birth attendants (range 40.4%–71.3%). The percentage of deliveries attended by a formally trained HBB provider in the past month ranged from 50.3% to 97.8% at 4 to 6 weeks and from 23.6% to 90.1% at 4 to 6 months.
HBB Training Coverage of Birth Attendants and Deliveries
Provider Skills
Jhpiego conducted 22 512 OSCEs as part of HBB implementation, of which 22 454 have complete OSCE data: 13 167 (58.6%) immediately after HBB training and 9287 (41.4%) at the 4- to 6-week follow-up visits. The evaluation team conducted among a stratified sample of facilities an additional 1004 OSCEs, of which 984 have complete OSCE data: 726 (73.8%) at the 4- to 6-week follow-up visits and 258 (26.2%) at the 4- to 6-month follow-up visits.
There were 681 providers that had OSCE assessments at 4 to 6 weeks by both the implementing partner and the evaluation team. A paired t test of this subset of data showed a statistically significant difference between the mean OSCE score of those conducted by the implementing partner (mean 16.4; SD 4.5) and the mean OSCE score of those conducted by the evaluation team (mean 16.8; SD 4.3) (t = −3.061; P = .002). Although the difference is statistically significant, the absolute difference (0.4 points out of 23 points) helps confirm a previously determined high interrater reliability and the ability to compare OSCE scoring across partners.6
The mean OSCE score was highest (mean 18.9; SD 3.3), and the greatest percentage of providers passed the OSCE (87.1%) immediately after HBB training, compared with the follow-up visits (Table 4). The mean OSCE score dropped at the 4- to 6-week follow-up visits conducted by the implementing partner (mean 17.9, SD 4.1; 79.4% passing) and by the evaluation team (mean 16.8, SD 4.2; 68.3% passing). By the 4- to 6-month visits, the mean OSCE score had decreased to 15.8 (SD 4.7), with 55.8% of providers passing the OSCE. However, there was a decrease in fall-off of scores over the course of the program that was associated with the introduction of interim interventions (Fig 2).
HBB OSCE Results by Region and Time Since Training
Implementation data on percentage of providers passing the HBB OSCE, by region and time since training. Regions are listed in the chronological order in which training occurred, beginning in May 2013 through November 2014.
Among the OSCE tasks that providers most often completed correctly were washing hands or using alcohol rub and then putting on gloves, drying the newborn, and improving ventilation with ≥1 technique (Table 5). Among the tasks that providers most often did not complete correctly were preparing an area for ventilation, checking the equipment before delivery, and stimulating breathing by rubbing the newborn’s back.
Percentage of Providers Correctly Completing Each OSCE Task, by Time Since Training
In unadjusted logistic regression, the odds of passing the OSCE significantly decreased with time since HBB training (4–6 week odds ratio [OR] 0.515, P < .001; 4–6 month OR 0.143, P < .001) (Table 6). Working at a dispensary, compared with at a hospital, decreased the odds of passing the OSCE (OR 0.716, P < .001), and medical assistants, compared with medical doctors, were less likely to pass the OSCE (OR 0.433, P = .002).
Unadjusted and Adjusted ORs For Passing the HBB OSCE, by Time Since Training, Facility Level, Health Cadre, and Region
In multivariate logistic regression, time since HBB training, facility level, and health cadre were all independent predictors of passing the OSCE, whereas region did not predict passing the OSCE. Providers were less likely to pass the OSCE at the 4- to 6-week visits and at the 4- to 6-month visits (4–6 week OR 0.520, P < .001; 4–6 month OR 0.126, P < .001) compared with immediately after HBB training. Providers working at dispensaries were less likely to pass the OSCE than providers working at hospitals (OR 0.850, P = .005), and medical assistants were less likely to pass the OSCE compared with medical doctors (OR 0.458, P = .006).
Availability of HBB Equipment
Bag-mask devices and penguin suction devices were highly available and functional over time (Table 7). The percentage of bag-mask devices that were available but in a location other than immediately accessible in the labor and delivery ward or obstetric theater decreased from 12% to 0% from the 4- to 6-week to 4- to 6-month visits. Over the same time, the percentage of facilities where bag-mask devices were available in all newborn resuscitation areas increased from 84% to 92%.
Availability and Functionality of Resuscitation Equipment, by Time
Focus Group Discussions
Over the course of the 3-year program, >200 focus group discussions were held with providers who received HBB training. Overall, the feedback was highly positive (Table 8). Providers reported feeling more confident and better equipped to resuscitate newborns. Birth attendants reported that the HBB equipment simplified resuscitation. Providers considered the supportive supervision visits and the follow-up visits critical for skill retention. They also appreciated the hands-on training on mannequins. However, they reported some dislikes and barriers, including that some providers still struggled to use the bag-mask device and thought that the 1-day training was too short to properly cover all the material. Additionally, providers thought that the financial incentives for training were too small and that intrafacility rotation of trained attendants limited the impact of the training.
Themes From Focus Group Discussions With HBB-Trained Providers
Discussion
This study adds valuable information on implementing and sustaining the HBB program, because it is one of the first and most extensive assessments of HBB at scale. We analyzed 3 years of monitoring and evaluation data from >13 000 health care providers in 15 of Tanzania’s mainland regions. Although the evaluation used several methods to assess the success of the program, including a cost analysis and qualitative interviews, the current study focuses largely on the assessment of providers’ resuscitation skills and equipment availability.
The initial success of HBB on developing provider-level resuscitation skills is evidenced by 87.1% of providers passing the HBB OSCE after training. However, to determine skill retention, we also assessed HBB skills over time. During the follow-up visits led by the implementing partner, the percentage of providers with passing OSCE scores decreased by 7.7% at 4 to 6 weeks and by 31.3% at 4 to 6 months. Early identification of this decline in skills prompted the development of interim interventions, which included structured tools to facilitate on-the-job HBB practice and supportive supervision by trained maternity ward leaders and district-level trainers. Similar to findings in a recent review of newborn resuscitation training approaches, we saw an improvement in HBB skill retention with the implementation of these interventions. Another remarkable accomplishment of the program was that the percentage of providers who passed the OSCE improved as the program matured (Fig 2). However, the odds of a provider passing the OSCE differed by the facility level in which they worked and their health cadre. The odds of passing the OSCE were 15% lower among providers working at dispensaries compared with hospitals. Furthermore, compared with medical doctors, medical assistants were less likely to pass the OSCE. However, medical assistants are usually unskilled birth attendants and perform very few deliveries. Skilled birth attendants such as nurse-midwives and clinical officers, who perform most of the deliveries, showed no significant difference in odds of passing the OSCE compared with medical doctors.
In addition to improving providers’ resuscitation skills, other successes of the HBB program in Tanzania were creating critical attention for newborn health and achieving widespread program coverage and implementation in an affordable way. In each region, by 4 to 6 weeks after implementation of the program, 68.7% of providers working in the maternity wards and obstetric theaters had received the HBB course. Coverage dropped to 57.1% at 4 to 6 months, probably because of attrition, new hires, and the common practice of interdepartmental staff rotation. Although the majority of providers were still trained in HBB at long-term follow-up visits, the decline in trained providers indicates the need for continued trainings. Coverage of resuscitation equipment, such as bag-mask devices and penguin suction devices, was high at >90% of facilities. Equipment also proved to be durable, in that almost all devices were still functional at the longer-term follow-up visits. The cost to train providers and equip the facilities was ∼$600 USD per health facility, equating to a national rollout cost of ∼$4 million, making HBB affordable in low-resource regions.10 A recent study examining HBB costs per lives saved in a rural missionary hospital in Tanzania found implementation costs of $233 USD per life saved ($4.21 USD per life year gained) and program maintenance costs of $80 USD per life saved ($1.44 USD per life year gained).12
There were challenges to at-scale implementation of the HBB program. Sustaining high coverage of trained providers was difficult, as discussed. Although attempts were made to measure program impact on neonatal mortality by using historic and existing health information systems, reliably quantifying this impact was not possible. Measuring mortality in resource-limited settings is challenging because deaths often go undocumented, especially before a program is implemented. Subsequently, with program implementation comes better documentation, which can initially lead to a perceived increase in mortality. HBB training also corrects providers’ often-misunderstood definitions of perinatal outcomes, which can lead to an apparent increase in newborn mortality (eg, previously miscategorized “stillbirths” are correctly categorized as newborn deaths). Options to help overcome some of these data challenges include parallel data collection systems, sentinel panel of districts or facilities, active surveillance, and composite indicators (eg, perinatal death). But these options come with significant financial and logistical costs for country-level programs.
Recognizing the challenges of resources and clinical coverage during trainings, the American Academy of Pediatrics supports 1- or 2-day HBB trainings for providers.9 The Tanzania MOHSW chose a 1-day training approach because of the desire to expeditiously and cost-effectively train a large number of providers across large geographic regions with finite resources. This 1-day training was complemented by the mentioned interim interventions, specifically facility-based supportive supervision visits and structured on-the-job training intended to mitigate any negative impact from the shorter training course. The ideal duration of training remains unclear because data show that frequent, short trainings are more effective than long, infrequent trainings.13 Therefore, the ideal neonatal resuscitation training program for large-scale implementation in resource-limited settings may permit 1-day courses if they are followed by frequent refresher or formal in-facility mentorship programs. Consistent with this approach, Jhpiego is currently implementing a new facility-based clinical mentorship program to address concerns with skill retention. Nevertheless, it is still possible that the shorter, pragmatic 1-day training approach reduced the potential impact of the HBB program on provider skills. The program was implemented largely as a stand-alone vertical training program; more effectively embedding HBB into comprehensive newborn programming and preservice training would probably help ensure cost-effectiveness and sustainability.
The current study does have limitations. First, this country-level HBB implementation in Tanzania may not necessarily be generalizable to other resource-limited settings. However, its 3-year design, large and diverse sample, and implementation within 15 large regions significantly increase the study’s representativeness. Second, social desirability bias could have occurred during focus group discussions. The use of an independent and experienced evaluation team and the confidential nature of the discussions probably mitigated this bias. Third, it is unknown whether good performance on OSCEs translates into improved care in the clinical setting.14,15 Fourth, some providers were lost to follow-up through attrition or rotation to other departments. Lastly, as discussed, we were not able to accurately measure the program’s impact on neonatal mortality because of the poor quality and availability of data in historic health information systems. Additional program focus and resources will probably be necessary to accurately measure mortality impact in these resource- and data-limited settings.
Conclusions
The HBB program in Tanzania trained >13 000 providers with high rates of knowledge and skill adoption. Although there was notable fall-off in skills over time, early identification of this fall-off and implementation of interim interventions probably contributed to the decline in skill fall-off. Importantly, the HBB program was highly praised and accepted among providers working in labor and delivery wards, although they request additional training to the 1-day program. Although the program has gained acceptability and shown success in equipping providers with neonatal resuscitation knowledge and skills, the program’s impact on neonatal mortality at scale is unknown because of the difficulty in measuring impact by using existing data systems.
Acknowledgments
The authors acknowledge and thank for their critical assistance Tanzania’s national and district HBB trainers, Tanzania’s regional and district health officials, Tanzania’s HBB Steering Committee and HBB Data Committee, and Jhpiego-Tanzania’s in-country leadership and newborn resuscitation program staff.
Footnotes
- Accepted February 9, 2017.
- Address correspondence to Brett D. Nelson, MD, MPH, DTM&H, Division of Global Health, Department of Pediatrics, Massachusetts General Hospital, 125 Nashua St, Room 8426, Boston, MA 02215. E-mail: bnelson1{at}mgh.harvard.edu
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This program and its independent evaluation were funded by the Children’s Investment Fund Foundation (London, UK).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
COMPANION PAPER: A companion to this paper can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2016-4172.
References
- Copyright © 2017 by the American Academy of Pediatrics
In this issue
