BACKGROUND: The basic knowledge and skill base to resuscitate a newborn infant is taught in the Neonatal Resuscitation Program (NRP). We hypothesize that caregivers will perform below current acceptable standards before the recertification period of two years.
METHODS: This is a prospective descriptive study evaluating performance of pediatric residents’ NRP knowledge and skills over time. NRP scores are used as baseline data. Follow‐up is performed before the resident's first NICU rotation. Differences in the mean scores are analyzed for degree of retention. Subset score analysis is also performed.
RESULTS: Eighty-eight subjects completed both evaluations. Knowledge scores maintained close to passing throughout the academic year. Subset evaluation revealed significant deficits within the intubation lesson. Alarming deficits were seen in skills evaluation starting at initial NRP certification with 39.1% residents having failing scores. Mean scores were below passing for every group on follow-up testing. Subgroup analysis of skills revealed deficits in the initial phases of resuscitation (lessons 1–3).
CONCLUSIONS: Deterioration of skills is seen shortly after training. It appears that knowledge is generally better retained. Discrepancies between areas of knowledge and skill deterioration indicate that proficiency in one does not necessarily indicate proficiency of the other.
- AAP —
- American Academy of Pediatrics
- NRP —
- Neonatal Resuscitation Program
What’s Known on This Subject:
Skills learned in standardized courses are estimated to last only a few months. Neonatal Resuscitation Program certification is mandatory for all pediatric residents and is valid for 2 years. Exact timing of when proficiency is lost is unknown.
What This Study Adds:
Neonatal Resuscitation Program skills deteriorate immediately after certification, whereas knowledge is better retained. Significant skill deficits were seen at baseline raising concerns regarding the efficacy of the current course structure. Discrepancies in knowledge and skill retention may impact caregiver performance.
Proficiency in neonatal resuscitation is paramount to ensuring the safety and well-being of newborn infants. Optimal care provided during the first few minutes of life plays a crucial role in reducing morbidity and mortality. Currently the Accreditation Council for Graduate Medical Education mandates completion of formal training, via the neonatal resuscitation program (NRP), by all pediatric residents. Historically this is taught at the beginning of the residency-program training period. The NRP was developed by the American Academy of Pediatrics (AAP) and the American Heart Association in 1987 with the original goal of having an individual available at every birth that was capable of initiating resuscitation if needed. Since its inception, >2.4 million people have been trained and certified.1
During NRP training, caregivers are taught the basic knowledge and technical skills needed to resuscitate a newborn infant. NRP efficacy has been validated by several studies showing significant improvement in knowledge regarding newborn resuscitation after course completion.2,3 However, there can often be a significant time lapse from initial NRP training to the time of resident NICU rotations. In addition, retention of the skills learned in standardized courses has been shown to last only 4 to 8 months.2–5 Studies have shown when senior residents are tested at a time distant from original training, they often perform subpar on key lifesaving skills regardless of their own perceived ability.6,7 These skills are vital as up to 10% of infants will require some assistance at birth and 1% require significant resuscitative efforts immediately after birth.8,9 Given this, it is even more concerning that the majority of resuscitation teaching done in the delivery room is provided by senior residents.10 Residents are increasingly limited by the Accreditation Council for Graduate Medical Education–mandated restrictions on the amount of time spent rotating through the NICU. After residency, 45% of general pediatricians will routinely attend deliveries, and 71% will remain involved in the care of sick newborns.11 This lack of exposure coupled with continued involvement in neonatal resuscitation is concerning given the potential of poor retention of NRP skills and knowledge.
The AAP and American Heart Association have currently set the certification period for NRP providers at 2 years. We hypothesize that pediatric residents will perform below the current acceptable standards before this time period. The purpose of this study is to describe the decline in both knowledge and skills with time, paying special attention to which subcategories decline the fastest.
This was a prospective descriptive study evaluating the natural progression of pediatric interns’ NRP knowledge and skills assessment (megacode) performance over time. Study subjects included all first year pediatric residents at the Children’s Hospital of Philadelphia during the academic years of 2009–2010 and 2010–2011. All incoming residents were eligible for participation in the study. The only exclusion criterion was a physical disability prohibiting the performance of resuscitative maneuvers. All study data were collected anonymously, and participation was voluntary. The hospital’s institutional review board approved this study.
The primary outcome of this study was the length of time for either knowledge or skill assessment scores to fall below the current recommended passing standards for NRP certification. Currently this is set as >80% on the knowledge assessment and >85% on the megacode assessment PLUS completion of 5 critical tasks.
Evaluations were performed at 2 time points. The first was during initial NRP certification, which was provided to all study subjects at the beginning of their academic year. Scores from the certification course were used as baseline data with the assumption that this would serve as the peak level of performance. The NRP course was provided during a 6-hour block in the resident orientation schedule and was structured with traditional lectures followed by small group skills training with an instructor to participant ratio of 1:6. Approximately 40% of time was spent in didactics and 60% in skills training. The NRP course allowed for all participants to have coached hands-on skills practice. The second assessment was performed at the start of the resident’s first NICU rotation before clinical exposure. Residents at our institution rotate through the NICU at 4-week intervals, generally with 4 residents every month, thus giving each group a different length of time from initial NRP training.
Knowledge was analyzed by using the standard written examination taken from the NRP instructor’s manual. The NRP manual is grouped into 9 lessons, each dealing with a specific aspect of neonatal resuscitation. Only the first 6 lessons of the test were used in the analysis as these were felt to be the most relevant to the everyday practice pattern of the residents. These same lessons correspond to the 6 lessons scored in the megacode assessment. The topics for each lesson are as follows: Lesson 1: Overview and principles of resuscitation, Lesson 2: Initial Steps of Resuscitation, Lesson 3: Use of resuscitation devices for positive pressure ventilation, Lesson 4: Chest compressions, Lesson 5: Endotracheal intubation, and Lesson 6: Medications.
Skills were analyzed by using the megacode assessment form provided in the NRP instructor’s manual. This standardized checklist is grouped into 17 to 19 items (depending on scenario) with each being scored from 0 to 2 (0, not performed; 1, performed incorrectly or out of order; 2, performed correctly). All study participants were given identical scenarios during NRP certification. Because of the large number of residents, megacode scenarios during NRP certification were done by using a low-fidelity mannequin and without assistants. Follow-up testing was done in smaller groups as part of an educational curriculum day and was completed by using a high-fidelity mannequin with advanced simulation and debriefing techniques. On follow-up testing, each study subject was given 1 of 2 scenarios with the only difference being the presence or absence of meconium-stained amniotic fluid. Subjects were given standardized orientation and instructions before all case scenarios. A third-party assistant who was an NRP instructor but not directly involved in the study analysis participated in all scenarios but was told only to intervene when instructed. Each scenario was video-recorded and scored by a study investigator. Access and review of the neonatal resuscitation textbook before assessments was not controlled.
Study subjects were given a questionnaire at each follow-up session that elicited previous experience in neonatal resuscitation, comfort level in performing various skills, and interest level in pursuing neonatology as a career. This questionnaire was kept anonymous for privacy concerns.
Test scores and all other continuous variables are described by using mean and standard deviations. Differences between continuous variables for each study subject were compared by using paired student t tests. Association between confounders and scoring were quantified by using 1-way analysis of variance testing. Pairwise testing comparison between groups was performed by using the Tukey-Kramer method. Results for all tests were considered statistically significant at an α level of .05. All statistical analysis was performed on Stata IC 11.0.
Initial NRP certification was completed by 92 subjects. Results for both academic years are shown in Table 1. Knowledge scores were above passing with no subject having a failing overall score. The lesson with the lowest performance was lesson 5 (intubation) with 14 of 92 (15.1%) receiving a failing score. All other lessons had <10% fail rate. Megacode scoring was more variable with overall mean of 86.8% (10.5) and range of 55.6% to 100%. Overall 36 of 92 (39.1%) subjects achieved a failing score on the megacode assessment. All subjects achieving a failing score on any knowledge or megacode subsection were remediated before achieving NRP certification.
Eighty-eight residents completed both NRP certification and follow-up testing. Follow-up scores for each time period are depicted in Table 2. Overall knowledge scores stayed close to the passing threshold through the entire academic year with only 2 groups achieving an overall failing score. Ten of 12 groups had a statistically significant drop in their scores when compared with baseline however each group had a starting average of >10% higher than the passing threshold.
Subgroup performance on the knowledge examination is depicted in Figure 1. The vast majority of groups achieved a passing score on all subsections of the written examination. The exception was the intubation subsection. Despite all failing subjects receiving remediation during NRP certification, it remained the most failed subsection on follow-up. Fifty-nine of 88 (67.0%) subjects failed this lesson on follow-up testing. Other lessons with significant fail rates were lesson 3 (23.9%) and lesson 4 (19.3%).
Follow-up scores for each time period are depicted in Table 3. Despite the overall group average over 2 years being above passing, when analyzed by individual group scores, it was found that 3 of 12 groups started with average scores that were below passing. On follow-up testing 12 of 12 (100%) groups had average scores that were below the passing threshold. On the individual level, only 8 of 88 (9.1%) subjects achieved a passing score on follow-up assessment, and only 1 passing score was achieved in the groups tested at >2 months from NRP certification. Eight of 12 groups had a statistically significant drop in their scores when compared with baseline.
Subgroup performance on the megacode assessment is depicted in Figure 2. Lessons 2 and 3 had an overall failing average at NRP certification. Follow-up testing shows a failing average in the majority of groups for lessons 1 through 3. Overall fail rates for these lessons were 59.1%, 85.2%, 87.5%, respectively. Problems in lesson 1 included failure to ask appropriate questions and check equipment properly. Lesson 2 deficits included failure to properly dry, stimulate, and perform oropharyngeal suctioning. The most common failed task in lesson 3 was failure to troubleshoot the efficacy of positive pressure ventilation. There was also a significant overall failure rate of lesson 6 (34.1%); however, this is largely due to the marked increase in the number of subjects that failed this section starting 4 months from NRP certification.
The results of 1-way analysis of variance testing show evidence of a statistically significant difference between overall comfort (low, medium, and high) in performing resuscitative maneuvers and performance on overall megacode (P = .03). This effect was not significant in analysis of knowledge performance (P = .12). There is an overall increase in megacode scores with increasing comfort level however pairwise comparison revealed there was only a statistically significant difference (P < .05) when the low-confidence group was compared with the high-confidence group. Additional testing also revealed there was no statistically significant association among (1) interest level in pursuing neonatology as a career, (2) previous resuscitation experience, and (3) subgroup confidence on either overall or subgroup performance.
Certification in NRP does not imply competence. This study and several others have shown across a variety of caregivers and time periods that skill and knowledge retention deteriorate before the recertification time period. What this study adds is a more detailed and comprehensive look at deterioration of skills and knowledge in US-trained residents using up-to-date guidelines and evaluation tools. It also analyzes the subsections of NRP in closer detail than has been done previously.
What we have observed is a marked discrepancy between retention of knowledge and skill by pediatric residents. Overall knowledge was retained much longer than skills with averages remaining above or close to the pass threshold throughout the academic year. Despite the large number of groups having a statistically significant fall in scores, the clinical significance of this was not felt to be of great impact. This high degree of knowledge retention is in contrast to what has been observed in other studies. Review of NRP materials before follow-up assessments was not controlled in this study, and this could serve as a reason for this. Subsection analysis revealed significant deficits in the intubation lesson. It is unclear why this section in particular caused so much difficulty; however, anecdotal experience would indicate that many residents have difficulty with interpretation of the numerous low-fidelity drawings in this testing section.
Analysis of the skills assessments revealed a markedly different pattern of deficiencies. Overall skill scores at NRP certification were much lower than anticipated. No group achieved a passing score on follow-up testing, and overall skill scores took a sharp fall starting 2 to 3 months from NRP certification. Previous studies have documented a fall in NRP skills when tested at a distant time; however, this study shows that (1) skills may not be taught well enough during NRP to establish adequate competency and (2) skills start to fall almost immediately after certification. Subsection analysis shows significant deficits in the performance of the initial phases of resuscitation. This is extremely concerning because the majority of steps in lessons 1 and 2 should be performed universally at all deliveries. Deficiencies in positive pressure ventilation could partly be explained by lack of experience with flow inflating bags; however, all study participants are given a short tutorial explaining the basic functioning of all equipment before assessments. Furthermore, the efficacy of chest compressions may have been overestimated given that detailed quantitative analysis (force, depth, flow time) was not available given the current software.
Overall the striking observation is the marked contrast between deficiencies in knowledge and skill. There seems to be poor correlation between the retention of each. Data from Basic Life Support show an indeterminate correlation between written tests and performance-based measures; however, it has been documented in Advanced Cardiovascular Life Support that there is poor correlation between written testing scores and corresponding skill performance.12,13 This has also been suggested in pediatric studies showing high levels of knowledge retention for pediatric advanced life support–related items but poor performance when basic and advanced skill maneuvers were tested.5,7
Another interesting finding in our study is the improved overall megacode performance with increasing confidence level. A previous report in pediatric residents showed high confidence levels in performing various pediatric advanced life support–related maneuvers; however, these same residents had significant deficits on skills training.6 Our study showed an improvement in skill scores with higher levels of confidence. Even with the overall trend toward improved scores with increasing confidence, the median scores for all 3 groups in the skills assessment was still below passing.
Recently the AAP has brought up concerns regarding the efficacy of the teaching methods used in traditional NRP teaching. The NRP steering committee has addressed these concerns by revamping the course structure. Currently the majority of the course is spent in lectures with a smaller percentage of time devoted to hands on training. Future courses will emphasize knowledge acquisition before the course and more emphasis on simulation training for skill acquisition. A recent study showed that subjects in a self-taught group followed by simulation training performed similarly to traditionally taught NRP subjects and had similar satisfaction scores, indicating that this method of teaching is likely just as effective.14 As an ongoing effort to improve NRP education, the AAP has set out specific areas with high research priority. Exploration and development in these areas will help guide how future NRP courses are developed and taught.
We anticipated performance to deteriorate quickly after NRP certification. Currently there is no consensus in the literature suggesting optimal timing and content of refresher courses. At our institution, we have given courses before NICU rotations; however, caregivers could be exposed to neonatal resuscitations in other locations of the hospital at variable time points before this refresher. Although this study was not specifically designed to assess for this, it would seem useful, on the basis of the results seen in this study, to perform skill refresher sessions every few months. Unfortunately, the logistics of performing these sessions with a large group of people is often difficult. One study did show residents given additional resuscitation exposure during training have improved confidence and performance on written and skills assessments.15 Another showed an improvement in knowledge and skill scores 6 to 8 months after an NRP “booster” course given 3 to 5 months after initial NRP training; however, the results were not significant and could not provide recommendations for timing and structure of such a course.3 On the basis of the data from these studies and our own, it seems as if skills quickly deteriorate after initial training and reach a plateau level. Unfortunately, the scores achieved at this plateau fall well below passing requirements for NRP certification.3,4
There are a few points to consider when interpreting our results. The evaluations were done on first-year pediatric residents, who have little previous neonatal resuscitation experience. It was also performed at a single tertiary-care center at an academic institution. Whether similar findings would be seen in caregivers who frequent the delivery room is still unknown. Multiple NRP instructors scored the initial NRP course, whereas a single observer scored the follow-up sessions. This resulted in some variation in scoring of the initial certification course; however, the AAP has designed the scoring sheet in anticipation of this variability. Furthermore, the scoring sheet is an objective scoring tool and should not involve any subjective measures. Every attempt to have subjects suspend disbelief was made; however, performing scenarios in an artificial environment can occasionally remove realistic cues and actions that would normally be seen and performed during actual resuscitations. This is an ongoing battle with simulation education.
Retention of information taught in standardized courses has always been in question. This study shows that although knowledge acquisition is fairly robust, the ability to acquire skills may be much lower given current teaching methods. Coupled with poor retention, this leads to drastically lower than expected performance at future dates. NRP instructors and caregivers in neonatal intensive care units should be aware of this decline and have measures set up for frequent refresher sessions.
Although it is not practical to teach a formal NRP course at the beginning of each rotation, a refresher course that reviews the core skills and knowledge base of NRP could serve as an adequate stimulus to help recover any lost skills. Although the timing of this course could be extrapolated from the results of this study, the optimal content and method of delivery of this refresher course has yet to be determined and should be the focus of future studies.
- Accepted May 3, 2012.
- Address correspondence to Jay Patel, MD, Department of Pediatrics, Division of Neonatology, Children’s Hospital of Philadelphia, 34th and Civic Center Blvd, Philadelphia PA 19104. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
- ↵NRP Instructor Update Fall/Winter. 2008. Available at: www.aap.org/nrp/newsletter/2008_fallwinter_iu.pdf. Accessed June 2012
- American Heart Association,
- American Academy of Pediatrics
- Katwinkel J
- Halamek LP,
- Kaegi DM
- Walton DM,
- Edwards MC
- Weiner GM,
- Menghini K,
- Zaichkin J,
- Caid AE,
- Jacoby CJ,
- Simon WM
- Copyright © 2012 by the American Academy of Pediatrics