BACKGROUND AND OBJECTIVES: Pediatric residency training programs are graduating residents who are not competent in neonatal intubation, a vital skill needed for any pediatrician involved in delivery room resuscitations. However, a precise definition of competency during training is lacking. The objective of this study was to more precisely define the trajectory toward competency in neonatal intubation for pediatric residents, as a framework for later evaluating complementary training tools.
METHODS: This is a retrospective single-center observational study of resident-performed neonatal intubations at Duke University Medical Center between 2005 and 2013. Using a Bayesian statistical model, intubation competency was defined when the resident attained a 75% likelihood of intubating their next patient successfully.
RESULTS: A total of 477 unique intubation attempts by 105 residents were analyzed. The path to proficiency was defined by a categorical or milestone learning event after which all learners move toward competency in a similar manner. In our cohort, 4 cumulative successes were needed to achieve competency. Only 24 of 105 (23%) achieved competency during the study period. Residents who failed their first 2 opportunities, compared with those successful on their first 2 opportunities, needed nearly double the intubation exposure to achieve competency.
CONCLUSIONS: Bayesian statistics may be useful to more precisely describe neonatal intubation competency in residents. Achieving competency in neonatal intubation appears to be a categorical or milestone learning event whose timing varies between residents. The current educational environment does not provide adequate procedural exposure to achieve competency for most residents.
What’s Known on This Subject:
Pediatric residents may not be achieving competency in neonatal intubation. Opportunities for intubation during residency are decreasing. A precise definition of competency during training is lacking.
What This Study Adds:
Bayesian statistics may be used to describe neonatal intubation competency in residents. At least 4 successful intubations are needed to achieve competency. The first 2 intubation opportunities appear to predict how many intubation opportunities are ultimately needed to achieve competency.
Current trainees in pediatric residency programs are graduating without achieving competency in procedures such as neonatal intubation.1–5 Neonatal intubation remains a critical skill for the general pediatrician charged with delivery room (DR) and nursery coverage. The amount of exposure for pediatric residents to neonatal intubation in the NICU has decreased over time for several reasons, including the following: the more frequent use of noninvasive mechanical ventilation,6,7 recent changes to duty hour regulations, and the expansion of nonphysician providers such as neonatal nurse practitioners in many academic medical centers.8 With the emergence of the Accreditation Council for Graduate Medical Education’s (ACGME) Next Accreditation System9–11 and an ongoing movement toward competency-based graduate medical education (GME), there is a great need to describe the real-world attainment of the psychomotor skills necessary for performing neonatal intubations.
Further complicating this problem is the absence of a practical definition of competence in neonatal intubation and the lack of useful predictors to identify those trainees in need of additional training opportunities before their final year of residency. Currently, there is no consensus definition of intubation competency, although many training programs have identified deficiencies in procedural competence among their recent trainees. A study in 2005 evaluated 3719 intubation attempts by pediatric residents over a 10-year period and revealed that these residents were provided inadequate opportunity to become proficient in intubation, based on number of total procedures attempted and rates of success.12 A similar study attempted to define a successful intubation as occurring on the first or second attempt >80% of the time,13 which was based on a definition used in previous gastroenterology literature.14 None of their resident groups met this definition of competence, though likelihood of success improved marginally through advancement in training years. These previous studies have all focused on outcomes related to neonatal intubation such as resident success rate, time to intubation, and total number of intubations by residents.1–4,12,13,15 Using a Bayesian statistical model, the objective of this study is to define the path to competency in neonatal intubation more precisely, as a framework for later evaluating complementary training tools and improving intubation training.
Records of endotracheal intubations performed by pediatric and medicine–pediatric residents, neonatology fellows, neonatal nurse practitioners, and respiratory therapists in the DR and NICU at a single academic medical center have been collected as a quality assurance (QA) measure since 2005. A standardized data collection form records resident postgraduate year (PGY), procedure dates, use of rapid sequence intubation (RSI), total number of attempts, and outcome of each attempt for each intubation opportunity. For this study, we only analyzed a cohort of pediatric and combined medicine–pediatric residents, who had completed 3 full academic years of residency, from the 2005–2006 to the 2011–2012 academic year. Categorical pediatric residents have in general a total of 3 months in NICU rotation time during residency. Combined medicine–pediatric residents have 1 less month of rotation time than categorical pediatric residents. With the ACGME duty hours changing during the study period, PGY-1 residents no longer rotated at night starting in the 2011 academic year, with PGY-2 residents covering the unit in alternating night shifts. The Duke University Institutional Review Board approved this study.
Residents receive no specific intubation training curriculum, besides the mandatory certification through the neonatal resuscitation program. We defined an intubation opportunity as any procedural opportunity where the resident inserted a laryngoscope blade into the oropharynx of an infant, regardless of whether an attempt at passing an endotracheal tube was made. Guidelines exist in our NICU that restrict residents from intubating infants who are <1000 g at the time of the procedure. Although chest auscultation and end tidal carbon dioxide detectors provided the most immediate clinical feedback on outcome of the procedure attempt, final determination of successful placement of the endotracheal tube was confirmed by a chest radiograph. The number of attempts allowed was at the discretion of the supervising physician. There was no time limit for attempts. A second attempt on the same patient was allowed, and treated as belonging to the same opportunity, meaning success on the second attempt would be considered a success for that opportunity. DR intubations performed for meconium stained amniotic fluid were not included in this study, because confirmation of endotracheal tube placement was not possible. Most resident intubations occur in the NICU and not the DR, because unit policy calls for the most experienced intubator to perform the procedure in the DR.
Achievement of intubation competency was defined by using a Bayesian statistical model. Bayesian statistics have been used in a variety of medical education paradigms including analyzing progress test scores, medical simulation, and health care analyses in general.16–18 Bayesian statistics are useful in this investigation for several reasons. Intubation is a skill for which previous experience is a determinant of future success. Bayesian models are better able to analyze how previous success or failure will affect future performance. In Bayesian statistics, outcomes can be predicted by using what are known as previous distributions. Previous distributions are known objective or subjective observations about a phenomenon that can be used to predict the likelihood of future outcomes.
For this analysis, we used the observed previous distribution of our novice learners’ known success or failure of intubation attempts on their first attempt as collected before this study in our QA database and thus defined our novice cohort as having a 30% likelihood of intubating the next patient. Our statistical model then assigned each resident at any given intubation opportunity, a Bayesian likelihood score (how likely they were to successfully intubate the next patient) based on their performance before that opportunity. The model was used to define how long it took a resident to move from the novice cohort toward the competent cohort, as a function of number of intubation opportunities encountered and their cumulative intubation successes over the course of their residency training. To ensure reliability of the Bayesian model, the Bayesian likelihood score (the predicted likelihood of success at the next intubation opportunity) was compared retrospectively, using logistic regression, to the actual outcome of the resident’s next opportunity captured in the database. Competency was defined when the provider was 75% likely to intubate the next patient successfully (at the next opportunity, whether the first or second attempt).
We then sought to determine if early intubation opportunities could be used to predict the number of intubation opportunities needed to achieve competency. To do so, we selected a cohort of residents who had achieved competency in intubation as defined by the parameters mentioned previously. Using a bivariate regression, we compared the number of cumulative successes required to become competent for residents in the competent cohort to the outcomes of their first 2 intubation opportunities.
A total of 477 unique intubation opportunities by 105 residents were analyzed (Table 1). The majority of intubation opportunities that included location information occurred in the NICU (97%). The number of opportunities per training year varied in our cohort with a reduction in overall attempts in the PGY-3 year as compared with the PGY-1 or PGY-2 years. The median number of intubation opportunities for any 1 individual over his or her training period was 3, (range, 1–13; interquartile range [IQR], 2–5).
We validated the Bayesian scoring method by comparing the Bayesian likelihood score assigned at any given opportunity to the actual future outcome of the next recorded intubation opportunity, by logistic regression. This revealed that a Bayesian score of 80 reflected a 75% likelihood of successful intubation of the next patient, our agreed upon definition of competency (Fig 1). To visualize the learning curves of individual residents, we plotted individual residents’ Bayesian scores over time with each opportunity (Fig 2). A negative slope represents decreasing Bayesian scores as residents demonstrate failed attempts, whereas a positive slope represents increasing Bayesian scores as successful attempts are achieved over time. Some residents never begin the process of attaining competency (Fig 2, resident A). Others, after mixed early outcomes, need varying exposure to opportunities before beginning to attain competency (Fig 2, residents B and C). Other residents begin the attainment of competency immediately, showing successes starting with their first 2 opportunities (Fig 2, resident D).
Having established 80 as the Bayesian score needed to achieve competency, we determined that 24 of 105 residents (23%) achieved competency during the study period. Of this competent group, 5 of 24 (21%) residents achieved competency during the PGY-1 year, 14 of 24 (58%) during the PGY-2 year, and 5 of 24 (21%) during the PGY-3 year. The competent cohort had a higher median number of intubation opportunities than the entire cohort, 7, ranging from 4 to 14 opportunities (IQR, 6–10). Further analysis of the competent cohort by logistic regression determined that the number of cumulative successful intubations needed to achieve a Bayesian score of 80 was at least 4 (Fig 3). We then grouped competent residents by the outcome of their first 2 intubation opportunities and analyzed their path to achieving 4 cumulative successes (Fig 4). Success during the first 2 intubation opportunities was directly related to the number of opportunities needed to achieve 4 cumulative successes. Residents successful on both of their first 2 opportunities only needed ∼5 opportunities to achieve competency. Residents only successful on 1 of 2 first opportunities needed 7 opportunities. Residents who failed both of their first 2 opportunities needed at least 8 opportunities to achieve the same number of cumulative successes as the other 2 groups.
A need for better understanding of the intubation learning process is of great importance in the current GME training environment, given the development of the Next Accreditation System and a trend toward competency-based programs in GME. The ACGME pediatric program requirements state that pediatric residents must have sufficient training in endotracheal intubation before completion of their training program.19 However, a clear and consistent definition of sufficient training has not been specified. Although previous studies have linked educational theory to procedural tasks and the acquisition of psychomotor skills in medical education,20–22 few have described a precise definition of a competency in procedural skills.
We have shown that Bayesian statistics may be used to define competency in neonatal intubation, and that we may be able to predict likelihood of future intubation success. In our cohort, a Bayesian score of 80 represented a 75% likelihood to intubate the next patient, which we believe to be a reasonable measure of competency for pediatric trainees to achieve before the completion of their training. Most residents begin intubation training with a series of failures or inconsistent successes. Each resident then, at a different point in training, arrives at a categorical learning milestone, after which they begin a steady climb toward competency. The learning curves of intermediate learners, the majority of residents in our cohort, demonstrates a “hockey stick” appearance, with early failures causing a downward slope followed by an upward infection at the categorical learning milestone. Some residents in our study begin the attainment competency after only 1 or 2 opportunities. For others, this upward climb did not begin until well after 6 or 7 opportunities. We hypothesize that the clinical relevance of the observation of this categorical learning moment is in fact the point at which the resident is reliably able to identify the anatomy of the neonatal airway, which is critical for successful intubation. Many residents never achieve this categorical learning event and subsequently fail to become competent. In fact, only 23% of residents in our cohort achieved this definition of competency. Given the nature of this retrospective study, we were not able to ascertain how intrinsic personal characteristics, such as confidence or persistence, may aid residents in this learning process, especially if faced with early failures. The initial feedback that is given to the resident regarding identification of the airway anatomy may be a key factor determining which residents achieve competency more rapidly. Future prospective studies investigating how immediate feedback to the learner at the bedside impacts intubation training would be timely.
In our cohort, at least 4 consecutive successful intubations were needed to become competent. The outcome of a resident’s first 2 intubation opportunities appear to have a major impact on how many total opportunities are then subsequently needed to achieve competency. A group of residents failing on both of their first 2 opportunities needed nearly double the amount of exposure to intubation opportunities compared with the group who was successful on both their first 2 attempts. Unfortunately, the median number of total intubation opportunities for any 1 trainee during residency in our cohort was only 3, though some residents recorded many more. The competent cohort had a much higher number of opportunities than the whole cohort. Therefore, the major barrier to producing competent intubators may be simply lack of procedure opportunity. Although the “real-world” difference between a resident having 5 vs 7 vs 8 opportunities may not be obvious, we argue that for program directors, this relates back to the problem of too few opportunities occurring during the training years. Had we found that our residents were being exposed to many more opportunities during training (such as with anesthesia residents23,24), this small difference in exposure may not be a major one. However, we see pediatric residents as a completely different group of learners, and with most residents having so few opportunities for success, the small difference in the number needed to become competent becomes hugely relevant.
Our study offers evidence that by using outcomes of early opportunities, educators might be able to predict which residents will need supplemental training to achieve competency. As more residents complete postgraduate training programs without adequate procedural skills, demand for procedural experience falls more often to the postgraduate fellow, in turn causing fewer opportunities for residents.25,26 In the future, the goal of educators may not be to increase the total number of intubation opportunities, but rather increase the quality of intubation attempts by helping residents identify the landmarks of the neonatal airway more easily, allowing them the opportunity to move toward procedural competency sooner. Prospective studies may allow researchers to investigate additional factors that impact intubation training, including those intrinsic to the learner such as learner confidence or learner persistence, visual acuity of the learner, supervision by faculty, and infant characteristics such as gestational age, weight, and emergent versus elective intubations. Although some small studies have only shown minimal improvement,27,28 complementary training measures such as video laryngoscopy, high fidelity simulation, video archival of intubations, and subsequent formatted feedback and debriefing may emerge as strategies to enhance the path to resident competency. Video laryngoscopy technology has been shown to shorten intubation procedure time and improve success rates in pediatric patients by inexperienced intubators when compared with standard laryngoscope blades.29 Improving other clinical conditions may be helpful, because a recent study revealed that resident intubation success improved after RSI versus intubations using no premedication.30 In the current study, we did not perform a factor analysis to determine how RSI may affect resident intubation competency, because RSI was not routine practice in our NICU until the 2012 academic year.
Bayesian statistics are useful in this analysis because they are able to capture how previous events may influence future outcomes. In contrast, standard statistical methods would treat each intubation opportunity as an independent event, which is not a true reflection of real-life psychomotor skills training. Although concerns exist regarding the validity of Bayesian prediction models, our model benefits from a degree of internal validity. As we tracked individual residents over time, we were able to validate the Bayesian scoring by comparing the predicted success/failure on the next opportunity to the actual procedure outcome. We believe that the Bayesian score is a better measure of resident competency as opposed to commonly published outcomes such as first pass intubation success rates or overall success rates.
One major drawback of this analysis is that time between opportunities was not analyzed. There is reason to believe that a significant degradation of skill may occur over time, and it has been hypothesized that novice learners are much more sensitive to this degradation than experts. In the future, we plan to investigate how length of time between procedure attempts may affect future success. Another weakness of this study is that the QA database may not capture all intubations in the NICU. We also did not capture information about procedural difficulty or grading of airway view, and therefore cannot comment on how residents may be affected by difficult intubations. However, a previous study that did analyze the effect of difficulty of intubation on resident success revealed overall poor success rates whether the intubation was graded difficult or not.4 Lastly, at our center, residents have some exposure to neonatal intubations in other areas of training including emergency medicine or pediatric intensive care and our data may not represent the full extent of intubation experience throughout a pediatric residency program.
In a GME environment moving toward competency-based training, few objective measures of procedural competence have been described. Bayesian statistics may be useful to more precisely describe neonatal intubation competency in pediatric residents. Achieving competency in neonatal intubation appears to be dependent on a categorical or milestone learning event whose timing varies between residents. Four cumulative successes were needed to achieve competency in our cohort. The outcome of a resident’s first 2 intubation opportunities may have a major impact on how many intubation opportunities are ultimately needed to achieve competency.
- Accepted January 29, 2015.
- Address correspondence to Stephen D. DeMeo, DO, Division of Neonatology, Duke University Medical Center, 2424 Erwin Rd, Suite 504, Durham, NC 27715. E-mail:
Dr DeMeo conceptualized and designed the study and drafted the initial manuscript; Dr Tanaka conceptualized and designed the study, performed the statistical analyses, and reviewed and revised the manuscript; Drs Katakam and Goldberg reviewed and revised the manuscript; and all authors approved the final manuscript for submission.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by a grant from the Duke University Graduate Medical Education Innovation Fund.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
COMPANION PAPER: A companion to this article can be found on page e1290, online at www.pediatrics.org/cgi/doi/10.1542/peds.2015-0571.
- Sanders RC Jr,
- Giuliano JS Jr,
- Sullivan JE,
- et al.,
- National Emergency Airway Registry for Children Investigators and Pediatric Acute Lung Injury and Sepsis Investigators Network
- Finer NN,
- Carlo WA,
- Duara S,
- et al.,
- National Institute of Child Health and Human Development Neonatal Research Network
- Lindner W,
- Vobeck S,
- Hummler H,
- Pohlandt F
- Smith SL,
- Hall MA
- Schumacher DJ,
- Spector ND,
- Calaman S,
- et al
- Falck AJEM,
- Escobedo MB,
- Baillargeon JG,
- Villard LG,
- Gunkel JH
- ↵ACGME Program Requirements for Graduate Medical Education in Pediatrics. https://www.acgme.org/acgmeweb/Portals/0/PFAssets/2013-PR-FAQ-PIF/320_pediatrics_07012013.pdf. Accessed February 9, 2014
- Gaies MG,
- Landrigan CP,
- Hafler JP,
- Sandora TJ
- Halamek LP,
- Kaegi DM
- Fonte M,
- Oulego-Erroz I,
- Nadkarni L,
- Sanchez-Santos L,
- Iglesias-Vasquez A,
- Rodriguez-Nunez A.
- Copyright © 2015 by the American Academy of Pediatrics