Interns' Success With Clinical Procedures in Infants After Simulation Training
BACKGROUND AND OBJECTIVE: Simulation-based medical education (SBME) is used to teach residents. However, few studies have evaluated its clinical impact. The goal of this study was to evaluate the impact of an SBME session on pediatric interns’ clinical procedural success.
METHODS: This randomized trial was conducted at 10 academic medical centers. Interns were surveyed on infant lumbar puncture (ILP) and child intravenous line placement (CIV) knowledge and watched audiovisual expert modeling of both procedures. Participants were randomized to SBME mastery learning for ILP or CIV and for 6 succeeding months reported clinical performance for both procedures. ILP success was defined as obtaining a sample on the first attempt with <1000 red blood cells per high-power field or fluid described as clear. CIV success was defined as placement of a functioning catheter on the first try. Each group served as the control group for the procedure for which they did not receive the intervention.
RESULTS: Two-hundred interns participated (104 in the ILP group and 96 in the CIV group). Together, they reported 409 procedures. ILP success rates were 34% (31 of 91) for interns who received ILP mastery learning and 34% (25 of 73) for controls (difference: 0.2% [95% confidence interval: –0.1 to 0.1]). The CIV success rate was 54% (62 of 115) for interns who received CIV mastery learning compared with 50% (58 of 115) for controls (difference: 3% [95% confidence interval: –10 to 17]).
CONCLUSIONS: Participation in a single SBME mastery learning session was insufficient to affect pediatric interns’ subsequent procedural success.
- clinical skills
- clinical competence/standards
- competency-based education/methods
- educational measurement/methods
- education/medical/graduate methods
- internship and residency/methods
- practice (psychology)
- prospective studies
- outcome assessment (health care)
- patient simulation
- randomized controlled trial
- spinal puncture
- CIV —
- child intravenous line
- CSF —
- cerebrospinal fluid
- ILP —
- infant lumbar puncture
- IV —
- intravenous line
- LP —
- lumbar puncture
- SBME —
- simulation-based medical education
What’s Known on This Subject:
Pediatric training programs use simulation for procedural skills training. Research demonstrates student satisfaction with simulation training, improved confidence, and improved skills when retested on a simulator. Few studies, however, have investigated the clinical impact of simulation education.
What This Study Adds:
This is the first multicenter, randomized trial to evaluate the impact of simulation-based mastery learning on clinical procedural performance in pediatrics. A single simulation-based training session was not sufficient to improve interns’ clinical procedural performance.
The Accreditation Council for Graduate Medical Education mandates pediatric residency programs to provide “sufficient training” for providers to develop competency in 16 procedures. Skills training has traditionally involved the apprenticeship model of “see one, do one, teach one.”1 There is concern that this paradigm of skills training is inefficient and not safe for patients.2 Furthermore, patients and parents are uncomfortable with trainees learning while performing procedures.3 However, patients report they would be more likely to allow trainees to perform a procedure after demonstrating mastery on a simulator.4 New medical school graduates have minimal exposure to and lack confidence in common procedures such as lumbar puncture (LP).5–14 In addition, training programs struggle to provide sufficient opportunities for procedural training in the setting of restricted work hours, increased attending oversight, and greater midlevel providers competing for experience.10–12
A growing number of training programs are using simulation-based medical education (SBME) to meet the challenges of procedural training. SBME allows for controlled training experiences and standardized learning outcomes. Experiences traditionally left to chance, such as when and where a procedure will occur, are standardized. Teaching is customized to individual learner needs.15–17 SBME provides opportunities for novice providers to practice procedural skills and make errors without harming patients. Research has demonstrated that SBME improves clinical skills and patient outcomes associated with procedures in adults,18 including laparoscopic surgery,19 bronchoscopy,20 emergency airway management,21 advanced cardiac life support,22–24 thoracentesis,25 and central venous catheter insertion.26,27 However, studies demonstrating improved clinical impact on pediatric patients are limited to obstetrical literature showing improved neonatal outcomes for shoulder dystocia after an SBME intervention.28,29
One of the components of a successful SBME intervention is incorporation of a sound educational theory. The majority of SBME studies with positive outcomes apply mastery learning, a form of competency-based education.30 Mastery learning starts with a baseline skill assessment by using a checklist to identify the trainee’s individual learning needs. Next, the learner participates in repetitive cycles of deliberate practice until he or she achieves a predefined passing score on the checklist. An ideal deliberate practice session consists of a highly motivated learner pursuing a well-defined learning objective via focused repetitive practice while receiving informative feedback from an expert.31,32 The duration of a learning session will vary based on the time it takes each learner to achieve the predefined outcome.33
In a recent study,34 interns randomized to SBME procedural training (LP, intravenous line [IV], and bag valve mask) demonstrated higher skill levels than interns randomized to conventional procedural training when retested on a simulator. However, participation in SBME was not associated with improved procedural success rates on patients. In contrast, our previous research at a single center demonstrated that residents randomized to SBME mastery learning for the infant lumbar puncture (ILP) procedure were more likely to be successful with their next patient ILP (71%) than the control group (27%).35 The current study aims to evaluate the impact of a single SBME mastery learning session on pediatric interns’ ILP and child intravenous line insertion (CIV) performance on patients across a network of pediatric training programs. We hypothesized that a single SBME mastery learning intervention improves clinical procedural success rates on patients.
Study Design and Setting
This was a randomized controlled trial of an SBME mastery learning intervention conducted in 10 tertiary care urban academic medical centers at the start of the 2009–2010 academic year. These institutions are members of the Patient Outcomes in Simulation Education Network, a pediatric SBME research network consisting of 10 institutions during the study period (currently known as INSPIRE).36 Investigators conducted a 4-month study-planning phase involving biweekly conferences. The network supported the purchase of simulator equipment, video development, and centralized data collection. Institutional review boards at each center approved this study, and participants provided the requisite consent before enrollment at each site.
We enrolled incoming postgraduate year 1 trainees from pediatric categorical or combined training programs.
Using a block randomization scheme, participants were randomized within each site to either the ILP mastery learning or CIV mastery learning group. Each group served both as an experimental group for the mastery learning intervention that it received as well as a control group for the procedural training that it did not receive. To keep outcome assessors blinded, each participant was assigned an anonymous study identification number for all data collection. Next, participants viewed a 40-minute set of LP and IV videos including both procedure videos published by New England Journal of Medicine and content developed by the study authors.37 The video content included indications, contraindications, complications, necessary equipment, and key steps (positioning, landmarks, preparation, early stylet removal technique, and use of local anesthesia and analgesia). Additional video content developed by the study authors covered pediatric-specific elements of these procedures along with expert modeling of both procedures on simulators.38
The SBME mastery learning session occurred at the start of internship and used bench-top simulators and trained facilitators who guided participants in deliberate practice until they achieved a predefined level of mastery of the skill being taught. All facilitators were clinician educators who participated in a standardized 30-minute train-the-trainer session before the study. The ILP group completed their mastery learning session by using a Laerdal Baby Stap ILP simulator (Laerdal Medical, Stockholm, Sweden). The CIV group completed their session by using a Laerdal Multivenous IV training arm simulator. Mastery was defined as achievement of a predetermined level of performance on a subcomponent skills checklist that was developed through a modified Delphi process among the network site directors (Appendix). The training sessions continued until learners demonstrated independent mastery performance on the checklist (ie, sessions ranged from 20–60 minutes depending on the learner). At the study’s conclusion 6 months later, each group received mastery learning for the alternate procedure to close the educational gap.
At the start of internship, participants completed an online, self-administered, 28-item questionnaire that collected baseline information on knowledge, attitudes, and experience with the ILP and CIV procedures. Knowledge was assessed by using multiple choice questions developed and pilot tested on nonstudy senior residents, fellows, and faculty. Attitudes were assessed on a 4-point Likert scale of confidence. ILP and CIV procedural experience was assessed based on numbers of procedures observed and performed.
Primary Outcome Measures and Potential Confounders
Throughout the 6-month study period, participants completed an online self-report questionnaire for all ILP procedures on patients aged <365 days and all CIV placements on patients 1 to 18 years of age performed as part of their patient care duties. Participants were sent monthly e-mail reminders with links to the online procedure logs. Site directors used varied methods, including checking computer medical records, reviewing local procedure logs, and holding conversations with staff, to provide feedback and emphasize the importance of reporting the procedures. Data collected for ILP included fluid acquisition, cell count, fluid appearance (bloody, blood-tinged, purulent, or clear), number of attempts, patient age, holder, clinical environment where the procedure was performed, use of local anesthetic, use of early stylet technique, and presence of supervisor. ILP success was defined as obtaining an adequate sample on the first attempt that had <1000 red blood cells per high-power field on microscopic examination or that was described as clear. Data were excluded from analysis if another provider had attempted the procedure before the subject’s attempt (thereby increasing the risk for a traumatic specimen). Data collected for CIV included success, gauge of needle, number of attempts, age of patient, clinical environment, and presence of supervisor. CIV success was defined as placement of a functioning intravenous catheter on the first attempt. Physicians supervising the interns during the procedures were unaware of the participants’ study group, as were the patients and parents.
Assessing for Reporting and Other Bias
We assessed reporting behaviors in a follow-up questionnaire at 6 months that assessed reporting behaviors and surveyed participants for any additional procedural training received during the study period.
Participant characteristics were analyzed by using descriptive statistics, and clinical outcomes were compared by using the Fisher exact test. Knowledge and confidence levels were compared between groups by using the Mann–Whitney U test and χ2 analysis. Other secondary variables were analyzed by using either χ2 tests for proportions or Mann–Whitney U tests for continuous variables. We powered the study to find an improvement from 60% success at obtaining cerebrospinal fluid (CSF) from the patient to 80%, estimating a sample size of 73 residents in each group to have 80% power to detect this difference at an α level of 5% (G*Power for Mac, version 3.0.10).39 Sample sizes were based on ILP frequency, expected to be rarer than IV insertions. Target enrollment was increased by 20% to accommodate interns who did not have an opportunity to do an ILP within the 6-month study period. All analyses were conducted by using SPSS version 19 (IBM SPSS Statistics, IBM Corporation, Armonk, NY).
A total of 210 interns were randomized to either the ILP or CIV group, and 200 interns completed the study (Fig 1). Baseline knowledge, attitudes, and experience collected on the initial survey were similar in both groups (Table 1).
A total of 203 participants achieved mastery during the session according to a predefined set of criteria (Appendix). Across study groups, clinical data were reported for 178 LPs performed by 89 participants and 230 IVs performed by 65 participants from all sites.
Participants who reported procedures were similar in terms of experience, knowledge, and attitudes to those who did not report procedures throughout the year. Groups who performed an ILP or CIV were similar with regard to hospital location, number of attempts, and whether another provider had made previous attempts in the preceding 24 hours, in addition to other characteristics (Table 2).
Primary Outcome Measures: Clinical Outcomes
In the ILP mastery learning group, 31 (34%) of 91 LP attempts were successful on the first try. In the CIV mastery learning group, 25 (34%) of 73 LP attempts were successful (difference: 0.2% [95% confidence interval: –0.1 to 0.1]). Due to another provider’s previous LP attempt, 10 reports from the ILP group and 4 from the CIV group were excluded. There were no differences between groups for rates of CSF acquisition, success after multiple attempts, or success with the first ILP performed after training (Table 3). There were also no statistical differences in success between groups on an individual hospital basis.
In the ILP mastery learning group, 58 (50%) of 115 IV attempts were successful on the first try compared with 62 (54%) of 115 attempts in the CIV mastery learning group. There were no differences between groups for rates of success after multiple attempts or success with the first IV performed on a child after the SBME training. There were also no statistical differences in success between groups on an individual hospital basis.
Sixteen subjects who did not complete demographic data were included in this intention-to- treat analysis for both outcome measures. A reanalysis with these subjects removed also did not result in any statistical differences between groups for either outcome.
Median elapsed days from training to reporting of LPs in the ILP mastery learning group were less (70 days compared with 90 days in the control group, P = .001 [Mann–Whitney U test]), whereas median elapsed days from training to reporting of IVs in the ILP mastery learning group were longer (91.5 days compared with 64 days in the CIV mastery learning group, P = .03 [Mann–Whitney U test]) (Fig 2).
A total of 137 participants responded to the 6-month follow-up questionnaire (69% response rate). Underreporting of procedures was not statistically different between groups. Self-reported experience and reporting rates are presented in Table 4. There were no statistically significant differences in additional procedural training obtained between the 2 groups.
We investigated whether a single SBME mastery learning session for a procedure could result in measurably higher success rates with the procedure on patients. Although we did not detect differences in procedural success rates in participants completing an SBME session, our work demonstrates the feasibility of testing the impact of SBME on clinical outcomes in a multi-institutional setting.
These results differ from our previous single-center study of ILP success after SBME (71% for intervention and 27% for controls).35 That cohort included residents from all 3 years and only 3 trainers, whereas our current study included only interns and 50 trainers. To reduce variability in the SBME intervention, we designed a train-the-trainer module that was viewed by all trainers; however, variability in learner–trainer interactions or other institutional practices may have led to differences in the intervention’s effectiveness.
In this study, we report success rates for ILP (34% for intervention and 34% for controls) that are lower than the descriptive reports of novice success rates in the literature (45%–63%).34,40 These differences may be due to the populations studied, variations in data acquisition, or the definitions of success. Our decision to define success as a single, atraumatic attempt was based on the observation that the CSF is more likely to be bloody after multiple attempts.41 This patient-centered definition was chosen because success on the first attempt would cause the least harm to patients. Other than our aforementioned single-center study,35 no research has demonstrated an SBME intervention leading to improved clinical success with the ILP procedure. A randomized trial of an ILP SBME intervention for pediatric residents at the start of their emergency medicine rotation did not find any significant difference in clinical success (70% for participants and 62% for controls).34 The high success rates reported in that study are likely due to the more liberal definition of success, comprising any CSF sample that was suitable for culture. A more recent single-center study found that 62% (13 of 21) of interns were successful on their first ILP attempt after an SBME intervention; however, conclusions were limited due to a lack of comparison group.42
The success rates reported in this study for CIV (62% for intervention and 58% for control) are similar to success rates in the literature. A previous SBME study of 53 interns reported no improvement in CIV success (66% for intervention and 50% for controls).34
Participants in both groups reported few procedures, even when accounting for underreporting. This shortage of clinical opportunities and the time-lapse between mastery learning and clinical performance may have contributed to the lack of improvement in clinical success rates. Studies on skill retention for other procedures, such as neonatal intubation, have demonstrated a significant deterioration of skills within months after simulator training.43–46 Current research into skills training suggests that distributed practice and just-in-time skills refreshers may be a better strategy to confront skill deterioration over time.47,48
Participants who received SBME for a skill were more likely to report that procedure earlier in the year compared with the control group. This finding may have been due to increased confidence of participants or their supervisors. Educational interventions often improve confidence more rapidly than skills.49,50 In our study, premature performance of the procedure may have occurred before an intern was truly competent due to increased confidence of participant or supervisor. Another possibility is that participants lost interest in reporting once they became proficient in a procedure. Both scenarios could have falsely deflated the success rates of the intervention group. However, the fact that first-time success rates did not differ between groups speaks against this variable being a major confounder.
An important limitation of our study was the reliance on self-report to document the primary outcome. We chose self-report in part due to limited feasibility of alternative data collection methods that could reliably capture all events and be standardized across multiple centers. More robust methods such as direct field observation were considered less feasible based on the heavy resources and the logistics of around-the-clock coverage that would have limited site involvement and significantly reduced our power to detect a clinical difference. Despite the propensity of underreporting seen with self-reported outcomes, we still had a sufficient number of reports to have detected a clinically significant difference had it existed. Another potential problem with self-report is recall bias in which reliance on memory leads to systematic errors (eg, reporting only successes); however, the success rate we observed is plausible and had sufficient variability to have allowed us to detect differences between groups. We cannot exclude the possibility that recall interacts with the group assignment. Future studies should consider verifying self-report data through chart review or supervisor-reporting forms.
Participants in both groups reported low levels of experience with ILP and CIV procedures before starting internship, confirming that current medical school experiences are not sufficient to prepare incoming interns to perform these procedures.14 However, we were also missing baseline data from 16 participants, thus limiting our ability to draw conclusions in this regard. The use of multiple trainers is an inherent limitation to any multicenter educational study that we sought to mitigate through the use of standardized training protocols and faculty development. Although this inevitably increases variability (despite best efforts for standardization), it also improves the generalizability of our findings.
This study demonstrated the feasibility of disseminating a standardized SBME intervention at multiple institutions. However, participation in a single SBME mastery learning session did not seem to impact pediatric interns’ clinical success rates with ILP or CIV. Despite training, interns were not adequately prepared for either procedure in the clinical setting as measured by low success rates. Although mastery learning may be a necessary component of skills education, it was not sufficient to affect clinical success rates. Future studies should focus on providing more time for deliberate practice and distributed training to deal with skill development and skill retention, respectively.
The authors thank Jillian Olsen for her help in editing and preparing the manuscript for publication; Karl Santiago, who provided technical assistance with organization of several orientation trainings and manikin distribution; and Laura Santry, for help with editing the figures. The authors also acknowledge the many faculty educators and interns who participated in making this educational intervention possible.
- Accepted November 19, 2012.
- Address correspondence to David O. Kessler, MD, MSc, RDMS, Columbia University Medical Center/New York Presbyterian Morgan Stanley Children's Hospital of New York, 622 West 168th St, PH-137, New York, NY 10032. E-mail:
Drs Kessler, Arteaga, Ching, Haubner, and Kamdar, Ms Krantz, Drs Lindower, Miller, Petrescu, Pusic, Rocker, Shah, Tilt, Weinberg, Chang, Fein, and Auerbach contributed substantially to the conception and design of this study; Drs Kessler, Arteaga, Ching, Haubner, Kamdar, Lindower, Miller, Petrescu, Pusic, Rocker, Shah, Strother, Tilt, Weinberg, Chang, and Auerbach contributed to the data acquisition and enrollment of study subjects; and Drs Kessler, Auerbach, and Pusic contributed to the analysis and interpretation of the data. All authors contributed to the drafting, editing, and preparation of the manuscript, and all authors approved of the final version of the manuscript and are responsible for the reported research.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This work was supported in part by the R Baby Foundation.
- Rodriguez-Paz JM,
- Kennedy M,
- Salas E,
- et al
- ↵Graber MA, Wyatt C, Kasparek L, Xu Y. Does simulator training for medical students change patient opinions and attitudes toward medical student procedures in the emergency department? Acad Emerg Med 2005;12(7):635–639
- Huang GC, Smith CC, Gordon CE, et al. Beyond the comfort zone: residents assess their comfort performing inpatient medical procedures. Am J Med. 2006;119(1):71.e17–71.e24
- Gaies MG,
- Landrigan CP,
- Hafler JP,
- Sandora TJ
- Kamdar G,
- Kessler D,
- Tilt L,
- et al
- ↵Auerbach MA, Chang T, Krantz A, Pratt A, Gerard J, Quinones C, Pusic M, Kessler D. Are pediatric interns prepared to perform infant lumbar punctures? A multi-institutional descriptive study. Pediatric Emergency Care. [accepted 2012, publication pending]
- ↵Andreatta PB, Woodrum DT, Birkmeyer JD, et al. Laparoscopic skills are improved with Lap Mentor™ training: results of a randomized, double-blinded study. Ann Surg. 2006;243:854–863
- ↵McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86(6):706–711
- ↵Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(suppl 10):S70–S81
- ↵Ericsson KA. Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med. 2008;15(11):988–994
- Airasian PWBB,
- Carroll JB
- Gaies MG,
- Morris SA,
- Hafler JP,
- et al
- ↵Kessler DO, Auerbach M, Pusic M, Tunik MG, Foltin JC. A randomized trial of simulation-based deliberate practice for infant lumbar puncture skills. Simul Healthc. 2011;6(4):197–203
- ↵International Network for Simulation-Based Pediatric Innovation, Research and Education (INSPIRE). Available at: www.InspireSim.com. Accessed November 1, 2012
- ↵Auerbach M, Chang T, Krantz A, et al. Infant lumbar puncture: POISE pediatric procedure video. Available at: https://www.mededportal.org/publication/8339. Accessed January 14, 2013
- ↵Faul, F., Erdfelder, E., Lang, A.-G. & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191
- Glatstein MM,
- Zucker-Toledano M,
- Arik A,
- Scolnik D,
- Oren A,
- Reif S
- Patel J,
- Posencheg M,
- Ades A
- ↵Kahn CE Jr, Ehlers KC, Wood BP. Radiologists' preferences for just-in-time learning. J Digit Imaging. 2006;19(3):202–206
- Stoler GB,
- Johnston JR,
- Stevenson JA,
- Suyama J
- Garrood T,
- Iyer A,
- Gray K,
- et al
- Copyright © 2013 by the American Academy of Pediatrics