OBJECTIVE. The objective of this study was to assess the opinions of pediatric program directors regarding procedural skills training of pediatric residents.
METHODS. We developed a survey based on the Residency Review Committee's guidelines for procedural training. It included items about the importance of 29 procedures encountered in pediatric training, estimates of residents' competence in performing them, and the teaching of procedural skills. The survey was sent to members of the Association of Pediatric Program Directors. The primary outcome was the perceived importance for residents to achieve competence in these procedures, rated on a 10-point Likert scale. Secondary outcomes included perception of resident competence to perform procedures and educational methods used by respondents for teaching procedural skills. Associations between demographic characteristics and perceived importance or competence were also assessed.
RESULTS. Surveys were sent to 139 programs, and 112 responded. Thirteen procedures were rated 8 or higher by >75% of program directors. Seven skills that were prioritized by the Residency Review Committee did not achieve this level of consensus. Respondents reported that many residents failed to achieve competence by the end of training in 9 of 13 procedures that they rated as very important, including venipuncture, neonatal intubation, and administering injections. Residents who perform the majority of venipunctures and intravenous catheter placements at their institutions were more likely to be judged competent in performing these skills than residents who do not.
CONCLUSIONS. The Residency Review Committee's list of procedures does not necessarily reflect the opinions of pediatric program directors on the most essential skills for trainees. Many residents may not develop competence in several important procedures by the end of residency, most notably vascular access and life-saving skills. A more robust and standardized method is needed for teaching procedural skills and for documenting competence.
The mandate for competency-based postgraduate medical training requires all residency programs to teach clinical skills formally.1,2 The Pediatric Residency Review Committee (RRC) is responsible for creating and modifying the general program requirements for pediatric residencies, including the recommended list of procedural skills. A review of these requirements occurs every 5 years. The RRC is mandated to inform program directors and program director associations when the contents of program requirements are created or modified. Program directors are then given a 45-day period to comment, either individually or through a program director association, on these proposed changes. The RRC can choose to accept or reject suggestions from program directors or program director associations, but there is no requirement to distribute these suggestions to the group of program directors in its entirety.3 As a result of this process, the RRC currently specifies that all pediatric residents should have “sufficient training” in 16 procedures (Table 1) and that they should be monitored by the residency program for appropriate skill development.3 However, previous studies have demonstrated that although many residents feel confident in their abilities to perform these procedures, they do not perform all skills competently when they are formally tested.4–6
There is a paucity of data describing pediatric residents' expertise in performing procedural skills. This knowledge gap presents a substantive challenge to those who intend to develop curricula and evaluation systems to teach and assess these skills. Although there are some data on internal medicine program directors' opinions about which procedure skills are essential,7 no recent studies have surveyed pediatric program directors or pediatric residents about which procedures they deem essential to pediatric training. More than 15 years ago, authors from the American Board of Pediatrics published a study that reported data from a survey of accredited pediatric training programs on the importance of developing technical skills in pediatric residency.8 However, the method of that survey was never published separately, and the opinions of respondents were only briefly described. Moreover, it is unclear whether these findings remain relevant in today's substantially changed pediatric training environment. Therefore, we conducted a study to assess the perceived importance of training pediatric residents in various procedures and to document the educational methods that are used to teach these skills in pediatric residency programs. We also sought to estimate the competence of residents in performing various procedures at the completion of residency. Our hypotheses were that (1) wide variation would exist in program directors' assessments of the importance of various procedures on the RRC's list and (2) program directors would report that graduating residents have a high level of competence to perform most procedures.
To determine pediatric program directors' opinions about the importance of various procedural skills, we conducted a survey of all registered members of the Association of Pediatric Program Directors (APPD). Our survey design was modeled on a previous survey study by Wigton et al,7 in which internal medicine program directors were asked to rate the importance of procedures for resident training. To develop a comprehensive list of procedures of high priority for pediatric residency training programs, we convened focus groups of Children's Hospital Boston faculty to define a list of procedures that merit inclusion in the final survey tool. These groups were made up of generalists, subspecialists, proceduralists, and nonproceduralists and included senior clinical fellows and staff. The RRC's list of recommended procedures3 was used as a starting point for the list. In collaboration with the participating physicians, it was determined that procedural skills that involved physical contact with patients would be the focus of this study; procedures such as developmental screening and pain management, although considered essential to pediatric training, were excluded from the final survey instrument. The focus groups also identified procedures that were not on the RRC's list but were believed to be important enough to include in the survey, and component skills of basic and advanced life support were specifically delineated and included. In addition, we included a few procedures on the survey that the groups agreed were not important to pediatric training, to be sure that our survey could identify both positive and negative opinions about individual procedures. After initial survey development, former chief residents from 3 different institutions were asked to pilot-test the survey; cognitive interviews were performed to assess item comprehension and to refine further the list of procedures included in the survey. Finally, in collaboration with the APPD, a Web-based version of the survey instrument was created.
The first part of the survey asked for demographic information on each respondent and program, including years since completing training, specialty, program size, number and types of fellowship programs associated with the institution, and various other identifiers. For the purposes of assessing the impact of specialty training on responses, we considered the following 4 specialties a priori to be “proceduralist” fields: cardiology, critical care, emergency medicine, and neonatology. For each of the 29 procedures included in the final survey, respondents were asked the following questions: “How important is it for pediatric residents to be able to perform the procedure competently by the end of their training?” and, “In your estimation, what percentage of residents in your program is competent to perform the procedure by the end of their residency?” For the question asking the respondents to rate the importance of the procedure, participants responded using a 10-point Likert scale, with 1 representing “not important at all” and 10 representing “extremely important.” We decided a priori to identify procedures that were rated ≥8 by >75% of program directors, because procedures that achieve this degree of consensus at that level of importance might reasonably be considered essential to pediatric training. A similar definition of importance has been used in previous studies.7,8
Regarding competence, respondents were asked to estimate the proportion of residents in their programs who were competent to perform the procedure by the end of their residency: “all or almost all,” “more than half,” “approximately half,” “less than half,” or “few or none.” Competence was defined for the participants as “the ability to perform a procedure independently, without supervision, and with a high likelihood of successful completion. This includes an understanding of the indications, contraindications, and risks of the procedure.” It is reasonable to expect that by the end of residency, all or almost all residents should be competent to perform procedures that are considered essential to pediatric training. Therefore, we sought to identify procedures for which fewer than two thirds of program directors thought “all or almost all” of their residents were competent, believing that such procedures may not be mastered by a significant portion of graduating residents. A final item asked about current educational practices for teaching and documenting procedural skills on a subset of 10 procedures identified by our focus groups as the most critical to resident training.
An e-mail that contained a link to the online survey was sent to the registered program directors and assistant program directors of all 139 active, dues-paying member programs of the APPD in September 2005. The first responding program director or assistant program director from each program was enrolled in the study. Subsequent responses from other members of the same program were not included to ensure that each participating program was equally represented in the study. Programs that did not respond initially received a total of 3 reminder e-mails at 2-week intervals.
Descriptive analyses of survey responses were performed to characterize demographics of respondents, the opinions about the importance of each procedure to residency training, estimates of the current level of competence of pediatric residents at the respondents' institutions, and the current teaching strategies used at pediatric training programs. χ2 and Fisher's exact tests were used as appropriate to assess whether perceived importance or estimated competence for a procedure was associated with respondent or program characteristics. Statistical analyses were performed using SAS 9.0 (SAS Institute, Cary, NC). Two-sided P < .05 indicated statistical significance.
We received at least 1 response from 112 (81%) of the 139 programs surveyed. Demographic data are presented in Table 2. Because the data from nonresponding programs were not known, no comparisons could be made between respondents and nonrespondents. The majority of our respondents represented training programs at freestanding children's hospitals, and most stated that ≥20% of their residents eventually pursue fellowship training.
Importance of Procedures to Training
Program directors' ratings of importance for the individual procedures are depicted in Fig 1. Notably, for several procedures currently included on the RRC's recommended list, <75% of respondents gave a rating of ≥8, including administration of injections (74%), intraosseous line placement (65%), nonneonatal intubation (65%), arterial puncture (63%), defibrillation (60%), umbilical arterial and venous catheter placement (53% and 62%, respectively), and management of dislocations and fractures (48%). Program size, program setting, respondents' specialty, and years since completion of training were not associated with the ratings of importance for any individual or group of procedures.
Program directors' estimates of graduating residents' competence to perform these same procedures are depicted in Fig 2. For 14 procedures currently recommended by the RRC, fewer than two thirds of program directors thought that all or almost all of their residents were competent to perform those skills at the completion of training. These procedures included Pediatric Advanced Life Support (PALS) and Neonatal Resuscitation Program resuscitation leadership (53% and 55%, respectively), neonatal and nonneonatal intubation (60% and 29%, respectively), defibrillation (23%), intraosseous line placement (23%), peripheral intravenous catheter placement (48%), arterial puncture (51%), venipuncture (61%), umbilical arterial and venous catheter placement (55% and 57%, respectively), bladder catheterization (65%), laceration repair (64%), and management of dislocations and fractures (22%). Furthermore, fewer than two thirds of program directors thought that all or almost all of their residents were competent at the end of their training to perform 9 procedures that program directors agreed were very important on the basis of their ratings.
Most of the respondent and program characteristics that we measured were not associated with perceived competence. Program size, program setting, percentage of residents entering fellowship, and number of fellowship programs associated with the institution did not predict response. However, program directors who identified residents as the primary performers of intravenous catheter placements and venipunctures at their institution were more likely to report that their residents are competent to perform intravenous catheter placement and venipuncture than respondents who reported that other practitioners (eg, nurses) usually perform these procedures (P < .0001).
Respondents were asked about the current educational practices being used at their program for 10 of the 29 procedures included in the survey. Figure 3 displays the percentage of program directors who responded that they had any formal method for teaching these skills and the percentage who responded that they used some form of simulation to teach these skills.
We assessed whether a report of using any method to teach the procedure formally was associated with program directors' perceived competence of their residents to perform that procedure. Use of any formal teaching method was associated with a response that all or almost all residents were competent to perform the skill for the following procedures: bag-mask ventilation (P < .0001), venipuncture (P = .03), peripheral intravenous catheter placement (P = .011), neonatal intubation (P = .03), and bladder catheterization (P < .0001). The use of simulation to teach these procedures was significantly associated with perceived competence only to perform bag-mask ventilation (P = .006). For all other procedures, perception of competence was not significantly different between respondents who used simulation versus other or no formal methods. More than 70% of respondents reported that they had some formal method for documenting their residents' competence to perform 7 of these 10 skills, excluding bladder catheterization (61%), intraosseous line placement (60%), and defibrillation (44%).
We found that wide variation exists in pediatric program directors' ratings of the importance of achieving competence in several procedures recommended by the RRC, including intraosseous line placement, arterial puncture, umbilical catheter placement, and administration of injections. Program directors also disagreed with the RRC on the importance of procedures that are part of PALS training and included in the broader rubric of resuscitation skills (eg, defibrillation, endotracheal intubation). For each of these procedures, <75% of program directors rated the procedure ≥8. Only 1 procedure that was included in our survey and was not on the RRC's list—fluorescein eye examination—was deemed very important for training by >75% of program directors.
When asked to estimate the competence of graduating residents to perform each of the procedures in the survey, fewer than two thirds of program directors thought that all or almost all of their residents were competent to perform several of the procedures at the end of residency training. This perception was especially true for procedures that would be classified as a life-saving or vascular access skill. Furthermore, several procedures were classified as very important by program directors and may not be performed competently by a significant number of graduating residents, based on program directors' estimates. These findings are consistent with the results from the study of internal medicine program directors by Wigton et al7 and from studies that directly assessed pediatric residents' ability to perform resuscitations5,6 and neonatal endotracheal intubation.4 To our knowledge, this is the first collective assessment of the procedural skills of pediatric trainees in the United States.
It is notable to compare the results of our survey with those included in a study by Oliver et al8 in 1991. In a survey conducted in 1986, members of accredited pediatric training programs were given a list of 101 technical skills (including procedural, laboratory, and physical examination skills) and asked which skills residents should be able to perform. Seventy-two of those procedures were considered “essential” by >80% of respondents. Several procedures that were deemed essential in that study received much lower ratings in our study, including arterial puncture, bladder aspiration, administration of injections, arterial catheter placement, cardioversion/defibrillation, umbilical artery and vein catheterization, and chest tube placement/thoracentesis.
There are many reasons that program directors may be de-emphasizing the teaching and learning of procedural skills during pediatric residency and that residents may be failing to achieve competence in these skills. Foremost among these reasons is the increasing skill of nonphysician clinicians and the development of ancillary services, such as phlebotomy and intravenous access teams, to perform procedures that traditionally have been done by residents. This possibility is supported by the fact that only 12% and 13% of respondents identified residents as the people who perform the majority of venipunctures and intravenous catheter placements, respectively, in the hospitals in which they rotate. Other procedures, such as administration of injections, bladder catheterization, and arterial puncture, are frequently performed by nurses, which undoubtedly decreases the motivation and opportunities for busy residents to learn these skills amid the other demands on their time.
When these factors are combined with the increasing amount of time spent on paperwork and documentation by residents, it is not surprising that many trainees will have an insufficient number of opportunities to learn and practice a procedure to achieve competence. We did find that the designation of residents as the clinician performing most venipunctures and intravenous catheter placements predicted in a statistically significant way the perceived competence to perform those skills. This finding suggests that when residents are forced to perform a skill, they may be more likely to master it.
The failure to develop procedural skills becomes self-perpetuating. As more residents graduate from pediatric training programs without these skills, the importance of having opportunities to perform these procedures during fellowship training increases. It is not uncommon at Children's Hospital Boston for fellows to be performing many procedures that traditionally were learned and mastered during residency training. This shift in practice results in fewer opportunities for residents to experience these procedures during training, thereby potentially continuing the cycle by further reducing competence. It is interesting that we did not find any statistically significant associations between the presence of fellowship programs at an institution and the rating of importance or perceived competence by respondents; neither did we find that the percentage of residents who entered fellowship training predicted perceived competence.
It is possible that program directors see procedural training as the responsibility of fellowship programs, where trainees will learn the skills necessary for practice in their field. Such a system may place an undue burden on fellowship programs to ensure adequate opportunities for their trainees to achieve competence in certain procedures and to provide remediation for skills that more appropriately are learned during residency training. Furthermore, if residents are not getting opportunities to learn and practice procedures, then fields such as critical care medicine and cardiology, which typically attract trainees who enjoy procedures, may have a more difficult time recruiting residents.
Another factor that may affect program directors' responses about the importance of teaching certain procedures is the perception that most residents will not perform these procedures after their training has been completed. More research is needed to determine exactly which procedures pediatricians (generalists and specialists alike) are performing in their everyday practice. Studies designed to answer this question were published >15 years ago.8,9 Newer data would help to direct future curricula and would allow training programs to focus on procedures that residents need to perform to provide adequate care during training and on those that most trainees will perform after they graduate.
Finally, it is possible that current educational methods that are used by programs for teaching procedures are not sufficient to ensure a high level of competence among graduating residents, taking into account the factors previously noted. Simulation allows trainees to gain experience without depending on encounters with live patients during their clinical rotations.10 Modular curricula using simulation have been shown to be effective in teaching trainees to achieve competence in arterial puncture,11 administration of injections,12 central venous catheterization,13 and resuscitation skills.14 We found that a majority of programs surveyed had some formal method for teaching several procedures, but a minority of respondents reported the use of simulation for non-PALS skills. We found that only for bag-mask ventilation was there a statistically significant association between the use of simulation to teach a procedure and the perceived competence of residents to perform that procedure. More research is needed to determine the optimal methods for teaching the skills that currently are required by the RRC.
At the time of the next RRC review of the general program requirements for pediatric training programs, it is essential that the consensus opinions of program directors be represented in the development of the list of recommended procedural skills. The RRC has worked closely with the APPD on past reviews and has actively sought the opinions of its membership to respond to proposed changes to the requirements (T Sectish, MD, Children's Hospital Boston, verbal communication, 2007). Continuation of this productive relationship will help to ensure that the new recommendations reflect the ideas of those who are deeply committed to the pediatric residency training process.
Until new data are available on which procedures are used by the majority of pediatric trainees after graduation, we believe that the list of recommended procedures should focus on those that are required to provide optimal patient care during training. There will be differences between institutions in defining that group (based on the availability of ancillary services, other skilled practitioners, and other factors), but we advocate for a list that more comprehensively reflects program directors' opinions. Whether residency training should be further tailored on the basis of the type of institution or residents' ultimate career goals remains a topic for ongoing discussion by pediatric educators. Until such questions are answered in a broader context, we believe that it is important to identify a set of procedural skills that should be mastered by all pediatric trainees.
Our study has several limitations. First, the survey did not reach all accredited programs; only active, dues-paying members of the APPD were surveyed. We do not know whether the nonactive members would differ significantly in their responses from those included in our study. Second, the assessment of resident competence is based on the estimates of program directors. It is not known whether program directors are more likely to over- or underestimate their residents' competence, but, as previously noted, available data suggest that basic procedures are not learned well by many residents. In addition, because no standardized method to assess competence has been developed for most of these procedural skills, objective assessment of competence cannot be easily obtained. Finally, despite extensive efforts to create a comprehensive list of procedures for the survey, it is possible that we failed to include other procedures that program directors find very important to pediatric training. If this possibility were true, however, then it would only further confirm our assertion that the RRC's list of recommended procedures needs reexamination and revision.
We have reported the results of a national survey of pediatric program directors that assessed the current state of procedural skills training in pediatric residency programs. There is not widespread agreement among program directors on which procedures should be mastered during training, and many residents may not be competent to perform several basic procedures at the conclusion of their training. The current era of postgraduate medical education has seen an increase in the demands on residency programs to document educational outcomes. The list of required procedures must be refined to reflect the consensus of pediatric educators, and standardized methods for teaching procedural skills and objectively documenting competence must be developed.
This study was supported by a Children's Hospital Boston internal grant.
We acknowledge the assistance of the APPD for technical and organizational support of this research study.
- Accepted April 30, 2007.
- Address correspondence to Michael G. Gaies, MD, MPH, C.S. Mott Children's Hospital, L1242 Women's, Box 0204, 1500 East Medical Center Dr, Ann Arbor, MI 48109-0204. E-mail:
The authors have indicated they have no financial relationships relevant to this article to disclose.
- ↵Sectish TC, Zalneraitis EL, Carraccio C, Behrman RE. The state of pediatric residency training: a period of transformation of graduate medical education. Pediatrics.2004;114 :832– 841
- ↵Accreditation Council for Graduate Medical Education. Outcome Project: Enhancing Residency Education Through Outcomes Assessment. Available at: www.acgme.org/Outcome. Accessed January 22,2007
- ↵Accreditation Council for Graduate Medical Education. Residency Review Committee section. Available at: www.acgme.org/acWebsite/about/ab_ACGMEpolicyProceed06_07.pdf. Accessed January 22,2007
- ↵Falck AJ, Escobedo MB, Baillargeon JG, Villard LG, Gunkel JH. Proficiency of pediatric residents in performing neonatal endotracheal intubation. Pediatrics.2003;112 :1242– 1247
- ↵Oliver TK Jr, Butzin DW, Guerin RO, Brownlee RC. Technical skills required in general pediatric practice. Pediatrics.1991;88 :670– 673
- Copyright © 2007 by the American Academy of Pediatrics