search text   Advanced Search

This Article Abstract Full Text (PDF) P3Rs: Submit a response Alert me when this article is cited Alert me when P3Rs are posted Alert me if a correction is posted Citation Map Services E-mail this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Add to My File Cabinet Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via CrossRef Citing Articles via ISI Web of Science (1) Citing Articles via Google Scholar Google Scholar Articles by Gaies, M. G. Articles by Sandora, T. J. Search for Related Content PubMed PubMed Citation Articles by Gaies, M. G. Articles by Sandora, T. J. Related Collections Office Practice

Assessing Procedural Skills Training in Pediatric Residency Programs

^a Department of Pediatrics and Communicable Diseases, Division of Pediatric Cardiology, C.S. Mott Children's Hospital, Ann Arbor, Michigan
^b Department of Medicine, Children's Hospital Boston, Boston, Massachusetts
^c Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts
^d Office of Educational Affairs, Tufts University School of Medicine, Boston, Massachusetts

	ABSTRACT

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 
OBJECTIVE. The objective of this study was to assess the opinions of pediatric program directors regarding procedural skills training of pediatric residents.

METHODS. We developed a survey based on the Residency Review Committee's guidelines for procedural training. It included items about the importance of 29 procedures encountered in pediatric training, estimates of residents' competence in performing them, and the teaching of procedural skills. The survey was sent to members of the Association of Pediatric Program Directors. The primary outcome was the perceived importance for residents to achieve competence in these procedures, rated on a 10-point Likert scale. Secondary outcomes included perception of resident competence to perform procedures and educational methods used by respondents for teaching procedural skills. Associations between demographic characteristics and perceived importance or competence were also assessed.

RESULTS. Surveys were sent to 139 programs, and 112 responded. Thirteen procedures were rated 8 or higher by >75% of program directors. Seven skills that were prioritized by the Residency Review Committee did not achieve this level of consensus. Respondents reported that many residents failed to achieve competence by the end of training in 9 of 13 procedures that they rated as very important, including venipuncture, neonatal intubation, and administering injections. Residents who perform the majority of venipunctures and intravenous catheter placements at their institutions were more likely to be judged competent in performing these skills than residents who do not.

CONCLUSIONS. The Residency Review Committee's list of procedures does not necessarily reflect the opinions of pediatric program directors on the most essential skills for trainees. Many residents may not develop competence in several important procedures by the end of residency, most notably vascular access and life-saving skills. A more robust and standardized method is needed for teaching procedural skills and for documenting competence.

Key Words: procedures • clinical skills • pediatrics • residency • education

Abbreviations: RRC—Pediatric Residency Review Committee • APPD—Association of Pediatric Program Directors • PALS—Pediatric Advanced Life Support

The mandate for competency-based postgraduate medical training requires all residency programs to teach clinical skills formally.^1,2 The Pediatric Residency Review Committee (RRC) is responsible for creating and modifying the general program requirements for pediatric residencies, including the recommended list of procedural skills. A review of these requirements occurs every 5 years. The RRC is mandated to inform program directors and program director associations when the contents of program requirements are created or modified. Program directors are then given a 45-day period to comment, either individually or through a program director association, on these proposed changes. The RRC can choose to accept or reject suggestions from program directors or program director associations, but there is no requirement to distribute these suggestions to the group of program directors in its entirety.³ As a result of this process, the RRC currently specifies that all pediatric residents should have "sufficient training" in 16 procedures (Table 1) and that they should be monitored by the residency program for appropriate skill development.³ However, previous studies have demonstrated that although many residents feel confident in their abilities to perform these procedures, they do not perform all skills competently when they are formally tested.^4–6

	METHODS

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 
Survey Design
To determine pediatric program directors' opinions about the importance of various procedural skills, we conducted a survey of all registered members of the Association of Pediatric Program Directors (APPD). Our survey design was modeled on a previous survey study by Wigton et al,⁷ in which internal medicine program directors were asked to rate the importance of procedures for resident training. To develop a comprehensive list of procedures of high priority for pediatric residency training programs, we convened focus groups of Children's Hospital Boston faculty to define a list of procedures that merit inclusion in the final survey tool. These groups were made up of generalists, subspecialists, proceduralists, and nonproceduralists and included senior clinical fellows and staff. The RRC's list of recommended procedures³ was used as a starting point for the list. In collaboration with the participating physicians, it was determined that procedural skills that involved physical contact with patients would be the focus of this study; procedures such as developmental screening and pain management, although considered essential to pediatric training, were excluded from the final survey instrument. The focus groups also identified procedures that were not on the RRC's list but were believed to be important enough to include in the survey, and component skills of basic and advanced life support were specifically delineated and included. In addition, we included a few procedures on the survey that the groups agreed were not important to pediatric training, to be sure that our survey could identify both positive and negative opinions about individual procedures. After initial survey development, former chief residents from 3 different institutions were asked to pilot-test the survey; cognitive interviews were performed to assess item comprehension and to refine further the list of procedures included in the survey. Finally, in collaboration with the APPD, a Web-based version of the survey instrument was created.

The first part of the survey asked for demographic information on each respondent and program, including years since completing training, specialty, program size, number and types of fellowship programs associated with the institution, and various other identifiers. For the purposes of assessing the impact of specialty training on responses, we considered the following 4 specialties a priori to be "proceduralist" fields: cardiology, critical care, emergency medicine, and neonatology. For each of the 29 procedures included in the final survey, respondents were asked the following questions: "How important is it for pediatric residents to be able to perform the procedure competently by the end of their training?" and, "In your estimation, what percentage of residents in your program is competent to perform the procedure by the end of their residency?" For the question asking the respondents to rate the importance of the procedure, participants responded using a 10-point Likert scale, with 1 representing "not important at all" and 10 representing "extremely important." We decided a priori to identify procedures that were rated 8 by >75% of program directors, because procedures that achieve this degree of consensus at that level of importance might reasonably be considered essential to pediatric training. A similar definition of importance has been used in previous studies.^7,8

Regarding competence, respondents were asked to estimate the proportion of residents in their programs who were competent to perform the procedure by the end of their residency: "all or almost all," "more than half," "approximately half," "less than half," or "few or none." Competence was defined for the participants as "the ability to perform a procedure independently, without supervision, and with a high likelihood of successful completion. This includes an understanding of the indications, contraindications, and risks of the procedure." It is reasonable to expect that by the end of residency, all or almost all residents should be competent to perform procedures that are considered essential to pediatric training. Therefore, we sought to identify procedures for which fewer than two thirds of program directors thought "all or almost all" of their residents were competent, believing that such procedures may not be mastered by a significant portion of graduating residents. A final item asked about current educational practices for teaching and documenting procedural skills on a subset of 10 procedures identified by our focus groups as the most critical to resident training.

Participants
An e-mail that contained a link to the online survey was sent to the registered program directors and assistant program directors of all 139 active, dues-paying member programs of the APPD in September 2005. The first responding program director or assistant program director from each program was enrolled in the study. Subsequent responses from other members of the same program were not included to ensure that each participating program was equally represented in the study. Programs that did not respond initially received a total of 3 reminder e-mails at 2-week intervals.

Statistical Analysis
Descriptive analyses of survey responses were performed to characterize demographics of respondents, the opinions about the importance of each procedure to residency training, estimates of the current level of competence of pediatric residents at the respondents' institutions, and the current teaching strategies used at pediatric training programs. ² and Fisher's exact tests were used as appropriate to assess whether perceived importance or estimated competence for a procedure was associated with respondent or program characteristics. Statistical analyses were performed using SAS 9.0 (SAS Institute, Cary, NC). Two-sided P < .05 indicated statistical significance.

	RESULTS

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 
We received at least 1 response from 112 (81%) of the 139 programs surveyed. Demographic data are presented in Table 2. Because the data from nonresponding programs were not known, no comparisons could be made between respondents and nonrespondents. The majority of our respondents represented training programs at freestanding children's hospitals, and most stated that 20% of their residents eventually pursue fellowship training.

View larger version (18K): [in this window] [in a new window]   FIGURE 1 Percentage of respondents who rated procedure 8. BLS indicates basic life support; Gyn, gynecologic; Admin, administration; SC, subcutaneous; IM, intramuscular; ID, intradermal; I&D, incision and drainage; Ortho, orthopedic; ED, emergency department.

View larger version (17K): [in this window] [in a new window]   FIGURE 2 Percentage of program directors who responded "all or almost all" are competent to perform the procedure at the end of training. BLS indicates basic life support; Gyn, gynecologic; Admin, administration; SC, subcutaneous; IM, intramuscular; ID, intradermal; I&D, incision and drainage; Ortho, orthopedic; ED, emergency department.

Educational Methods
Respondents were asked about the current educational practices being used at their program for 10 of the 29 procedures included in the survey. Figure 3 displays the percentage of program directors who responded that they had any formal method for teaching these skills and the percentage who responded that they used some form of simulation to teach these skills.

View larger version (18K): [in this window] [in a new window]   FIGURE 3 Educational methods used for teaching procedural skills.

	DISCUSSION

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 
We found that wide variation exists in pediatric program directors' ratings of the importance of achieving competence in several procedures recommended by the RRC, including intraosseous line placement, arterial puncture, umbilical catheter placement, and administration of injections. Program directors also disagreed with the RRC on the importance of procedures that are part of PALS training and included in the broader rubric of resuscitation skills (eg, defibrillation, endotracheal intubation). For each of these procedures, <75% of program directors rated the procedure 8. Only 1 procedure that was included in our survey and was not on the RRC's list—fluorescein eye examination—was deemed very important for training by >75% of program directors.

When asked to estimate the competence of graduating residents to perform each of the procedures in the survey, fewer than two thirds of program directors thought that all or almost all of their residents were competent to perform several of the procedures at the end of residency training. This perception was especially true for procedures that would be classified as a life-saving or vascular access skill. Furthermore, several procedures were classified as very important by program directors and may not be performed competently by a significant number of graduating residents, based on program directors' estimates. These findings are consistent with the results from the study of internal medicine program directors by Wigton et al⁷ and from studies that directly assessed pediatric residents' ability to perform resuscitations^5,6 and neonatal endotracheal intubation.⁴ To our knowledge, this is the first collective assessment of the procedural skills of pediatric trainees in the United States.

It is notable to compare the results of our survey with those included in a study by Oliver et al⁸ in 1991. In a survey conducted in 1986, members of accredited pediatric training programs were given a list of 101 technical skills (including procedural, laboratory, and physical examination skills) and asked which skills residents should be able to perform. Seventy-two of those procedures were considered "essential" by >80% of respondents. Several procedures that were deemed essential in that study received much lower ratings in our study, including arterial puncture, bladder aspiration, administration of injections, arterial catheter placement, cardioversion/defibrillation, umbilical artery and vein catheterization, and chest tube placement/thoracentesis.

There are many reasons that program directors may be de-emphasizing the teaching and learning of procedural skills during pediatric residency and that residents may be failing to achieve competence in these skills. Foremost among these reasons is the increasing skill of nonphysician clinicians and the development of ancillary services, such as phlebotomy and intravenous access teams, to perform procedures that traditionally have been done by residents. This possibility is supported by the fact that only 12% and 13% of respondents identified residents as the people who perform the majority of venipunctures and intravenous catheter placements, respectively, in the hospitals in which they rotate. Other procedures, such as administration of injections, bladder catheterization, and arterial puncture, are frequently performed by nurses, which undoubtedly decreases the motivation and opportunities for busy residents to learn these skills amid the other demands on their time.

When these factors are combined with the increasing amount of time spent on paperwork and documentation by residents, it is not surprising that many trainees will have an insufficient number of opportunities to learn and practice a procedure to achieve competence. We did find that the designation of residents as the clinician performing most venipunctures and intravenous catheter placements predicted in a statistically significant way the perceived competence to perform those skills. This finding suggests that when residents are forced to perform a skill, they may be more likely to master it.

The failure to develop procedural skills becomes self-perpetuating. As more residents graduate from pediatric training programs without these skills, the importance of having opportunities to perform these procedures during fellowship training increases. It is not uncommon at Children's Hospital Boston for fellows to be performing many procedures that traditionally were learned and mastered during residency training. This shift in practice results in fewer opportunities for residents to experience these procedures during training, thereby potentially continuing the cycle by further reducing competence. It is interesting that we did not find any statistically significant associations between the presence of fellowship programs at an institution and the rating of importance or perceived competence by respondents; neither did we find that the percentage of residents who entered fellowship training predicted perceived competence.

It is possible that program directors see procedural training as the responsibility of fellowship programs, where trainees will learn the skills necessary for practice in their field. Such a system may place an undue burden on fellowship programs to ensure adequate opportunities for their trainees to achieve competence in certain procedures and to provide remediation for skills that more appropriately are learned during residency training. Furthermore, if residents are not getting opportunities to learn and practice procedures, then fields such as critical care medicine and cardiology, which typically attract trainees who enjoy procedures, may have a more difficult time recruiting residents.

Another factor that may affect program directors' responses about the importance of teaching certain procedures is the perception that most residents will not perform these procedures after their training has been completed. More research is needed to determine exactly which procedures pediatricians (generalists and specialists alike) are performing in their everyday practice. Studies designed to answer this question were published >15 years ago.^8,9 Newer data would help to direct future curricula and would allow training programs to focus on procedures that residents need to perform to provide adequate care during training and on those that most trainees will perform after they graduate.

Finally, it is possible that current educational methods that are used by programs for teaching procedures are not sufficient to ensure a high level of competence among graduating residents, taking into account the factors previously noted. Simulation allows trainees to gain experience without depending on encounters with live patients during their clinical rotations.¹⁰ Modular curricula using simulation have been shown to be effective in teaching trainees to achieve competence in arterial puncture,¹¹ administration of injections,¹² central venous catheterization,¹³ and resuscitation skills.¹⁴ We found that a majority of programs surveyed had some formal method for teaching several procedures, but a minority of respondents reported the use of simulation for non-PALS skills. We found that only for bag-mask ventilation was there a statistically significant association between the use of simulation to teach a procedure and the perceived competence of residents to perform that procedure. More research is needed to determine the optimal methods for teaching the skills that currently are required by the RRC.

At the time of the next RRC review of the general program requirements for pediatric training programs, it is essential that the consensus opinions of program directors be represented in the development of the list of recommended procedural skills. The RRC has worked closely with the APPD on past reviews and has actively sought the opinions of its membership to respond to proposed changes to the requirements (T Sectish, MD, Children's Hospital Boston, verbal communication, 2007). Continuation of this productive relationship will help to ensure that the new recommendations reflect the ideas of those who are deeply committed to the pediatric residency training process.

Until new data are available on which procedures are used by the majority of pediatric trainees after graduation, we believe that the list of recommended procedures should focus on those that are required to provide optimal patient care during training. There will be differences between institutions in defining that group (based on the availability of ancillary services, other skilled practitioners, and other factors), but we advocate for a list that more comprehensively reflects program directors' opinions. Whether residency training should be further tailored on the basis of the type of institution or residents' ultimate career goals remains a topic for ongoing discussion by pediatric educators. Until such questions are answered in a broader context, we believe that it is important to identify a set of procedural skills that should be mastered by all pediatric trainees.

Our study has several limitations. First, the survey did not reach all accredited programs; only active, dues-paying members of the APPD were surveyed. We do not know whether the nonactive members would differ significantly in their responses from those included in our study. Second, the assessment of resident competence is based on the estimates of program directors. It is not known whether program directors are more likely to over- or underestimate their residents' competence, but, as previously noted, available data suggest that basic procedures are not learned well by many residents. In addition, because no standardized method to assess competence has been developed for most of these procedural skills, objective assessment of competence cannot be easily obtained. Finally, despite extensive efforts to create a comprehensive list of procedures for the survey, it is possible that we failed to include other procedures that program directors find very important to pediatric training. If this possibility were true, however, then it would only further confirm our assertion that the RRC's list of recommended procedures needs reexamination and revision.

	CONCLUSIONS

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 
We have reported the results of a national survey of pediatric program directors that assessed the current state of procedural skills training in pediatric residency programs. There is not widespread agreement among program directors on which procedures should be mastered during training, and many residents may not be competent to perform several basic procedures at the conclusion of their training. The current era of postgraduate medical education has seen an increase in the demands on residency programs to document educational outcomes. The list of required procedures must be refined to reflect the consensus of pediatric educators, and standardized methods for teaching procedural skills and objectively documenting competence must be developed.

	ACKNOWLEDGMENTS

 
This study was supported by a Children's Hospital Boston internal grant.

We acknowledge the assistance of the APPD for technical and organizational support of this research study.

	FOOTNOTES

 
Accepted Apr 30, 2007.

Address correspondence to Michael G. Gaies, MD, MPH, C.S. Mott Children's Hospital, L1242 Women's, Box 0204, 1500 East Medical Center Dr, Ann Arbor, MI 48109-0204. E-mail: mgaies{at}med.umich.edu

The authors have indicated they have no financial relationships relevant to this article to disclose.

	REFERENCES

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 

1. Sectish TC, Zalneraitis EL, Carraccio C, Behrman RE. The state of pediatric residency training: a period of transformation of graduate medical education. Pediatrics. 2004;114 :832 –841[Abstract/Free Full Text]
2. Accreditation Council for Graduate Medical Education. Outcome Project: Enhancing Residency Education Through Outcomes Assessment. Available at: www.acgme.org/Outcome. Accessed January 22,2007
3. Accreditation Council for Graduate Medical Education. Residency Review Committee section. Available at: www.acgme.org/acWebsite/about/ab_ACGMEpolicyProceed06_07.pdf. Accessed January 22,2007
4. Falck AJ, Escobedo MB, Baillargeon JG, Villard LG, Gunkel JH. Proficiency of pediatric residents in performing neonatal endotracheal intubation. Pediatrics. 2003;112 :1242 –1247[Abstract/Free Full Text]
5. White JR, Shugerman R, Brownlee C, Quan L. Performance of advanced resuscitation skills by pediatric housestaff. Arch Pediatr Adolesc Med. 1998;152 :1232 –1235[Abstract/Free Full Text]
6. Nadel FM, Lavelle JM, Fein JA, Giardino AP, Decker JM, Durbin DR. Assessing pediatric senior residents' training in resuscitation: fund of knowledge, technical skills, and perception of confidence. Pediatr Emerg Care. 2000;16 :73 –76[CrossRef][ISI][Medline]
7. Wigton RS, Blank LL, Nicolas JA, Tape TG. Procedural skills training in internal medicine residencies: a survey of program directors. Ann Intern Med. 1989;111 :932 –938[ISI][Medline]
8. Oliver TK Jr, Butzin DW, Guerin RO, Brownlee RC. Technical skills required in general pediatric practice. Pediatrics. 1991;88 :670 –673[Abstract/Free Full Text]
9. Weinberger HL. An attempt to identify the frequency of use of technical skills and procedures by the primary care physician. J Pediatr. 1976;88 (pt 1):671 –677[CrossRef][ISI][Medline]
10. Vozenilek J, Huff JS, Reznek M, Gordon JA. See one, do one, teach one: advanced technology in medical education. Acad Emerg Med. 2004;11 :1149 –1154[CrossRef][ISI][Medline]
11. Fagan MJ, Cece R. Using respiratory therapists to teach arterial puncture for blood gas procedures to third-year medical students. Acad Med. 1999;74 (5):594 –595[CrossRef][Medline]
12. Szumlas GA. Development of an office-based curriculum of common pediatric primary care skills for residents. Acad Med. 2002;77 :749[CrossRef][Medline]
13. Velmahos GC, Toutouzas KG, Sillin LF, et al. Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. Am J Surg. 2004;187 :114 –119[CrossRef][ISI][Medline]
14. Nadel FM, Lavelle JM, Fein JA, Giardino AP, Decker JM, Durbin DR. Teaching resuscitation to pediatric residents: the effects of an intervention. Arch Pediatr Adolesc Med. 2000;154 :1049 –1054[Abstract/Free Full Text]

This article has been cited by other articles:

				M. G. Gaies, C. P. Landrigan, J. P. Hafler, and T. J. Sandora Assessing Procedural Skills Training in Pediatric Residency Programs: In Reply Pediatrics, March 1, 2008; 121(3): 650 - 651. [Full Text] [PDF]

This Article Abstract Full Text (PDF) P3Rs: Submit a response Alert me when this article is cited Alert me when P3Rs are posted Alert me if a correction is posted Citation Map Services E-mail this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Add to My File Cabinet Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via CrossRef Citing Articles via ISI Web of Science (1) Citing Articles via Google Scholar Google Scholar Articles by Gaies, M. G. Articles by Sandora, T. J. Search for Related Content PubMed PubMed Citation Articles by Gaies, M. G. Articles by Sandora, T. J. Related Collections Office Practice

	Home | My Pediatrics | Journal Information | Current Issue | Past Issues | Subscriptions & Services | Contact Us Click here for faster international access © 2007 American Academy of Pediatrics. All rights reserved.