OBJECTIVE: To measure the effects of participating in structured oral presentation evaluation sessions early in pediatric clerkships on students' subsequent presentations.
METHODS: We conducted a single-blind, 3-arm, cluster randomized controlled trial during pediatric clerkships at Boston University School of Medicine, University of Maryland School of Medicine, Oregon Health & Science University, and Case Western Reserve University School of Medicine. Blocks of students at each school were randomly assigned to experience either (1) no formal presentation feedback (control) or a small-group presentation feedback session early in pediatric clerkships in which students gave live presentations and received feedback from faculty who rated their presentations by using a (2) single-item (simple) or (3) 18-item (detailed) evaluation form. At the clerkship end, overall quality of subjects’ presentations was rated by faculty blinded to randomization status, and subjects reported whether their presentations had improved. Analyses included multivariable linear and logistic regressions clustered on clerkship block that controlled for medical school.
RESULTS: A total of 476 participants were evenly divided into the 3 arms, which had similar characteristics. Compared with controls, presentation quality was significantly associated with participating in detailed (coefficient: 0.38; 95% confidence interval [CI]: 0.07–0.69) but not simple (coefficient: 0.16; 95% CI: −0.12–0.43) feedback sessions. Similarly, student self-report of presentation improvement was significantly associated with participating in detailed (odds ratio: 2.16; 95% CI: 1.11–4.18] but not simple (odds ratio: 1.89; 95% CI: 0.91–3.93) feedback sessions.
CONCLUSIONS: Small-group presentation feedback sessions led by faculty using a detailed evaluation form resulted in clerkship students delivering oral presentations of higher quality compared with controls.
- BUSM —
- Boston University School of Medicine
- CI —
- confidence interval
- CWRUSM —
- Case Western Reserve University School of Medicine
- OHSU —
- Oregon Health & Science University
- OP —
- oral presentation
- OR —
- odds ratio
- UMSM —
- University of Maryland School of Medicine
What’s Known on This Subject:
Delivering competent oral case presentations is an important clinical communication skill, yet effective means of improving trainees’ presentations have not been identified.
What This Study Adds:
Oral presentation feedback sessions facilitated by faculty by using an 18-item competency-based evaluation form early in pediatric clerkships improved medical students’ subsequent oral presentations. Medical schools should consider implementing this evidence-supported practice.
While caring for patients, physicians communicate by delivering structured oral case presentations (OPs). Because the ideal method of teaching presentation skills is unknown, students starting clerkships often struggle to fulfill their teachers’ expectations for delivering OPs. Learning presentation skills can be stressful; pilot studies of medical students reveal substantially elevated cortisol concentrations before and after giving OPs.1 Students frequently perceive OPs to be “a rule-based, data-storage activity governed by order and structure,”2 whereas clinicians often view OPs “as a flexible form of communication with content determined by the clinical context and audience.”3 Because of shortcomings in study design, previous research has not conclusively identified effective methods of teaching this skill during required clerkships.4–7
Our specific aim was to compare the effects of 3 different approaches to teaching students OP skills early in pediatric clerkships on students' subsequent presentations. To address this aim, we conducted a multicenter cluster randomized controlled trial (RCT) in third-year medical students rotating in pediatrics that had 3 study arms: (1) no feedback session (control), (2) a presentation feedback session with the use of a simple evaluation form (simple), and (3) a presentation feedback session with the use of a 18-item evaluation form (detailed). We hypothesized that students who participated in simple or detailed presentation feedback sessions early in their pediatric clerkship would subsequently deliver higher quality OPs than controls.
We conducted a single-blind, stratified, cluster RCT8,9 at 4 medical schools to compare 2 interventions with no intervention. Before the study began, randomization was performed by using a random-number generator, and then study arm assignment for each block was disseminated to sites. Because randomizing by block rather than by student minimizes contamination of study arms while providing the same educational experience within a clerkship block, the units of randomization of this cluster RCT were clerkship blocks (eg, clusters of all students simultaneously rotating in pediatrics during the same academic block at 1 school). At each school, clerkship blocks were stratified into groups of 2 to 3 successive blocks; within each stratum, these clusters were randomly assigned to experience either (1) no feedback session (control), (2) an early OP feedback session with the use of a single-item evaluation form (simple), or (3) an early OP feedback session with the use of an 18-item evaluation form (detailed). Because their clerkship experiences included the scheduled feedback sessions, simple- and detailed-group participants were not blinded; however, faculty who assessed the primary outcome were blinded to intervention status.
Participants were medical students rotating in the third-year pediatric clerkship at Boston University School of Medicine (BUSM), University of Maryland School of Medicine (UMSM), Oregon Health & Science University (OHSU), or Case Western Reserve University School of Medicine (CWRUSM) over a 365-day period. Because of scheduling differences, the study start and end dates differed by school. The institutional review boards at each school approved this study. Informed consent was provided by all participants; the institutional review boards at BUSM and CWRSM required documentation of informed consent, whereas those at UMSM and OHSU did not.
Students in all 3 arms attended a lecture during the clerkship’s first week that reviewed important aspects of delivering OPs in pediatrics. The control arm had no other scheduled OP experiences. Students in the other 2 arms participated in an early OP feedback session held once during the first 10 days of the clerkship during which they gave a “live presentation” (eg, not recorded) of any patient they evaluated during the clerkship to a session facilitator. In the vast majority of sessions, 3 to 6 students took turns presenting and receiving feedback; few sessions were conducted one-on-one.
Feedback session facilitators were “faculty” (eg, pediatric residents, fellows, or attending physicians) volunteers who received advance training on form completion and session facilitation. All facilitators were instructed to provide students with constructive feedback about their presentations but were not explicitly told to use the form to guide their verbal feedback; students were allowed to see the completed evaluation form with all of their scores. As a result, the “simple” and “detailed” groups differed only in the form facilitators completed to rate the presentations. Of the 54 facilitators, 25 assisted only simple sessions, 22 only detailed sessions, and 6 assisted both sessions; in sum, 13% of intervention subjects (7 detailed and 39 simple) experienced sessions led by a facilitator who previously led the other type of session.
In the simple sessions, facilitators rated each student’s overall performance on a single-item 9-point Likert scale on which 1 = “needs significant help”, 3 = “needs some help”, 5 = “mostly on target”, 7 = “above expectations”, and 9 = “well above expectations.” In the detailed sessions, facilitators evaluated presentations by using an 18-item form that assessed presentation quality overall on the same 9-point Likert scale as well as 17 presentation-specific domains (see Table 1) rated on 5-point Likert scales previously shown have a reliable intraclass correlation coefficient of 0.9.10 Participants did not receive copies of either evaluation form.
The primary outcome was the overall quality of participants’ OPs during the last week of the clerkship, assessed by “outcome evaluators” blinded to the study arm. Outcome evaluators were 46 pediatric “faculty” volunteers who received a brief training in rating presentations with the use of the 18-item form. Subsequently, outcome evaluators listened to a student’s live presentation in one-on-one sessions before rating their overall performance on the 9-point Likert scale detailed above. During each evaluation session, a student presented a clinical case from their clerkship experience to an outcome evaluator in an office or conference room. Outcome evaluators rated from 3 to 30 outcome presentations during the study.
The secondary outcome was student report of whether their presentations improved during the clerkship. Specifically, on a confidential survey completed during the last week of the clerkship before the outcome assessment session, participants were invited to rate their level of agreement with the following question: “I feel that my ability to give effective oral case presentations improved during the clerkship” on a 10-point Likert scale, on which 1 = “strongly disagree,” 5 = “neutral,” and 10 = “strongly agree.” Ratings >7 reflected that subjects felt that their ability to deliver a presentation improved. Students were offered a coffee gift card of $5 value as an incentive to complete the survey, which was collected by nonfaculty clerkship staff.
The survey also collected demographic information, including participant age, gender, graduate degrees, previous clerkships completed, and timing of their pediatrics clerkship.
For each independent measure (see Table 2), we conducted χ2 analyses to compare rates between the control, simple, and detailed groups. Because cluster randomization of unequal numbers of subjects typically produces uneven group sizes11,12 and clusters in our study ranged from 5 to 25 students per block, all unadjusted and adjusted regressions were clustered on clerkship block. To examine relationships between demographic characteristics and presentation quality scores, we conducted unadjusted linear regressions; to assess factors associated with student report of presentation improvement, we conducted unadjusted logistic regressions. To determine independent predictors of outcomes, we conducted multivariable linear and logistic regressions controlling for study arm, as well as medical school as a fixed effect. All reported P values are 2-sided. Power calculations using the SD from preliminary analyses of 1.1, 2-sided α = 0.05, and β = 0.10 determined that 133 participants per group would result in detecting a statistically significant effect size in presentation quality of ≥0.4 on a 10-point scale.13 All analyses were conducted by using Stata SE 13.0 (StataCorp LP, College Station, TX).
To address between-group crossovers, we conducted 3 sets of analyses of the data categorized as (1) intention-to-treat, (2) as treated, and (3) per protocol. We present the more conservative intention-to-treat analyses as our primary results and include all 3 sets of results in the Supplemental Information and Supplemental Tables 6 and 7. To maintain the “single-blind” status of this RCT, presentations evaluated by nonblinded faculty were excluded from analyses of this outcome.
The 476 participants were evenly divided among the control (158; 33.3%), simple (160; 33.4%), and detailed (158; 33.3%) groups. Two clusters at 1 school and 1 cluster at another (29 participants in total) did not experience their randomly assigned intervention status due to miscommunications among our research team (see Supplemental Information and Supplemental Tables 6 and 7). In addition, cancellation by faculty scheduled to serve as outcome evaluators resulted in 32 participants being evaluated by coinvestigators who were unblinded to intervention status. To maintain the single-blind status of the RCT, these 32 subjects’ presentation quality was excluded from analyses.
The vast majority of participants returned completed surveys (N =473; response rate = 99.4%). At BUSM, 98.3% of students consented to participate, as did 98.7% of CWRSM students. More participants were students at BUSM or UMSM than at OHSU or CWRUSM, and few were MD/PhD students (4.6%). The proportions of participants in each study arm significantly varied at BUSM and UMSM, and among MD/PhD students (Table 3).
In unadjusted analyses (Table 4), mean presentation quality was significantly higher in the detailed group than in the control group but was not statistically significantly different between the simple group and either the control or detailed groups. Among controls, the mean presentation quality at BUSM (6.18) was significantly higher than at UMDSM (5.76; P = .03) but was not significantly different from OHSU (5.82; P = .07) or CWRUSM (5.55; P = .17). We found no statistically significant associations between presentation quality and other demographic characteristics.
Controlling for school in multivariable linear regressions clustered on block revealed that presentation quality scores were 3.8% higher in the detailed group than in the control group, which was a statistically significant difference (regression coefficient: 0.38; 95% confidence interval [CI]: 0.07 to 0.69; see Table 4), but were not significantly different between the simple group and either the control (regression coefficient: 0.16; 95% CI: −0.12 to 0.43) or the detailed groups (regression coefficient: 0.22; 95% CI: −0.09 to 0.53). Results of the as-treated and per-protocol analyses did not meaningfully differ from the intention-to-treat analyses (see Supplemental Information and Supplemental Tables 6 and 7).
In unadjusted analyses, participants in the detailed group had significantly higher odds of reporting presentation improvement than did controls, but there were no statistically significant differences between either the simple and control groups or the detailed and simple groups (Table 5). Students at BUSM (the reference site) had significantly lower odds of reporting presentation improvement than did students at OHSU (odds ratio [OR]: 0.39; P = .03) or CWRUSM (OR: 0.23; P = .03); no significant difference was observed between BUSM and UMDSM students (OR: 0.63; P = .17).
Multivariate analyses controlling for school revealed that detailed participants had a significantly higher odds of reporting presentation improvement than did controls (Table 5) but did not find a significant difference between either the simple and control groups or the detailed and simple groups. Results of the as-treated and per-protocol analyses were very similar to those of the intention-to-treat analyses (Supplemental Information and Supplemental Tables 6 and 7).
Delivering constructive criticism to medical students rotating in pediatrics about their OPs during feedback sessions in which facilitators used an 18-item evaluation form improved the quality of students’ subsequent presentations. However, feedback on presentations given by session facilitators who completed a single-item evaluation form did not improve students’ subsequent presentations. Although the “detailed” session facilitators completed a reliable 18-item instrument that has face validity,10 the tool has not been formally validated, so the precise educational impact of the measured 3.8% higher faculty-rated presentation quality scores is unclear. Although this effect size is a real difference, some teachers may dismiss it as insufficient to justify adding OP feedback sessions to the clerkship. Others will feel that any real improvement is worth a modest investment of teaching time, especially if it is simply incorporated into attending physicians’ duties. We feel that a 1-hour feedback session that improves a key clinical skill is a worthwhile investment of time, given the extensive time dedicated to a student’s medical education overall. Because this study implemented rigorous research methods and was conducted in a generalizable study population, schools should consider adopting this approach to improve students’ presentation skills.
To our knowledge, this multicenter RCT is the first study of interventions designed to improve third-year students’ OPs that has convincingly identified an effective teaching method. Although a Web-based curriculum followed by deliberate practice improves preclinical students’ case presentations,4 tools designed to improve third-year students’ presentations have seldom been tested by using rigorous research methods. A noncontrolled, nonblinded “before/after” study of a multimodal presentation teaching program implemented during an internal medicine clerkship at a single school found that students’ presentation skills were rated higher during the intervention year than in the previous year.5 A clinical reasoning curriculum implemented during a medicine clerkship at a single institution showed improved presentation skills among intervention students compared with controls.6 Finally, a single-center RCT of weekly assessments of students’ presentations with the use of encounter cards over the 2-month medicine clerkship did not reveal a statistically significant impact on students’ performance.7
Our study differs from these previous studies, which were all conducted at single institutions4–7; in addition, the 2 studies that identified effective presentation tools were either conducted in preclinical students4 or without randomizing subjects.5 Our decision to conduct a multicenter RCT was validated by our finding that our results varied significantly by schools in unadjusted analyses, and that controlling for school improved the precision of our point estimates. Reasons for the between-school variance are unclear but may involve a complex interplay of institutional culture and expectations rather than absolute differences in student abilities. Regardless of the etiology of such institutional differences, medical education research should be conducted in multiple centers.
Many potential explanations exist for the increased effectiveness of the “detailed” feedback sessions. Although students’ clinical performance may be mediated by factors that differ among schools, multivariate analyses controlling for school revealed that presentations improved when students participated in feedback sessions with session facilitators referring to the 18-item evaluation form but not in sessions that used a single-item form. Although feedback session facilitators were oriented to running the sessions and completing the forms, they were not instructed to treat the forms as scaffolding for the feedback they delivered. Regardless, the impact of the 2 feedback sessions presumably differed because the forms differed in content. Although both forms include the overall presentation quality scale, the detailed form also includes 17 items presented with the use of behavioral anchors designed to incorporate elements of clinical reasoning into the feedback.10
Several limitations to this study warrant consideration. First, miscommunications among our study team resulted in 3 clerkship blocks not receiving their randomly assigned experience, which we conservatively addressed by reporting results of intention-to-treat analyses as our primary findings; therefore, our results may underestimate the impact of the interventions on presentation quality, as the Supplemental Information and Supplemental Tables 6 and 7 show. Second, although the primary outcome was assessed with a reliable instrument10 by evaluators oriented to its use, we did not record the feedback or the outcome assessment sessions; doing so could have confirmed the feedback provided during the detailed and simple sessions differed, and clarified what aspects of presentations improved during the study. However, because the only difference between simple and detailed sessions was the form faculty completed, our results are consistent with the feedback given during the sessions being different. Third, although outcome evaluators at all sites received similar training before the sessions, we did not perform cross-institutional checks for uniformity in outcome evaluation standards, so differing expectations of presentations at the 4 schools may have influenced outcome ratings. Fourth, although a 5-point Likert scale version of questions assessing overall presentation quality was shown to be reliable in rating taped presentations,10 neither the reliability of the 9-point Likert scale version of this question nor its use in rating “live” presentations has been assessed. In addition, although the detailed form has face validity, it has not been formally validated, nor has the instrument used to assess student report of presentation improvement. Fifth, although “randomization generally provides excellent control over confounding,”14 even a randomized design cannot completely eliminate the possibility that measured or unmeasured factors, such as the numbers of patients evaluated or presentation given, may have also influenced presentation quality. Finally, although we applied a robust study design to a generalizable study population of students attending 2 private and 2 public medical schools in different regions of the United States, specific results might differ if the study was replicated at different schools. However, because multivariate analyses controlling for school improved the precision of our results, the overall findings of studies designed to replicate our findings would likely be similar.
Our study has several implications. Presentation skills are critical to learn during medical school, and our methodologically rigorous study of presentation feedback sessions identified an effective approach to teaching presentation skills. Although the intervention’s effect size was modest and all participants were likely exposed to a morass of possible favorable influences during the study, this work shows that presentations by students who heard a lecture on presentation skills followed by a small-group session devoted to providing them feedback on their skills were rated higher than those who only experienced the lecture. A structured approach to evaluating skills appears to be necessary to achieve this result. Medical schools should implement evidence-supported practices to improve student performance. We observed variations in outcomes by medical school, which emphasizes the importance of implementing multicenter RCTs to test medical education interventions. Although the sessions only required an hour-long faculty commitment per block, variable faculty availability made it challenging to reliably implement feedback sessions during the clerkship. As a result, a priority for future research is to identify effective and practical interventions that would be easier to implement within the time confines of clerkship schedules. These considerations aside, we showed that special sessions in which facilitators gave students specific, detailed feedback on their presentations improved a critical clinical skill.
We acknowledge the voluntary contributions of effort made by clerkship coordinators and numerous faculty at each center who facilitated feedback sessions or evaluated the primary outcome. We also appreciate feedback provided during the study design and manuscript preparation phases of this work by Dr Michael Silverstein, Boston University School of Medicine, and Dr Harold Sox, Geisel School of Medicine at Dartmouth.
- Accepted August 28, 2014.
- Address correspondence to Colin M. Sox, MD, MS, Boston University School of Medicine, Vose Hall #307, 88 East Newton Street, Boston, MA 02118. E-mail:
Dr Sox conceptualized and designed the study, secured grant funding to support conducting the study, had full access to all of the data in the study, conducted all data analyses, coordinated and supervised data collection at 1 of the 4 sites, and drafted the initial manuscript; Drs Dell, Lewin, and Phillipi each coordinated and supervised data collection at 1 of the 4 sites and reviewed and revised the manuscript; Dr Cabral designed the analytic plan, provided guidance during data analyses, and reviewed and revised the manuscript; Dr Vargas designed the data collection instruments, coordinated data collection at the 4 sites, had full access to all of the data, created the data set, and critically reviewed the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: Drs Dell and Phillipi receive modest compensation for serving as members of the CLIPP editorial board; the other authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This study was supported by the Joel & Barbara Alpert Endowment for the Children of the City (Boston, MA). The Alpert Foundation had no role in the design or conduct of the study; the collection, management, analysis, or interpretation of the data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Dell M,
- Lewin L,
- Gigante J
- Browner WS,
- Newman TB,
- Hulley SB
- Koepsell TD,
- Weiss NS
- Copyright © 2014 by the American Academy of Pediatrics