Clerkship Curricular Revision Based on the Ambulatory Pediatric Association and the Council on Medical Student Education in Pediatrics Guidelines: Does It Make a Difference?
Objective. To compare general pediatric knowledge acquisition and clinical problem-solving skills by students pre- and postcurricular reform based on the 1994 Ambulatory Pediatric Association and the Council on Medical Student Education in Pediatrics (APA–COMSEP) curricular guidelines.
Setting. A large, urban academic medical center.
Subjects. Third-year medical students on a required clerkship in Pediatrics.
Intervention. Pre- and postcurricular revision, the students were given both the National Board of Medical Examiners (NBME) Pediatric Subject Examination and an objective examination, which was developed in-house, based on the APA–COMSEP guidelines (Pediatric Clerkship Examination [PCE]). Baseline data before curricular revision were obtained on 52 students from May 1995 to May 1996. After curricular redesign in May 1996, data were obtained on 42 students from May 1996 to May 1997. Curricular revision focused on the following: defining educational principles, selecting teaching strategies, defining learning objectives, implementing the curriculum, and evaluating the students with an examination.
Results. Before curricular revision, the average NBME score was 521 ± 122. The average PCE score was 53.7% ± 10.1%. After curricular revision, the average NBME score was 520 ± 109, and the average PCE score was 67.7% ± 8.4%. Content areas showing the greatest improvement were fluids and electrolytes, issues pertaining to the newborn, and health supervision.
Conclusions. Our baseline data indicate that despite spending two thirds of the clerkship in the ambulatory setting, students did not acquire adequate general pediatric knowledge or clinical problem-solving ability. After broad clerkship revision based on the APA–COMSEP Core Curriculum, students' acquisition of general pediatric knowledge and clinical problem-solving improved significantly, as measured by the PCE. The overall NBME Pediatric Subject Examination scores did not reflect this increased acquisition of general pediatric knowledge.
Health care delivery in the United States has undergone dramatic change in the past 15 years. Medical education, however, has been slow to react to this new paradigm; despite the increasing shift of medicine into the ambulatory setting, students often spend a majority of their time on the wards and have limited exposure to ambulatory patients. Clerkship curricula often focus on diseases encountered in the inpatient setting, a problem that has grown more severe as the nature of hospitalized patients has become limited to those with complex or multiple coexisting diseases, those who have failed to respond to outpatient treatment, or those who need the advanced technology of intensive care units.1
Leaders in medical education have recognized that undergraduate medical education has not provided adequate teaching of important generalist competencies. The Association of American Medical Colleges produced a Policy on the Generalist Physician that states: “Medical schools should require that all medical students have meaningful curricular experiences in the generalist specialties.”2 The physician's role has been redefined to that of caring for the health and wellbeing of a group of patients over time, providing patient care that is comprehensive and longitudinal, and incorporating health-promoting tasks, in addition to the care of acute and chronic illnesses.3 In addition, the principles of prevention, therapeutics, and supportive care should be taught to meet the needs of patients, their families, and their communities.
The Ambulatory Pediatric Association (APA) and the Council on Medical Student Education in Pediatrics (COMSEP) recognized a need for curricular reform in pediatric undergraduate medical education. A task force with broad national representation developed a Core Curriculum in General Pediatrics to help pediatric clerkship directors plan and implement curricular change. This curriculum outlined goals and objectives in the following 10 areas: health supervision, growth and development, behavior, nutrition, prevention of illness and injury, genetics, common pediatric problems, therapeutics, fluids and electrolytes, and pediatric emergencies.4 It also recommended that ∼50% of a student's time during the pediatric rotation be spent in ambulatory settings. A resource manual that addressed implementation, teaching, and evaluation strategies, and a clinical problems supplement that translated the objectives and competencies of the curriculum into real-life patient problems were provided along with the curriculum.
Substantial time and resources were required to develop the APA–COMSEP Core Curriculum, and more time and resources would be necessary for pediatric departments across the country to implement it. Given these requirements, it is reasonable to ask whether curricular reform is needed and whether curricular revision based on the APA–COMSEP guidelines would make a difference in student learning. Only one study to date has examined the issue of the impact of curricular change on student acquisition of knowledge in a pediatric clerkship.5 However, this curricular revision preceded the APA–COMSEP guidelines and focused on only six selected general pediatric objectives. It did not assess comprehensive reform of a clerkship curriculum.
In this article, we describe an effort at comprehensive curricular revision of the UCLA–CHS Pediatric clerkship that was based on the APA–COMSEP guidelines and sound educational principles. In this study, two written examinations, the National Board of Medical Examiners (NBME) Pediatric Subject Examination and a written examination based entirely on the APA–COMSEP objectives, were used to address the following two questions:
What was the nature of pediatric knowledge acquisition and clinical problem-solving skills by students in the clerkship before curricular reform?
Would broad-based curricular revision lead to improvement in general pediatric knowledge acquisition and clinical problem-solving skills?
Two outcome measures were used to evaluate the impact of the pediatric clerkship before and after curriculum revision at UCLA–CHS: 1) the Pediatric Clerkship Examination (PCE) and (2) the NBME Pediatric Subject Examination. One year of baseline data using the two examination measures was gathered on 52 students from May 1995 to May 1996, before curricular revision. After implementation of the revised clerkship curriculum in May 1996, another year of data were gathered on 42 students from May 1996 to May 1997, with the same two evaluation measures as before curricular revision. Of note, before curricular revision, the NBME Pediatric Subject Examination scores were used in determination of students' final grades for the clerkship. After curricular revision, the PCE scores were used.
The pediatric clerkship at UCLA–CHS is a 6-week rotation. Before curricular revision, students spent a majority of their time (4 weeks) in the ambulatory setting and 2 weeks on a general pediatric ward team in a tertiary care academic medical center. The 4 weeks of ambulatory care were divided into 2 weeks in a county hospital acute care clinic and 2 weeks in a university hospital acute care clinic. In addition, students spent a half-day in a community preceptor's office.
All the students rotating through UCLA–CHS for their required pediatric clerkship were tested. Pretest students were from the class of 1997 and posttest students were from the class of 1998. The assumption was made that students entering UCLA are typically the same each year, as evidenced by their class profiles. The class of 1997 had a mean undergraduate GPA of 3.58, mean USMLE Part I score of 213, and mean USMLE Part II score of 208. The class of 1998 had a mean undergraduate GPA of 3.58, mean USMLE Part I score of 211, and mean USMLE Part II score of 210.
The PCE was designed to measure a medical student's mastery of pediatric competencies based on the APA–COMSEP guidelines. The topic areas of the PCE were patterned after the knowledge categories within the guidelines. The categories and the proportion of questions represented on the PCE were as follows: Common Pediatric Illnesses (28%); Health Supervision (19%); Growth and Development (13%); Issues Unique to the Newborn (12%); Fluids and Electrolytes (7%); Therapeutics (6%); and Behavior, Nutrition, and Pediatric Emergencies each with 5%. Of the 51 total questions on the PCE, 31 were taken directly from the Clinical Problems Supplement to the APA–COMSEP guidelines. The other 20 questions were written by the authors of this article. The examination was 2 hours long and consisted primarily of short-answer questions, with only five multiple-choice questions. Short-answer questions were used to assess students' clinical problem-solving ability with case-based scenarios. One author scored all the examinations for the pilot and study period. Partial or total credit was given based on completeness of answers as evidenced by predetermined criteria established by us.
The NBME Pediatric Subject Examination is a nationally standardized examination consisting of 120 objective multiple-choice questions. Students are given 2 hours to complete the examination. The questions on this examination cover a broad range of topics that reflect inpatient and outpatient experiences in general and subspecialty pediatrics.
The PCE was piloted for 6 consecutive months at four different UCLA-affiliated sites (CHS, Cedars–Sinai Medical Center, Harbor–UCLA Medical Center, and Martin Luther King/Drew Medical Center) and was determined to be reliable based on the Cronbach-α value of 0.75. Expert validity was obtained by having the clerkship directors at each of the three other UCLA-affiliated sites review the examination. All the directors agreed that the examination was fair and valid and have subsequently substituted the PCE for the NBME Pediatric Subject Examination as the end-of-rotation evaluation tool.
The didactic series before revision consisted of 23.5 hours of lecture and case-based discussions, of which 7 hours were devoted to general pediatric topics and 16.5 hours were devoted to subspecialty topics. As part of our revision, suggested general pediatric topics from the APA–COMSEP curriculum were distributed to a member of each general and subspecialty division identified as a medical education liaison. Subspecialty discussions were reduced by 3 hours to a total of 13.5 hours to make time for more general pediatric topics such as nursery, pediatric emergencies, dermatology, and well-child care. General pediatrics discussion hours increased from 7 to 12 hours to accommodate these areas. Faculty were encouraged to use case-based teaching and/or problem-based learning to stimulate student participation in discussions whenever possible.
Previously, students attended weekly sessions with the clerkship director to discuss cases that they had seen during the rotation. These sessions were redesigned to include in-depth discussions on six different general pediatric topics: evaluation of the febrile child, newborn sepsis, respiratory illnesses, vomiting and diarrhea, growth and development, and behavioral issues in pediatrics.
In the past, the clerkship had no required textbook. Two recently published textbooks that have a strong primary care focus are now recommended to students. Each student is expected to choose one of these texts as their primary reference for the rotation. In addition, supplemental readings of recent review articles and topics not covered adequately in the textbooks were provided with the syllabus.
Before curricular revision, specific learning objectives for students had not been identified. We reviewed the APA– COMSEP curriculum and selected 55 objectives in 13 of the 16 content areas. Two of the content areas (“Medical Genetics and Congenital Malformations” and “Child Advocacy”) were omitted intentionally because we felt that they were beyond the scope of what a 3rd-year medical student should master in a core clerkship. A third content area, “Prevention and Treatment of Illness and Injury,” was incorporated into the “Health Supervision” content area.
These objectives were reviewed and agreed on by faculty and housestaff at a division meeting and noon conference. Revisions were made, and the selected objectives were included in a syllabus for every student on the rotation. Information was provided in the syllabus about where and how each objective should be mastered, for example, in a particular clinical setting, or in a lecture or a textbook, or in supplemental readings.
Before revision, the NBME Pediatric Subject Examination was used to determine students' grades for the rotation. For the curricular revision, the PCE, based on the learning objectives for the clerkship, was used instead of the NBME Subject Examination to determine students' grades.
From May 1995 to May 1997, every student rotating through Pediatrics at UCLA–CHS (n = 52 before revision;n = 42 after revision) completed the PCE and the NBME Pediatric Subject Examination.
Before curricular revision, the mean score for the PCE was 53.7% (±10.1%). The mean score after curricular revision increased to 67.7% (±8.4%) (P < .001). Table 1 presents differences between pre- and postrevision scores for the various topic areas covered by the curriculum. Scores in every topic area except pediatric emergencies increased significantly (P < .01). Topic areas that showed the greatest improvement were fluids and electrolytes, health supervision, behavior, and growth and development.
Before curricular revision, the mean score for the NBME examination was 521 (±122), which corresponds to 72% (±9%) correct, as indicated by the Score Interpretation Guide provided by the NBME. The national average for the NBME examination is 500 (±100). After curricular revision, the mean score for the NBME examination was 520 (±109) (P = .98).
As measured by the PCE before curricular revision, students in the UCLA–CHS core clerkship did not acquire an adequate general pediatric knowledge base. This outcome occurred despite having two thirds of the clerkship based in the ambulatory setting. The poor performance on this examination before reform may reflect several factors: the lack of identified learning objectives for students and faculty, the relative weighting of time in didactic sessions toward subspecialty topics, and a perception by students that the NBME examination did not focus on general pediatric topics.
In contrast, performance on the NBME Pediatric Subject Examination was adequate. This performance does not necessarily contradict the poor performance on our PCE. COMSEP members have suspected that the NBME examination focused on subspecialty pediatrics rather than on general pediatric topics: at the March 1996 Annual COMSEP meeting, clerkship directors met with NBME representatives and reviewed a sample NBME Pediatric Subject Examination. Only 56% of the questions were viewed as “appropriate for junior medical students and based on core competencies contained within the APA–COMSEP curriculum.”6 UCLA students' success on the NBME examination before curricular revision may have reflected a mastery of more subspecialty-oriented content areas.
After curricular revision, students scored significantly higher on the PCE, whereas NBME scores remained unchanged relative to prerevision levels. These findings contrast with those of Potts and Phelan, who found a drop in NBME Pediatric Subject Examination scores after curricular revision.5 The differences seen in the two studies may be attributable to the specific nature of curricular revision at the two institutions. At the University of Illinois (Rockford), curricular change was not based on the APA–COMSEP guidelines; rather, increased emphasis was placed on only six core general pediatric objectives. One possible explanation for why the NBME scores did not fall at UCLA is that students may have performed better on general pediatric questions and less well on subspecialty questions. Unfortunately, additional analysis of the NBME examination results was not possible because of unavailability.
The postrevision PCE average score of 67.7% could be viewed as a disappointing result. The examination, however, was challenging because of the case-based, short-answer essays that were used. In this format, unlike multiple-choice examinations, students cannot guess or be cued to answers.
The potential limitations of the study should be addressed. First, the use of historical controls may be questioned. One could speculate that the improvement in performance on the PCE was caused by factors other than the curricular revision. However, the two cohorts of students studied (class of 1997 and class of 1998) appear academically equivalent, based on GPA and USMLE Parts I and II scores. In addition, there were no major changes in the faculty involved in teaching the clerkship. Finally, there were no modifications of the UCLA School of Medicine curriculum that could have led to the improvement of scores on the PCE.
A second potential criticism is that the PCE did not affect students' clerkship grades before curricular revision but did after; therefore, the improvement in scores may have been attributable to the students' not taking the examination seriously enough in the former group. Before revision, the NBME examination was the method used to assess student knowledge. Students often study what they expect will be on the test. If the PCE had counted before curricular revision, student learning may have been altered, and the results may not have been an accurate assessment of the baseline status. Until clerkship directors are confident that the NBME examination provides an appropriate assessment of mastery of the APA–COMSEP objectives, they may consider using evaluation methods that are more directly tied to the APA–COMSEP objectives.
Finally, the study involved a clerkship at only one institution, and the results therefore may not be generalizable. Curricular revision at most institutions is incremental and ongoing, making multiinstitutional assessment difficult. The goal of the study was to demonstrate that curricular revision based on the APA–COMSEP curriculum can affect student outcomes.
It is important to recognize that the curricular changes described in this study are feasible to implement. The revision built on fundamental educational principles: defining appropriate learning objectives, sharing these learning objectives with faculty and students, designing and implementing didactic series and case discussions that focused on these objectives, and evaluating students with a final written examination that was clearly tied to the learning objectives for the clerkship. Our results demonstrate a significant improvement in general pediatric knowledge acquisition. Of note, the pediatric clerkship at UCLA–CHS was somewhat progressive even before curricular change because students spent 4 of their 6 weeks in an ambulatory setting. Therefore, the same intervention might bring even greater changes in programs with more traditional, inpatient formats. These findings should encourage clerkship directors who are in the midst of or considering curricular change.
We acknowledge Samia Kronfli and Ava Yajima for their assistance with the clerkship, and Brenda Hanning and Dr Janet P. Hafler for their editorial assistance with the preparation of this manuscript.
- APA =
- Ambulatory Pediatric Association •
- COMSEP =
- Council on Medical Student Education in Pediatrics •
- NBME =
- National Board of Medical Examiners •
- PCE =
- Pediatric Clerkship Examination
- Burke W,
- Baron RB,
- Lemon M,
- Losh D,
- Novack A
- ↵Ambulatory Pediatric Association and Council on Medical Student Education in Pediatrics. Core Curriculum in General Pediatrics. APA; 1994
- ↵Berkow R. Evaluation task force. The Pediatric Educator. 1996;3. Newsletter
- Copyright © 1999 American Academy of Pediatrics