Development and Evaluation of a Method for Evaluating Pediatric Residents' Knowledge and Skill in Performing Physical Examinations of the Ankle and Knee
Background. Pediatric residents need the knowledge and physical examination skills to evaluate common musculoskeletal injuries. The ankle and the knee are the 2 most common sites of musculoskeletal injury in young athletes. Methods for evaluating pediatric residents' knowledge and skills in examining the ankle and knee are needed.
Objectives. 1) To describe the development of a method for evaluating pediatric residents' knowledge and skill in performing physical examinations of the ankle and knee, and 2) to report the reliability of this method.
Methods. A written test and a Clinical Skills Assessment Examination (CSAE) with a rating index were developed by the investigators to evaluate pediatric residents' knowledge and skills in examining the ankle and knee. Fifty-eight pediatric residents completed the written test and examined the ankle and knee of one standardized patient at the beginning of a required 1-month adolescent medicine rotation. Forty-eight residents repeated the evaluation at the end of the month. The investigators rated the residents' performance of the CSAE and then assessed interrater reliability using Cronbach's α. Test–retest correlation was calculated to assess the reliability of the written test.
Results. Test–retest correlation for the written test was 0.72, establishing its reliability. Interrater reliability for rating the CSAE of the ankle and knee was 0.98 and 0.90, respectively.
Conclusion. Pediatric residents' knowledge and skills in examining the ankle and knee can be reliably evaluated using the written test and CSAE described in this article. These could be used to assess the effectiveness of current curricula in improving pediatric residents' knowledge and skill in evaluating ankle and knee complaints and to assist in the design of future curricula. musculoskeletal, evaluation methods, resident curriculum.
- OSCE =
- Objective Structured Clinical Examination •
- CSAE =
- Clinical Skills Assessment Examination •
- T0 =
- baseline evaluation •
- T1 =
- immediately after the intervention •
- T2 =
- at the end of the 1-month rotation
Musculoskeletal complaints are common in primary care practice.1 Annually, there are ∼3 million injuries in children and adolescents in the United States requiring time away from sports participation.1 The estimated direct and indirect costs of recreational/sports injuries in <19-year-olds in the United States in 1994 was $22 billion.2 There have been recommendations for more training in the diagnosis and treatment of musculoskeletal injuries during medical school and residency.3–9 Physical examination skills are the cornerstone of making diagnoses of musculoskeletal injuries.
The Objective Structured Clinical Examination (OSCE) is a method that has been used to evaluate residents' competency in assessing clinical problems.10 The OSCE format has been rated highly by pediatric residents and medical students; however, an OSCE involving pediatric residents examining the musculoskeletal system has not been published.11 McGaghie et al12 reported methods in which rheumatologists designed detailed checklists for an ideal performance of the physical examination of the knee (57 items), shoulder (87 items), back (73 items), and general musculoskeletal system (152 items) by medical students. The amount of detail, in terms of the number of items required for each joint examination, limits their use as practical evaluation tools. Lawry et al13reported that teaching a 3-minute screening musculoskeletal examination to first-year medical students increased skill in performing the examination at follow up 3 and 16 months later. A checklist was used to evaluate the physical examination skills (43 items). The Society of Teachers of Family Medicine published a curriculum guide for teaching sports medicine to family practice residents suggesting that the evaluation of the residents' ability to perform a focused musculoskeletal examination include direct observation complemented by checklists with component skills.14 A checklist was not provided and it does not seem that one was developed; rather, a reference was given under the evaluation section of physical examination skills in the bibliography. That reference was used to develop the checklists in this project.15 Checklists were not published in the studies by McGaghie et al and Lawry et al, although we reviewed the checklist in the former study sent to us by the author. Methods for evaluating pediatric residents' knowledge and skill in performing musculoskeletal examinations, with reliability established psychometrically, are needed.3 We are not aware of any such methods that have been tested and found to be reliable in the context of a pediatric residency training program. The ankle and knee are the 2 most common sites of injury in adolescent athletes.16
Purpose of the Study
The purposes of this study were: 1) to describe the development of a method to assess pediatric residents' knowledge and skill in performing ankle and knee physical examinations, and 2) to describe the reliability of this method. This study was conducted within a larger study evaluating the effectiveness of a teaching intervention in improving residents' knowledge and skill in performing ankle and knee physical examinations. The results of the teaching intervention are reported in an accompanying article.
Between September 1998 and December 1999, there were 60 second-year pediatric and medicine–pediatric residents assigned to a 1-month adolescent medicine rotation, under the supervision of one of the investigators who is board certified in adolescent medicine and sports medicine. The focus of the rotation is adolescent medicine; however, residents see sports medicine patients in the sports medicine clinic of the investigators, which is held in conjunction with the adolescent clinic. Fifty-eight residents participated in the evaluation of a written test and a Clinical Skills Assessment Examination (CSAE), discussed below, at baseline and 48 participated at the end of their rotation. All but 2 of the 10 not available at 1 month were on vacation at the end of their rotation.
Development of the Evaluation Method
The evaluation method tested in this project included a written test and a CSAE.
A 20-question, multiple-choice written test was developed by the investigators, 2 of whom are pediatricians with board certification in Sports Medicine, a third is a medical educator, and the fourth is an athletic trainer. Ten ankle questions (the ankle component of the written test) and 10 knee questions (the knee component of the written test) were developed. The content validity was based on techniques described in standard textbooks and the investigators' consensus of the key elements of the physical examination of the ankle and knee.15,17 The test was given to a pilot group of 4 second-year pediatric residents. Poor questions were identified by item analysis and rewritten. The revised test was given to a second pilot group of 5 second-year pediatric residents. After item analysis, the written test was finalized; the final instrument is included as “Appendix 1.”
The residents' performance of physical examinations of the ankle and knee were assessed using CSAE checklists developed by the investigators (see “Appendix 2”). The checklists were developed based on information in standard textbooks and were modifications of the physical examination forms used in the investigators' sports medicine clinic.16,17 Development of the checklists was accompanied by the development of written indices, which included criteria for judging whether correct techniques in performing the ankle and knee examinations were demonstrated (see “Appendix 3”). The ankle CSAE checklist had a maximum score of 34 from 29 items and the knee CSAE checklist had a maximum score of 31 from 31 items. Some items had a potential score of 2, because of their importance in the examination, whereas some pairs of items had a 1-point maximum.
The 3 raters for the CSAE were the 2 sports medicine physicians and a certified athletic trainer who also served as the study coordinator and the standardized patient. The certified athletic trainer trained with the 2 physicians before the pilot phase of the study and after each pilot group to standardize her performance as the standardized patient. The CSAE checklists and indices were evaluated in the same 2 groups of residents used to pilot test the written test. After each pilot group, the 3 raters compared CSAE ratings of each resident for each item on the checklists. The raters discussed their reasoning when discrepancies among raters existed during the pilot groups, and based on these discussions, final criteria for correct performance of the physical examination techniques were established. Written notes were added to the checklists to cue the raters to the correct criteria to improve rater consistency.
Administration of the Evaluation Tools
The baseline evaluation (T0) occurred just before the teaching intervention, which was within the first 2 days of the 1-month adolescent medicine rotation. The residents were given an identification number and that number, not their name, was recorded on the checklist evaluation sheet and written test. The residents completed the written test and the CSAE. The written test was administered 2 more times: 1) immediately after the intervention (T1) and 2) at the end of the 1-month rotation (T2). The questions for the written tests at each time were identical; however, the order of the questions was changed between T0 and T1. The T1 and T2 written tests were identical and the results of these 2 tests were used to evaluate test–retest reliability.
The CSAEs were conducted in an examination room in the sports medicine clinic of the investigators at T0 and T2. There was no CSAE performed at T1, because this was a time for the teaching intervention, not evaluation. The investigators explained to the residents that the residents' examination of the standardized patient was the first step in the teaching intervention and that the evaluation method and teaching intervention were being tested, not the residents. Each resident was introduced to the standardized patient and short patient scenarios were presented (see “Appendix 2”). The residents were asked to demonstrate physical examination techniques during the CSAE and not to collect additional history or to make a diagnosis. During the CSAE, the resident was encouraged to talk aloud, explaining what they were doing. All CSAEs were audio recorded. In rating each resident's performance for each CSAE item, the standardized patient combined the scores and formulated one composite score. The standardized patient listened to the audiotape after the CSAE to settle discrepancies between raters' ratings, especially for physical examination techniques performed by inspection. For instance, if one rater indicated that a resident inspected for swelling and discoloration of the ankle and the others did not, the rating was done in favor of what was said or not said on the audiotape. The audiotape was reviewed and the CSAE evaluations of the 3 raters were pooled as soon as possible after the CSAE, usually within 24 hours.
The 3 raters recorded their CSAE ratings separately and did not compare rating scores. It was these separate CSAE ratings, recorded before a composite score was formulated, that were used to calculate the interrater reliability for the CSAE ratings. After the raters recorded their CSAE scores for each resident, the study coordinator pooled and recorded the ratings. If ≥2 raters concurred in their rating of the resident's performance of an item on the CSAE, then the resident was given credit for that item, unless there was a discrepancy among raters about an inspection technique. These discrepancies were resolved by reviewing the audiotape as discussed in the previous paragraph. The audiotape was reviewed for all CSAEs, regardless of the number of raters. If 2 raters rated the residents' CSAE and they disagreed about whether the resident had performed an item correctly, then an average of the 2 scores for that item was recorded.
After the CSAEs, the teaching intervention began. The teaching intervention included watching a videotape demonstrating the faculty performing the ankle and knee examinations. The video was followed by a skills-based session in which the residents observed one of the physicians demonstrating the techniques on the standardized patient in an examination room, followed by correct demonstration of the techniques on the standardized patient by the resident under the supervision of 1 of the 3 raters. The intervention is discussed in more detail in the accompanying article.
Data from the 2 pilot groups were excluded from analysis. Test–retest reliability for equivalent tests was assessed by calculating the simple correlation between the written test scores at T1 and T2.18Cronbach's α was used to assess interrater reliability for the CSAE at T0 and T2. The raters recorded whether each item was performed correctly by the resident.
Test–Retest Reliability of the Ankle and Knee Components of the Written Test
The test–retest correlation for the ankle component of the written test was 0.62 (P < .001). The test–retest correlation for the knee component of the written test was 0.72 (P < .001). For the overall written test, combining the ankle and knee components, the test–retest correlation was 0.72 (P < .001).
Interrater reliability among the 3 raters for the CSAE ratings for the ankle and knee was α = 0.98 and 0.90, respectively, at T0. At T2, the interrater reliability among the 3 raters for the CSAE ratings for the ankle and knee was 0.99 and 0.84, respectively.
The purposes of the study, to describe the development of a method to assess pediatric residents' knowledge and skill in performing ankle and knee physical examinations and to describe the reliability of this method, were accomplished. We are not aware of another method with published reliability data in the context of a pediatric residency training program. The reliability of the written test was high, especially considering that the test and retest were conducted 1 month apart. The correlation coefficient reported here may have been attenuated by many factors that could effect residents' acquisition or retention of information during that month. Thus, the reliability reported here may be an underestimate. The approach used in this study, starting with standard techniques, pilot testing, and then revising the evaluation method twice before implementation, resulted in excellent interrater reliability for the CSAE. McGaghie et al12reported a weighted κ-value of 0.72 for interrater reliability in rating a screening musculoskeletal CSAE. Our CSAE was less detailed than that of McGaghie et al and may be more practical for use in pediatric residency training programs. The next step will be to test this method in other residency programs. This will be facilitated through the use of the teaching videotape that demonstrates the techniques assessed in the CSAE and was developed as part of the teaching intervention. This videotape could fill the gap between written descriptions of technique and practical demonstration of techniques. The videotape and skills-based teaching session are described in the accompanying article. A concern may be that because the 3 raters in this project worked together closely in the pilot phase of the study and because the resultant CSAE interrater reliability was so high, it will be difficult to replicate these methods elsewhere. We anticipated this concern and in response developed the detailed CSAE Indices (“Appendix 3”), which describe correct performance of the individual maneuvers. These indices, coupled with the teaching videotape described in the companion manuscript, will facilitate implementation in other settings. The authors will be available for consultation as well.
The written test and CSAE could evaluate residents' performance in these areas and identify areas for curriculum improvement. Residency directors and the residency review committee of the Accreditation Council on Graduate Medical Education could consider requiring the demonstration of these skills as a requirement during residency, similar to the requirement of performing a pelvic examination.19 The presence of a proven evaluation method may be an incentive for residency programs to make teaching musculoskeletal examination techniques a higher priority. Identifying residents' needs through the method described in this article could be an incentive to increase resources for teaching about the diagnosis of musculoskeletal injuries.
Inspection skills should be easy to evaluate if the residents are prompted to talk aloud during their examination of the patient. The audiotape was helpful as an aid in assessing inspection skills but not performance of other skills because some residents verbalized that they were performing a particular examination technique, yet were observed by the raters to be doing so incorrectly. Another potential source of rater disagreement is inherent in the standardized patient being one of the raters, having the added benefit of proprioceptive input into whether the resident performed the examination correctly. This was not an issue in this study.
Although residents were told that they did not have to diagnose the standardized patient's ankle or knee problem and the emphasis was on the physical examination skills, several residents focused their initial examination, after the patient scenarios were read, on making a diagnosis by attempting to obtain additional history rather than performing the physical examination. This may also have been an attempt to organize their thoughts about conducting the examination, which is understandable. A reminder at the beginning of the session focused the resident's attention on the examination.
Pediatric residents have the need to improve their skills in performing physical examinations of the ankle and knee. This article describes a reliable method to evaluate pediatric residents' knowledge and skills in performing these examinations. There is no other published method that we are aware of that has been tested in this manner and in the context of a pediatric residency program. The next steps will be to test these evaluation methods in other primary care residency programs, with primary care physicians in practice, and to extend these methods to assess pediatric residents' knowledge and skill in performing physical examinations of the remainder of the musculoskeletal system.
- Received August 3, 2000.
- Accepted November 1, 2000.
Amy C. Fetterhoff is currently at Wyle Laboratories, Houston, Texas.
Reprint request to (A.C.H.) Adolescents Medicine and Sports Medicine Section, Baylor College of Medicine, Texas Children's Hospital, 6621 Fannin St MC3-3340, Houston, TX 77030-2399. E-mail:
- Requa RK. The scope of the problem: the impact of sports-related injuries. In: Conference on Sports Injuries in Youth: Surveillance Strategies. Bethesda, MD: National Institutes of Health; 1992. National Institutes of Health Publication No. 93-3444
- Danseco ER, Miller TR, Spicer RS. Incidence and costs of 1987–1994 childhood injuries: demographic breakdowns. Pediatrics. 2000;105(2). URL: http://www.pediatrics.org/cgi/content/full/105/2/e27
- Centers for Disease Control and Prevention. Injury Control in the 1990s: A National Plan for Action. Atlanta, GA: National Center for Injury Prevention and Control; 1993
- Harden RG,
- Stevenson M,
- Downie WW,
- Wilson GM
- Society of Teachers of Family Medicine. Sports Medicine: Curriculum Guide for Family Practice Residents. Kansas City, MO: The Society of Teachers and Family Medicine; 1990. ISBN No. 0-942295-21-8
- Hoppenfield S. Physical Examination of the Spine and Extremities. Norwalk, CT: Appleton-Century-Crofts; 1976
- Garrick JG,
- Requa RK
- Starkey C, Ryan J. Evaluation of Orthopedic and Athletic Injuries. Philadelphia, PA: F. A Davis Company; 1996
- Gronlund NE. Assessment of Student Achievement. Boston, MA: Allyn and Bacon; 1998
- American Medical Association. Graduate Medical Education Directory: 2001. Chicago, IL: American Medical Association; 2000
- Copyright © 2001 American Academy of Pediatrics