 |
|
 |
|
 |
 |
|
 |
 |
 |
 |
 |
 |
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
PEDIATRICS Vol. 107 No. 4 April 2001, p. e51
,
From the * Department of Pediatrics, Adolescent Medicine
and Sports Medicine Section; the Department of Medicine, Baylor
College of Medicine; and the § University of Texas Houston Health
Science Center School of Public Health, Houston, Texas.
 |
ABSTRACT |
|---|
Top
Abstract
Methods
Results
Discussion
Conclusion
References
|
|---|
Background. Pediatric residents need the knowledge and physical examination skills to evaluate common musculoskeletal injuries. The ankle and the knee are the 2 most common sites of musculoskeletal injury in young athletes. Methods for evaluating pediatric residents' knowledge and skills in examining the ankle and knee are needed.
Objectives. 1) To describe the development of a method for evaluating pediatric residents' knowledge and skill in performing physical examinations of the ankle and knee, and 2) to report the reliability of this method.
Methods. A written test and a Clinical Skills Assessment
Examination (CSAE) with a rating index were developed by the
investigators to evaluate pediatric residents' knowledge and skills in
examining the ankle and knee. Fifty-eight pediatric residents completed the written test and examined the ankle and knee of one standardized patient at the beginning of a required 1-month adolescent medicine rotation. Forty-eight residents repeated the evaluation at the end of
the month. The investigators rated the residents' performance of the
CSAE and then assessed interrater reliability using Cronbach's 
.
Test-retest correlation was calculated to assess the reliability of
the written test.
Results. Test-retest correlation for the written test was 0.72, establishing its reliability. Interrater reliability for rating the CSAE of the ankle and knee was 0.98 and 0.90, respectively.
Conclusion. Pediatric residents' knowledge and skills in examining the ankle and knee can be reliably evaluated using the written test and CSAE described in this article. These could be used to assess the effectiveness of current curricula in improving pediatric residents' knowledge and skill in evaluating ankle and knee complaints and to assist in the design of future curricula. musculoskeletal, evaluation methods, resident curriculum.
Musculoskeletal complaints are common in primary care
practice.1 Annually, there are ~3 million injuries in
children and adolescents in the United States requiring time away from
sports participation.1 The estimated direct and indirect
costs of recreational/sports injuries in <19-year-olds in the United
States in 1994 was $22 billion.2 There have been
recommendations for more training in the diagnosis and treatment of
musculoskeletal injuries during medical school and
residency.3-9 Physical examination skills are the
cornerstone of making diagnoses of musculoskeletal injuries.
The Objective Structured Clinical Examination (OSCE) is a method that
has been used to evaluate residents' competency in assessing clinical
problems.10 The OSCE format has been rated highly by
pediatric residents and medical students; however, an OSCE involving
pediatric residents examining the musculoskeletal system has not been
published.11 McGaghie et al12 reported
methods in which rheumatologists designed detailed checklists for an
ideal performance of the physical examination of the knee (57 items),
shoulder (87 items), back (73 items), and general musculoskeletal
system (152 items) by medical students. The amount of detail, in terms
of the number of items required for each joint examination, limits
their use as practical evaluation tools. Lawry et al13
reported that teaching a 3-minute screening musculoskeletal examination to first-year medical students increased skill in performing the examination at follow up 3 and 16 months later. A checklist was used to
evaluate the physical examination skills (43 items). The Society of
Teachers of Family Medicine published a curriculum guide for teaching
sports medicine to family practice residents suggesting that the
evaluation of the residents' ability to perform a focused
musculoskeletal examination include direct observation complemented by
checklists with component skills.14 A checklist was not
provided and it does not seem that one was developed; rather, a
reference was given under the evaluation section of physical
examination skills in the bibliography. That reference was used to
develop the checklists in this project.15 Checklists were
not published in the studies by McGaghie et al and Lawry et al,
although we reviewed the checklist in the former study sent to us by
the author. Methods for evaluating pediatric residents' knowledge and
skill in performing musculoskeletal examinations, with reliability
established psychometrically, are needed.3 We are not
aware of any such methods that have been tested and found to be
reliable in the context of a pediatric residency training program. The
ankle and knee are the 2 most common sites of injury in adolescent
athletes.16
Purpose of the Study
The purposes of this study were: 1) to describe the development of
a method to assess pediatric residents' knowledge and skill in
performing ankle and knee physical examinations, and 2) to describe the
reliability of this method. This study was conducted within a larger
study evaluating the effectiveness of a teaching intervention in
improving residents' knowledge and skill in performing ankle and knee
physical examinations. The results of the teaching intervention are
reported in an accompanying article.
Study Participants
Between September 1998 and December 1999, there were 60 second-year pediatric and medicine-pediatric residents assigned to a
1-month adolescent medicine rotation, under the supervision of one of
the investigators who is board certified in adolescent medicine and
sports medicine. The focus of the rotation is adolescent medicine;
however, residents see sports medicine patients in the sports medicine
clinic of the investigators, which is held in conjunction with the
adolescent clinic. Fifty-eight residents participated in the evaluation
of a written test and a Clinical Skills Assessment Examination (CSAE),
discussed below, at baseline and 48 participated at the end of their
rotation. All but 2 of the 10 not available at 1 month were on vacation
at the end of their rotation.
Development of the Evaluation Method
The evaluation method tested in this project included a written
test and a CSAE.
Knowledge Assessment
A 20-question, multiple-choice written test was developed by the
investigators, 2 of whom are pediatricians with board certification in
Sports Medicine, a third is a medical educator, and the fourth is an
athletic trainer. Ten ankle questions (the ankle component of the
written test) and 10 knee questions (the knee component of the written
test) were developed. The content validity was based on techniques
described in standard textbooks and the investigators' consensus of
the key elements of the physical examination of the ankle and
knee.15,17 The test was given to a pilot group of 4 second-year pediatric residents. Poor questions were identified by item
analysis and rewritten. The revised test was given to a second pilot
group of 5 second-year pediatric residents. After item analysis, the written test was finalized; the final instrument is included as "Appendix 1."
CSAE
The residents' performance of physical examinations of the
ankle and knee were assessed using CSAE checklists developed by the
investigators (see "Appendix 2"). The checklists were developed based on information in standard textbooks and were modifications of
the physical examination forms used in the investigators' sports medicine clinic.16,17 Development of the checklists was
accompanied by the development of written indices, which included
criteria for judging whether correct techniques in performing the ankle
and knee examinations were demonstrated (see "Appendix 3"). The
ankle CSAE checklist had a maximum score of 34 from 29 items and the
knee CSAE checklist had a maximum score of 31 from 31 items. Some items
had a potential score of 2, because of their importance in the
examination, whereas some pairs of items had a 1-point maximum.
METHODS
Top
Abstract
Methods
Results
Discussion
Conclusion
References
Administration of the Evaluation Tools
Baseline Evaluations The baseline evaluation (T0) occurred just before the teaching intervention, which was within the first 2 days of the 1-month adolescent medicine rotation. The residents were given an identification number and that number, not their name, was recorded on the checklist evaluation sheet and written test. The residents completed the written test and the CSAE. The written test was administered 2 more times: 1) immediately after the intervention (T1) and 2) at the end of the 1-month rotation (T2). The questions for the written tests at each time were identical; however, the order of the questions was changed between T0 and T1. The T1 and T2 written tests were identical and the results of these 2 tests were used to evaluate test-retest reliability.
The CSAEs were conducted in an examination room in the sports medicine clinic of the investigators at T0 and T2. There was no CSAE performed at T1, because this was a time for the teaching intervention, not evaluation. The investigators explained to the residents that the residents' examination of the standardized patient was the first step in the teaching intervention and that the evaluation method and teaching intervention were being tested, not the residents. Each resident was introduced to the standardized patient and short patient scenarios were presented (see "Appendix 2"). The residents were asked to demonstrate physical examination techniques during the CSAE and not to collect additional history or to make a diagnosis. During the CSAE, the resident was encouraged to talk aloud, explaining what they were doing. All CSAEs were audio recorded. In rating each resident's performance for each CSAE item, the standardized patient combined the scores and formulated one composite score. The standardized patient listened to the audiotape after the CSAE to settle discrepancies between raters' ratings, especially for physical examination techniques performed by inspection. For instance, if one rater indicated that a resident inspected for swelling and discoloration of the ankle and the others did not, the rating was done in favor of what was said or not said on the audiotape. The audiotape was reviewed and the CSAE evaluations of the 3 raters were pooled as soon as possible after the CSAE, usually within 24 hours. The 3 raters recorded their CSAE ratings separately and did not compare rating scores. It was these separate CSAE ratings, recorded before a composite score was formulated, that were used to calculate the interrater reliability for the CSAE ratings. After the raters recorded their CSAE scores for each resident, the study coordinator pooled and recorded the ratings. If
2 raters concurred in their rating of the
resident's performance of an item on the CSAE, then the resident was
given credit for that item, unless there was a discrepancy among raters
about an inspection technique. These discrepancies were resolved by
reviewing the audiotape as discussed in the previous paragraph. The
audiotape was reviewed for all CSAEs, regardless of the number of
raters. If 2 raters rated the residents' CSAE and they disagreed about whether the resident had performed an item correctly, then an average
of the 2 scores for that item was recorded.
After the CSAEs, the teaching intervention began. The teaching
intervention included watching a videotape demonstrating the faculty
performing the ankle and knee examinations. The video was followed by a
skills-based session in which the residents observed one of the
physicians demonstrating the techniques on the standardized patient in
an examination room, followed by correct demonstration of the
techniques on the standardized patient by the resident under the
supervision of 1 of the 3 raters. The intervention is discussed in more
detail in the accompanying article.
Statistical Analyses
Data from the 2 pilot groups were excluded from analysis.
Test-retest reliability for equivalent tests was assessed by
calculating the simple correlation between the written test scores at
T1 and T2.18
Cronbach's was used to assess interrater reliability for the CSAE
at T0 and T2. The raters
recorded whether each item was performed correctly by the resident.
| |
RESULTS |
|---|
|
Top
Abstract
Methods
Results
Discussion
Conclusion
References
|
|---|
Test-Retest Reliability of the Ankle and Knee Components of the Written Test
The test-retest correlation for the ankle component of the written test was 0.62 (P < .001). The test-retest correlation for the knee component of the written test was 0.72 (P < .001). For the overall written test, combining the ankle and knee components, the test-retest correlation was 0.72 (P < .001).
CSAE
Interrater reliability among the 3 raters for the CSAE ratings for
the ankle and knee was = 0.98 and 0.90, respectively, at
T0. At T2, the interrater
reliability among the 3 raters for the CSAE ratings for the ankle and
knee was 0.99 and 0.84, respectively.
| |
DISCUSSION |
|---|
|
Top
Abstract
Methods
Results
Discussion
Conclusion
References
|
|---|
The purposes of the study, to describe the development of a method
to assess pediatric residents' knowledge and skill in performing ankle
and knee physical examinations and to describe the reliability of this
method, were accomplished. We are not aware of another method with
published reliability data in the context of a pediatric residency
training program. The reliability of the written test was high,
especially considering that the test and retest were conducted 1 month
apart. The correlation coefficient reported here may have been
attenuated by many factors that could effect residents' acquisition or
retention of information during that month. Thus, the reliability
reported here may be an underestimate. The approach used in this study,
starting with standard techniques, pilot testing, and then revising the
evaluation method twice before implementation, resulted in excellent
interrater reliability for the CSAE. McGaghie et al12
reported a weighted 
-value of 0.72 for interrater reliability in
rating a screening musculoskeletal CSAE. Our CSAE was less detailed
than that of McGaghie et al and may be more practical for use in
pediatric residency training programs. The next step will be to test
this method in other residency programs. This will be facilitated
through the use of the teaching videotape that demonstrates the
techniques assessed in the CSAE and was developed as part of the
teaching intervention. This videotape could fill the gap between
written descriptions of technique and practical demonstration of
techniques. The videotape and skills-based teaching session are
described in the accompanying article. A concern may be that because
the 3 raters in this project worked together closely in the pilot phase
of the study and because the resultant CSAE interrater reliability was
so high, it will be difficult to replicate these methods elsewhere. We
anticipated this concern and in response developed the detailed CSAE
Indices ("Appendix 3"), which describe correct performance of the
individual maneuvers. These indices, coupled with the teaching
videotape described in the companion manuscript, will facilitate
implementation in other settings. The authors will be available for
consultation as well.
The written test and CSAE could evaluate residents' performance in these areas and identify areas for curriculum improvement. Residency directors and the residency review committee of the Accreditation Council on Graduate Medical Education could consider requiring the demonstration of these skills as a requirement during residency, similar to the requirement of performing a pelvic examination.19 The presence of a proven evaluation method may be an incentive for residency programs to make teaching musculoskeletal examination techniques a higher priority. Identifying residents' needs through the method described in this article could be an incentive to increase resources for teaching about the diagnosis of musculoskeletal injuries.
Measurement Issues
Inspection skills should be easy to evaluate if the residents are prompted to talk aloud during their examination of the patient. The audiotape was helpful as an aid in assessing inspection skills but not performance of other skills because some residents verbalized that they were performing a particular examination technique, yet were observed by the raters to be doing so incorrectly. Another potential source of rater disagreement is inherent in the standardized patient being one of the raters, having the added benefit of proprioceptive input into whether the resident performed the examination correctly. This was not an issue in this study.
Although residents were told that they did not have to diagnose the standardized patient's ankle or knee problem and the emphasis was on the physical examination skills, several residents focused their initial examination, after the patient scenarios were read, on making a diagnosis by attempting to obtain additional history rather than performing the physical examination. This may also have been an attempt to organize their thoughts about conducting the examination, which is understandable. A reminder at the beginning of the session focused the resident's attention on the examination.
| |
CONCLUSION |
|---|
|
Top
Abstract
Methods
Results
Discussion
Conclusion
References
|
|---|
Pediatric residents have the need to improve their skills in performing physical examinations of the ankle and knee. This article describes a reliable method to evaluate pediatric residents' knowledge and skills in performing these examinations. There is no other published method that we are aware of that has been tested in this manner and in the context of a pediatric residency program. The next steps will be to test these evaluation methods in other primary care residency programs, with primary care physicians in practice, and to extend these methods to assess pediatric residents' knowledge and skill in performing physical examinations of the remainder of the musculoskeletal system.
| |
FOOTNOTES |
|---|
Amy C. Fetterhoff is currently at Wyle Laboratories, Houston, Texas.
Reprint request to (A.C.H.) Adolescents Medicine and Sports Medicine Section, Baylor College of Medicine, Texas Children's Hospital, 6621 Fannin St MC3-3340, Houston, TX 77030-2399. E-mail: alberth{at}bcm.tmc.edu
Received for publication Aug 3, 2000; accepted Nov 1, 2000.
| |
ABBREVIATIONS |
|---|
OSCE, Objective Structured Clinical Examination; CSAE, Clinical Skills Assessment Examination; T0, baseline evaluation; T1, immediately after the intervention; T2, at the end of the 1-month rotation.
| |
APPENDIX 1 |
|---|
| |
APPENDIX 2 |
|---|
| |
APPENDIX 3 |
|---|
| |
REFERENCES |
|---|
|
Top
Abstract
Methods
Results
Discussion
Conclusion
References
|
|---|
This article has been cited by other articles:
 |
|
 |
|
  P. Weisberg, K. S Scanlon, R. Li, and M. E Cogswell Nutritional rickets among children in the United States: review of cases reported between 1986 and 2003 Am. J. Clinical Nutrition, December 1, 2004; 80(6): 1697S - 1705S. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||
|
||
Home | My Pediatrics | Journal Information | Current Issue | Past Issues | Subscriptions & Services | Contact Us Click here for faster international access © 2001 American Academy of Pediatrics. All rights reserved. |
||
|
||
