Objective. To assess the knowledge and skills of residents in health maintenance and to measure the effectiveness of a new curriculum.
Design. Longitudinal cohort study of first- and second-year pediatric residents over 2 academic years (AY). Residents in AY 1995–1996 (n = 32) comprised a control group. Residents in AY 1996–1997 (n = 36) served as a study group who completed a structured 12-week program of reading materials, small group discussions, and case scenarios.
Methods. A needs assessment identified 7 topic areas in health maintenance: health screening, immunizations, nutrition, development, behavior, injury prevention, and dental health. Control and study residents' cognitive knowledge, chart documentation, and clinical performance were assessed using a 50-item multiple choice examination, medical records audit, and standardized patient evaluations.
Results. Mean examination scores, chart documentation, and clinical performance did not differ significantly between first- and second-year control residents. In the study group, mean examination scores of first-year residents were not significantly better than controls. Second-year residents significantly improved their knowledge about health screening, nutrition, and dental health compared with controls but showed no improvement in other areas. Medical record documentation increased significantly among study residents compared with control residents. Clinically, first-year residents significantly improved in the task of health screening (relative risk [RR] 6.33) and markedly improved (36.8% vs 66.7%) their injury prevention counseling compared with controls. Second-year residents showed significant increases in health screening (RR 6.09) and taking a nutritional history (RR 1.44) compared with controls. Assessment of the curriculum by residents showed a high degree of satisfaction with the program.
Conclusions. Pediatric residents who had clinical experience and completed the curriculum demonstrated significant improvements in their medical record documentation and selective gains in knowledge and clinical performance of health maintenance compared with residents with clinical experience only. A structured curriculum in health maintenance and a multifaceted assessment system can identify and enhance the skills of pediatric residents.
Pediatric health maintenance visits comprise a significant part of general pediatric practice time. Surveys of pediatricians in the United States have estimated that 33% to 65% of all office visits were for well-child care.1–3 The content of these visits usually consists of a physical examination and a discussion about nutrition, developmental status, behavior problems, family history, immunizations, and anticipatory guidance.4,,5 The American Academy of Pediatrics has affirmed the importance of these visits to the health of children in several policy statements.6,,7 In addition, program requirements for residency education in pediatrics require instruction in health promotion and disease prevention as part of any accredited training program.8 The Ambulatory Pediatric Association's Educational Guidelines for Residency Training recommends that training programs establish curricula to allow residents to become proficient in the core concepts of health maintenance. These include anticipatory guidance, developmental surveillance, behavioral issues, immunization practices, and health promotion.9Recently published clinical guidelines for health supervision have emphasized the need to incorporate these core concepts into practice.10 However, surveys of graduates from residency programs and program directors in the United States reported that training in health supervision and well-child care was deficient in quality and quantity compared with other experiences during residency.11,,12
Given the importance of health maintenance visits in pediatric practice and the need to train residents in this subject, we embarked on a program to develop specific curricula in this area. Designing effective instruction in health maintenance should incorporate strategies of adult learning that have been advocated in medical education.13,,14 These strategies include establishing the needs of the learners, developing curricula to meet these needs, and designing evaluation methods that measure instructional effectiveness. Although a number of studies of curricula in preventive health care have been conducted, little data have been published on curricula for pediatric health maintenance.15,,16 Furthermore, few studies have addressed the outcome of a curriculum in well-child care that measured variables such as knowledge, interviewing skills, and written documentation.
The purpose of this study was to assess pediatric residents' learning needs in the area of health maintenance and to design, implement, and evaluate a new curriculum to be used by primary care providers.
MATERIALS AND METHODS
Participants in this study included alumni and residents of the pediatric training program of the University of Texas Health Science Center San Antonio (UTHSCSA). This program trains approximately 50 to 60 physicians per year in all aspects of pediatric care and has been accredited by the Accreditation Council for Graduate Medical Education since its inception in 1971. The program's curriculum included a series of didactic lectures for first-year residents on well-child examinations, nutrition, immunizations, behavior, and development totaling about 30 hours of instruction. After training is completed, most graduates practice general pediatrics in communities across Texas.
All 32 pediatric residents in academic year (AY) 1995–1996 (19 first-year and 13 second-year) and 36 residents from AY 1996–1997 (19 first-year and 17 second-year) were eligible to participate in this research project. Except for 2 first-year residents from AY 1995–1996 who withdrew from the program, all eligible residents completed the study. First- and second-year residents from AY 1995–1996 served as a control group to collect baseline data. First- and second-year residents from AY 1996–1997 (which included all but 2 first-year residents from the previous AY) comprised the study group. Demographically, control and study residents were not significantly different in mean age, male/female ratio, number of own children, or mean in-training examination scores.
As a prelude to this study, a survey of learning needs of third-year residents and recent graduates from our program in the area of health maintenance was conducted. Seven topics in pediatric health maintenance were chosen for this survey. Topics were selected from criteria developed by a task force of the Maternal and Child Health Bureau, US Department of Health and Human Services10 and were used throughout the project to develop our needs assessment survey, resident performance tools, and the outline of our curriculum. The 7 topics were:
1. General health screening
3. Childhood nutrition
4. Behavioral issues
5. Developmental surveillance
6. Injury prevention
7. Dental health
We developed a questionnaire that asked about learning issues and training deficits in 33 specific tasks associated with health maintenance. Using a 6-point Likert scale, participants rated on a scale of 1 (very little) to 6 (very well) the degree to which their residency training program prepared them to deal with these issues. The questionnaire was distributed to graduates from the last 3 years of the UTHSCSA pediatric residency program whose primary activity was general pediatrics. Third-year UTHSCSA pediatric residents from AY 1995–1996 completed the same questionnaire. Results from this needs assessment were incorporated into the content of a new curriculum in health maintenance.
We defined goals and objectives for a health maintenance curriculum using data from the needs assessment survey. The curriculum was divided into modules corresponding to the 7 topic areas described previously. Educational goals and objectives for each module were based on those recommended by the Ambulatory Pediatric Association'sEducational Guidelines For Residency Training in General Pediatrics.9
Each of the 7 modules had a similar design. A title page listed the goals and objectives for the module, followed by a list of specific reading assignments from the pediatric literature. A self-assessment quiz was included after the readings as a way of emphasizing important points. Finally, the module included a case scenario that presented a clinical problem to be solved. Learners completed the readings and the quiz before meeting with a faculty member in small groups before the start of their weekly continuity clinics. The purpose of the groups was to discuss questions related to the topic and the readings. The group could then attempt to solve the clinical problem presented in the case scenario using the principles just learned. Memory aids consisting of pocket cards and handouts were designed for several of the modules. A videotape of a health maintenance visit produced by one of us (J.O.L.) was available to aid in demonstrating health maintenance interviewing techniques. At the conclusion of these group meetings, learners could practice their health maintenance skills with several patients in their continuity clinic. Nine faculty members from the department of pediatrics facilitated the curriculum. Faculty development sessions were held to familiarize the staff with how to moderate a small group session and to review the curriculum content before distribution to the residents. The standardized teaching format for each module (reading, quiz, and case scenario) helped to standardized the teaching strategy among continuity clinic groups.
Tools to Assess Residents' Skills
A. Multiple Choice Examination
A 50-question multiple choice test was constructed using questions from the last 5 years of the American Academy of Pediatrics' Review and Education Program. One best-answer questions corresponding to the 7 topic areas in this study were chosen for inclusion. One validity check included having the examination reviewed by the pediatric faculty at UTHSCSA to obtain an equal mix of factual recall and higher order thinking questions. The test was administered to all residents in the control group in January 1996. An identical examination was administered to study residents 1 year later (February 1997) so that measurements from both groups were taken at the same point in the AY. Kuder Richardson 20 reliability coefficients (a measure of internal consistency that can vary between 0 and 1.0) for these examinations were 0.41 and 0.53, respectively.
B. Medical Records Review
The general pediatric faculty routinely completed review forms on medical records from the clinics where residents saw patients. The review form consisted of a checklist containing specific items assessed in the record and an overall rating of its contents. Items in the checklist that corresponded to the 7 topic areas were identified for auditing and to measure the documentation of health maintenance tasks. Only charts that contained a health maintenance assessment as part of the visit were examined. A data manager abstracted audit items from a random selection of faculty record review forms from control residents during the period of January to March 1996 and from study residents during November 1996 to February 1997. Residents were aware that medical record audits were part of this study, however, control residents did not receive results of audits. Audit data were reported as the percentage of charts that contained documentation of items in the checklist.
C. Standardized Patient Mothers
We used standardized patient mothers to assess the clinical performance of residents during health maintenance visits. Three mothers with children <5 years old were recruited from the community to be trained as standardized patient mothers. The mothers were paid for their participation. Training consisted of a faculty member who role played a resident performing a health maintenance assessment in the presence of the volunteer mother. Another faculty member observed the interaction between the volunteer mother and the physician. A checklist of health maintenance tasks was constructed that corresponded to the 7 topic areas in the curriculum. This checklist was completed by the volunteer mothers, the physician interviewer, and the observer at the end of the role play visit. The training ended when all 3 participants had identical items recorded on the checklist. These standardized mothers used this checklist to assess the performance of residents during a health maintenance visit with their own children. Control residents were evaluated by 1 of the 3 standardized mothers in March 1996. Study residents were evaluated 1 year later, in February 1997. Residents had previously signed consent forms to allow evaluation of their performance by standardized patients but were unaware of which of their patient visits were standardized patient mothers. All mothers recorded their observations on the health maintenance checklist immediately after the visit. Results were reported as the percentage of residents in each group who completed tasks on the checklist during an office visit.
Implementation of the Study
In January 1996, the health maintenance skills of the control group of pediatric residents (n = 32) were assessed using the examination, medical record reviews, and standardized patient methodologies. The 50-item multiple choice examination was completed by first- and second-year control residents in January 1996. We randomly selected medical records of patients seen by control residents for health maintenance visits between January and March 1996 for review. During this same time period, each resident was scheduled to see one of the three standardized patient mothers and their children.
During a 12-week period from July 1996 to September 1996, the new curriculum was implemented among the study group residents (n = 36). Faculty and residents met once per week before their weekly continuity clinic session to complete sections of the curriculum. At the conclusion of the training session, residents saw patients in the pediatric clinic for health maintenance as well as follow-up care of chronic medical conditions. The pediatric faculty supervised all cases. All study residents participated in the entire 12-week curriculum. In February 1997, 4 months after completion of the curriculum, study residents underwent evaluation of their health maintenance knowledge and skills. Multiple choice examinations, medical record reviews, and standardized patient evaluations were performed in the same manner as control residents.
Data obtained from control and study residents were compiled using descriptive statistics. Differences between first- and second-year residents in each group were determined by χ2 or paired t tests using Epi-info version 6.0 (Centers for Disease Control and Prevention, Atlanta, GA) and Minitab statistical software packages. Significance was defined as a P value of <.05.
Needs Assessment Survey
We mailed a total of 61 questionnaires to alumni and third-year pediatric residents. All 11 third-year residents and 38/50 alumni returned completed questionnaires. Participants were asked to choose which of 4 methods they felt they used to learn the 7 health maintenance topics previously described. Forty-six percent of respondents reported that they learned about health maintenance from reading textbooks, 38% learned from viewing others do health maintenance visits, and 8% learned from either their peers or from getting feedback from patients. Participants then rated several areas in which training was needed. The topics in which alumni felt the least prepared for after pediatric residency training were in the areas of common behavioral problems of children and injury prevention (mean Likert score = 4). The third-year residents reported the same needs as alumni and added items in the areas of: interpretation of newborn metabolic tests, screening for hearing and vision abnormalities, counseling on immunizations, using developmental screening tools, and dental hygiene.
Multiple Choice Examination
A total of 32/32 control residents and 36/38 study residents completed the multiple choice examination (Table 1). Before implementation of the new curriculum, the mean percentage of correct answers from first- and second-year residents did not differ significantly (first-year control residents, 61.5%; second-year control residents, 65.1%;P = .21). When subgroups of examinees were analyzed, first-year residents had <60% correct responses in 4 of 8 areas surveyed: health screening, immunizations, growth and development, and behavior. Second-year residents had <60% correct responses in 3 areas: health screening, growth and development, and behavior. After implementation of the curriculum, the mean percentage of correct responses was 62.0% for first-year residents and 69.5% for second-year residents (P = .01). There was no statistically significant difference in scores between first-year residents in any subset before or after the implementation of the curriculum. Second-year residents who completed the new curriculum scored significantly better in knowledge about health screening, nutrition, and dental health; but showed no significant gains in other subsets.
Medical record reviews (Table 2) were performed on 116 charts of control residents (75 first-year residents and 41 second-year residents) and 82 charts of study residents (49 first-year and 33 second-year). In the control group, there was no statistically significant difference in the percentage of charts documenting items in the audit checklist among first- and second-year residents (P = .76). Items documented <60% among first-year control residents included: growth charting, immunization status, diet history, developmental screening, and dental health. Second-year control residents did slightly better in taking a diet history, but were otherwise deficient in the same areas as first-year residents. After implementation of the curriculum, there were significant increases in the percentage of charts documenting items in the audit checklist between control and study residents. Among first-year residents, this increase was evident in all 10 areas included in the audit criteria. The percentage of first-year resident charts rated good to excellent increased from 46.7% in control residents to 97.6% in the study group (P = .01). Second-year residents showed similar significant increases in documentation except in the areas of prescribing fluoride supplements and injury prevention. Overall, good to excellent chart ratings of second-year residents increased from 42.8% to 78.8% (P = .01). For all residents in the study group, the percentage of charts documenting any one of the specific health maintenance items ranged from 69.7% to 100%.
Standardized Patient Encounters
Standardized patient encounters (Table 3) were recorded on 32 control and 36 study residents. As with the data from chart reviews, there were no statistically significant differences between task completion among first- and second-year residents in the control groups. Among first-year residents, the tasks least often performed during the office visit were asking about hearing and vision problems (10.5% of first-year residents), car seat use (36.8%), and giving anticipatory guidance (47.4%). Second-year residents least often completed tasks in the same 3 areas.
Among the study groups, there were few significant differences in task completion between first-year control and study residents. First-year residents significantly improved in the task of asking about hearing and vision problems (relative risk [RR] 6.33;P < .01) and markedly improved their counseling on car seat use (36.8% vs 66.7%) compared with control residents. However, there were no significant improvements in any of the other task areas. Second-year residents showed significant increases in asking about health concerns and taking a diet history compared with controls and markedly improved their mention of growth status to the parent (69.2% of residents vs 93.8%). All other task areas failed to show significant improvement.
All residents who participated in the curriculum completed a written anonymous questionnaire at the conclusion of the course. Ninety-five percent of residents felt the course materials and method of instruction were useful. Over 90% of residents agreed or strongly agreed that the curriculum was helpful, related to their clinical activities, and placed an appropriate emphasis on important topics. Over two thirds of residents agreed that the curriculum stimulated them to read more about health maintenance and to discuss related topics more frequently. Eighty percent of residents felt the curriculum helped them to develop a greater self-confidence in health maintenance knowledge and skills.
According to modern educational theory,17 linking learning points to current job requirements helps to motivate adult learners. This study used a survey of experienced pediatricians to identify learning needs associated with the pediatric health maintenance visit. In most of the topic areas surveyed, pediatricians felt they needed additional instruction during their training years in the areas of health screening, behavior, normal development, and dental health. Other surveys of training programs and training directors12,,13 have documented the need for more instruction in health maintenance, but this study was the first to examine specific health maintenance categories.
Using survey instruments, multiple choice examinations, medical record reviews, and standardized patient mothers, we documented deficiencies in knowledge, medical record documentation, and clinical skills among pediatric residents. The needs identified by practicing physicians and pediatric residents were combined with current national guidelines9,,10 to develop a new curriculum for pediatric residents.
To assess our residents' performance, we developed a system to measure knowledge of health maintenance concepts, documentation of health maintenance in the medical record, and clinical skills during health maintenance visits. Cognitive knowledge of health maintenance topics varied between and within groups. Second-year control residents demonstrated only small gains in examination scores compared with first-year control residents despite an additional year of clinical experience. However, there was a significant 13% increase in knowledge scores between first-year control and second-year study residents, suggesting that our new curriculum in health maintenance improved clinical knowledge above that gained by clinical experience alone. Within year groups, we noted a lack of significant improvement in cognitive knowledge performance. This finding may reflect high baseline scores in some areas such as injury prevention, dental health, and nutrition where little improvement would be expected. In other areas, the absence of significant differences between control and study residents may be attributable to the lack of immediate consequences for failing to master a subject area and no requirement to commit principles to long-term memory. This finding is consistent with other studies comparing medical student and resident performance.18
Chart documentation of health maintenance visit content was poor among control residents but dramatically improved for residents exposed to the new curriculum. The new awareness of health maintenance issues by residents completing the curriculum and the feedback received through weekly chart reviews by the faculty most likely contributed to the improvement in medical record documentation. A study by Greene et al19 showed that peer review alone did not significantly affect resident documentation of health maintenance issues. The reliability and consistency of measuring resident performance using chart audits was demonstrated by Erviti et al.20 They used an audit system similar to ours and showed a high correlation (r = .8) between audit criteria and actual behavior among pediatric residents.
We decided to use standardized patients to assess clinical performance rather than direct observation or videotaping. Standardized patients are less intrusive to the clinical examination and are less likely to bias resident performance. Data from control residents showed no significant difference between first- and second-year residents in clinical performance. Areas of particular weakness included eliciting parental concerns, taking a family history, injury prevention advice, and anticipatory guidance. First- and second-year study residents significantly improved the frequency of asking about health concerns. In addition, second-year residents asked more frequently about family history and the nutritional status of the child. Other topics showed little or no improvement between controls and study groups. Many topics had high baseline frequencies and little improvement could be detected with the sample size. An explanation for lack of improvement in other areas may be attributable to inadequate attention to this skill in the curriculum or inadequate reinforcement by the faculty during presentation of clinical cases. Those areas where we did not note improvement will help us in future revisions of the teaching plan.
Measuring resident learning needs via several assessment methods has implications for curriculum development and faculty teaching. We noted a different pattern of deficiencies in resident performance within the same topic area depending on the assessment method used (Table 4). Measuring needs using written examination scores may identify somewhat different learning needs than using standardized patients. This is most likely attributable to the measurement of different skills by each of the assessment methods. Previous studies of standardized patient assessments have confirmed this lack of correlation with cognitive knowledge measures.21 Alternatively, the standardized patient scores may focus on a more narrow performance domain than the written examination, which can assess a wide variety of issues.
Using multiple assessment measures helped us to identify specific performance areas for improvement. For example, we devoted more time in the new curriculum in the areas of health screening, developmental assessment and behavior because our needs assessment and performance measures both suggested that these were the weakest among our residents. In addition, our curriculum served to focus our faculty teaching on specific areas of instruction in health maintenance and build consensus through a common set of teaching modules.
Resident satisfaction with the curriculum was very high. Most residents believed that the curriculum represented topics that contributed to their learning and that emphasis was placed on important topics. The methods of instruction and clinical activities were well-received by the residents. Overall, most residents viewed the curriculum and its objectives as worthwhile.
There are several limitations to this study. The total number of residents in our training program required the use of historical control groups rather than concurrent controls to achieve an adequate sample size. Although the use of historical controls is a valid research method, the number of residents enrolled in each study group was small. This small sample size may not have enough statistical power to detect subtle differences in knowledge or clinical performance even if they did exist. Some of the differences we did observe were so small that sample sizes many times larger than practical in one training program would be necessary. Although the use of standardized patients has been validated in the literature, only 1 standardized patient evaluation was possible for each resident. Resident performance can vary from encounter to encounter and the results of several observations over time may be necessary to make an assessment of clinical skills. However, the aim of this study was to measure changes that occurred in an entire group of residents rather than changes in individual performance.
Using survey instruments, multiple choice examinations, medical record reviews, and standardized patients, we documented deficiencies in cognitive knowledge, medical record documentation, and clinical skills among pediatric residents. Additional clinical experiences alone did not significantly improve health maintenance skills in our residents. The needs identified by practicing physicians and pediatric residents were combined with current national guidelines to develop a new curriculum using specific goals and objectives, selected readings, self-assessment quizzes, case scenarios, and learning aids. We have showed that this curriculum in combination with clinical experience can improve performance in some health maintenance tasks. The high satisfaction scores for this curriculum suggests that it may be useful in other training programs.
We wish to thank the general pediatrics staff and residents of the University of Texas Health Science Center, San Antonio, for their support and guidance in this research project and to commend their professional spirit in the practice of general pediatrics.
- Received July 30, 1999.
- Accepted December 28, 1999.
Reprint requests to (J.O.L.) Department of Pediatrics, National Naval Medical Center, 8901 Wisconsin Ave, Bethesda, MD 20889-5600. E-mail:
This work was the 1998 winner of the Ray E. Helfer Award for Innovation in Pediatric Education and presented in part at the annual meeting of the Pediatric Academic Societies; May 1998; New Orleans, LA.
Dr Lopreiato is an active duty Navy physician.
The views expressed in this article are those of the authors and do not reflect the official policy or position of the Department of the Navy or the US government.
- UTHSCSA =
- University of Texas Health Science Center San Antonio •
- AY =
- academic year •
- RR =
- relative risk
- Breese BB,
- Disney FA,
- Talpey W
- Bergman AB,
- Dassel SW,
- Wedgwood RJ
- Reisinger KS,
- Bires JA
- Foye H,
- Chamberlin R,
- Charney E
- American Academy of Pediatrics, Committee on Practice and Ambulatory Medicine
- Green M
- ↵Accreditation Council for Graduate Medical Education. Program Requirements for Residency Education in Pediatrics. Chicago, IL: Accreditation Council for Graduate Medical Education; 1997
- ↵Kittredge D, ed. Educational Guidelines for Residency Training in General Pediatrics. Washington, DC: Division of Medicine, Bureau of Health Professions; 1996. Publication No. 103HR940857P000-000
- ↵Green M, ed. Bright Futures: Guidelines for Health Supervision of Infants, Children, and Adolescents. Arlington, VA: National Center for Education in Maternal and Child Health; 1994
- ↵Schwenk TL, Whitman N. Residents as Teachers: A Guide to Educational Practice. 2nd ed. Salt Lake City UT: University of Utah School of Medicine; 1993
- ↵Whitman N. Creative Medical Teaching. Salt Lake City, UT: University of Utah School of Medicine; 1990
- ↵Barker WH, ed. Teaching Preventive Medicine in Primary Care. New York, NY: Springer; 1983
- ↵Jensen NM, Dirkx JM. A Curriculum for Internal Medicine Residency: The University of Wisconsin Program. Philadelphia, PA: American College of Physicians; 1989
- ↵Knowles MS. The Modern Practice of Adult Education. Englewood Cliffs, NJ: Cambridge Adult Education; 1980
- Norman GR,
- Shannon SI
- Greene JW,
- Altemeier WA,
- Moroney DM,
- O'Conner SM
- Copyright © 2000 American Academy of Pediatrics