Objective. Surveys of residency graduates and employers have suggested that residency programs do not prepare residents well for practice. Since 1988, pediatric residents at the University of Massachusetts have been paired one-on-one with an office-based pediatrician for their 3-year continuity experience. This survey was conducted to determine if graduates of such a program are prepared to enter pediatric practice.
Methodology. Graduates of the program from 1991 through 1995 who entered primary care practice were surveyed about their preparedness for practice. The questionnaire was also sent to the residents' first employers. The 32 questions were directed to overall sense of preparedness, ability to manage the pace of practice, common illnesses, common behavior problems, anticipatory guidance, office management, and subspecialty problems.
Results. Data from all 25 residents who entered practice and the employers of 20 of the 25 residents were obtained and analyzed. Both groups rated overall resident preparedness to be “well-prepared” or “very well-prepared” and gave high scores on working at the pace of practice, diagnosing and treating common illnesses, diagnosing and treating common behavior problems, and providing anticipatory guidance. Areas in which residents were considered to be less well-prepared included anticipatory guidance about nutrition, managing problems by telephone, office management, gynecology, and orthopedics.
Conclusions. The results suggest that continuity experiences in office practices are associated with preparation for the pace and types of visits that occur commonly in primary care practice, abilities which previous surveys of residency alumni and employers have found lacking. Some areas may benefit from a formal curriculum which may be implemented in the office practice, at the medical center, or at both sites. Preceptors may benefit from faculty development and continuing medical education that is directed not only at teaching skills but also at content areas which were not addressed in their own residencies.
- preparedness for practice
- internship and residency
- evaluation studies
- health maintenance organizations
Throughout the past 20 years, multiple surveys have documented that graduates of pediatrics residency programs have not felt prepared to practice primary care pediatrics.1Concurrent assessments of the curriculum of residency programs, beginning with the report of the American Academy of Pediatrics (AAP) Task Force on Pediatric Education in 1978, demonstrated that primary care aspects of pediatric practice were given little time and emphasis.2,5,6 Recently, directors of managed care organizations have added their dissatisfaction with the preparedness of graduates of pediatrics residency programs to practice in their settings.7,8
Multiple changes to the traditional residency curriculum have been proposed to address the Resi-dency-Practice Training Mismatch.7 At the University of Massachusetts Medical Center (UMMC), the response included moving the resident continuity experience from hospital clinics to practices. This report describes a survey of UMMC residency graduates and their first employers, to determine resident preparedness for primary care practice.
DESCRIPTION OF RESIDENCY PROGRAM
UMMC is the only tertiary care center in central Massachusetts for children beyond the neonatal period. Rotations are offered in every pediatric subspecialty. In 1987, the Division of General and Community Pediatrics changed its model for the continuity experience, recruiting community practitioners as preceptors and initiating a faculty development program.13 Since July 1988, residents have been paired one-on-one with office-based pediatricians for their entire 3-year continuity experience. Matching is based on interview visits and mutual selection. The hospital-based clinic was closed, and a faculty group practice was begun, with a pediatrician recruited from office practice as the director. This practice was not used as a training site for incoming residents between 1988 and 1990. Once the transformation from clinic to practice was accomplished, faculty whose primary function was primary care practice were recruited as preceptors and paired one-on-one with residents in the same fashion as practitioners in the community. Thus, some residents have been in the community and some at the hospital, but all have been in office practices. The community sites have included offices of a staff model health maintenance organization (HMO), traditional “private practices,” and two sites affiliated with the medical center (a community health center and an office practice).
The goals of the office-based portion of the residency are to prepare graduates to practice primary care pediatrics and to provide an office-based perspective to graduates who enter a subspecialty. The residents go to their office site 1 to 2 half-day sessions a week and have a 4-week block rotation in each year of training. Residents build their own panel of patients. A data system which tracks each patient visit is used to assure that residents see children of various ages, from newborns to adolescents. First year residents initially are allotted 60 minutes for health supervision visits and 30 minutes for sick visits; before the end of the 3 years, these visits are being scheduled for no longer than 30 minutes and 15 minutes, respectively, closer to the pace expected in practice; the ability to increase the pace is facilitated by a low broken appointment rate in the practices and the opportunity to add acute illness visits. The preceptors are graduates of the year-long basic faculty development program on teaching skills and meet regularly throughout the year to review program and resident goals. Additional “nuts and bolts” of the program are described elsewhere.14
THE SURVEY INSTRUMENT
A questionnaire was mailed to the 37 residents who graduated from the residency between 1991 and 1995 and who completed a minimum of 2 years in the program. Questionnaires were also sent to the first employers of the 25 residents who entered primary care practice.
The survey form instructed graduates to reflect back to their first few months in primary care practice and employers to reflect on the new employee's first few months of job performance. The survey included a question about overall sense of preparedness; employers were also asked to compare the resident to graduates of other residency programs they may have hired (if any). An additional 31 questions were directed to specific areas: preparedness to perform at the pace expected in practice, preparedness to diagnose and treat common illnesses and behavior problems, preparedness to provide anticipatory guidance, preparedness to practice without resources of a medical center, preparedness to work with special populations, preparedness in office management issues, and preparedness to treat subspecialty problems.
RESULTS (TABLE 1)
The responses of both graduates and employers to “overall preparedness for practice” were either “well-prepared” or “very well-prepared.” The employers of 15 graduates also had experience with graduates of other programs and rated all 15 University of Massachusetts graduates as “very well-prepared” compared with the graduates of other programs. Of the other 31 questionnaire items, which attempted to measure particular components of the primary care curriculum, the means of the resident self-ratings exceeded “well-prepared” for 11 components and were less than “fairly well-prepared” for only 2 items, gynecology and managing problems by telephone. Only one of the employer rating means was below “well-prepared” (gynecology, mean of 3.9).
Both groups rated preparedness in independently diagnosing and managing common illnesses and behavior problems to be in the “well-prepared” to “very well-prepared” range. Written comments by residents identified the opportunity to see both acute illness visits and health supervision visits in the office to be valuable; in particular, learning the office treatment of common acute illnesses provided a useful contrast to the emergency department treatment of the same complaints. In the five anticipatory guidance areas, resident means were between “well-prepared” and “very well-prepared” for all except nutrition; employers rated residents “well-prepared” to “very well-prepared” for all five. Practice management issues received means between “fairly well-prepared” and “well-prepared” from residents and employer means were between “well-prepared” and “very well-prepared.” Managing problems by telephone was one of only two mean scores in the resident's self ratings below “fairly well-prepared” (2.9); employers rated them between “well-prepared” and “very well-prepared.”
Among the subspecialty problems represented in primary care practice, residents felt “well-prepared” in infectious diseases and pulmonology; the two mean scores at “fairly well-prepared” or below were in gynecology (2.7) and orthopedics (3.0). Employers also rated resident preparedness lower in gynecology (3.9) than in the other nine subspecialty areas, which received mean scores between “well-prepared” and “very well-prepared.”
The traditional approach to assessing the primary care component of residency programs has been to focus on readily available quantitative measures, such as the number of patients seen or the amount of time spent, rather than on outcomes. Residents assigned to private practices have been reported to see more patients per session than those assigned to hospital clinics or publicly-funded community clinics,15,16 but the significance of the additional experience per se is not clear. Osborn et al15 compared the performance of residents in different types of continuity sites on the American Board of Pediatrics In-Training Examination and the Behavioral Pediatrics Examination. Residents in private offices had equivalent scores on the In-Training Examination and higher scores on the Behavioral Pediatrics Examination than residents at the university clinic or in public clinics, but the process of assignment was by choice rather than random and the amount of time in the various settings differed, precluding a definitive conclusion about the independent effect of the setting or of the number of patients seen.
The current study attempted to assess preparedness to practice primary care pediatrics upon graduation from a residency in which all residents participated in an office-based continuity experience, whether at the hospital or in the community. Resident self-evaluation was used as one measure, recognizing the limitation that residents tend to rate themselves lower than other evaluators rate them on the same task.17,18 Proposed explanations of this phenomenon are that trainees have difficulty defining behavioral criteria against which to measure themselves; that they tie self-assessments to stable, globalized attributions of ability; and that they fault themselves for not having lived up to unrealistic expectations.19
Employer evaluation was used as another important outcomes-oriented measure. Previous surveys have not found such employer satisfaction with new graduates.7,8 In this survey, employers rated the residency graduates “well-prepared” or “very well-prepared” in absolute terms and relative to graduates of other programs.
The UMMC resident response on question 1, “overall sense of preparedness” was a mean of 4.3, or between “well-prepared” and “very well-prepared.” This would seem to echo other studies in which graduates express satisfaction with their residency program,1,4,11 but the current survey differs from previous ones in an important way. In previous surveys, all of the graduates were asked to rate their satisfaction with various aspects of their residency experience. In the UMMC survey, the subset of residents who chose to practice primary care pediatrics were asked to rate their preparation for practice. Comparable elements of the questionnaires from previous surveys suggest that preparation for primary care practice was not rated as highly as overall satisfaction with the program. For example, in the most recent survey published, the score given to the quality of preparation in primary pediatric care clinic (noted by the authors to be equivalent to continuity clinic) was 2.86 on a 5-point scale similar to the one used in the University of Massachusetts survey.1
A most interesting finding was the employer ratings of resident “overall sense of preparedness.” Previous studies of employer satisfaction have indicated that employers are highly critical of the training of their pediatricians. A 1993 survey of managed care organizations, conducted by the Group Health Association of America under contract to the federal Health Resources Services Administration, asked the medical directors of managed care practices the question: “How well do residency programs prepare primary care physicians for practice in managed care settings?” The responses of the medical directors were: 62%, “poorly prepared”; 32%, “prepared”; and 6%, “well-prepared” (Fig 1).7 In 1994, a separate study contracted by Health Resources Services Administration reported that 13 of 23 HMOs did not believe their primary care physicians were well trained for HMO settings and respondents from 4 additional HMOs had serious concerns in specific areas such as resource management, working in a team, managing a patient's care, prevention, and patient education.8
In contrast to these previous surveys, the UMMC study indicates high employer satisfaction, both in absolute terms and compared with graduates of other programs, with no rating below “well-prepared.” The employer responses for the seven residents who went to work in managed care settings were similar to the overall group, with two responses of “well-prepared” and five responses of “very well-prepared.”
Developmental and Behavioral Pediatrics (DBP)/Anticipatory Guidance
The 1978 AAP Task Force Report identified goals for residency curriculum, including increased emphasis on biosocial and developmental problems of children and adolescents. To train pediatricians who are skilled in these areas, the report suggested that residents spend more time in ambulatory experiences, provide continuous care for a group of their own patients, and develop skill in anticipatory guidance, developmental appraisal, screening, and referral.5Studies assessing progress in the years after the Task Force Report identified that change came very slowly.2,6 Although a 1989 survey similar to the one used by the 1978 Task Force found that graduates after 1984 thought their training was better in DBP and adolescent medicine, 50% of the 1785 member sample still considered their training insufficient in biosocial areas.2
In comparison to these studies, UMMC residents rated themselves “well-prepared” to provide anticipatory guidance and manage common behavioral issues; employers also rated residents between “well-prepared” and “very well-prepared.” In the UMMC residency curriculum, DBP received a two-pronged approach, taught by preceptors in the offices and by faculty at the medical center, with each resident having two 4-week DBP rotations. There was considerable interaction between DBP faculty and the preceptors at faculty development sessions. Groups were begun for preceptors (and other pediatricians in the area) to present cases for discussion with DBP and child psychiatry faculty; a similar group was conducted for senior residents, in which monthly sessions were conducted jointly by a senior DBP faculty member and a practitioner-preceptor. It is likely that the combined approach contributed to the graduates' sense of preparation in this area.
Residency programs have been criticized for not preparing graduates in practice management issues, particularly those related to managed care. The Council on Graduate Medical Education reports that, despite concern by many physicians about managed care, more than 75% of physicians have at least one managed care contract, and almost 50% are involved with at least one HMO.20 Of the preceptors of UMMC residents, 45% are employed by a staff-model HMO, and the other 55% have more than one managed care contract. Studies of practicing physicians affirm the need for more training in cost-effective care. Graduates from the University of California, San Diego, between 1979 to 1988 reported that their lowest perceived comfort level was in “economics of pediatric practice.”3 Cantor et al11 found that 81% of physicians were satisfied overall with their training, but only 46% rated training in “cost-effective medical care” as excellent or good, and only 4% thought they were prepared for “management aspects of your practice.” Studies of internal medicine residents and family physicians report similar results.21,22
The results of the UMMC study show that residents rate themselves lower in management issues than in other areas. Immersion in the clinical activities of a practice may prepare the resident to work at the pace expected in practice and to function without the resources of a medical center immediately available, but it does not guarantee recognition of the structure of the practice or management issues such as the use of the telephone. These issues require formal attention, such as a curriculum on telephone triage skills.
For primary care pediatricians to be the initial managers of subspecialty problems, they need a firm basis in subspecialty areas.23 Regarding “office management of subspecialty areas,” residents felt least prepared in gynecology (2.7) and orthopedics (3.0). Women graduates gave slightly higher scores in gynecology than men, but both genders reported feeling less than “fairly well-prepared” (women 2.8, men 2.5). Gynecology and orthopedics have been identified as weaknesses in previous surveys. In the 1978 AAP Task Force Report, 66.9% of practicing physicians surveyed reported a low feeling of competence when dealing with gynecology, and 73.9% said they had insufficient training in that area.5 In the same survey, 55.9% of physicians reported a low feeling of competence when dealing with orthopedics, with 60% reporting insufficient training; a recent study also identified orthopedics as an area of weakness.3
Employers perceived residents to be “well-prepared” to “very well-prepared” in 9 of the 10 subspecialty items, the exception being in gynecology. The largest difference between employer responses and resident self-ratings was in orthopedics.
The similarity between the subspecialty areas in which the residents felt relatively unprepared and those identified by the AAP Task Force on Pediatric Education and subsequent studies as inadequately stressed in residency programs is provocative. It is quite possible that the preceptors may have limitations teaching in these areas because of their own lack of preparation in residency or because they have little opportunity to maintain skills in their practices or both. In support of this hypothesis are the results of a separate survey of preceptors, which asked what content areas the preceptors would most like to address in monthly Continuing Medical Education sessions designed specifically for them (“Community Teaching Rounds”). The preceptors chose topics related to gynecology, orthopedics, nutrition, psychopharmacology, and behavior. The implication for faculty development programs for community preceptors is that, in addition to building teaching skills, consideration should be given to the clinical content and skills practitioners might need.24
A combined approach (medical center and office practices), such as described above in DBP, may be useful, as suggested by the responses regarding preparedness in gynecology; coincident with the recruitment of additional adolescent medicine faculty and the initiation of required rotations in adolescent medicine, the scores in “gynecology” increased from an mean of 2.5 for the residents who graduated in 1991 to a mean of 3.3 for those who graduated in 1995. In both DBP and adolescent medicine, the strategy chosen has been to develop an integrated curriculum with primary care rather than to depend on totally independent rotations.
Limitations of This Study
Because there was not a uniform period of time in the present study from completion of residency to the time of the survey, it is possible that graduates with a longer period may not have rated their preparation the same as those with a shorter period. The direction of possible bias is hard to predict: perhaps those in practice for a few years have a better idea of what they did not know at first and, therefore, rate their preparation lower; or those in practice for only a few months might still feel anxious and, therefore, rate their preparation lower. In fact, the variation from year to year was slight, with the means of overall preparation for the five cohorts being 4.3, 4.3, 4.3, 4.2, and 4.2.
The high rating of resident preparedness for practice is associated with the change in residency curriculum of conducting the continuity experiences in office practices. However, no data are available before the change in curriculum to permit a comparison; moreover, other changes were made in the curriculum as well, in areas such as DBP and adolescent medicine, which likely contributed to the outcome.
The number of resident graduates is small, because of the small size of the program (7 residents per year) and the restriction to graduates who entered primary care practice. Confirmation is required from larger programs or from multiple programs using community-based primary care education.
It could be argued that residents who were oriented toward primary care preferentially selected the University of Massachusetts residency because of the opportunity to establish a continuity practice in the community. This selection bias is unlikely to account for the high preparedness scores in the early years of the program, before the reputation of the program was established, and the overall preparedness means do not vary by year of graduation as demonstrated above. Moreover, the rate of graduates entering primary care practice was 67.5% during the years of the study, comparable to the percentage of first-time candidates for certification by the American Board of Pediatrics in 1995 (67.2%).25
There may well be bias in favor of the residents, because the residency program was the source of the questionnaire rather than a more anonymous source. However, even the employers who have no relationship with the medical center (eg, are located in distant states) considered the residents to be well-prepared. The two employers who hired the residents who worked in their offices had reason to be biased, because the residents were “known quantities,” who were “trained” in their office; their scores did not differ from the others. The director of the staff model HMO system had some previous knowledge of the residents, but he was not himself a preceptor, and his assessment reflects his experience with new employees from several residency programs.
The most important finding of the survey is that both the residency program graduates feel prepared and their employers rate the graduates as well-prepared to very well-prepared for practice. After 3 years in the office setting for their continuity experience, the residents feel prepared for the pace of practice and the evaluation and management of the types of visits that occur commonly in office practice.
One notable difference between the results of this survey and the 1978 AAP Task Force Report and related studies was in preparedness to evaluate and manage developmental and behavioral issues and to provide anticipatory guidance. The relatively high ratings may relate to the combined teaching approach taken in the residency program, utilizing both medical center faculty and community practitioners, a strategy that may be useful in other areas as well.
The survey demonstrates that practice management education does not happen by osmosis; physical presence in the practices is insufficient to assure recognition of the structure of the practice or management issues such as the use of the telephone. A more formal process, including curriculum, is required.
Preceptors, who may themselves be graduates of programs that gave little attention to areas such as orthopedics and gynecology, may have difficulty teaching in those areas without assistance. Faculty development and continuing medical education programs directed in these areas may result in increased teaching abilities and expanded clinical abilities.
Three years of primary care continuity experience in an office practice is associated with a high level of resident preparedness for practice, as determined by the graduates and their employers. Foci of relative weakness remain, however; practice management and certain subspecialty areas, which need to be addressed explicitly, either by combining office-based training with medical center programs or by developing curricula for practitioner-preceptors. Preceptors may themselves benefit from continuing medical education in the same areas.
- Received February 27, 1997.
- Accepted May 13, 1997.
- Address correspondence to Kenneth B. Roberts, MD, Pediatric Teaching Program, Moses Cone Health System, 1200 North Elm Street, Greensboro, NC 27401.
↵* From the Department of Pediatrics, University of Massachusetts Medical Center, Worcester, Massachusetts.
Dr Roberts is currently at Moses Cone Health System, Greensboro, North Carolina.
Dr DeWitt is currently at Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio.
Reprints not available.
- AAP =
- American Academy of Pediatrics •
- UMMC =
- University of Massachusetts Medical Center •
- HMO =
- health maintenance organization •
- DBP =
- developmental and behavioral pediatrics
- Wender EH,
- Bijur PE,
- Boyce WT
- Taras H,
- Nader P
- DeAngelis C,
- Duggan A,
- Oski F
- ↵American Academy of Pediatrics. Report of the Task Force on Pediatric Education. Chicago, IL: American Academy of Pediatrics; 1978
- Weinberger HL,
- Oski FA
- ↵Palsbo S, Sullivan C. The Recruitment Experience of Health Maintenance Organizations for Primary Care Physicians: Final Summary Report. Washington, DC: Group Health Associates of America, Inc; 1993
- ↵Felt S, Frazer H, Gold M. HMO Primary Care Staffing Patterns: A Cross-Site Analysis of 23 HMOs. Rockville, MD: Health Resources Services Administration; 1994
- Sargent JR,
- Osborn LM,
- Roberts KB,
- DeWitt TG
- Osborn LM,
- Sargent JR,
- Williams SD
- ↵Council on Graduate Medical Education. Sixth Report. Managed Health Care. Implications for the Physician Workforce and Medical Education. Rockville, MD: Council on Graduate Medical Education; 1995
- Charney E
- Roberts K,
- DeWitt T
- Stockman, J. Annual Report of the President of the American Board of Pediatrics to the Association of Pediatric Program Directors. Chapel Hill, NC: American Board of Pediatrics; 1996
- Copyright © 1997 American Academy of Pediatrics