Throughout the past 20 years, multiple surveys have
documented that graduates of pediatrics residency programs have not
felt prepared to practice primary care pediatrics.1
Concurrent assessments of the curriculum of residency programs,
beginning with the report of the American Academy of Pediatrics (AAP)
Task Force on Pediatric Education in 1978, demonstrated that primary
care aspects of pediatric practice were given little time and
emphasis.2,5,6 Recently, directors of managed care
organizations have added their dissatisfaction with the preparedness of
graduates of pediatrics residency programs to practice in their
settings.7,8
Multiple changes to the traditional residency curriculum have been
proposed to address the Resi-dency-Practice Training
Mismatch.7 At the University of Massachusetts
Medical Center (UMMC), the response included moving the resident
continuity experience from hospital clinics to practices. This report
describes a survey of UMMC residency graduates and their first
employers, to determine resident preparedness for primary care
practice.
DESCRIPTION OF RESIDENCY PROGRAM 
UMMC is the only tertiary care center in central Massachusetts for
children beyond the neonatal period. Rotations are offered in every
pediatric subspecialty. In 1987, the Division of General and Community
Pediatrics changed its model for the continuity experience, recruiting
community practitioners as preceptors and initiating a faculty
development program.13 Since July 1988, residents have
been paired one-on-one with office-based pediatricians for their entire
3-year continuity experience. Matching is based on interview visits and
mutual selection. The hospital-based clinic was closed, and a faculty
group practice was begun, with a pediatrician recruited from office
practice as the director. This practice was not used as a training site
for incoming residents between 1988 and 1990. Once the transformation
from clinic to practice was accomplished, faculty whose primary
function was primary care practice were recruited as preceptors and
paired one-on-one with residents in the same fashion as practitioners
in the community. Thus, some residents have been in the community and
some at the hospital, but all have been in office practices. The
community sites have included offices of a staff model health
maintenance organization (HMO), traditional "private practices,"
and two sites affiliated with the medical center (a community health
center and an office practice).
The goals of the office-based portion of the residency are to prepare
graduates to practice primary care pediatrics and to provide an
office-based perspective to graduates who enter a subspecialty. The
residents go to their office site 1 to 2 half-day sessions a week and
have a 4-week block rotation in each year of training. Residents build
their own panel of patients. A data system which tracks each patient
visit is used to assure that residents see children of various ages,
from newborns to adolescents. First year residents initially are
allotted 60 minutes for health supervision visits and 30 minutes for
sick visits; before the end of the 3 years, these visits are being
scheduled for no longer than 30 minutes and 15 minutes, respectively,
closer to the pace expected in practice; the ability to increase the
pace is facilitated by a low broken appointment rate in the practices
and the opportunity to add acute illness visits. The preceptors are
graduates of the year-long basic faculty development program on
teaching skills and meet regularly throughout the year to review
program and resident goals. Additional "nuts and bolts" of the
program are described elsewhere.14
THE SURVEY INSTRUMENT
A questionnaire was mailed to the 37 residents who graduated from
the residency between 1991 and 1995 and who completed a minimum of 2 years in the program. Questionnaires were also sent to the first
employers of the 25 residents who entered primary care practice.
The survey form instructed graduates to reflect back to their first few
months in primary care practice and employers to reflect on the new
employee's first few months of job performance. The survey included a
question about overall sense of preparedness; employers were also asked
to compare the resident to graduates of other residency programs they
may have hired (if any). An additional 31 questions were directed to
specific areas: preparedness to perform at the pace expected in
practice, preparedness to diagnose and treat common illnesses and
behavior problems, preparedness to provide anticipatory guidance,
preparedness to practice without resources of a medical center,
preparedness to work with special populations, preparedness in office
management issues, and preparedness to treat subspecialty problems.
A 5-point Likert scale was used. A score of 1 indicated not at all prepared, a score of 2 indicated minimally
prepared, a score of 3 indicated fairly well-prepared, a score of 4 indicated well-prepared, and a score of 5 indicated very well-prepared. Additional narrative comments were solicited.
Fig. 1.
Overall preparedness of residency graduates for practice: responses of
residences and employees using a 5-point Likert scale. (Literature:
Resident responses from reference 1; employer responses calculated from
reference 7.)
[View Larger Version of this Image (18K GIF file)]
RESULTS (TABLE 1)
Surveys from all 25 residents who entered primary care practice
were completed and used in the analysis. Of the 25 residents, 12 (48%)
had their continuity site in various offices of an HMO, 6 (24%) had
been in "private practices," and 7 (28%) had been either in the
UMMC practice or at a UMMC-sponsored community site. The distribution
of employment sites differed from this pattern: 7 (28%) practiced in
HMO sites, 13 (52%) in "private practices," and 5 (20%) in other
settings, such as the Indian Health Service and an academic medical
center. Of the 25, 4 stayed in the same office in which they had their
continuity experience. The employers of 20 of the 25 graduates
responded; of the 5 who did not respond, 4 were no longer in the
practice and could not be located (including 3 from the Indian Health
Service), and 1 did not respond despite multiple requests.
|
Table 1.
Means of the Responses of Residents and Their First
Employers to 32 Questions Regarding the Residents' Preparedness for
Primary Care Practice in the First 1 to 2 Months After Graduation From Residency*
[View Table]
|
The responses of both graduates and employers to "overall
preparedness for practice" were either "well-prepared" or "very well-prepared." The employers of 15 graduates also had experience with graduates of other programs and rated all 15 University of Massachusetts graduates as "very well-prepared" compared with the
graduates of other programs. Of the other 31 questionnaire items, which
attempted to measure particular components of the primary care
curriculum, the means of the resident self-ratings exceeded
"well-prepared" for 11 components and were less than "fairly
well-prepared" for only 2 items, gynecology and managing problems by
telephone. Only one of the employer rating means was below
"well-prepared" (gynecology, mean of 3.9).
Both groups rated preparedness in independently diagnosing and managing
common illnesses and behavior problems to be in the "well-prepared"
to "very well-prepared" range. Written comments by residents
identified the opportunity to see both acute illness visits and health
supervision visits in the office to be valuable; in particular,
learning the office treatment of common acute illnesses provided a
useful contrast to the emergency department treatment of the same
complaints. In the five anticipatory guidance areas, resident means
were between "well-prepared" and "very well-prepared" for all
except nutrition; employers rated residents "well-prepared" to
"very well-prepared" for all five. Practice management issues received means between "fairly well-prepared" and
"well-prepared" from residents and employer means were between
"well-prepared" and "very well-prepared." Managing problems by
telephone was one of only two mean scores in the resident's self
ratings below "fairly well-prepared" (2.9); employers rated them
between "well-prepared" and "very well-prepared."
Among the subspecialty problems represented in primary care practice,
residents felt "well-prepared" in infectious diseases and
pulmonology; the two mean scores at "fairly well-prepared" or below
were in gynecology (2.7) and orthopedics (3.0). Employers also rated
resident preparedness lower in gynecology (3.9) than in the other nine
subspecialty areas, which received mean scores between
"well-prepared" and "very well-prepared."
DISCUSSION
The traditional approach to assessing the primary care component
of residency programs has been to focus on readily available quantitative measures, such as the number of patients seen or the
amount of time spent, rather than on outcomes. Residents assigned to
private practices have been reported to see more patients per session
than those assigned to hospital clinics or publicly-funded community
clinics,15,16 but the significance of the additional experience per se is not clear. Osborn et al15 compared the performance of residents in different types of continuity sites on the
American Board of Pediatrics In-Training Examination and the Behavioral
Pediatrics Examination. Residents in private offices had equivalent
scores on the In-Training Examination and higher scores on the
Behavioral Pediatrics Examination than residents at the university
clinic or in public clinics, but the process of assignment was by
choice rather than random and the amount of time in the various
settings differed, precluding a definitive conclusion about the
independent effect of the setting or of the number of patients seen.
The current study attempted to assess preparedness to practice primary
care pediatrics upon graduation from a residency in which all residents
participated in an office-based continuity experience, whether at the
hospital or in the community. Resident self-evaluation was used as one
measure, recognizing the limitation that residents tend to rate
themselves lower than other evaluators rate them on the same
task.17,18 Proposed explanations of this phenomenon are
that trainees have difficulty defining behavioral criteria against
which to measure themselves; that they tie self-assessments to stable,
globalized attributions of ability; and that they fault themselves for
not having lived up to unrealistic expectations.19
Employer evaluation was used as another important outcomes-oriented
measure. Previous surveys have not found such employer satisfaction
with new graduates.7,8 In this survey, employers rated the
residency graduates "well-prepared" or "very well-prepared" in
absolute terms and relative to graduates of other programs.
Overall Preparedness
The UMMC resident response on question 1, "overall sense
of preparedness" was a mean of 4.3, or between "well-prepared"
and "very well-prepared." This would seem to echo other studies in which graduates express satisfaction with their residency
program,1,4,11 but the current survey differs from
previous ones in an important way. In previous surveys, all of the
graduates were asked to rate their satisfaction with various aspects of
their residency experience. In the UMMC survey, the subset of residents
who chose to practice primary care pediatrics were asked to rate their
preparation for practice. Comparable elements of the questionnaires
from previous surveys suggest that preparation for primary care
practice was not rated as highly as overall satisfaction with the
program. For example, in the most recent survey published, the score
given to the quality of preparation in primary pediatric care clinic (noted by the authors to be equivalent to continuity clinic) was 2.86 on a 5-point scale similar to the one used in the University of
Massachusetts survey.1
A most interesting finding was the employer ratings of resident
"overall sense of preparedness." Previous studies of employer satisfaction have indicated that employers are highly critical of the
training of their pediatricians. A 1993 survey of managed care
organizations, conducted by the Group Health Association of America
under contract to the federal Health Resources Services Administration,
asked the medical directors of managed care practices the question:
"How well do residency programs prepare primary care physicians for
practice in managed care settings?" The responses of the medical
directors were: 62%, "poorly prepared"; 32%, "prepared"; and
6%, "well-prepared" (Fig 1).7 In 1994, a separate
study contracted by Health Resources Services Administration reported that 13 of 23 HMOs did not believe their primary care physicians were
well trained for HMO settings and respondents from 4 additional HMOs
had serious concerns in specific areas such as resource management, working in a team, managing a patient's care, prevention, and patient
education.8
In contrast to these previous surveys, the UMMC study indicates high
employer satisfaction, both in absolute terms and compared with
graduates of other programs, with no rating below "well-prepared." The employer responses for the seven residents who went to work in
managed care settings were similar to the overall group, with two
responses of "well-prepared" and five responses of "very
well-prepared."
Developmental and Behavioral Pediatrics (DBP)/Anticipatory Guidance
The 1978 AAP Task Force Report identified goals for residency
curriculum, including increased emphasis on biosocial and developmental problems of children and adolescents. To train pediatricians who are
skilled in these areas, the report suggested that residents spend more
time in ambulatory experiences, provide continuous care for a group of
their own patients, and develop skill in anticipatory guidance,
developmental appraisal, screening, and referral.5 Studies assessing progress in the years after the Task Force Report identified that change came very slowly.2,6 Although a 1989 survey similar to the one used by the 1978 Task Force found that graduates after 1984 thought their training was better in DBP and
adolescent medicine, 50% of the 1785 member sample still considered their training insufficient in biosocial areas.2
In comparison to these studies, UMMC residents rated themselves
"well-prepared" to provide anticipatory guidance and manage common
behavioral issues; employers also rated residents between "well-prepared" and "very well-prepared." In the UMMC residency curriculum, DBP received a two-pronged approach, taught by preceptors in the offices and by faculty at the medical center, with each resident
having two 4-week DBP rotations. There was considerable interaction
between DBP faculty and the preceptors at faculty development sessions.
Groups were begun for preceptors (and other pediatricians in the area)
to present cases for discussion with DBP and child psychiatry faculty;
a similar group was conducted for senior residents, in which monthly
sessions were conducted jointly by a senior DBP faculty member and a
practitioner-preceptor. It is likely that the combined approach
contributed to the graduates' sense of preparation in this area.
Office Management
Residency programs have been criticized for not preparing
graduates in practice management issues, particularly those related to
managed care. The Council on Graduate Medical Education reports that,
despite concern by many physicians about managed care, more than 75%
of physicians have at least one managed care contract, and almost 50%
are involved with at least one HMO.20 Of the preceptors of UMMC residents, 45% are employed by a staff-model HMO,
and the other 55% have more than one managed care contract. Studies of
practicing physicians affirm the need for more training in
cost-effective care. Graduates from the University of California, San
Diego, between 1979 to 1988 reported that their lowest perceived comfort level was in "economics of pediatric
practice."3 Cantor et al11 found that 81% of
physicians were satisfied overall with their training, but only 46%
rated training in "cost-effective medical care" as excellent or
good, and only 4% thought they were prepared for "management aspects
of your practice." Studies of internal medicine residents and family
physicians report similar results.21,22
The results of the UMMC study show that residents rate themselves lower
in management issues than in other areas. Immersion in the clinical
activities of a practice may prepare the resident to work at the pace
expected in practice and to function without the resources of a medical
center immediately available, but it does not guarantee recognition of
the structure of the practice or management issues such as the use of
the telephone. These issues require formal attention, such as a
curriculum on telephone triage skills.
Subspecialty Areas
For primary care pediatricians to be the initial managers of
subspecialty problems, they need a firm basis in subspecialty areas.23 Regarding "office management of
subspecialty areas," residents felt least prepared in gynecology
(2.7) and orthopedics (3.0). Women graduates gave slightly higher
scores in gynecology than men, but both genders reported feeling less
than "fairly well-prepared" (women 2.8, men 2.5). Gynecology and
orthopedics have been identified as weaknesses in previous surveys. In
the 1978 AAP Task Force Report, 66.9% of practicing physicians
surveyed reported a low feeling of competence when dealing with
gynecology, and 73.9% said they had insufficient training in that
area.5 In the same survey, 55.9% of physicians reported a
low feeling of competence when dealing with orthopedics, with 60%
reporting insufficient training; a recent study also identified
orthopedics as an area of weakness.3
Employers perceived residents to be "well-prepared" to "very
well-prepared" in 9 of the 10 subspecialty items, the exception being
in gynecology. The largest difference between employer responses and
resident self-ratings was in orthopedics.
The similarity between the subspecialty areas in which the residents
felt relatively unprepared and those identified by the AAP Task Force
on Pediatric Education and subsequent studies as inadequately stressed
in residency programs is provocative. It is quite possible that the
preceptors may have limitations teaching in these areas because of
their own lack of preparation in residency or because they have little
opportunity to maintain skills in their practices or both. In support
of this hypothesis are the results of a separate survey of preceptors,
which asked what content areas the preceptors would most like to
address in monthly Continuing Medical Education sessions designed
specifically for them ("Community Teaching Rounds"). The preceptors
chose topics related to gynecology, orthopedics, nutrition,
psychopharmacology, and behavior. The implication for faculty
development programs for community preceptors is that, in addition to
building teaching skills, consideration should be given to the clinical
content and skills practitioners might need.24
A combined approach (medical center and office practices), such as
described above in DBP, may be useful, as suggested by the responses
regarding preparedness in gynecology; coincident with the recruitment
of additional adolescent medicine faculty and the initiation of
required rotations in adolescent medicine, the scores in
"gynecology" increased from an mean of 2.5 for the residents who
graduated in 1991 to a mean of 3.3 for those who graduated in 1995. In
both DBP and adolescent medicine, the strategy chosen has been to
develop an integrated curriculum with primary care rather than to
depend on totally independent rotations.
Limitations of This Study
Because there was not a uniform period of time in the present
study from completion of residency to the time of the survey, it is
possible that graduates with a longer period may not have rated their
preparation the same as those with a shorter period. The direction of
possible bias is hard to predict: perhaps those in practice for a few
years have a better idea of what they did not know at first and,
therefore, rate their preparation lower; or those in practice for only
a few months might still feel anxious and, therefore, rate their
preparation lower. In fact, the variation from year to year was slight,
with the means of overall preparation for the five cohorts being 4.3, 4.3, 4.3, 4.2, and 4.2.
The high rating of resident preparedness for practice is associated
with the change in residency curriculum of conducting the continuity
experiences in office practices. However, no data are available before
the change in curriculum to permit a comparison; moreover, other
changes were made in the curriculum as well, in areas such as DBP and
adolescent medicine, which likely contributed to the outcome.
The number of resident graduates is small, because of the small size of
the program (7 residents per year) and the restriction to graduates who
entered primary care practice. Confirmation is required from larger
programs or from multiple programs using community-based primary care
education.
It could be argued that residents who were oriented toward primary care
preferentially selected the University of Massachusetts residency
because of the opportunity to establish a continuity practice in the
community. This selection bias is unlikely to account for the high
preparedness scores in the early years of the program, before the
reputation of the program was established, and the overall preparedness
means do not vary by year of graduation as demonstrated above.
Moreover, the rate of graduates entering primary care practice was
67.5% during the years of the study, comparable to the percentage of
first-time candidates for certification by the American Board of
Pediatrics in 1995 (67.2%).25
There may well be bias in favor of the residents, because the residency
program was the source of the questionnaire rather than a more
anonymous source. However, even the employers who have no relationship
with the medical center (eg, are located in distant states) considered
the residents to be well-prepared. The two employers who hired the
residents who worked in their offices had reason to be biased, because
the residents were "known quantities," who were "trained" in
their office; their scores did not differ from the others. The director
of the staff model HMO system had some previous knowledge of the
residents, but he was not himself a preceptor, and his assessment
reflects his experience with new employees from several residency
programs.
Implications
- The most important finding of the survey is that
both the residency program graduates feel prepared and their employers
rate the graduates as well-prepared to very well-prepared for practice. After 3 years in the office setting for their continuity experience, the residents feel prepared for the pace of practice and the evaluation and management of the types of visits that occur commonly in office practice.
- One notable difference between the results of this survey and
the 1978 AAP Task Force Report and related studies was in preparedness to evaluate and manage developmental and behavioral issues and to
provide anticipatory guidance. The relatively high ratings may relate
to the combined teaching approach taken in the residency program,
utilizing both medical center faculty and community practitioners, a
strategy that may be useful in other areas as well.
- The survey demonstrates that practice management education does
not happen by osmosis; physical presence in the practices is
insufficient to assure recognition of the structure of the practice or
management issues such as the use of the telephone. A more formal
process, including curriculum, is required.
- Preceptors, who may themselves be graduates of programs that
gave little attention to areas such as orthopedics and gynecology, may
have difficulty teaching in those areas without assistance. Faculty
development and continuing medical education programs directed in these
areas may result in increased teaching abilities and expanded clinical
abilities.
Conclusions
Three years of primary care continuity experience in an office
practice is associated with a high level of resident preparedness for
practice, as determined by the graduates and their employers. Foci of
relative weakness remain, however; practice management and certain
subspecialty areas, which need to be addressed explicitly, either by
combining office-based training with medical center programs or by
developing curricula for practitioner-preceptors. Preceptors may
themselves benefit from continuing medical education in the same
areas.
Address correspondence to Kenneth B. Roberts, MD, Pediatric
Teaching Program, Moses Cone Health System, 1200 North Elm Street,
Greensboro, NC 27401.
AAP, American Academy of Pediatrics.
UMMC, University of Massachusetts Medical Center.
HMO, health maintenance
organization.
DBP, developmental and behavioral pediatrics.
ABSTRACT
