Throughout the past 20 years, multiple surveys have documented that graduates of pediatrics residency programs have not felt prepared to practice primary care pediatrics.¹ Concurrent assessments of the curriculum of residency programs, beginning with the report of the American Academy of Pediatrics (AAP) Task Force on Pediatric Education in 1978, demonstrated that primary care aspects of pediatric practice were given little time and emphasis.^2,5,6 Recently, directors of managed care organizations have added their dissatisfaction with the preparedness of graduates of pediatrics residency programs to practice in their settings.^7,8

Multiple changes to the traditional residency curriculum have been proposed to address the Resi-dency-Practice Training Mismatch.⁷ At the University of Massachusetts Medical Center (UMMC), the response included moving the resident continuity experience from hospital clinics to practices. This report describes a survey of UMMC residency graduates and their first employers, to determine resident preparedness for primary care practice.

DESCRIPTION OF RESIDENCY PROGRAM

The goals of the office-based portion of the residency are to prepare graduates to practice primary care pediatrics and to provide an office-based perspective to graduates who enter a subspecialty. The residents go to their office site 1 to 2 half-day sessions a week and have a 4-week block rotation in each year of training. Residents build their own panel of patients. A data system which tracks each patient visit is used to assure that residents see children of various ages, from newborns to adolescents. First year residents initially are allotted 60 minutes for health supervision visits and 30 minutes for sick visits; before the end of the 3 years, these visits are being scheduled for no longer than 30 minutes and 15 minutes, respectively, closer to the pace expected in practice; the ability to increase the pace is facilitated by a low broken appointment rate in the practices and the opportunity to add acute illness visits. The preceptors are graduates of the year-long basic faculty development program on teaching skills and meet regularly throughout the year to review program and resident goals. Additional "nuts and bolts" of the program are described elsewhere.¹⁴

THE SURVEY INSTRUMENT

The survey form instructed graduates to reflect back to their first few months in primary care practice and employers to reflect on the new employee's first few months of job performance. The survey included a question about overall sense of preparedness; employers were also asked to compare the resident to graduates of other residency programs they may have hired (if any). An additional 31 questions were directed to specific areas: preparedness to perform at the pace expected in practice, preparedness to diagnose and treat common illnesses and behavior problems, preparedness to provide anticipatory guidance, preparedness to practice without resources of a medical center, preparedness to work with special populations, preparedness in office management issues, and preparedness to treat subspecialty problems.

A 5-point Likert scale was used. A score of 1 indicated not at all prepared, a score of 2 indicated minimally prepared, a score of 3 indicated fairly well-prepared, a score of 4 indicated well-prepared, and a score of 5 indicated very well-prepared. Additional narrative comments were solicited.

RESULTS (TABLE 1)

Table 1. Means of the Responses of Residents and Their First Employers to 32 Questions Regarding the Residents' Preparedness for Primary Care Practice in the First 1 to 2 Months After Graduation From Residency* [View Table]

The responses of both graduates and employers to "overall preparedness for practice" were either "well-prepared" or "very well-prepared." The employers of 15 graduates also had experience with graduates of other programs and rated all 15 University of Massachusetts graduates as "very well-prepared" compared with the graduates of other programs. Of the other 31 questionnaire items, which attempted to measure particular components of the primary care curriculum, the means of the resident self-ratings exceeded "well-prepared" for 11 components and were less than "fairly well-prepared" for only 2 items, gynecology and managing problems by telephone. Only one of the employer rating means was below "well-prepared" (gynecology, mean of 3.9).

Both groups rated preparedness in independently diagnosing and managing common illnesses and behavior problems to be in the "well-prepared" to "very well-prepared" range. Written comments by residents identified the opportunity to see both acute illness visits and health supervision visits in the office to be valuable; in particular, learning the office treatment of common acute illnesses provided a useful contrast to the emergency department treatment of the same complaints. In the five anticipatory guidance areas, resident means were between "well-prepared" and "very well-prepared" for all except nutrition; employers rated residents "well-prepared" to "very well-prepared" for all five. Practice management issues received means between "fairly well-prepared" and "well-prepared" from residents and employer means were between "well-prepared" and "very well-prepared." Managing problems by telephone was one of only two mean scores in the resident's self ratings below "fairly well-prepared" (2.9); employers rated them between "well-prepared" and "very well-prepared."

Among the subspecialty problems represented in primary care practice, residents felt "well-prepared" in infectious diseases and pulmonology; the two mean scores at "fairly well-prepared" or below were in gynecology (2.7) and orthopedics (3.0). Employers also rated resident preparedness lower in gynecology (3.9) than in the other nine subspecialty areas, which received mean scores between "well-prepared" and "very well-prepared."

DISCUSSION

The current study attempted to assess preparedness to practice primary care pediatrics upon graduation from a residency in which all residents participated in an office-based continuity experience, whether at the hospital or in the community. Resident self-evaluation was used as one measure, recognizing the limitation that residents tend to rate themselves lower than other evaluators rate them on the same task.^17,18 Proposed explanations of this phenomenon are that trainees have difficulty defining behavioral criteria against which to measure themselves; that they tie self-assessments to stable, globalized attributions of ability; and that they fault themselves for not having lived up to unrealistic expectations.¹⁹

Employer evaluation was used as another important outcomes-oriented measure. Previous surveys have not found such employer satisfaction with new graduates.^7,8 In this survey, employers rated the residency graduates "well-prepared" or "very well-prepared" in absolute terms and relative to graduates of other programs.

Overall Preparedness

A most interesting finding was the employer ratings of resident "overall sense of preparedness." Previous studies of employer satisfaction have indicated that employers are highly critical of the training of their pediatricians. A 1993 survey of managed care organizations, conducted by the Group Health Association of America under contract to the federal Health Resources Services Administration, asked the medical directors of managed care practices the question: "How well do residency programs prepare primary care physicians for practice in managed care settings?" The responses of the medical directors were: 62%, "poorly prepared"; 32%, "prepared"; and 6%, "well-prepared" (Fig 1).⁷ In 1994, a separate study contracted by Health Resources Services Administration reported that 13 of 23 HMOs did not believe their primary care physicians were well trained for HMO settings and respondents from 4 additional HMOs had serious concerns in specific areas such as resource management, working in a team, managing a patient's care, prevention, and patient education.⁸

In contrast to these previous surveys, the UMMC study indicates high employer satisfaction, both in absolute terms and compared with graduates of other programs, with no rating below "well-prepared." The employer responses for the seven residents who went to work in managed care settings were similar to the overall group, with two responses of "well-prepared" and five responses of "very well-prepared."

Developmental and Behavioral Pediatrics (DBP)/Anticipatory Guidance

In comparison to these studies, UMMC residents rated themselves "well-prepared" to provide anticipatory guidance and manage common behavioral issues; employers also rated residents between "well-prepared" and "very well-prepared." In the UMMC residency curriculum, DBP received a two-pronged approach, taught by preceptors in the offices and by faculty at the medical center, with each resident having two 4-week DBP rotations. There was considerable interaction between DBP faculty and the preceptors at faculty development sessions. Groups were begun for preceptors (and other pediatricians in the area) to present cases for discussion with DBP and child psychiatry faculty; a similar group was conducted for senior residents, in which monthly sessions were conducted jointly by a senior DBP faculty member and a practitioner-preceptor. It is likely that the combined approach contributed to the graduates' sense of preparation in this area.

Office Management

The results of the UMMC study show that residents rate themselves lower in management issues than in other areas. Immersion in the clinical activities of a practice may prepare the resident to work at the pace expected in practice and to function without the resources of a medical center immediately available, but it does not guarantee recognition of the structure of the practice or management issues such as the use of the telephone. These issues require formal attention, such as a curriculum on telephone triage skills.

Subspecialty Areas

Employers perceived residents to be "well-prepared" to "very well-prepared" in 9 of the 10 subspecialty items, the exception being in gynecology. The largest difference between employer responses and resident self-ratings was in orthopedics.

The similarity between the subspecialty areas in which the residents felt relatively unprepared and those identified by the AAP Task Force on Pediatric Education and subsequent studies as inadequately stressed in residency programs is provocative. It is quite possible that the preceptors may have limitations teaching in these areas because of their own lack of preparation in residency or because they have little opportunity to maintain skills in their practices or both. In support of this hypothesis are the results of a separate survey of preceptors, which asked what content areas the preceptors would most like to address in monthly Continuing Medical Education sessions designed specifically for them ("Community Teaching Rounds"). The preceptors chose topics related to gynecology, orthopedics, nutrition, psychopharmacology, and behavior. The implication for faculty development programs for community preceptors is that, in addition to building teaching skills, consideration should be given to the clinical content and skills practitioners might need.²⁴

A combined approach (medical center and office practices), such as described above in DBP, may be useful, as suggested by the responses regarding preparedness in gynecology; coincident with the recruitment of additional adolescent medicine faculty and the initiation of required rotations in adolescent medicine, the scores in "gynecology" increased from an mean of 2.5 for the residents who graduated in 1991 to a mean of 3.3 for those who graduated in 1995. In both DBP and adolescent medicine, the strategy chosen has been to develop an integrated curriculum with primary care rather than to depend on totally independent rotations.

Limitations of This Study

The high rating of resident preparedness for practice is associated with the change in residency curriculum of conducting the continuity experiences in office practices. However, no data are available before the change in curriculum to permit a comparison; moreover, other changes were made in the curriculum as well, in areas such as DBP and adolescent medicine, which likely contributed to the outcome.

The number of resident graduates is small, because of the small size of the program (7 residents per year) and the restriction to graduates who entered primary care practice. Confirmation is required from larger programs or from multiple programs using community-based primary care education.

It could be argued that residents who were oriented toward primary care preferentially selected the University of Massachusetts residency because of the opportunity to establish a continuity practice in the community. This selection bias is unlikely to account for the high preparedness scores in the early years of the program, before the reputation of the program was established, and the overall preparedness means do not vary by year of graduation as demonstrated above. Moreover, the rate of graduates entering primary care practice was 67.5% during the years of the study, comparable to the percentage of first-time candidates for certification by the American Board of Pediatrics in 1995 (67.2%).²⁵

There may well be bias in favor of the residents, because the residency program was the source of the questionnaire rather than a more anonymous source. However, even the employers who have no relationship with the medical center (eg, are located in distant states) considered the residents to be well-prepared. The two employers who hired the residents who worked in their offices had reason to be biased, because the residents were "known quantities," who were "trained" in their office; their scores did not differ from the others. The director of the staff model HMO system had some previous knowledge of the residents, but he was not himself a preceptor, and his assessment reflects his experience with new employees from several residency programs.

Implications

1. The most important finding of the survey is that both the residency program graduates feel prepared and their employers rate the graduates as well-prepared to very well-prepared for practice. After 3 years in the office setting for their continuity experience, the residents feel prepared for the pace of practice and the evaluation and management of the types of visits that occur commonly in office practice.
2. One notable difference between the results of this survey and the 1978 AAP Task Force Report and related studies was in preparedness to evaluate and manage developmental and behavioral issues and to provide anticipatory guidance. The relatively high ratings may relate to the combined teaching approach taken in the residency program, utilizing both medical center faculty and community practitioners, a strategy that may be useful in other areas as well.
3. The survey demonstrates that practice management education does not happen by osmosis; physical presence in the practices is insufficient to assure recognition of the structure of the practice or management issues such as the use of the telephone. A more formal process, including curriculum, is required.
4. Preceptors, who may themselves be graduates of programs that gave little attention to areas such as orthopedics and gynecology, may have difficulty teaching in those areas without assistance. Faculty development and continuing medical education programs directed in these areas may result in increased teaching abilities and expanded clinical abilities.

Conclusions