Evaluation of a Peer-Reviewed Career Development and Compensation Program for Physicians at an Academic Health Science Center
Objective. The Department of Pediatrics at the Hospital for Sick Children, which is funded by an alternative payment plan, has implemented a novel career development and compensation program (CDCP). Job activity profiles were used to more clearly define job expectations, benchmarks guided career development, and peer review was used to assess performance. The objective of this study was to evaluate the departmental pediatricians’ satisfaction with the CDCP.
Methods. Pediatricians, all of whom had undergone CDCP annual reviews, could participate if they had undergone the in-depth triennial CDCP review. Each received a 5-point Likert scale-based questionnaire that asked how well the CDCP had conformed to the principles identified by the department during the development of the CDCP. Anonymous, confidential responses were collated and used to guide focus groups that discussed areas of greatest concern and attempted to identify solutions. Focus groups were led by external facilitators who were experienced in qualitative research. They audiotaped the sessions, transcribed the comments, and analyzed the data with the assistance of a qualitative analysis application.
Results. Sixty of the eligible 88 pediatricians participated, and 74% of their responses were that the CDCP had addressed the original principles “somewhat,” “to a great extent,” or “extremely well.” The remainder indicated that some of the principles were either “not addressed” or “only to a small extent” by the CDCP. Results from the 11 focus groups (46 participants) indicated that the CDCP was an improvement over the previous method of career development and determination of the rate of remuneration. Most were also still in agreement with the purpose and design principles. Although they did not want the CDCP to undergo a major redesign, they identified areas that need improvement. Short-, medium-, and long-term action plans to address these areas are under way.
Conclusion. Pediatricians at the health science center of the Hospital for Sick Children remain supportive of the CDCP.
In 1990, academic pediatricians in the Department of Pediatrics at the Hospital for Sick Children (HSC) entered into an innovative alternative funding plan (AFP) with the government of Ontario.1,2 The HSC and the University of Toronto’s Faculty of Medicine both were signatories on the original AFP wherein remuneration arising from the fee-for-service payment for clinical care was exchanged for a set amount of funding. The AFP had a novel characteristic in that 50% of the funds were allocated for patient care and related administrative activities, with the remainder devoted to pediatricians’ research (30%) and educational activities (20%). This recognized that physicians within academic health science centers not only provide direct patient care, but also have a responsibility to carry out education and research activities. However, there was no formal system to promote the individual pediatrician’s career development, enhance his or her academic performance, and fairly evaluate and competitively financially reward all of the activities of an academic pediatrician. To address this issue, the members of the Department’s Pediatric Consultants Partnership practice plan created and implemented a career development and compensation program (CDCP).3,4 The CDCP used job activity profiles (JAPs) to define more clearly job expectations, developed benchmarks to guide career development, and implemented peer review to assess performance. The JAPs are as follows: 1) Clinician Teacher, major (50%–65%) commitment to provide, advance, and promote clinical care; usually significant bedside teaching and some research activities; 2) Clinician-Educator, major (≥50%) commitment to education administration and educational development or research in education; participates in clinical care and bedside teaching; 3) Clinician-Scientist, major (75%) commitment to research; participates in clinical care and education; 4) Clinician-Investigator, significant (30%–70%) research commitment and contributes to education and/or research; 5) Clinician-Administrator, major (>50%) administrative responsibilities and contributes to clinical care, education, and research; and 6) Clinician-Specialist, predominate (≥70%) commitment to provide, advance, and promote excellence in clinical care with contributions to education and/or research. The CDCP determines the total remuneration of individual pediatricians by using 2 processes: 1) an annual bonus related to short-term performance, the amount of which ranges from 0% to 10% of the base compensation rate, and 2) a triennial review process that leads to changes in the pediatrician’s assigned “level” within the CDCP (each of the 8 levels within the CDCP is associated with an incremental amount of guaranteed base compensation).
The annual review-bonus process begins with the staff pediatrician and his or her Division Chief reviewing his or her JAP and setting goals and objectives for the upcoming year. Twelve months later, the Division Chief reviews the pediatrician’s performance, both from a general point of view and relative to the previously established goals and objectives. The Division Chief then makes a recommendation as to the amount of bonus to the Department Chief.
The triennial review process is undertaken by pediatricians who have been on staff at the HSC for the preceding 3 years. The pediatrician creates and submits separate clinical, medical education, and research dossiers to the department’s Clinical, Medical Education, and Research Advisory Committees. Each committee assigns a “category of achievement” based on previously developed benchmarks. The resultant confidential peer evaluation of their performance is then reviewed by the Chief of Pediatrics, who places the evaluation into context by considering other factors. These factors include the number of years on staff at an academic health science center; his or her JAP; the amount of time allocated for clinical, education, and research activities; and other pertinent information to decide whether the pediatrician’s “level” should be altered. Because one third of the eligible pediatricians underwent this triennial review process each year by the summer of 2001, all of the pediatricians who had been on staff at the time of the CDCP implementation completed this process. During this period, there were only minor changes to the CDCP.
We wanted to revisit the CDCP’s overall structure to determine whether it should be continued and, if so, what modifications should be made to improve it. To identify the strengths and weaknesses of the CDCP, we first conducted an anonymous, confidential survey. The results of the survey were collated to guide additional inquiry through focus groups. These focus groups, led by external facilitators who were experienced in qualitative research, were developed to discuss areas of most concern and to identify solutions. However, no restrictions were placed on the content of the discussions. The results show that the pediatricians preferred the CDCP over the department’s previous methods of career development and allocation of income; although they identified areas for improvement, they did not believe that major changes were required.
All of the department’s pediatricians had participated in the annual review process. However, only those physicians who had completed the triennial review were eligible to participate in this evaluation of the CDCP.
We focused the questionnaire on the issues and principles that had guided the development of the CDCP (Table 1). To determine whether the CDCP had addressed these principles, the authors developed questions that used a 5-point Likert-based scale. Although the exact wording of the responses varied slightly depending on the question, the potential responses to the questions were essentially that the CDCP had “not addressed,” “only addressed to a small extent,” “somewhat addressed,” “addressed to a great extent,” or “addressed extremely well” the theme or principle of interest. Questions relating to the workload required to prepare for the CDCP and the quality and amount of feedback from the CDCP process were also included. Comments were invited on any aspect of the CDCP. The names of the 88 eligible pediatricians were provided by our department to the HSC’s Department of Human Resources, which mailed copies of the survey to each eligible participant. For ensuring confidentiality and anonymity, the completed surveys were returned directly to Human Resources for analysis. Respondents were asked to identify their JAP to enable analysis of the responses by these categories. No other identifying information was requested. The survey was completed during August and September 2001. In an effort to maximize the response rate, 2 reminder messages and an additional electronic copy of the survey were sent to participants during the survey completion period.
Focus Group Sessions
The results from the questionnaire were collated and used to identify areas of strength and concern regarding the CDCP. This information was used as a general guide for subsequent focus group sessions that were held with the same group of participants to discuss the areas of concern in greater detail and, when possible, to identify solutions. Physicians who participated in these focus groups were also encouraged to identify any additional issues that had not been identified by the questionnaire. A total of 11 focus group sessions were held, with separate groups by JAP and for division chiefs. The focus group sessions were held during late November and December 2001. Many contacts were sent out by e-mail during this period to arrange convenient times and to encourage participation. Focus groups were led by facilitators from Smaller World Communications (Toronto, Ontario, Canada), a performance and evaluation company that is experienced in qualitative research techniques and that had frequently worked in partnership with academia and health care. The rate of remuneration of the external consultants was established before beginning the focus group sessions, and no member of the pediatric executive or hospital administration attended the focus group sessions. The facilitators audiotaped the sessions and took notes during the focus group sessions. All notes and tapes were transcribed. The NU*DIST qualitative analysis application (Version 5, QSR International Pty Ltd, Melbourne, Victoria, Australia) was used. Three researchers from Smaller World Communications were involved in the analysis of the data. The 3 researchers designated a first iteration of theme names based on the feedback from the questionnaire. One researcher then independently entered relevant information from each of the 11 focus groups into NU*DIST and indexed the information into themes. A second researcher then reviewed the themes and quotes and recommended changes. A third researcher resolved any discrepancies between the first and second researchers. A summary from each focus group session, including supporting quotes with no names, was sent for validation to each participant of a given specific focus group session. These validated group summaries were then analyzed to identify common themes across all focus groups and those specific to JAP and division chief categories.
Sixty of the 88 eligible pediatricians completed the questionnaire, yielding an overall 68% response rate. The participation of members from each JAP, with number of eligible participants in parenthesis, was as follows: 11 (11) Clinician Educators, 11 (15) Clinician Investigators, 15 (22) Clinician Scientists, 21 (33) Clinician Teachers, and 2 (6) Clinician Specialists. Because of the small number of Clinician Administrators (1), they did not participate in this process. Very few Clinician Specialists were eligible for participation as this is a new JAP and very few individuals had been on staff for 3 years (see Methods section). Seventy-four percent of the responses were that the CDCP had addressed the original principles “somewhat,” “to a great extent,” or “extremely well” (Fig 1, Table 2). The remainder indicated that some of the principles were either “not addressed” or addressed “only to a small extent” by the CDCP. Of all questions, the most positive responses came from the question, “How well has the CDCP assessed your performance through the annual career review and performance evaluation process?” (Table 2). The most negative response came from the question, “Has the benefit of the CDCP process merited the work required on your part to prepare your submissions?” (Table 2); the same question also had the largest number of written comments.
Focus Group Sessions
Forty-six pediatricians participated in the 11 focus group sessions. Overall, participants indicated that the CDCP was an improvement over the previous approach used by the department for career development and determination of their rate of remuneration. They indicated that they were still in agreement with the CDCP’s purpose and design principles and that the CDCP did not require major redesign. As described below, they also noted that some of the themes and principles were not being fully met by the current CDCP.
The external consultant, Smaller World Communications, consolidated the themes from the focus groups into 5 key areas. The first was “General Understanding of the CDCP”; the participants wanted some clarification on several issues, including how the Department Chief determined the individual’s final level, the role of the Division Chief and peer review process in the evaluation process, and the linkage between annual and triennial reviews. They also requested clarification of and an improvement in the linkage between their perception of the division’s and the department’s goals. The second theme was “Enhancement of Career Development”; they requested increased mentorship, assistance in addressing the challenges that affect career development, and recognition of how the provision of supports affected career development. The third theme was “Fairness and Equity of the Program.” Many pediatricians believed that research continued to be valued more than the provision of clinical care. They also thought that it was important that the context of their work situation and actual proportion of time available for clinical, education, and research activities was considered when achievements were being assessed. The fourth theme was the “Work Required for the CDCP”; they believed that there could be improved coordination and streamlining of review processes that occur within the HSC and that additional refinement of expectations and documentation was required for the review process. The final theme was “Improving the Measurement of Performance”; discussions largely focused on the need for improved measures of the quantity and quality of clinical work.
The Pediatric Consultants Partnership, the practice plan for pediatricians at the HSC, first developed and then implemented a CDCP.3 By 2001, the majority of the full-time members of the partnership had completed the annual review process on 3 occasions and completed a triennial review. An anonymous questionnaire and subsequent focus group sessions indicated that the department’s pediatricians perceived the CDCP as an improvement over past methods for career development and determination of their rate of remuneration. They also indicated that there was no need for a major redesign of the program. However, there were aspects of the CDCP that required improvement, the solutions for which could be simplified into 3 areas: 1) those that simply required clarification and communication of existing principles and/or processes, 2) those that were important to address but would take months to years to develop and implement, and 3) others for which no immediate solutions or strategies could be identified. On the basis of these findings, the Pediatric Consultants Partnership has now developed short-, medium-, and long-term action plans that will lead to additional improvements, where possible, in the CDCP.
Issues that could be addressed within 3 months are included in our short-term action plan. Examples include additional communication and clarification of existing approaches. Specific emphasis will be placed on the process used by the Department Chief for the determination of the CDCP level, the role of annual review in determining the CDCP level, and increasing the amount of detailed feedback provided to the pediatrician who undergoes triennial review. Also, the appeal process will be revised and additional assistance will be provided for the preparation of the dossiers. Examples of medium-term goals are to streamline the process for dossier preparation and to improve the transparency of the annual and triennial review decision-making processes. We will also enhance the role and effectiveness of the Division Chief in mentoring the individual pediatrician and in the provision of evaluations to the Department Chief. Strategies will be developed to improve the linkage between personal, divisional, and departmental goals. Long-term (1–3 years) goals included improvements in the department’s mentorship programs, enhancing the department’s assessment of clinical performance, and better incorporating the work context into the final evaluation.
Our preset goal was to obtain a 60% return rate on our questionnaire. We exceeded this goal (68%), so it is likely that the responses accurately reflect the perceptions of the entire group of eligible pediatricians. Our participation rate in the focus group sessions was lower (52%) than that for the questionnaire and lower than the 1996 focus group participation rate that was part of the development of the CDCP (unpublished observations). We have no evidence to support or reject potential explanations, including apathy, individuals who reject the CDCP and the review process, or disinterest as a result of the improved financial position of the department and rate of remuneration for the department’s pediatricians.
When the department developed the CDCP,3 the focus was on the principles or themes that were identified during the initial focus group sessions held in 1996 (Table 1). Examples include our goal to reward comparable performance equally in each of the newly created JAPs; the underlying assumption was that it would be equally rigorous and challenging to develop and improve one’s capabilities within each JAP. Because the desired goals included the promotion and provision of excellent evidence-based patient care and effective utilization of resources, it was also essential to develop a CDCP that, although influenced by, would not be entirely determined by the university’s academic promotion track. For the CDCP to be fair, a structured evaluation process that was open and understood by the physicians and valid and respected by all members was required.
This study provides insight as to how the department’s pediatricians view the CDCP. The results indicate that the benefits outweighed the negative aspects of the program, such as the significant work required by the pediatricians. The study, however, cannot assess the CDCP’s contribution to the department’s present morale or its overall achievements in the clinical, education, and research arenas. For example, our ability to renegotiate the AFP has permitted an increase in the amount of remuneration for each CDCP level, which likely has had a positive impact on morale. There has been a recent increase in the amount of research funding provided by the government of Canada. Although we are aware of the establishment of other AFPs for academic physicians,5 we are unaware of previous publications regarding physician satisfaction with the resultant approaches to remuneration.
Should other institutions wish to develop an analogous CDCP, there are several points that are worth emphasizing. First and most important, we found that the 1996 focus group sessions led by expert external consultants were 1 of the major enablers that led to the development and subsequent success of the CDCP. These sessions permitted our pediatricians to express freely their opinions, and the themes identified during these discussions also provided us with criteria against which we can subsequently assess the CDCP. This led us to use the anonymous questionnaire followed by confidential focus group sessions led by external consultants with expertise in qualitative research techniques to assess the process on completion of 1 full cycle. It is also important that there be clarity in defining the relative roles and responsibilities of the Division and Department Chiefs. Finally, frequent communication and dialogue with department members optimize the potential for the understanding of the principles and processes for a successful CDCP.
The Department of Pediatrics at the HSC implemented a CDCP to develop the careers of its pediatricians and determine individual remuneration derived from an AFP. Through an anonymous questionnaire and focus groups, we have obtained our pediatricians’ perceptions of the program. In general, they remain supportive of the CDCP and indicate that they do not want it to undergo a major redesign process. Strategies and implementation plans are being developed to address areas of the CDCP that require change or improvements.
This assessment of the CDCP program represents the work of many members of the Department of Pediatrics at the HSC. Special thanks go to the Department of Pediatrics’ Clinical, Medical Education, and Research Advisory Committees. The Department’s Finance Committee and Pediatric Consultants Partnership Executive Committee (Drs Marvin Gans, James Hilton, Debra Katzman, and Dennis Scolnik) also provided valuable insight and advice. The study could not have been completed without the valuable assistance of Kim Mundy and Linda Wan. We also thank Barb Van Maris, Jennifer Yessis, and Lisa Stockton from Smaller World Communications (Toronto, Ontario, Canada).
- ↵Haslam RHA, Walker NE. Alternative funding plans: is there a place in academic medicine? Can Med Assoc J.1993;148 :1141– 1146
- ↵Haslam RHA. Alternative funding plan, Department of Pediatrics, University of Toronto: is the AFP still alive? Ann R Coll Physicians Surg Can.1996;29 :219– 222
- ↵O’Brodovich H, Pleinys R, Walker NE. Peer-reviewed career development and compensation program for physicians in an academic health science centre. Ann R Coll Physicians Surg Can.2000;33 :88– 96
- ↵Thorne S. Queen approves alternative funding arrangement. Can Med Assoc J.1994;151 :1637– 1638
- Copyright © 2003 by the American Academy of Pediatrics