PEDIATRICS Vol. 117 No. 3 March 2006, pp. 796-802 (doi:10.1542/peds.2005-1403)
Assessment of Pediatricians by a Regulatory Authority
a Department of Community Health Sciences
b Office of Continuing Medical Education, Faculty of Medicine, University of Calgary, Calgary, Canada
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODSRESULTSDISCUSSIONREFERENCES |
|---|
OBJECTIVE. To determine whether it is possible to develop feasible, valid, and reliable multisource feedback data for pediatricians.
METHODS. Surveys with 40, 22, 38, and 37 items were developed for assessment of pediatricians by patients, co-workers, medical colleagues, and themselves, respectively, using 5-point scales with an "unable to assess" category. Items addressed key competencies related to communication skills, professionalism, collegiality, continuing professional development, and collaboration. Each pediatrician was assessed by 25 patients, 8 medical colleagues, and 8 co-workers. Feasibility was assessed with response rates for each instrument. Validity was assessed with rating profiles, the percentage of participants unable to assess the physician for each item, and exploratory factor analyses to determine which items grouped together into scales. Cronbach's 
and generalizability coefficient analyses assessed reliability.
RESULTS. One hundred pediatricians participated. The mean number of respondents per physician was 23.4 (93.6%) for patients, 7.6 (94.8%) for co-workers, and 7.6 (95.5%) for medical colleagues. The mean ratings ranged from 4 to 5 for each item on each scale. Few items had high percentages of "unable to assess" responses. The factor analyses revealed a 4-factor solution for the patient survey, a 3-factor solution for the co-worker survey, and a 4-factor solution for the medical colleague survey, accounting for at least 64% of the variance. All instruments had high internal consistency. The generalizability coefficients were .85 for patients, .87 for co-workers, and .78 for medical colleagues.
CONCLUSION. Surveys can be developed to provide feedback data on key competencies.
Key Words: pediatricians • assessment
Abbreviations: PAR—Physician Achievement Review • CIS—Customer Information Services • MSF—multisource feedback • CPSA—College of Physicians and Surgeons of Alberta • Ep2—generalizability coefficient
Approaches to maintaining competence and certification for physicians have received worldwide attention. In the United States, the Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties have identified the core competencies for US physicians,1 as has the Royal College of Physicians and Surgeons of Canada for Canadian physicians.2 The focus of these essential competencies involves delivering quality care that extends beyond medical knowledge and clinical expertise. These competencies, which include communication skills, collaboration, and professionalism, need to be maintained throughout physicians' careers.1–4 Pediatric residency programs can set up structures to assess and to provide feedback on these competencies with strategies such as direct observation and feedback, preceptor evaluations, and objective structured clinical evaluations. It is more challenging to enhance these competencies for practicing pediatricians.
In many disciplines of medicine, multisource feedback (MSF) or 360° evaluation is being used to assess and to provide feedback to physicians about a broad range of competencies,5,6 by licensing authorities,7–9 professional organizations,10 and health care facilities.11,12 The purpose of MSF is to guide self-development by identifying behaviors that can be addressed. MSF relies on questionnaires completed by patients, medical colleagues (eg, peers and referral physicians), and co-workers (eg, pharmacists and nurses) to provide feedback to physicians about their communication skills, interpersonal skills, collegiality, medical expertise, and ability to learn and to improve practice patterns continually.5–12 The American Board of Internal Medicine10 has developed a patient and peer MSF system as a component of its program for maintenance of competence. The American Board of Pediatrics has indicated that patient and peer surveys will be one of the ways in which pediatricians will provide evidence of satisfactory performance.3,4 Early work in the United Kingdom with pediatric middle-grade and senior house officers showed that this is a feasible, reliable, and practical approach to assessment.13
Studies of MSF show that reliable valid instruments (questionnaires) can be developed.5–13 It seems feasible to develop quality improvement programs in which most of the physicians in the discipline can be assessed by 8 to 10 co-workers, 8 to 10 medical colleagues, and 25 patients.6–12 This number of raters produces an acceptable reliability for both the overall instrument and the physician being assessed.7,9,10 Furthermore, with the intent of MSF being to guide professional development, studies have shown that participating physicians use their feedback data to guide the changes they make.6,8,10 Studies show that physicians increase their explanations to patients, improve the printed material in their offices, change their strategies for communication with their peers, and improve their psychosocial skills after feedback.10,14
The College of Physicians and Surgeons of Alberta (CPSA) Physician Achievement Review (PAR) began developing MSF instruments in 1996.15 The program requires that every physician participate in a 5-year cycle. The original goal of the program was to provide feedback to physicians about 6 broad categories of performance, namely, medical knowledge and skills, attitudes and behavior, professional responsibilities, practice improvement activities, administrative skills, and personal health.7 Instruments have been developed and tested for family physicians7 and surgeons.8 As part of this work, one set of instruments was developed to be used for the combined medical specialties of pediatrics, internal medicine, and psychiatry.9 The instruments were developed by a working group that included pediatricians. Every pediatrician in the province had an opportunity to review all of the items on all instruments and to provide feedback, which was incorporated into the final set of instruments. Our earlier examination of the medical colleague instrument component demonstrated that it was reliable and appropriate for use for all 3 disciplines.9
The main purpose of the present study was to extend the examination of the pediatrician data, specifically examining the co-worker assessment, patient assessment, and self-assessment data to assess the feasibility, validity, and reliability of a MSF system for pediatric practice, in conjunction with the previously reported data from the medical colleague assessment.9 Several questions were of interest. (1) What is the feasibility of an assessment system for practicing pediatricians that provides feedback from patients, co-workers, medical colleagues, and self? (2) What questions about a pediatrician's practice can patients, co-workers, and medical colleagues answer? (3) What are the score profiles (ie, mean and SD) for each of the items in the surveys? (4) Do the items in a survey group together into meaningful scales to guide performance improvement? (5) Are the instruments reliable for both the practice of pediatrics and the individual pediatrician?
|
METHODS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
The pediatricians were recruited by Customer Information Services (CIS), a private company that handles the PAR work. With direction from the CPSA, CIS recruited a combination of generalists and subspecialty pediatricians who were licensed to practice and had been in practice for 5 years. They used the CPSA list of licensed pediatricians and drew a random sample from that list. In Canada, most pediatricians work as consultant pediatricians, rather than functioning as primary care pediatricians. The CPSA database does not separate those with subspecialty practices from other pediatricians.
The final instrument for patients consisted of 40 items (Table 1). Raters were asked to use a 5-point rating scale (from 1 = strongly disagree to 5 = strongly agree). The instruments for co-workers (Table 2) and medical colleagues (Table 3) consisted of 22 and 38 items, respectively, with a 5-point rating scale (from 1 = among the worst to 5 = among the best). The self-assessment instrument (Table 3) was identical to the medical colleague instrument except that all items were written in the first person and the last item on the colleague questionnaire ("If a member of my family needed care, I would rate this physician") was omitted. All questionnaires provided respondents with the option of being able to indicate they were unable to assess the physician on the item.
|
|
|
Each participating pediatrician was responsible for completing a self-assessment and identifying the 8 medical colleagues and 8 co-workers who could answer the questions on the survey. Previous work established that raters chosen by the people being assessed do not provide significantly different evaluations than do those selected by a third party.11 Furthermore, studies examining how well the assessor and the assessed physician knew each another showed that familiarity contributed very little to the variance in ratings.7,10 Each pediatrician was provided with 25 copies of the patient survey, because previous studies6,7 showed that surveys from 25 patients would be required to produce a generalizability coefficient (Ep2) of
.70. The physicians were provided with sealable envelopes so that the patients could complete the questionnaires anonymously in the physician's office and the physician's staff could send all of the forms to CIS for processing. CIS provided co-workers and medical colleagues with copies of the questionnaire, and responses were returned directly to CIS.
A number of statistical analyses were undertaken to address the research questions posed. Response rates were used to determine feasibility for each of the respondent groups (question 1). For each item on each survey, the percentage of "unable to assess" responses and the mean and SD were computed to determine the viability of items and the score profiles (questions 2 and 3, respectively). When the percentage of unable to assess responses exceeds 20% on a survey, it suggests a need to examine the item for revision or deletion. We used exploratory factor analysis to determine which items on the patient and co-worker surveys belonged together (ie, became a factor or scale) (question 4). This analysis allowed us to identify the factors and numbers of factors for each instrument and to describe the relative variance accounted for by each factor and the coherence. These factors or scales could then be used to establish the key domains (eg, communication) for improvement, whereas the items within each factor could provide more precise information about behaviors (eg, is courteous to co-workers). The factor analysis for the medical colleague instrument was reported previously.9 The factors for the self-assessment are not reported in this study because they were identical to the medical colleague factors. Finally, reliability was assessed (question 5). Internal consistency reliability was examined by using Cronbach's coefficient for each of the rater groups and for each of the scales/factors for each rater group. This enables an assessment of overall instrument stability. This analysis was followed by a generalizability analysis to determine Ep2, to ensure that there were sufficient numbers of items and raters to provide stable data for each individual pediatrician on each instrument. Normally, an Ep2 value of .70 suggests that data are stable.6,7,10,11 If the Ep2 is low, then it suggests that more raters or more items are required to enhance stability. The study received approval from the Calgary Health Research Ethics Board.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
A total of 100 pediatricians participated in the study. These physicians provided a total of 2341 patient surveys (mean: 23.41 surveys; range: 20–25 surveys). A total of 758 co-worker surveys were provided (mean: 7.58 surveys; range: 6–8 surveys). There were 764 medical colleague surveys (mean: 7.64 surveys; range: 5–8 surveys).9 All of the 100 pediatricians returned self-assessment forms. The mean response rates were 94.8% for co-worker assessments, 93.6% for patient assessments, and 100.0% for self-assessments. The response rate for medical colleague assessments, as reported earlier, was 95.5%.9
The majority of items on the questionnaires could be answered by respondents. As presented in Tables1 to 3, the assessment of unable to assess items showed that 13 of 40 items on the patient survey, 1 of 22 items on the co-worker survey, and 8 of 38 items on the medical colleague survey had unable to assess rates of >20%. The mean ratings for all items on the patient, medical colleague, and peer surveys were between 4 and 5.
The factor analysis identified 4 factors on the patient survey that accounted for 77.6% of the variance, ie, patient care, technical communication, staff, and office function. The factor analysis identified 3 factors on the co-worker instrument that accounted for 63.8% of the variance, ie, humanistic and psychosocial skills, collegiality, and written communication. As noted elsewhere, the medical colleague assessment identified 4 factors that accounted for 67.6% of the variance, ie, patient management, clinical assessment, professional development, and communication skills.9
Cronbach's was calculated to determine the internal reliability of the instruments. Patient surveys had an value of .99, co-worker surveys a value of .95, medical colleague surveys a value of .98,9 and self-assessments a value of .98. The Ep2 values were .85 and .87 for the patient and co-worker surveys, respectively. The medical colleague questionnaire had an Ep2 value of .78.9
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
This study evaluated the use of questionnaire-based assessments of pediatrician practice. We think these data show that it is feasible to design a MSF program for pediatric practice that includes medical colleague, co-worker, patient, and self components. The PAR program is mandatory, and the response rates were high. These rates are consistent with the response rates for other groups of physicians that we studied7–9 and higher than those achieved in studies in the United States and United Kingdom.10–13
The majority of the items could be answered by the pediatricians' assessors. There were some items that proved difficult for respondents to assess. For the medical colleagues, the most difficult items were those related to professional development and practice-based improvement, as well as some items that might have been difficult to assess for some types of pediatricians (eg, "contributes to quality improvement programs" and "handles emergency situations effectively"). For the patient questionnaires, the items that proved most challenging were related to the aspects of care that patients might not have experienced (eg, "provides, reports, files, or copies of letters in a timely manner" and "refers me to appropriate educational resources"). These items were ones for which the CPSA, as a regulatory authority, receives many complaints. However, given the span of pediatric practice from generalist to subspecialist referral practice, these may need to be reassessed, because not all participants could observe them.
The range and mean ratings were similar to those of other groups, with most physicians receiving all of their ratings between 4 and 5.7–9 Although these scores are high, they are consistent with the range of scores found in most assessments of residents and medical students. Similarly, the self-assessment ratings were lower than those provided by medical colleagues, a finding that is consistent with other studies of this nature.7,8 The factor analysis identified that each questionnaire had several items that grouped together as factors. These factors were consistent with the intent of the PAR program. They provided the general direction for the physicians, because each physician received descriptive data (means and SDs) on the scales and individual items for himself or herself and the group as a whole.
The reliability analyses indicated that overall the instruments and scales were reliable, as indicated by Cronbach's . Furthermore, the Ep2 values indicated that the data provided to each physician were also stable. These data were similar to those found in our previous work7 and better than those achieved in the American Board of Internal Medicine work, which used shorter instruments.10,11 These data suggest that the mixtures of items and raters on the surveys are appropriate.
Overall, we think it is possible to develop high-quality MSF instruments for pediatric practice. Our instrument was developed by a regulatory authority as a quality improvement program. The items on our instruments are focused on broadly based licensure issues. Furthermore, the relatively small total population of physicians in the province (
5500 licensed physicians) meant that instruments specific to the practice of pediatrics were not feasible. Nonetheless, we think that this work demonstrates that reliable, valid, feasible surveys can be developed for pediatricians, including assessments by patients, co-workers, medical colleagues, and the pediatrician. Our instruments may provide a basic set of items with which to assess communication skills, practice-based improvement, professionalism, and patient care, the domains identified by the Accreditation Council for Graduate Medical Education and American Board of Medical Specialties as key to pediatrician and physician performance.1–4
|
ACKNOWLEDGMENTS |
|---|
Funding for the study was provided by the CPSA. Data collection was provided by CIS (Edmonton, Canada).
We offer special thanks to Robert Burns, John Swiniarski, and Bryan Ward at the CPSA for allowing us to continue to be part of this work. We thank our colleagues Drs M. Atkinson, I. Buka, D. Butzner, A. Gordon, A. Leung, R. McLeod, N. Roberts, W. Dickout, D. Urness, and R. Lewkonia, who served on the working group.
|
FOOTNOTES |
|---|
Accepted Jul 29, 2005.
Address correspondence to Claudio Violato, PhD, Department of Community Health Sciences, Faculty of Medicine, University of Calgary, 3330 Hospital Dr NW, Calgary, AB, Canada, T2N 4N1. E-mail: violato{at}ucalgary.ca
The authors have indicated they have no financial relationships relevant to this article to disclose.
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
- Accreditation Council for Graduate Medical Education. Competencies. Available at: www.acgme.org/outcome. Accessed January 9, 2006
- Royal College of Physicians and Surgeons of Canada. Canadian Medical Education Directions for Specialists (CanMEDS). Available at: http://rcpsc.medical.org/canmeds. Accessed January 9, 2006
- Stockman JA, Miles PV, Ham HP. The Program for Maintenance of Certification in Pediatrics (PMCP). J Pediatr. 2003;143 :292 –295[CrossRef][Web of Science][Medline]
- Carraccio C, Englander R, Wolfsthal S, Martin C, Ferentz K. Educating the pediatrician of the 21st century: defining and implementing a competency-based system.
Pediatrics. 2004;113
:252
–258
[Abstract/Free Full Text] - Evans R, Elwyn G, Edwards A. Review of instruments for peer assessment of physicians.
BMJ. 2004;328
:1240
[Abstract/Free Full Text] - Lockyer J. Multisource feedback in the assessment of physician competencies. J Contin Educ Health Prof. 2003;23 :4 –12[CrossRef][Medline]
- Hall W, Violato C, Lewkonia R, et al. Assessment of physician performance in Alberta: the Physician Achievement Review.
CMAJ. 1999;161
:52
–57
[Abstract/Free Full Text] - Violato C, Lockyer J, Fidler H. Multisource feedback: a method of assessing surgical practice.
BMJ. 2003;326
:546
–548
[Free Full Text] - Lockyer J, Violato C. An examination of the appropriateness of using a common peer assessment instrument to assess physician skills across specialties. Acad Med. 2004;10 (suppl):S5 –S8[CrossRef]
- Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med. 2002;77 (suppl):S64 –S66[CrossRef][Web of Science][Medline]
- Ramsey PC, Wenrich MD, Carline JD, Inui TS, Larson EB, LeGerfo JP. Use of peer ratings to evaluate physician performance.
JAMA. 1993;269
:1655
–1660
[Abstract/Free Full Text] - Ramsey PG, Carline JD, Blank LL, Wenrich MD. Feasibility of hospital-based use of peer ratings to evaluate the performances of practicing physicians. Acad Med. 1996;71 :364 –370[Web of Science][Medline]
- Archer JC, Norcini J, Davies HA. Use of SPRAT for peer review of pediatricians in training.
BMJ. 2005;330
:1251
–1253
[Abstract/Free Full Text] - Fidler H, Lockyer JM, Toews J, Violato C. Changing physicians' practices: the effect of individual feedback. Acad Med. 1999;74 :702 –714[Web of Science][Medline]
- College of Physicians and Surgeons of Alberta. Physician Achievement Review. Available at: www.par-program.org/. Accessed January 9, 2006
PEDIATRICS (ISSN 1098-4275). ©2006 by the American Academy of Pediatrics
CiteULike 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
This article has been cited by other articles:
|
|
 |
|
  J L Campbell, S H Richards, A Dickens, M Greco, A Narayanan, and S Brearley Assessing the professional performance of UK doctors: an evaluation of the utility of the General Medical Council patient and colleague questionnaires Qual. Saf. Health Care, June 1, 2008; 17(3): 187 - 193. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. M. Lockyer, C. Violato, and H. M. Fidler Assessment of Radiology Physicians by a Regulatory Authority Radiology, June 1, 2008; 247(3): 771 - 778. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
