 |
|
 |
|
 |
 |
|
 |
 |
 |
 |
 |
 |
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ARTICLE |
Department of Pediatrics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
METHODS. Thirty-six pediatric residents were evaluated by parents, nurses, and attending physicians during their first year of training. For analysis, the percentage of responses in the highest response category was calculated for each resident on each item. Differences between attending physician ratings and those of parents and nurses were compared using the signed rank test.
RESULTS. Parent and attending physician ratings were similar on most items, but attending physicians indicated that they frequently were unable to observe the behaviors of interest. Nurses rated residents lower than did attending physicians on items that related to respecting staff (69% vs 97%), accepting suggestions (56% vs 82%), teamwork (63% vs 88%), being sensitive and empathetic (62% vs 85%), respecting confidentiality (73% vs 97%), demonstrating integrity (75% vs 92%), and demonstrating accountability (67% vs 83%). Nurse responses were higher than attending physicians on anticipating postdischarge needs (46% vs 25%) and effectively planning care (52% vs 33%).
CONCLUSIONS. Expanding resident evaluation procedures to include parents and nurses does enhance information that is gathered on resident communication skills and professionalism and may help to target specific behaviors for improvement. Additional research is needed to determine whether receiving feedback on parent and nurse evaluations will have a positive impact on resident competency.
Key Words: medical education • resident education/training • multisource feedback • patient–doctor communication • professionalism
Abbreviations: ABIM—American Board of Internal Medicine • PSQ—Patient Satisfaction Questionnaire
Communication and professionalism are fundamental to the patient–physician relationship. Effective communication improves patient satisfaction and health outcomes,1,2 whereas poor communication and unprofessional behavior are linked to patient complaints and malpractice claims.3,4 Nonetheless, according to a nationally representative survey by the Agency for Healthcare Research and Quality, 1 in 3 parents reported that their child's doctor did not always listen carefully, show respect, or explain things well.5 These findings highlight the need to ensure competence in communication skills, interpersonal skills, and professionalism among pediatricians.
The Accreditation Council for Graduate Medical Education Outcomes Project requires that residency-training programs assess 6 core competencies. Among these are interpersonal and communication skills that "result in effective information exchange and teaming with patients, their families, and other health professionals" and professionalism, "as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population."6 Traditionally, residents have been evaluated only by supervising attending physicians, but attending physicians often are asked to evaluate residents on behaviors that were not witnessed.7–10 Moreover, even when residents are observed directly, attending physicians still must infer how someone else is affected by the encounter to judge the effectiveness of the interaction.11
Encountering similar problems in evaluating employee performance, business organizations have used the multisource evaluation process (also known as 360-degree evaluation) for nearly 30 years.12 Multisource evaluation is a questionnaire-based assessment that gathers perspectives from multiple people within an employee's sphere of influence. The rationale for multisource evaluation is that a more comprehensive account of performance may help to guide improvement and learning.
Multisource evaluation has been adapted in a variety of internal medicine settings by using nurses and patients to evaluate residents and practicing physicians.13–15 Nurse and patient ratings have been compared with those that were obtained from physician raters. Most studies used different instruments for each rating source and found significant correlations between sources. These correlations, however, generally were low, suggesting that each rater provided a different insight.16–24
Incorporating multisource evaluation in pediatric residency training could involve creating a formal mechanism for nurses and parents, rather than adult patients, to evaluate residents. Recently, parent evaluations of interpersonal skills were shown to identify correctly medical students who subsequently performed poorly on a summative observed structured clinical skills examination.25 To our knowledge, no published reports have examined the relationship between resident ratings that are obtained from nurses and parents with those that are obtained from attending physicians in pediatric settings. Communicating with children and their parents is likely to be different from communicating with a single adult who is able to make independent decisions about health care.26 Whether this difference in communication dynamics has an impact on the concordance between rating sources is not known.
The purpose of this study was to determine whether parent and nurse ratings of specific pediatric resident behaviors differed from the ratings that were obtained from attending physicians. We hypothesized that parent and nurse ratings of resident performance would differ significantly from ratings that were obtained from attending physicians, who have served as the traditional source of evaluation during residency training.
|
METHODS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
Potential parent evaluators were those whose child was assigned to a resident participant. For this study, "parent" refers to any caregiver who primarily was responsible for the child during the hospital stay. Only English-speaking parents were eligible to participate. Potential nurse evaluators were those on the active staffing roster for the ward. No attempt was made to identify how much contact nurses had with a specific resident during a given month. Potential attending physician evaluators were identified from the published monthly call schedule. Attending physicians generally spent at least 2 weeks with each group of residents whom they subsequently were asked to evaluate.
Evaluation Instruments
Evaluation instruments for parents, nurses, and attending physicians were adapted from American Board of Internal Medicine (ABIM) surveys of communication skills and humanistic qualities (available from author on request). The Patient Satisfaction Questionnaire (PSQ) was developed and tested by the ABIM in 1989.27 Internal consistency (Cronbach's 
coefficient) for the PSQ is high ( = .98).28 The PSQ, which was designed for adult patients to rate their physicians, consists of 10 behavior-specific questions. The items relate to being friendly, showing interest, listening carefully, encouraging questions, communicating effectively during the physical examination, being respectful, using plain language, explaining problems, being truthful, and sharing decisions. The PSQ uses a rating format. Patients rate the performance of the physician by choosing among 5 ordinal responses (poor, fair, good, very good, and excellent). This survey tool was chosen as the primary outcome measure for the larger, randomized, controlled trial because of its known psychometric properties and the availability of published data from past applications. We made minor wording modifications to 3 of the 10 items, with the substitution of "your child" for "you" when appropriate so that the PSQ would be applicable to parent raters.
The nurse evaluation also was adapted from an ABIM instrument. It contains different items than the PSQ. Although it has been used by internal medicine residency training programs, its psychometric characteristics have not been published by the ABIM Foundation. The items relate to communicating effectively with staff, being a good team member, treating staff with respect, respecting confidentiality, demonstrating honesty and integrity, responsibility and accountability, completing tasks reliably, timeliness of work, planning the course of care effectively, anticipating postdischarge needs, accepting suggestions, establishing rapport with patients and families, being sensitive and empathetic, and communicating effectively with patients and families. This evaluation was modified to use a reporting format, which asked nurses to "report" whether or how often a particular experience occurred by choosing among 5 ordinal responses (never, rarely, sometimes, usually, or always).
For this study, the questions from both the parent and the nurse questionnaires were combined to create a resident evaluation for use by attending physicians. We made modifications on the parent items so that they were applicable to attending physician raters (substitute "their child" for "your child" when appropriate), and a reporting format (frequency rather than perceived quality of behavior) was adopted. The attending physician items that mirror nurse items are identical in wording, format, and scaling.
Resident participants and parent, nurse, and attending physician evaluators all were surveyed regarding their personal characteristics (gender, age, and race/ethnicity). Unlike parents, who evaluated only 1 resident, nurses and attending physicians may have evaluated multiple residents. For maintaining nurse and attending physician anonymity, this information was collected with each evaluation rather than linked to an individual.
Data Collection Methods
A trained research assistant approached parents using a standardized technique on the day of anticipated discharge or the last day of the resident's rotation. The research assistant informed parents that the evaluations would be reviewed by the resident and a coach to help guide improvement efforts. The parent was informed that the evaluation was anonymous and confidential. Parents who wished to participate gave verbal consent. The name and the picture of the resident who primarily was responsible for the patient during the hospital stay was displayed. The parent was asked whether he or she recognized and was able to evaluate the resident. The research assistant read items to parents and displayed answer options using a laminated card. Parents received no incentive to participate.
Nurses received an e-mail informing them of the study and their potential role in evaluating residents. In addition, the project was discussed at nursing staff meetings, where the evaluation forms were reviewed and questions about the project were answered. Like parents, nurses were informed that the purpose was to help residents identify strengths and areas for improvement. Nurses received no direct incentive to participate, although intermittently throughout the course of the study, food was delivered to the ward break room to thank them for their participation.
In contrast to parents and nurses, attending physician evaluations were solely for research purposes and were not shared with the residents. This was stated explicitly when attending physician participation was requested. Regardless of whether attending physicians chose to complete an evaluation for this study, they still were responsible for completing the standard end-of-rotation evaluation for each resident on their service. Attending physicians received no incentive to participate.
Nurses and attending physicians had the opportunity to evaluate between 2 and 4 (mean: 3) residents per month. On the last day of the rotation, evaluators received an e-mail requesting their participation. There was 1 e-mail for each resident to be evaluated. The name and the picture of the resident was not revealed until after the evaluator completed the demographics section of the questionnaire. Each evaluator had to indicate that he or she recognized the resident and felt comfortable providing ratings. Nurses and attending physicians were informed that the evaluations were anonymous and confidential. Aside from written instructions, evaluators did not receive formal training on how to complete the evaluation. Participants entered responses using a Web-based evaluation instrument.
Analytical Strategy
Dichotomous outcome variables were created, grouping responses into the highest possible response category versus responses in any other category. For each item, we calculated ratings as percentage in the highest response category. For example, if a resident was evaluated by 10 parents and 7 of the 10 marked "excellent" in response to the item related to using understandable language, then the percentage of highest response was 70%. Differences in this percentage were calculated for each resident on items that were common to the parent and the attending physician evaluations and items that were common to the nurse and the attending physician evaluations. These differences were compared using the signed rank test. Parent and nurse evaluations contained different items and therefore could not be compared directly. To facilitate comparisons with previously published studies, we calculated mean scores for each parent item and internal consistency (Cronbach's coefficient) for each evaluation form. SAS version 9.1 (SAS Institute, Inc, Cary, NC) was used for statistical analyses.
Traditionally, results from the ABIM PSQ have been reported as mean scores for each item by assigning a numerical value to each ordinal response (poor = 1, fair = 2, good = 3, very good = 4, excellent = 5).15,28,29 Comparisons that involve mean scores can be difficult to interpret, however, because the measurement distance between ratings of "fair" and "good" may be different from between ratings of "very good" and "excellent."30 For example, someone who has a mean score of 4 may be a more skilled communicator than someone who has a mean score of 2, but the individual is not necessarily twice as skilled. In addition, PSQ ratings are not normally distributed, because they tend to cluster at the high end of the scale.30,31 This phenomenon is common to satisfaction ratings and may result from respondents' finding it difficult to provide negative evaluations regardless of how they feel about their experience. As a result, analysts in the business sector have come to view any response other than the highest possible response as indicating that some element of the experience was not acceptable.32 We applied this rating philosophy through our use of the percentage of highest response.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
|
coefficient) was calculated for each evaluation instrument: parent evaluation ( = .95), nurse evaluation ( = .96), and attending physician evaluation ( = .91).
Parent Ratings
Parent evaluations were available for all 36 resident participants. Residents were evaluated by a median of 10 parents (range: 5–19). Eighty-eight percent (355 of 400) of eligible parents agreed to participate and completed an evaluation. Seven non–English-speaking parents were ineligible to participate in the study. Parents provided the highest possible response ("excellent") between 48% and 67% of the time, depending on the item (Table 2). Mean scores also were calculated for each of the parent items. These ranged from 4.24 to 4.58. These are similar, although slightly higher, than those that were reported in a study in which adult patients used the ABIM PSQ to rate a group of internal medicine residents (4.17–4.24).28 On 3 items, parents indicated that they were "unable to evaluate" the resident >10% of the time. These items related to sharing decisions, explaining problems, and how the resident related to the child and the family during the physical examination.
|
|
Differences Between Attending Physician and Parent Ratings
The item related to sharing decisions with parents was rated significantly higher by parents than by attending physicians (48% vs 28%; P = .03; Table 2). Both parents and attending physicians provided the lowest percentage of highest response on this item. For the remaining items, attending physician and parent responses did not differ significantly.
Differences Between Attending Physician and Nurse Ratings
In contrast to the parent–attending physician comparison, nurse responses on multiple items were significantly lower than those of attending physicians (Table 3). Nurses gave residents lower ratings on items that related to treating staff with respect (69% vs 97%; P < .01), accepting suggestions (56% vs 82%; P < .01), being a good team member (63% vs 88%; P < .01), being sensitive and empathetic (62% vs 85%; P < .01), respecting confidentiality (73% vs 97%; P < .01), demonstrating honesty and integrity (75% vs; 92%; P < .01), and demonstrating responsibility and accountability (67% vs 83%; P = .02). Nurse responses were significantly higher than those of attending physicians on anticipating postdischarge needs (46% vs 25%; P < .01) and effectively planning the course of care (52% vs 33%; P = .03). Attending physician and nurse ratings were similar on items that related to communicating effectively with staff, completing tasks reliably, timeliness of work, and establishing rapport and communicating effectively with the patient and the family.
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
Contrary to our hypothesis, attending physicians who indicated that they observed the resident–patient/parent interactions in question provided responses that were similar to those that were obtained from parents. This differs from an internal medicine study that asked attending physicians to rate residents as they thought the average patient would and then compared the responses with those that were obtained from patients using the same evaluation form.22 Attending physician ratings were not significantly correlated with those of patients (r = 0.26). In our study, parents and attending physicians differed only on how well residents shared decisions with parents, with the parents rating residents higher than attending physicians. It is interesting that this item received the lowest rating from both sources, and both parents and attending physicians frequently were "unable to observe" this behavior. There are several possible explanations for this finding: (1) no decisions were made; (2) residents were not involved in the decisions that were made; (3) the family did not share in the decisions that were made; (4) the types of decisions that were faced were not well suited for a shared decision-making approach; or (5) participants had differing experiences, standards, or expectations regarding the sharing of decisions. Additional research is needed to understand better the preferences and use of shared decision making among families and physicians in pediatric settings.
Attending physicians frequently marked "unable to observe" in response to 5 of the 10 items that mirrored parent items. This may indicate that they did not have the opportunity to observe the behavior in question directly. This seems likely, because these items relate to interactions that might be difficult to observe in the context of traditional rounds. For example, asking, "How often does this doctor ... let parents tell their story; listen carefully; ask thoughtful questions; not interrupt them while they're talking," would require an attending physician to observe the initial interview with a family. To respond to other items, attending physicians might need to observe the resident directly performing a physical examination, educating family members about their child's problem, and counseling a family about diagnostic and/or therapeutic options. In some areas (eg, history, physical examination), direct observation by attending physicians may be important, whereas in other areas (eg, interpersonal skills), parents may be uniquely positioned or simply the best source for the information. Finally, for family education, counseling, and decision sharing, gathering information from a combination of sources may be desirable. To our knowledge, this is the first study in which attending physicians self-reported their inability to observe the resident behaviors of interest on an evaluation. As highlighted by the Accreditation Council for Graduate Medical Education requirements, there is a clear need for resident evaluations to be informed by the direct observation of attending physicians. In the absence of an "unable to observe" category, it is possible that ratings would be provided for unobserved behaviors by extrapolating from resident performance in other domains (eg, presentations skills). Other researchers have addressed this issue by requiring raters to indicate the data source (note review, case discussion, and/or direct observation) that served as the basis for their rating.33 Such methods may promote increased direct observation among attending physicians. Practical strategies are needed to incorporate efficiently direct observation into the daily workflow of busy inpatient services.
Consistent with our hypothesis, we found that nurses commonly rated residents significantly lower than did attending physicians. On the item, "How often does this doctor treat staff with respect; not verbally abusive when under stress," the response of "always" was provided 97% of the time by attending physicians and 69% of the time by nurses. There were 6 additional items for which nurses perceived a lower performance than did attending physicians. Our findings are similar to previous studies in internal medicine settings in which nurses provided significantly lower ratings than did physician raters, especially in the areas of respect, integrity, and responsibility.34,35 This suggests that nurses are observing different behaviors or using different standards or criteria to evaluate residents. Interactions between residents and nurses may be different from those between residents and attending physicians. Standards may vary according to many factors, including personal values and previous experience and expectations.36 Different nurses may place value on different aspects of their interactions with residents. Previous experiences in a particular situation or with a specific resident may influence the impact of subsequent nurse–resident interactions. The reasons for these differences, however, have not been elucidated fully and are an area for future research. Regardless, identifying interactions that are not effective offers an opportunity for improvement that otherwise might go unnoticed.
Although the PSQ is an established tool, it does have limitations. The "poor" to "excellent" rating format that is used by the PSQ was shown during the development of the Consumer Assessments of Health Plans Study to be less favorable than a reporting format (eg, whether or how often a particular experience occurred) for assessment of physician communication skills.37 In addition, respondents are more comfortable reporting about certain aspects of care than they are rating them.38 Moreover, reports of experience are easier to interpret and use in quality improvement efforts. On the basis of the Consumer Assessments of Health Plans Study experience, a reporting format was adopted for the nurse and attending physician evaluations that were used in this study. Use of this format in the attending physician questionnaire, however, introduced scaling differences between the parent and attending physician items that subsequently were compared. We addressed this issue by using the analytic approach to satisfaction surveys that are used by many business organizations: either respondents were "completely satisfied," or they were not. The percentage of highest response captures whether some element of the performance was unacceptable, independent of scale. This approach also addresses the tendency for responses to cluster at the high end of the scale by creating a continuous variable that is distributed more broadly.
This study is limited by inclusion of only parent, nurse, and attending physician raters. Future studies should include self-assessment as well as evaluations by peer residents and adolescent patients. Potential sources of bias include our exclusion of non–English-speaking families. We chose to exclude non–English-speaking families, who constitute <2% of all our hospital admissions, to standardize data collection in a way that would not rely on multiple interpreters. This may limit the applicability of our findings in more diverse settings. Another potential source of bias is the low attending physician response rate. This low response rate may have been attributable to time constraints and their awareness that the evaluation was solely for research purposes. In addition to the time spent completing the end-of-rotation evaluations, attending physicians who participated likely spent 3 to 5 minutes completing each research evaluation. Response bias should be limited by our concealing the identity of the resident being evaluated until after the evaluation was initiated. Personal characteristics of evaluators and resident participants and the health status of the patient may influence responses.16 Although these analyses were beyond the scope of the current study, this is an important area for future research. In addition, qualitative studies are needed to examine (1) whether parents are concerned that providing low ratings will have a negative impact on their child's future care and (2) the meaning that different raters attach to different responses. Future studies should include formal rater training to increase the likelihood that raters apply uniform performance standards.
|
CONCLUSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
|
ACKNOWLEDGMENTS |
|---|
We acknowledge the data collection efforts of Azadeh Namakydoust, who served as a part-time research assistant on this project.
|
FOOTNOTES |
|---|
Address correspondence to William B. Brinkman, MD, Cincinnati Children's Hospital Medical Center, MLC 70353333 Burnet Ave, Cincinnati, OH 45229-3039. E-mail: bill.brinkman{at}cchmc.org
The authors have indicated they have no financial relationships relevant to this article to disclose.
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONREFERENCES |
|---|
This article has been cited by other articles:
 |
|
 |
|
  K. P. Zimmer, B. S. Solomon, G. K. Siberry, and J. R. Serwint Continuity-Structured Clinical Observations: Assessing the Multiple-Observer Evaluation in a Pediatric Resident Continuity Clinic Pediatrics, June 1, 2008; 121(6): e1633 - e1645. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. B. Brinkman, S. R. Geraghty, B. P. Lanphear, J. C. Khoury, J. A. Gonzalez del Rey, T. G. DeWitt, and M. T. Britto Effect of Multisource Feedback on Resident Communication Skills and Professionalism: A Randomized Controlled Trial Arch Pediatr Adolesc Med, January 1, 2007; 161(1): 44 - 49. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||
|
||
Home | My Pediatrics | Journal Information | Current Issue | Past Issues | Subscriptions & Services | Contact Us Click here for faster international access © 2006 American Academy of Pediatrics. All rights reserved. |
||
|
||
