 |
|
 |
|
 |
 |
|
 |
 |
 |
 |
 |
 |
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ARTICLE |
Department of Pediatrics, Johns Hopkins University, Baltimore, Maryland
 |
ABSTRACT |
|---|
 TOP ABSTRACT PATIENTS AND METHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
METHODS. Pediatric residents in a large continuity clinic participated between August and December 2005. The continuity-structured clinical tool included items in the following domains: opening the interview, history taking, relationship skills, personal manner, negotiation or management, and physical examination. Each resident was directly observed during 1 entire patient encounter. Parents, preceptors, and residents completed evaluations by selecting 1 of 4 possible responses for performance of each item. We dichotomized responses as "yes" versus "no/partial" and analyzed aggregate scores for individual items and domains among the 3 evaluators by McNemar test, percentage agreement, and interoberserver agreement (
).
RESULTS. Fifty-four of 57 eligible residents had all 3 evaluations completed. Parents rated residents the highest and showed least variability (only 2 items for which parents indicated the task was completed in <90% of the encounters). Residents rated themselves the lowest (35 items with <90%). In comparing the residents and preceptors, the domains that had the lowest percentage of agreement were history taking (range: 61%–91%) and negotiation or management (range: 51%–88%). All of the evaluators scored residents the lowest in the domain of negotiation or management, with the following lowest-score items: probe for decision-makers, assess willingness and barriers, and use of visual aids.
CONCLUSIONS. Compared with parents, residents and preceptors demonstrated greater variability in resident performance evaluations. All of the evaluators scored residents lowest in the domain of negotiation or management during continuity-clinic visits. Residency programs should strongly consider emphasizing skill development in this area.
Key Words: multiple-observer assessments • resident education • continuity-clinic experience • continuity-structured clinical observations • CSCO
Abbreviations: ACGME—Accreditation Council for Graduate Medical Education • OSCE—objective structured clinical examination • CSCO—continuity-structured clinical observation • CI—confidence interval
In 1999, the Accreditation Council for Graduate Medical Education (ACGME) began a long-term initiative focused on restructuring the way residency programs assess their residents in 6 general competencies: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. The ACGME now requires residency programs to fully integrate the competencies into resident assessment. To assist programs in this process, the ACGME and the American Board of Medical Specialties created a toolbox of assessment methods,1 which includes the objective structured clinical examination (OSCE) and the multiple-observer assessment (with which several evaluators complete a survey to gather summative information on several topics). In pediatrics, the use of an OSCE has been described by Carraccio and Englander2 in which residents were directly observed during simulated patient encounters with standardized patients. Lane and Gottlieb3 developed the structured clinical observation, which involved a series of brief direct observations of medical students for a portion of a real patient encounter by a preceptor. Although OSCEs can be standardized, implementation requires great cost and preparation time, because they involve the development of cases and standardized patient training. Direct observation of real patient encounters, however, is thought to provide similar reliability to OSCEs, yet encompass a greater breadth of problems by which to evaluate4 and to be more cost-effective.3
Few published studies have used multiple observers to directly evaluate learners, and only recently has 1 study included both medical educators and patients or families.5 There are few reports on the use of multiple-observer assessment in graduate medical education using the same instrument and even fewer regarding integration of family feedback on resident performance.1,4 Although family input is rarely included in resident evaluation, there is a growing body of research emphasizing the importance of family and patient-centered communication.6–9 Studies have also shown that building parental rapport is essential to promote continuity of care.10,11 A recently published report emphasizing the importance of using a family approach in pediatrics12 highlights the need to begin to understand how parents perceive their resident physicians and how to best incorporate family input in the resident evaluation process.
The objectives of our study were to (1) develop, implement, and evaluate a standardized continuity-structured clinical observation (CSCO) tool completed by preceptors, residents, and parents (family members) during 1 directly observed, complete encounter in a continuity-clinic setting, (2) compare CSCO evaluations among preceptors, residents, and parents, and (3) identify major strengths and areas needing improvement in resident performance during patient encounters in a continuity clinic.
|
PATIENTS AND METHODS |
|---|
|
TOPABSTRACTPATIENTS AND METHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
|
|
|
1 prompted a review of the item, rewording when required for better clarification and approval by the preceptors. These revisions were completed, and the tool was retested before the initiation of the study. We developed the "parent" evaluation tool (parent CSCO) and the self-evaluation tool (resident CSCO) with similar items and domains to directly compare resident skill observed by the 3 different evaluators. In this study, "parent" represented any primary caregiver or guardian who accompanied the child to the clinic for a visit with their primary care provider. The wording of the parent CSCO was modified to ensure that it was at a fifth-grade reading level. In addition, 4 items were removed from the parent CSCO because they were thought to be difficult for parents to understand and evaluate: "followed with open-ended questions," "avoided leading questions," "ensured accuracy of content of information," and "prioritized order of examination based on age of the child."
For all 3 of the evaluation tools, there were 4 possible responses to each of the items regarding completion of a behavior. "No" defined as "almost never," "partial" as "about half of the time," "yes" as "majority of the time," and not applicable. The responses were scored based on a 3-point, Likert scale, where 0 indicated "no," 1 indicated "partial," and 2 indicated "yes," whereas "not applicable" was not given a numerical value.
The project was piloted in May 2005 to assess overall study flow, as well as resident feedback, on the evaluation tools and process. The pilot included 8 graduating senior residents and 4 preceptors (2 senior residents in 4 continuity-clinic groups). In addition, 10 individual parents were asked to read the parent CSCO and to identify confusing items. After the pilot was complete and modifications were made, the final parent CSCO was implemented.
Study Population
Fifty-seven pediatric residents were assigned to the Harriet Lane Clinic for continuity clinic during the time of the study. The Harriet Lane Clinic is a large, urban, general pediatric clinic that serves as the medical home for 
9000 children and adolescents. Approximately 94% of patients are black, and 82% are enrolled in Medicaid for health insurance coverage. Each resident is assigned as the primary care provider to a continuity panel of patients for their 3 years of training.
There were a total of 10 half-day clinic sessions per week with 10 faculty or fellow preceptors (each assigned a specific clinic session) throughout the year. In addition, each preceptor supervised the same group of residents throughout the year. Resident groups included trainees in all 3 levels of training in the afternoon sessions and second and third year residents in morning sessions.
Data Collection
The CSCO project was implemented biannually (September to December 2005 and March to June 2006). This article focuses on resident performance outcomes from the first observation that occurred in fall 2005. For all of the observed visits, residents were the patient's primary care provider.
Each resident was directly observed for 1 entire patient encounter (a well-child or follow-up visit) during a continuity-clinic session. To reduce selection bias, direct observations and evaluations typically occurred during a visit early in the clinic session (preferably, the first visit). This was also a time when preceptors were most available. Just before the visit, the resident informed the parent that his or her preceptor would be in the room observing the resident.
Residents asked the parent to complete the parent CSCO at the end of the visit to prevent the parent from paying more attention to particular components of the visit than usual. The resident read a scripted paragraph when describing the purpose of the parent CSCO to ensure that parents received consistent information. If the parent agreed, the resident handed the parent an evaluation form on a clipboard to be completed after the resident left the examination room. Parents were asked to hand their clipboards to the registration staff before leaving clinic. After the observation, the preceptor and resident completed their individual CSCO evaluation forms. Preceptors and residents were asked to complete their evaluations directly after the clinic visit or at least by the end of that day's clinic session for that day.
To provide anonymity to the parent who was involved in the direct observation, parents from the next 2 consecutive unobserved visits were also asked to complete a parent CSCO. The additional parent CSCOs usually occurred on the same day but sometimes occurred during a future session, depending on patient volume. To provide additional anonymity, there were no patient identifiers on the parent CSCO forms.
After all of the evaluations were completed (parent CSCOs, resident CSCO, and preceptor CSCO), the preceptor removed the "age of patient" from the parent CSCO, compiled responses, and reviewed them directly with the resident. The original evaluation forms were kept in a locked cabinet that only the primary investigator and research assistant could access. Copies of the evaluations were given to the program director's office for the resident's educational portfolio. A research assistant entered all of the CSCO responses into a secure database. Only the parent CSCO that was used in the direct observation was used in the current study.
Data Analysis
For all of the analyses, we dichotomized responses as "yes" (2) or "no/partial" (0 and 1) and analyzed scores for individual items and domains. A response of "not applicable" or a blank response was treated as "missing" data.
First, the distribution of responses among each of the evaluators for each item and domain was analyzed. For each item, we tallied the percentage of evaluator types (ie, preceptor, resident, and parent) that answered the question "yes." These scores were pooled across all of the participating evaluators.
To assess for consistency and differences among evaluators, comparisons among groups were conducted by a matched-pair analysis with resident as the unit of analyses to assess whether each of the evaluators had the same perception of that resident's performance. For individual items and domains, we compared responses of preceptor to resident, preceptor to parent, and resident to parent. Because some surveys were missing responses to some items, it was not possible to compare all 3 of the evaluators on each item.
We examined similarities in responses using percentage of agreement with confidence intervals (CIs) and values. Although was calculated, it was less reliable for data with this nonuniform distribution.14 In addition, we examined the significance of the different responses by using McNemar's test. SAS 9.1 (SAS Institute Inc, Cary, NC) was used. The project was determined to be exempt by the Johns Hopkins Institutional Review Board.
|
RESULTS |
|---|
|
TOPABSTRACTPATIENTS AND METHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Parents consistently scored the residents highly, whereas preceptors and residents had greater variability. Parents rated residents highest and showed least variability (only 2 items where parents indicated that the task was completed in <90% of encounters). Residents rated themselves lowest (35 items with <90%); preceptors scored the residents <90% on 28 items. Figure 1 shows sample items from 1 domain (relationship-skills management) that displays the variability among each of the evaluators. This pattern was seen across each of the domains.
|
|
|
|
|
DISCUSSION |
|---|
|
TOPABSTRACTPATIENTS AND METHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
It is known that physician interviewing style can either enhance or impede medical care, as well as satisfaction with care.15 Assessment of communication skills has become a major focus of medical education, and many evaluation tools have been proposed.4 We chose to combine the use of a multiple-observer assessment tool with a structured clinical observation of a full patient visit in a continuity-clinic setting. The inclusion of a family evaluation in a multiple-observer tool was a novel component that has not been studied previously. As a result, the evaluators were focusing on the same encounter, and the feedback became more specific and less summative.
Overall, parents scored residents highest and demonstrated the least amount of variability in rating the residents' performance. In contrast, residents routinely scored themselves the lowest and were most critical of themselves. The 2 domains where residents and preceptors had the greatest discrepancy in assessing resident performance were in the areas of history taking and negotiation and management, with the latter having the lowest scores.
The doctor-patient interaction remains 1 of the most crucial areas of a patient encounter. Techniques to enhance the relationship may also improve compliance with a medical regimen.16
Historically, there has been a great deal of focus on how to elicit a thorough medical history with an emphasis on communication skills.17,18 In 1987, Roter and Hall17 described how physicians only elicited slightly >50% of the medical information considered important. In 1999, Marvel et al19 described that physicians solicited patient concerns as often as 75% of the time but only allowed the patient response to be completed 28% of the time. Brinkman et al5 reaffirmed that parents and nurses significantly differ in their assessment of communication skills and support the concept that a multiple-observer evaluation adds depth to resident assessment. Our results show that the residents performed far better in this realm in the continuity-clinic setting. In these areas, all 3 of the evaluators scored residents highly. Specifically, residents were able to elicit parent concerns and use open-ended questions in >90% of the observed encounters.
Other domains were not as reassuring. In the domain of negation of management, "uses visual or written aids" and "probes for decision-makers" were the 2 lowest-scoring items overall. This is a concern both from an educational and a medical standpoint. Historically, social science researchers have debated whether high-quality care requires solicitation of patient perspectives along with negotiation toward an agreed-on plan.20 Studies have shown that if parents understand and agree with a plan, there is a greater likelihood of adhering to recommendations and improve medical outcomes.21–23
In this study, the majority of participating residents did well in eliciting the concerns of parents, but there was variability in perceptions of skill for engaging parents in discussions or "negotiations" of the plan. Our results were consistent with a study in adult medicine that characterized the consent process in a primary care office.24 Braddock et al24 found that physicians routinely failed to provide adequate information for informed decision-making. Specifically, physicians in this study rarely assessed the patient's understanding of the decision. Another study found that visit closure is a "distinctive phase" and, given the frequency of new problems that arise at the end, suggests that physicians may have the potential to improve their effectiveness in this area.25,26 In addition, understanding perspectives from those who assume care of the child, as well as the use of visual aids, may help with patient compliance. This is a concern because there may be additional influential individuals who do not attend a clinic visit but help to make decisions for that child's care.
In our study, compared with parents and preceptors, residents were far more critical of themselves in all of the domains, resulting in a low percentage of agreement. Residents may be less objective and perhaps more self-effacing. As residents graduate from a training environment, it is important to develop the skill of self-assessment. The American Board of Pediatrics has strongly encouraged lifelong learning and education to provide the highest quality of care to patients. As stated by the American Board of Pediatrics, "one must seek to learn from errors and aspire to excellence through self-evaluation and acceptance of the critiques of others."27 Although self-assessment was not valued as highly as parent or preceptor feedback by the resident,28 it is a necessary skill and should continue to be encouraged.
Although there was a lack of variability in the parents' overall ratings, the lowest parent scores also came in the area of negotiation and management. However, given the narrow range of parent responses, there needs to be continued work to develop a tool that may more accurately reflect parent opinion. There are a few possible reasons for the disproportionately high ratings from parents. Although the form and process were piloted, parents may not have fully understood items on the evaluation tool, and no standardization took place. Also, whereas residents read the "standardized paragraph," which informed that the evaluation was anonymous and that their responses would not impact their child's care, parents could have responded with artificially inflated scores because of fear of retribution for lower scores. It is essential that we are meeting the needs of our patients. Thus parents should have the opportunity to provide feedback to physicians about their clinical performance and interviewing styles. For many of our residents, this was the first time they had received any systematic feedback from the families, and it was considered the most important.28 Thus, more effort is needed to improve the parent feedback tool to increase its utility.
The limitations of this study are that it was conducted at a single site, which may limit generalizability to other continuity clinics, and residents may behave differently during an encounter when they know that they are being observed. In addition, with use of 1 site, we had a small number of residents. However, the tool that we developed may serve as a foundation for future work in this area. Our CSCO tool was developed from other validated assessment tools but had not been separately validated. There were a few limitations related to the implementation of this project. This took additional time and sometimes required extra preceptor staffing to ensure that other resident's questions could be answered in a timely manner while the preceptor was in a room observing another visit. However, when the observation was conducted as the first visit of the day, the additional time and precepting was minimized. Lastly, whereas demographics and cultural background are additional factors that may influence communication, we deidentified the parent CSCO forms to keep the evaluations anonymous. As such, we are unable to look at patient specific factors that may have influenced our findings.
|
CONCLUSIONS |
|---|
|
TOPABSTRACTPATIENTS AND METHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
|
ACKNOWLEDGMENTS |
|---|
We are indebted to the pediatric residents, continuity preceptors, and patients and families in the Harriet Lane Clinic who participated in this project.
|
FOOTNOTES |
|---|
Address correspondence to Karen P. Zimmer, MD, MPH, Johns Hopkins University, Department of Pediatrics, 200 North Wolfe St, Baltimore, MD 21287. E-mail: kzimmer1{at}jhmi.edu
The authors have indicated they have no financial relationships relevant to this article to disclose.
| What's Known on This Subject Structured clinical observation is a recommended method for assessing resident performance. To date, most research involving direct observation has utilized simulated patients to assess medical student performance with only a few published reports of attendings and families assessing residents.
|
| What This Study Adds This is the first study using multiple observers (preceptors, parents, and resident self-assessments) to evaluate resident performance of a full patient encounter in continuity clinic. Multiple observer assessment can provide insight into residents’ strengths and areas for improvement..
|
|
REFERENCES |
|---|
|
TOPABSTRACTPATIENTS AND METHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||
|
||
Home | My Pediatrics | Journal Information | Current Issue | Past Issues | Subscriptions & Services | Contact Us Click here for faster international access © 2008 American Academy of Pediatrics. All rights reserved. |
||
|
||
