OBJECTIVE: To measure the impact of an objective evaluation of provider Spanish-language skills on self-reported language proficiency and comfort using Spanish in a range of clinical scenarios.
METHODS: We enrolled pediatric residents with any self-reported Spanish language ability from 3 residency programs. Participants completed a baseline survey, objective language testing, and a posttest survey. We gathered demographics, self-reported Spanish ability, and comfort using Spanish in various clinical scenarios, which were grouped and analyzed by degree of complexity. Between surveys, a language testing service administered a 20-minute, telephone-based assessment of general Spanish proficiency. Scores were reported on a scale from 1 to 12, with scores ≥9 designated “proficient.” Participants received a numeric score and brief qualitative feedback on their language ability.
RESULTS: Following testing, residents (n = 76) were significantly less likely to report comfort using Spanish in straightforward clinical scenarios, from 64% to 51% (P = .007). That difference was accounted for entirely by residents who tested at a non-proficient level (56% to 39%, P = .006). Testing had no impact on comfort using Spanish in complex or medical-legal scenarios, at any proficiency level. We found no change in self-reported Spanish proficiency in any resident group.
CONCLUSIONS: Objective Spanish-language testing decreased nonproficient resident comfort using Spanish in straightforward clinical encounters, but it did not change comfort in complex or legal scenarios. In combination with education and enforceable policies, language testing may play an important role in decreasing nonproficient Spanish use and improving care for patients with limited English proficiency.
- communication barriers
- Hispanic Americans
- physician-patient relations
- LEP —
- limited English proficiency
What’s Known on This Subject:
Providers who speak Spanish, regardless of their proficiency level, may use Spanish for clinical care without seeking professional interpretation. Failure to use professional interpretation increases the risk for miscommunication and can lead to patient harm.
What This Study Adds:
Providing residents with objective feedback on Spanish language proficiency decreased willingness to use Spanish in straightforward clinical scenarios. Language proficiency testing, coupled with institutional policies requiring professional interpretation, may improve care for patients with limited English proficiency.
The US population is becoming increasingly diverse. In 2006, 13.7% of US children aged <5 lived in a home where a parent or guardian spoke English less than “very well.”1 Medical providers are challenged with ensuring effective communication even when providers and patients have discordant language skills.
Language barriers negatively affect medical care, including patient satisfaction2,3 and adherence,4,5 cost,6,7 medical errors,8–11 and risk of litigation.12 Professional interpretation has been shown to improve communication and quality of care.13–16 In contrast, the common use of family members or bilingual staff as ad hoc interpret-ers carries significant risk of miscommunication from interpretation errors.9,13,17,18
Many US medical providers have some ability to communicate in a language other than English.19,20 Patient-physician language concordance is associated with improvements in patient satisfaction, health care delivery, and outcomes,5,21–24 but no standards exist for determining what degree of nonnative language proficiency is sufficient to qualify as language concordant.25,26 Providers who are not proficient may fail to use professional interpretation, which increases the risk of significant miscommunication.3,27,28 Residents routinely use less-than-proficient second-language skills in clinical settings.29–33
Our aim was to measure the impact of an objective evaluation of Spanish language ability on self-reported language proficiency and comfort using Spanish in a range of clinical scenarios. We hypothesized that providers with lower levels of Spanish proficiency would be reluctant to deliver care in Spanish after objective feedback on language proficiency, particularly in situations with complex medical and legal implications.
This was a multisite, prospective study of a language proficiency testing intervention among pediatric residents recruited from 3 programs: University of Washington/Seattle Children’s Hospital (Seattle, WA), Johns Hopkins School of Medicine (Baltimore, MD) and Children’s Mercy Hospitals and Clinics (Kansas City, MO). This study was approved by institutional review boards at each site.
Pediatric residents were notified of the study opportunity and enrolled between March and June 2010.
Residents were sent an e-mail (n = 243) inviting those speaking any Spanish to participate. Eighty-four residents (35%) completed the baseline survey. Six (7%) were unable to schedule the language test and were excluded from analysis. An additional 2 (2.4%) completed testing but not the follow-up survey; they are included in the baseline results (n = 78) but not in the pretest-posttest comparisons (n = 76). Residents who participated in any portion of the study received a small gift of chocolates.
Written consent was obtained at the time of enrollment. Participants were not informed of the study hypothesis but also were not blinded. Each person completed a baseline survey containing demographic information, self-reported Spanish proficiency, and self-reported comfort using Spanish in standardized clinical scenarios. Within 1 to 3 weeks, participants took a 20-minute, telephone-based test of spoken proficiency and comprehension of Spanish. Scores were confidentially shared with each participant several days later. Participants completed a follow-up survey within 2 weeks, reporting comfort using Spanish in the same clinical scenarios.
Aggregate demographic data from participating residency programs (academic year 2009–2010) were obtained from program coordinators.
Participants completed an online survey with 13 questions regarding demographics (gender, year of residency training, age, self-reported race/ethnicity) and second-language skills. Questions were developed for this study and pilot-tested before use. Participants rated their ability to speak about health and pediatric medicine in Spanish (rudimentary, basic, conversational, proficient, and fluent). Next, respondents rated their comfort using Spanish in 14 clinical scenarios of varying complexity, grouped into 4 categories: no clinical content (eg, introducing yourself), straightforward care (eg, normal newborn care), complex care (eg, transition to comfort care), and medicolegal content (eg, medical error disclosure). Responses ranged from 1 (“very uncomfortable”) to 5 (“very comfortable”); the numbers in-between were not assigned descriptors. Survey questions and scenarios are listed in Appendix.
Language Proficiency Test
After the baseline survey, each participant took a validated telephonic oral proficiency test, the ALTA Spanish Speaking and Listening Assessment (www.altalang.com/language-testing/medical.aspx). This standardized test of general Spanish contained no medical content (unavailable at the time) and was not customized for this study. Each test was conducted by a native Spanish speaker and consisted of 12 questions selected from a pool of 120 items. Each question was chosen to evaluate ≥1 expression or comprehension skills. The test lasted 15 to 20 minutes and cost the sponsoring hospital $50.
Tests were recorded and scored by a trained native Spanish speaker, educated at the university-level in a Spanish-speaking country. Evaluations were based on general vocabulary, grammar, speaking speed, sentence structure complexity, and ability to convey ideas. Evaluators had no information about participants except their name and organization. Scores were reported on a scale from 1 (no knowledge of the language) to 12 (native speaker) that can be mapped to other language proficiency test scales, such as the Interagency Language Roundtable scale.32 On the basis of previous research33 and detailed descriptions provided by ALTA,32 a score of ≥9 was considered “proficient,” and a score of 11 or 12 as “highly proficient.” ALTA defines 9 as the level at which grammatical errors are no longer likely to cause misunderstanding, and 11 as near-native communication abilities. Because few individuals scored in the proficient and highly proficient ranges, we treated highly proficient as a subgroup of proficient.
Scores and brief qualitative evaluations were reported by ALTA to members of the study team. Results were shared confidentially with each resident. Individual scores were not disclosed to program directors or other faculty supervisors. Participants had an opportunity to provide feedback on the language test, both verbally at the time of score receipt and in a semistructured format on the posttest survey.
Following receipt of the score, participants were e-mailed a link to a posttest survey. Residents were asked to reassess their spoken Spanish proficiency and comfort using Spanish in clinical scenarios described above. Participants were asked if the test altered their general comfort in speaking Spanish with patients and families (no change, more comfortable, or less comfortable) and if they would change their frequency of interpreter use (no change, more frequently, less frequently, or unsure). We sought feedback on the testing tool, asking if they felt it was a fair test (yes or no, followed by space to explain their answer) and if they would be willing to have their language skills formally assessed by their hospital (yes or no).
Demographic data for participants and the all-resident cohort were analyzed with basic descriptive statistics.
Comfort Using Spanish in Clinical Contexts, Before and After Testing
Responses to each clinical scenario were dichotomized, with a Likert score of 4 or 5 labeled as “comfortable” using Spanish in a scenario and all other responses labeled “not comfortable.” We dichotomized participant responses within each scenario and by scenario type: no clinical content, straightforward clinical content, complex clinical content, and medicolegal content.
In our primary analysis, if the respondent answered that he or she was “comfortable” in any scenario within a category, that category was labeled “comfortable” in the analysis. We compared the proportion comfortable using Spanish within each category of clinical scenarios in the baseline and follow-up surveys by using the paired t test. Changes after testing were evaluated for all participants, as well as by self-reported proficiency level (rudimentary, basic, conversational, proficient, or fluent) and tested proficiency subgroups (highly proficient, proficient, and nonproficient). We dichotomized participant responses to evaluate changes across the threshold from comfortable to uncomfortable because these seemed most likely to affect actual use. Given that any clinical use of nonproficient Spanish poses a potential risk for miscommunication, we chose a low threshold for defining comfort within a category (ie, a positive response for any scenario in the category).
In a secondary analysis, we assigned participants a point for each scenario in which they reported comfort on the dichotomized Likert scale, then evaluated points per category before and after objective language proficiency testing with the paired t test. This additional analysis aimed to provide a more detailed look at changes within categories of clinical Spanish use.
We evaluated responses to direct questioning about anticipated changes in clinical Spanish and interpreter use with descriptive statistics and χ2 analysis for comparison between tested proficiency groups.
Self-Reported Spanish Proficiency, Before and After Testing
To evaluate changes in self-reported Spanish proficiency after testing, self-reported levels were assigned numeric value (1 through 5, corresponding to levels from rudimentary to fluent), and the pretest and posttest mean proficiency scores were compared with the paired t test. We evaluated changes in the overall study population and in subgroups broken down by tested proficiency and by pretest self-reported proficiency levels.
Participant Response and Feedback
Participant willingness to be formally tested and report of test fairness were evaluated with descriptive statistics. Responses by tested proficiency level were compared by using Fisher’s exact test. Written and verbal feedback were grouped into themes and are summarized in the next section.
Across the 3 institutions, 78 pediatric residents participated and completed language testing; 76 completed all study procedures. Participant characteristics are detailed in Table 1 and are similar to those of the overall residency program profiles.
Changes in Comfort and Self-Assessed Ability After Language Testing
Tested language proficiency scores ranged from 2 to 12. Twenty-four percent of participants scored at a level considered proficient (9–12), and 6% were highly proficient (≥11). Mean ALTA test scores by self-reported proficiency are given in Table 2, along with reported comfort before and after testing.
Table 3 shows pretest-posttest comfort using Spanish in clinical scenario groups. After language testing, providers were significantly less likely to report comfort using Spanish in straightforward clinical scenarios (64% pretest to 51% posttest, P = .007). Among tested nonproficient providers, comfort using Spanish in straightforward clinical scenarios decreased from 56% to 39% (P = .006). No significant changes occurred among nonproficient residents in nonclinical, complex clinical or legal situations. Testing was not associated with changes in comfort using Spanish in any type of scenario for providers who were proficient or highly proficient. Highly proficient residents reported comfort using Spanish in all types of clinical scenarios both before and after testing (data not shown).
After receiving objective results, 39% of residents with ALTA scores in the “not proficient” range were still comfortable using Spanish in straightforward clinical encounters, 18% in complex medical encounters, and 11% in situations with potential legal implications.
Secondary analysis showed no significant change after proficiency testing in the mean number of scenarios within each category in which residents reported comfort, either overall or stratified by tested proficiency group. In the straightforward clinical category, residents who tested at a nonproficient level (n = 57) reported comfort, on average, in 1.4 of 5 scenarios before testing and 1.3 of 5 after testing (P = .39). Although the means did not differ significantly, the distribution is informative: the percent who reported comfort in none of the scenarios rose from 44% before testing to 61% after testing, and those who reported comfort in only 1 scenario dropped from 28% to 7%. Comfort in 2 or 3 scenarios increased from 10% to 14%, and comfort in 4 or 5 scenarios remained unchanged at 18%. The remainder of the data are not shown.
In the follow-up survey, 29% percent of participants stated that they were more likely to use a professional interpreter after testing, whereas 64% reported no intended change, and 7% were unsure. This finding did not vary by tested proficiency level.
Objective language testing did not significantly alter self-reported Spanish proficiency among participants overall or in any subgroup defined by tested or self-reported proficiency level (data not shown).
Resident Perceptions After Language Testing
Participants felt that the ALTA assessment was a fair test of their Spanish proficiency (83%), and most would be willing to have such formal assessment by their hospital or employer (81%). These results did not vary by tested proficiency level (data not shown). Respondents felt the assessment was not onerous, but investigators noted difficulties in scheduling each test, requiring multiple reminder pages for busy residents. Most residents felt the test would be improved by including medical content, and many felt that they would have scored better had their medical Spanish been tested. A few participants using mobile phones reported difficulty hearing the tester, but there were no other technical difficulties.
In this multisite study, we found that residents not proficient in Spanish were less likely to feel comfortable providing clinical care without an interpreter in some settings once their proficiency was objectively measured and reported. Given that nonproficient residents are at high risk for miscommunication when providing care without interpretation, these findings suggest that objective testing might be a useful tool in a comprehensive strategy to decrease the risk of miscommunication by altering provider comfort with providing care in another language.
The impact of objective testing on resident self-assessment and comfort with clinical Spanish was less marked than hypothesized. In the category of scenarios in which comfort changed, most of the change occurred among residents initially reporting comfort in only 1 scenario, suggesting they were already less comfortable than their colleagues. We found no change in resident comfort in medically complicated and potentially legal scenarios, even among those testing as nonproficient. This is concerning given the potential for increased patient harm and physician liability should miscommunications occur.
Language testing did not alter self-reported Spanish proficiency or anticipated interpreter use. We previously reported that a substantial number of residents inaccurately assess their Spanish skills when compared with objective testing (K.C.L., D.A.T., J.D.C., E.M., S.A.R., R.F.H., E.F.K., J.F., B.E.E., unpublished data, 2011). In this study, of 18 residents who reported themselves to be proficient or fluent, 33% tested at a “not proficient” level, and 78% tested at a “not highly proficient” level (K.C.L., D.A.T., J.D.C., E.M., S.A.R., R.F.H., E.F.K., J.F., B.E.E., unpublished data, 2011). Our finding that self-assessment rarely changed, even when inconsistent with objective testing results, may help explain the modest impact of language testing. In particular, it provides a potential explanation for the finding that testing did not change nonproficient residents’ comfort using Spanish in complex or medicolegal scenarios. Those who believed themselves proficient before testing continued to trust their abilities, regardless of test results. In addition, many providers reporting clinical Spanish use did so despite self-reported lack of proficiency. Confirmation of lack of proficiency did little to change comfort or anticipated behavior for this group.
A possible explanation for the trend toward increased comfort in complex scenarios among nonproficient residents is that some may have obtained a better numerical score on the proficiency test than expected, and so testing may have left them feeling more comfortable using Spanish rather than less. An alternative approach might provide only qualitative feedback to nonproficient residents to emphasize that current proficiency is unsuitable for clinical care. Future research should explore the impact of such an approach.
We were interested that residents testing as proficient were not universally comfortable using Spanish in complex and legal scenarios, either before or after objective testing. We suspect this reflects the broad range of scores used to define proficiency, such that a level 9 speaker may not feel confident conducting a nuanced discussion of a sensitive subject. As has been found in other studies of physician self-assessment,34,35 less skilled providers were more likely to overestimate their proficiency and report an inappropriately high degree of comfort in complex scenarios, whereas more skilled but not fully fluent providers tended to rate their comfort in such scenarios more conservatively relative to their actual skill.
Most residents found testing to be efficient and straightforward and were willing to be formally assessed by their hospital.
Our results are consistent with previous reports that pediatric residents at all levels of Spanish proficiency, including those self-identifying as nonproficient, provide care directly to limited English proficiency (LEP) families without professional interpretation, even when hospital policies require interpreter use (K.C.L., D.A.T., J.D.C., E.M., S.A.R., R.F.H., E.F.K., J.F., B.E.E., unpublished data, 2011).29–,31 There remain barriers to changing provider use of interpreters, including lack of awareness of hospital policy, time constraints, and failure to appreciate the importance of effective 2-way communication even when no complex medical discussion is anticipated (Yolanda N. Evans, MD, MPH, S.A.R., E.M., B.E.E., unpublished data, 2011).30 Without clear guidelines and incentives, busy providers are unlikely to adopt behavior that is seen as time-consuming or unnecessary, even in the face of objective feedback.
Our study had several limitations. We asked residents to report comfort using Spanish and anticipated behavior change, rather than directly observing their behavior. Another important limitation was the lack of medical content in the ALTA Speaking and Listening Assessment. General proficiency and comprehension are necessary but not sufficient to provide safe medical care. Most participants felt they would have performed better had the test contained medical content. Without measuring skill in those aspects of vocabulary and cultural nuance necessary for successful medical communication, however, we cannot predict whether our results would have differed, and if so, how. Since completing this study, a test instrument with medical content has been developed and is being adopted by large clinical groups.36 A test including specific medical content, in addition to measures of general fluency, would be preferable when introducing a hospital-wide policy.
Our sample size was relatively small, which may have limited our ability to detect posttest differences, especially among proficient and highly proficient subgroups. Generalizability may be limited by the fact that participants were at medium-to-large academic pediatric training programs with ready access to professional interpreter services. Finally, our definition of proficiency was based on the best information available, but consensus is lacking on how proficient is proficient enough. Future research should focus on defining the threshold for nonnative language proficiency.
The US Department of Health and Human Services’ National Standards on Culturally and Linguistically Appropriate Services mandate that all organizations receiving federal funding “assure the competence of language assistance provided” to LEP families “by interpreters and bilingual staff.”37 In addition to the moral imperative to ensure effective communication with patients and families, a legal imperative also exists. The challenge to hospitals and residency programs will lie in how they comply with the mandate.
Our results demonstrate that proficiency testing decreases provider comfort using nonproficient Spanish in some clinical settings. As the number of providers who speak Spanish as a second language grows, proficiency testing may play an important role in a multifaceted approach to ensuring the competence of language services provided. Combining such testing with clear, enforceable policies and increased education may help to ensure high-quality, safe, equitable care for LEP patients and families.
Text of self-reported proficiency question and the clinical scenarios for which participants were asked to rate their comfort using Spanish with patients and families, grouped by analysis category. On the survey, the scenarios were not grouped by category.
How would you describe your ability to speak in Spanish about health and pediatric medicine?
Please rate your level of comfort in providing the following types of care in spoken Spanish on a scale of 1 to 5, with 1 meaning very uncomfortable and 5 meaning very comfortable:
Introducing yourself and making small talk with a patient’s family
Giving a family verbal directions to the cafeteria
Asking about a child’s pain overnight
Discussing the initiation of care for a wheezing child before the interpreter arrives
Midnight rounds with a family who speaks some English
Discussing treatment of constipation in a hospitalized patient
Answering brief questions about healthy newborn care
Discussing the initiation of care for a wheezing child with midline thoracic scar before the interpreter arrives
Discussing concern for intimate partner violence with a patient’s mother
Discussing a transition to comfort care for a child in the ICU
Clinical Scenarios With Legal Implications
Discussing risks and benefits with a family who is refusing the recommended treatment
Consent for peripherally inserted central catheter line placement
Consent for biopsy of a lymph node
Discussing a medication error without clinical consequence
- Accepted September 15, 2011.
- Address correspondence to Beth E. Ebel, MD, MSc, MPH, Associate Professor of Pediatrics, University of Washington, Director, Harborview Injury Prevention & Research Center, 325 Ninth Ave, Box 359960, Seattle, WA 98104-2499. E-mail:
All authors made substantial intellectual contributions to the study conception and design. Funding was provided by Drs Ebel, Thompson, and Cowden. Drs Lion and Ebel performed data analysis and interpretation; Drs Lion, Thompson, Cowden, and Ebel were responsible for manuscript drafting and critical revisions; all authors approved the final manuscript for submission.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Project support was provided by the Robert Wood Johnson Foundation grant 65127 and the Seattle Children’s Hospital Center for Diversity, Seattle, Washington, the Johns Hopkins Hospital and Johns Hopkins Bayview Hospital, Baltimore, Maryland, and Children’s Mercy Hospitals and Clinics, Kansas City, Missouri.
- ↵US Census Bureau. American Community Survey 2006. Available at http://factfinder.census.gov/servlet/DatasetMainPageServlet?_program = DEC&_submenuId = &_lang = en&_ts =. Accessed June 21, 2010
- Cohen AL,
- Rivara F,
- Marcuse EK,
- McPhillips H,
- Davis R
- Flores G,
- Laws MB,
- Mayo SJ,
- et al
- Divi C,
- Koss RG,
- Schmaltz SP,
- Loeb JM
- ↵Quan K, Lynch J. The High Costs of Language Barriers in Medical Malpractice. Berkeley, CA: National Health Law Program School of Public Health, University of California; 2010
- Flores G
- ↵Laraque D, Mendoza F, Dreyer B, Frintner MP, Cull W. Cross Cultural and Linguistically Appropriate Care: Pediatric Resident Preparedness. Presented at the 2010 Pediatric Academic Societies Annual Meeting, Vancouver, BC; 2010
- Ferguson WJ
- Burbano O’Leary SC,
- Federico S,
- Hampers LC
- ↵Oral Language Proficiency Testing. Atlanta, GA: ALTA Language Testing Services, 2003
- ↵ALTA Language Testing Services. Clinician Cultural and Linguistic Assessment. Available at www.altalang.com/language-testing/ccla.aspx. Accessed August 12, 2011
- ↵Office of Minority Health. National Standards for Culturally and Linguistically Appropriate Services in Health Care. Washington, DC: US Department of Health and Human Services Office of Minority Health; 2001
- Copyright © 2012 by the American Academy of Pediatrics