Validation of the Rapid Estimate for Adolescent Literacy in Medicine Short Form (REALM-TeenS)
BACKGROUND: This study was designed to develop and validate a brief adolescent health literacy assessment tool (Rapid Estimate of Adolescent Literacy in Medicine Short Form [REALM-TeenS]).
METHODS: We combined datasets from 2 existing research studies that used the REALM-Teen (n = 665) and conducted an item response theory analysis. The correlation between scores on the original 66-item REALM-Teen and the proposed REALM-TeenS was calculated, along with the decision consistency across forms with respect to grade level assignment of each adolescent and coefficient α. The proposed REALM-TeenS was validated with original REALM-Teen data from a third independent study (n = 174).
RESULTS: Items with the largest discriminations across the scale, from low to high health literacy, were selected for inclusion in REALM-TeenS. From those, a set of 10 items was selected that maintained a reasonable level of SE across ability estimates and correlated highly (r = 0.92) with the original REALM-Teen scores. The coefficient α for the 10-item REALM-TeenS was .82. There was no evidence of model misfit (root mean square error of approximation < 0.001). In the validation sample, REALM-TeenS scores correlated highly with scores on the original REALM-Teen (r = 0.92), and the decision consistency across both forms was 80%. In pilot testing, administration took ∼20 seconds.
CONCLUSIONS: The REALM-TeenS offers researchers and clinicians a brief validated screening tool that can be used to assess adolescent health literacy in a variety of settings. Scoring guidelines ensure that reading level assessment is appropriate by age and grade.
- AAP —
- American Academy of Pediatrics
- DIF —
- differential item functioning
- IRT —
- item response theory
- REALM-Teen —
- Rapid Estimate of Adolescent Literacy in Medicine
- REALM-TeenS —
- Rapid Estimate of Adolescent Literacy in Medicine Short Form
What’s Known on This Subject:
Research with adolescents suggests that many youth have limited health literacy skills. A brief medical word recognition tool would expand the ability to assess relationships between health literacy and health behaviors and outcomes in various settings.
What This Study Adds:
This study documents the development of a short form of the widely used Rapid Estimate of Adolescent Literacy in Medicine. The new Rapid Estimate of Adolescent Literacy in Medicine Short Form offers a valid alternative that can identify youth health literacy limitations in <30 seconds.
According to the only national health literacy survey to date, 1 in 3 adults have low health literacy, indicating that they lack adequate skills to obtain, understand, and use health information and services.1 Numerous studies have indicated limited health literacy is linked to reduced use of preventive services, poor medication adherence, lower self-management of chronic disease, and worse health outcomes in adults.2–5 In recent years, more health literacy research has included adolescent populations, documenting links between health literacy and health behaviors and outcomes, including the use of online disease management tools, medication misuse, and asthma emergency department and inpatient visits.6–8 Yet, adolescent health literacy research remains limited compared with adult populations.
At the same time, there has been a growing demand in both adolescent and adult health care and research settings for screening tools that can be quickly and easily used to assess health literacy. The availability of a brief assessment tool could facilitate health literacy research in multiple settings. In particular, such a tool has great utility for clinical research in time-constrained primary care and specialty pediatric and adolescent clinical settings. Identifying low health literacy in adolescent patients could also be useful clinically because it might signal a need to tailor health information, medication instructions, and follow-up support.
Several health literacy tools have been used in research and practice in the United States, but few have been validated and developed specifically for use with adolescents (Table 1); all have advantages and disadvantages. It is important for researchers and clinicians to select a tool that meets their needs. Currently, the Rapid Assessment of Adolescent Literacy in Medicine (REALM-Teen) is one of the most commonly used tools to assess health literacy for adolescents. This 66-item adolescent health word recognition test was developed using words found in the American Academy of Pediatrics (AAP) adolescent health education materials. The test takes ∼3 to 4 minutes to administer and score.9 Although the tool is not lengthy, it can still require too much time for research studies, especially those conducted in busy clinical settings. The purpose of this study was to develop and validate a short form of the previously validated REALM-Teen (REALM-TeenS).
Study Participants and Data Collection
We used item response theory (IRT) to develop and test the REALM-TeenS using existing data sets from diverse ethnic and socioeconomic cohorts from several distinct geographical, clinical, and community sites. Development data came from 2 sources: a study conducted in New York (sample 1) to develop a self-report health literacy measurement tool11 and a project Ohio (sample 2) studying adolescent health literacy and transition readiness.15 A total of 665 youth 12 to 19 years of age were included.
Sample 1 included 272 English-speaking participants ages 12 to 19 years in upstate New York. Youth were recruited through a pediatrics clinic at a teaching hospital (37%) and from the community (63%) through organizations and schools, flyers, and word-of-mouth.11 The average age of the sample was 15.6 years (range, 12–19 years). A majority of the participants were female (63%) and 93% were born in the United States. Twelve percent received special education services. The sample was racially, ethnically, and economically diverse; 41% were non-Hispanic white, 30% were non-Hispanic black, 18% were Hispanic, and 11% were other, and 45% received a free or reduced lunch. Sample 2 included data from 393 adolescents with special health care needs, ages 12 to 18 years, identified through claims data from Partners for Kids, a pediatric Medicaid accountable care organization covering >300 000 children in central and south eastern Ohio. Half of the sample subjects were girls (51.3%), and the mean age was 16.8 years. Racially, the sample was 63% white and 37% African American. Four percent identified as Hispanic.
Data for the REALM-TeenS validation (n = 174) was extracted from a study on adolescent health risk behaviors, which included youths ages 14 to 19 years recruited from the waiting rooms of 2 Midwestern adolescent medicine clinics.16 Girls made up 72.4% of the sample and the mean age was 16.6 years. The racial distribution was 45.5% white, 50.0% African American, and 4.5% other race. Nearly 10% (9.7%) identified as Hispanic.
All studies were approved by their respective institutional review boards.
The original REALM-Teen assessment is a word recognition literacy test consisting of 66 adolescent-appropriate health words arranged in order of increasing difficulty.9 All words were chosen from AAP materials. Word recognition tests, although not designed to measure comprehension, are useful predictors of general reading ability in English. If an individual has difficulty pronouncing words in isolation, which is a beginning-level reading skill, he or she is likely to have difficulty with comprehension (a higher level skill). An individual’s reading grade level is commonly used as an indicator of their health literacy.17
The REALM-Teen was developed and validated with 1533 6th to 12th graders in 11 clinical, educational, and community sites in 2 states. The tool provides an estimate of each subject’s reading grade range and detects below-grade reading level. Although it can be administered with other measures that assess skills, such as reading comprehension or numeracy, the REALM-Teen is designed to serve as a stand-alone measurement tool.
To create the REALM-TeenS, an IRT analysis was conducted by using IRTPRO.18 Item parameters from the IRT analysis describe how much information each item provides and the relative position of each item according to the easiness or difficulty of the item. It is important that a scale contain items of varying degrees of difficulty so that the scale is informative for respondents all along the scale of interest. The fit of the IRT model to the data was evaluated with standard fit statistics to identify possible item-specific misfit or undue correlations among pairs of items.
When selecting items for a scale, the discrimination and difficulty of each item are considered together. In general, highly discriminating items are most desirable; however, items must be selected all along the scale so that a person’s ability can be precisely estimated.19 The decision to select particular items for the REALM-TeenS was based on statistical properties of the items, as well as the word itself and its appropriateness for the audience, especially when deciding among several words that were similar in difficulty and discrimination.
A differential item functioning (DIF) analysis was conducted on the potential items to determine whether any items functioned differently for different groups. For example, if an item functions differentially for different sexes, a male and female, who are identical in ability (or health literacy) would have different probabilities of correctly responding to the item. This inconsistency is a violation of the IRT assumption that the probability of a correct response is a function of the respondent’s ability only and not another characteristic, such as sex, race, or language spoken at home. DIF analyses were conducted on the following pairs of groups: male and female participants, white participants and those of all other race/ethnicities, Hispanic and non-Hispanic participants, and native English speakers and non-native English speakers. Any items demonstrating DIF were not included in the REALM-TeenS.
The correlation between scores on the original 66-item REALM-Teen and the proposed REALM-TeenS was calculated, as was the decision consistency across forms with respect to the grade level assignment of each child. The coefficient α was also computed for the proposed set of items. The total test information function was also inspected to ensure that the set of items provided adequate coverage across the entire scale of the construct. Finally, a conversion chart, consistent with the original REALM-Teen, was established from raw scores on the REALM-TeenS to the estimated reading level.
Once the list of 10 words was chosen for the REALM-TeenS, the REALM-TeenS was validated with a different sample who had also previously taken the full REALM-Teen. To additionally support the validity of the new REALM-TeenS, it should be noted that the confirmatory sample was based on a very different population from the sample used to develop the REALM-TeenS. The correlation between scores on the REALM-Teen and the new REALM-TeenS was calculated, along with the decision consistency across both forms.
Finally, the newly developed REALM-TeenS was piloted in a convenience sample of 10 teenage patients in an adolescent medicine clinic waiting room to estimate the administration time.
Administration and Scoring
REALM-TeenS administration is based on instructions for the original REALM-Teen. An examiner provides the list of words to an adolescent in a private setting. The examiner asks the youth to read each word aloud beginning with the first word. If they struggle with a word, the examiner can suggest they move to the next word they recognize. Dictionary pronunciation is the scoring standard. The examiner counts a word as correct if the word is pronounced correctly and no additions or deletions have been made to the beginning or ending of the word. For example, a patient who says “adolescence” would not receive credit for the word “adolescent.” Words pronounced with a dialect or accent are counted as correct provided there are no additions or deletions to the word. If a patient indicates that he/she knows the meaning of the word, but is unable to say it, no credit is given. Scoring instructions are discussed in the Results.
Developing the REALM-TeenS
The initial psychometric properties of each item were evaluated before conducting the IRT analysis, including proportion correct, item-to-total correlation, and coefficient α. No items were found to be negatively correlated with the total score nor did any items have all correct responses, so no items were removed from consideration for the REALM-TeenS at this stage.
The IRT analysis revealed that most items were highly discriminating; however, with all 66 items in the model, there was considerable model misfit at both the item level and when evaluating local dependence. This misfit was limited to a few items that were highly correlated to several others. These items were not considered as possible candidates for the REALM-TeenS.
After considering several possible sets of items, a set of 10 items was selected that had adequate spread across the construct scale, maintained a reasonable level of SE across the ability estimates, and correlated highly (r = 0.92) with the original scores on the full REALM-Teen. The coefficient α for the 10-item REALM-TeenS was .82. The items’ IRT-based parameter estimates, along with classic psychometric properties, are shown in Table 2. The coverage of the items can be seen in the total test information curve shown in Fig 1. The spike in the test information curve, about 1.5 SDs below the mean ability, is a function of the large discrimination of the word, “exercise.” In addition, the decision consistency across the 2 forms, with respect to the estimated reading levels, was 72%. However, most of the misclassifications occurred for those who were on the border between 2 reading level categories in the original REALM-Teen. The raw score to estimated reading level conversion is provided in Table 3. Table 4 provides scoring adjustments based on the grade level of the adolescent. Both Tables 3 and 4 can be used for scoring results of the REALM-TeenS.
None of the items on the REALM-TeenS were found to exhibit DIF in any of the 4 separate analyses comparing groups based on sex, race/ethnicity, and language spoken at home. This indicates that the selected words functioned the same for different groups of people. There was no evidence of model misfit for the proposed REALM-TeenS with a root mean square error of approximation < 0.001.
When tested on the validation sample, the REALM-TeenS scores correlated highly with the scores on the original REALM-Teen (r = 0.92). The decision consistency across both forms was 80% for the validation sample. When administered to a convenience sample of 10 youths, the mean administration time for the short form was 13.6 seconds (range, 7.8–23.0 seconds) with no unexpected difficulties.
The REALM-TeenS offers researchers a simplified, validated, and efficient instrument for assessing health literacy in diverse adolescent research settings. The REALM-TeenS can help researchers gain a greater understanding of how adolescent health literacy impacts health outcomes for this important population. Youth make health and wellness decisions using health information from many sources, including interpersonal clinical communication, media, print, and the internet, so their capacity to understand and evaluate health information is important to consider. Research focused on the relationship between adolescent health literacy and health behaviors and outcomes could be used to develop literacy-appropriate materials and interventions designed to improve health outcomes.
Clinically, the REALM-TeenS could be used as a rapid screen for low health literacy to identify adolescent patients who may need clinicians to slow down, write plain language take-home messages, and use teach-back to confirm understanding. However, current recommendations suggest using a universal precautions approach, which means providing easy-to-understand information to all patients.20 Given the time limitations on clinic visits and mixed support for standard screening,21 we do not envision REALM-TeenS being used as a routine assessment tool for clinic visits. Still, clinics could use the tool for time-limited data collection to determine aggregate literacy if there is a desire to go beyond a universal precautions approach. Findings might inform the development of intake and patient forms and handouts, the selection of appropriate patient education materials, and the identification of technologies to recommend, such as smartphone applications that are user-friendly for low-literacy populations.
Before deciding to screen adolescents for below–grade-level literacy, health professionals need to consider where patients will be tested, who will do the testing, and how they will be trained, as well as how the results will be used and documented. For some adolescents, the administration of a literacy test in a health care setting may feel like a school test and create anxiety or stress. Even adults with low literacy often try to hide their problem.17,22,23 Screeners should be sensitive to and aware of this possibility when administering the tool.
The REALM-TeenS is a valuable addition to the health literacy measurement landscape because it allows for an assessment of health literacy in <1 minute. Additionally, it offers an adolescent analog to the brief adult tool (Rapid Estimate of Adult Literacy in Medicine–Short Form)24 that can allow for additional exploration of the relationship between parent and adolescent health literacy using similar measures. The development of the REALM-TeenS mirrors the development of the brief versions of adult health literacy assessments, including the Rapid Estimate of Adult Literacy in Medicine–Short Form and the Short Test of Functional Health Literacy in Adults, which are now widely used and help support the expansion of health literacy research.24,25
Our study has limitations. Data for the development of this tool were extracted from 3 larger studies on adolescent health literacy, and there was not a single approach to eligibility, sampling, data collection, or recruitment. Although all of the studies implemented the REALM-Teen in a manner consistent with its documentation, variations in the overall context of data collection could yield differential responses across studies. Although we note these differences as a limitation, we also believe that the diversity of the youths included in our samples strengthens the generalizability of our findings. The item-level DIF analysis demonstrated that each item in the REALM-TeenS functioned the same when the following groups were compared with each other: males and females, the white population and those of all other race/ethnicities, Hispanics and non-Hispanics, and native English speakers and non–native English speakers. This finding indicates that group membership did not influence the likelihood of a correct response. Also, our item-reduction approach was built on the development of the original REALM-Teen, which was validated in 11 clinical, educational, and community sites in 2 states, and, therefore, all of the individual words included in the REALM-TeenS were thoroughly tested in a sample larger than those used in this study. The generalizability of the REALM-TeenS is thus supported by the original work that developed the REALM-Teen, and the IRT analysis presented in this study, which included 2 additional states and new clinical, community, and health plan–based recruitment sites. The main threat to generalizability is the limitation of our samples to youths who could complete the REALM-Teen in English. Additional work will be needed in limited–English-proficiency populations to determine whether a language-specific tool is needed, such as the Short Assessment of Health Literacy for Spanish-speaking Adults, which has not been validated with adolescents.26
The REALM-TeenS may be useful for researchers in various settings, including busy clinics. The availability of this shorter health literacy assessment tool may increase the likelihood that researchers will include a health literacy measure in studies, even when literacy is not the primary focus. Future research is needed to continue to investigate the relationship between health literacy and adolescent health behaviors and outcomes.
- Accepted February 16, 2017.
- Address correspondence to Jennifer A. Manganello, PhD, MPH, Department of Health Policy, Management, and Behavior, University at Albany School of Public Health, One University Pl, #165, Rensselaer, NY 12144. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This work was supported by funding from multiple grants, including the following from the National Institutes of Health: National Institute of Child Health and Human Development grant 1R03HD059757-01A1, National Institute on Minority Health and Health Disparities grant 1R01MD007160-01A1, National Institute on Drug Abuse grant 5R01DA026312-02, and National Center for Advancing Translational Sciences grant UL1TR001070. Drs Davis and Arnold were supported in part by grant 1 U54 GM104940 from the National Institute of General Medical Sciences of the National Institutes of Health, which funds the Louisiana Clinical and Translational Science Center. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Funded by the National Institutes of Health (NIH).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest relevant to this article to disclose.
- Kutner M,
- Greenberg E,
- Jin Y,
- Paulsen C; US Department of Education
- Bennett IM,
- Chen J,
- Soroui JS,
- White S
- Nielsen-Bohlman LPA,
- Kindig DA
- Valerio MA,
- Peterson EL,
- Wittich AR,
- Joseph CL
- Davis TC,
- Wolf MS,
- Arnold CL, et al
- Clayman ML,
- Pandit AU,
- Bergeron AR,
- Cameron KA,
- Wolf MS
- American Academy of Pediatrics
- American Academy of Family Physicians
- American College of Physicians-American Society of Internal Medicine
- Chisolm DJ,
- Sarkar M,
- Kelleher KJ,
- Sanders LM
- ↵IRTPRO for Windows [computer program]. Version 3.3. Lincolnwood, IL: Scientific Software International; 2011
- Hambleton RK,
- Swaminathan H,
- Rogers H
- Agency for Healthcare Research and Quality
- US Department of Health and Human Services
- Copyright © 2017 by the American Academy of Pediatrics