Assessing Materials

Featured Resource :

Teaching Patients with Low Literacy Skills, Second Edition by Doak, Doak & Root, 1996.

updated July 2024

Assessment Tools

We have many tools at our disposal to help us assess materials. Reading Grade Level (RGL) is a common assessment often used in the education field to rank the level of a book for children. However, RGL is considered a fairly superficial assessment of information presented to the public. There are several tools and many guidelines available to help us make a broader and more meaningful assessment of written health materials. At the same time, a RGL tool offers an initial measure and serves well for the process of rewriting exiting texts because it helps highlight complex words [such as utilize]. This offers the reviewer an opportunity to substitute a more common and more easily accessible word [such as use]. As is noted below, tools such as the Suitability of Materials [SAM] tool moves beyond a look at vocabulary to examine important qualities of a text that ease or hinder reading and comprehension. Health information or data is often presented through a visual display. A unique tool such as the PMOSE/IKIRSCH enables one to rate the quality of such displays, lists, or charts, or graphs.

Reading Grade Level [RGL] Assessments: These tools offer some insight into the level of difficulty of materials written in English. Most reading grade level assessment tools focus on the vocabulary [generally assessed by number of syllables in words] in a text. The focus on the number of syllables is of particular use in English because of the frequency of silent letters in long words. Note that this makes a reading grade level tool inappropriate for many other languages – especially when all letters in a word are pronounced.

Several widely available tools enable us to rate the reading grade level of a text. The formulas fall into two major categories: those that are computer generated and those that are calculated by hand. The most frequently used computerized formulas include the Flesch Grade Level Formula and the Flesch-Kincaid Index. Word processing programs often have the Flesch formula built into the software for easy use. For those who conduct readability tests in the field, there are a number of formulas that can be calculated by hand. Hand-calculated formulas include the FOG method, Fry formula and the SMOG. The SMOG was created by McLaughlin in 1969 and uniquely considers sentence length as well as word length and offers a stronger predictor of comprehension. The Health Literacy Studies Group chose the SMOG in its research because of this factor. In addition, we have found the formula easy to compute and well designed for field work. Once you know the formula, you do not need to carry any charts for reference.

The SMOG, G. McLaughlin. Click here for an overview of the SMOG and directions for use. There is also an online calculator. This Java application calculates a SMOG score for any length of text that is typed or pasted in. It can be accessed at http://www.harrymclaughlin.com/SMOG.htm.

Notes: Multi-syllabic words in the English Language are considered an indicator of vocabulary difficulty [e.g. utilize instead of use]. Poor readers often need to sound out words and longer words present a bigger challenge. In addition, vocabulary development is based on exposure to a topic and on background knowledge. Long sentences in any language often contain phrases set aside by commas, multiple ideas, and/or lists. Think of these phrases as side comments that may distract a poor reader from the main idea of the sentence. On the other hand, these phrases may be important and so deserve a sentence of their own. Very poor readers struggle with reading because they are still sounding out words and may not be able to derive meaning from even short sentences because of the slow pace required for sounding out individual words. Furthermore, they have problems with long sentences because they can lose the main idea part way through the sentence.

Assessment Tools Beyond Reading Level Assessment

We highlight two tools here: The SAM, developed by Doak, Doak, and Root and the PMOSE/IKIRSCH developed by Mosenthal and Kirsch. The SAM has inspired many subsequent [and shorter] materials assessment tools. The PMOSE/IKIRSCH is the first tool to focus on documents (documents are defined as information presented in displays such as lists, graphs, charts).

Suitability of Materials [SAM]: Grade-level readability is only one of many factors that contribute to the overall readability of materials. Even materials written on a low grade level may be difficult to comprehend if proper attention is not also paid to organization, layout, and design. To address the overall suitability of materials, including reading grade level, Doak and Doak developed the Suitability Assessment of Materials (SAM). Although the SAM was developed for use with print materials, it has also been used to assess video- and audio-taped instructions to patients.

The SAM, Doak, Doak and Root. The SAM, developed by Len and Ceci Doak, offers a detailed approach for assessing texts. The SAM enables reviewers to move beyond mere readability assessments and consider the many important aspects of materials such as organization format, design and culture-that ease or hinder reading, comprehension and use. Information on SAM can be found on pg 51-60 of the Doak, Doak and Root book Teaching Patients with Low Literacy Skills, Second Edition.

The SAM scores materials in 6 categories: content, literacy demand, graphics, layout and typography, learning stimulation and cultural appropriateness. The SAM yields final percentage score. This score falls into one of three categories: superior, adequate or not suitable. The SAM can be used to identify specific shortcomings that reduce the suitability of materials- either in the development stages or with existing materials. The tool also provides a strong guideline for developing materials – with attention paid to each of the 6 categories. A full description of the SAM and a scoring sheet is available in Doak, Doak & Root’s Teaching Patients with Low Literacy Skills, Second Edition, J.B. Lippincott Company, 1996.

PMOSE/IKIRSCH: Readability of Charts and Graphs:

A good deal of health directions and information is presented as data in lists, graphs, or charts. The readability tools noted above are used for prose documents – information presented through sentences organized in paragraph format. However, adults must often grapple with data displays or what literacy researchers call ‘documents’. Document literacy was one of the measures on the international assessments of adult literacy conducted amongst a wide swath of nations. Documents include forms, tables, graphs, charts, and lists. The PMOSE/IKIRSCH is a tool that measures the readability of information organized in rows and columns or general displays.

The PMOSE/IKIRSCH Document Readability Formula, P. Mosenthal and I. Kirsch.

This innovative assessment tool focuses on documents-materials in the form of lists, charts and graphs. Unlike prose, which is full sentences, documents provide different challenges based on format. This formula offers a scoring mechanism for assessing documents.Researchers Mosenthal and Kirsch developed a measure for assessing document complexity, called the PMOSE/ IKIRSCH document readability formula (Journal of Adolescent & Adult Literacy, 41(8), May 1998). The formula uses the number of rows and columns, the structure, and the number of labels and items to assess the chart or table. Scores range from Level 1 to Level 5 Proficiency. The Proficiency Level can be translated into a grade-level equivalent.

The Health Literacy Studies Group uses the PMOSE/ IKIRSCH in research and finds it to be a useful tool. Attention to information presented in rows and columns is especially critical given the large number of forms and charts needed to navigate health care. It is important to note that the formula does not examine the vocabulary (such as long, technical words) that may be used in charts and tables. A separate commentary or notation is needed to capture this quality of the text.

Additional Tools in Common Use

A variety of tools and guidelines have been developed more recently to aid in the assessment of health materials for factors related to health literacy. Descriptions and links to two of these are provided below.

The CDC Clear Communication Index, CDC’s Office of the Associate Director for Communication. This 20-item research-based index is intended for use in the development and assessment of public health communication materials that are clear and can be easily understood by the public. The tool includes a focus on organization of key ideas and on the important public health concept of risk. The CDC has developed an Index Widget, which can be incorporated into other websites.

The Patient Education Materials Assessment Tool (PEMAT), S. Shoemaker, M. Wolf and C. Brach, Agency for Healthcare Research and Quality. The PEMAT was developed by the AHRQ to allow for the systematic assessment of the understandability and actionability of various patient education materials. Two different versions of the PEMAT exist: one for the evaluation of printable materials and one for the evaluation of audiovisual materials.