Context-Aware Automatic Text Simplification of Health Materials in Low-Resource Domains (2020)

Abstract: Healthcare systems have increased patients’ exposure to their own health materials to enhance patients’ health levels, but this has been impeded by patients’ lack of understanding of their health material. We address potential barriers to their comprehension by developing a context-aware text simplification system for health material. Given the scarcity of annotated parallel corpora in healthcare domains, we design our system to be independent of a parallel corpus, complementing the availability of data-driven neural methods when such corpora are available. Our system compensates for the lack of direct supervision using a biomedical lexical database: Unified Medical Language System (UMLS). Compared to a competitive prior approach that uses a tool for identifying biomedical concepts and a consumer-directed vocabulary list, we empirically show the enhanced accuracy of our system due to improved handling of ambiguous terms. We also show the enhanced accuracy of our system over directly-supervised neural methods in this low-resource setting. Finally, we show the direct impact of our system on laypeople’s comprehension of health material via a human subjects’ study (n=160).

## introduction healthcare practices have granted patients increased access to their health information to support self-care @xcite . but, the benefits have been hindered by patients' low comprehension of their own health data @xcite , as a study shows that readability measures of online health information is significantly higher than patient health literacy abilities @xcite . moreover, older adults, the largest demographic group interacting with the healthcare system, are often the least health-literate @xcite . with low levels of health literacy resulting in worse health outcomes @xcite , there is an urgent need to reduce the gap between the health literacy of patients and the health literacy demands of healthcare systems. accordingly, we explore text simplification methods for health text, taking medication instructions as a use case (see this mismatch in patient literacy levels and health documents is due in part to the differing language used by healthcare professionals and patients @xcite . for example, what professionals refer to as "po", patients might refer original: take 50 mg po daily for 2 days. hold for @xmath0. gold simplification: take 50 milligrams by mouth daily for 2 days. hold for systolic blood @xmath1. dr. babel fish: take 50 milligrams orally daily for 2 days. hold for systolic blood @xmath2. to as "by mouth". while previous works have addressed this by performing local word replacement @xcite , their context-free frameworks lacked the accuracy. in a health document, "mg" could mean "milligrams" or "magnesium", and harnessing the contextual information, for example in "take 50 mg" or "mg reacts with", aids accurate simplification. our approach is a context-aware medical text simplification system, named dr. babel fish (dbf). we design our system to be independent of the availability of annotated datasets as scarcity of such data is expected due to privacy and proprietary concerns. to compensate for annotated datasets, we instead rely on a structured knowledge base in the form of the unified medical language system (umls) @xcite . taking inspiration from the modular and context-aware frameworks of phrase-based statistical machine translation (pbsmt) systems @xcite , our system, dbf, first identifies hard (low frequency) words such as "sbp", then collects possible simplifications of these words from the umls such as {systolic blood pressure, and serotonin-binding protein}, and finally chooses the simplification that best reflects patients' preferred medical terms and best fits the context ("systolic blood pressure" in this case), by relying on a patient language model trained on a suitable monolingual corpus. although neural machine translation (nmt) frameworks @xcite constitute the state-of-the-art, they suffer in the low-resource settings of the clinical (medical) domains, and we accordingly present our system to complement neural methods in domains lacking the appropriate parallel corpus. although we take medication instructions as a use case, our system is general enough by construction, to handle any medical text. all code and materials associated with this study are released to the public foot_0 . this paper makes the following contributions: ## previous work efforts to improve patient comprehension of health information in the biomedical informatics community can be categorized into: developing standards @xcite , curating dictionaries @xcite , annotating text with additional information @xcite @xcite , normalizing terms @xcite , syntactic simplification @xcite @xcite , and finally, lexical simplification @xcite @xcite . our work belongs to the final category. one popular previous attempt @xcite of health material text simplification relies on the consumer health vocabulary (chv) @xcite for mapping the hard term to its simpler counterpart, disregarding context information. other word-replacement systems @xcite have relied on metamap @xcite to map medical terms to their simpler counterparts by either utilizing chv as a thesaurus @xcite , or relying on an inhouse equivalent resource (codemed) @xcite . although, metamap performs word sense disambiguation (wsd) by relying on the context, its creators admit its low wsd quality @xcite . therefore, we rely on a language model instead of metamap. nonetheless, since metamap followed by a chv (or another dictionary) is a popular method in previous works, we include it as a baseline in our experiments. beyond health materials, lexical simplification is highly researched. a thorough survey of this field is present in @xcite , which divides work in this field into four stages of a pipeline: (1) complex word identification, (2) substitution generation, (3) substitution selection, and (4) substitution ranking. we also perform complex word identification as a first stage by relying on word frequencies, which is a popular method among previous work @xcite @xcite . we also generate substitutions as a second stage by relying on umls, similar to how previous work relied on word taxonomies @xcite . the last two stages are performed in one shot in dbf, where instead of finding which candidate substitutions fit the context and then select the simplest based on a certain metric, we let the language model decide which is the most probable substitution in terms of meaning and simplicity. our work is the first work to combine these stages in a context-aware method tailored for the healthcare domain. finally, text simplification has been modeled previously as a machine translation task where parallel corpora are available @xcite @xcite . accordingly, we compare against these methods in this study to assess their capacity in the low resource setting and their capability to generalize across healthcare domains. ## methods in this section, we describe our method, dbf, along with the established baselines it was quantitatively evaluated against: metamap+chv, seq2seq-w-attention, and pointer-generator. ## experiments this section describes the results of two studies. the first study uses automated evaluation metrics to assess dbf's output in comparison to the baselines considered. the second study evaluates the impact of dbf on laypeople comprehension. we first describe the data used in our experiments and then present the results. ## discussion to better understand the functioning of the knowledge-aware systems, we study the effect of f t on their first stage of identifying hard words. upon tuning the systems on the validation dataset, f t was set to 672 for both systems coincidentally. based on figure 5 , we deduce that around 18% of words in the original instructions were attempted for translation, reflecting a high recall of hard words. along the same lines, we check the effect of f t on dbf and metamap+chv in terms of the two evaluation scores (see figure 6 ). in terms of pinc scores, we observe an expected pattern of increase as we increase f t for both systems. as f t increases, both systems attempt to modify more of the original sentence (including easy words) leading to a lower overlap with reference sentences. metamap+chv introduces more novelty as we increase f t since dbf can retain easy words even if they were identified as hard, unlike metamap+chv. as for sari scores, we observe the significance of tuning the first stage, where too low of an f t results in reduced performance due to lack of attempted translations (low pinc scores), and too high of an f t results in reduced performance due to translating easy words. moreover, we observe consistent enhances in performance for dbf over metamap+chv for all f t considered. moving our attention to the effect of k and r on the performance of dbf, we show in figure 2 , dbf's sari score when varying k and r from 1 to 5, and fixing f t to 672. our first observation is a positive trend as we increase k, particularly for @xmath3. this shows the aptitude of the language model at selecting the best translation even when faced with a plethora of options given equal trans-lation probabilities. as for r, we notice reduced performances for any r value different from 1. we thus conclude that the model we use for estimating translation probabilities is not benefiting translation quality. moreover, the ranking of the concepts returned by the umls search function has insignificant value, when an appropriate language model is present. ## conclusion despite significant efforts to keep patients informed of their health condition, the gap in health literacy remains an issue, calling for a necessary text simplification system to bridge the gap. in this work, we suggest a context-aware framework to ensure high accuracy in such a critical domain, while also showing its positive impact on comprehension through a human subjects' study. we also conclude that, while supervised nmt methods are well-suited for the task, several healthcare subdomains lack suitable parallel corpora, which limits the performance of these supervised methods. to overcome this, we offer a knowledge-aware text simplification system to robustly operate in a low-resource setting. looking forward, we see great potential in adapting deep learning architectures to utilize the rich and highly curated content of umls, and exploring better methods or implementations to copy easy words to the target side. these two adaptations would make the high performing supervised methods of nmt less dependent on direct supervision and more generally applicable to the multiple healthcare domains. this would also address the limitation of dbf to word-level simplifications.