Medical Text Simplification: Optimizing for Readability with Unlikelihood Training and Reranked Beam Search Decoding (2023)

Abstract: Text simplification has emerged as an increasingly useful application of AI for bridging the communication gap in specialized fields such as medicine, where the lexicon is often dominated by technical jargon and complex constructs. Despite notable progress, methods in medical simplification sometimes result in the generated text having lower quality and diversity. In this work, we explore ways to further improve the readability of text simplification in the medical domain. We propose (1) a new unlikelihood loss that encourages generation of simpler terms and (2) a reranked beam search decoding method that optimizes for simplicity, which achieve better performance on readability metrics on three datasets. This study’s findings offer promising avenues for improving text simplification in the medical field.

## introduction in recent years, text simplification has become an increasingly useful application of ai @xcite particularly in healthcare @xcite @xcite , where text can be technical and difficult to understand. by automating this process, we can help healthcare professionals explain key medical texts (e.g. doctor's reports, findings) to patients. previous work in text simplification in medical domain has explored use of pretrained language models @xcite @xcite @xcite @xcite , reinforcement learning @xcite , and zero-shot prompting @xcite . despite this progress, simplification sometimes results in the generated text having lower quality and diversity @xcite . further as we find some simplification models copy sentences from the source, and thus remain do not sufficiently improve the readability (see appendix b). in this work, we seek to further improve medical text simplification. we first propose a new unlikelihood loss that penalizes words in proportion to their reading level using a well-established readability index. second, we propose a modified beam search method at decoding time to rerank intermediate candidates based on their readability. despite simplicity, our methods improve readability based on automated metrics (up to 2.43 points on flesch-kincaid) and human evaluation, while maintaining similar performance in terms of factual consistency and overall simplification. we make the following contributions: (1) we propose a new form of unlikelihood loss based on well-established readability index to improve medical text simplification (2) we propose a decoding strategy that optimizes for readability in medical text simplification (3) we provide evaluation results for previous state-of-the-art on three datasets in terms of readability and factual consistency. we make our code publicly available at https://github.com/ljyflores/ simplification-project . related work text simplification research primarily focuses on sentence-level @xcite @xcite @xcite , with some attempts at paragraph or document-level datasets @xcite . most datasets have been sourced from accessible wikipedia or news articles, which are already quite accessible. however, the medical field, laden with technical jargon, can greatly benefit from simplification. initial methods in medical text simplification employed lexical and syntactic techniques @xcite , while recent work includes finetuning language models like bart @xcite and a two-stage summarizethen-simplify approach @xcite . medical simplification has also expanded to multilingual settings @xcite . in this work, following devaraj et al. (2021) we use unlikelihood (ul) training @xcite to encourage the generation of simplified terminology. this strategy has been used in other domains to penalize inaccuracy @xcite , complexity @xcite , and redundancy @xcite ) in text generation. unlike @xcite , our work adapts ul to optimize for both readability and factual consistency. to improve simplification, we also intervene at the decoding stage. previous work uses modified decoding methods to address factual inconsistency @xcite @xcite , or optimize fluency and diversity in text generation @xcite . our work extends this by optimizing the decoder for readability in medical text simplification. ## methods we propose two simple but effective approaches for improving medical text simplification, one during the training phase, and the other during decoding. specifically, we propose a modified unlikelihood loss @xcite to incorporate readability index and encourage the model to favor the generation of simpler words. then, we introduce a decoding approach that evaluates and re-ranks the candidate beams by considering both readability and factuality. we detail these approaches below: ## experiments datasets we run our experiments on three datasets: cochrane @xcite consists of 4,459 pairs of abstracts from the cochrane database of systematic reviews and their corresponding summaries written by domain experts. medeasi @xcite consists of 1,697 pairs of human-annotated sentences sourced from the merck manuals @xcite and simpwiki @xcite . finally, the radiology reports dataset foot_0 consists of 2,269 radiology reports collected from a large urban hospital and simplified by medical residents and doctors. baselines we compare against a bart-xsum @xcite ) model which we further finetune on our datasets, and state-of-the-art models by @xcite ; @xcite , all of which we fine-tune on each of the three datasets; we chose bart-xsum to align it with previous work, in order to provide an apples-to-apples comparison and isolate the impact of our methods. we also compare with state-of-the-art large language model gpt-4-0314 (openai, 2023) foot_1 . evaluation metrics we evaluate the readability, consistency, and overall performance as follows: for readability, we use the standard fk @xcite and ari scores @xcite , which use the average word and sentence length to estimate the complexity of texts. for factual consistency, we use bertscore (zhang et al., 2020) and gpt-eval @xcite (see appendix d), as these correlated well with human judgement @xcite @xcite . for gpt-eval, we evaluate 50 summaries, and report the fraction of samples in which a factual inconsistency was found. we additionally use sari (xu et al., 2016), an edit-based metric for text simplification, and rouge-lsum @xcite for overall fluency. ## results we fine-tune a bart model using our methods and present the results in effect of unlikelihood loss and decoding on cochrane and radiology, our proposed methods achieve better readability scores in terms of fk and ari. in particular, combining unlikelihood loss with the decoding strategy achieves a 2.43/1.74 point improvement in fk/ari upon the next best model for cochrane, and a 0.12/0.17 point improvement for radiology. note that in the radiology dataset, the sentences are typically short, resulting in a lower (better) baseline readability score. see sample comparison of outputs in appendix b. on medeasi, our methods slightly underperform napss @xcite . we find that it sometimes generates phrases instead of full sentences, which lowers fk/ari, since these scores depend on sentence length. in contrast, our models generate complete sentences, which improve fluency at the cost of worse (i.e. higher) fk/ari scores. our methods generally improve over the prior sota in terms of sari and bertscore, however, interestingly on the radiology dataset all methods underperform a fine-tuned bart model. we observe that using ul or the decoder indi-vidually results in fewer hallucinations than both bart-ul @xcite and napss @xcite on radiology, and against napss on medeasi. when the baseline models perform well, we find that it is because they tend to copy information from the input, and hence are less prone to hallucinations. in contrast, our strategies force the model to use simpler words and not copy the input, but may introduce inconsistencies with the source. we confirmed this with an experiment: we compute the % 4-gram overlap of the model written summaries with the source, and observe that large portions of previous works' output is copied from the text, whereas output in our models are not (see note that some of the identified hallucination errors are relatively minor as we find gpt-eval to be very strict. for example the phrase "26 selftreatments of 26 chinese herbal medicine prescriptions" is found to be factually inconsistent with the source having the phrase "26 self concocted chinese herbal compound prescriptions" by gpt-eval (see ## conclusion in this paper, we propose methods to improve simplicity in medical text simplification; this improves the readability of generated summaries, and achieves comparable bertscore and sari scores. however, hallucination remains a challenge.