Generating Summaries with Controllable Readability Levels (2023)

Abstract: Readability refers to how easily a reader can understand a written text. Several factors affect the readability level, such as the complexity of the text, its subject matter, and the reader’s background knowledge. Generating summaries based on different readability levels is critical for enabling knowledge consumption by diverse audiences. However, current text generation approaches lack refined control, resulting in texts that are not customized to readers’ proficiency levels. In this work, we bridge this gap and study techniques to generate summaries at specified readability levels. Unlike previous methods that focus on a specific readability level (e.g., lay summarization), we generate summaries with fine-grained control over their readability. We develop three text generation techniques for controlling readability: (1) instruction-based readability control, (2) reinforcement learning to minimize the gap between requested and observed readability and (3) a decoding approach that uses lookahead to estimate the readability of upcoming decoding steps. We show that our generation methods significantly improve readability control on news summarization (CNN/DM dataset), as measured by various readability metrics and human judgement, establishing strong baselines for controllable readability in summarization.

## introduction summaries convey salient pieces of information and their understanding depends on the reader's world and domain knowledge. the readability of a text plays a crucial role in how easily it can be understood and consumed for learning and education. higher readability lowers reading efforts and increases the speed for any reader, and it is particularly beneficial for those who lack high comprehension @xcite . on the other hand, figure 1 : summaries generated with different readability levels using our lookahead method (sec. 3.3). we requested summaries with flesch reading ease (fre) readabilities of 90, 70, 30 (corresponding to the levels of 11-year-old, middle-school and college, respectively). the readability scores of the generated summaries (87.1, 70.3, and 30.9) are close to the requested targets. significant progress has been made in abstractive summarization using large language models (llms) @xcite @xcite . this approach involves making various generation decisions, such as determining which content to paraphrase and how specific a summary should be. the goal is to generate high-quality summaries that are cohesive, readable, and factually consistent. nevertheless, current methods provide limited mechanisms to specify stylistic preferences such as readability @xcite . while readability assessment, which measures the level of difficulty to comprehend a text, is a well-established field within nlp @xcite , the control of readability in natural language generation (nlg) tasks, such as summarization, has not been extensively explored and current readability control performance is low @xcite . while previous work @xcite @xcite focused on binary readability control, such as expert versus lay summaries, we instead focus on generating summaries of various fine-grained reading grade levels @xcite . figure 1 presents an example of three summaries with diverse educational levels of readability generated with our lookahead method (sec. 3.3), given the same document. while the easier-to-understand summary, with the highest flesch reading ease score @xcite , uses simpler words, shorter sentences, and less specialized knowledge, the summary with the lowest score requires the reader to understand more complex sentence structures and words (e.g., "saturated", "cholesterol", "cognitive") and contains more specific details (e.g., "university of illinois researchers"), assuming the readers' familiarity with necessary domain and world knowledge. we study three methods to control the readability of generated summaries: first, we present an instruction-prompting method that prompts the model to output summaries with particular target readability scores or categories, enabling fine-grained control. next, we develop an approach based on reinforcement learning (rl), using proximal policy optimization (ppo; @xcite ) with a novel gaussian-based reward that strongly penalizes significant variations in readability, optimizing for summaries with the desired readability. finally, inspired by @xcite , we propose a readability-aware lookahead decoding method that selects tokens at each decoding step based on the readability of expected future token generations. our contributions in this paper are: (1) we propose three readability-controllable methods for text generation, using instruction-prompting, rl, and lookahead decoding, and (2) show that readability can be explicitly controlled for generating abstractive summaries with finely adjustable levels. finally, (3) we explore the relation between summary readability and aspects such as specificity, abstractiveness, factuality and informativeness (e.g., more readable summaries tend to have lower specificity and informativeness). our results show considerable improvements over gpt3.5, a state-of-the-art approach for controlling the readability level @xcite . ## controllable methods for readability in what follows, we present our three approaches that employ readability metrics (sec. 2.2) for controllable summary generation, based on finegrained instructions (sec. 3.1), reinforcement learning (rl, sec. 3.2), and lookahead decoding (sec. 3.3). figure 2 illustrates the methods. ## experimental setup we evaluate on cnn/dm @xcite , which contains news articles and their bullet-point highlights considered as summaries. in line with previous work @xcite where summaries have similar or easier readability level than the input document, we assess the subset of the cnn/dm test set with documents with a fre score below 50 (high school and college student levels). the readability of the summaries are computed using fre metric and used during training for constructing the instructions for categoryinstruct and scoreinstruct, and as the desired readability r in scoreinstruct+rl and scoreinstruct+la. for categoryinstruct, we map the fre score to the instruction for each reference summary as shown in our models are initialized with flan-t5-large @xcite and fine-tuned using a context size of 1024. we include as baselines gpt3.5 (text-davinci-003) and a flan-t5 model without readability control. as an additional baseline, which we call best-gpt3.5, we sample k summaries from gpt3.5 for each instruction and select the summary whose readability level is closest to the requested level. 3 we evaluate the readability with the fre, gfi and cli metrics (sec. 2.2). we use bertscore @xcite and the f1 measure of rouge-l (rg-l, lin, 2004) for evaluating the summary quality; and factcc @xcite and unieval @xcite for faithfulness assessment. ## conclusion in this work, we propose three methods for finegrained control of the readability level of summaries. we showed that instruction-based methods can be used to guide llms to generate summaries with fine-grained readability degrees. we thus presented a rl approach that uses a gaussian-based reward and a new decoding method that allows to control the readability during inference. we provided an extensive evaluation of our approaches and showed that they significantly improves the control of the summaries' readability. future work includes adapt the methods to different nlg tasks and combine different metrics in order to capture distinct summary aspects that impact readability.