I nfo L oss QA : Characterizing and Recovering Information Loss in Text Simplification (2024)

Abstract: Text simplification aims to make technical texts more accessible to laypeople but often results in deletion of information and vagueness. This work proposes InfoLossQA, a framework to characterize and recover simplification-induced information loss in form of question-and-answer (QA) pairs. Building on the theory of Questions Under Discussion, the QA pairs are designed to help readers deepen their knowledge of a text. First, we collect a dataset of 1,000 linguist-curated QA pairs derived from 104 LLM simplifications of English medical study abstracts. Our analyses of this data reveal that information loss occurs frequently, and that the QA pairs give a high-level overview of what information was lost. Second, we devise two methods for this task: end-to-end prompting of open-source and commercial language models, and a natural language inference pipeline. With a novel evaluation framework considering the correctness of QA pairs and their linguistic suitability, our expert evaluation reveals that models struggle to reliably identify information loss and applying similar standards as humans at what constitutes information loss.

## introduction technical texts, many of which exist in high-stake domains (e.g., medicine), are often written in a language incomprehensible to lay readers. improving the accessibility of such texts may help address wider social issues, e.g., disinformation (unsdg, 2021) and access to higher education @xcite . automatic text simplification that rewrites text into plain language may therefore be a technology for good. with the adoption of llms, document-level text simplification has significantly progressed in recent years @xcite @xcite . original: in this study, the benefit of preoperative nutritional support was investigated for non-small cell lung cancer patients who underwent anatomic resection. [...] patients who were malnourished, diabetic or who had undergone bronchoplastic procedures or neoadjuvant therapy were excluded from the study. simplification: this study looked at if eating a protein-rich diet before surgery could help lung cancer patients recover more quickly after surgery. [...] we didnʼt include patients who were already not eating well, had diabetes, or had received other treatments for their lung cancer. lossy! 2 identify information loss and recover it in reader-focused qa (this paper) (i) non-small cell lung cancer patients; (ii) bronchoplastic procedures or neoadjuvant therapy (i) lung cancer patients; (ii) other treatments ... simplified into ... q: what kind of lung cancer do these patients have? a: non-small cell lung cancer, which is a group of lung cancers that behave similarly. non-small cell lung cancer, compared to small cell lung cancer, is most common. q: what patients were not eligible for the study? a: patients who had bronchoplastic procedures (surgical repair of a part of the lung) or neoadjuvant therapy (therapy given before the main treatment). simplification is an inherently lossy process: even when done by professional editors, the resulting plain language tends to lose details and become more generic @xcite , and some content is omitted @xcite . the over-simplification of content-including excessive deletion and vagueness-may lead to reduced comprehension @xcite or in its worst case to misinterpretations and factual errors @xcite . therefore, we consider the following question: how can we characterize information loss and help readers recover what is lost in an intuitive and understandable manner? this paper presents a question generation and answering framework to characterize and recover information loss during simplification in a readercentric way. inspired by the theoretical work of "potential questions" @xcite and expectationdriven questions under discussion (qud, @xcite , information loss is characterized in the form of pragmatically felicitous ques-tions rooted in simplified texts, without assuming access to the original text. information loss is then recovered by answering these questions in plain language, based on the original (technical) text (figure 1 ). informed by qud, the questions are curiosity-driven and information-seeking, and are subject to well-defined linguistic constraints @xcite . while prior work used factoid qas to describe specific factual changes @xcite , the qud framework suggests higher-level questions, and allows us to characterize general yet subtle language changes pervasive in factually equivalent texts. these include lower specificity @xcite and greater degrees of vagueness @xcite . we envision the resulting qa pairs being directly incorporated in interactive simplification tools @xcite . contribution 1: task formulation and dataset. we first introduce infolossqa, a linguistcurated dataset grounded in the scenario of lay readers digesting simplified abstracts of randomized controlled trials (rcts) in medicine. we focus on rcts because they combine important and illustrative properties: they use highly technical/inaccessible language; lay audiences often having keen interest in their content; and misinterpretations are risky in the healthcare domain. infolossqa consists of 1,000 qa pairs across 104 rct abstracts simplified by gpt-4 (prior work has shown that gpt-3.5 is a capable simplifier of medical texts; @xcite . the qa pairs provide a rich description of what information is lost both on a pragmatic and semantic level. our analyses show that questions often elaborate about why and how an rct was conducted, going beyond frequently asked questions in this domain @xcite . contribution 2: automatic methods. we then present two methods to perform the question generation and answering task. first, an end-to-end model, where we prompt several open-source and api-based llms with detailed task instructions and a one-shot example. second, a natural language inference (nli) pipeline which reasons over entailment of atomic facts @xcite . contribution 3: evaluation framework and human quality judgments. finally, we design a comprehensive framework to evaluate models at this task. the framework includes 10 quality desiderata evaluating the correctness of qas, their linguistic suitability, and the recall of humanwritten qas. expert judgments are collected on more than 400 generated qa pairs. we find that models have good question generation and answering capabilities but fall short at reliably identifying information loss. in this regard, the nli pipeline is more effective than open-source llms, but produces qa pairs with a smaller granularity. task formulation. given a pair of documents, the original text x src and the simplified text x tgt , the task entails generating qa pairs that each reveal to lay readers one instance of information loss. we define an instance of information loss as a tuple (x src , x tgt , q, a, e, r). a potential question q asks for information absent from x tgt . the answer a provides the missing information. we additionally require two forms of localization or grounding to support richer analysis and scrutiny of errors in the generated qa pairs. first, each answer a must be supported by evidence @xmath0. second, each qa pair may be supported by an extractive rationale @xmath1. note that while e always exists, r may not (some information in x src may be completely absent from x tgt ). this formulation permits multiple information loss instances for a given x src and x tgt pair. we do not constrain the length or linguistic unit(s) of the extractive spans (e and r): they can comprise words, phrases, sentences, or entire paragraphs, and a single e or r can be one or more spans. types of information loss. for the purposes of this study, we define two types of information loss: ## automatic information loss detection no system in literature is directly applicable to solve all parts of infolossqa. therefore, we propose and evaluate two systems: (1) end-to-end prompting, contrasting several strong llms; (2) an nli pipeline. we describe the methods below and provide detailed parameters in table a.1. method 1: llm prompting. we design a oneshot prompt that specifies the task, closely resembling the manual annotation guidelines. we ask all models to localize information loss instances, to provide a short motivation for why it is considered information loss, akin to chain-of-thought prompting @xcite , and finally to generate the qa pairs. we benchmark three llms: mistral-7b, llama2-70b, and gpt-4. 9 all prompts were manually tuned for each model, though this was a heuristic process and better prompts may exist. ## evaluation framework we next propose a comprehensive framework to evaluate automatic systems for infolossqa. all evaluation is done manually by trained annotators. we describe the evaluation criteria below and appendix c provides full annotator guidelines. aspect 1: correctness of generated qa pairs. our first criterion, answerability (3-point), states that the generated questions (qs) should be about an information loss. valid questions must be answerable with the original text (i.e., there exists an evidence e answering q), but unanswerable or only vaguely answerable with the simple text. the second criterion, answer accuracy, states that questions should be correctly answered by the extracted and generated answers. we evaluate this in three parts. first, accuracy -evidence (3-point) captures how well the highlighted evidence e answers q. second, accuracy -answer (3-point) evaluates how well a answers q. finally, we ask if a contains any hallucinations (binary). aspect 2: linguistic suitability. we aim for qas that are pragmatically felicitous and comprehensible for readers (cf. section 2). we propose evaluating this through four linguistic criteria. first, we evaluate givenness (binary), closely following the constraints of qud @xcite : a question should only contain concepts that are directly mentioned in the question context, inferable, or generally known. we define the question context as the entire simple text x tgt for qas on deletions, and everything up to and including the rationale span r for qas on oversimplifications. 11 second, when a question seeks clarification about an oversimplified concept, there should be a rationale r in x tgt as an opportunity to show users that the question addresses said vagueness (rationale, 3-point). third, we evaluate if a contains any unexplained jargon (4-point). finally, standalone (binary) states that answers must be de-contextualized @xcite , i.e., they should not contain unresolved coreferences or acronyms that would require reading the original text. aspect 3: recall of human qa. lastly, we evaluate if models identify similar information loss as the references in infolossqa. a challenge in this evaluation is that both humans and models produce a set of qas without a strict 1-to-1 correspondence. we found that directly comparing two question sets is too cognitively demanding with typical sets of more than five qas. therefore, we opt for a pairwise setting, comparing each reference qa with one model qa at a time, and marking the reference as either fully recalled, partially recalled or not recalled. as comparing all possible pairs is intractable, we exclude unlikely matches following a paraphrase mining approach @xcite . we report the detailed procedure to estimate a minimum similarity threshold in appendix a.3. evaluation process. we sample 100 qas per model for aspect 1+2, and 50% of the documents for aspect 3. all samples were independently evaluated by two of the experts described in section 3.2. evaluation took 64 hours for aspect 1+2, and 20 hours for aspect 3, with additional training hours. we measure randolph's kappa @xcite for inter-evaluator agreement and observe a substantial average agreement of κ = 0.70 ( ## results and discussion overall we find that infolossqa is a challenging task. while all models have good questiongeneration/answering abilities, the main difficulty lies in reliably identifying information loss and ensuring the answers are comprehensible for readers. furthermore, models identify different information loss than humans. quantitative results are given in tables 3 and 4 and qualitative examples in figure 4 . models generate valid questions but not all are about an information loss ( while most questions are valid, all models could improve at generating questions that are not or only vaguely answerable with x tgt . see example 2 (figure 4 ), where the generated qa does not add any new information. notably, the fact-based nli pipeline outperforms open-source llms in identifying unanswerable questions, indicating a promising direction for addressing information loss. considering answer accuracy, gpt-4 provides near perfect results both in extracting the correct evidence span (98% correct) and in rewriting it into a full answer (94% correct). surprisingly, mistral and llama2 more often generate correct abstractive answers than extracting the right evidence, implying internally inconsistent llm responses. lastly, across all models, we only observe a small rate of hallucinations (worst: llama2, 7.8%). models often fail to localize qas in the simple text ( qas are well-framed, with room for improvement in terms of jargon ( concerning the answers, most can be understood without referring to the original text, i.e., they are standalone. qualitatively, undefined abbreviations are frequent reasons for answers to not be standalone (ex. 3 in figure 4 ) which could be solved with a dedicated decontextualization step @xcite . lastly, many answers contains unexplained jargon (--, best nli 28.4%, worst mistral 55.9%). despite instructing all models to generate simple answers, the task likely has too many constraints for a single end-to-end prompt. by decoupling information loss detection and qa generation, the nli pipeline can better follow this constraint. models have a low recall of human qas ( automatic evaluation: prompting llms to evaluate linguistic suitability is difficult ( the previous results rely on manual evaluation by trained annotators, which is expensive and not scalable. therefore, we run a preliminary investigation of automatic llm-based evaluation. for each of the 10 evaluation criteria, we created a prompt with evaluation instructions, the generated qa pair, and additional context for assessment. the model outputs a brief rationale followed by the rating, akin to chain-of-thought @xcite . no further prompt engineering was done. agreement between human ratings and ratings by representative llms (gpt-4o and llama3-70b) 12 are given in on average, llama3-70b and gpt-4o agree with humans evaluators to a fair and moderate de12 specifically, we use the checkpoint gpt-4o-2024-05-13 and meta-llama/meta-llama-3-70b-instruct. full evaluation suite is provided in code release. ## related work deletion in text simplification. professional editors commonly use deletion to make text more accessible @xcite @xcite . @xcite found that even professional example 1: vague question with multiple valid ways of answering; the answer is not standalone because acronym "vs" is only defined in xsrc. xsrc: ... two experiments were conducted. in experiment 1, cocs (@xmath3: ... in experiment 2, cocs (@xmath4, but those in vs1, vs1 + vs2 and vitrified groups were sub-divided and exposed to the warming solution for either 1 or 5 min. ... in experiment 2, there was no effect of time in the warming solution... xtgt: ... we did two experiments. in the first experiment, we had 420 [cow egg cells (also called cocs)] separated into four groups: ... in the second experiment, we treated 581 cocs the same way but also had some of the cocs from groups 2, 3 and 4 spend either 1 or 5 minutes in a warming solution.... q: what were the details of experiment 2? a: in experiment 2, 581 cocs were assigned to the same groups as in experiment 1, but those in vs1, vs1 + vs2 and vitrified groups were sub-divided and exposed to the warming solution for either 1 or 5 minutes. ## conclusion we propose infolossqa: a task and dataset to describe and recover simplification-induced information loss as reader-centric qas. our analyses show that qas following the qud theory provide a rich description that can mitigate overdeletion and vagueness in text simplification. we also establish automatic pipelines for the task and propose a rigorous evaluation framework considering correctness of qas and user-centric constraints. looking ahead, this work opens new avenues in interactive simplification tools @xcite and for quality assessment of automatic simplifications.