Explainable Prediction of Text Complexity: The Missing Preliminaries for Text Simplification (2021)

Abstract: Text simplification reduces the language complexity of professional content for accessibility purposes. End-to-end neural network models have been widely adopted to directly generate the simplified version of input text, usually functioning as a blackbox. We show that text simplification can be decomposed into a compact pipeline of tasks to ensure the transparency and explainability of the process. The first two steps in this pipeline are often neglected: 1) to predict whether a given piece of text needs to be simplified, and 2) if yes, to identify complex parts of the text. The two tasks can be solved separately using either lexical or deep learning methods, or solved jointly. Simply applying explainable complexity prediction as a preliminary step, the out-of-sample text simplification performance of the state-of-the-art, black-box simplification models can be improved by a large margin.

## introduction text simplification aims to reduce the language complexity of highly specialized textual content so that it is accessible for readers who lack adequate literacy skills, such as children, people with low education, people who have reading disorders or dyslexia, and non-native speakers of the language. mismatch between language complexity and literacy skills is identified as a critical source of bias and inequality in the consumers of systems built upon processing and analyzing professional text content. research has found that it requires on average 18 years of education for a reader to properly understand the clinical trial descriptions on clinicaltrials.gov, and this introduces a potential self-selection bias to those trials @xcite . text simplification has considerable potential to improve the fairness and transparency of text information systems. indeed, the simple english wikipedia (simple.wikipedia.org) has been constructed to disseminate wikipedia articles to kids and english learners. in healthcare, consumer vocabulary are used to replace professional medical terms to better explain medical concepts to the public @xcite . in education, natural language processing and simplified text generation technologies are believed to have the potential to improve student outcomes and bring equal opportunities for learners of all levels in teaching, learning and assessment @xcite . ironically, the definition of "text simplification" in literature has never been transparent. the term may refer to reducing the complexity of text at various linguistic levels, ranging all the way through replacing individual words in the text to generating a simplified document completely through a computer agent. in particular, lexical simplification @xcite is concerned with replacing complex words or phrases with simpler alternatives; syntactic simplification @xcite alters the syntactic structure of the sentence; semantic simplification @xcite paraphrases portions of the text into simpler and clearer variants. more recent approaches simplify texts in an end-toend fashion, employing machine translation models in a monolingual setting regardless of the type of simplifications @xcite @xcite . nevertheless, these models are limited on the one hand due to the absence of large-scale parallel (@xmath0, and on the other hand due to the lack of interpretibility of their black-box procedures @xcite . given the ambiguity in problem definition, there also lacks consensus on how to measure the goodness of text simplification systems, and automatic evaluation measures are perceived ineffective and sometimes detrimental to the specific procedure, in particular when they favor shorter but not necessar-ily simpler sentences @xcite . while end-to-end simplification models demonstrate superior performance on benchmark datasets, their success is often compromised in out-of-sample, real-world scenarios @xcite . our work is motivated by the aspiration that increasing the transparency and explainability of a machine learning procedure may help its generalization into unseen scenarios @xcite . we show that the general problem of text simplification can be formally decomposed into a compact and transparent pipeline of modular tasks. we present a systematic analysis of the first two steps in this pipeline, which are commonly overlooked: 1) to predict whether a given piece of text needs to be simplified at all, and 2) to identify which part of the text needs to be simplified. the second task can also be interpreted as an explanation of the first task: why a piece of text is considered complex. these two tasks can be solved separately, using either lexical or deep learning methods, or they can be solved jointly through an end-to-end, explainable predictor. based on the formal definitions, we propose general evaluation metrics for both tasks and empirically compare a diverse portfolio of methods using multiple datasets from different domains, including news, wikipedia, and scientific papers. we demonstrate that by simply applying explainable complexity prediction as a preliminary step, the out-of-sample text simplification performance of the state-of-the-art, black-box models can be improved by a large margin. our work presents a promising direction towards a transparent and explainable solution to text simplification in various domains. ## an explainable pipeline for text simplification we propose a unified view of text simplification which is decomposed into several carefully designed sub-problems. these sub-problems generalize over many approaches, and they are logically dependent on and integratable with one another so that they can be organized into a compact pipeline. figure 1 : a text simplification pipeline. explainable prediction of text complexity is the preliminary of any human-based, computer assisted, or automated system. the first conceptual block in the pipeline (figure 1 ) is concerned with explainable prediction of the complexity of text. it consists of two sub-tasks: 1) prediction: classifying a given piece of text into two categories, needing simplification or not; and 2) explanation: highlighting the part of the text that needs to be simplified. the second conceptual block is concerned with simplification generation, the goal of which is to generate a new, simplified version of the text that needs to be simplified. this step could be achieved through completely manual effort, or a computer-assisted approach (e.g., by suggesting alternative words and expressions), or a completely automated method (e.g., by selftranslating into a simplified version). the second building block is piped into a step of human judgment, where the generated simplification is tested, approved, and evaluated by human practitioners. one could argue that for an automated simplification generation system the first block (complexity prediction) is not necessary. we show that it is not the case. indeed, it is unlikely that every piece of text needs to be simplified in reality, and instead the system should first decide whether a sentence needs to be simplified or not. unfortunately such a step is often neglected by existing end-to-end simplifiers, thus their performance is often biased towards the complex sentences that are selected into their training datasets at the first place and doesn't generalize well to simple inputs. empirically, when these models are applied to out-of-sample text which shouldn't be simplified at all, they tend to oversimplify the input and result in a deviation from its original meaning (see section 5.3). one could also argue that an explanation component (1b) is not mandatory in certain text simplification practices, in particular in an end-to-end neural generative model that does not explicitly identify the complex parts of the input sentence. in reality, however, it is often necessary to highlight the differences between the original sentence and the simplified sentence (which is essentially a variation of 1b) to facilitate the validation and evaluation of these black-boxes. more generally, the explainability/interpretability of a machine learning model has been widely believed to be an indispensable factor to its fidelity and fairness when applied to the real world @xcite . since the major motivation of text simplification is to improve the fairness and transparency of text information systems, it is critical to explain the ra-tionale behind the simplification decisions, even if they are made through a black-box model. without loss of generality, we can formally define the sub-tasks 1a, 1b, and 2-in the pipeline: definition 3.1. (complexity prediction). let text @xmath1...w n . the task of complexity prediction is to find a function f : @xmath2, 1} such that f (d) = 1 if d needs to be simplified, and f (d) = 0 otherwise. definition 3.2. (complexity explanation). let d be a sequence of tokens w 1 w 2 ...w n and f (d) = 1. the task of complexity explanation/highlighting is to find a function h : @xmath3, 1} n s.t. h(d) = c 1 c 2 ...c n , where c @xmath4. we denote d|h(d) as the highlighted part of d and d|¬h(d) as the unhighlighted part of d. definition 3.3. (simplification generation). let d be a sequence of tokens w 1 w 2 ...w n and f (d) = 1. the task of simplification generation is to find a function g : @xmath5.t. g(d, f (d), h(d)) = d , where @xmath6...w m and f (d ) = 0, subject to the constraint that d preserves the meaning of d. in this paper, we focus on an empirical analysis of the first two sub-tasks of explainable prediction of text complexity (1a and 1b), which are the preliminaries of any reasonable text simplification practice. we leave aside the detailed analysis of simplification generation (2-) for now, as there are many viable designs of g(•) in practice, spanning the spectrum between completely manual and completely automated. since this step is not the focus of this paper, we intend to leave the definition of simplification generation highly general. note that the definitions of complexity prediction and complexity explanation can be naturally extended to a continuous output, where f (•) predicts the complexity level of d and h(•) predicts the complexity weight of w i . the continuous output would align the problem more closely to readability measures @xcite . in this paper, we stick to the binary output because a binary action (to simplify or not) is almost always necessary in reality even if a numerical score is available. note that the definition of complexity explanation is general enough for existing approaches. in lexical simplification where certain words in a complex vocabulary v are identified to explain the complexity of a sentence, it is equivalent to highlighting every appearance of these words in d, or ∀w @xmath7, c @xmath8. in automated simplification where there is a self-translation function g(d) = d , h(d) can be simply instantiated as a function that returns a sequence alignment of d and d . such reformulation helps us define unified evaluation metrics for complexity explanation (see section 4). it is also important to note that the dependency between the components, especially complexity prediction and explanation, does not restrict them to be done in isolation. these sub-tasks can be done either separately, or jointly with an end-toend approach as long as the outputs of f, h, g are all obtained (so that transparency and explainability are preserved). in section 4, we include both separate models and end-to-end models for explanatory complexity predication in one shot. ## empirical analysis of complexity prediction and explanation with the pipeline formulation, we are able to compare a wide range of methods and metrics for the sub-tasks of text simplification. we aim to understand how difficult they are in real-world settings and which method performs the best for which task. ## conclusions we formally decompose the ambiguous notion of text simplification into a compact, transparent, and logically dependent pipeline of sub-tasks, where explainable prediction of text complexity is identified as the preliminary step. we conduct a systematic analysis of its two sub-tasks, namely complexity prediction and complexity explanation, and show that they can be either solved separately or jointly through an extractive adversarial network. while pre-trained neural language models achieve significantly better performance on complexity prediction, an extractive adversarial network that solves the two tasks jointly presents promising advantage in complexity explanation. using complexity prediction as a preliminary step reduces the error of the state-of-the-art text simplification models by a large margin. future work should integrate rationale extractor into the pre-trained neural language models and extend it for simplification generation.