(Psycho-)Linguistic Features Meet Transformer Models for Improved Explainable and Controllable Text Simplification (2022)

Abstract: State-of-the-art text simplification (TS) systems adopt end-to-end neural network models to directly generate the simplified version of the input text, and usually function as a blackbox. Moreover, TS is usually treated as an all-purpose generic task under the assumption of homogeneity, where the same simplification is suitable for all. In recent years, however, there has been increasing recognition of the need to adapt the simplification techniques to the specific needs of different target groups. In this work, we aim to advance current research on explainable and controllable TS in two ways: First, building on recently proposed work to increase the transparency of TS systems (Garbacea et al., 2020), we use a large set of (psycho-)linguistic features in combination with pre-trained language models to improve explainable complexity prediction. Second, based on the results of this preliminary task, we extend a state-of-the-art Seq2Seq TS model, ACCESS (Martin et al., 2020), to enable explicit control of ten attributes. The results of experiments show (1) that our approach improves the performance of state-of-the-art models for predicting explainable complexity and (2) that explicitly conditioning the Seq2Seq model on ten attributes leads to a significant improvement in performance in both within-domain and out-of-domain settings.

## introduction text simplification (henceforth ts) is a natural language generation task aimed at transforming a text into an equivalent that is more readable and understandable for a target audience, while preserving the original information and underlying meaning. it involves a number of transformations applied at different linguistic levels, including lexical, syntactic and discourse aimed at reducing the complexity of content for the purpose of accessibility and readability (see @xcite @xcite @xcite . simplification techniques have been shown to be beneficial as reading supports across a wide range of populations, from children @xcite , individuals with language disorders such as aphasia @xcite , dyslexia @xcite or autism @xcite ; language learners and nonnative english speakers @xcite , and people with low literacy skills @xcite @xcite . moreover, ts techniques have also been successfully employed as a preprocessing step to improve the performance of various downstream nlp tasks such as parsing @xcite , machine translation @xcite , summarization @xcite , semantic role labeling @xcite , and information extraction @xcite . ts approaches typically learn simplification transformations using parallel corpora of matched original and simplified sentences and can be classified into six categories (for recent overviews see @xcite : early approaches relied on either (1) manually generated rules for splitting and reordering sentences @xcite or (2) learned simple lexical simplifications, i.e., oneword substitutions @xcite . subsequent work has introduced (3) phrasebased and syntax-based statistical machine translation techniques @xcite , (4) grammar induction @xcite , and (5) semantics-assistance, i.e., obtaining semantic representations of the original sentences @xcite . more recently, ts tasks have been approached with @xcite neural machine translation methods, in particular sequence-to-sequence (seq2seq) models using an attention-based encoder-decoder architecture @xcite @xcite . while the performance of seq2seq ts models is impressive, most of these models are black-box models characterized by the lack of interpretability of their procedures @xcite . in recent years, there have been growing calls to a move away from blackbox models toward explainable (white-box) models @xcite @xcite . moreover, recent work in ts suggests that the performance of state-of-the-art ts systems can be improved by conducting explainable complexity prediction as a preliminary step @xcite . another important trend in current ts research is the growing recognition that the concept of 'text complexity' is not homogeneous for different target populations @xcite . that is, rather than viewing ts as a general task where the same simplification is appropriate for everyone (one-fitsall approach), researchers are placing a greater emphasis on the need to develop ts systems that can flexibly adapt to the needs of different audiences: for example, while second language learners might struggle with texts with rare or register-specific vocabulary, aphasic patients might be overwhelmed by a high cognitive load associated with long, syntactically complex sentence structures. in response, recent ts research has begun to adopt methods proposed in controllable text generation research (see the 2 section for further discussion). controllable text generation refers to the task of generating text according to a given controlled property of a text. more generally, the development of controllable text generation systems makes an important contribution to the general development of ethical ai applications. this requires the ability to avoid biased content such as gender bias, racial discrimination, and toxic words. in addition, it is widely seen as critical to the development of advanced text generation technologies that better address specific needs in real-world applications @xcite . for example, the task of dialog response generation requires effective control over text attributes associated with emotions @xcite and persona @xcite . in the context of ts, the relevant attributes involve various linguistic aspects of text complexity @xcite . by combining multiple attributes, a natural language generation system can theoretically achieve not only greater controllability but also greater interpretability. this requires the inclusion not only of surface features, but also of more sophisticated features. traditionally, ts has used readability measures that consider only surface features. for example, the flesch reading ease score @xcite , a commonly used surface feature, measures the length of words (in syllables) and sentences (in words). while readability has been shown to correlate to some degree with such features @xcite , there is general consensus that they are insufficient to capture the full complexity of a text. in a nutshell, despite significant progress in datadriven text simplification, the development of explainable and controllable models for automatic text simplification remains a challenge. in this paper, we advance current research on explainable and controllable text simplification in two ways: the remainder of the paper is organized as follows: section 2 provides a concise overview of related work in the field of explainable and controllable text generation with a focus on ts. section 3 outlines the experimental setup including the description of three benchmark datasets used (section 3.1), the type of features extracted from these datasets (section 3.2), and the models performed to improve explainable and controllable ts (sections 3.3-3.5). section 4 presents and discusses the results of our experiments before presenting conclusions and future work in section 5. sections 6 and 7 address the limitations of the study and point out ethical considerations. ## related work state-of-the-art systems for controllable text generation typically use a sequence-to-sequence (seq2seq) architecture. these systems follow either a learning-based or a decoding-based approach: in the learning-based approaches, the seq2seq model is conditioned on the attribute under consideration at training time and then used to control the output at inference time. within this approach, controlled text generation can be achieved by disentangling the latent space representations of a variational autoencoder between the text representation and the controlled attributes @xcite . decoding-based methods, on the other hand, are based on a seq2seq training setup that is modified to control specific attributes of the output text @xcite . for instance, @xcite controlled the length of the text output in the encoder-decoder framework by preventing the decoder from generating the end-of-sentence token before the desired length was reached, or by selecting only hypotheses of a certain length during the beam search. recently, @xcite adapted a discrete parameterization mechanism to the task of sentence simplification by conditioning on relevant attributes. building on the earlier work of ts @xcite , their model, called access -short for audience-centric sentence simplification -provides explicit control of ts by conditioning the output returned by the model on specific attributes. these attributes and their values are prepended as additional inputs to the source sentences at train time as plain text 'parameter tokens'. results of experiments on the wiki-large corpus @xcite show that with carefully chosen values of three attributes -(i) character length ratio between source sentence and target sentence, (ii) normalized character-level levenshtein similarity between source and target, and (iii) wordrank, a proxy to lexical complexity, the access model outperformed previous ts systems on simplification benchmarks, achieving state-of-the-art at 41.87 sari, corresponding to a +1.42 improvement over the best previously reported score. another recently introduced line of research, on which the present work builds, explores how the transparency and explainability of the ts process can be facilitated by decomposing the task into several carefully designed subtasks. more specifically, @xcite propose that ts benefits from a preparatory task aimed at the explainable prediction of text complexity, which in turn is divided into two subtasks: (1) classifying whether a given text needs to be simplified or not (complexity prediction) and (2) highlighting the part of the text that needs to be simplified (complexity explanation). @xcite focuses on empirical analysis of the two subtasks of explicable prediction of text complexity. specifically, they conduct experiments using a broad portfolio of deep and shallow classification models in combination with model-agnostic explanatory techniques, in particular lime @xcite and shap (lundberg and @xcite . the results of their experiment show that a combination of a long short-term memory network at the word level and lime explanations can achieve strong performance on datasets. as a next step, they conduct follow-up experiments with state-of-the-art controllable endto-end text generation systems, including access. the results of these experiments suggest that the performance of state-of-the-art ts models can be significantly improved in out-of-sample text simplification simply by applying explainable complexity prediction as a preliminary step. ## experimental setup in this section, we first introduce the three datasets used in our experiments (section 3.1) and the type of (psycho-)linguistic features used in our models (section 3.2). we then describe the methods used to address the three subtasks, i.e., (1) complexity prediction, (2) complexity explanation, and (3) simplification generation. for subtask (1), we perform experiments with five complexity prediction models described in section 3.3: (1) a word-level long short-term memory (lstm) network, (2) a fine-tuned pre-trained bert-based model, ( 3 ) and ( 4 ) two hybrid bidirectional long-term memory (blstm) classifiers that integrate glove word embeddings with (psycho-)linguistic features using different fusion methods, and (5) a hybrid classifier that integrates the those features with bert representations. in subtask (2), we apply these five models to identify the complex parts of a given input set to facilitate model validation and evaluation (section 3.4). in section 3.5, we turn to subtask (3) and introduce an extended access model, which we refer to as access-xl, containing a total of ten control features (parameter tokens) covering several dimensions of linguistic complexity. ## results an overview of the results of the three subtaskscomplexity prediction, complexity explanation and simplification generation -is presented in complexity prediction: our best-performing models outperformed the classification accuracy of explainable model -the word-level lstm -predicting complexity in all three benchmark datasets reported in @xcite . since the pattern of results is consistent across all evaluation metrics, we focus here on classification accuracy: on wikilarge, we improve on the wordlevel lstm presented in @xcite by +8.08% by extracting attention weights from the pre-trained bert model. on newsela, our glovebased lstm model outperforms the word-level lstm by +6.68%. on the biendata dataset, our hybrid model that integrates bert representations with linguistic features leads to an improvement of +4.43%. overall, our results replicate the general pattern of results reported in @xcite in that the best-performing models achieve approximately 80% accuracy on the wikilarge and newsela datasets and much higher -approximately 95% accuracy -on the biendata dataset. these results support the conclusion drawn in @xcite that complexity prediction is influenced by the application domain, with the distinction between scientific content and public domain press releases (biendata) being much easier on the wikilarge dataset, the bert model performed the best, with a +7.4% performance increase over the lstm. on the newsela dataset, however, the lstm achieved the highest accuracy, outperforming both the bert and bert-hybrid models by +1.9% and +1.5%, respectively. on the biendata dataset, the highest performance was achieved by our bert-hybrid model, which improved the already high performance of the lstm by +3.6%. across all datasets, the glove word embedding-based models consistently ranked between the lstm and bert-based models, suggesting that the use of contextualized word embeddings of the bert-based model may reduce the generalizability of the model, leading to variations in model performance across datasets. ## conclusion and future work in this work, we have advanced research on explainable and controllable text simplification in two ways: first, we have shown that performance on a prior task of explainable complexity prediction can be significantly improved by the combined use of (psycho-)linguistic features and pre-trained neural language models. and second, by extending the audience-centric sentence simplification model to explicitly control ten text attributes, we have achieved a new state of the art in text simplification in both within-domain and out-of domain settings. in future work, we plan to apply our modeling approach to another key text style transfer task, that of formality transfer, and evaluate it on existing benchmark datasets such as the gyafc dataset @xcite . moreover, we intend to explore the role of (psycho-)linguistic features for controllable ts in unsupervised settings using a variational auto-encoder and a content predictor in combination with attribute predictors @xcite .