Text Simplification via Adaptive Teaching (2024)

Abstract: Text simplification is the process of rewriting a piece of text using simpler vocabulary and grammatical structure in order to make the text more accessible and understandable for a larger audience. In this paper, we introduce a new text simplification model based on the notion of adaptive teaching using a teacher network and a text generation network. We name this new model Simplification via Adaptive Teaching (SAT). Our proposed model sets a new state-of-the-art performance in terms of standard simplification metrics such as SARI and D-SARI with a significant improvement over the previous state of the art on the D-Wikipedia dataset and the Wiki-Doc benchmark dataset. Moreover, we conduct a human evaluation in terms of text simplicity, correctness, and fluency to substantiate SAT’s performance.

## introduction text simplification is an important area of research which aims at improving the understandability and accessibility of written texts for individuals with limited reading skills and education level (e.g., children @xcite , dyslexic people @xcite , those suffering from autism @xcite ), non-native speakers of a language, and generally people without expertise in a specific domain (e.g., in legal or medical documents). that is, highly technical texts with specialised vocabulary can be made more accessible to a wider audience by simplifying the text. text simplification (al-thanyyan and azmi, 2021) is generally categorized as (1) lexical simplification, (2) syntactic simplification, (3) statistical machine translation style and finally, the more recent (4) neural machine translation style using transformer models @xcite . ## related work text simplification can be seen as a special form of controlled text generation @xcite where a given input text is converted to its simplified equivalent. recent text simplification research has been more focused on sentence simplification @xcite , which deal with simplification of short single sentence texts. the most commonly used datasets for text simplification such as wikilarge @xcite , turkcorpus @xcite , and newsela @xcite are originally designed for sentence simplification. on the other hand, various real-world applications require document-level simplification rather than sentencelevel processing. this stems from the necessity to understand the main ideas across multiple sentences simultaneously and rewrite them in a simplified vocabulary and grammar structure without respecting a number of sentences. thus, documentlevel text simplification may have more applications than text simplification at the sentence level. @xcite investigated the task of document-level text simplification, provided a large-scale dataset called d-wikipedia, and proposed a more suitable evaluation metric than sari @xcite named d-sari in the documentlevel simplification task. @xcite further preprocessed and cleaned the two datasets to remove faulty samples from both datasets. furthermore, they also introduced a document-level simplification model named simsum @xcite for document-level simplification. this model consists of a two stage setup of first summarizing and then simplifying a document end-to-end. in this paper, we also focus on document-level text simplification. analogous to most other nlp tasks, recent text simplification research has heavily relied on transformer-based models @xcite . this recent research trend considers text simplification as a sequence-to-sequence problem, resembling machine translation @xcite or document summarization @xcite . as a result, language models such as t5 @xcite or bart @xcite have been extensively used in text simplification research. for instance, the current state of the art on text simplification is control_prefixes @xcite , which is developed on an underlying t5 model as well as a bart-based variant. due to the success of these three models we compare our new model sat, against all three models. the control_prefixes model uses a dynamic method which allows for the inclusion of conditional input-dependent information, combining the benefits of prompt tuning and controlled generation. this method incorporates attribute-level learnable representations into different layers of a pre-trained transformer, namely either t5 or bart. another recent top-performing baseline is brio @xcite . this method holds the current state of the art for abstractive summarization, outperforming bart and and t5 by significant margins. due to the fact that abstractive summarization and document simplification share notable similarities, such as both being sequence-to-sequence nlp tasks, and both aiming at generating the main concepts and the gist of any given input article, we also compare our simplification model against this model. brio, unlike most abstractive summarization models which assume a deterministic (onepoint) target distribution in which an ideal model will assign all the probability mass to the reference summary, proposes a novel training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality. the authors show that their method performs well during inference. another framework which has shown to achieve state-of-the-art performance on sentenceto-sentence simplification tasks is muss @xcite . muss utilizes sentence-level paraphrase data instead of simplification data for training. furthermore, it leverages unsupervised pre-training and controllable generation mechanisms to flexibly adjust attributes such as length and lexical complexity at inference time. other efforts focus on the correctness and factual accuracy in automatic text simplification by introducing a taxonomy of frequent errors @xcite . in this paper, we propose a novel model using an adaptive teaching framework where a transformerbased language model learns from a teacher network, the degree to which it needs to modify an input document in order to produce a simplified version of the input. ## proposed new model our new model, sat, consists of two neural networks: (1) a transformer network based on an encoder-decoder architecture for sequence-tosequence generation and (2) a teacher network which is a feed-forward neural network. the two networks are trained in an end-to-end fashion. figure 1 illustrates the sat architecture. we initialize the transformer network using a pre-trained bart model. the teacher network is initialized with random weights as a feed-forward network optimizing cross-entropy loss. both networks receive as input the same documents written in potentially complex language (i.e. in terms of vocabulary and discourse), and as the target of the training they receive the corresponding documents in simplified language. whether a text is complex or simple, is determined by human-annotated datasets explained in section 5.1. we elaborate on the details of each network and describe the learning process. the transformer network uses an encoder-decoder architecture to map complex text to its simplified equivalent in a sequence-to-sequence text generation setup. on the other hand, the teacher network receives as input documents in potentially complex vocabulary in the form of a binary bag-of-words feature set, and the target of its training is mapping this input to its simplified corresponding document, again in the form of binary bag-of-words. therefore, the feedforward teacher network learns a mapping between complex vocabulary and its simplified equivalent via a cross-entropy loss function in a supervised fashion. after the teacher network is trained, it is integrated into the transformer model to force the transformer to use simple vocabulary. in order to do that, the loss function of the transformer network is modified as follows. the transformer model aims to learn a mapping between complex and simplified text that results in generating highquality simplified text. in order to achieve this goal it uses a maximum likelihood estimation (mle) step. that is, it aims to maximize the likelihood of the reference simple text, s * : at training time, the teacher network is initially trained for a number of epochs. subsequently, the transformer network is trained using equation 2. that means that each document in complex language is split into inputs and given to both the transformer and teacher networks; first, the teacher network loss is computed and subsequently, the transformer loss is updated. the teacher loss value indicates the degree of vocabulary simplification needed to reach the lowest loss, and therefore a reasonable simplified text. this loss value is directly added to the transformer network's loss before error backpropagation. this is to encourage sat to simplify complex words more than simpler ones. at inference time, the transformer model takes in the complex input documents and generates simplified versions. the teacher network is no longer used in this phase. the pairs of system outputs and ground truth simple texts are then used for evaluation. ## detailed explanation of model training and testing the sat model is composed of a feed-forward teacher network embedded in a transformer model. in this section we describe the model in detail and elaborate on the training and testing phases. we will release our implementation of sat in the final version of this paper. the teacher network contains one hidden linear layer followed by a softmax layer. its purpose is to determine the complexities of sentences and map them to corresponding loss values, which are used to improve training and target more difficult simplification tasks. this is achieved by inputting binary encoded vectors representing the vocabulary in source (complex) sentences and expecting outputs of similar binary encoded vectors representing the vocabulary in target (simplified) sentences. we create the inputs and labels for the teacher network by converting source and simplified documents in the training data to binary encoded vectors of size 15869, matching the vocabulary size. crossentropy loss is calculated on the outputs and labels after softmax activation; higher loss values should indicate higher degrees of simplification needed. after pre-training the feed-forward network for 7 epochs, it is integrated into the transformer model to assist training. as it can be seen in the model architecture in figure 1 , there is a linear layer combining the loss terms of both networks. that means that a forward-pass of both the teacher network and the transformer network does go through the linear layer, both loss terms are combined and then the calculated total loss is back-propagated through the linear layer onto the transformer network, thus the gradients of the transformer network are influenced by the forward pass of the teacher network and updated accordingly. finally, model parameters are determined using validation sets. the transformer model implements an encoderdecoder architecture initialized by bart. its purpose is to process complex sentences and generate their simplified versions. ## evaluation we base our evaluation on common standard metrics @xcite ) for text simplification, namely, sari @xcite , d-sari @xcite , and bleu @xcite . we use easse @xcite , a python3 package created to standardize the evaluation of text simplification methods. sari @xcite ) compares the system output against references and the input document, explicitly measuring the quality of words that are added, deleted, and kept by the systems. sari is the most popular used metric for text simplification task. d-sari @xcite is a modified sari score, adding different penalty factors based on the length. it is targeted towards documentlevel text simplification tasks. bleu @xcite was originally designed for machine translation tasks. it com-putes the similarity between the system output and the reference simple document. fkgl @xcite is used to measure readability, but does not take grammar and semantic meanings into account. ## conclusions and future work in this paper, we proposed sat, an innovative adaptive teaching approach to document simplification. sat outperforms existing top baselines by a considerable margin in simplification scores. the architecture of sat is composed of a feedforward teacher network integrated into a transformer sequence-to-sequence network via a joint loss function. our model has shown merit as a practical text simplification tool, not only in the results demonstrated in this paper but also in parallel applied research work simplifying informed consent forms in the healthcare domain. we showed through extensive experimentation both on automatic evaluation metrics such as sari, d-sari, bleu, and fkgl, as well as, human evaluations of simplicity, correctness, and fluency that this light-weight model has a remarkable performance achieving top results. moreover, we conducted an extensive ablation study testing the effect of the teacher network and multiple different settings to corroborate the effectiveness of sat model's design in its current form. in the future, we plan to study controlled document simplification, tailored towards specific needs and preferences of various audiences, e.g., children, non-native speakers, dyslexic, etc. another interesting direction for future exploration would be studying adding multiple teacher networks to build a more complex objective function for train-ing the text generation network. such objective function could for instance control the style or the topic-focus(bahrainian et al., 2021a) of the final text output. this could open a path to studying such effects along-side various fine-tuning schemes beyond linear addition of loss terms.