Cognitive Simplification Operations Improve Text Simplification (2022)

Abstract: Text Simplification (TS) is the task of converting a text into a form that is easier to read while maintaining the meaning of the original text. A sub-task of TS is Cognitive Simplification (CS), converting text to a form that is readily understood by people with cognitive disabilities without rendering it childish or simplistic. This sub-task has yet to be explored with neural methods in NLP, and resources for it are scarcely available. In this paper, we present a method for incorporating knowledge from the cognitive accessibility domain into a TS model, by introducing an inductive bias regarding what simplification operations to use. We show that by adding this inductive bias to a TS-trained model, it is able to adapt better to CS without ever seeing CS data, and outperform a baseline model on a traditional TS benchmark. In addition, we provide a novel test dataset for CS, and analyze the differences between CS corpora and existing TS corpora, in terms of how simplification operations are applied.

## introduction text simplification (ts) is the task of converting text into a form that is easier to understand by modifying its syntax and/or the words used in it, while maintaining the original text's meaning @xcite . ts is a very diverse task that can include simplifications aimed at different target audiences. ts is often operationalized in nlp using a number of particular corpora to train and evaluate neural models (see §3), whose target audiences are mostly second language learners, primary school students or adults with learning disabilities. for brevity, this paper will refer by ts to this concrete formulation of the simplification task, rather than the abstract, general notion of the task of simplifying text. cognitive simplification (cs) is the task of converting text to a form that is clear, simple, and readily understood by people with cognitive disabilities @xcite @xcite . 1 the procedure includes structural and lexical modifications that reduce the text's complexity, while preserving as much of the meaning and information content as possible, and without rendering it childish or simplistic. see figure 1 . the following example illustrates the differences and similarities between cs and ts. the sentence "some indigenous groups living in palm-rich areas use palms to make many of their necessary items and food." from the asset (alva-manchego et al., 2020a) validation set was simplified by one of the annotators as "groups who live in palm-rich areas use palms to make basic items and food.". a cs, in this case, could be "people who live in areas with a lot of palm trees use the trees for many things. people can eat the dates that grow on palm trees. people can make many things from palm trees, for example, baskets and plates.". foot_1 this is an example of the common need in cs to explicitly state assumed prior knowledge, and the need to make the text "closer" to the reader ("people" vs. "groups"). see §5.1. cs and ts appear to be similar tasks, as similar modifications can be applied in both. cs could even be considered a sub-task of ts, with a target audience of people with cognitive disabilities. however, there are two main differences between the two, that we believe motivate further investigation into cs as an independent task. first, cs is a well-defined procedure with manuals in multiple languages @xcite , while ts has general guidelines and, to the best of our knowledge, no common standards. second, the goal of cs is to simplify texts to provide cognitive accessibility @xcite . figure 1 : illustration of our approach on an example sentence from the cs dataset festability transcripts. the modified sources and targets for each model architecture include special operation tokens (see §5.2) added in the method appropriate for the model. for demonstration purposes in the original source and target, we boldface and color match areas @xmath0, we italicize areas @xmath1, and underline areas @xmath2. this goal of cs can also be at odds with ts's more general goal of improving comprehension, such as when simplifying an article for school students vs. for adults with cognitive disabilities at a similar language proficiency. as a first step, we explore cs and ts in english, and leave exploration of other languages and intralanguage comparisons for future work. there are very few nlp works that tackle cs. as such, scarce data is available for training potential cs models. we propose a methodology to address this gap, by introducing an inductive bias to a model trained on ts, in the form of simplification operations. we propose a set of simplification operations based on cs manuals, and show that adding inductive bias regarding their use improves performance on the asset test set, compared to a strong baseline model. in addition, we present an english parallel corpus aimed at cs, which we use as a test set. 3 we show that when fine-tuning models on ts data, our method improves the models' sari score on the cs dataset, allowing better task adaptation from ts to cs. finally, we compare how the operations are used in the new cs dataset and existing ts corpora, and show that cs differs from ts not only in goal, but also in data statistics. ## cognitive simplification the field of cognitive accessibility @xcite is derived from defining accessi-bility to include the ability to use services, receive information, and participate in activities, in addition to the more commonly accepted physical ability to reach, navigate, and move in a place. this definition codified the accessibility measure of simplifying textual information to address the need of people with cognitive disabilities to understand textual information, i.e., cognitive simplification. subsequent operationalizations of this notion were carried out by @xcite and @xcite . in particular, they emphasize the need to preserve as much of the meaning of the original text as possible, without rendering it childish or simplistic, while using the same written language as the original text. although cognitively simplified texts can be easier to read for people with learning disabilities (such as dyslexia), people with learning disabilities are not the main target audience for them. nlp research into ts for people with cognitive disabilities is relatively scarce. most works focus on measuring the effect of cognitively simplified text on the comprehension of people with cognitive disabilities @xcite and without them @xcite . a different line of work explored how people with different cognition react to texts at different simplification levels @xcite . several works @xcite ) detail parallel corpora of regular and easyread documents, documents that are created via the process of cs. although these works provide details regarding linguistic phenomena in their corpora, we were not able to find any of the corpora detailed therein to run evaluations on. in addition, we were not able to find any recent works that report results on these corpora, using neural techniques for ts. although some preliminary works reference the use of contemporary nlp methods for cs to generate simplification examples (e.g., @xcite , to the best of our knowledge none provide details regarding the model used, model hyperparameter choices, and evaluation methodology. as such, we consider our work to be one of the first to tackle cs as a rigorous, distinct nlp task. 4 two other tasks that are related to cs, and use contemporary nlp methods, are text2picto @xcite and picto2text @xcite . these are the tasks of converting text to the sclera foot_5 and beta foot_6 pictogram languages, designed for people with idd (intellectual or developmental disabilities), and vise versa. while the output of both tasks can improve access to information for people with cognitive disabilities, we believe this task to be distinct from cs and especially ts, that focus on written and spoken language. ## other related work we would like to highlight key points from @xcite relevant to our work that relate to training and evaluation datasets and evaluation metrics. the main datasets used to train and evaluate ts models are wikilarge @xcite and newsela @xcite . both corpora contain matching complex-simple document pairs, whose sentences are automatically or manually aligned to create the datasets. in wikilarge, the matching document pairs are taken from english wikipedia foot_7 and simple english wikipedia, foot_8 that aims to be more accessible to people with lower english skills, mainly language learners. in newsela, 9 the matching document pairs are articles written professionally at four different reading levels, and are originally intended to be used to teach language skills at different school grade levels. the latest training datasets, and the current de facto standard for ts training, are wikiauto and newselaauto, created by @xcite by using a neural crf sentence alignment model. both are split into training and validation sets. to train their neural crf aligner, @xcite also compiled two manually aligned datasets, wikimanual and newselamanual, split into development, train, and test sets. the two main datasets used for validation and evaluation of ts models are turkcorpus @xcite and asset @xcite . both contain multiple references for each source sentence (8 and 10 respectively). they are crowdsourced and validated professionally. the main metric used for evaluating ts models is sari @xcite , which is computed based on three token-level operations: add, keep, and delete. for the full calculation, see appendix d. many previous works in ts also report bleu @xcite . however, several works @xcite , have shown that bleu scores are not suitable for the evaluation of ts models. nevertheless, bleu is still reported, and so we also report it for completeness. a contemporaneous work @xcite argued for the value of manual evaluation in ts rather than automatic metrics. we defer this exploration for cs for future work. recent works have proposed methods to control ts outputs by prepending special tokens to the input of a ts model, in a similar manner to the one explored in this work. such control allows adjusting the model's outputs to different target audiences, and to control what aspects of the simplification process are applied. access @xcite , and muss @xcite both use four structural features of the input-output pairs to define what tokens to prepend during training, and at inference they predefine which tokens to use for all inputs. sheang and @xcite add a fifth token to this methodology. @xcite use a combination of tokens to specify the type of simplification to perform and the grade level to which to simplify to. similarly to these works, we also define special tokens to add to the input at training, while at inference we take a different approach (see §6). other recent work on ts focuses on particular simplification operations @xcite , or on combining different operation modules in a joint model @xcite . @xcite define elaborative simplification as simplification by adding information to the source text, rather than just removing redundant information. this aligns with some of our proposed simplification operations (adding information and explicitation, see §5.1). similarly, @xcite focus on whole sentence deletion, which aligns with some operations from our proposed list (deleting information, and operations on sentences). @xcite combine a module for sentence deletion and splitting with a paraphrasing module to generate final simplifications. we discuss all three operations in §5.1. ## our approach to learn how to simplify a text, a model needs to learn what types of modifications to apply to the input and how to apply each one. these modifications can be categorized into operations. moreover, since ts has multiple large-scale datasets commonly used for training, while there are hardly any such datasets for cs, incorporation of some form of cs-focused inductive bias into a ts-trained model would be useful to allow it to adapt to the cs task. the inductive bias could also be useful for improving ts on its own, given the similarities between the two tasks (see §7 and §8). as such, our hypothesis is that a ts-trained model that was trained to be aware of the use of cs simplification operations, will perform better at ts and adapt better to cs than a model that was trained end-to-end. we will now turn to testing this hypothesis empirically. ## simplification operations we adapt existing cs manuals (plain, 2011a,b; u.s. opm, 2011a,b; u.s. dep. hhs, 2020; @xcite into a list of eight main types of simplification operations. seven of these apply to the simplification instance (si) level, and the final main type applies to a whole document. an si is a set of one or more sentences in regular language (source) aligned to one or more sentences in simplified language (target). 10 each main type of operation has multiple sub-operations. for full details, see appendix a. previous work define different lists of simplification operations @xcite or focus on word-level operations (keep, add, delete and sometimes also move @xcite ). our list is based on independent sources (the cs manuals) and focus on intra-and inter-sentence operations applied mainly to a si. §5.1 provides theoretical definitions for each operation. §5.2 describes how we integrate operations into a ts model. ## simplification experiments we use the huggingface foot_13 api to fine-tune pretrained language models. we select t5 @xcite and bart @xcite ) model architectures of two sizes each, base and large, to align with the recently published gem benchmark's @xcite official baseline for ts that uses these two model architectures. in addition, we wanted to test if results are consistent across model architectures. ## results focusing on cs performance, we find that the +classifier variants achieved superior results for all model architectures and sizes. the improvement differs by architecture and size, with the largest difference being of 5.74 sari point for the t5base models trained on wikiauto. the best performance is again obtained by the bart-large+classifier model, and is at least 2.01 sari points higher than the score obtained by any baseline variant. with respect to the manual dataset training setting, we see similar trends. in particular, the +classifier models outperform baseline models, and the best performing model is still bart-large+classifier. due to space limitations, we discuss the results on this dataset in appendix c. taken together, our results demonstrate the effectiveness of incorporating inductive bias using simplification operations for both ts and cs. in order to ensure that the experimental setup we use is comparable in performance with the standard practice in the field of ts, we experiment with the original gem baseline code-base, and our hyperparameter settings were chosen according to it. the results of models trained according to this codebase are indeed comparable to models of matching sizes of the baseline variants. we further validated our results with significance tests, following the guidelines of @xcite . we used the wilcoxon signed-ranked @xcite test as our main significance test. we compared each vanilla and +classifier model pair, and also each model of a particular type (t5 and bart) to their respective gem baselines. the results are shown in we attribute the improved performance of all +classifier models to improvements in the tokenlevel operations scores for add and delete. in the standard training setting on wikiauto, all +classifier models achieve substantially higher add and delete scores than their same-sized baseline counterparts, while all models achieve similar keep scores. interestingly, for the bart models, the difference in add scores is less substantial than for the t5 models. ## simplification dataset comparison we compare simplification datasets with respect to how the simplification operations are used in each. we show that simplification operations can also be used to better characterize such datasets. we analyze all available sub-sets (development, train, validation, and test) of all datasets, to provide a fine-grained analysis. we consider test sub-sets of datasets, to better understand the results of §7. this analysis was done after-the-fact, and did not influence the development of the models. 17 the results presented in this section show that cs is different from ts in how the operations are applied. they also surface the known relationships between the datasets, validating our analysis. we believe that this type of aggregate analysis can be confidently performed given the validation at the end of §5.2, but acknowledge that the token assignment is noisy. to understand how each simplification operation is applied individually, we compute the frequency with which each operation is applied in a given sub-set. these frequencies can be viewed as defining random variables x s o , stating the probability that each simplification operation o is used in a particular si in sub-set s. as such, to understand the distance between sub-sets with respect to the individual application of each operation, we can compute the mean jensen-shannon distance @xcite ) (which we mark jsd) between matching random variables in different sub-sets. for further details on the action distributions for each dataset, see appendix h. as can be seen in figure 2a , all sub-sets have 17 we analyze the test sets also because the cs dataset only contains a test set at this point, due to their small size. @xmath3.1 from one another, which is not a large distance. however, we are still able to see distinct clusters for each dataset, with subsets having @xmath4.04 within clusters and @xmath5.04 to other sub-sets. 18 interestingly, wikiauto-test is closer to the wikimanual cluster than it is to wikiauto-valid, which could be explained by the fact that wikiauto was created based on the matching of complex-simple sentences presented in wiki-manual. in addition, wikiauto-valid and assetvalid appear to be identical, which could be explained by the fact that the source for asset-valid was taken from wikiauto-valid. regarding the cs dataset festability, it is @xmath6.07 from all other sub-sets, and is the farthest sub-set from wikiauto, asset, and wikimanual clusters, and the second or third farthest from sub-sets in the newselamanual cluster. to understand how simplification operation are applied together, we computed the pearson correlations of the co-occurrence of each operation pair in a given subset s, to create a correlation matrix m s . we then computed the pair-wise ℓ 2 -distance between matrices. results are in figure 2b . as can be seen in figure 2b , the clusters of closest sub-sets are maintained for newselamanual, and for asset and wikiauto-val , while the subsets of wikimanual are no longer closest to one another. also, wikiauto-train is similarly distant from both wikiauto-val and the wikimanual subsets, unlike when comparing with jsd. in this setting, the festability dataset is the most distant sub-set from all other sub-sets, with d ℓ 2 > 0.88 from all of them. all other sub-sets are d ℓ 2 < 0.75 from one another, except newselamanual-test from wikiauto-train and wikimanual-test with d ℓ 2 = 0.85 and d ℓ 2 = 0.88 respectively. taken together, these results show that while each individual operation is applied with similar probability in every dataset, the operations are applied together differently. in cs in particular, they are applied in a more distinct fashion than in ts. the difference in operation application could be attributed to the different domains from which each dataset pulls its sentences. in our cs dataset, all sentences are transcripts of human speech, taken from a formal conference. thus, they may contain more informal language than a wikipedia article. given our datasets, we therefore cannot differentiate between domain difference and task difference. however, we are currently compiling a larger dataset for cs that contains more formal language, that will enable such analysis. the analysis here can provide additional insight as to the performance patterns of the different models ( §7). since each operation is applied individually under a similar distribution in ts and cs, the +classifier models could have potentially learned indicators of when to apply each action individually when training on ts. this could have been useful when adapting to cs, especially given that the operations co-occur differently in ts and cs. ## conclusion and future work we formulated the task of cognitive simplification as an nlp task, and discussed its similarities and dissimilarities from the well-researched task of ts. the two tasks are similar in the types of simplification operations that are applied in each, and different in the distribution in which the operations are applied. they also differ in their target audience, at least when using standard datasets. we further release with this paper a readily available dataset directed at cs, providing a test set to evaluate cs models on. attempting to overcome the absence of training data for cs, we showed that by introducing to a ts-trained model inductive bias as to the simplification operations that need to be performed on the input, the model is able to better adapt to cs. we also showed that ts-trained models that are trained to predict simplification operations perform better than their baseline counterparts on ts. we believe that comparing how simplification operations are applied in different languages can provide valuable insights into understanding the task of text simplification better. future work will further explore the relation between the distribution of operations and the ability of the model to generalize to different domains and task formulations. such an inquiry may reveal that simplification operations provide not only inductive bias, but also an analytical tool for comparing datasets and variants of ts. there are ts datasets in many languages, including swedish @xcite , spanish @xcite , german @xcite , danish @xcite , portuguese @xcite , and russian @xcite . we plan to compare these datasets in terms of their distribution of operations, so as to empirically characterize whether the notion of text simplification implicit in these datasets is similar or not. we hope that our findings will spark interest in cs, as there is much more to solve in creating automatic simplification systems for people with cognitive disabilities. as stated above, we are currently working on compiling a larger and more robust cs dataset, that will enable improvements in cs technology, and allow to tease apart domain effects in the differences between ts and cs from more fundamental differences between the tasks.