Multi-Word Lexical Simplification (2020)

Abstract: In this work we propose the task of multi-word lexical simplification, in which a sentence in natural language is made easier to understand by replacing its fragment with a simpler alternative, both of which can consist of many words. In order to explore this new direction, we contribute a corpus (MWLS1), including 1462 sentences in English from various sources with 7059 simplifications provided by human annotators. We also propose an automatic solution (Plainifier) based on a purpose-trained neural language model and evaluate its performance, comparing to human and resource-based baselines.

## introduction text simplification is the task of automatically modifying natural language text to improve its ease of understanding, whilst preserving the overall meaning. the diverse applications range from second language learners to lay readers of scientific texts to stroke victims. this challenge is often defined in the framework of lexical text simplification (ls), where individual words are replaced with their simpler equivalents @xcite . however, single-word substitutions do not cover the full complexity of techniques humans use to approach text simplification, including replacements, deletions, addition and sentence splits. they could be modelled implicitly, by sentence-to-sentence monolingual translation approaches @xcite , or explicitly, by collecting data @xcite and developing methods @xcite focusing on particular types of transformations. with a similar motivation, we propose in this work a new task going beyond single word substitution, namely multi-word lexical simplification (mwls). the aim of mwls is to replace a given fragment (short sequence of words) with its simpler version, so that the enclosing sentence retains its meaning, but becomes easier to understand. see this paper explores the task of mwls in two directions. firstly, we contribute a dataset of 1462 sentences with 7059 simplifications obtained through crowdsourcing. secondly, we design a method for generating such simplifications automatically. our solution, called the plainifier, is inspired by a recently proposed method for ls utilising language models @xcite ), which we extend so that multi-word simplifications can be obtained. in order to encourage more research on the problem, we make the dataset 1 , the language model 2 and the plainifer code 3 openly available. ## background typical lexical simplification systems have followed a four stage pipeline of complex word identification @xcite , substitution generation, word sense disambiguation and synonym ranking @xcite . however, in the past these systems have mostly focused on identifying and replacing single words @xcite . in the recent shared task on complex word identification @xcite the english portion of the dataset @xcite contained a proportion (26%) of multi-word expressions (mwes). this presented a challenge to the participants who needed to adapt their systems to identify these cases. whilst some did not provide any specific treatment of mwes, a common technique was to average features of the sub-words in an mwe to give an overall feature set @xcite . the winning submission however @xcite used a "greedy" approach whereby they assigned all mwes to the complex class. clearly, mwes deserve further attention in the lexical simplification world. there are a few prior attempts to integrate mwes into lexical simplification such as the use of decompositional rules to identify key words in mwes that can then be used for ranking @xcite . in other languages which involve heavy compounding, such as swedish, lexical simplification is multi-word by nature. lexical complexity can be assessed by identifying relevant lexical substrings in a target word and using these to compute features @xcite . more recently, recls @xcite ) was developed to perform lexical simplification recursively. if 2 consecutive words are marked as complex by the algorithm then they will be simplified one at a time, with the algorithm deciding whether to stop at each iteration. although this does allow the handling of mwes, it does not allow for the generation of phrases to replace the mwe, instead simplifying each word in place. sentence simplification @xcite is a combination of syntactic and lexical simplification. typically, it is accomplished using techniques from machine translation. the baseline dataset for this form of simplification is sari @xcite , which comprises target sentences and their reference simplifications. it is worth noting however, that many transformations are in fact multi-word lexical simplifications as we are trying to produce here, or combinations thereof. to the best of our knowledge, no corpora or methodology has been previously published on simplifying text in english by replacing multi-word fragments, which is the focus of this study. ## task and dataset the goal of multi-word lexical simplification is to simplify a given sentence in a natural language by replacing its fragment (a sequence consisting of one or several words) with another fragment, such that: note that we impose no restrictions on the grammatical roles of words included in the fragment, e.g. it does not have to constitute a multi-word expression. moreover, while it would be desirable for the fragment being replaced to be the hardest part in the sentence, we do not consider selecting such a fragment part of the mwls task, since it belongs to a problem complex word identification (cwi), which has recently been explored in a multi-word setting @xcite . given that this problem formulation is novel and no resources for it are available, we have decided to prepare a new dataset that would include a large number of such multi-word simplifications provided by human annotators. the rest of this chapter shows how we (a) collect sentences from corpora with complex language, (b) use a cwi solution to select fragments worth simplifying and (c) obtain manual replacements from crowdsourcing workers. finally, we briefly describe the characteristics of the obtained dataset. ## simplification method our approach to the mwls problem, called the plainifier, is an extension of the unsupervised method for single word lexical simplification by @xcite . following their work, we generate candidate replacements using bert predictions for a given context and rank them according to language-model probability, simplicity and similarity of meaning to the original text. nevertheless, there are important differences caused by the fact that both replaced and replacement text can be of any length. specifically, the candidate generation procedure (section 4.2) is a multi-step recursive procedure, which requires a specially trained version of bert (section 4.1). also, during candidate ranking (section 4.3), our system needs to compare the quality of the generated fragments even though they differ in length. note that plainifier does not perform cwi and thus its input consists of the the full sentence text and coordinates for the fragment that should be simplified. ## evaluation the evaluation of plainifier is performed by comparing its output to the replacements provided by humans in the crowdsourcing process. we only use the 1277 sentences, which have all 5 replacements. given that our method does not need training data, but offers possibility of tuning, we randomly select 100 sentences for this purpose, leaving the remaining 1177 for computing evaluation metrics. ## discussion our main motivation in this work was to improve the simplification process by introducing a task that goes beyond word-for-word substitution. the fact that we have been able to obtain replacements from annotators in a vast majority of cases suggests that mwls is indeed a valid task for human simplification. nevertheless, the cases with missing replacements indicate the possible limitations of this framework. apart from understandable cases of insufficient knowledge (especially in the biomedical domain), there have been many cases of words, such as circumcision, for which no simplification within the allowed 3 words is possible and they could only be made understandable by providing a longer explanation. we think this phenomenon points to a potential for future work on the cwi task -namely, in differentiating complex words that could be simplified through substitution (e.g. foes to enemies) from those requiring explanation (e.g. circumcision). another challenging aspect of the task is evaluation: when so many substitutions are possible, a value provided may be valid even if it was not included in gold-standard human annotations. the situation is similar to other generative tasks, such as summarisation or translation, where relaxed matching measures were developed (e.g. bleu). we have proposed the bow@k metric in similar spirit, but more work is necessary to assess, to what degree they correlate with manual assessment of quality. regarding automatic simplification, while the plainifier achieves better results than resource-based baselines, it leaves much room for improvement. the comparison of results in terms of prec@5 and potential shows that many relevant replacements are available in the candidate list, but the ranking method fails to assess them properly. manual inspection shows that the most challenging aspect is preserving the original meaning when computing the cosine of word vectors blends the semantic and stylistic similarity. our corpus is not user specific, although other studies have shown that user effects are present in simplification @xcite . we would hope that this work fits into the larger picture of ongoing lexical simplification research to adapt simplification to further users, genres and problem-types. ## conclusion in this article we have defined the task of multi-word lexical simplification and explored its nature while obtaining both human and automatic solutions to the problem. we hope that the resources we contribute, including the gathered data, developed code and trained models, will be of use for future researchers taking up this challenge.