G ene S is: A G enerative A pproach to S ubstitutes in C ontext (2021)

Abstract: The lexical substitution task aims at generating a list of suitable replacements for a target word in context, ideally keeping the meaning of the modified text unchanged. While its usage has increased in recent years, the paucity of annotated data prevents the finetuning of neural models on the task, hindering the full fruition of recently introduced powerful architectures such as language models. Furthermore, lexical substitution is usually evaluated in a framework that is strictly bound to a limited vocabulary, making it impossible to credit appropriate, but out-of-vocabulary, substitutes. To assess these issues, we proposed GeneSis (Generating Substitutes in contexts), the first generative approach to lexical substitution. Thanks to a seq2seq model, we generate substitutes for a word according to the context it appears in, attaining state-of-the-art results on different benchmarks. Moreover, our approach allows silver data to be produced for further improving the performances of lexical substitution systems. Along with an extensive analysis of GeneSis results, we also present a human evaluation of the generated substitutes in order to assess their quality. We release the fine-tuned models, the generated datasets, and the code to reproduce the experiments at https://github.com/SapienzaNLP/genesis .

## introduction the lexical substitution task @xcite requires a system to provide adequate replacements for a target word in a given context. through the years, two lexical substitution variants have been proposed, i.e., candidates ranking and substitutes prediction @xcite . while the former aims at ranking a list of predefined candidate substitutes for a word in a given context, the latter is more challenging, requiring a system to output a sorted list of replacements without any predefined substitutes inventory. although it is not explicitly required by either of the two tasks, a good substitution system is expected to capture the semantics of its input and implicitly perform a soft disambiguation. for example, denoting bright as target word in the context sentence "she is a bright student", we expect a good substitution system to provide a set of substitutes closer to {intelligent, clever, smart} than to {luminous, clear, light}. thanks to this implicit disambiguation capability, lexical substitution has shown its usefulness in several scenarios, such as word sense induction @xcite @xcite , data augmentation @xcite , word sense disambiguation @xcite and semantic role labeling @xcite . however, despite having been employed in numerous downstream tasks, the lexical substitution task still presents unresolved issues that need to be addressed. first, the shortage of large-scale corpora annotated with the expected substitutes hinders the use of supervised techniques, including powerful transformer-based language models, thus leaving the task in a possibly sub-optimal setting. second, the evaluation metrics provided for the task are bound to the test vocabulary, hence they fail to capture the quality of substitutes outside the vocabulary; moreover, the vocabulary is usually small and often biased by the particular linguistic style and background of the annotators who developed the datasets. 1 in this paper, we focus on substitutes prediction and address the above problems by proposing gen-esis, a generative approach to lexical substitution. we find that not only is this novel approach effec-tive when tested on the lexical substitution task, but also that it can be applied to generate substitutes from raw text, enabling the effortless construction of large-scale silver data. moreover, we conduct an annotation task to analyze the results of our model and to validate out-of-vocabulary generations. our contribution is threefold: through the years, several approaches have been developed to tackle the lexical substitution tasks, but, to the best of our knowledge, ours is the first attempt to apply a generative approach to it. in what follows, we first review the principal approaches and resources for the lexical substitution tasks, and then provide a brief overview of generative methods across different fields. ## genesis the task of substitutes prediction requires finding replacements for a target word in a context that ideally do not modify the overall meaning of that context. more formally, given a target word w t occurring in a context sentence @xmath0, . . . , w n , a substitution system has to assemble a ranked list s of possible replacements for w t according to its context x. consider as an example the context the roses are bright. we tackle the lexical substitution task with a two-stage process: first, we use a seq2seq model that takes as input both the context and the target, and generates several possible lists of substitutes (substitutes generation, section 3.1); second, we process the substitutes collected with the first step to obtain the final, ranked list (substitutes ranking, section 3.2). the whole process is described in figure 1 . throughout the paper, we will consider each target word w t to be univocally associated with its part-of-speech (pos) tag. to improve readability we discard pos tags from the notation. ## vocabulary definition one of the challenges in the lexical substitution task is the lack of a predefined substitute inventory, i.e., for each target word we lack a reference list of possible replacements. importantly, with gen-esis we can produce approximately any word in the english vocabulary as substitute, although standard evaluation benchmarks consider valid only the words in the test vocabulary. to reach a suitable trade-off between the generative power of the model and the necessity of a fair evaluation, we define an output vocabulary that the model has to stick to, i.e., we discard any generated word that is not contained in it (section 3.2). to build our vocabulary, we take advantage of wordnet 3.0, a widely-used lexicographic resource structured as a graph. each wordnet node is a synset, i.e., a set of different lexicalizations with the same meaning and pos, while edges represent semantic relations between synsets, such as hyponymy and hypernymy. for example, one of the synsets for the adjective bright is {bright, brilliant, vivid}, that is connected through the similar-to semantic relation to the synset {colorful, colourful}. for each target w t we compute a set of synsets d wt that defines the output vocabulary. we initialize d wt as the set of synsets s wt where w t appears. 3 then, for each s @xmath1; finally, for each neighbor n that is connected to s wt through a hyponymy, hypernymy, similar-to or see-also relation, we add all the neighbors of n to d wt . we define as possible substitutes for w t the union of all the lexicalizations appearing in d wt . this procedure, visualized in figure 2 , builds a vocabulary that covers all the senses enumerated in wordnet for a given target, defining a reduced range of available substitutions that is still challenging for the task. to provide a quantification of the coverage of the output vocabulary, we report that, when computed for the lst targets, it includes 25 842 distinct substitute words, while the figure 2: a visual example of how the vocabulary is constructed. we consider s w , one of the two synsets where the noun wine appears (purple oval). first, we consider all its neighbors (orange ovals), then for each neighbor n of s w connected through hypernymy, hyponymy, similar-to and see-also relations (double orange oval), we include all the neighbors of n (green ovals). the neighbors of synsets connected through different relations to s w are discarded (grey ovals). ## dataset generation genesis is able to generate substitutes starting from a word in its context. thus, starting from a source dataset c of target words in context, we can exploit genesis to produce ranked lists of substitutes and, associating the generated substitutes with the targets, obtain silver datasets for the lexical substitution task. to this end, first, we finetune genesis on a gold dataset for the lexical substitution task; then, we give as input to the finetuned model the corpus c, generating as output a list of replacements for each input instance. the input instances, associated with the generated substitutes, constitute the silver corpus. to ensure the quality of the generated substitutes, we apply a similarity threshold λ on the ranking step of genesis (cf. section 3), keeping only the substitutes whose similarity to the target is higher than λ. as source dataset c we exploit semcor @xcite , a manually annotated corpus where instances are sense-tagged according to the wordnet sense inventory foot_3 . while it is typically used as a training corpus for english word sense disambiguation (wsd), as we show, its manually-curated sense distribution is also beneficial for lexical substitution. indeed, having a frequency of target words that model we use bart @xcite as seq2seq model, trained through the radam optimiser (lr 10 - foot_4 ); we train it for a maximum of 100 epochs, with early stopping and patience set to 2 epochs. the input is fed to the model in batches of up to 600 tokens. to obtain the contextualized representations in equations ( 2 ) and ( 3 ) we use the average of the last four 5 hidden layers of bert large cased. both bart and bert are used through the huggingface @xcite implementations. ## experiments once all the parameters have been tuned, we train genesis on ct t , cut the substitutes generated according to the output vocabulary, apply the fallback strategy and test on the lst dataset. baseline using a predefined vocabulary limits the possible outputs of our model; therefore, to assess whether the performances of genesis are mainly influenced by the restricted vocabulary, we comparison systems we choose as comparison systems the two most recent approaches to the task, i.e., the bert-based system proposed by zhou et al. (2019) and the best-performing solution presented by arefyev et al. (2020), i.e., an xlnetbased model enhanced with the injection of specific embedding information about the target word. these two models achieve the currently highest reported results on the task. in these approaches the language models are used in a feature-based approach, i.e., they are not finetuned for the task. as already noted by @xcite , both models output a probability distribution over a bpe-based vocabulary, making it tricky to reconstruct words at inference time. genesis, instead, overcomes this limit by relying on the decoding strategy of the generative model. ## results we report our results in as regards the results obtained with genesis, in each configuration we report the average of five runs with their standard deviation. first, we inspect the performances of genesis without output vocabulary and without fallback strategy (genesis - ## qualitative evaluation our quantitative analysis shows that the substitutes produced by genesis are good enough to outperform the previous approaches to the task. the evaluation setting, though, is inherently limited to a fixed vocabulary 7 . hence, we devise an annotation task to assess the quality of generated substitutes, investigating whether genesis is able to generate substitutes that, even when not appearing in the gold standard, are judged good replacements by human annotators. annotation task we set up a test where an annotator is provided with a target word in context and a set of substitutes that are equally distributed among the gold and the generated ones. the annotator is required to select, if there are any, all the substitutes that are not suitable replacements for the target in the given context. we select three annotators with certified proficiency in english and previous experience in linguistic annotation tasks and present them with a sample of 322 test instances drawn from the lst dataset 8 . the annotators are asked to select the inappropriate substitutes from an anonymized shuffled set of three gold and three generated substitutes, obtained with genesis trained on ct t . for all the instances, the gold substitutes do not appear in the generated ones and vice versa. the annotation guidelines are reported in the supplementary material (section d). inter-annotator agreement since each annotator may select more than one substitute, we measure the inter-annotator agreement (iaa) using kraemer's κ coefficient @xcite , an extension of the better known cohen's κ @xcite that allows multiple answers to be provided by annotators. we follow @xcite to interpret κ values in the range (0.4, 0.6] as moderate agreement, values in (0.6, 0.8] as substantial agreement and those in (0.8, 1.0) as almost perfect agreement. annotations are usually considered reliable if their iaa agreement is equal or higher than 7 we recall from section 4 that the vocabulary of lst has slightly more than 3000 words. 8 the sample size is significant with respect to the source dataset with confidence level of 95% and a margin error @xmath2. the annotation interface was developed through label studio https://github.com/heartexlabs/ label-studio#try-out-label-studio . 0.67 @xcite . ## conclusions in this paper we presented genesis, the first generative approach to lexical substitution. the method is simple but versatile: by finetuning a seq2seq model and post-processing its output we are able to generate appropriate substitutes for target words in contexts. testing genesis on the lexical substitution task, we show performances that surpass the state of the art on several measures. at the same time, our approach can be used to produce large-scale silver data, which, when used as train- context gold substitutes generated substitutes i think this idea has tremendous promise. great, wonderful, enormous, terrific, huge, vast huge, great, big, abundant, major, enormous, terrific, wonderful, excellent, ample, plentiful a little bit of rest and relaxation is waiting for michelle after the college national finals rodeo. respite, repose, quiet, leisure, inactivity, sleep recuperation, repose, reprieve, respite, relaxation, break well worth buying, well worth reading, but skip the bits you find irrelevant -another type of blended learning! miss out, disregard, jump over, miss, avoid, jump, ignore, leave ditch, disregard, dismiss, omit, neglect, leave use market tools to address environmental issues, such as eliminating subsidies for industries that severely harm the environment, like coal. badly, gravely, seriously, dramatically extremely, highly, significantly, hugely, greatly, really, extensively, tremendously, exceptionally very late in the movie, he finally does marry scarlett. far, on, near, end, near the end of, well into, latterly, last minute very, soon, further, too, extra if you wish to collect your robes earlier you should contact the above number to arrange collection. he was bright and independent and proud. let your child pick one bug to glue on the lid. insect fly, insectoid, critter, insect, creature