CIL ex: An Investigation of Context Information for Lexical Substitution Methods (2022)

Abstract: Lexical substitution, which aims to generate substitutes for a target word given a context, is an important natural language processing task useful in many applications. Due to the paucity of annotated data, existing methods for lexical substitution tend to rely on manually curated lexical resources and contextual word embedding models. Methods based on lexical resources are likely to miss relevant substitutes whereas relying only on contextual word embedding models fails to provide adequate information on the impact of a substitute in the entire context and the overall meaning of the input. We proposed CILex, which uses contextual sentence embeddings along with methods that capture additional context information complimenting contextual word embeddings for lexical substitution. This ensured the semantic consistency of a substitute with the target word while maintaining the overall meaning of the sentence. Our experimental comparisons with previously proposed methods indicated that our solution is now the state-of-the-art on both the widely used LS07 and CoInCo datasets with P@1 scores of 55.96% and 57.25% for lexical substitution. The implementation of the proposed approach is available at https://github.com/sandaruSen/CILex under the MIT license.

## introduction lexical substitution is an important natural language processing (nlp) task, which aims to generate and rank suitable candidate words to replace a given target word, while maintaining the meaning of the given sentence. lexical substitution is used in a wide range of nlp tasks like data augmentation, paraphrase generation, word sense induction, or text simplification @xcite . through the years, different approaches have been introduced for lexical substitution but, due to the paucity of annotated data, most of the lexical substitution systems rely on unsupervised methods based on lexical resources or pre-trained language models @xcite . earlier, methods typically relied entirely on manually curated lexical resources like wordnet @xcite . the synonyms obtained from such resources were then ranked based on their suitability evaluated by a similarity metric and predefined rules. some approaches used vector-based modelling and distributional vectors based on syntactic context to obtain the most suitable synonyms @xcite . recent advances in contextual language models like bidirectional encoder representations from transformers (bert) @xcite , embeddings from language models (elmo) @xcite , and xlnet @xcite have resulted in major breakthroughs in nlp. because these models carry contextual information and have the ability of context-sensitive modelling of word probabilities, they have achieved the stateof-the-art (sota) results in lexical substitution as well. some recent research efforts have improved lexical substitution by modifying the architecture of contextual embedding models @xcite whereas others integrated lexical resources to contextual embeddings to obtain the most suitable set of substitutes @xcite . methods based on lexical resources may fail to obtain the most relevant substitutes given that they predominantly focus on synonyms, hypernyms, and hyponyms. moreover, they fail to consider the influence of the substitute on the global context of the given sentence @xcite . even though the contextual word embedding models consider the given context, they are unable to provide sufficient knowledge about the effect of the substitute on the overall meaning of a sentence. to address these issues, the aim of this paper was to investigate the effect of introducing contextual sentence embeddings alongside contextual word embeddings in the lexical substitution task. we first analysed the impact of the addition of contextual sentence information and then, investigated other methods to improve lexical substitution @xcite . the proposed solution achieved the sota results on the ls07 and coinco datasets. the main contributions of the paper were as follows: ## related work researchers have identified different subtasks under lexical substitution, namely substitution generation, substitution selection, and substitution ranking @xcite . out of these, substitution generation and substitution ranking are considered as the two main subtasks, where the former focuses on generating possible substitutes for a target word given the context, and the latter aims to rank the substitutes @xcite . ranking of the substitutes may include ranking of the generated substitutes by the lexical substitution method or a much simpler ranking problem with ranking of the set of substitutes obtained from the human-annotated data given in the dataset @xcite . early efforts on lexical substitution relied mainly on manually curated lexical resources like word-net @xcite which evolved to the use of unsupervised methods and models based on distributional similarity. word embeddings, such as word2vec @xcite were used to obtain substitutes by selecting words with embeddings residing near the target word. the embedding similarity obtained from these models was used to rank the substitutes @xcite . the model context2vec, introduced by @xcite , produced the contextual embeddings for a given target word by combining the output of two bidirectional long short-term memory networks (lstms) using a feedforward neural network. this model was successfully applied for the ranking of given substitutes in the lexical substitution task. elmo used a similar approach with bidirectional lstms where the embedding of a given word was created based on the meaning of the context it appeared @xcite . elmo was used in the lexical substitution task to rank the candidates by calculating the cosine similarity between the contextual embeddings from the elmo for the target word and all the substitutes for the target word @xcite . the introduction of transformers resulted in major advances in a wide range of nlp tasks @xcite . transformer-based language models trained on extra large corpora like bert @xcite and a robustly optimised bert (roberta) @xcite used a masked language modelling objective where tokens were replaced by a special token @xcite in the training process. further improving on the bertbased language models, xlnet was introduced; it used an autoregressive pre-training method with a permutation-based language modelling objective without corrupting the input with masks @xcite . these contextual embedding models were extensively used for lexical substitution. the authors in @xcite relied on contextual word embeddings for lexical substitution. they modified the bert architecture with a dropout embedding policy where the target word was partially masked with the aim of providing some information of the target word in the prediction. to evaluate the fitness of possible candidates, the authors introduced a validation score which was computed using representations in the top four layer's of bert. the proposed method achieved the sota results for lexical substitution. @xcite presented an extensive analysis on different contextual embedding models for lexical substitution. the authors, in addition to the model probability predictions for the target word, computed the word embedding similarity of the target with all the words in the model's vocabulary for final predictions. their experimental comparisons indicated that xlnet had superior performance compared to other contextual word embedding models like elmo, bert, and roberta at providing substitutes given no changes in the basic architecture of the models. title case prepend token list preprocessors model prediction score proximity similarity score probability estimators lower case lemmatize exclude target word + postprocessors candidates, probabilities xlnet candidates sentence similarity score wordnet score/ gloss score validation score + final candidates michalopoulos et al. (2022) presented a framework, which integrated external knowledge from wordnet to bert for lexical substitution. the authors computed a proposal score based on bert and wordnet, a gloss sentence similarity score based on wordnet definitions, a sentence similarity score using a contextual sentence embedding model, and the validation score introduced by @xcite to obtain the final set of substitutes. compared to @xcite and @xcite , additionally their approach integrated both wordnet and contextual sentence embeddings for lexical substitution. these authors' most recent methods on lexical substitution relied mostly on contextual word embedding models @xcite and the use of a variety of methods that provide contextual information @xcite . in our study, we specifically focused on the added value of sentence context for lexical substitution based on contextual sentence embeddings. we based our experiments on @xcite which gave evidence of xlnet outperforming other contextual embedding models for lexical substitution, and analysed the impact of adding sentence context information. additionally, we introduced a wordnet-based metric and investigated the methods proposed by @xcite ; @xcite for lexical substitution. ## the cilex solution in this study, we investigated the impact of sentence context and the methods that capture context information, and proposed a lexical substitution solution called cilex (figure 1 ). we followed @xcite and @xcite as the basis of our work. ## experiments this section gives an overview of the datasets used in the experiments, evaluation metrics, the experimental setup, and the results from performance evaluations. ## discussion in this paper, we analysed the impact of introducing contextual sentence embeddings and methods which provide additional context information for lexical substitution, thereby ensuring that the substitutes are semantically consistent while preserving the overall meaning of the sentences. the results from the proposed cilex solution outperformed previous sota methods for lexical substitution on ls07 and coinco datasets. our results indicated that accounting for sentence context information has improved the performance on the substitution generation task. this is demonstrated by our approach (i.e., cilex) outperforming @xcite sota contextual word embedding-based method on two datasets. however, interestingly based on our results, the performance did not improve in the candidate ranking task, which requires further investigation. the results from our ablation study on the methods, which provide additional context information, indicated that they improve the lexical substitution task. analogously with @xcite , our results implied that the model achieves the worst performance when gloss sentence similarity score was used as additional context information. this could be mainly due to wordnet being manually curated and definitions for words obtained from wordnet might not reflect the meaning of the words in the given context. furthermore, for certain words, definitions may not be available on wordnet. based on our results, we also observed an increase in the final results when sentence similarity score was used to obtain additional context information as opposed to wordnet score, gloss sen-tence similarity score, and validation score. this is likely because the sentence similarity can appropriately identify if a substitute fits the context and ensures that the overall meaning of the sentence is unchanged. however, when compared with @xcite , their approach yielded an improvement in results when the validation score introduced by @xcite was used. these observations hindered us to come to a conclusion as to which of the components contributes most to lexical substitution and illustrated that different components contributed differently. our experiments gave evidence of the importance of the initial model/method used to obtain the first set of substitutes. the proposed cilex solution, which relied on an xlnet-based method to obtain the initial set of substitutes, outperformed lexsubcon @xcite which used a bert-based method to obtain the initial set of substitutes. cilex showed an improvement of ∼4% on ls07 dataset and ∼6.75% on coinco dataset. these insights guided us to conclude that the initial model used to obtain the possible candidates also had a direct impact to the lexical substitution which requires a thorough investigation. the proposed cilex solutions are based on pretrained language models and therefore are generalisable. for domain specific applications, the proposed approach can be easily transferable by replacing the pre-trained language models with respective domain specific models. further analysis of the substitution generation results indicated that samples which did not have successful predictions may contain potential substitutes based on wordnet. this illustrated the impact of the annotation subjectivity in the interpretation of the performance of the lexical substitution task. as future work, we intend to extend our experiments and analyse methods that can provide context information to improve lexical substitution. moreover, the impact on the candidate ranking task will be explored further as future work. we also plan to look into the applicability of the proposed approach for other tasks like word sense induction @xcite and word sense disambiguation. ## conclusion we have presented and released a solution for lexical substitution investigating the impact of sentence context obtained from contextual sentence embed-dings. we have introduced and further integrated methods which capture additional context information as proposed by @xcite ; @xcite . the unified solution has achieved the sota results on two benchmark datasets; ls07 and coinco. we have also analysed and evaluated effects of different methods that provide contextual information and their contribution for lexical substitution. the results have demonstrated the importance of sentence context information obtained using contextual sentence embeddings in lexical substitution.