L ex S ub C on: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution (2022)

Abstract: Lexical substitution is the task of generating meaningful substitutes for a word in a given textual context. Contextual word embedding models have achieved state-of-the-art results in the lexical substitution task by relying on contextual information extracted from the replaced word within the sentence. However, such models do not take into account structured knowledge that exists in external lexical databases. We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models that can identify highly-accurate substitute candidates. This is achieved by combining contextual information with knowledge from structured lexical resources. Our approach involves: (i) introducing a novel mix-up embedding strategy to the target word’s embedding through linearly interpolating the pair of the target input embedding and the average embedding of its probable synonyms; (ii) considering the similarity of the sentence-definition embeddings of the target word and its proposed candidates; and, (iii) calculating the effect of each substitution on the semantics of the sentence through a fine-tuned sentence similarity model. Our experiments show that LexSubCon outperforms previous state-of-the-art methods by at least 2% over all the official lexical substitution metrics on LS07 and CoInCo benchmark datasets that are widely used for lexical substitution tasks.

## introduction lexical substitution @xcite is the task of generating appropriate words which can replace a target word in a given sentence without changing the sentence's meaning. the increased research interest in lexical substitution is due to its utility in various natural language processing (nlp) fields including data augmentation, paraphrase generation and semantic text similarity. contextual word embedding models (such as elmo @xcite and bert @xcite ) have achieved state-of-art results in many nlp tasks. these models are usually pre-trained on massive corpora and the resulting context-sensitive embeddings are used in different downstream tasks @xcite . @xcite have achieved state-of-the-art results on the lexical substitution task by improving the bert's standard procedure of the masked language modeling task. however, the current state-of-the-art contextual models have yet to incorporate structured knowledge that exists in external lexical database into their prediction process. these lexical resources could boost the model's performance by providing additional information such as the definitions of the target and candidate words (in order to ensure that the candidate word is semantically similar to the target word and not only appropriate for the sentence's context) or by enriching the proposed candidate word list so it will not only be based on the vocabulary of the contextual model. in this paper, we present and publicly release foot_0 a novel framework for the lexical substitution task. specifically, (i) we are the first, to the best of our knowledge, to propose a novel mix-up embedding strategy that outperforms the previous state-of-the-art strategy of word embedding dropout for the input embedding of the target word in a contextual model @xcite for the task of predicting accurate candidate words; (ii) we propose the combined usage of features from contextual embedding models and external lexical knowledge bases in order to determine the most appropriate substitution words without modifying the meaning of the original sentence, such as introducing a new gloss (definition) similarity metric which calculates the similarity of the sentence-definition embeddings of the target word and its proposed candidates; (iii) we generate a highly accurate finetuned sentence similarity model by taking advantage of popular data augmentation techniques (such as back translation), for calculating the effect of each candidate word in the semantics of the original sentence; and, (iv) finally, we show that lexsub-con achieves state-of-the-art results on two popular benchmark lexical substitution datasets @xcite . ## related work the lexical substitution task consists of two subtasks: (i) generating a set of meaning preserving substitute candidates for the target word and (ii) appropriately ranking the words of the set by their ability to preserve the meaning of the initial sentence @xcite . however, lexical substitution models can also be tested in a "simpler" problem where the set of substitute candidates is composed of human-suggested words and the task is to accurately rank the substitute words that are provided @xcite . the authors in @xcite proposed the use of a word2vec model which utilizes word and context embeddings to represent the target word in a given context. their model ranked the candidate substitutions by measuring their embedding similarity. in @xcite the context2vec model was introduced where the context representation of the word was calculated by combining the output of two bidirectional lstm models using a feed-forward neural network. @xcite introduced contextualized word embeddings in a bidirectional language model (elmo). this allowed the model to change the embedding of a word based on its imputed meaning which is derived from the surrounding context. subsequently, @xcite proposed the bidirectional encoder representations from transformers (bert) which uses bidirectional transformers @xcite to create context-dependent representations. the authors in @xcite used elmo in the lexical substitution task by calculating the cosine similarity between the elmo embedding of the target word and all the candidate substitutes. in addition, @xcite achieved state-of-the-art results on the lexical substitution task by applying a dropout embedding policy to the target word embedding and by taking into account the similarity between the initial contextualized representation of the context words and their representation after replacing the target word by one of the possible candidate words. an analysis of state-of-the-art contextual model on the lexical substitution task was presented in @xcite . finally, external knowledge from knowledge bases has been used to enhance the performance of deep learning models. sense-bert @xcite was pre-trained to predict the semantic class of each word by incorporating lexical semantics (from the lexical database wordnet @xcite ) into the model's pre-training objective. furthermore, @xcite and @xcite used external knowledge (namely wordnet) in order to enhance word embeddings and to create more accurate representations of rare words. ## lexsubcon framework in the lexical substitution task, a model aims to firstly generate a set of candidate substitutions for each target word and secondly to create an appropriate ranking of the elements of the candidate set. in addition, there are two main conditions for a lexical substitute model to satisfy: (i) to be semantically similar to the target word and (ii) and to be compatible with the given context (sentence) @xcite . we present the lexsubcon framework which achieves state of the art results on the lexical substitution task by combining contextual information with knowledge from structured external lexical resources. the architecture of lexsubcon is depicted in figure 1 . the key characteristic of lexsubcon is its capability of unifying different substitution criteria such as contextualized representation, definition and sentence similarity in a single framework in order to accurately identify suitable candidates for the target words in a specific context (sentence). ## conclusion this paper presents lexsubcon, an end-to-end lexical substitution framework based on contextual embedding models. lexsubcon establishes a new mix-up embedding strategy that outperforms the previous sota strategy of word embedding dropout for the embedding of the target word for the task of predicting accurate candidate words. lexsubcon introduces the combined usage of features from contextual embedding models and external lexical knowledge bases in order to calculate accurately the semantic similarity between a target word and its candidates. we confirm that these features can improve the lexsubcon's performance as it outperforms other state-of-the-art results on two benchmark datasets. as for future work, we plan to address the limitations of this study including: (i) examining the effect of using other models as the basis of our features (e.g. albert @xcite ); (ii) exploring other candidate features for the ranking of the candidates (e.g. parser information @xcite ) (iii) testing lexsubcon in datasets of other languages using multi-language lexical databases (e.g. multiwordnet @xcite or @xcite ) to investigate further the model's general availability.