Grounding the Lexical Substitution Task in Entailment (2023)

Abstract: Existing definitions of lexical substitutes are often vague or inconsistent with the gold annotations. We propose a new definition which is grounded in the relation of entailment; namely, that the sentence that results from the substitution should be in the relation of mutual entailment with the original sentence. We argue that the new definition is well-founded and supported by previous work on lexical entailment. We empirically validate our definition by verifying that it covers the majority of gold substitutes in existing datasets. Based on this definition, we create a new dataset from existing semantic resources. Finally, we propose a novel context augmentation method motivated by the definition, which relates the substitutes to the sense of the target word by incorporating glosses and synonyms directly into the context. Experimental results demonstrate that our augmentation approach improves the performance of lexical substitution systems on the existing benchmarks.

## introduction lexical substitution is the task of finding appropriate replacements for a target word in a given context sentence. this task was first introduced as an application-oriented alternative to word sense disambiguation (wsd) that does not depend on a predefined sense inventory @xcite . lexical substitution has been applied in various tasks, such as word sense induction @xcite , lexical relation extraction @xcite , and text simplification @xcite . lexical substitution continues to be an important area of research in nlp. for instance, it can be used to probe the ability of nlp models to capture contextual meaning, as substitutes can vary depending on the sense of the word. furthermore, professional writers often need good substitutes in a specific context, which cannot be found by simply looking them up in a thesaurus. many definitions used in the literature to describe lexical substitution are either vague or inconsistent with the evaluation datasets. for example, @xcite and @xcite leave the criteria for lexical substitution to the discretion of human annotators. studies such as @xcite and @xcite require substitutes to be synonyms, which creates a discrepancy with established lexical substitution benchmarks that allow annotators to provide slightly more general terms (hypernyms) @xcite . for example, while the two words are not synonyms, vehicle can be considered as a valid substitute for car if the context clearly refers to a car. most prior work requires substitutes to preserve the meaning of the original sentence @xcite @xcite @xcite @xcite @xcite @xcite . however, as we show in this work, not all gold substitutes necessarily preserve the meaning of the sentence taken in isolation. we propose a definition of lexical substitution that is more precise and well-founded. our aim is not only to address the inconsistency in the literature but also to align the task definition with established evaluation datasets. we draw on insights from natural language inference (nli), which provides a framework for understanding the semantic relationship between sentences and words. according to our definition, the sentence that results from a lexical substitution must be in the relation of mutual entailment with the original sentence. for example, position is a suitable substitute for post in the sentence "i occupied a post in the treasury" because the two sentences entail each other. the entailment criterion takes into account the implicit background knowledge @xcite , which allows lexical substitution to generalize over simple synonym replacement, encompassing a wider range of semantic relations, such as hypernymy and meronymy @xcite . the classification of the entailment relation between two sentences requires the identification of the target word's sense. for example, position is a proper substitute for post only if it is used in the sense corresponding to "job in an organization". based on this observation, we develop an augmentation method that helps to ground the substitutes by incorporating glosses and synonyms of the target word's sense directly into the context. since the word sense is latent, the method leverages a wsd system to account for the probabilities of each candidate sense. we show the effectiveness of the proposed definition and our augmentation method through experiments on existing lexical substitution datasets. our analysis indicates that the proposed definition encompasses gold substitutes that could not previously be explained by existing definitions. furthermore, our empirical evaluation shows that our augmentation method improves the performance on the lexical substitution benchmarks by up to 4.9 f1 points, surpassing the previous state-of-the-art models in certain settings. the main contributions of this paper are as follows. ## related work on lexical substitution in this section, we review the available datasets and provide a brief overview of the prior work. ## entailment-based lexical substitution in this section, we provide background information about entailment, present the theoretical formulation of the proposed definition, and demonstrate its suitability through empirical validation. ## sense-based augmentation method in this section, we describe our sense-based augmentation method for lexical substitution. our approach is based on the observation that knowing the sense of the target word is key to deciding whether a substitution induces an entailment relation between the two sentences. for example, position is a proper substitute for post in some context only if the latter is used in the sense corresponding to "job in an organization". we posit that inserting sense glosses directly into the context will help lexical substitution systems identify substitutes that are mutually entailed by the original context. our hypothesis is supported by prior findings that this technique works well for semantic tasks such as wsd @xcite and idiomaticity detection @xcite . our method is based on two stand-alone modules: a wsd system and a lexical substitution generation system. the method is sufficiently flexible to incorporate new systems as the state of the art on those two tasks continues to improve. the only requirement is that these systems output probabilities for each candidate sense or substitute. the formula below is used to combine the probabilities from the two systems. figure 1 shows an example of soft constraint augmentation. let c t be a context sentence containing the target word t, w be a candidate substitute, and @xmath0. under the assumption that the substitutes depend on the sense of the target word, the conditional probability p (w|c t ) can be derived by marginalizing the senses out: in the equation above, we model p (s|c t ) using a wsd system, and obtain p (w|c t , s) from a lexical substitution system that operates on the context augmented with sense information. motivated by the work of luan et al. ( 2020 ), we experiment with two types of constraint: hard and soft. in the hard-constraint approach, a wsd system is used to identify the most likely sense of the target word, which is effectively assigned the probability of 1.0. next, the glosses and synonyms corresponding to this sense are retrieved from a lexical resource and inserted in parentheses after the target word. this augmented context is then passed to a lexical substitution system, which generates substitutes along with their substitute probabilities. in the soft-constraint approach, for each possible sense of the target word, a wsd system first computes its probability, the context is augmented with glosses and synonyms of that sense, and finally a lexical substitution system generates and assigns final probabilities to candidate substitutes using the formula above. soft constraint allows grounding of lexical substitutes in the target word senses, while taking into account the probability of each candidate sense. we posit that considering all candidate senses and their probabilities should work better than committing to a single most likely sense, by improving robustness against wsd errors. in addition, in some cases, the context itself may not provide enough information to reliably disambiguate the sense of the target word. we verify this hypothesis experimentally in the next section. ## experiments in this section, we investigate the effectiveness of our dataset and augmentation method in improving the performance of lexical substitution systems. the experiments were conducted on a machine with two nvidia geforce rtx 3090 video cards. ## conclusion we consider the new entailment-based definition and formalization of lexical substitution as the principal contribution of this paper. the new wnsub dataset and the context augmentation method are inspired by our theoretical analysis. the experiments demonstrate that both innovations lead to performance improvements on the standard lexical substitution benchmarks, which we interpret as empirical validation of the theoretical approach. in the future, we plan to explore the generalizability of our approach to other languages, as well as cross-lingual lexical substitution.