W i C = TSV = WSD : On the Equivalence of Three Semantic Tasks (2022)

Abstract: The Word-in-Context (WiC) task has attracted considerable attention in the NLP community, as demonstrated by the popularity of the recent MCL-WiC SemEval shared task. Systems and lexical resources from word sense disambiguation (WSD) are often used for the WiC task and WiC dataset construction. In this paper, we establish the exact relationship between WiC and WSD, as well as the related task of target sense verification (TSV). Building upon a novel hypothesis on the equivalence of sense and meaning distinctions, we demonstrate through the application of tools from theoretical computer science that these three semantic classification problems can be pairwise reduced to each other, and therefore are equivalent. The results of experiments that involve systems and datasets for both WiC and WSD provide strong empirical evidence that our problem reductions work in practice.

## introduction this paper answers an open question about the the relation between two important tasks in lexical semantics. word sense disambiguation (wsd) is the task of tagging a word in context with its sense @xcite . the word-in-context (wic) problem is the task of deciding whether a word has the same meaning in two different contexts @xcite . a crucial difference between the two tasks is that wsd depends on a pre-defined sense inventory foot_0 while wic does not involve any identification or description of word meanings. despite ongoing interest in both tasks, there is substantial disagreement in the literature as to whether wic is a re-formulation of wsd (e.g. @xcite ) or an entirely distinct task (e.g. @xcite ). by establishing that wsd and wic are equivalent, we construct a theoretical foundation for the transfer of resources and methods between the two tasks. wsd has been intensively studied for decades, while wic has recently attracted considerable attention from the research community. for example, the mcl-wic semeval shared task @xcite attracted 48 teams, and wic instances have been integrated into the superglue benchmark @xcite . understanding how the two tasks relate to each other allows us to correctly interpret and confidently build upon those results, including prior work on using wsd systems for wic (e.g. @xcite ). we establish the theoretical equivalence of wic and wsd by specifying reduction algorithms which produce a solution for one problem by applying an algorithm for another. in particular, we employ the target sense verification (tsv) task @xcite as an intermediate step between wsd and wic, and specify three reductions: wic to wsd, wsd to tsv, and tsv to wic. we formalize the three problems using a common notation, and provide both theoretical and empirical evidence for the correctness of our reductions. while we focus on english in this paper, we make no language-specific assumptions. 2 the soundness of all three tasks hinges on the consistency of judgments of sameness of word meaning, whether with respect to discrete sense inventories (wsd), a representation of a single sense (tsv), or two occurrences of a word (wic). we posit that different instances of a word have the same meaning if and only if they have the same sense. this empirically falsifiable proposition, which we refer to as the sense-meaning hypothesis, implies that wic judgements induce sense inventories that correspond to word senses. this counter-intuitive finding has intriguing implications for the task of word sense induction (wsi), as well as algorithmic wordnet construction. we empirically validate our hypothesis by conducting multiple experiments and analyzing the results. in particular, we test our wsd-to-wic and wic-to-wsd reductions on standard benchmark datasets using state-of-the-art systems. we find that our reductions perform remarkably well, revealing no clear counter-examples to our hypothesis in the process. our contributions are as follows: (1) we answer the open question of the relation between wic and wsd by constructing a theoretical argument for their equivalence, which is based on a novel sensemeaning hypothesis. (2) we carry out a series of validation experiments that strongly support the correctness of our reductions. (3) we release the details of our manual analysis and annotations of the instances identified in the validation experiments. ## theoretical formalization in this section, we formally define the three problems, present a theoretical argument for their equivalence, and specify the reductions. ## wic datasets in this section, we discuss and analyze the existing wic datasets with the aim of finding a dataset suitable for validating our equivalence hypothesis. an instance that contradicts one of the reduction equivalences in section 2.3 would be an exception to the hypothesis. since natural language is not pure logic, falsifying the hypothesis would require finding that such exceptions constitute a substantial fraction of instances, excluding errors and omissions in lexical resources. ## empirical validation in this section, we aim to quantify and analyze any apparent counter-examples to the sense-meaning hypothesis which are identified in the process of testing the wsd-to-wic and wic-to-wsd reductions. we are particularly interested in the exceptions that cannot be attributed to errors in the resources that are used to implement the reductions, because such exceptions represent potential evidence against our hypothesis. ## discussion having presented theoretical and empirical evidence for the equivalence of wic, wsd, and tsv, we devote this section to the discussion of the relationship between wordnet and wic. most english wic and tsv datasets are based, in whole or in part, on wordnet. if no sense inventory is used for grounding decisions about meaning, the inter-annotator agreement is reported to be only about 80% @xcite . for the mcl-wic dataset, however, annotators consulted other dictionaries, and obtained "almost perfect agreement" @xcite . this suggests that sense inventories, and semantic resources in general, are crucial to reliable annotation for semantic tasks. however, because the exact mcl-wic procedure for resolving differences between dictionaries is not fully specified, and because such dictionaries vary in their availability, the correctness of the annotations cannot be readily verified (c.f. section 4.4). our experiments provide evidence that, even when the wordnet sense inventory is not explicitly used in constructing wic datasets, wic annotations nevertheless tend to agree with word-net sense distinctions, as our hypothesis predicts. namely, the mcl-wic instances in which both focus tokens have the same sense are almost always annotated as positive by the mcl-wic annotators. the converse also holds, with any exceptions being explainable by errors in the resources. thus, empirical validation confirms our sense-meaning hypothesis, which implies that the meaning distinctions induced by wic judgements closely match wordnet sense inventories. this is a remarkable finding given the high granularity of wordnet. we postulate that the adoption of wordnet as the standard sense inventory for wic would have several practical benefits: (1) it has been adopted as the standard inventory for wsd, and so would simplify multi-task evaluation; (2) it allows seamless application of systems across datasets; (3) it facilitates rapid creation of new wic datasets based on existing sense-annotated corpora; (4) it is freely available; (5) it can be modified and extended to correct errors and omissions @xcite ; and finally (6) it can be extended to facilitate work with other languages, as in the xl-wic dataset @xcite . in addition, wordnet has strong theoretical advantages. its fine granularity is a consequence of its grounding in synonymy and lexical concepts. therefore, the sense distinctions found in other dictionaries either already correspond to different wordnet concepts, or should lead to adding new concepts to wordnet. furthermore, unlike in dictionaries, senses of different words in wordnet are linked via semantic relations such as synonymy and hypernymy, which facilitate an objective assignment of every word usage to a single wordnet concept. this property of wordnet may be the reason that the wsd methods based on sense relation information have surpassed the inter-annotator agreement ceiling of around 70% @xcite . ## conclusion we formulated a novel sense-meaning hypothesis, which allowed us to demonstrate the equivalence of three semantic tasks by mutual reductions. we corroborated our conclusions by performing a series of experiments involving both wsd and wic tools and resources. we have argued that these relationships originate from the wordnet properties, which are highly desirable in semantics research. we expect that our findings will stimulate future work on system development, resource creation, and joint model optimization for these tasks.