WiC-TSV : A n Evaluation Benchmark for Target Sense Verification of Words in Context (2021)

Abstract: We present WiC-TSV, a new multi-domain evaluation benchmark for Word Sense Disambiguation. More specifically, we introduce a framework for Target Sense Verification of Words in Context which grounds its uniqueness in the formulation as binary classification task thus being independent of external sense inventories, and the coverage of various domains. This makes the dataset highly flexible for the evaluation of a diverse set of models and systems in and across domains. WiC-TSV provides three different evaluation settings, depending on the input signals provided to the model. We set baseline performance on the dataset using state-of-the-art language models. Experimental results show that even though these models can perform decently on the task, there remains a gap between machine and human performance, especially in out-of-domain settings. WiC-TSV data is available at https://competitions.codalab.org/competitions/23683 .

## introduction word sense disambiguation (wsd) is a longstanding task in natural language processing and artificial intelligence. while progress has been made in recent years, mainly thanks to the surge of transformer-based language models such as bert @xcite @xcite , the evaluation of wsd models has been limited to a set of (mostly semeval-based) standard wsd datasets @xcite . these datasets usually come in one of the two forms: lexical sample, in which a target word is placed in various contexts, triggering different senses, and all-words, in which all the content words in a given text are to be disambiguated. both settings, however, come with a major restriction: word senses in the datasets are linked to external sense inventories such as wordnet @xcite . therefore, existing benchmarks are limited to only those wsd systems in which sense distinctions are defined according to an underlying sense inventory. this not only gives restrictions to the model's flexibility, but also enforces the assumption of the availability of complete data. however, as general sense inventories are complex to maintain they often lag behind in being up-to-date foot_0 , yielding to the absence of novel terms and term usages. furthermore, the coverage of domain-specific terms and named entities in general sense inventories is quite limited, while domain-specific sense inventories are rare and in most cases incomplete. as a motivating example, let us assume technology as the target domain and the collection of information on the current technology landscape as a goal. therefore, the following context needs to be disambiguated in order to evaluate its relevance: from 1970 to 2007, apple's chief executive was former beatles road manager neil aspinall. even when incorporating a general sense inventory (which would include senses for the fruit and the tree) and a technology-specific sense inventory (which would include the sense for apple inc. the technology company), the actual target sense of this context (i.e., apple corps limited, a multimedia corporation founded by the beatles) may still be missing, which makes the annotation of the correct sense impossible. for these reasons, the current wsd task formulation and existing benchmarks are not fully able to evaluate the suitability of disambiguation systems in realistic domain-specific and/or enterprise settings. in this paper, we try to fill this gap by proposing a re-formulation of the existing wsd task as well as a new benchmark for evaluating wsd systems under this paradigm. target sense verification (tsv) formulates the disambiguation of a word as a binary classification task where the equivalence of the intended sense of a word in context and a single given sense is evaluated. for instance, in the example above, the system would need to decide whether the sentence refers to apple inc. the technology company or not, by being provided with a sense indicator for solely apple inc. (e.g., the hypernym technology company or the definition). a system able to efficiently solve the tsv task could be effectively used in the scenario of collecting and tagging large amounts of textual data; e.g., from social media, news agencies, blogs and for downstream tasks such as information retrieval, sentiment analysis or relation extraction. furthermore, such a system could be a good candidate for entity linking (el) as the task statement of tsv resembles the usage of enterprise knowledge graphs @xcite for el: typically, small domain-specific enterprise knowledge graphs only contain entities from the domain of interest, partially or completely missing the general purpose senses of the contained labels. in order to train and evaluate models for tsv we constructed wic-tsv (word in context -target sense verification) a multi-domain dataset and evaluated standard unsupervised and supervised approaches (including language models). while wic-tsv's training and development set consist of general purpose instances, the test set contains domain-specific instances from three different domains. therefore, this dataset aims at evaluating the ability of a model to (1) disambiguate the word in context without an external sense inventory, (2) deal with unseen instances and incomplete data, and (3) transfer the intrinsic knowledge (gained on general domain data) into a specific domain. this facilitates the development of systems for specific domains or settings, as no general-domain knowledge resource is required to perform this task. for instance, an indonesian company may want to retrieve all sentences referring to the java island and not other unrelated senses. this framing of the task is frequent in business and data mining settings where domain-specific knowledge resources or inventories may be available, without the need for modeling instances from other domains. wic. the task closest to the proposed wic-tsv is probably word-in-context (pilehvar and camacho-collados, 2019, wic), which our dataset is based on. wic is a binary classification dataset where a target word is presented within two different contexts. the task consists of deciding whether the word is associated with the same sense in the two contexts or not. wic is also one of the tasks included in the general language understanding framework superglue @xcite . wic-tsv inherits some of the desirable properties of wic, such as independence from external sense inventories and the binary classification nature of the task. however, though our benchmark draws ideas from the word-in-context benchmark, it provides a different evaluation setting with additional flavors. the main difference with respect to our dataset lies in the presence of relevant information such as hypernyms and definitions, which makes our dataset more realistic and a direct proxy for downstream evaluation: in wic-tsv the ambiguous target word in a single context is compared against a specific target sense (indicated by provided hypernyms and definitions), in contrast to the comparison of the intended senses of the target word in two different contexts. also, the task is more targeted at word-level representation, as in one of the tasks (i.e. hypernymy task) the model is not provided with any contextual information and, therefore, needs to have a clear understanding of the word to be able to make correct judgements. moreover, wic-tsv includes instances from three domains (cocktails, medicine, and computer science) in its test set, which makes the benchmark more challenging and comparable to a real setting. to this end, we constructed a benchmark satisfying following requirements: formally, each instance in the dataset consists of a target word w, a context c containing the target word w, and its corresponding target sense s represented by either its definition (task 1), its hypernym/s (task 2), or both definition and hypenyms (task 3). the task aims to determine whether the intended sense of the word w used in the context c matches the target sense s. ## experimental results in this section we evaluate the performance of different baseline models on our wic-tsv benchmark. for our experiments we considered two main systems, namely bert @xcite and fasttext @xcite , as well as unsupervised baselines adapted to the corresponding tasks in wic-tsv. ## conclusions and future work in this paper we have introduced the target sense verification task, a re-formulation of wsd where the equivalence of the intended sense of a word in context and a single given sense is evaluated. furthermore, we presented wic-tsv, a multidomain benchmark which differs from existing wsd datasets in three main ways: (1) it is based on tsv and therefore framed as a binary classification task where only one target sense needs train ctl msh cps all see uns all see uns all see uns wnt 76.5 74.7 75.5 77.4 82.9 76.0 74.9 77.8 74.8 wnt+d 80.2 89.5 75.5 78.5 86.2 76.6 73.6 88.9 72.9 dom 84.2 88.8 82.6 78.7 88.6 75.5 70.7 93.3 69.1 base true 58.5 54.8 59.1 68.2 70.9 67.5 62.6 58.3 62. 8 to be verified, (2) it is independent from external sense inventories, and (3) its test set contains instances from three specific and heterogeneous domains are included: cocktails, medical subjects and computer science. our benchmark therefore opens the floor for different disambiguation algorithms that do not require modeling the entirety of a sense inventory. this characteristic also provides a crucial advantage in enterprise and domain-specific settings as it facilitates the development of systems which are only aimed at modelling the domain at hand. moreover, having these out-of-domain test instances makes our benchmark more robust and generalisable, preventing (or making it harder) for statistical models to learn spurious correlations from the training set, which has been proven to be an issue in standard nlp tasks @xcite @xcite . in our initial experiments we found that current state-of-the-art disambiguation techniques based on pre-trained language models such as bert are very accurate at handling ambiguity, even in specialised domains. however, there is still room for improvement as highlighted by the gap with the human performance. this benchmark therefore opens up avenues for future research on domain-transfer and on developing general-purpose solutions which can perform well on a variety of domains without the need for large amounts of training data. as future work, we are planning to further investigate and analyse the robustness of pre-trained models with respect to domain changes. also, it would be interesting to develop hybrid models which take both definition and hypernymy information into account -in this paper we combined both sources in bert in a simple manner, but more complex models should lead to further improvements.