S em E val-2023 Task 1: Visual Word Sense Disambiguation (2023)

Abstract: This paper presents the Visual Word Sense Disambiguation (Visual-WSD) task. The objective of Visual-WSD is to identify among a set of ten images the one that corresponds to the intended meaning of a given ambiguous word which is accompanied with minimal context. The task provides datasets for three different languages: English, Italian, and Farsi.We received a total of 96 different submissions. Out of these, 40 systems outperformed a strong zero-shot CLIP-based baseline. Participating systems proposed different zero- and few-shot approaches, often involving generative models and data augmentation. More information can be found on the task’s website: \url{https://raganato.github.io/vwsd/}.

## introduction word sense disambiguation (wsd) is the task of associating a word in context with its intended sense, generally from a pre-defined sense inventory. while there has been significant progress in the last few years @xcite , mainly powered by progress in language models, wsd has been mainly limited to settings with textual content only. in most settings, wsd uses sense inventories obtained from lexical resources, such as wordnet @xcite . however, in real-world scenarios, wsd is often associated with other modalities, such as images. the visual word sense disambiguation (visual-wsd) task aims at filling this gap: given a word and some limited textual context (often a single word), the task is to select among a set of candidate images the one which corresponds to the intended meaning of the target word. figure 1 provides a simplified overview of the task. to make the task more challenging, the dataset is constructed with the following objectives in mind: the target (ambiguous) word (in bold) is provided with minimal context (one or two words). the task is to associate the intended sense with the relevant image (underlined). note: for the task, nine negative images were provided, but for the sake of simplicity, we only show three negative images. the visual-wsd task can be viewed from two different perspectives. the task is a fine-grained image classification task (in a constrained setting) where the model needs to have knowledge about the different meanings of an ambiguous word. alternatively, visual-wsd is a novel disambiguation task where the sense distinctions are given in the visual modality rather than the conventional textual definitions obtained from lexical sense inventories @xcite . task formulation. given a target ambiguous word (e.g., coach) and a limited context (e.g. passenger) along with ten images, the visual-wsd task consists of identifying the most appropriate image for the intended meaning of the ambiguous word. figure 1 shows a simplified summary of the task with three examples. applications. multimodality is highly intertwined with different aspects of our day-to-day lives. in fact, retrieving and understanding text with textual data is relevant for a myriad of tasks, from object detection to image retrieval @xcite @xcite . our task enables the in-depth investigation of one aspect often neglected, which is the inherent ambiguity in language @xcite . multilinguality. in addition to english, we provide labeled data (context words for given ambiguous targets) for italian and farsi. this data serves to analyse the performance of models in languages other than english, and in practical cross-lingual settings where initial training data might not be available for a given language. ## data in this section we detail the data collection and annotation process carried out for the training and testing data. we then provide some statistics on the datasets. ## evaluation we first introduce the metrics that are used to evaluate the participating systems (section 3.1). then, we provide a brief description of the baseline system (section 3.2) as well as the participting systems (section 3.3), and present the overall results of the task (section 3.4). ## conclusions in this paper, we presented the first task on visual word sense disambiguation. instead of the usual single-modality tasks of textual word sense disambiguation and image recognition, our proposal merges these two paradigms into a single unified task. in particular, wsd is reframed by replacing the usual sense inventory based on lexical resources by a dynamic inventory based on images. the task received 96 submissions. the evaluation showed promising results with 40 submissions outperforming a very competitive image recognition baseline based on a zero-shot clip model for different languages. nonetheless, given the recency and challenging nature of the task, there is clear room for improvement for future work.