UA lberta at S em E val-2023 Task 1: Context Augmentation and Translation for Multilingual Visual Word Sense Disambiguation (2023)

Abstract: We describe the systems of the University of Alberta team for the SemEval-2023 Visual Word Sense Disambiguation (V-WSD) Task. We present a novel algorithm that leverages glosses retrieved from BabelNet, in combination with text and image encoders. Furthermore, we compare language-specific encoders against the application of English encoders to translated texts. As the contexts given in the task datasets are extremely short, we also experiment with augmenting these contexts with descriptions generated by a language model. This yields substantial improvements in accuracy. We describe and evaluate additional V-WSD methods which use image generation and text-conditioned image segmentation. Some of our experimental results exceed those of our official submissions on the test set. Our code is publicly available at https://github.com/UAlberta-NLP/v-wsd .

## introduction this paper addresses our work on semeval-2023 task 1: visual word sense disambiguation foot_0 @xcite . the v-wsd task is closely related to wsd, and similarly involves understanding and classifying the meaning of a polysemous word in context. the distinction is in how classes are defined: in wsd, a system has access to a sense inventory that enumerates the possible senses of each word, and the task is to classify the focus word according to the sense that best corresponds to its intended meaning. in v-wsd, a system is given a set of candidate images, and the task is to select the image which depicts the intended meaning of the focus word. the multi-modal nature of v-wsd introduces challenges not encountered in wsd. first, image figure 1 : the task is to select the image that best represents the meaning of the focus word (e.g., bat) in the context (e.g., "baseball bat.") processing is generally more computationally intensive than text processing. second, a v-wsd system must represent the meanings of both images and text, and must have mechanisms to compare these multi-modal semantic representations. last, since the candidate images in v-wsd are not restricted to a sense inventory, they may exhibit highly variable levels of sense granularity. the v-wsd task is motivated by cases where textual context alone is insufficient to disambiguate a word. in such cases, visual context may be available to facilitate disambiguation. for example, the word play is ambiguous in the context "that was a good play," as it may refer to a theatrical performance or an action in a sport. however, an associated image of a stage or a sports field would enable a v-wsd system to disambiguate play. we propose a novel v-wsd algorithm that ranks candidate images by embedding images and wordsin-context in a shared semantic space, while also taking advantage of lexical knowledge bases commonly used in wsd. in particular, our method uses sense glosses of the focus word to create representations of the possible meanings that word may have. our algorithm is flexible, it includes several optional modules, as well as hyper-parameters that facilitate customization, optimization, and detailed analysis. we test various configurations of our method and analyze their performance. our three principal conclusions are as follows: first and foremost, the augmention of the original textual context plays a crucial role in improving performance. second, there is a considerable gap between english and non-english performance, indicating that bias towards english models extends to the multi-modal setting. third, we observe a major distribution shift between the train and test sets, which is confirmed by our ablation study. ## related work recent work on wsd can be divided into supervised and knowledge-based systems. supervised wsd methods depend on large training corpora in which some or all of the content words have been tagged with their correct senses @xcite . knowledgebased methods depend on other sources of linguistic knowledge @xcite . in general, knowledge-based methods are outperformed by contemporary supervised methods @xcite . today, state-of-the-art wsd systems approach accuracy limits imposed by inter-annotator agreement @xcite . research on the incorporation of visual information for wsd is relatively sparse. @xcite propose a statistical model that associates image regions and words to predict word senses. @xcite apply spectral clustering to group similar images corresponding to the same senses. @xcite employ an unsupervised approach to assign senses to images using surrounding texts and dictionary definitions, and then train a visual svm classifier to disambiguate unseen images. @xcite introduce the task of visual verb sense disambiguation, in which one image is selected based on a given context. @xcite propose a graph-based semisupervised transductive learning method for visual verb sense disambiguation. multi-modal foundation models @xcite such as clip @xcite and align @xcite can represent both text and images in a shared embedding space. recent work foot_1 @xcite improves the text encoder by using a pretrained text-only encoder such as bert @xcite . dataset: the training data provided for this shared task consists of a silver dataset with 12,869 v-wsd instances. each sample is a 4-tuple ⟨f, c, i, i * ∈ i⟩ where |i| = 10. the contexts are generally very short, often just a single word in addition to the focus word. we randomly select 10% of the training data for use as a development set. the test dataset consists of 968 instances, of which 463 are english, 200 are farsi, and 305 are italian. we observe that many of the incorrect candidate images in the training data have nothing to do with any sense of the focus word. however, in the test data, we observe that this is less often the case, making the test set considerably more difficult. evaluation metrics: the primary metric is the hit rate, which is equivalent to top-1 accuracy, or simply accuracy. this is the proportion of instances for which the system selects the correct image. we also compute the mean reciprocal rank @xcite which represents how highly v-wsd systems rank the ground-truth image, on average. language % example english 82 waxflower wildflower latin 15 shorea genus german 2 truppenübungsplatz workplace french 1 brumaire month language distribution: we observed some instances where the context contained non-english words. to estimate the prevalence of this phenomenon, we randomly selected 100 instances from the training set and manually identified the language of each. for example, the focus word shorea in "shorea genus" comes from new latin, and refers to a genus of mainly rainforest trees. ## method in this section, we describe the key components of our systems, including an algorithm that combines text and image similarity measures. ## systems in this section, we describe our systems for the v-wsd task, our official system submissions are based on our primary systems: tr and langspec. we also describe two alternative systems, which do not use algorithm 1. both perform worse than the primary systems, but their results are nevertheless valuable for the purpose of analysis. we also present a supplementary method, which can be optionally used in combination with our other systems. non-english instances are translated using deepl foot_2 for italian and chatgpt foot_3 for farsi. ## experiments in this section, we present, discuss, and analyze our results. ## conclusion in this paper, we outlined our work on the recently proposed task of v-wsd. we found that many ideas from traditional wsd are difficult to adapt to v-wsd, and, moreover, wsd systems are generally not useful for v-wsd. we were particularly surprised to find that, unlike in wsd, glosses appear to be unhelpful for v-wsd. contrariwise, our innovation of augmenting the context did yield substantial gains in accuracy. further research will be needed to establish the connection between v-wsd and the broader field of lexical semantics. we speculate that developing systems for joint wsd and v-wsd may yield improvements in one or both tasks. our work here serves as a proof-of-concept establishing the utility of language models and lexico-semantic resources in the developing task of v-wsd.