OPI at S em E val-2023 Task 1: Image-Text Embeddings and Multimodal Information Retrieval for Visual Word Sense Disambiguation (2023)

Abstract: The goal of visual word sense disambiguation is to find the image that best matches the provided description of the word’s meaning. It is a challenging problem, requiring approaches that combine language and image understanding. In this paper, we present our submission to SemEval 2023 visual word sense disambiguation shared task. The proposed system integrates multimodal embeddings, learning to rank methods, and knowledge-based approaches. We build a classifier based on the CLIP model, whose results are enriched with additional information retrieved from Wikipedia and lexical databases. Our solution was ranked third in the multilingual task and won in the Persian track, one of the three language subtasks.

## introduction visual word sense disambiguation (vwsd) is a task in the field of multimodal natural language processing, in which the goal is to identify the intended meaning of a target word in a given context by selecting the most appropriate image from a set of candidate images. finding images corresponding to the correct meaning of the word might improve the performance of methods combining text and visual information such as image search engines, visual question answering, or image generation models. semeval-2023 workshop hosted a task on visual word sense disambiguation. the task involved selecting the best matching image out of ten candidates given a short textual description. the descriptions usually consisted of two words: the target word and the context word @xcite . for example, the phrase andromeda tree contains the ambiguous target word andromeda and the context word tree, which indicates a specific meaning of the target word. the task organizers provided three datasets, of which the trial and training datasets contained phrases in english, while the test dataset was multilingual and consisted of english, italian, and persian subsets. participants were allowed to submit their solutions for a particular language or for all three languages. the systems were ranked according to the average accuracy score from three language-specific subtasks. in this paper, we describe our system for the vwsd shared task. the backbone of our solution is a classifier using multimodal clip embeddings @xcite , which has been enriched with features extracted from wikipedia and dictionaries. these knowledge sources are used to retrieve textual and image data, providing additional information useful for determining the correct meaning of the target word. our system was ranked third in the multilingual task and took first place in the persian subtask. the source code of our system is publicly available, as well as the fine-tuned models and other resources required to reproduce our experiments. foot_0 ## system description the proposed approach consists of several modules which together constitute a visual word sense disambiguation system. the three core components of our approach are: 1) a classifier based on clip image-text embeddings, 2) a wikipedia retrieval module, 3) a learning to rank (ltr) model whose role is to generate the final ranking of images based on the features provided by the other modules. a high-level overview of the system is shown in figure 1 . ## experiments and results this section contains a discussion of the official results of the visual word sense disambiguation task. we have also included a description of other variants of our system which were not used in the submitted solution. we conducted post-evaluation experiments using the gold labels provided by the organizers to analyze the results obtained by alternative versions of our approach. ## conclusion in this paper, we described our solution for the visual word sense disambiguation shared task. our system was ranked third in the multilingual track and won in the persian track, one of the three language-specific subtasks. in the publication, we demonstrated how to build a system, which incorporates different approaches to the problem: imagetext embeddings, lexical resources, image and text retrieval. we showed that each of these components can improve the performance of the overall solution. we have also pointed out some drawbacks of our approach, which can be a starting point for creating better methods in the future.