PMC oders at S em E val-2023 Task 1: RA lt CLIP : Use Relative A lt CLIP Features to Rank (2023)

Abstract: Visual Word Sense Disambiguation (VWSD) task aims to find the most related image among 10 images to an ambiguous word in some limited textual context. In this work, we use AltCLIP features and a 3-layer standard transformer encoder to compare the cosine similarity between the given phrase and different images. Also, we improve our model’s generalization by using a subset of LAION-5B. The best official baseline achieves 37.20% and 54.39% macro-averaged hit rate and MRR (Mean Reciprocal Rank) respectively. Our best configuration reaches 39.61% and 56.78% macro-averaged hit rate and MRR respectively. The code will be made publicly available on GitHub.

## introduction wsd (word sense disambiguation) is the task of identifying which sense of a word is meant in a sentence or other segment of text. in the vwsd (visual-wsd) task @xcite , given a potentially ambiguous word and some limited textual context, the image corresponding to the intended meaning should be selected from a set of candidate images. such a model can operate better as a text-to-image retrieval system. clip @xcite uses a contrastive loss function to map text and image features into a common space by minimizing the distance between the related caption and image and maximizing the distance between negative pairs. altclip @xcite alters the language encoder in clip and uses xl-r lm (language model) to extend its language capabilities. we use altclip as the base model as the dataset includes italian and farsi languages in addition to english. the official dataset consists of 12869, 16, 463, 305, and 200 samples as training, trial, english test, italian test, and farsi test data respectively. each sample has a word (potentially ambiguous), a phrase including the word, and 10 images. to improve model generalization, we also use a subset of the laion-5b @xcite ## system description figure 1 shows an overview of the proposed model and the following subsections discuss it in more detail. ## results to investigate the effect of the modifications, we examine the effect of using each one. you can see the results in ## future work there are some modifications that can be made to improve the overall accuracy of the proposed system, here are some of them: ## conclusion in this work, we proposed a system to relatively compare the similarity between a phrase and 10 images. also, we used an extra feature vector to disambiguate the ambiguous word, further more, we used a subset of laion-5b to pretrain the proposed model to improve the generalization. the proposed system beat the official baseline by 2.41% and 2.39% in macro-averaged hit rate and mrr respectively. the results show that using a massive amount of data for pretraining, the proposed model can be used to improve text-to-image retrieval systems by re-ranking the most related images.