Rahul Patil at S em E val-2023 Task 1: V - WSD : Visual Word Sense Disambiguation (2023)

Abstract: Semeval 2023 task 1: VWSD, In this paper, we propose an ensemble of two Neural network systems that ranks 10 images given a word and limited textual context. We have used openAI Clip based models for the English language and multilingual text-to-text translation models for Farsi-to-English and Italian-to-English. Additionally, we propose a system that learns from multilingual bert-base embeddings for text and resnet101 embeddings for the image. Considering all the three languages into account this system has achieved the fourth rank.

## introduction visual word sense disambiguation (visual-wsd) @xcite is an important task for understanding and measuring machine's ability to find correct image from a set of candidate images given the word and the limited text context. word sense disambiguation (wsd) has long been formulated and studied as an important problem in natural language processing since 1940s. visual word sense disambiguation (visual-wsd) is a combination of natural language processing and computer vision that involves identifying the correct visual image of an ambiguous word using limited context. for example, the word "bank" can refer to a financial institution or the side of a river. in the context of limited text data, such as "bank erosion", visual-wsd aims to identify the image displaying the correct meaning of the word in its context. visual-wsd is an important task in natural language processing and computer vision, as it helps to improve the accuracy of systems that use text and visual information together, such as image captioning, video analysis and visual question answering. successful visual-wsd can help improve the accuracy of natural language processing applications that rely on visual information. we used an ensemble of two different approaches 1) fine-tune clip @xcite base architecture 2) joint embedding. the first approach involves finetuning clip based model with additional data and the second approach involves mapping image and text-based features into a common vector space, which allows direct comparison of the two features. this enables the model to learn the relationship between visual and text features and use this knowledge to determine the correct sense of a word. the proposed system has achieved the 4 th overall rank in this competition with the hit rate of 69.805 and mean reciprocal rank score of 78.230. the system has achieved the second rank in english language with the hit rate of 83.153 and mrr rate of 88.802, first rank in italian with the hit rate of 84.262 and mrr rate of 89.048, lastly achieved 16th rank in farsi language with the hit rate of 42.000 and mrr rate of 56.841. the system lacked understanding of farsi language and struggle with farsi to english conversion. ## background three different types of datasets were released in this competition. trial and train dataset contained words, limited context, 10 candidate images and correctly labeled gold data. test dataset consists of word, limited context and 10 candidate images. in the figure1: bank is an ambiguous word. it can have multiple meanings like river bank, erosion of bank, money bank, piggy bank, turning of airplane. in this figure, context is mentioned as an erosion bank hence the gold label. the train and the trial data consists of english language only, whereas test data includes english, italian and farsi languages. additionally we utilized the ms-coco @xcite dataset in conjunction with the wikipedia image text (wit) dataset @xcite , a vast multimodal and multilingual dataset sourced from wikipedia. as mentioned in in the clip architecture (figure 2 ), we take an image-text caption as input. during prediction, we provide a collection of candidate images (i) and a set of text captions (t). model generates similarity score with i*t dimensional matrix. we used this output for identifying the best matching image for a given text caption. in our case we had 12,885 labeled image-text pairs. clip architecture consists of 3 main components: ## results while fine-tuning the clip model, we placed greater emphasis on loss values rather than hit rate or mrr. additionally, we followed the approach outlined in the openai paper and overfitted our model, resulting in enhanced mrr and hit scores, despite observing an increase in validation loss. unlike a fine-tuned clip model, we opted to stop training our joint embedding 1 vs 10 model once the validation loss ceased to improve. this approach helped us prevent overfitting and ensured that our model remained robust. furthermore, we evaluated best-performing model using multiple metrics including mrr, hit rate and validation loss. ## conclusion our proposed solution for task semeval-2023 task 1: v-wsd is able bridge the gap between computer vision and natural language processing for english and italian language. as mentioned in