SUT at S em E val-2023 Task 1: Prompt Generation for Visual Word Sense Disambiguation (2023)

Abstract: Visual Word Sense Disambiguation (V-WSD) identifies the correct visual sense of a multi-sense word in a specific context. This can be challenging as images may need to provide additional context and words may have multiple senses. A proper V-WSD system can benefit applications like image retrieval and captioning. This paper proposes a Prompt Generation approach to solve this challenge. This approach improves the robustness of language-image models like CLIP to contextual ambiguities and helps them better correlate between textual and visual contexts of different senses of words.

## introduction there are different senses associated with polysemous english words @xcite . the task of determining a specific sense of a word has been one of the most significant challenges in natural language processing. several studies have been conducted on determining a word's sense based on its sentence. with the emergence of multimodal works, visual and acoustic information can be used to solve a word's ambiguity. these models can be used to expand capability of models in limited textual context. using a proper visual word sense disambiguation model can contribute to the advancement of different areas, such as image captioning, image retrieval, and other multimodal problem settings. our main approach is to tackle the problem of v-wsd @xcite using prompt engineering. we used chatgpt (openai, 2022) to generate a desired prompt for a given sense. using clip model @xcite we extract text and image embeddings for the given input and compare their cosine distance. our model works for * the first two authors contributed equally and listed randomly. english, farsi and italian inputs with average mrr of 71.13% ranking 9th in semeval final results. using chatgpt to expand contextual information increased model's accuracy. however, since chatgpt usage has limitation on number of requests, the model input is limited. our code is publicly available on github foot_0 . ## experimental setup and evaluation for our experiments, each dataset instance has three main parts: the ambiguous word, limited textual context, and ten image candidates, one of which corresponds to the desired visual context, and there were three languages in the test dataset: english, italian, and farsi. we first batched the dataset words and their textual contexts and got their corresponding prompts. we then used different backbones of the clip model as the multimodal module of our system. additionally, we took a mean pooling of all of these backbones to predict the correct image. for the farsi and italian words, we translated them into english and then used the translated text as the input for our multimodal module. we used resnet-50 and resnet-101 @xcite and vit @xcite backbones for the clip models. we also used fine-tuned models on the muwosed dataset @xcite , a multimodal word sense disambiguation dataset. we fine-tuned models in 5 epochs on this dataset. we test various models using the same two ranking metrics specified in the task. the first metric is hits@1. this metric measures the frequency with which the ground truth appears in the top position. mean reciprocal rank (mrr) is our second metric, calculated using the following formula: ## results for the farsi language, our model achieved hits@1 of 0.50 and mrr of 0.64, suggesting that this language is more challenging for the model. moreover, our model produced hits@1 of 0.53 and mrr of 0.68 for the italian language. overall, our model achieved hits@1 of 0.57 and mrr of 0.71. this suggests that our zero-shot approach performs fairly well, given the fact that it is not fine-tuned on a specific dataset and can perform fairly well across different domains. in addition, it consumes less energy since it does not need any data for training. we also evaluated the pre-trained muwosed on this task dataset and observed that the pre-trained model on the same task but with different data increased the accuracy in the zero-shot setting. ## discussion and conclusions in this paper, we proposed a model to solve the visual word sense disambiguation challenge using large language models to generate prompts for different textual contexts of words to add extra information to them to make multimodal models select the corresponding image of that context better. for the english language, we used chatgpt to generate those prompts and then gave those prompts to different backbones of clip models to detect the related image of that sense both in zeroshot and fine-tune settings. it shows that fine-tuning on the training dataset improves the model's performance. moreover, using a pre-trained model on the muwosed dataset in zero-shot settings leads to better results than the original clip model in the zero-shot setting. we first translated textual contexts for the farsi and italian languages and then used the result as the text input of our multimodal models in zero-shot settings. using a zero-shot setting significantly reduces the consumed energy while maintaining its performance at a fairly good level in different domains. future works can include prompt generation in multilingual settings to add extra information to the textual input of the multimodal models.