SLT at S em E val-2023 Task 1: Enhancing Visual Word Sense Disambiguation through Image Text Retrieval using BLIP (2023)

Abstract: Based on recent progress in image-text retrieval techniques, this paper presents a fine-tuned model for the Visual Word Sense Disambiguation (VWSD) task. The proposed system fine-tunes a pre-trained model using ITC and ITM losses and employs a candidate selection approach for faster inference. The system was trained on the VWSD task dataset and evaluated on a separate test set using Mean Reciprocal Rank (MRR) metric. Additionally, the system was tested on the provided test set which contained Persian and Italian languages, and the results were evaluated on each language separately. Our proposed system demonstrates the potential of fine-tuning pre-trained models for complex language tasks and provides insights for further research in the field of image text retrieval.

## introduction visual word sense disambiguation (vwsd) organized by @xcite 1 is the task of selecting the correct image from a set of candidate images that corresponds to the intended meaning of a target word, given a limited textual context. semeval 2023 task 1 focuses on vwsd in the context of images and textual descriptions. vwsd helps to improve the performance of many natural language processing (nlp) applications that involve both textual and visual information. for example, in image captioning @xcite , the system needs to identify the correct sense of a word in a given context to generate an accurate and meaningful caption. similarly, in visual question answering (vqa) @xcite , the system needs to understand the meaning of the question and the visual context to provide a correct answer. image retrieval aims to retrieve images that are relevant to a given textual query, such as a caption 1 https://raganato.github.io/vwsd/ or a keyword. this involves learning a representation of both images and text in a shared embedding space, where the similarity between images and text is determined by their proximity in the embedding space. the objective is to locate the most relevant images based on the textual query and present them in a ranked order. @xcite vision-language pre-training has emerged as a powerful approach for multimodal understanding in recent years (e.g., clip @xcite , albef @xcite , simvlm @xcite ). this approach involves jointly training models on large-scale image and text datasets to learn a shared representation that captures the semantic relationship between visual and textual inputs. task organizers selected clip as a baseline approach. clip has emerged as a powerful approach for multimodal understanding and leverages a vast amount of supervision by pre-training on a dataset of 400 million (image, text) pairs collected from the internet. using clip as a baseline, we aim to demonstrate the effectiveness of our proposed approach for the vwsd task. vision-language pretraining has achieved significant success in various downstream tasks, such as image captioning, visual question answering @xcite , and image-text retrieval @xcite . by leveraging the joint representation learned during pretraining, these models can effectively reason about the relationships between visual and textual information @xcite , leading to improved performance on these tasks. our approach to vwsd in this task is based on vision-language pre-training, specifically bootstrapping language-image pre-training (blip) @xcite . blip trains a model to predict the relationship between an image and its corresponding textual description, and has been shown to be effective for a range of vision-language tasks, including vwsd. in this paper, we describe our system for se-meval 2023 task 1, which uses blip pre-training to generate image embeddings that capture the semantic information of the images. our system then utilizes these embeddings to perform vwsd by computing the similarity between the target word's embedding and the embeddings of the candidate images. that also incorporates an attention mechanism to further refine the similarity scores. we implemented the bootstrapping language-image pre-training (blip) method by adapting an open-source implementation available on github foot_0 . we made some modifications to the code to suit our specific requirements, like finetuning and evaluating the method on the vwsd task. ## system overview blip is a vision-language pre-training (vlp) framework that transfers flexibly to both visionlanguage understanding and generation tasks. it is designed to effectively utilize noisy web data by bootstrapping the captions. the framework consists of two stages: bootstrapping and refinement. during the bootstrapping stage, synthetic captions are generated for noisy image-text pairs from the web. the captioner network is trained to generate captions for a given image, and a filter network is used to remove noisy captions. this process is shown in figure 1 . this results in a dataset of highquality image-text pairs for further pre-training. in the refinement stage, the pre-trained model is fine-tuned on downstream tasks such as image captioning, visual question answering, and image text retrieval. this stage involves training the model on a specific task dataset to fine-tune the pre-trained model parameters to improve task performance. ## conclusion based on the experimental results, we can conclude that our system, which fine-tunes blip's image text retrieval model on the vwsd task dataset using the itc and itm losses, performs well in estimating similarity between image and text. the candidate selection approach employed in our system also speeds up the inference process, allowing for more efficient processing of image-text pairs. overall, the proposed system shows promising results in vwsd and has the potential to be applied in various domains such as image retrieval, recommendation systems, and virtual assistants.