P oli T o at S em E val-2023 Task 1: CLIP -based Visual-Word Sense Disambiguation Based on Back-Translation (2023)

Abstract: Visual-Word Sense Disambiguation (V-WSD) entails resolving the linguistic ambiguity in a text by selecting a clarifying image from a set of (potentially misleading) candidates. In this paper, we address V-WSD using a state-of-the-art Image-Text Retrieval system, namely CLIP. We propose to alleviate the linguistic ambiguity across multiple domains and languages via text and image augmentation. To augment the textual content we rely on back-translation with the aid of a variety of auxiliary languages. The approach based on finetuning CLIP on the full phrases is effective in accurately disambiguating words and incorporating back-translation enhance the system’s robustness and performance on the test samples written in Indo-European languages.

## introduction visual-word sense disambiguation (v-wsd) is a seminal task in the field of multimodal learning involving both natural language processing and computer vision @xcite . it entails the identification of the correct sense of a given text containing ambiguous words by selecting a clarifying image among a set of candidates, possibly including misleading options. tackling the aforesaid task is relevant to address well-known real-world challenges such as multimodal machine translation @xcite , image caption generation @xcite , visual search and object recognition @xcite . the key issues in v-wsd are the linguistic ambiguity of the expressions occurring in the text, which may subtend multiple senses, and the high variability of the candidate images across different domains, which makes the problem of image retrieval non-trivial. to address the above issues, we propose a novel approach to v-wsd based on clip @xcite , i.e., a state-of-theart vision-language model trained on massive data consisting of image-caption pairs. leveraging the outstanding capability of clip to capture associations between visual and textual content, we first fine-tune the pretrained model on the raw v-wsd data to assess model robustness across multiple domains and languages. next, we explore the use of data augmentation on both text and images, where text is augmented via back-translation. considering text translations in multiple languages has the twofold aim of enriching the model with new text variants generated via machine translation and making the clip model suitable for multilingual scenarios. we also empirically analyze the effect of considering different snippets of the input text, i.e., the ambiguous words only, the main topic description, or the full phrase. the results confirm the suitability of the finetuned clip version to effectively tackle the v-wsd by extracting the key information from the full phrases. specifically, they show a performance boost on the family of indo-european languages that are predominant in the training data whereas confirming the expected issues with underrepresented indo-iranian languages. our results achieve the 40 th rank out of 91 submissions for the english test split, the 28 th rank out of 58 submissions for the farsi test split, and the 29 th rank out of 64 submissions for the italian test split. the rest of the paper is organized as follows. section 2 formalizes the addressed task. section 3 reviews the related literature. section 4 presents the proposed methodology. section 5 describes the preliminary results, whereas section 6 draws the conclusions of the work. ## task overview the v-wsd task frames an image-text retrieval problem, where the goal is to retrieve a target image given a textual query. the challenge is based on a bespoke dataset that comprises 12869 instances, the v-wsd task also takes language diversity into consideration. specifically, although the training corpus is exclusively written in the english language, the test set includes text snippets written in english, farsi, and italian. hence, effectively handling multilingual textual data is particularly relevant. ## related works the v-wsd task concerns the polysemy between textual and visual information thus requiring a joint effort of computer vision and natural language processing experts. a word can be deemed as ambiguous if expresses multiple concepts and thus can be associated with different images. the ultimate goal of the v-wsd task is to provide a comprehensive and coherent understanding of multimedia data, reconciling the ambiguity between textual and visual content. we can disentangle the v-wsd task into two well-known sub-problems, which need to be jointly addressed. ## methodology in this section, we describe the core elements of the methodology proposed to tackle the v-wsd task. hereafter, we will separately discuss the definition of the text snippet to provide as an input to the clip model (see the definition in section 4.1 and the related experiments in section 5.1) and the data augmentation procedure (see the details in section 4.2 and the related experiments in section 5.2). ## experimental results we empirically analyze the impact of each core element of the proposed methodology on the v-wsd dataset. to this end, we use the official pretrained large version of the clip model foot_2 throughout the experimental session. to ensure the robustness and generalizability of the model, we randomly select a validation set comprising 20% of the labeled training data to monitor the model's performance during training. we subsequently evaluate the test set by selecting the best performing model according to the results obtained on the validation set. ## conclusions and discussion the proposed visual-word sense disambiguation (v-wsd) system has demonstrated a remarkable ability to accurately disambiguate lexical items and select the corresponding image. our empirical results have highlighted the robustness of the system when pieces of text written in indo-european languages are translated to english and given as input. the preliminary results underscore the potential of leveraging the clip model for natural language processing tasks that demand multimodal understanding. future research endeavors will entail exploring the use of a multilingual clip model to obviate the translation process during inference and broaden the system's scope to accommodate additional language families, including more indo-iranian and other underrepresented languages. we also plan to observe the effect of back-translation individually for each intermediate language, also varying its aggressiveness. ultimately, our proposed system has far-reaching applications, including multimodal machine translation, image caption generation, and visual question answering.