GPL at S em E val-2023 Task 1: W ord N et and CLIP to Disambiguate Images (2023)

Abstract: Given a word in context, the task of VisualWord Sense Disambiguation consists of select-ing the correct image among a set of candidates. To select the correct image, we propose a so-lution blending text augmentation and multi-modal models. Text augmentation leverages thefine-grained semantic annotation from Word-Net to get a better representation of the tex-tual component. We then compare this sense-augmented text to the set of image using pre-trained multimodal models CLIP and ViLT. Oursystem has been ranked 16th for the Englishlanguage, achieving 68.5 points for hit rate and79.2 for mean reciprocal rank.

## introduction many and very common words in language are polysemous, i.e., they have more than one meanings associated with the lexical object @xcite . the task of word sense disambiguation (wsd) consists of identify the correct meaning of a word in a specific context, its sense. for example, the word andromeda has three different meanings according to the cambridge dictionary: the princess from the greek mythology, the large galaxy and a particular plant with flowers. the objective of wsd is to determine which sense is active in a specific context, such as the sentence "we found the andromeda growing on the banks of a lagoon of fresh water". visual word sense disambiguation (vwsd) is the extension of the traditional wsd task that incorporates visual information in addition to textual context @xcite . in semeval-2023 task 1 proposed by @xcite , the objective is to identify the correct image from a pool of candidates that corresponds to the intended meaning of the word in context. for example, the word is andromeda, the context is andromeda tree, and three candidate images are presented in figure 1 . the image located on the left displays a photograph of an andromeda tree, which is the correct choice, whereas the others images show other objects and are therefore incorrect. we tackle this task with an elaborated text augmentation strategy and the power of pre-trained language-vision models. in particular, our text augmentation strategy employs the fine-grained linguistic resource of wordnet @xcite to get better textual representations. multimodal models are then leveraged to select the correct image in the pool of candidates in a zero-shot. our system has been ranked 16 th for the english language (50 submissions, 125 participants), achieving 68.5 points for hit rate and 79.2 for mean reciprocal rank. we also achieved 37.7 for hit rate and 55.7 for mean reciprocal rank on the italian dataset. this report is composed of six sections. section 2 provides an overview of the history of wsd task and vwsd task. section 3 outlines our approach to the vwsd task. section 4 presents the experiment setup. section 5 presents the experiment settings. the results are presented and evaluated in section 5. finally, conclusions are drawn in section 6. ## related work in the traditional word sense disambiguation task @xcite , a keyword, the context in which it is utilized, and a collection of potential senses are supplied. the objective of the task is to determine the accurate sense from the set of senses provided, based on the contextual information. previous studies in the field of wsd mainly leverage knowledge-based methods that relied on manually crafted rules or knowledge resources @xcite . the lesk algorithm @xcite ) was one such approach that select the sense with the highest n-gram overlap with the context of the target word, and has been widely recognized as a robust baseline method for wsd. some knowledgebased methods also use graph-based algorithms for wsd, such as the approach presented in @xcite , which employs random walks over a lexical knowledge base created using word-net. more recent studies have shifted towards using machine learning techniques for wsd. these approaches make use of both annotated and unannotated datasets, with semcor being the most commonly used one. it is a dataset containing over 220,000 words that were manually annotated with senses from wordnet @xcite . one widely used method involves comparing contextual word embeddings with sense embeddings. this method requires either a corpus annotated with word senses (such as semcor) or a word net (e.g., loureiro and jorge 2019; @xcite . visual word sense disambiguation (vwsd) can be considered as a multimodal extension of the conventional word sense disambiguation task. being a relatively new topic, there are not many available studies on this subject. an initial version of the task have been introduced by @xcite , where the image was used as an additional resource to disambiguate the sense of the word. researchers employ statistical models to extract features from the co-occurrence of image regions and nouns in context. most vwsd tasks involves identifying the sense of a word depicted in an image, given the image and the word. recent studies on vwsd use a supervised wsd approach. for instance, @xcite generated two sense representations for each candidate sense, one based on text and the other on image. the researchers then applied an adapted lesk algorithm to measure the similarity between the image and senses in the sense inventory to determine the correct sense of the target word. the semeval 2023 vwsd task is unique in the sense that: given a word in its context and a list of images to choose from, it requires participants to select the correct image. other wsd tasks, instead, typically ask to identify the textual sense of a word depicted in an image. ## system overview the participants of the shared task were provided with two inputs: a text file and a folder that contains all candidate images. at each line, the text file comprises: a) the target word to disambiguate (e.g., andromeda in figure 1 ); b) the target word in context (e.g., andromeda tree in figure 1 ); c) the 10 candidate image references for that word. the linguistic context in this task is typically short, comprising one or two words. moreover, around 18% of instances in the train set involve the following context words: family, genus, species, phylum, class, order. for some of the instances, the challenge is not textual ambiguity but selecting the correct image based on the limited textual information provided. for example, "malaxis genus" refers to a specific genus of orchid with only one sense, making disambiguation unnecessary. for these cases, the challenge lays in selecting the correct image from ten candidate options rather than disambiguation. due to the limitation of the linguistic context for disambiguation, we propose a two-step strategy that strongly rely on text augmentation. firstly (3.1), we augmented the textual context using the external resources of wordnet and wikipedia, leveraging sentence encoders to select the correct sense. secondly (3.2), we make use of languagevision models to identify the correct image based on the extended context. text augmentation has been found to be advantageous for the task. ## experimental setup for text augmentation, our approach combines wordnet and sentence transformers (sbert). to access wordnet, nltk toolkit has been employed. for sbert, we conducted a preliminary research to identify the best pre-trained sentence encoder for our augmentation strategy. for the 16 samples in the trial set, we manually annotated the correct synsets in wordnet, then testing different sentence encoders. the best performing one, all-mpnet-base-v2, achieves an accuracy of 0.625 (10/16 samples). we consider this result satisfying for augmentation, either considering the extremely short context and the large number of synsets for common words (e.g., bank). for the multimodal match, we conducted experiments in zero-shot settings. in the zero-shot setting, we ranked the images using the vanilla clip model based on clip-vit-base-patch32 and the vilt model based on vilt-b32-finetuned-coco, following the approach described in section 3. the maximum text length was set to 40 for vilt and 77 for clip. the image resolution was 384 × 640 with patch projection of 12 × 20 = 240 patches for the vilt model and 1440 × 1810 for the clip model. for each dataset, we use the augmented/unaugmented text with 10 images and ranked them according to their similarity (image, text) pair according to clip or vilt. in detail, we first compute the word embedding of the text and the feature embedding of the 10 images with respect to text. then, we ranked these 10 images based on the similarity scores of these embeddings. ## results two evaluation metrics are used to measure the performance of our model: hit rate at 1 and mean reciprocal rank. the hit rate at 1 evaluates the accuracy of the first ranked image, with a score of 1 if the correct image is in the first position and 0 otherwise. the mean reciprocal rank is the average of the reciprocals of the rank position of the correct image (e.g., if the correct image is ranked second, the reciprocal rank would be 0.5). the baseline model provided by the task organizers is very similar to our system. they used clip to compute the text ("this is" + phrase to disambiguate) and image embeddings, and rank the candidate images based on the cosine similarity between the text and image embeddings. the results shown in wikipedia), with a hit rate of 68.47 and a mean reciprocal rank of 79.17. without text augmentation, the performance decreased to 58.96 for hit rate and 72.73 for mean reciprocal rank. for clip, we further try text augmentation only using wikipedia: the low results (56.81 hit rate, 71.86 mrr) clearly demonstrate the positive effect of using wordnet on the task. while text augmentation produced an improvement in clip's overall performance, it actually worsened the performance of the vilt model. this suggests that the clip is probably better equipped to handle diverse and potentially noisy text inputs while vilt is more sensitive to changes in the input text and not able to handle text variations effectively. also, clip uses large and seperate transformer embeddings for each modality which leads to a remarkable performance in text-to-image retrieval in zero-shot setting. in contrast, the singlestream architecture might be a reason of relatively poor performance in vilt zero-shot setting. our study highlights that text augmentation can benefit image retrieval, particularly for the clip model. in both languages, our system outperform the baseline system. however, it is crucial to evaluate the impact of text augmentation on each model, as it may not always lead to improved performance. we assessed the performance of both models using a trial dataset consisting of 16 instances, and our approach resulted in 10 accurate augmentations. the clip model correctly predicted 10 images, 9 of which were augmented with correct augmentation and 1 with incorrect augmentation. interestingly, sometimes clip can retrieve the correct image using incorrect text augmentation. for example, "breaking wheel" refers to a medieval instrument of torture and execution, the text augmentation obtained from wordnet is "wheel around wheel somebody or something", which is obviously incorrect. however, clip model still selected the correct image. in contrast, the vilt model only made 4 correct predictions, all of which were augmented with correct augmentation. we speculate the different performance of these two models may be due to the semantic encoder used in clip, which could enable the model to better capture the semantic relationship between words and images. ## conclusion overall, this project provides a brief overview of the word sense disambiguation (wsd) task and its visual counterpart (vwsd). it thoroughly discusses the differences and relationships between the two tasks. the vwsd task is formulated into two steps: text augmentation and image retrieval. the first step utilizes external resources such as wordnet. for the second step, the vilt and clip models are experimented with, both achieving good results. we also tried to fine-tune the clip model, however fine-tuning did not help with model's performance.