S t FX NLP at S em E val-2023 Task 1: Multimodal Encoding-based Methods for Visual Word Sense Disambiguation (2023)

Abstract: SemEval-2023’s Task 1, Visual Word Sense Disambiguation, a task about text semantics and visual semantics, selecting an image from a list of candidates, that best exhibits a given target word in a small context. We tried several methods, including the image captioning method and CLIP methods, and submitted our predictions in the competition for this task. This paper describes the methods we used and their performance and provides an analysis and discussion of the performance.

## introduction semeval-2023's task 1: visual word sense disambiguation (v-wsd) involves selecting an image from a list of candidates, that best exhibits a given target word in a small context. in this task @xcite , each sample will contain one target word, a limited context, and ten candidate images. the ten candidate images contain one gold image -the image that best matches the sense of the target word in its context. in addition to this, the ten candidate images also contain images related to other senses (i.e., the meaning of the word) of the target word as well as images not related to the target word. the limited context contains two or three words (including the target word), and the vast majority of these contexts are strongly related to a sense of the target word. figure 1 shows one example in the dataset. in this example, the target word is andromeda and the context is andromeda tree. the first image is the gold image (the tree with white flowers). the second image (whale) and the third image (snake) are some of the candidate images not related to the target word. the fourth image (stars) is the candidate image related to another sense of the word andromeda. the dataset for this task contains a trial set, a training set, and a testing set. the context and target words in the samples of the trial set and the training set are in english. the testing set contains samples in english, italian, and farsi. . this task involves both natural language processing and computer vision. this is because we need to acquire semantic features of both text and images. for this task, we chose to map text and images to the same vector space and then compare them using a similarity metric. to perform this feature mapping, we require the use of models that have been trained on other datasets. we approached this task with several pre-trained visual language models. this includes the image caption model @xcite and contrastive language-image pre-training (clip) @xcite models. ## related work word sense disambiguation (wsd), a task in natural language processing, is about identifying the sense of a target word in the context. the most commonly used sense inventory for word senses in wsd is wordnet @xcite . wordnet is a large lexical database of english senses, which organises word senses in the form of synsets(sets of synonyms). for each word sense(synset), wordnet provides a text definition to describe it. in the wsd task, we can represent the usage of a word through embeddings, which are word embeddings, the most commonly used representations of words. word embeddings represent words as dense vectors(low dimensional vectors). there are some commonly used word embeddings: word2vec @xcite , glove @xcite , elmo (matthew et al., 1802), bert @xcite . researchers such as @xcite have shown that simply using word embeddings can achieve good performance on wsd tasks. similar to word embedding, sentence embeddings can represent sentences in a low-dimension vector. the sentence embedding technique used in this study is sentence bert (reimers and gurevych, 2019), also known as the sentence transformer. it is trained on multiple corpora, such as reddit comments @xcite and s2orc @xcite , and used to generate sentence embeddings for sentence similarity. in recent years, progress has been made in the development of multimodal models, which take multiple types of data(like images and text) as learning subjects. multimodal models have been applied to image captioning tasks such as @xcite and @xcite . a state-of-the-art taskagnostic and modality-agnostic framework, called ofa @xcite has outstanding results on a range of vision-language tasks. microsoft coco captions @xcite is one of the most commonly used datasets for image captioning tasks. it contains 330,000 images with roughly 1.5 million captions, making it the largest image caption dataset available, with captions generated by human annotation. ofa framework has achieved great performance on microsoft coco captions. in 2021, openai released a pre-trained neural network model for matching images and text, named clip (contrastive language-image pre-training) @xcite . the model is trained on over 400 million image text pairs collected by openai on the internet to encode images and text. it then trains with the aim of improv-ing the similarity between the encodings, resulting in a multimodal pre-trained model that generates vision-language embeddings. ## methods in this section, we will introduce the methods that we applied to the v-wsd task. a common method used in comparing the similarity of vector spaces is cosine, which we use in each of our models. the cosine similarity between numerical vectors that represent semantic features is first calculated, and then the similarity is ranked according to the cosine value. although this method is relatively simple, it is very effective in comparing similarities. ## experimental results this section presents the results of our experiments. we use the hit rate and mean reciprocal rank (mrr) as our evaluation metrics. we submitted the prediction results on the english test set for two methods using the vit-b/32 clip model. one is the method that uses cosine which achieved a hit rate of 58.3%, an mrr of 72.1, and placed 62nd in the task, and one is the method that uses linear regression which achieved a hit rate of 59.2%, an mrr of 73.0, and placed 57th in the task. they performed slightly below the organizer's baseline model (also using clip), which achieved a hit rate of 60.5%, an mrr of 73.9, and placed 54th on the english test set. since we did not try the vit-l/14 clip model before submission, we only submitted the method using the vit-b/32 clip model. these results are shown in ## conclusion in this study, we used several methods for the v-wsd task in semeval-2023. the main approaches include the one using the image caption model and the one using the clip model. the method using the image caption model does not perform well, probably because the generated image captions are too simple and some word definitions are difficult to associate with image captions. the methods using the clip model are better, reflecting the powerful ability of the clip model to generate uniform vision-language embeddings and its effectiveness in comparing image and text similarity. we also used other methods, including a neural network and a linear regression, to reassign the weights of the elements in the summation phase when calculating the cosine similarity. they all have better performance than simply using cosine similarity.