OPI PIB at S em E val-2023 Task 1: A CLIP -based Solution Paired with an Additional Word Context Extension (2023)

Abstract: This article presents our solution for SemEval-2023 Task 1: Visual Word Sense Disambiguation. The aim of the task was to select the most suitable from a list of ten images for a given word, extended by a small textual context. Our solution comprises two parts. The first focuses on an attempt to further extend the textual context, based on word definitions contained in WordNet and in Open English WordNet. The second focuses on selecting the most suitable image using the CLIP model with previously developed word context and additional information obtained from the BEiT image classification model. Our solution allowed us to achieve a result of 70.84% on the official test dataset for the English language.

## introduction visual word sense disambiguation (vwsd) is a classification problem that involves selecting the one image from a set of candidates that most accurately represents the meaning of a target word. this task is important because many words have multiple meanings, and the intended meaning of a word can be ambiguous based on its context. the problem of word ambiguity is a well-established one @xcite , but vwsd is relatively new and contributions to the subject remain sparse. to popularise this issue, semeval 2023 task 1: v-wsd: visual word sense disambiguation @xcite was organised. the organisers prepared the task for three languages: english, farsi, and italian. it was possible to participate in the competition for each language separately. our solution is designed exclusively for english. this article describes our solution for the competition. we focus chiefly on describing our attempts to extend the context for given ambiguous words. this is crucial for selecting the most appropriate image. the remainder of the publication is organised as follows: section 2 describes the datasets provided by the organisers for each phase of the competition; section 3 presents the design of the system in detail; section 4 offers information on the evaluation metrics used and describes the experiments; section 5 analyses the results of the experiments; section 6 contains summarises and presents our conclusions. ## dataset description the datasets were prepared by the organisers as tsv files. each line represents a sample that comprises a single word, limited textual context, and ten image file names. a separate directory that contains image files was attached to each collection. in addition, for the trial and training set, a so-called gold file was included. it contains information about the best-matching image for each word. after the competition, the organisers released the gold file for the test datasets. the trial data comprises only sixteen samples and 160 images. the training data comprises 12869 samples and 12999 images. both collections were prepared for the english language. however, in the training dataset, when single words in another language, such as polish, chinese, or latin were found, the word was replaced by an icon. the test dataset has 463 samples and 8100 images provided in two versions: original size and resized. the collection was prepared for three languages: english, farsi, and italian. ## system description the architecture of our system is presented in figure 1 from the best-to the worst-matching context. detailed descriptions of these modules are presented in the proceeding subsections. ## experiments all of our experiments focused on a sub-task for english. to implement the context extension module, we used wordnet v3.1 and open english word-net 2021, a derivative solution of the princeton wordnet developed under open-source methodology. microsoft's beit model 1 was used to classify candidate images. to implement the ranking module, we used the ready-made clip model foot_1 from laion. for both beit and clip model we used the original model weights, without any finetuning. ## error analysis the results of the experiments for our system enable us to distinguish three categories of error: errors in matching definitions, errors in classifying images, and subjective judgment in selecting the correct image. the methods of extending the context with definitions in wordnet or open english wordnet were based on the assumption that occurrences of keywords (target words, context words, image class names, and their related terms) would appear among them. however, an analysis of the results highlighted that keywords alone were frequently insufficient. a larger dictionary of synonyms would improve the effectiveness of this approach markedly, but unfortunately we didn't have access to such resource. the beit model used to classify the images also did not always return the correct labels; for that reason, its use should be considered as supportive in the selection of the right definitions. nevertheless (leaving aside the correctness of the image classification), given the number of occurrences of class names in extended contexts (tables 3 and 5 ), the use of this approach is justified. in assessing the correctness of the model, there is a risk that the image that the model should return fails to represent the given phrase clearly. some phrases do not refer to issues that are easily represented by objects, such as species of animals, plants, or names of objects; they may instead refer to abstract concepts, such as moods ( ## conclusion this article presents our solution for semeval-2023 task 1: visual word sense disambiguation. we first explained the essence of the vwsd task. next, we presented the system architecture as a proposed solution to the problem. we described the principles of operation of individual modules. we presented an error analysis conducted on the basis of the results of our experiments. we demonstrated that the relatively simple construction of our solution enabled us to achieve a fairly good final result. the error analysis enables us to propose further work on the issue, which should focus primarily on the expansion of the context extension module. wikipedia resources and further dictionaries of synonyms, such as roget's 21st century thesaurus @xcite could be used for this purpose. our proposed solution is adapted only to english. taking into account the other languages, it would be worth considering a solution that would address the problem of multilingualism. this is a good opportunity and topic for further research.