team PN at S em E val-2023 Task 1: Visual Word Sense Disambiguation Using Zero-Shot M ulti M odal Approach (2023)

Abstract: Visual Word Sense Disambiguation shared task at SemEval-2023 aims to identify an image corresponding to the intended meaning of a given ambiguous word (with related context) from a set of candidate images. The lack of textual description for the candidate image and the corresponding word’s ambiguity makes it a challenging problem. This paper describes teamPN’s multi-modal and modular approach to solving this in English track of the task. We efficiently used recent multi-modal pre-trained models backed by real-time multi-modal knowledge graphs to augment textual knowledge for the images and select the best matching image accordingly. We outperformed the baseline model by ~5 points and proposed a unique approach that can further work as a framework for other modular and knowledge-backed solutions.

## introduction as per semeval-2023 task 1 definition for visual- @xcite . "given a word and some limited textual context, the task is to select among a set of candidate images the one which corresponds to the intended meaning of the target word." the final evaluation data in this task has 3 language tracks english, italian, and farsi. an initial trial data set was released with 16 samples, whereas the final training dataset was released with 12869 samples, where each pair of text and contextual text were mapped to a target image that has to be retrieved from a set of 10 images. the images attached to each of the training samples overlap with other samples. and thus, a total of 12999 unique images were found in the training data pool. these images vary from a wide variety of genres. a sample input data example can be seen in fig. 1 . here, 'white' is the term, and 'white yolk' is the context. the model has to retrieve the right image from the set of input images. ## results the results of model performance on various approaches are tabulated in all these above-mentioned metrics were reported only for contextual text-image pairs as they performed well against text-image pairs. fusing the results of text-image and contextual text-image led to improving the top 1 accuracy to 64.7% and mrr to 77.6%. approach top 1 accuracy mrr baseline 60.47% 73.87% blip itm 61.1% 75.4% blip itm + te 62.4% 75.2% blip itm + fte 64.7% 77.6% mmkg 49.0% 65.5% mmkg + blip itm + fte 66.9% 78.6% multi-modal kgg was able to get the top 1 accuracy of 49.0% with a mean retrieval rank of 65.5%. however, aggregation of multi-modal kgg, blip itm, and fusion term expansion yielded in top 1 accuracy of 66.9% with an mrr of 78.6%