LT at S em E val-2023 Task 1: Effective Zero-Shot Visual Word Sense Disambiguation Approaches using External Knowledge Sources (2023)

Abstract: The objective of the SemEval-2023 Task 1: Visual Word Sense Disambiguation (VWSD) is to identify the image illustrating the indented meaning of a target word and some minimal additional context. The omnipresence of textual and visual data in the task strongly suggests the utilization of the recent advances in multi-modal machine learning, i.e., pretrained visiolinguistic models (VLMs). Often referred to as foundation models due to their strong performance on many vision-language downstream tasks, these models further demonstrate powerful zero-shot capabilities. In this work, we utilize various pertained VLMs in a zero-shot fashion for multiple approaches using external knowledge sources to enrich the contextual information. Further, we evaluate our methods on the final test data and extensively analyze the suitability of different knowledge sources, the influence of training data, model sizes, multi-linguality, and different textual prompting strategies. Although we are not among the best-performing systems (rank 20 of 56), our experiments described in this work prove competitive results. Moreover, we aim to contribute meaningful insights and propel multi-modal machine learning tasks like VWSD.

## introduction this paper presents and analyses effective zero-shot approaches for the semeval-2023 task 1: visual word sense disambiguation (vwsd) @xcite . in traditional word sense disambiguation (wsd), the context or sentence in which ambiguous words, i.e., words with multiple meanings, occur is used for disambiguation by identifying the correct sense in a sense inventory. frequently used as sense inventories are dictionaries or knowledge bases such as wordnet or dbpedia. as opposed to traditional wsd, in the vwsd shared task, images are used to disambiguate a word given a context. precisely, given a word and another word serving as context, the task is to identify the image that corresponds to or illustrates the correct meaning in a set of ten images. in the trial phase of the task, 12869 samples in english, including gold labels, were provided. however, besides 463 english samples, the final phase test data also contains 305 italian and 200 farsi samples. a random vwsd figure 1 : an illustration of a random vwsd sample with the target word 'bonxie', the context 'bonxie skua', and the correct image highlighted by a a golden border sample is illustrated in figure 1 . due to the multi-modal nature of the task, it requires methods or models to understand textual semantics contained in the target word and context word and visual semantics contained in the images. therefore, our approach leverages state-of-the-art pretrained visiolinguistic models (vlms) in a zeroshot fashion, i.e., we do not continue pretraining or finetune. this is motivated by several reasons based on the tasks data: first, the samples are not restricted to a particular topic, e.g., animals or plants, but can belong to any topic (open-domain), which rules out domain adaption strategies. further, since current vlms are trained on massive amounts of text-image pairs crawled from the internet, continuing pretraining on additional open-domain data will likely have no effect. second, the textual context is minimal and contains only a single or, at most, two additional words, which are often rare english words like the latin names of certain plants. additionally, it frequently requires expert knowledge to identify the correct image because the set of ten images often contains images very similar to the gold image. due to this, finetuning a vlm on the provided data is ineffective, which we also confirmed in conducted but not reported finetuning experiments. third, recent pretrained vlms have proven capable of grasping textual and visual semantics out of the box by demonstrating strong zero-shot performance in many vision-language downstream tasks. the central strategy of the approaches presented by this work is to utilize given information to acquire additional context from external knowledge sources. a pretrained vlm then computes embeddings for the acquired context and all images to find the image with the maximum similarity. see section 3 algorithmic details and an illustrative overview. pretrained visio-linguistic models the combination of recent advances in natural language processing and computer vision has greatly increased interest and performance in the emerging field of multi-modal machine learning, especially in visio-linguistic models (vlms) with strong zero-shot performance on many downstream tasks @xcite . in this work, we specifically focus on vlms referred to as clip (radford et al., 2021), which we utilize to compute semantic representations of text and images in a joint vector space. there exist various versions of clip, all having the following in common: a clip model implements a dual encoder architecture with two separate encoders: a textual encoder, typically a bert-based @xcite language model, and a visual encoder, typically cnns like resnet @xcite or convnext @xcite or a transformer @xcite like vit @xcite . the encoders are jointly trained on massive amounts of textimage pairs to maximize the similarity between matching pairs in large batches via contrastive loss in an unsupervised fashion. while there are other vlms with strong zero-shot capabilities, such as align @xcite , flava @xcite , or coca @xcite , we prefer clip because there exist many publicly available pretrained versions, it is conveniently usable via multiple open-source libraries and frameworks like huggingface @xcite or open-1 https://github.com/uhh-lt/vwsd_semeval_23 clip @xcite , and it has an active and large community. specifically, we evaluate the performance of our vwsd approaches using different sizes of the original model @xcite , models trained on the publicly available datasets laion @xcite , and multi-lingual versions from sentencetransformer @xcite and openclip @xcite ). an overview with more details about the clip models employed in this work is given in external knowledge since the provided context in a vwsd sample is minimal, we use different external knowledge sources to acquire additional contextual information. one of the sources is wikipedia, from which we retrieve article summaries using the target word and the additional context word(s). another source is a large-scale corpus @xcite , containing 252b tokens based on english commoncrawl data, which we have indexed using elasticsearch @xcite . the only multi-modal external knowledge source we employ is visualsem @xcite , a high-quality knowledge graph containing 90k nodes with 1.3m glosses in 14 languages and 930k images associated with the nodes. unfortunately, our request to use the large-scale multimodal knowledge graph babelnet @xcite was rejected. babelnet arguably would have improved our results significantly since it contains 1.4m senses described by 135m glosses in 500 languages and illustrated by 51m images. ## approaches this section provides details for the zero-shot approaches to vwsd presented by this work. section 4 then analyzes and discusses the evaluation results. the general strategy, illustrated in figure 2 , comprises five primary steps: in (1), we acquire additional context from an external knowledge source (see section 2) using the textual information provided in a vwsd sample. in (2) and (3), we leverage a pretrained clip model to compute embeddings from the acquired context and all ten images contained in the sample. (4), we compute the cosine similarity of the text embedding and all image embeddings and (5) select the image with the maximum similarity as the best matching image. depending on the method, we use different external knowledge sources, employ different pretrained clip models, or compute the textual or visual embeddings differently. baseline -no external knowledge our baseline method does not use external knowledge but computes the textual embedding only from the target word and or context in a vwsd sample. however, we test different template sentences or prompts to compute the textual embedding (see ## evaluation and analysis in this section, we present and analyze the evaluation results of our approaches described in section 3. the evaluation is based on the final multilingual evaluation data, including the gold labels released in the google group after the competition 3 . evaluation results for the approaches discussed in this section are depicted in figure 3 . ## conclusion this work presents various zero-shot visual word sense disambiguation approaches using different external knowledge sources. across all approaches, we analyzed different pretrained versions of the clip model varying in size, training data, and multi-lingual capabilities. further, we assessed the suitability of three external knowledge sources: wikipedia, a large-scale english common crawl corpus, and the multi-modal knowledge graph visu-alsem. our best-performing approach involved the common crawl corpus which we queried for sentences containing the target word and context, serving as additional context. by translating farsi and italian samples into english, we achieved strong competitive results not only for english samples.