Augmenters at S em E val-2023 Task 1: Enhancing CLIP in Handling Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt Augmentation and Text-To-Image Diffusion (2023)

Abstract: This paper describes our zero-shot approachesfor the Visual Word Sense Disambiguation(VWSD) Task in English. Our preliminarystudy shows that the simple approach of match-ing candidate images with the phrase usingCLIP suffers from the many-to-many natureof image-text pairs. We find that the CLIP textencoder may have limited abilities in captur-ing the compositionality in natural language. Conversely, the descriptive focus of the phrasevaries from instance to instance. We addressthese issues in our two systems, Augment-CLIPand Stable Diffusion Sampling (SD Sampling).Augment-CLIP augments the text prompt bygenerating sentences that contain the contextphrase with the help of large language mod-els (LLMs). We further explore CLIP modelsin other languages, as the an ambiguous wordmay be translated into an unambiguous one inthe other language. SD Sampling uses text-to-image Stable Diffusion to generate multipleimages from the given phrase, increasing thelikelihood that a subset of images match theone that paired with the text.

## introduction the task of visual word sense disambiguation, as set out in semeval-2023 task 1 overview paper @xcite , can be described as follows: given a target word (the target word) in the context of a two or more word phrase (the full phrase) and ten candidate images, pick the image (the gold image) among the ten candidate images that correctly corresponds to the target word. this competition was run in three languages, english, farsi and italian. we participated in the english version of the task. this task is in line with previous tasks connecting images to text, such as @xcite . we explore two distinct systems to tackle this task. both systems use contrastive language-image pre-training (clip) @xcite as a foundation. clip was trained to associate text and related images, through increasing the cosine similarity (clip-similarity) between the normalized text-embedding and image-embedding of related text and image pairs and decreasing that for unrelated pairs. our first system (augment-clip) augments the clip text embedding by introducing additional context (through key-to-text) and accessing clip text and image embedding in other languages, through third-party implementation of clip for various languages. the second system (sd sampling) samples stable diffusion @xcite to generate multiple images illustrating the semantics of the full phrase and then applies a distance metric to select the candidate image that is closest to the generated images. as standalone systems, augment-clip and sd sampling do not outperform base-clip as the additional context may not correctly extend the target word meaning, but they offer complementary benefits and improve base-clip through ensembling. we ensemble models by first calculating a new probability (or score) for each candidate image by taking the equally weighted average of the probability calculated from the underlying models. each individual model can output a probability for a candidate image. for clip-based models, the probability is the softmax of the candidate image logits. we then rank the candidate images based on the new probability in descending order, with the highest probability candidate image being the predicted image from the ensembled model. see ## experimental setup the trial, train, and test datasets consist of multiple instances. an instance is a target word and a full phrase (containing the target word) and ten candidate images, with one image (the gold image) capturing the semantic meaning of the target word as exemplified in the full phrase. train, trial, and test have 12869, 16, and 463 instances, respectively. for the test data, there are two versions of the dataset provided by the task organizers, differing in the image size @xcite . we perform our predictions on the dataset with the smaller image size. we do not train or fine-tune our models on training data to demonstrate the zero-shot property of our approach, although we do use the training data in part to inform us of which augment-clip system and which sd sampling system to select for task submissions. based on trial data performance, among the three k2t systems, we choose k2t 2, and among the sd sampling systems, we choose textto-image with clip-similarity. as measurements of the performance of the models, hit rate and mean reciprocal rank (mrr) are applied to the model predictions on the trial dataset and test dataset. based on the inputs, the model assigns a score to each candidate image. the model can output one predicted image, with the highest score, or it can output a list of images ordered in decreasing order of score. hit rate is the percentage of instances where the predicted image is the gold image. mean reciprocal rank is the average of the reciprocal of the rank of the gold image in the list of images, ordered based on score. ## conclusion the base-clip system is a strong solution to the task challenge. our system augment-clip complements base-clip to resolve text ambiguity and improves text compositionality. our system sd sampling provides pictorial diversity in ambiguous and unambiguous text interpretation. these two methods offer additional ways to connect text and images.