Ebhaam at S em E val-2023 Task 1: A CLIP -Based Approach for Comparing Cross-modality and Unimodality in Visual Word Sense Disambiguation (2023)

Abstract: This paper presents an approach to tackle the task of Visual Word Sense Disambiguation (Visual-WSD), which involves determining the most appropriate image to represent a given polysemous word in one of its particular senses. The proposed approach leverages the CLIP model, prompt engineering, and text-to-image models such as GLIDE and DALL-E 2 for both image retrieval and generation. To evaluate our approach, we participated in the SemEval 2023 shared task on “Visual Word Sense Disambiguation (Visual-WSD)” using a zero-shot learning setting, where we compared the accuracy of different combinations of tools, including “Simple prompt-based” methods and “Generated prompt-based” methods for prompt engineering using completion models, and text-to-image models for changing input modality from text to image. Moreover, we explored the benefits of cross-modality evaluation between text and candidate images using CLIP. Our experimental results demonstrate that the proposed approach reaches better results than cross-modality approaches, highlighting the potential of prompt engineering and text-to-image models to improve accuracy in Visual-WSD tasks. We assessed our approach in a zero-shot learning scenario and attained an accuracy of 68.75\% in our best attempt.

## introduction visual word sense disambiguation (visual-wsd) is a challenging task in natural language processing that aims to disambiguate words or phrases with multiple possible meanings and retrieve the image that best matches the semantic content of the input text. the semeval 2023 shared task on "visual word sense disambiguation (visual-wsd)" provides a valuable benchmark for evaluating the performance of visual-wsd systems @xcite . in this paper, we propose an approach to visual-wsd that utilizes the clip @xcite model for image retrieval and prompt engineering with openai's completion model for generating images from textual descriptions. we describe our methodology and experimental results, including our use of zero-shot learning and comparison of two text-to-image models, glide @xcite and dall-e 2 @xcite , for image synthesis. furthermore, we compare the performance of our approach with image retrieval using clip encoders for both input text and candidate images, finding that our best result using generated images reaches better results than using text and images for evaluation. our findings suggest that prompt engineering and text-to-image models have the potential to significantly improve the accuracy of image retrieval tasks. finally, we discuss the implications of our results for future research in this area. we should note that we only test our model with english texts, and not the optional persian and italian languages as part of the competition. the outline of the paper consists of four sections: background, system overview, experimental setup, and conclusion. in the background section, the authors provide an overview of the problem of visual word sense disambiguation and its importance in natural language processing. the system overview section outlines the proposed approach, which involves using clip, prompt engineering, and text-to-image models for image retrieval and synthesis. the experimental setup section describes the evaluation methodology and the results obtained using different combinations of tools. finally, the conclusion section summarizes the findings and highlights future research opportunities in visual-wsd. ## background visual word sense disambiguation (visual-wsd) is a challenging problem in natural language processing that aims to disambiguate word meanings using visual cues. traditional wsd methods rely on textual data, but they may not always be effective in multiple meanings or ambiguous contexts. visual-wsd approaches use visual information, such as images or videos, to provide additional context for word disambiguation @xcite . the clip model is a state-of-the-art model for zero-shot image classification and retrieval that has the potential for visual-wsd tasks. clip stands for contrastive language-image pre-training, and it was developed by openai. clip is a neural network that is pre-trained on a large corpus of text and images and is able to encode both modalities into a shared embedding space. this allows for efficient retrieval of images that are semantically similar to input text, even in zero-shot settings @xcite . prompt engineering is a technique used to generate more effective prompts for models such as clip (?). in the context of visual-wsd, prompt engineering involves generating textual prompts that include limited textual context and the target word or phrase. these prompts are used to synthesize images that are semantically related to the input text @xcite ). openai's completion models are one example of a prompt engineering technique that can generate high-quality prompts @xcite . text-to-image models like glide and dall-e 2 can be used for image retrieval and synthesis in visual-wsd tasks. glide is a text-to-image model developed by openai that is able to generate high-quality images from textual descriptions @xcite . dall-e 2 is another text-toimage model developed by openai that can generate complex images from textual prompts @xcite . these text-to-image models can be used to generate images that are semantically related to the input text, providing additional context for word disambiguation. our goal in this study is to contribute to the growing body of research on visual-wsd and to inspire further development in this important area of natural language processing. ## system overview our visual-wsd system consists of three main components: prompt engineering, text-to-image systems, and evaluation using clip. figure 2 provides a high-level overview of our system architecture. ## experiment setup the study involved an image selection task using a dataset that requires selecting an image that corresponds to a given word and limited textual context. two types of prompts were used: "simple prompts", which involved concatenating the word and limited textual context to create a text prompt, and "generated prompts," which involved using openai's "text-davinci-003" completion model to "generated prompts" by adding the phrase ", can you describe it as a picture?" to the end of the "simple prompts". two text-to-image models, glide and dall-e 2, were used to generate images from the text prompts. the generated images and candidate images were then processed using the "vit-b/32" clip model to obtain embeddings. cosine similarity was used to compare the embeddings and determine the closest image to the input text. ## results and conclusions we conducted experiments to evaluate the performance of our visual word sense disambiguation (visual-wsd) system using two different text-toimage models: glide and dall-e 2. the system was evaluated with two different prompts: a "simple prompt" that combines the target word and limited textual context, and a "generated prompt" that is generated using a completion model and also conduct an experiment to evaluate a system that uses "simple prompt" and candidate images for visual-wsd task. our results show that the "generated prompt" with dall-e 2 leads to the highest accuracy of 68.75%, while the lowest accuracy of 18.75% was achieved using the "simple prompt" with glide. in our research, it was observed that dall-e 2 outperformed glide with statistically significant results when evaluated using "simple prompts" and "generated prompts", exhibiting a respective improvement. while using cross-modality embeddings achieve 56.25% accuracy. moreover, we observed that changing the prompts from "simple prompts" to "generated prompts" can substantially improve the accuracy of glide from 18.75% to 43.75%, and dall-e 2 from 62.5% to 68.75%. the results are summarized in