Query Generation Using GPT -3 for CLIP -Based Word Sense Disambiguation for Image Retrieval (2023)

Abstract: In this study, we propose using the GPT-3 as a query generator for the backend of CLIP as an implicit word sense disambiguation (WSD) component for the SemEval 2023 shared task Visual Word Sense Disambiguation (VWSD). We confirmed previous findings — human-like prompts adapted for WSD with quotes benefit both CLIP and GPT-3, whereas plain phrases or poorly templated prompts give the worst results.

## introduction the semeval 2023 shared task vwsd 1 combines wsd and image retrieval (ir), which aims to select a correct image among ten candidates using a phrase containing ambiguous words. neural models are likely to be attracted by frequent tokens, labels, and senses of ambiguous words, particularly in limited contexts. we determined that clip @xcite fails to find the correct images using phrases with the ambiguous words of frequent senses even enhanced with contrastive learning on large-scale data. as shown in figure 1 , given phrase "andromeda tree", the pretrained clip selected incorrect images that focused on either constellation "andromeda" or part of "tree", neglecting the phrase's meaning entirely. this sample demonstrates ambiguity as a challenge for state-of-the-art neural models. therefore, we exploited gpt-3 @xcite , a large language model (llm), for its pretrained knowledge as implicit sense disambiguation and phrase context enrichment for this task. prompt engineering boosts model performance and plays a crucial role in applying llms to many nlp tasks, which lessens training and testing discrepancies resulting from human-like languages figure 1 : clip ranks ten images with respective relevance to the phrase "andromeda tree", which consists of ambiguous word "andromeda" and limited context "tree". the goal is to select the correct image from ten images of different relevance corresponding to the intended meaning of "andromeda tree". note that the first (blue) and second (green) images ranked higher than the correct third image (red). @xcite . a prompt refers to a text or a set of instructions that guides the model to generate a specific type of response or output. a prompt can be a question, statement, keyword, or sequence of words that provide context and information to the model. recent research on prompt techniques demonstrates that well-designed prompts spur the potential of neural models without modifying their parameters @xcite . we are curious about how prompts can improve the performance of clip on vwsd. our main contributions are as follows: ## method each vwsd phrase contains an indication of the ambiguous word(s). as illustrated in figure 2 , we first introduce a baseline clip that takes text input as either vwsd short phrases or a longer query enhanced with templates for ambiguous words. then, we leverage gpt-3 to further enrich these queries for clip. ## conclusion and future work we explored the effects of a query on vwsd and used gpt-3 as a key to generate queries. after finetuning clip with queries generated from gpt-3, we determined that queries generated by gpt-3 using prompts improved the performance in terms of accuracy. in the future, we plan to apply some other multimodal models and compare the results with those of existing works. we also intend to adopt gpt-4 foot_0 to generate knowledge considering both textual and visual cues for vwsd.