Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation (2023)

Abstract: Visual Word Sense Disambiguation (VWSD) is a novel challenging task with the goal of retrieving an image among a set of candidates, which better represents the meaning of an ambiguous word within a given context. In this paper, we make a substantial step towards unveiling this interesting task by applying a varying set of approaches. Since VWSD is primarily a text-image retrieval task, we explore the latest transformer-based methods for multimodal retrieval. Additionally, we utilize Large Language Models (LLMs) as knowledge bases to enhance the given phrases and resolve ambiguity related to the target word. We also study VWSD as a unimodal problem by converting to text-to-text and image-to-image retrieval, as well as question-answering (QA), to fully explore the capabilities of relevant models. To tap into the implicit knowledge of LLMs, we experiment with Chain-of-Thought (CoT) prompting to guide explainable answer generation. On top of all, we train a learn to rank (LTR) model in order to combine our different modules, achieving competitive ranking results. Extensive experiments on VWSD demonstrate valuable insights to effectively drive future directions.

## introduction visual word sense disambiguation (vwsd) is a recently introduced challenging task where an ambiguous target word within a given context has to retrieve the proper image among competitive candidates @xcite . for example, the phrase andromeda tree contains the ambiguous target word andromeda accompanied by the context tree which resolves this ambiguity. out of the 10 candidates presented in fig. 1 , a vwsd framework attempts to retrieve the ground truth image, denoted with colored border. even though vwsd is essentially a text-image retrieval task, there are some fundamental differences. first of all, the context given for an ambiguous word is minimal, most often limited to a single word, upon which a retrieval module should rely to retrieve the proper candidate. additionally, the candidate images themselves pose significant challenges; for example, by observing the candidates of fig. 1 , which are related to several meanings of the ambiguous word andromeda (can be either a constellation, fish species, tree, reptile etc), a successful retrieval system should be fine-grained enough and highly sensitive to the contextualization of the ambiguous target word. in that case, the context word tree should be prominent enough -from the perspective of the retrieval module-to resolve ambiguity. at the same time, this retrieval module should not excessively rely on the tree context: images containing flowers and green grass induce some expected visual bias in the retrieval process, thus the image having the highest probability of containing a tree may be selected, ignoring the ambiguous andromeda attribute. of course, there is also the possible scenario that a retrieval model has never been trained on the ambiguous word at all: the rarity of the concepts present in the target word vocabulary increases the probability of exclusively relying on the well-known context word, resulting in significant randomness during selection. to this end, vwsd trustworthiness also arises as a critical point, raising the need for explainable solutions. in this work, we showcase a vast variety of implementations for vwsd. multiple experiments are conducted for each of the implemented approaches, achieving one of the first extensive contributions to this interesting task: our code can be found at https://github.com/anastasiakrith/multimodalretrieval-for-vwsd/ . visual word sense disambiguation (vwsd) @xcite is only recently introduced as part of the semeval 2023 challenge. so far, the concurrent work of @xcite can act as a measure of comparison to ours. we adopt and extend some ideas presented in their paper, while further expanding the experimental suite to cover variable approaches to the task. moreover, we step upon the usage of llms for vwsd as in @xcite to address both performance improvement and explainability aspects. llms as knowledge bases is a core idea followed throughout our paper, as enriching the short phrases of the vwsd dataset can facilitate target word disambiguation, and thus improve retrieval. traditionally, knowledge enhancement for disambiguation was performed via knowledge graphs @xcite . the usage of knowledge graphs was also favored for knowledge enhancement of multimodal tasks @xcite . nevertheless, large language models (llms) as knowledge bases (llm-as-kb) @xcite is a novel paradigm, presenting some interesting capabilities compared to traditional knowledge graphs. knowledge retrieval from llms is implemented via prompting @xcite , which attempts to appropriately trigger the llm in order to provide the fact requested. to this end, recent vl breakthroughs favor the llm-as-kb paradigm for knowledge enhancement, even though some unresolved shortcomings may be inherited @xcite . we opt for incorporating the llm-as-kb paradigm within our experimentation to investigate performance gains over knowledge-free baselines, following @xcite . ## method we followed 6 approaches to investigate the vwsd task from several different perspectives. all our approaches were tested exclusively on english. ## experimental results the vwsd dataset is comprised of 12869 training samples and 463 test samples; each sample contains 10 images (appendix a). all our approaches are evaluated on the vwsd test split using accuracy and mean reciprocal rank (mrr) metrics. llms for phrase enhancement in tab. 3, we present results regarding llm-based phrase enhancement involving all vl retrieval models (with and without penalty p(i)). baselines refer to vl retrieval with non-enhanced phrases t. additional quantitative results including more llms, as well as qualitative examples of how knowledge enhancement benefits disambiguation and thus retrieval are provided in appendix c. we can easily observe that different prompts contribute towards better results per llm, while blip-l f is the most successful vl retriever; however, align (with penalty) achieves top performance, together with gpt-3 and "meaning" as the prompt. in general, large scale llms (gpt-3, gpt-3.5-turbo) are able to clearly surpass the respective non-enhanced baselines in most cases. interestingly, for smaller llms (up to 6.7b parameters), scaling-up does not necessarily imply advanced performance: opt-6.7b (tab. 12) and galactica-6.7b (tab. 13) fall below their smaller bloomz-3b competitor when the same prompts are used. nevertheless, contrary to gpt-3 and gpt-3.5-turbo, the phrase enhancement that few-billion parameter llms can offer in total is only marginal in the best case, and sometimes even fail to compete against their non-enhanced baselines, indicating that they do not contain the necessary knowledge to enrich rare target words with respect to their given context. therefore, our llm-enhancement analysis reveals that the necessary enrichment for vwsd may be only achieved when employing large-scale llms, most probably being on par with other emergent llm abilities @xcite . image captioning in tab. 4 we present results on text retrieval between extracted image captions c i and given phrases t, which are achieved us- prompt name qa prompt template think (greedy) "q: what is the most appropriate caption for @xmath0: (a) <caption for image 1> (b) <caption for image 2> ... a: let's think step by step. " think (beam) "q: what is the most appropriate group of captions for @xmath1: (a) <captions for image 1 (separated with comma)> (b) <captions for image 2> ... a: let's think step by step. " cot "<@xmath2, among a through j, the answer is " no_cot (greedy) "q: what is the most appropriate caption for @xmath3: (a) <caption for image 1> (b) <caption for image 2> ... a: " no_cot (beam) "q: what is the most appropriate group of captions for @xmath4: (a) <captions for image 1> (b) <captions for image 2> ... a: " wikipedia & wikidata image retrieval in tab. 5 results regarding image-to-image retrieval between candidates i and web retrieved images i w are presented. out of the 463 samples of the test set, wikipedia api and wikidata api returned results for 460 and 324 phrases respectively. even best results for image-to-image retrieval are not competent against our previous approaches; we assume that exclusively visual representations are not expressive enough to distinguish fine-grained details between semantically related candidates. learn to rank in tab. 6 we exhibit results using align as the vl retriever. the presented feature combinations involve the following: (1) baseline features: the choice for incorporation (or not) of penalty p(i) in score(t, i) for the vl retrieval; (2) llm-enhancement features: the prompt to produce enhanced phrases t e (or an ensemble of prompts leading to multiple t e ) and the choice for incorporation (or not) of p(i) in score(t e , i); (3) text retrieval features: the captioner to generate c i , together with the text embedding model and the similarity measure (cosine/euclidean/manhattan) for text-to-text retrieval, as well as the phrase (original t, or enhanced t e , or an ensemble of enhanced phrases t e derived from different llms); (4) image retrieval features: image embedding model and similarity measure (cosine/euclidean/manhattan) for imageto-image retrieval. additional results for the ltr are provided in appendix e. for all the experiments of tab. 6 we used the following hyperparameter configuration: n_estimators: 500, early_stopping: 100, learning rate: 0.03, feature_fraction: 0.25, max_bin: 100, min_child_samples: 50 and reg_alpha: 0.05. an 80-20 train/validation split was followed, allocating 2514 samples in the validation set. the ablations of feature combinations presented in tab. 6 are highly informative, indicating that different features pose a varying influence on the final result, while the amount of incorporated features is also significant. in general, incorporating llm-based phrase enhancement in ltr is highly beneficial, offering optimal metric results compared to other feature combinations, or our other approaches presented in tab. 3, 4. overall best results are achieved when including all features (colored instances of tab. 6). this is an interesting observation since standalone text retrieval (tab. 4) and image retrieval (tab. 5) experiments did not provide competitive metric results; nevertheless, considering the respective features in ltr training benefits performance. moreover, ensembling of features is highly beneficial. this applies on both ensembling the llm-enhanced prompt features (e.g. all prompts combines features from t exact , t what_is , t describe , t meaning ), as well as ensembling phrase features for text-to-text retrieval (all t e + t refers to combining features from all the aforementioned 4 enhancements, plus the original given phrase t). as demonstrated in tab. 6, most ensemble feature combinations help surpassing baselines and other implementations @xcite . the implemented ltr module is computationally efficient, as it only requires a cpu for training, while achieving state-of-the-art performance (tab. 9 ). the clear pattern that arises from the ablation cot prompting results can reveal the internal steps followed by an llm to reach an answer. in fig. 2 , we present the candidate images for the phrase t "rowing dory", with candidate c serving as the golden image. captions are extracted using git-l captioner with greedy search, thus returning one caption c i per candidate i. we then transform "rowing dory" in a question q, following the think prompt template, as presented in tab. 2, with captions c i , @xmath5, ..., j forming the answer choices. consequently, we produce answers via gpt-3.5turbo with and without cot, as demonstrated in tab. 7. in the presented case, cot encourages choosing the correct answer c over j, which is preferred by gpt-3.5-turbo when cot is omitted. furthermore, cot elicits the reasoning process of the llm, thus outputting a detailed explanation regarding the choice of a candidate against semantically similar candidates; for example, c (a small boat in the water with two oars) and j (a row boat with people on it in the water) are indeed closely related and can be easily confused. as one of the basic vwsd challenges is the fine-grained discrimination between semantically similar candidates, we consider such llm-derived explanations to be highly informative and accurate for discerning the subtle details between them. more cot/no_cot examples are presented in appendix f. ## discussion after the announcement of the semeval-2023 proceedings, more works achieving state-of-the-art results on vwsd for english have been released. zhang et al. (2023b) utilize online dictionaries and biencoder architectures for disambiguation and a matching module that selects the most appropriate modality to guide final retrieval. their approach significantly differs from ours, both in the disambiguation and the ranking part. @xcite rely on an ensemble approach involving fine-tuning clip and extracting a joint image-text representation. in their paper they make no usage of knowledge sources but rather resort on more computationally heavy approaches; instead, in our approach we only train a lightweight ranking module, while the usage of computationally demanding modules such as vl retrievers and llms is restricted to first, we need to identify the subject of the caption, which is the rowing dory. then, we need to choose a descriptive and informative caption that accurately represents the image. out of all the answer choices, (c) "a small boat in the water with two oars" would be the most appropriate caption for the rowing dory as it accurately describes the image and provides relevant information about the boat. answers (d), (f), and (j) also provide some information about the boat and the people in it, but they either include irrelevant information or are too vague to be an appropriate caption for the image. therefore, among a through j, the answer is (c) "a small boat in the water with two oars." ## conclusion in this work, we provided an extensive foundational experimentation and analysis on the novel visual word sense disambiguation (vwsd) task. specifically, we employed several state-of-the-art models for vl retrieval to build strong baselines, showcased the merits of enhancing ambiguous phrases with external knowledge stored in llms and achieved competitive ranking results by training a lightweight retrieval module using features extracted from our independent experiments. moreover, we obtained useful explanations that unveil the reasoning process behind vwsd via chain-of-thought prompting. our results surpassed concurrent implementations and given baselines, while demonstrating valuable insights that can drive future state-of-the-art implementations. we plan to expand our experimentation towards exploring the use of soft prompting to advance knowledge enhancement, and explore the possibility of harnessing large-scale knowledge graphs for further enrichment of ambiguous phrases. finally, we view explainability aspects of vwsd as a critical direction to be studied as future work.