Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information (2023)

Abstract: Visual Word Sense Disambiguation (VWSD) is a task to find the image that most accurately depicts the correct sense of the target word for the given context. Previously, image-text matching models often suffered from recognizing polysemous words. This paper introduces an unsupervised VWSD approach that uses gloss information of an external lexical knowledge-base, especially the sense definitions. Specifically, we suggest employing Bayesian inference to incorporate the sense definitions when sense information of the answer is not provided. In addition, to ameliorate the out-of-dictionary (OOD) issue, we propose a context-aware definition generation with GPT-3. Experimental results show that the VWSD performance significantly increased with our Bayesian inference-based approach. In addition, our context-aware definition generation achieved prominent performance improvement in OOD examples exhibiting better performance than the existing definition generation method.

## introduction with the development of deep learning technology, research on multimodality such as visio-linguistic models (vlms) has been actively conducted @xcite . in particular, state-of-the-art vlms, such as image-text matching (itm) models @xcite ) and text-to-image generation models @xcite , are employed in many industrial projects, including image retrieval systems @xcite and ai-assisted image generators @xcite . visual word sense disambiguation (vwsd) is a multimodal task of natural language processing (nlp) and computer vision that selects the image which corresponds to the intended meaning of the target word among a set of candidate images (ra- @xcite . we can see that even if the target word ('angora') is the same, different images should be selected according to the context. @xcite . figure 1 is an example of vwsd. for the ambiguous target word foot_0 'angora', we can notice that the answer image should be conditionally changed regarding the context. vwsd can play an important role in several downstream tasks including image retrieval @xcite , action recognition @xcite and visual question answering @xcite . unsupervised vwsd can be formulated in the same way as the itm task @xcite , that is, finding the images that best match the given context. however, vwsd often requires more complex reasoning on both text and images than conventional itm models. the example in figure 2 demonstrates that clip @xcite , a state-of-the-art (sota) itm model, fails to recognize the answer image for the given context foot_1 . this limitation of vlms, where they fail to handle ambiguous words, was also reported in another study on an image generation model @xcite . to ameliorate this problem, we propose to disambiguate visual words with the assistance of a glossary of lexical knowledge-bases (lkbs) without the use of any further training or dataset. specifically, we utilize the sense definitions of an ambiguous word that have been widely exploited in previous lexical semantic tasks @xcite @xcite . herein, since the answer sense of the target word is not provided in the vwsd setting, we propose an approach derived from bayesian inference, using pretained itm models. moreover, in order to deal with out-of-vocabulary (oov) words that cannot find the sense definitions of the target word in lkbs, we suggest the concept of context-aware definition generation (cadg). the definitions of a target word are generated by a large language model, gpt-3 @xcite , as auxiliary information for vwsd. experiments were conducted on semeval-2023 (se23) task 1-visual-wsd @xcite , a publicly available vwsd dataset. furthermore, in the experiments, we utilized two pretained sota itm models: (1) clip @xcite and ( 2 ) flava @xcite . experiments showed that our proposed approach significantly improved the performance of baseline itm models. in addition, we demonstrated that our concept of cadg not only significantly increased the performance of oov cases but is also more advantageous than the previous definition generation approach. we implement experimental codes in https://github.com/soon91jae/uvwsd . the contributions of this paper can be summarized as follows: visual verb sense disambiguation (vvsd) is another task relevant to vwsd. vvsd is a multimodal sense disambiguation task that selects the correct sense of a pair of a given ambiguous verb word and image @xcite . @xcite suggest an unsupervised vvsd approach that takes advantage of various visio-linguistic features (image representation, object label, image caption features) together and calculates the matching score between an image and a sense definition image encoder text encoder 𝐜, 𝑡 𝐬𝐨𝐟𝐭𝐦𝐚𝐱 𝑉 ! text encoder angora city : the capital of turkey; … angora city : a domestic breed of goat … angora city : domestic breed of rabbit … angora city : a long-haired breed of cat … angora city wordnet 𝑡 𝐷 ! 𝐷 ! , 𝐜, 𝑡 argmax 𝐯∈# ! p(𝐯|𝐜, t) = 94.90% 𝐝 % & < 𝐡 𝟏 𝐯 𝐝 % & < 𝐡 𝟐 𝐯 … 𝐝 % , < 𝐡 |# ! | 𝐯 … 𝐝 ) & < 𝐡 𝟏 𝐯 𝐝 ) & < 𝐡 𝟐 𝐯 … 𝐝 ) & < 𝐡 |# ! | 𝐯 ? ./% ) p(𝐯|𝐷 𝒊 𝒕 , 𝐜, 𝑡) < p(𝐷 𝒊 𝒕 |𝐜, 𝑡) 𝐬𝐨𝐟𝐭𝐦𝐚𝐱 𝐡 #… ' ! 𝐯 𝐜 d2i c2d 𝑉 & = candidate images 𝐜 = context 𝑡= target word 𝐷 & =sense definitions of 𝑡 @xmath0. vascon et al. (2021) propose a semi-supervised vvsd method based on game theoretic transduction for inference. meanwhile, gella et al. (2019) demonstrate that a vvsd model trained on multi-lingual vvsd dataset not only benefit the performance on verb sense disambiguation but also boost the performance of a downstream task, the multi-modal machine translation task. our work is related to gloss-enhanced wsd models in that we are using both sense definition and context together. however, our study differs from previous wsd studies in that it tackles a multi-modal task. it is also relevant to vvsd in terms of multi-modal sense disambiguation. however, vvsd systems @xcite are usually designed to analyze a small number of verb words, while the vwsd task contains a lot of nouns and adjectives. finally, our work tackles a new vwsd task and we introduce a method of implementing sense definitions with sota itm models based on bayesian inference where sense definitions as a latent variable. ## unsupervised vwsd incorporating gloss information usually, zero-shot itm models are pretrained without much consideration of polysemous words. for example, figure 2 demonstrates that clip fails to predict the correct answer for the target word 'angora', although it is provided with a clear hint of 'city' in the given context. therefore, the zero-shot performance of pretrained itm models may be limited in the vwsd task. one solution is to use gloss information of a lexical knowledge-base (lkb), particularly exploiting sense definitions. this is because the definitions in lkbs elaborate on each sense for readers who do not know the meaning. thus, we assume that the sense definitions in lkbs can boost itm models to conduct vwsd, by injecting the meaning of the correct sense on the input of these models. however, since there is no correct sense information for the target word, it is difficult to apply it directly. for this reason, we suggest a novel gloss-incorporated vwsd approach inspired by bayesian inference, as presented in eq. 2. suppose d t is a set of definitions for the target word t extracted from an lkb. herein, by using the chain rule, the posterior can be divided into two conditional probabilities associated with a latent variable d t . in this case, the right term p (d t i |c, t) (context to definition; c2d) is predicting the conditional probability over the given ith sense definition d t i for the given target word t and context c which is similar to the gloss-enhanced wsd models @xcite . meanwhile, the left term p (v|d t i , c, t) (definition to image; d2i) is the conditional probability of v for a given the ith sense definition, the context and the target word. in doing so, it allows for the development of sophisticated itms by enriching the context with its relevant sense definition. finally, we can calculate p (v|c, t) by marginalizing over all available sense definitions d t 1...|d t | . figure 3 demonstrates an illustrative concept of our gloss-incorporated vwsd approach with a pretrained clip. first, similar to the original clip, a set of candidate images v t and a context c are input to the image encoder and the text encoder, respectively. meanwhile, a set of definitions of the target word d t is extracted from an lkb. in our work, we utilize wordnet @xcite which has been widely used in previous semantic analysis tasks (pilehvar and camacho-collados, 2019; bevilacqua et al., 2021) as our source of lkb. then d t , c, and t are jointly inputted to the text encoder with the following template. {context} : {ith sense's definition} c2d is computed by the inner product of the hidden representations on the definitions d t 1...|d t | and the context h c⊺ . d2i is then calculated by the inner product of the hidden representations of the input images h v 1...|v t | and d t 1...|d t | ⊺ . both c2d and d2i input to the softmax function transformed into probability distributions. then, we choose the image with the highest probability as the prediction. as a result, for the example in figure 3 , our model can predict the correct answer of the given context 'angora city', whereas the original clip wrongly selects an image of 'angora cat' that produced the highest probability (as shown in figure 2 ), even though the network topology and the pretrained parameters in our model are the same as the original clip model. template: prompt: definition: {target wod} ({pos}): angora (n): a angora is a variety of long-haired domestic cat that originated in turkey. … (a) malkin et al. (2021)'s definition generation. define "{target word}" in {context}. {target wod} ({pos}): define "angora" in angora city. angora (n): a city in turkey that stands on the banks of the angora river, near … definition: figure 4 : examples of gpt-3 generated definitions when context, target word, and part-of-speech are 'angora city', 'angora' and 'noun' (n) respectively. ## handling oov with the context-aware definition generation not all words have their definitions available in a lexical knowledge-base. in particular, proper nouns, compound words, and foreign words frequently induce oov issues. for example, in the se23 dataset, about 14.33% of target words' definitions are not found in the english wordnet. therefore, we propose a solution to tackle the oov issue with the definition generation approach. a previous study showed that gpt-3 can generate the definition of a novel word @xcite . however, since this study does not consider the context of the word, it may not generate the definition in the correct sense. thus, we suggest generating a definition with the prompt that considers both the context and the target word together. figure 4 presents the generated definitions by the approach of @xcite (figure 4a ) and ours (figure 4b ). here, we add a conditional sentence that inputs the context of a target word. for example, when the target word is 'angora' and the context is 'angora city', we use a conditional sentence, "define "angora" in angora city.", in front of the previous input "angora (n)". indeed, in the example, the definition generated with our method shows a better description compared to the previous method. ## conclusion and future work this paper introduces a novel vwsd methodology to effectively incorporate gloss information from an external resource. our work mainly has two innovations: 1) bayesian style inference for sota itms, and 2) context-aware definition generation with gpt-3 to overcome the oov issue. experimental results show that our proposed bayesian style inference-based wordnet integration significantly improves vwsd performance without additional training. for the ambiguous target words, the performance of our approach is comparable to pipeline systems using finetuned wsd models. moreover, context-aware definition generation helps mitigate oov issues in the downstream vwsd tasks and shows higher performance compared to the previous definition generation approach. in the future, we plan to tackle the error cascading caused by over-confidence in c2d probability. for this, we may explore a prompting that is known to have good performance in zero-shot prediction @xcite . in addition, to deal with the hallucination and misdisambiguation problems of gpt-3 generated definitions, we may employ controllable generation by resampling @xcite .