SRCB at S em E val-2023 Task 1: Prompt Based and Cross-Modal Retrieval Enhanced Visual Word Sense Disambiguation (2023)

Abstract: The Visual Word Sense Disambiguation (VWSD) shared task aims at selecting the image among candidates that best interprets the semantics of a target word with a short-length phrase for English, Italian, and Farsi. The limited phrase context, which only contains 2-3 words, challenges the model’s understanding ability, and the visual label requires image-text matching performance across different modalities. In this paper, we propose a prompt based and multimodal retrieval enhanced VWSD system, which uses the rich potential knowledge of large-scale pretrained models by prompting and additional text-image information from knowledge bases and open datasets. Under the English situation and given an input phrase, (1) the context retrieval module predicts the correct definition from sense inventory by matching phrase and context through a biencoder architecture. (2) The image retrieval module retrieves the relevant images from an image dataset.(3) The matching module decides that either text or image is used to pair with image labels by a rule-based strategy, then ranks the candidate images according to the similarity score. Our system ranks first in the English track and second in the average of all languages (English, Italian, and Farsi).

## introduction word sense disambiguation (wsd) is a challenging task of natural language processing, which aims at identifying the polysemic word with correct interpretation @xcite . it has numerous applications, such as machine translation, search engines, and text understanding. the traditional wsd task provides the predicting word with its context, which is supposed to be a sentence containing the word, and chooses the proper sense from the predefined inventories @xcite . the model is asked to extract the semantic feature of the target word from the particular text and match it with the most practical sense. the wsd task can be processed as a classification problem among the senses as labels in the given inventory by taking advantage of the neural language model's powerful semantic understanding capability. the appearance of visual wsd connects the different modalities and gives the word sense a new presentation form. the sense inventory now is a set of images instead of text definitions, and the target word has to be paired with the corresponding image. the vision-language model, clip @xcite , has shown its outstanding performance on unifying image-test modalities, and it provides potential knowledge for semantic understanding required by disambiguation. the semeval-2023 task-1 visual word sense disambiguation (vwsd) @xcite is a multilingual wsd task, for english, italian, and farsi, with the phrase as textual context and images as visual senses. most of the system setup in this paper is under the english track. the other two languages are translated to english and only apply a partial sub-system without image sense involvement. under the english situation, the phrase contains the target word and has a length of 2-3, which differs from the traditional wsd task with the sentence as context. this challenges the encoder to extract sufficient semantic information from the target word from the short text. image sense labels require a highly unified image-text connection to produce the final output. for farsi and italian, although the specific language version of clip and multilingual clip are available @xcite @xcite , the actual performance in the current task is limited by the scale of pre-training data. our system makes the best use of the potential knowledge of the pre-trained model by prompting rather than continuous training and fine-tuning. we gather information on the target word from semantic networks and online dictionaries. the most pertinent information is selected by matching the augmented phrase through a biencoder architecture and regarded as the prompt. this biencoder architecture was proposed in bem @xcite ,which jointly trains context and gloss in the same vector space. simcse @xcite involves contrastive loss to maximize the similarity of sentences with identical meanings, which makes it a powerful tool for matching phrases and definitions. therefore, we use simcse as the backbone of our biencoder architecture. the later work @xcite improves the performance of matching text with different lengths. there is a recent multimodal wsd dataset @xcite ) that uses images to represent the sense of the concept. beyond the text information, we use additional images from the open image dataset. different types of words require different modalities to express. therefore, we design a rule-based process that decides whether text prompt or images are used for final sense representation. finally, we use a huge version of clip to extract features and rank the candidate images according to similarity scores. ## system overview the system description in this section is under the english situation. italian and farsi are translated to english and only apply a partial sub-system without image sense involved. for english, we divide the vwsd task into three main modules, including (1) context retrieval module, (2) image retrieval module foot_0 , and (3) matching module, as shown in figure 1 . the two retrieval modules perform disambiguation among senses and are regarded as a wsd-specific modules. ## experimental setup this section will discuss the detailed data usage and experiment setup. as there is a distribution gap between training and testing data, most of the setup is based on the testing and customized validation sets. ## conclusion we describe our prompt and biencoder-based visual wsd system and investigate how the potential knowledge of large-scale pre-trained vlm contributes to disambiguation and modal alignment. the well-designed prompt template connects the input phrase and the potential knowledge of the pretrained model, which also prevents the zero-shot generalization capability. the sense inventory is the fundamental element for wsd tasks, and context quality determines the performance boundary. the involvement of images in the wsd module further extends the coverage of sense interpretation ability. with the combination of text context and image representation, our system achieves a hit rate of 84 and ranks first in the english track. in future work, more research on the modal fusion strategy will be completed, and the automatic fusion approach that applies to the different datasets will be designed. rather than selecting a single modality based on rules, extracting useful information from both text and images could benefit the disambiguation performance. the captions of image labels are regarded as an augmented way to represent image in textual modality. the generated text of ofa @xcite has a high quality and can express most features on the image. for exmaple, the generated text of the bottom right image in figure 4 indicates the specific name of the food and the rack that is mostly hidden. the knowledge reserve and object extraction ability is acceptable. however, the results keep decreasing by fuse a higher weight of captions into the images. two simple approaches are applied, one is add the text clip feature by a ratio to the image feature, in which the result decreases as the fusion ratio increases. another one is fuse the similarities of image labels and its captions with the given phrase, where same results occurs.