Rutgers Multimedia Image Processing Lab at S em E val-2023 Task-1: Text-Augmentation-based Approach for Visual Word Sense Disambiguation (2023)

Abstract: This paper describes our system used in SemEval-2023 Task-1: Visual Word Sense Disambiguation (VWSD). The VWSD task is to identify the correct image that corresponds to an ambiguous target word given limited textual context. To reduce word ambiguity and enhance image selection, we proposed several text augmentation techniques, such as prompting, WordNet synonyms, and text generation. We experimented with different vision-language pre-trained models to capture the joint features of the augmented text and image. Our approach achieved the best performance using a combination of GPT-3 text generation and the CLIP model. On the multilingual test sets, our system achieved an average hit rate (at top-1) of 51.11 and a mean reciprocal rank of 65.69.

## introduction polysemous words are common in human language. these words are ambiguous and can be interpreted in variant ways under different contexts. although it is easy for humans to distinguish different word senses, machines need to transform the word senses into data structure and analyze the differences. word sense disambiguation (wsd) is a task for machines to identify the meaning of words given limited text contexts @xcite . wsd is widely used in different nlp tasks, including information retrieval, machine translation, information extraction, content analysis, and lexicography. with the rapid growth of the multimodal datasets, wsd task has expanded from language to the visual field, which aims to improve the tasks such as image description, visual question answering, object detection, and image retrieval @xcite @xcite . for example, when we search for an image of "andromeda tree" with an ambiguous word "andromeda", the search engine needs to identify the meaning of "andromeda" under the context "tree", i.e., a species of plant instead of the greek goddess or the galaxy. the semeval-2023 task-1 describes this task as the visual word sense disambiguation task (vwsd): given an ambiguous target word with limited textual context, select among a set of candidate images the one which corresponds to the intended meaning of the target word @xcite . word embedding has been widely used to represent words in numerical vectors for machines to interpret the word sense @xcite . a word is transformed into high-dimensional vector space where each dimension represents a unique feature of the word. word embeddings can capture the semantic and syntactic relations between words, e.g., words with similar meanings will be closer in the space, and antonyms words will be orthogonal to each other. however, word embeddings have a limitation that polysemous words with multiple meanings cannot be captured by a single vector. embedding the words with context is important for a model to identify the specific sense of the word @xcite . extending to the vision-language domain, the vision-language pre-trained (vlp) models aim to represent the joint features for both text and image. the textual and visual information is transformed into a shared feature space where the text and image with similar meanings are close to each other. existing vlp models were usually pre-trained on large-scale corpus of image-text pairs where the text data is usually a sentence that describes the image @xcite . to find the best matching image-text pairs for the vwsd task, we propose an approach that extends the context of the target word using different text-augmentation methods, such as prompting @xcite , the wordnet synonyms set @xcite , and text generation @xcite . for image selection, we experimented with different vlp models to extract the features for the augmented text and the candidate images @xcite ., augme nted text describe "andromeda tree" in one sentence: this is andromeda tree. andromeda generative model any of several shrubs of … broad-leaved evergreen … (greek mythology) an … a constellation in … vector the andromeda tree is an evergreen shrub with small, fragrant, white flowers that have diamondshaped leaves and silvery-gray bark. vector 1 vector 2 vector 3 vector 4 prompting wordnet synonyms set text generation this is a picture of andromeda tree. example of andromeda tree. … language model wordnet vision language pre-trained model cos sim andromeda tree img emb 1 vlp model text emb vlp model img emb 2 img emb 3 img emb 4 img emb 5 img emb 6 img emb 7 img emb 8 img emb 9 img emb 10 similarity 1 similarity 2 similarity 3 similarity 4 similarity 5 similarity 6 similarity 7 similarity 8 similarity 9 similarity 10 language model @xcite . the image corresponding to the target word is selected based on the similarity between the embeddings of the augmented text and the images. our contributions are: ## system overview our proposed approach consists of two parts (figure 1 ): (1) text augmentation and (2) image selection. here we describe our text augmentation approaches and the vlp models we used in our experiments. ## conclusion we introduced a text-augmentation based approach for semeval-2023 task-1 on visual word sense disambiguation. we explored different text augmentation methods such as prompting, the wordnet synonyms set, and text generation. we experimented with different vision-language pre-trained models using zero-shot learning. our system achieved the best performance with the combination of gpt-3 text generation and the clip model. although our team only experimented with the text-augmentation idea, we believe other ideas may also be effective for vwsd task, such as (1) imageaugmentation: using the state-of-the-art image generative models to generate images given the target phrase, and calculate the similarity between the generated image and candidate images @xcite ; (2) image captioning: using pre-trained image captioning model to generate descriptions of the candidate images, and calculate the similarity between the descriptions and the target phrase @xcite ; (3) ensemble model: assemble different vlp models to boost the performance from different models.