RCLN at S em E val-2023 Task 1: Leveraging Stable Diffusion and Image Captions for Visual WSD (2023)

Abstract: This paper describes the participation of the RCLN team at the Visual Word Sense Disambiguation task at SemEval 2023. The participation was focused on the use of CLIP as a base model for the matching between text and images with additional information coming from captions generated from images and the generation of images from the prompt text using Stable Diffusion. The results we obtained are not particularly good, but interestingly enough, we were able to improve over the CLIP baseline in Italian by recurring simply to the generated images.

## introduction word sense disambiguation (wsd) is a natural language processing task consisting in the identification of the correct sense of a word having multiple possible meanings, i.e. an ambiguous word, based on the context in which it appears. semeval-2023 task 1 @xcite introduces a wsd task that includes a visual representation of the word meaning, named visual wsd. it consists in determining, out of a set of candidate images, which one is the most pertinent to a given input text. the challenge is that the input text contains an ambiguous word with some of its possible interpretations figured in the set of candidate images. for instance, when presented with the context "andromeda tree" and the focus word "andromeda", a visual wsd system should choose an image of the japanese andromeda plant, rather than an image of the andromeda galaxy, as the former is a better representation of the meaning of "andromeda" in this context. we approached this task from two directions: the first one was to use image captions to compare to the input text; the second one was to use the input text as a prompt to create an image using a generative model. image captioning is the task of describing the content of an image in natural language. various models have been proposed in the past to address this task, from description retrieval to template filling and hand-crafted natural language generation techniques. these techniques have been superseded by modern deep-learning based generative models @xcite . current image captioning models are based on an encoder-decoder architecture. the encoder can be any pre-trained vision transformer, whose purpose is to produce one or multiple one or multiple feature vectors. the decoder is based on a large language model (usually bert or gpt and derivatives) which uses the embeddings produced in the previous step to produce a sequence of words. in this work, we applied the captioning model that combines the vit image transformer @xcite with gpt-2 @xcite as encoder and decoder respectively, proposed in @xcite . in figure 1 we show examples of captions obtained with this model for some of the images in the trial dataset. note that the model in some cases gives a description that only partially matches the content of the picture (e.g. "fruits and vegetables" instead of "melons"). however, we can take advantage of word and sentence embeddings to obtain a good score even if the match is not perfect. for instance, "andromeda tree" has a relatively higher similarity score with the image depicting flowers, as flowers and tree are close in the embedding space. image generation from textual prompts has been the object of increasing attention recently. in particular, models such as dall-e 1 and stable diffusion 2 have captured the attention of researchers and the general public for their ability to produce realistic images with short textual inputs. for this work we used stable diffusion given its availability in the huggingface repository 3 . stable diffusion itself is based on an encoder-decoder architecture in "a flower arrangement in a flower pot in a garden", 0.173 "a lake with a river and a bench", 0.086 "a variety of fruits and vegetables on a table", 0.135 which the encoding phase consists in adding noise to the image while the decoding step learns to remove the noise. the decoding is guided by the text given as input to the model ("prompt"), which conditions the final output thanks to a cross-attention mechanism in place between a text transformer and the denoising module. in the remainder of this article we describe first the initial experiments we carried out with the trial data and subsequently the models we used for the submission to the task, with some analysis of the errors we identified. ## methodology our first idea was to use textual embeddings obtained from the captions of the images via sentence-bert (sbert) @xcite and compare them to the sbert embeddings of the context sentence. as it can be noted from figure 1 , even if the text "andromeda tree" is not present from the captions, it could be possible to match "andromeda tree" with the right image by taking the one with the highest similarity score. we tested this idea on the trial collection, applying the vit-gpt2 image captioning model to the trial candidate images, and subsequently applying the sbert minilm-l6-v2 model to transform the image caption and the context query into embeddings. with this setup, we obtained an mrr of 0.575, inferior to the clip baseline (mrr 0.734). unfortunately, the results are affected both by errors in the captioning and the fact that sbert model sometimes provides lower scores for the right images as the contexts are too short. the second experiment was to use the stable diffusion model on the query to produce an image and compare this image to the candidate images. to obtain an embedding for the images we applied the same clip model and calculated similarity using cosine measure. with this setup, we obtained a mrr of 0.580. given these preliminary results, it was clear that these 0-shot approaches from pre-trained models were inferior to clip alone. with the availability of training data, we planned to build a neural network to fine-tune the sbert and stable diffusion models on them, also in combination with clip. inspired by the model used for sentence bert, we built a siamese network where the vectors coming from the target sentence and the candidate images are encoded in the same way, following a fully connected network with a first layer of 256 units and a second one of 512 units. a dropout with probability 0.2 is in place between the two fully connected layers. we tested in the preliminary phase different activation functions, in particular leakyrelu and tanh, without any significant differences. the loss function for the siamese network is a cosine embedding loss with margin 0.8 (we tested margins of 0.2, 0.5 and 0.8 on trial data, with the last one achieving the best results). the cosine embedding loss is defined as: ## results and discussion we submitted two runs with each of the models above for english and we submitted only the diffusion model run for italian. this was due to the fact that, given that no reliable captioning model was available at the time of participation, the only possibility to apply the captions+clip model was to translate the contexts into english. this option seemed to potentially introduce more noise into the model and we discarded it. the obtained results are shown in we took a more detailed look into what happened with the diffusion model in the italian dataset. we were able to identify some situations in which stable diffusion was not able to produce a good representation of the original text input. please refer to figure 4 for the visual examples mentioned in the list. some of these results depend on the way stable diffusion is trained. in particular, the self-attention mechanism may have been misled by giving too much weight to the ambiguous parts of the context. for instance, "diagonal" is what is important to focus on to disambiguate "alfiere" (bishop). instead, it looks like it's defaulting to the most common sense of the target word. the last error (no.5 in the above list) seems a problem of the transfomer tokenizer which is unable to differentiate "box" from "boxe", the term commonly used in italian to refer to the sport. ## conclusions although our participation was unsuccessful in terms of results, we were able to learn something interesting regarding the diffusion models, which seem particularly affected by the ambiguity problem, yielding bad example images even when the context in the input text should be enough. probably a good strategy would have been to use the training data to improve the diffusion models instead of fine-tuning the clip embeddings obtained from the images. the code used for this participation is available at the following address: https:// github.com/dbuscaldi/visualwsd23 . the automatically generated data (captions, generated images) are stored on zenodo at the following address: https://doi.org/10.5281/zenodo.7860213 .