U niversity of H ildesheim at S em E val-2023 Task 1: Combining Pre-trained Multimodal and Generative Models for Image Disambiguation (2023)

Abstract: Multimodal ambiguity is a challenge for understanding text and images. Large pre-trained models have reached a high level of quality already. This paper presents an implementation for solving a image disambiguation task relying solely on the knowledge captured in multimodal and language models. Within the task 1 of SemEval 2023 (Visual Word Sense Disambiguation), this approach managed to achieve an MRR of 0.738 using CLIP-Large and the OPT model for generating text. Applying a generative model to create more text given a phrase with an ambiguous word leads to an improvement of our results. The performance gain from a bigger language model is larger than the performance gain from using the lager CLIP model.

## introduction ambiguity is one of the big challenges for natural language processing (nlp) and can often be solved by context. in a multimodal setting, the ambiguity can be solved by identifying the correct visual information. this is the goal of the system implemented in this work. the goal of task 1: visual word sense disambiguation (v-wsd) is to find the most similar image among ten candidate images based on textual input (see task overview @xcite ). the scientific aim is the development of technology for disambiguation in a multimodal context. the textual input consisted of a: the ambiguous target word, and b: an additional phrase that clarified the intended meaning of the target word. some of the candidate images share visual similarities, which increases the importance of a precise representation of the model. the task was split into three different languages: english, farsi, and italian. the training dataset consists of nearly 13,000 training examples, where each includes an ambiguous word, an additional phrase, and ten candidate images. the correct image was located in a separate file. the test dataset has a similar structure, including 8,100 images for 463 examples @xcite . for this work, we focused only on the english dataset. our approach requires no training and uses only pre-trained models. the progress in large scale language as well as multimodal models in recent years has been rapid. these models capture the semantics of language and images and allow many applications @xcite . the main focus of our approach aims at the reduction of the ambiguity of the textual input by using "large" generative language models to create more text when given the input phrases. we used four different text inputs, including the phrase, the ambiguous word, the phrase without the ambiguous word, and the input created by the generative model. by using the multimodal model clip @xcite , we mapped these texts into the joint embedding space. then we used the closest image from the given set to identify the correct candidate image. ## background word sense disambiguation is an important task in natural language processing. it typically involves the identification of a word in a given context and requires the selection of the correct sense given several options @xcite . textual ambiguity has been approached from several different perspectives and with various goals. in information retrieval, the ambiguity of a query can lead to unsatisfactory results for users @xcite . ambiguity is also very frequent in technical domains @xcite . even in patents, it is widespread, as a large-scale analysis showed @xcite . ambiguity is often used for expressing humor but also to express hate by exploiting multimodal semantics @xcite . visual word sense disambiguation can be seen from different angles. an image could indicate the correct sense for a word. in the visual word sense disambiguation, several senses of a word are represented by images. the correct image is indicated by an additional context word for the ambiguous word @xcite . progress in large scale multimodal models facilitates visual word sense disambiguation. modern multimodal models are able to combine different input modalities and perform a variety of tasks. by mapping input from different modalities into a joint embedding space, such models create a bridge between a visual and textual representation @xcite . these multimodal models can be used for a variety of applications. these include visual question answering, image generation from text (e.g. dall-e) and image to caption generation @xcite . besides clip, there are other models like google align @xcite and microsoft unicl @xcite . they differ in their training data or procedures as well as in their architecture. one big obstacle often mentioned for supervised multimodal models is the availability and quality of natural language training data. datasets had to be heavily curated @xcite or must be extremely large when relying on noisy text image relations. for example, for google align, over 1.8 billion examples were used @xcite to generate useful joint embeddings in which visual and textual representations can be aligned. ope-nai's clip is the oldest but most renown model in this field, released in february 2021. shortly after, google's align model was released in may 2021. both models build pairs of text inputs and images to gain joint representations. microsoft's unicl from april 2022 combines a third layer by also using labeled images, which are used e.g. for image classification, to gain a more robust image representation @xcite . we decided to use clip as the basis for our approach because various model sizes are made available with the capability to modify inputs and outputs rather easily. ## system overview the key components used in our approach are clip and the text generation frameworks gpt-2 and opt. the generated text is additionally used in clip to disambiguate the given terms since only a few words were provided by the task. we assume that text generation models are powerful at providing additional text. by generating an additional short paragraph of text based on the ambiguous target word and the provided phrase, we exploit the knowledge included in the language generation models. the objective of this approach is that the generative model solves the ambiguity and adds more words for the correct semantics. we used the clip model, which contains extensive knowledge due to the pre-training on the relationship between text and visual information. we applied the visual transformer, which was pretrained on imagenet. again, we assume that the pretrained model contains the relevant knowledge. the processing pipeline for our approach contains the following stages. the pipeline is configured in various settings that particularly use multiple textual inputs. overall, four text inputs are made available within our approach. the following list shows them with an example in italics. ## experimental setup we implemented several configurations to examine the effect of using various textual inputs. our different experiment setups are displayed in table id models inputs sequ. leng. penalty cb-1 clip-b target --cb-2 clip-b target, phrase, second word --cb-3 clip-b, gpt2 target, phrase, second word, generated text 50 -cb-4 clip-b, opt target, phrase, second word, generated text 40 1.0 cb-5 clip-b, opt generated text 40 1.0 cl-1 clip-l target --cl-2 clip-l target, phrase, second word --cl-3 clip-l, opt target, phrase, second word, generated text 40 1.0 cl-4 clip-l, opt generated text 40 1.0 cl-5 clip-l, opt target, phrase, second word, generated text 60 1.5 cl-6 clip-l, opt generated text 60 1.5 cl-7 clip-l, opt target, phrase, second word, generated text 70 2.0 cl-8 clip-l, opt generated text 70 2.0 since the clip model is trained on sequences of words as inputs and not only keywords, we conducted separate runs with only the generated text as input and the generated text, target word, additional phrase, and the non-ambiguous word of the additional phrase. the generative models all used the same seed set for better comparison when possible. the text generation models we initially used were gpt-2 and later opt. we used the gpt-2 model with 355 million parameters and the opt model with 2.7 billion parameters. we additionally included two model-specific configurations to observe: the maximum number of tokens to generate and the penalty for repetition. the maximal sequence length was thus set to 40, 60, and finally 70 tokens. to fit all of our inputs into clip, we had to limit the text generation to a maximum of 70 tokens, since the default limit for clip is 77 tokens. opt has the possibility of penalizing repetition within a generation. by increasing the penalty, the model is forced to write more diverse sentences. when inferring with multiple textual inputs, every input creates a softmax distribution for the ten candidate images. by averaging the results over all four inputs, we created the final ranking for the candidates. during the processing, some issues were observed. a few images in the training dataset are corrupted in some way, mainly truncated. we imported truncated images as they are, which might affect the model's results. text has to be imported with the right encoding since the data includes letters from different languages like chinese, arabic, latin, and emojis. the metrics proposed by the task organizers are used to evaluate the performance of the experiments. the first metric is the hit rate. it measures the correctness of the top-1 prediction. the second metric is the mean reciprocal rank (mrr). it is a measurement that reciprocally considers the correct ranking position across all results. ## results overall, our approach reached the 77th rank out of 98 submissions within the task. it was ranked 55th in the rankings for only english, which we focused on. our pipeline achieved robust performance in zero-shot settings, that is, without training. when compared to the leader board in english only, our approach is positioned one rank below the baseline (bl in in general, the results reveal significant differences between the performance of our approach on the two sets, although we performed no specific training. this tendency is observed regardless of the inclusion of generated text. in accordance with the reported clip results in @xcite , robust zero-shot performance can be observed from the training set. the best performance is seen from cl-5 with a hit rate of 83.2%. comparing these results to the available training phase results on the competition page of the task, this would have been in the top three results out of 26 submissions 1 . however, a much lower performance can be noticed on the final test set. the hit rate of 57.3% is seen from cl-3. other submissions had some very good results on the test data, like samsung research china with 84% hit rate. specific optimization for the task can increase performance. the results also reveal that additional textual inputs do improve the disambiguation performance 1 https://codalab.lisn.upsaclay.fr/ competitions/8190#results of the model. in particular, the use of averaged embeddings of multiple text inputs tends to show performance improvements. however, the text that is generated with a greater repetition penalty and number of tokens seems to make the model lose focus on the target term's semantics. further improvements are noticed when the averaged embeddings are used in a larger clip model. interestingly, the improvements are clearly identified only from the training results. finally, the use of larger language generative models improved the disambiguation ability by a small margin. the effect of using different language models for generation is noticed by comparing the results of cb-3 and cb-4. a slight performance improvements are identified from both train and test sets. the opt model, which consists of a much larger number of parameters (2.7 b) than gpt-2 (355 m), is considered suitable for target term disambiguation based on its extensive knowledge of word semantics. the third entry of table a1 shows a more abstract example. the given input phrase is "lift, raising" and the target image displays a person lifting another person. since this describes an activity, it is very hard for our approach to put this input in the correct context, and defaults to the most common example. an image captioning model like blip might be able to describe the activity in the scene and give more valuable insights. the best results for the test data were produced with a shorter generated sequence of 40 tokens and no repetition penalty (1.0). even though the low repetition penalty model scored the best on the test dataset, the generated text can be misleading for the model's prediction. in the globe example of table a1 and table a2 the model predicted the wrong candidate image. this is likely due to the additional attribute "wood" in the generator text. the higher penalized model instead includes the part "[. . . ] a spherical object that represents earth in space." such an explanation of a globe with a focus on visual aspects can direct the model to the correct answer. as the prediction accuracy drops with a higher penalty, as earlier described, this could indicate that the more complexly generated texts are misleading in some cases. it can be concluded that the text generation models provide more knowledge for ambiguous words than clip itself. they are able to put the ambiguous word into the right context and, as a result, improve overall prediction accuracy. ## conclusion this paper showed a robust approach for image disambiguation relying only on pre-trained models and without any training. in future work, we intend to linearly weigh the four text inputs differently and find optimal settings through a grid search. we expect that such a straightforward approach, which was applied similarly for other cases with several text inputs @xcite can improve the performance.