ML Mob at S em E val-2023 Task 1: Probing CLIP on Visual Word-Sense Disambiguation (2023)

Abstract: Successful word sense disambiguation (WSD)is a fundamental element of natural languageunderstanding. As part of SemEval-2023 Task1, we investigate WSD in a multimodal setting,where ambiguous words are to be matched withcandidate images representing word senses. Wecompare multiple systems based on pre-trainedCLIP models. In our experiments, we findCLIP to have solid zero-shot performance onmonolingual and multilingual data. By em-ploying different fine-tuning techniques, we areable to further enhance performance. However,transferring knowledge between data distribu-tions proves to be more challenging.

## introduction determining the correct meaning of an ambiguous word in a given textual context is a fundamental element of successful understanding of natural language. formulated in the task of word sense disambiguation (wsd), it remains a key challenge of natural language processing (nlp). recent successful approaches tackling this task often leverage large pre-trained language models @xcite . with the rise of large pre-trained vision-and-language (v&l) models such as clip @xcite , the question arises how well these models can tackle wsd in a multimodal setting. towards answering this question, task 1 of semeval-2023 proposes a visual wsd task @xcite , combining ambiguous contextualized words with images encoding different word senses. instead of selecting the correct word sense from a sense inventory, a system now has to retrieve the best matching image from a set of candidate images, thereby bridging the gap between vision and language. in this paper, we present our contributions to the semeval-2023 visual-wsd task. our work focuses on empirically comparing the wsd performance of multiple clip models in different transfer learning settings. this includes the investigation of zero-shot transfer and adapters @xcite as a parameter-efficient means of transfer learning. we especially aim to highlight to which degree simple techniques are sufficient for successful task performance. our best submitted system ranks 9th out of 54 on the official task leaderboard (counting only one submission per person). ## methodology while @xcite show that clip has strong zero-shot capabilities on downstream tasks, subsequent work such as clip-vil @xcite incorporates the frozen clip vision encoder into existing architectures for various visionlanguage tasks. we directly leverage both pretrained clip encoders for transfer to the downstream task. as the formulation of the visual wsd task is similar to the pre-training task of clip, our fine-tuning objective closely follows clip's pretraining objective. instead of passing n imagetext pairs as done during pre-training, our input consists of one (w, c) pair alongside 10 images. the symmetric cross entropy loss employed during pre-training therefore collapses to a regular cross entropy loss (ce) over the candidate images. in simplified form, the full fine-tuning objective therefore can be formulated as: