ECNU _ MIV at S em E val-2023 Task 1: CTIM - Contrastive Text-Image Model for Multilingual Visual Word Sense Disambiguation (2023)

Abstract: Our team focuses on the multimodal domain of images and texts, we propose a model that can learn the matching relationship between text-image pairs by contrastive learning. More specifically, We train the model from the labeled data provided by the official organizer, after pre-training, texts are used to reference learned visual concepts enabling visual word sense disambiguation tasks. In addition, the top results our teams get have been released showing the effectiveness of our solution.

## introduction word sense disambiguation (wsd) consists of associating words in context with their most suitable entry in a predefined sense inventory, which is a historical and hot point in natural language processing and artificial intelligence. supervised @xcite and knowledgebased @xcite are two common research methods that are widely used. however, visual word sense disambiguation aims to select among a set of candidate images the one which corresponds to the intended meaning of the target ambiguous word, which is completely different from the previous work @xcite . in the real world, information often exists in different modalities, therefore, the fusion of multimodal information plays an important role in the research of deep learning. this task is also useful for vision-and-language tasks @xcite and text-to-image generator @xcite . in this paper, we focus on the computing of similarity between text and image track. as shown in figure 1 , given the ambiguous word "andromeda" and some limited textual context "andromeda tree", select one of the best corresponding images from candidate pictures to match the semantics according to the similarity. apart from that, it is worth noting that this task includes datasets in three languages: english, farsi, and italian. in recent years, pre-training methods have revolutionized nlp and achieved state-of-the-art performance in many fields @xcite @xcite . to solve the above problems effectively, we use a selfsupervised pre-training method, the text is used as a supervisory signal to train a better visual model @xcite . for the first, we use an image encoder and text encoder to extract feature representations of each modality separately. the second is joint multimodal embedding and calculating cosine similarities. the experimental results show that our approach has achieved competitive results.