TAM of SCNU at S em E val-2023 Task 1: FCLL : A Fine-grained Contrastive Language-Image Learning Model for Cross-language Visual Word Sense Disambiguation (2023)

Abstract: Visual Word Sense Disambiguation (WSD), as a fine-grained image-text retrieval task, aims to identify the images that are relevant to ambiguous target words or phrases. However, the difficulties of limited contextual information and cross-linguistic background knowledge in text processing make this task challenging. To alleviate this issue, we propose a Fine-grained Contrastive Language-Image Learning (FCLL) model, which learns fine-grained image-text knowledge by employing a new fine-grained contrastive learning mechanism and enriches contextual information by establishing relationship between concepts and sentences. In addition, a new multimodal-multilingual knowledge base involving ambiguous target words is constructed for visual WSD. Experiment results on the benchmark datasets from SemEval-2023 Task 1 show that our FCLL ranks at the first in overall evaluation with an average H@1 of 72.56\% and an average MRR of 82.22\%. The results demonstrate that FCLL is effective in inference on fine-grained language-vision knowledge. Source codes and the knowledge base are publicly available at https://github.com/CharlesYang030/FCLL .

## introduction the problem of word polysemy, first recognized in machine translation, is one of challenging tasks in natural language processing (nlp). in the field of nlp, resolving word polysemy is generally regarded as word sense disambiguation (wsd), and it still remains as one of the most challenging and pervasive linguistic phenomena in nlp at present @xcite . to promote the research of the issues, international competitions have been held such as semeval 1 to advance the state-of-theart in word sense analysis and to help create highquality annotated datasets in wsd. since semeval-2023, visual wsd has been introduced and it contains three tracks including english, farsi and italian @xcite . given a target word and some limited textual context, the visual wsd task is to select the one which relevant to the intended meaning of the word among a set of candidate images. theoretically, the general wsd task aims at making explicit the semantics of a word in context by identifying the most suitable meaning (called sense) from a pre-defined sense inventory @xcite , which only involves words and senses. differently, visual wsd requires models to learn more fine-grained image-text knowledge to establish the relationship among words, senses and images. meanwhile, the limitations of visual wsd are significant. because it is difficult in most cases to determine the specific sense of a word from a phrase containing few words without additional contextual information, and even more difficult to further identify the images related to the word. in addition, words in text of different languages may have different senses, and the syntactic and structural differences among languages make models ineffective during textual processing. therefore, cross-language background knowledge is another major challenge in visual wsd. in this paper, we propose a fine-grained contrastive language-image learning (fcll) model for learning fine-grained image-text knowledge. the main contributions of this work lies on three aspects: in vision-language pre-training, contrastive learning @xcite focuses on learning common features between similar instances. clip @xcite employs a classical two-stream structure using more than 400 million image-caption pairs collected from the internet as training data, which is equipped with an outstanding performance for zero-shot. blip @xcite is an extension of clip, learning from noisy image-text pairs by bootstrapping the captions. moreover, flava @xcite ) is an alignment model, which obtains image-text features separately using unimodal encoders, followed by multimodal pre-training. however, most of the existing multimodal pre-trained models use manually managed datasets that are coarse-grained, such as coco @xcite and @xcite . furthermore, these models are studied in the monolingual background. ## conclusion this paper proposes a fcll model for inference on fine-grained image-text knowledge and complementing additional english contextual information to phrases composed by limited words. besides, we demonstrate that our fine-grained contrastive language-image learning approach and sense autocomplementing module support fcll for understanding fine-grained image-text knowledge. moreover, we construct a new multimodal-multilingual fine-grained image-text knowledge base, which can be applied to visual wsd to improve fcll performance. the results on the benchmark test set from semeval-2023 task 1 show that fcll achieves an average h@1 of 72.56% and an average mrr of 82.22%, ranking at the first in overall evaluation. source codes and the knowledge base have been released to facilitate future fine-grained languageimage research.