P ol CLIP : A Unified Image-Text Word Sense Disambiguation Model via Generating Multimodal Complementary Representations (2024)

Abstract: Word sense disambiguation (WSD) can be viewed as two subtasks: textual word sense disambiguation (Textual-WSD) and visual word sense disambiguation (Visual-WSD). They aim to identify the most semantically relevant senses or images to a given context containing ambiguous target words. However, existing WSD models seldom address these two subtasks jointly due to lack of images in Textual-WSD datasets or lack of senses in Visual-WSD datasets. To bridge this gap, we propose PolCLIP, a unified image-text WSD model. By employing an image-text complementarity strategy, it not only simulates stable diffusion models to generate implicit visual representations for word senses but also simulates image captioning models to provide implicit textual representations for images. Additionally, a disambiguation-oriented image-sense dataset is constructed for the training objective of learning multimodal polysemy representations. To the best of our knowledge, PolCLIP is the first model that can cope with both Textual-WSD and Visual-WSD. Extensive experimental results on benchmarks demonstrate the effectiveness of our method, achieving a 2.53% F1-score increase over the state-of-the-art models on Textual-WSD and a 2.22% HR@1 improvement on Visual-WSD.

## introduction understanding and identifying the intended meaning of words with multiple senses (i.e., polysemy) is a significant challenge in natural language processing @xcite . this promotes in-depth research on word sense disambiguation (wsd), which has recently been extended to multimodal downstream tasks @xcite . techniques for wsd are critical for enhancing the accuracy and effectiveness of text understanding and * corresponding author. information retrieval tasks such as machine translation @xcite , image-text retrieval @xcite , and large language model inference @xcite . theoretically, wsd can be divided into two subtasks: textual word sense disambiguation (textual-wsd) @xcite and visual word sense disambiguation (visual-wsd) @xcite . given a context containing an ambiguous target word, the goal of textual-wsd is to select the most semantically appropriate one from a set of candidate senses, while the goal of visual-wsd is to choose the most semantically suitable one from a set of candidate images. due to the distinct modalities, these two subtasks typically require specialized training datasets and methods @xcite @xcite . nevertheless, they can be unified as a multimodal-wsd task if the senses and images in existing wsd datasets are aligned, as shown in figure 1 . the task objectives of multimodal-wsd cover: 1) in cases where candidates include both senses and images, a multimodal-wsd model is required to identify the most semantically relevant senses and images simultaneously for ambiguous target words within a given context; 2) in cases where candidates include either senses or images, the model is still required to identify the most semantically relevant senses or images for ambiguous target words within a given context. such task objectives are also in line with the practical scenarios of current large visionlanguage models @xcite . technically, developing a generic multimodal-wsd model can realize the unification of wsd tasks and activate the potential of multimodal applications in understanding polysemy knowledge. in the textual-wsd datasets @xcite , merely textual senses serve as candidates (i.e., image-missing), while in the visual-wsd datasets @xcite , only images serve as candidates (i.e., sense-missing). this results in the challenge of modality missing at the data level, limiting the unification of these two wsd subtasks. furthermore, multimodal representations have been demonstrated to carry richer semantic information compared to unimodal representations in recent wsd works @xcite . however, constrained by model architecture, existing textual-wsd models @xcite @xcite cannot supplement candidate senses with image information, and visual-wsd models @xcite @xcite cannot supplement candidate images with descriptions. this poses technical difficulties in developing a unified framework for multimodal-wsd. to address these issues, we propose polclip, a unified image-text wsd model which is proficient in multimodal polysemy processing and is built upon clip @xcite architecture. by employing an image-text complementarity strategy, it simulates stable diffusion models @xcite (generating images based on texts) and image captioning models @xcite (generating descriptions based on images). the core idea of this strategy is to make polclip initially focus on the key information of original unimodal senses or images, and then re-utilize the text or image encoder to generate implicit imagetext complementary representations. two widely used wsd datasets (semcor @xcite and @xcite ) are integrated into a disambiguation-oriented image-sense dataset for the training objective of learning aligned multimodal representations. moreover, a fine-tuned gpt-3.5 model is utilized to generate lexical definitions for semantic enhancement in testing phase. the main contributions of this work are summarized as follows: ## related work textual-wsd was mainly tackled by knowledgebased methods and supervised methods @xcite . knowledge-based methods @xcite typically used external dictionary resources to provide sense lists for ambiguous words to resolve polysemy. supervised methods @xcite generally used pre-training language models to maximize the similarity probabilities between contexts and candidate senses in a feature space. bem @xcite and sace @xcite adopted biencoders and only retained the representations corresponding to ambiguous words. they achieved state-of-the-art results on english all-words benchmarks at that time. moreover, the full utilization of visual features for verb sense disambiguation has attracted increasing interest @xcite . evilbert @xcite obtained better results by learning task-agnostic multimodal sense representations, compared to methods built solely on language models. although these methods primarily leveraged visual information to bolster performance on textual-wsd, they could not be applied to visual-wsd straightforwardly. visual-wsd was introduced in semeval-2023 task 1 @xcite . the visual-wsd mainstream approaches employed vision-language pre-training models (vlps) for imagetext retrieval. fcll @xcite proposed a fine-grained image-text contrastive learning mechanism and won first place in semeval-2023 task 1. moreover, large language models (llms) were widely used to enrich the semantic information of contexts @xcite . calling apis was a commonly adopted strategy, where simple prompts were designed to guide llms to return the interpretations of ambiguous target words in contexts @xcite . however, visual-wsd models depended on the prior knowledge of vlps, which were pretrained with objectives biased towards image-text understanding rather than wsd. this resulted in visual-wsd models struggling to effectively resolve textual-wsd. data was a key factor to unify these two wsd subtasks and develop a generic multimodal-wsd model. wordnet @xcite ) was a large lexicographic database and a standard inventory for english wsd. it contained approximately 120,000 synsets. babelnet @xcite was the most popular multilingual dictionary, which was semi-automatically mapped to other resources to acquire encyclopedic terms. it covered over 500 languages and was upgraded to version 5.3 recently. researchers could access various possible resources about ambiguous words by babelnet, including example sentences, parts of speech, textual senses, and images. by linking wordnet with wikipedia through babel-net, babelpic @xcite expanded non-concrete image-sense pairs, paving the way for our work to construct larger disambiguationoriented multimodal datasets. inspired by fcll, we used a dual-stream architecture with contrastive learning to align the knowledge of word-sense pairs and word-image pairs from the proposed disambiguation-oriented dataset. previous research viewed llms as common tools for text semantic enhancement. thus, llms were also adopted in our testing phase.