S zeged AI at S em E val-2023 Task 1: Applying Quasi-Symbolic Representations in Visual Word Sense Disambiguation (2023)

Abstract: In this paper, we introduce our submission in the task of visual word sense disambiguation (vWSD). Our proposed solution operates by deriving quasi-symbolic semantic categories from the hidden representations of multi-modal text-image encoders. Our results are mixed, as we manage to achieve a substantial boost in performance when evaluating on a validation set, however, we experienced detrimental effects during evaluation on the actual test set. Our positive results on the validation set confirms the validity of the quasi-symbolic features, whereas our results on the test set revealed that the proposed technique was not able to cope with the sufficiently different distribution of the test data.

## introduction multi-modal text-image encoders, such as clip @xcite have a demonstrated capability of performing zero-shot image classification tasks over diverse benchmark datasets. the contrastive learning objective applied during the pretraining of such models make them a natural choice for the v-wsd shared task @xcite , where the goal is to choose from a set of candidate images the one which suits the best some query expression including a potentially ambiguous term. that is, the shared task problem can be framed as an image retrieval task, for where the transfer learning capabilities of clip can be arguably exploited. the kind of continuous hidden representations produced by neural models, including clip, are limited by their lack of translucency from a human perspective. our earlier work demonstrated that by using an unsupervised technique relying on sparse coding, it becomes possible to connect static word embedding representations with human interpretable concepts and properties @xcite . in @xcite , we successfully employed these representations for hypernym discovery @xcite , surpassing the performance obtained by relying on traditional and less interpretable dense word embeddings. in @xcite , we demonstrated that the technique can be extended to contextualized word representations, substantially improving the performance in classic unimodal, text-only word sense disambiguation. our approach for inducing implicit semantic information to words within their context built on the observation that performing sparse coding over the contextual representations of pre-trained language models result in representations that align well with semantic categories. with our shared task participation, our goal was to investigate if similar benefits that we saw for the classic uni-modal, textonly wsd task can be observed for multi-modal clip-based representations in v-wsd. our experiments delivered mixed results, as the sparse multi-modal representations provided substantial performance gain during the development phase, when evaluation was carried out on a heldout part of the training set, but the same quasisymbolic features proved to be detrimental for the unseen test set with a distribution shift relative to the training set. ## our approach we first introduce our notation ( §2.1), then describe our methodology for determining multimodal quasi-symbolic representations of texts and images ( §2.2). finally, we reveal how we incorporated those during ranking ( §2.3). ## experiments in the following subsections, we first introduce the datasets ( §3.1) and report our quantitative and qualitative results ( §3.2 and 3.3). ## conclusion in this paper, we have investigated the use of quasi-symbolic representations derived from the latent representations produced by multi-modal, text-image encoders. our qualitative evaluation suggests that the individual latent representations that we derived from the original representations have translucency from a human point of view. additionally, the features that we extracted from these quasi-symbolic representations were helpful in boosting v-wsd performance when the distribution of the inputs was closer to that of the data that we used for training our ranking module with. on the negative side, our test scores dropped substantially compared to the ones we observed on the development data. the test set performance could be improved if more diverse training data was used, perhaps, from alternative sources such as ms coco @xcite or gcc @xcite , something we consider as a potential future research direction. our source code for replicating our experiments can be accessed via https://github.com/szegedai/vwsd/ .