CM _ CLIP : Unveiling Code-Mixed Multimodal Learning with Cross-Lingual CLIP Adaptations (2024)

Abstract: In this paper, we present CMCLIP, a Code-Mixed Contrastive Linked Image Pre-trained model, an innovative extension of the widely recognized CLIP model. Our work adapts the CLIP framework to the code-mixed environment through a novel cross-lingual teacher training methodology. Building on the strengths of CLIP, we introduce the first code-mixed pre-trained text-and-vision model, CMCLIP, specifically designed for Hindi-English code-mixed multimodal language settings. The model is developed in two variants: CMCLIP-RB, based on ResNet, and CMCLIP-VX, based on ViT, both of which adapt the original CLIP model to suit code-mixed data. We also introduce a large, novel dataset called Parallel Hybrid Multimodal Code-mixed Hinglish (PHMCH), which forms the foundation for teacher training. The CMCLIP models are evaluated on various downstream tasks, including code-mixed Image-Text Retrieval (ITR) and classification tasks, such as humor and sarcasm detection, using a code-mixed meme dataset. Our experimental results demonstrate that CMCLIP outperforms existing models, such as M3P and multilingual-CLIP, establishing state-of-the-art performance for code-mixed multimodal tasks. We would also like to assert that although our data and frameworks are on Hindi-English code-mix, they can be extended to any other code-mixed language settings.

## introduction in recent years, large pre-trained transformer-based language models (plms), such as bert @xcite , roberta @xcite , and gpt @xcite , have revolutionized the field of natural language processing (nlp), achieving state-of-the-art performance across a wide range of tasks. this progress marks a significant shift towards a paradigm where general linguistic understanding is first learned from large-scale unsupervised data, followed by fine-tuning for task-specific applications @xcite . these pre-training techniques, which derive representations directly from raw text, have fundamentally transformed the landscape of nlp, enabling models to perform exceptionally well on both traditional and emerging challenges in the field. initially, monolingual pre-trained language models like bert @xcite , gpt-2 @xcite , and roberta @xcite laid the groundwork for subsequent expansions into multilingual models such as m-bert @xcite , unicoder @xcite , and xlm/xlm-r @xcite . in parallel, there has been a growing interest in pre-trained vision-and-language representations, as demonstrated by models like uniter @xcite , ernie-vil @xcite , unicoder-vl @xcite , oscar @xcite , villa @xcite , and m3p @xcite . these multimodal pre-trained models have made remarkable strides in aligning natural language and visual data, achieving state-of-the-art results in tasks such as image retrieval, text-to-image generation, image captioning, and visual quality assessment. however, extending multimodal models to multilingual and code-mixed scenarios remains a challenge despite these advancements @xcite . cross-lingual alignment in such models often involves adapting the multimodal representations learned from english corpora to other languages. recent studies have shown that multilingual textual representations often fail to learn equally high-quality representations across all languages, especially for low-resource languages and code-mixed languages @xcite . code-mixed languages, in particular, have been significantly underrepresented in such models. while multimodal pretrained models like clip @xcite have demonstrated their ability to bridge vision and language tasks, their application in code-mixed settings has been limited. clip, trained using a contrastive learning objective, maximizes the similarity between corresponding image-text pairs. it measures the cosine similarity between image and text features extracted from their respective encoders, using this similarity to quantify the log odds of related pairs. while multilingual-clip @xcite attempts to address this issue by replacing the english text encoder with a pre-trained multilingual language model, such as m-bert @xcite , the handling of code-mixed languages still remains inadequate. as social media platforms have gained widespread popularity, the prevalence of codemixed language, particularly in informal communication, has surged @xcite . despite a large section of the global population using code-mixed languages, the development of pre-trained models for such data has been hindered by the scarcity of parallel data and the complexity of the language structures. existing multimodal models are typically trained on high-resource languages, making them unsuitable for code-mixed languages, which involve more complex linguistic phenomena, including free word order and morphological diversity @xcite . although some previous works have attempted to create synthetic datasets, they primarily rely on formal sources such as news articles, which rarely capture the codemixing found in everyday speech and online communication. to the best of our knowledge, no prior work has focused on creating a pre-trained vision-andlanguage model for code-mixed languages. this lack of attention can be attributed to several challenges, including the scarcity of annotated resources, the linguistic complexity of code-mixed languages, and the difficulty of aligning multimodal data in low-resource settings. translating data from high-resource languages like english to code-mixed formats (e.g., hindi-english) could mitigate the data scarcity problem. leveraging transfer learning from state-of-the-art english models can enhance performance on various code-mixed tasks @xcite . with the rapid advancements in machine translation (mt), there is now potential to bridge these gaps in low-resource, code-mixed language settings. in this paper, we build upon the work of @xcite , which demonstrated an effective way to incorporate new languages into pre-trained vision-and-language models such as clip. we extend this work to code-mixed languages, focusing on hindi-english as a case study but laying the groundwork for broader application to other codemixed languages. we introduce cmclip, a vision-and-language model that explicitly aligns images with codemixed text representations. our model enforces the learning of universal representations, mapping both images and code-mixed text into a joint semantic space. the training process of our model is based on two key principles (c.f. figure 1 ): (i) a translated code-mixed sentence should occupy the same vector space as the original english sentence, and (ii) a translated code-mixed sentence should occupy the same vector space as its corresponding image in cross-lingual scenarios. in summary, the main contributions of this paper are as follows: ## dataset we create a parallel corpus of english and codemixed [hindi-english] sentences in order to train our model. large part of this data also has parallel tagged images to exercise multimodal evaluation after training. for our downstream application evaluation, we also create another independent multimodal meme dataset. ## methodology teacher learning @xcite ) is a method to train a student model under the supervision of a teacher model. in recent years, it has become a powerful process to distill knowledge from teacher model a to student model b with low resource data and with minimal infra requirement. clip has two different encoders for vision and text, which are aligned in the same vector space for a context in both modalities using contrastive learning. another encoder for a different language could be aligned with the clip textual encoder using teacher learning. alignment of the image encoder with the second language is expected if the alignment between two text encoders is achieved using teacher learning. in other words, we can say that for a given set of parallel (translated) sentences as well as its corresponding image ((s 1 , t 1 , i 1 ), ..., (s n , t n , i n )), where t i is the code-mixed [hi-en] translation of english sentence s i and i i is corresponding image-description of s i and a teacher model t for the source language s, we employ mean squared loss to train a new student model m such that m (t i ) ≈ t (s i ) and m (t i ) ≈ t (i i ). this method is known as knowledge transfer since student m transmits teacher t knowledge (c.f. figure 1 ). the loss function objective for cross-lingual teacher training is defined in equation 1 . where n is the total number of samples, s i is an english sentence, t i is a code-mixed translation of an english sentence s i , t (i) is a representation of i th english sentence from the teacher model t and m (i) is the representation of i th code-mixed sentence from student model m. when we train our teacher model to align the english and translated code-mixed [hi-en] text encoders, we expect the same alignment between the translated code-mixed [hi-en] text encoder and image encoder. we propose two codemixed [hi-en] clip encoders: (i). cmclip-rb (code-mixed clip with bert and resnet) and (ii). cmclip-vx (code-mixed clip with xlm @xcite and visual transformer[vit] @xcite ). in the case of cmclip-rb, the rn50x4 variation of clip is used as a teacher model, and multilingual bert uncased is the base student text encoder. meanwhile, in the case of cmclip-vx, the vit-l-14 variation of clip is used along with xlm-roberta-large as a student text encoder. we propose these two variations since resnet is a base clip variation and vit's (visual transformers) are advanced clip variations. ## results and analysis this section compares the performance of both proposed models on the training and validation dataset, analyses teacher training alignments, and discusses the evaluation results. ## conclusion in this paper, we presented cmclip, a novel pre-trained model designed to learn joint visualsemantic embeddings through teacher-student train experimental results across various multilingual and code-mixed models demonstrated the superior performance of cmclip, especially in downstream tasks like image-text retrieval and multimodal classification. in the future, our framework could be extended to other code-mixed languages and additional vision-and-language tasks.