Learning Multimodal Contrast with Cross-modal Memory and Reinforced Contrast Recognition (2024)

Abstract: In many practical scenarios, contents from different modalities are not semantically aligned; for instance, visual and textual information may conflict with each other, resulting in non-compositional expression effects such as irony or humor. Effective modeling and smooth integration of multimodal information are crucial for achieving good understanding of the contrast across modalities. Being focusing on image-text matching, most current studies face challenges in identifying such contrast, leading to limitations in exploring the extended semantics when images and texts do not match. In this paper, we propose an LLM-based approach for learning multimodal contrast following the encoding-decoding paradigm, enhanced by a memory module with reinforced contrast recognition, and use a series of tasks that have the nature of multimodal contrast to verify our approach. The memory module learns the integration between visual and textual features with trainable memory vectors and the reinforced contrast recognition uses self-rejection sampling to optimize the memory to further enhance learning multimodal contrast. The resulted information, accompanied with visual and text features, is finally fed into the LLM to predict corresponding labels. We experiment our approach on four English and Chinese benchmark datasets, where it outperforms strong baselines and state-of-the-art studies.

## introduction multimodal information have become a widespread form of expression in many real-world applications, such as news feeding, social media, and instance messaging, etc., @xcite @xcite @xcite , where in most cases they are fused with image and text pairs. in order to enhance the expressive effect, images and texts are not always semantically aligned, resulting in irony or humor expressions, causing hateful or joyful emotions spreading around groups of people. such misalignment has the non-compositional effect that similar to idioms, where the overall meaning is not the combination of the meanings from its components. 2 for example, figure 1 shows three multimodal memes: (a) is a hateful meme; (b) and (c) are not. note that meme (a) has the same image as (c) and the same text as (b). in other words, a meme can be hateful even when neither its image nor its text is. this example demonstrates the importance of understanding the relationship between image and text, as the hateful attitude in a meme arises from the multimodal contrast foot_2 , which allows the integrated information from different modalities to convey a message (e.g., hateful information) that cannot be expressed by any single modality alone. existing approaches for multimodal understanding mainly focus on image-text matching @xcite @xcite @xcite @xcite and utilize advanced visual and text encoders (such as clip @xcite , flamingo @xcite , flava @xcite , and slip @xcite , etc.) to extract multimodal features, and subsequently align or fuse them by vector concatenation, outer production, or attentions to perform downstream tasks, such as hateful meme detection (hmd) @xcite @xcite @xcite . these hmd models are good at identifying hateful memes when images or text present explicit biases, but are unable to effectively recognize hateful information that is derived from the contrast between image and text. although there are efforts in utilizing additional resources or using model ensemble to improve tasks like hmd @xcite @xcite @xcite , they mainly enhance the generalization ability through more training data or ensemble of multiple models, overlooking the contrast between multiple modalities that lead to better understanding of the non-compositional effect of cross-modal information fusion. in this paper, we propose an approach with llm to learn multimodal contrast through cross-modal memory and reinforced contrast recognition (rcr). the cross-modal memorizing module learns how to capture the information from multiple modalities, and the reinforced contrast recognition utilizes self-rejection training to enhance the memory in learning the contrast by further optimizing the loss function. we use a series of representative tasks that require to understand the contrast from multimodalities to verify our approach. 4 evaluations on four benchmark datasets show that our approach outperforms strong baselines and existing approaches, demonstrating the benefits of memory and reinforcement learning with self-rejection training for all the tasks. ## the approach figure 2 illustrates the framework of our approach, which follows the encoding-decoding paradigm to perform multimodal classification, which predicts a label y based on the image v and embedded text t in a given pair (v, t ). it contains three essential components: the backbone model, the cross-modal memory, and the rcr. the memory module is inserted in between the visual encoding and llm decoding in the backbone, illustrated at the top of figure 2 . the rcr, demonstrated at the bottom of figure 2 , enhances the memory by enhancing the memory module with contrast information via self-rejection training, where an additional term is added to the loss function. in the following subsections, we firstly illustrate the backbone model, then the cross-modal memory, and finally presents the rcr. ## related work recent studies for image-text understanding generally utilize advanced pre-trained visual and text en-coders, such as vilbert @xcite , visual-bert @xcite , mmbt @xcite , flava @xcite , flamingo @xcite , slip @xcite , blip2 @xcite , etc. with llms demonstrating strong language modeling capabilities and achieving stateof-the-art results on many nlp tasks @xcite @xcite @xcite @xcite , there are studies that combines visual encoder and llms, such as minigpt-4 @xcite and llava @xcite . these studies mainly train the models to align image and text features, focusing on image-text matching and achieving satisfying performance accordingly @xcite @xcite . although there are some studies modeling the contrast between multiple modalities @xcite @xcite , their focus is still on the alignment of image and text features. however, besides image-text matching, there are multimodal tasks that rely on the contrast rather than the shared content between multiple modalities, such as hmd, multimodal sentiment analysis, offensive content detection, and humor recognition @xcite @xcite @xcite . to perform these tasks, most existing studies still follow the paradigm of simply aligning and fusing image and text features with a particular module or operation, such as vector concatenation, attentions, and contrastive learning @xcite @xcite @xcite @xcite @xcite . to further enhance these tasks, there are studies that ensembles models to benefit from the outputs of different models from various aspects @xcite @xcite or utilize additional training data and features to enhance the capability of models to capture multimodal information @xcite . compared with existing studies, our approach differs from them by explicitly modeling the contrast between multiple modalities rather than modeling how well images and texts are aligned. particularly, we design a memory module with rcr to learn the contrast, where a reward model is trained to assess how well the memory module learns the contrast and an effective learning approach with self-rejection sampling is applied, which, to our best knowledge, is not used before in previous studies for similar tasks. ## conclusion in this paper, we propose an llm-driven approach for learning contrast with cross-modal memory and rcr, which learns and enhances the contrast information between visual and text features that helps final classification results. we perform several tasks that require modeling multimodal contrast, including hmd, multimodal sentiment analysis, offensive content detection, and humor recognition, etc. experimental results on english and chinese benchmark datasets confirm the validity of the proposed approach, which outperforms strong baselines and existing studies and achieves state-of-theart performance. further analysis also confirms that the combination of memory and rcr demonstrates their superiority in learning contrast between multimodalities and thus facilitating downstream tasks.