Uncertainty-Guided Modal Rebalance for Hateful Memes Detection (2024)

Abstract: Hateful memes detection is a challenging multimodal understanding task that requires comprehensive learning of vision, language, and cross-modal interactions. Previous research has focused on developing effective fusion strategies for integrating hate information from different modalities. However, these methods excessively rely on cross-modal fusion features, ignoring the modality uncertainty caused by the contribution degree of each modality to hate sentiment and the modality imbalance caused by the dominant modality suppressing the optimization of another modality. To this end, this paper proposes an Uncertainty-guided Modal Rebalance (UMR) framework for hateful memes detection. The uncertainty of each meme is explicitly formulated by designing stochastic representation drawn from a Gaussian distribution for aggregating cross-modal features with unimodal features adaptively. The modality imbalance is alleviated by improving cosine loss from the perspectives of inter-modal feature and weight vectors constraints. In this way, the suppressed unimodal representation ability in multimodal models would be unleashed, while the learning of modality contribution would be further promoted. Extensive experimental results demonstrate that the proposed UMR produces the state-of-the-art performance on four widely-used datasets.

## introduction memes, a form of user-generated content on social media platforms, have become a prevalent way for expressing opinions. generally, memes consist of an image paired with a humorous caption. however, against a backdrop of current political and socio-cultural fragmentation, a sharply increasing * corresponding author figure 1 : examples demonstrate the inherent uncertainty in hateful memes, which is the degree of contribution between modalities to hate sentiment. the left example indicates that identifying hate sentiment should focus on cross-modal features, while the right example should focus on unimodal features. previous research on hateful memes detection has employed pre-trained vision-language models for learning vision, language, and cross-modal interactions comprehensively @xcite @xcite . meanwhile, sophisticated fusion techniques @xcite @xcite and external knowledge enhancement methods @xcite @xcite have been proposed to further learn the discriminative features of memes. although the above studies have produced promising progress, they excessively rely on multimodal fusion features, where the inherent uncertainty and imbalance between modalities have not been explicitly considered. the modality uncertainty is caused by the contribution degree of each modality to hate sentiment. as illustrated in figure 1 , the text in the left meme narrates an incredible story, while the image shows two smiling individuals. the text and image convey completely opposite sentiments. in this case, the multimodal fusion feature can provide additional discriminative information to detect hate sentiment against religion in the meme. conversely, the text and image in the right meme convey a consistent sentiment, allowing the meme to be identified as non-hateful. however, the learning of the crossmodal fusion features would cause an interaction between black in the text and woman in the image, leading the model to incorrectly classify it as sexist @xcite . therefore, quantifying the inherent uncertainty is crucial for determining when unimodal information suffices and when the integration of cross-modal information becomes necessary. the modality imbalance is caused by the dominant modality suppressing the optimization of another modality. through experimental analysis, we find that there is a modality imbalance phenomenon between the unimodal features in hateful memes. as shown in figure 2 , the performance of text is closer to multimodal representation compared to image, but the performance of text and image modality in multimodal models is clearly worse than that of the image-only and text-only models, respectively. the above phenomenon indicates that the dominant text modality leads to the suppression of image modality optimization, further resulting in the inability of multimodal models to fully unleash the corresponding discriminative capabilities, thereby affecting the judgment of modality contributions. to address the above issues, this paper proposes an uncertainty-guided modal rebalance (umr) framework for hateful memes detection. specifically, we incorporate a probability distribution to produce a stochastic representation for individual samples, diverging from the deterministic point embeddings employed in current approaches. to simplify modeling, we associate each meme with a gaussian distribution in a latent space defined by mean and variance parameters. the mean represents the feature, whereas the variance measures the uncertainty. by modeling uncertainty, we flexibly combine discriminative cross-modal and unimodal features. furthermore, we introduce a crossmodal feature fusion module based exclusively on mlp to capture semantic between images and texts, providing complementary features for hateful memes. finally, we improve the cosine loss to alleviate the modality imbalance by considering both weight norm and inter-modal constraints. releasing the unimodal representation ability in multimodal models through modal rebalancing further promotes the learning of modality contributions. the main contributions are summarized as follows: ## conclusion in this paper, a hateful memes detection framework (umr) is proposed to address the challenges of modality uncertainty and modality imbalance. by extracting stochastic embeddings from a gaussian distribution to quantify uncertainty, the framework adaptively aggregates cross-modal and unimodal features. by improving cosine loss from weight norm and inter-modal constraints, modality imbal-ance can be alleviated. moreover, umr demonstrates scalability by accommodating various multimodal models as backbones. experimental results on four widely-used datasets reveal that umr consistently outperforms baselines and achieves competitive performance compared to existing methods for hateful memes detection. quantitative analysis further validates the rationality of each component.