Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks (2024)

Abstract: Effectively leveraging multimodal information from social media posts is essential to various downstream tasks such as sentiment analysis, sarcasm detection or hate speech classification. Jointly modeling text and images is challenging because cross-modal semantics might be hidden or the relation between image and text is weak. However, prior work on multimodal classification of social media posts has not yet addressed these challenges. In this work, we present an extensive study on the effectiveness of using two auxiliary losses jointly with the main task during fine-tuning multimodal models. First, Image-Text Contrastive (ITC) is designed to minimize the distance between image-text representations within a post, thereby effectively bridging the gap between posts where the image plays an important role in conveying the post’s meaning. Second, Image-Text Matching (ITM) enhances the model’s ability to understand the semantic relationship between images and text, thus improving its capacity to handle ambiguous or loosely related posts. We combine these objectives with five multimodal models, demonstrating consistent improvements of up to 2.6 F1 score across five diverse social media datasets. Our comprehensive analysis shows the specific scenarios where each auxiliary task is most effective.

## introduction multimodal content including text and images is prevalent in social media platforms @xcite . the content of both text and images has been widely used to improve upon single modality approaches in various downstream tasks such as sentiment analysis @xcite @xcite , hate speech and rumor detection @xcite @xcite when @user gets more followers than you in 12 hours ## multimodal auxiliary tasks image-text contrastive (itc) modeling textimage pairs in social media posts involves capturing hidden cross-modal semantics (vempala and preoţiuc-pietro, 2019; @xcite . for instance, in figure 1 (left) the visible mood of the person on the photo is related to the text of the post. instead of directly matching images with textual descriptions (e.g., a man wearing a helmet), we aim to encourage the model to capture the dependencies between the image and text within the posts. for this purpose, we use the itc objective @xcite @xcite which pushes towards a feature space in which image and text representations of a post are brought closer together, while image and text representations that appear in different posts are pushed further apart. let l n and i n be the n-th (normalized) representation of text and accompanying image of a post in a training batch. while the cosine similarity of the pair l n and i n is minimized, the cosine similarity of all other random pairs (e.g., l n and i m ; i m is an image from a different post in the current batch) is maximized. given n posts within a training batch, itc loss is defined as follows: image-text matching (itm) in social media posts, unrelated or weakly related text-image pairs are common @xcite such as the post depicted in fig. 1 (right). to address this, we use the itm objective @xcite @xcite during fine-tuning to understand the semantic correspondence between images and text. itm involves a binary classification loss that penalizes the model when a given text and image do not appear together in a post. let i n and l n be the image and text representation of the n-th post in a training batch, we randomly replace i n with an image of another post from the current batch with a probability of 0.5 following @xcite . if i n is replaced, then the image and text do not match, otherwise i n and l n match. thus, the itm loss corresponds to the cross-entropy loss for penalizing incorrect predictions, l it @xmath0. ## analysis we analyze ber-vit-att's predictions on tir to understand when each auxiliary task benefits different image-text relations as categorized by vempala and preoţiuc-pietro (2019) based on image contribution and text representation (figure 3 and 4 ). when the text is represented in the image using both auxiliary tasks (models denoted with +c+m), the model achieves the best performance, text is represented in image text is not represented in image image adds to the meaning image adds to the meaning text is represented in image text is not represented in image image does not add to the meaning image does not add to the meaning especially when the visual content is not semantically relevant to the post. we observe that 80.2% of the tweets are correctly classified achieving a substantial improvement over the ber-vit-att baseline where only 59.3% of the posts are correctly classified. when text is not represented on the image, we find that including itc performs best when the visual content is relevant, with 59.3% of the tweets correctly classified compared to 49.2% using ber-vit-att. finally, in cases where the image does not enhance the semantic meaning, ber-vit-att+m ex- text is represented in image text is not represented in image image adds to the meaning image adds to the meaning new years resolution. when @user gets more followers than you in 12 hours att:✗ | +c:✓| +m:✓| +c+m:✓ att:✗ | +c:✓| +m:✗ | +c+m:✗ text is represented in image text is not represented in image image does not add to the meaning image does not add to the meaning hibits the highest performance, correctly classifying 65% of the posts. this validates our hypothesis that incorporating itm helps models to effectively identify posts with weaker image-text relationships. ## conclusion we presented an extensive study on the effectiveness of using two auxiliary tasks, image-text contrastive and image-text matching when fine-tuning multimodal models for social media posts classification. this approach addresses the challenges of hidden cross-modal semantics and weak image-text relationships in social media content.