Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification (2022)

Abstract: Social media is daily creating massive multimedia content with paired image and text, presenting the pressing need to automate the vision and language understanding for various multimodal classification tasks. Compared to the commonly researched visual-lingual data, social media posts tend to exhibit more implicit image-text relations. To better glue the cross-modal semantics therein, we capture hinting features from user comments, which are retrieved via jointly leveraging visual and lingual similarity. Afterwards, the classification tasks are explored via self-training in a teacher-student framework, motivated by the usually limited labeled data scales in existing benchmarks. Substantial experiments are conducted on four multimodal social media benchmarks for image-text relation classification, sarcasm detection, sentiment classification, and hate speech detection. The results show that our method further advances the performance of previous state-of-the-art models, which do not employ comment modeling or self-training.

## introduction interpersonal communications in multimedia are gaining growing popularity on social media. more and more social media users are turning to pair images to text and vice versa to better voice opinions, exchange information, and share ideas, exhibiting rich and ever-updating resources in multimedia. while potentially benefiting people's everyday decision making, the huge volume of multimedia content might also challenge users in finding what they need. towards a more efficient and effective way to process the online multimodal data, substantial efforts have been made to automatically understand the vision and language on social media through a broad range of multimodal classification tasks for predicting image-text relations @xcite , sarcasm @xcite , text: was going to take mollie for a walk in the park today . pennsylvania weather cooperated as per usual retrieved comments: 1. damn ! that 's a lot of snow to still be around for this time of year , no 2. snow in your area we 'd love to see pictures of it ! 3. heavier snow now shifting to your east … figure 1 : a sample tweet with its image on the left. on the right, the tweet text is shown on the top, followed by the comments retrieved from similar tweets. the word "snow" (in blue) in comments helpfully hint the implicitly shared semantics between image and text. despite the success of visual-lingual understanding witnessed in common domains @xcite @xcite , existing models' performance is likely to be compromised on social media posts. the possible reason lies in the relatively more implicit and obscure imagetext relations therein @xcite , whereas the image-text pairs in the widelyused datasets outside social media (e.g., coco dataset @xcite , vqa dataset @xcite , vcr dataset @xcite ) tend to present explicit information overlap. such issue is nevertheless ignored in many previous solutions, which follow the common practice to fuse visual and lingual features @xcite @xcite , making it hard for a multimodal model to well align cross-modal semantics attributed to their weak correlations @xcite . nonetheless, human readers seem to have no problem in digesting the cross-modal meanings on social media; in response to what they capture from a post, some readers may drop user comments, where some clues to hint cross-modal understanding may be hidden, e.g., an echo of the keypoints. for instance, in figure 1 , "snow" in retrieved comments might strengthen the connection of "weather" in text with the snowing visual scene. the helpfulness of user comments has also been demonstrated in the previous nlp practice for text-only posts @xcite . inspired by that, we propose "borrowing" the senses from human readers and modeling user comments to learn the hinting features therein to bridge the image-text gap. to further benefit posts without user comments, a comment retrieval algorithm is designed to gather comments from other posts in similarity, which is measured via balancing the visual and lingual semantics (henceforth cross-modal similarity). for the related experimental studies, a large-scale dataset is constructed to mimic the open environment (henceforth wild dataset). it contains over 27m multimodal tweets, each with 3 comments on average. then, we explore how to leverage the retrieved comments in multimodal classification and exploit a self-training framework to identify comments' hints which shape the cross-modal understanding (henceforth comment-aware self-training). this considers method feasibility in scenarios where large-scale labeled data is unavailable, which commonly appears in the realistic practice, because the annotation for multimodal data from social media is extremely expensive @xcite . concretely, we adopt a teacher-student prototype @xcite and tailor-make it to learn multimodal understanding with the help of user comments. a teacher model is first trained with the labeled data and pseudo-label the similar posts with comments retrieved from the wild dataset. then, a student model is trained in guidance of both the knowledge gained by the teacher model and hinting features offered by user comments. to evaluate our method in practice, it is comprehensively experimented on four popular social media multimodal benchmarks for varying classification tasks. in the setup of each benchmark, our comment-aware self-training module is customized to bert-based state-of-the-art (sota) architectures. ablation studies then exhibit the individual benefits provided by comments and self-training. then, we analyze the effects of retrieved post number and find the use of more posts would result in both the benefits and noise. next, we probe into comment retrieval and explore the contributions of visual and lingual modality in cross-modal similarity measure, where we observe the joint effects allow the best results. at last, a case study shows how the retrieved comments mitigate cross-modal semantic gap, followed by the an error analysis to discuss the existing limitation. in summary, our contributions are three fold. ## related work our work is in line with multimodal learning and self-training, which are discussed below in turn. multimodal learning. previous work in this field focuses on fusing features from different modalities (e.g., vision and language) @xcite to tackle cross-modal classification tasks, such as visual question answering (vqa) @xcite @xcite , visual commonsense reasoning @xcite , and imagetext retrieval @xcite . most benchmark data in vision and language assumes strong image-text correlations, and many multimodal models are hence designed to explore the common semantics shared by the two modalities. however, it has been recently pointed out that many real-world scenarios, including social media, tend to present image-text pairs with weak and intricate cross-modal interactions @xcite @xcite . despite the substantial efforts made in applying multimodal learning in social media to tackle var-ious visual-lingual tasks, e.g., sarcasm detection @xcite , hate speech detection @xcite , metaphor detection @xcite , etc., most existing methods follow the common practice to fuse visual and lingual features. it is hence challenging for them to figure out the cross-modal meanings exhibit with implicit imagetext links. we thus resort to the retrieved user comments and study how models can find the hinting features therein to mitigate the cross-modal gaps. our work is also related to @xcite , where the retrieved similar data show helpful in better aligning visual-lingual features for multi-modal classification. different from them, we additional retrieve comments and learn the hints therein via self-training, which allows easy integration to most multi-modal classification archetectures. ## comment-aware self-training this section presents the entire comment-aware self-training workflow illustrated in figure 2 . in the following, we first describe how we gather the wild dataset for retrieval, followed by the related analysis in §3.1. then we introduce the retrieval module to find similar posts and their comments in §3.2. next, we describe the usage of comments in the multimodal architecture in §3.3. at last, §3.4 presents the comment-aware self-training framework based on retrieved results. ## conclusion we have presented the potential of employing comments to better form visual-lingual understanding on social media, where 27m tweets with comments are contributed for the related study. a novel framework is proposed to retrieve comments from similar posts and explore comments' hinting capabilities via self-training. experimental results on four social media benchmarks show the universal benefit of leveraging retrieved comments and conduct comment-aware self-training on various multi-modal classification tasks and architectures.