M ulti C limate: Multimodal Stance Detection on Climate Change Videos (2024)

Abstract: Climate change (CC) has attracted increasing attention in NLP in recent years. However, detecting the stance on CC in multimodal data is understudied and remains challenging due to a lack of reliable datasets. To improve the understanding of public opinions and communication strategies, this paper presents MultiClimate, the first open-source manually-annotated stance detection dataset with 100 CC-related YouTube videos and 4,209 frame-transcript pairs. We deploy state-of-the-art vision and language models, as well as multimodal models for MultiClimate stance detection. Results show that text-only BERT significantly outperforms image-only ResNet50 and ViT. Combining both modalities achieves state-of-the-art, 0.747/0.749 in accuracy/F1. Our 100M-sized fusion models also beat CLIP and BLIP, as well as the much larger 9B-sized multimodal IDEFICS and text-only Llama3 and Gemma2, indicating that multimodal stance detection remains challenging for large language models. Our code, dataset, as well as supplementary materials, are available at https://github.com/werywjw/MultiClimate.

## introduction as climate change (cc) gains global attention, measuring human stance towards cc becomes increasingly important. numerous large language models (llms) and deep learning models have been developed to address these challenges. these models can help detect public opinions and assist stakeholders to improve decision-making, thus providing valuable insights into public perception regarding climate change @xcite @xcite . stance detection is a task to determine whether authors of a document support, oppose, or take a neutral stance toward a specific target @xcite @xcite anytime you hear the word harabagiu, 2023b). it enhances information management by efficiently categorizing diverse opinions. identifying varying public opinions helps promote societal understanding and communication, thus reducing conflicts and enhancing public discourse. previous research explored stance detection on climate change @xcite @xcite , but focusing on text-only data. images and videos can shape public awareness of climate change by vividly illustrating its social impacts. recently, multimodality gained significant traction for connecting cc to discourse @xcite , as images and videos largely impact cc perceptions @xcite . investigating stance through multimodality also becomes essential. however, due to the lack of datasets, developing multimodal models for stance detection on climate change remains challenging, as previous studies mainly focused on language texts rather than the visual modality. to fill this gap, this paper proposes the first opensourced dataset, multiclimate, that integrates both visual and textual modalities, specifically 4, 209 image frames and transcripts, to label stances in 100 cc-related youtube videos ( §3). §4 evaluates text-only, image-only, and multimodal models on multiclimate stance detection. §5 concludes the paper and proposes future directions. our results show that the text-only bert model outperforms image-only models on multimodal stance detection, and the best performance is achieved by fusing models from both modalities. we further experiment with 9b-sized large language and multimodal models and illustrate that these larger models deliver unsatisfactory zeroshot results, much lower than our state-of-the-art (sota) fusion models. fine-tuning a large multimodal model brings about some improvements, but it is resource-heavy and encounters the green nlp problems. current multimodal stance detection datasets emphasize different aspects such as communication frames and trending topics like covid-19 @xcite @xcite . these data rely solely on static images or extract the first frame from a video or gif for visual input. despite these advancements, research on stance detection using visual input remains limited. our multiclimate dataset addresses climate change topics by utilizing frames from full videos as visual input and transcripts as text, allowing for a more comprehensive information coverage. ## the multiclimate dataset we propose a new open-source multiclimate dataset, the first climate change stance detection corpus on multimodal data. to the best of our knowledge, mendeley provides a closest dataset to ours (aharonson, 2023). however, it only provides a simple excel file with 168 youtube links and some basic statistics on each video. 1 most of these videos are not under the creative commons license and none is annotated with stance labels. our multiclimate dataset consists of 100 ccrelated youtube videos in english with 4, 209 frame-transcript pairs. to ensure a balanced distribution, we partition multiclimate into 80% train, 10% dev, and 10% test sets, not only in terms of the number of videos but also the number of frametranscript pairs. video collection. 100 youtube videos are collected by querying "climate change" and filtering the results by creative commons license. the videos are downloaded using the pytube foot_1 and yt-dlp foot_2 apis, while the transcripts are extracted using the youtube-transcript-api. 4 frame extraction and alignment. to effectively use our video data, we extract the initial frame from each 5-second segment using ffmpeg @xcite . we then manually align each frame to its corresponding full sentence in the transcripts to form a frame-transcript pair. longer sentences can be aligned with multiple time-stamped frames, foot_4 and we allow different stance labels for individual frame-transcript pairs, as shown in figure 1 . ## multiclimate stance detection this section evaluates sota text-only, image-only, and multimodal models on multiclimate stance detection. we use both accuracy and weighted f1 scores since support/neutral/oppose labels are unbalanced in the dataset. the models are run on the cpu, google t4 gpu, and nvidia geforce rtx 2080. ## conclusion and future directions this paper curates multiclimate, a novel multimodal stance detection dataset to support video research on climate change. our results show that bert-fusion models can achieve considerably higher performance than large multimodal models. textual information is vital on multiclimate, while the visual modality can compensate for the drawbacks of language models, and the best is achieved when combining image and text information. simi-larly, sota llms beat large multimodal models in zero-shot. our work provides a foundation for multimodal stance detection in cc. we plan to extend our dataset with more videos and annotations and further investigate the interactions between visual and textual features, as well as the informativeness of each modality in manual stance labeling. we are also interested in expanding to audio and video modalities, as well as unimodal models using transfer learning techniques, to improve performance and explain stance detection in cc. we hope our work fosters the positive social impact of cc stance detection in the nlp field.