Identification of Multimodal Stance Towards Frames of Communication (2023)

Abstract: Frames of communication are often evoked in multimedia documents. When an author decides to add an image to a text, one or both of the modalities may evoke a communication frame. Moreover, when evoking the frame, the author also conveys her/his stance towards the frame. Until now, determining if the author is in favor of, against or has no stance towards the frame was performed automatically only when processing texts. This is due to the absence of stance annotations on multimedia documents. In this paper we introduce MMVax-Stance, a dataset of 11,300 multimedia documents retrieved from social media, which have stance annotations towards 113 different frames of communication. This dataset allowed us to experiment with several models of multimedia stance detection, which revealed important interactions between texts and images in the inference of stance towards communication frames. When inferring the text/image relations, a set of 46,606 synthetic examples of multimodal documents with known stance was generated. This greatly impacted the quality of identifying multimedia stance, yielding an improvement of 20% in F1-score.

## introduction frames of communication select particular aspects of an issue and make them salient in communicating a message @xcite . for example, when discussing the confidence in the covid-19 vaccine, selecting the aspects concerning the development of the vaccine may convey the message that the vaccine is safe because (a) scientists have worked for decades on coronavirus vaccines, and (b) the vaccine has been tested and tracked, as shown in the frame illustrated in figure 1 . frames of communication are ubiquitous in social media discourse and can impact how people understand issues and, more importantly, how they form their opinions @xcite . previous who is going to have the covid vaccine ? computational methods @xcite @xcite have focused on the problem of discovery of communication frames, or more precisely, identifying when a frame is evoked in a text. however, frames are also evoked in images, not only in texts, as advocated in @xcite , as text/images combinations convey memorable, emotionally-charged messages. in documents where a frame of communication is evoked, the authors of the documents also express their stance towards the frame. this allows us to consider the multimodal stance towards frames as an extension of the original notion of frame evocation introduced in @xcite , as shown in figure 2 . ## frames of vaccine hesitancy as reported in @xcite , the recent covid-19 pandemic has given rise to a large variety of opinions regarding the public health mandates designed to contain the spread of the virus. while a stance-annotated dataset including tweets targeting these measures was introduced in @xcite , no opinions determined by the emergence of the covid-19 vaccines were captured. these vaccines were hailed with enthusiasm by many, but substantial opposition and hesitancy towards vaccines also emerged on social media. this motivated our interest in capturing the stance towards multiple frames of communication that highlight aspects of vaccination. attitudes towards vaccination, e.g. vaccine readiness or hesitancy, are modeled by a set of seven components or factors that increase or decrease an individual's likelihood of getting vaccinated @xcite . these components are listed in in our work, we have considered the same 113 frames of communication that were reported in weinzierl and harabagiu (2022c), resulting from complex answers to questions that addressed various aspects of each of the components used in the 7c model @xcite . the majority of the 113 frames that we considered address more than one of the components listed in ## results and discussion to evaluate the quality of stance identification on the test collection from mmvax-stance we use the precision (p), recall (r), and f 1 metrics for detecting the accept and reject values of stance. we also compute a macro averaged precision, recall, and f 1 score. the evaluation results for mmvax-stance are listed in among all text-based baseline systems, the lacrscore system achieved the highest macro f 1 score, demonstrating the value of incorporating attitude consistency between tweets. imageinformed text-based systems demonstrated a clear pattern: ocr did not improve the f 1 scores of textbased systems, while image captioning increased the macro f 1 scores by 3 to 4 points over the noncaption-informed systems. image-text aligned systems illustrated a clear performance benefit when incorporating image data directly along with text data for stance identification, with the clip-joint system achieving an improved macro f 1 score over the best text-based systems. multimodal models further demonstrated this phenomenon, with vilt, flava, and bt systems matching or exceeding all text-based systems. bt achieved remarkable improvements in macro f 1 score, increasing 8.5 points above the best text-based system, and increasing 6.1 points above the clip-joint system. bt + s all -d t immediately demonstrates the value of introducing synthetic multimodal examples, as this system was not trained on any of the original training examples from mmvax-stance, only the synthetic examples constructed in section 4. the performance of bt along with training on each set of synthetic examples is provided to demonstrate how each type of synthetic example contributes to improved stance detection. bt + s same shows moderate improvements in f 1 score across both the accept and reject stance values by providing more examples to learn from where the image does not contain stance information. bt + s accept demonstrates a significant increase in the accept f 1 score, as these synthetic examples provide additional opportunities for bt to identify the accept stance in both the text and the images in a tweet. similarly, bt + s reject demonstrates a significant increase in the reject f 1 score, which has a large impact on the macro f 1 score, as the reject stance tends to be more difficult to identify by stance detection systems. finally, bt + s all incorporates all synthetic examples, providing significantly more silver examples for bt to use to identify all stance values. bt + s all achieves the best macro f 1 score of all systems, with a score of 71.32. we also perform an ablation experiment over the benefits of the size of the synthetic dataset. we train bridgetower on various randomly shuffled proportions of the full synthetic dataset s all , from 10%, corresponding to bt + s 10% , to 90%, corresponding to bt + s 90% . furthermore, we compare these systems with bridgetower trained on zero synthetic examples, corresponding to bt, and bridgetower trained on all the synthetic examples, bt + s all , the same systems reported in results of this ablation experiment are provided in several stance detection datasets using social media postings are available, all of them considering only the text of the postings. very well-known is the stance detection dataset shared publicly with se-meval2016 task 6 @xcite . this dataset consists of 4,163 tweets considered politicsrelevant targets (e.g., hillary clinton) when annotating the stance of a tweet's text. a newer and much larger stance-annotated dataset (approximately 50,000 tweets), introduced in conforti et al. (2020), used targets focused on financial transactions that involve mergers and acquisitions. stanceannotated twitter datasets covering topics related to the recent covid-19 pandemic are also currently available. for example, mutlu et al. (2020) published covid-cq, a dataset of approximately 14,000 tweets manually annotated to capture each tweet author's stance regarding the use of hydroxychloroquine in the treatment of covid-19 patients. a dataset of 1,629 tweets annotated with stance was also introduced in miao et al. (2020), targeting lockdown regulations in new york city. in glandt et al. (2021) a stance-annotated dataset of 7,122 tweets focused on 4 targets related to the pandemic (e.g. "wearing a face mask"). the covidlies dataset (hossain et al., 2020) of 6,761 stanceannotated tweets targeted 86 common misconceptions about covid-19. additionally, the cov-axlies dataset (weinzierl and harabagiu, 2022a) is a stance-annotated dataset of 7,152 tweets focusing on 47 frames of misinformation targeting the covid-19 vaccines, which was expanded into the covaxframes dataset (weinzierl and harabagiu, 2022c), containing 14,180 stance-annotated tweets focusing on 113 covid-19 vaccine hesitancy framings. all these datasets ignored the presence of images in tweets when judging the stance values. although the multimedia stance annotations provided by mmvax-stance consider only 11,300 tweets, the synthetic examples that we have generated provide significantly more examples. previous stance identification on twitter was cast either as (1) a classification problem, learning to predict the stance value of a tweet towards a given target; or (2) an inference problem, when a tweet may entail, contradict, or does not imply the target. classification methods: neural architectures, relying on rnns, cnns, and/or attention mechanisms, dominate these methods @xcite @xcite . bert was later shown to be the best model overall for stance detection on the semeval2016 task 6 @xcite . on the covidlies dataset, @xcite demonstrated that bert can be further improved when using graphic attention networks @xcite . further improvements were reported in weinzierl and harabagiu (2022a) when operating on co-vaxlies, by considering the implementation of pragmatics properties of stance into knowledge embeddings. inference methods: when the covidlies dataset of stance annotations was released in @xcite , stance identification was presented as a natural language inference problem, which can benefit from existing textual inference datasets. bidirectional lstm encoders and sentence-bert (sbert) (reimers and gurevych, 2019) were trained on three common nli datasets: snli @xcite , multinli @xcite , and mednli @xcite , helping to identify the stance of tweets. however, classification methods were able to obtain superior results on stance detection. nevertheless, none of these methods consider the information conveyed by images present in tweets for stance detection, which our experiments on mmvax-stance demonstrate. ## conclusion in this paper we introduce mmvax-stance, the first multimodal dataset of stance-annotated tweets towards 113 frames of communication. these frames communicate various aspects of covid-19 vaccination, informed by the 7c model @xcite . baselines were established using several text-based and multimodal stance detection systems. substantial interactions between the text and the image(s) of a tweet were found to be key when inferring the tweet author's stance towards a frame of communication. the inference of the text/image relations led to the generation of a large set of synthetic examples of multimodal tweets, annotated with stance values. baseline methods were greatly improved when the synthetic examples were employed, indicating the role of massive sets of examples and relations between text and images when learning to identify the values of stance in multimodal documents.