Cross-modal Transfer Between Vision and Language for Protest Detection (2022)

Abstract: Most of today’s systems for socio-political event detection are text-based, while an increasing amount of information published on the web is multi-modal. We seek to bridge this gap by proposing a method that utilizes existing annotated unimodal data to perform event detection in another data modality, zero-shot. Specifically, we focus on protest detection in text and images, and show that a pretrained vision-and-language alignment model (CLIP) can be leveraged towards this end. In particular, our results suggest that annotated protest text data can act supplementarily for detecting protests in images, but significant transfer is demonstrated in the opposite direction as well.

## introduction information published on the web, and in particular social media, has become a crucial source for understanding the world and how it develops. systems for the automatic detection and extraction of socio-political events are an important tool for processing this stream of information at scale. traditionally, these systems are primarily designed to process information in the form of text, but with the growing use of multimedia content (such as images and video) on the web and social media especially, there is a great potential for extending the analysis to additional data modalities as well @xcite . a question is however how this can be done in the most efficient manner, and whether existing data in one modality can be reused for extending analysis to another. in this work, we take a focused look at the task of protest detection, and investigate whether data from different modalities can act both supplementarily as well as complementarily for this task. we do so by seeking to answer the following research questions: * equal contribution. rq1 to which extent can the performance of a unimodal protest detection model transfer from one modality to another? rq2 can unimodal detection of protests be improved by using a multi-modal protest detection model? considering the natural way text and images complement each other, the hypothesis is that a multi-modal model trained on both text and images would have a broader understanding of the concept of protests. the investigation has been carried out by combining two existing open datasets for protest event detection, namely the textual clef 2019 protest news dataset @xcite and the ucla protest image dataset @xcite . our contributions are: ## datasets we took use of two open-source datasets: the ucla protest image dataset @xcite for images and clef 2019 protest news @xcite for texts. the ucla dataset consists of a training set of 32,612 images and a test set of 8,154. in our experiments we only consider the binary protest/not protest prediction task. meanwhile, for the protest news dataset we only consider the binary sentence-level classification task in english. it comprises 22,825 sentences in total, retrieved from news articles, which was split into a training and test set with a 75-25 ratio. ## experiments we begin by exploring rq1, that is, whether text and image representations can be interchanged in the task of protest detection. in practice, this would mean that a classifier trained on protest images is tested on protest texts, and vice versa. this is made possible by using a pretrained encoder that is able to represent both modalities in a common feature space. we denote such experiments as cross-modal, where training and test data are of different modalities, meaning zero-shot classification. the extent to which this capability can be transferred between modalities can then be evaluated by comparing to a unimodal baseline, which essentially means we train and test on the same modality. to get a lower bound we also compare against a random classifier baseline. in all experiments we evaluate using the auc-pr metric. to adress rq2, we consider the case where training data for both text and images are available and investigate whether these datasets can synergistically complement each other. specifically, we explore training a model jointly on the protest news and ucla datasets, but evaluate on each modality separately, similarly to the above experiments. we denote this experiment multi-modal, because it is trained on both modalities. this experiment is implemented by combining sentences and images in each training batch. to make use of all the image data, each batch contained 65% images and the remaining 35% sentences. these results can then be also compared to the unimodal baselines explained the training was done on three different datasets, as described in section 2, resulting in three trained classifiers: one trained on the pure image dataset (henceforth refered to as im), one on the pure text dataset (henceforth refered to as txt) and a third trained jointly on both, i.e. the multi-modal mixed dataset (henceforth refered to as mm). for im, the learning rate (lr) was set to 0.01, and for both txt and mm it was set to 0.001. the lr-scheduler for im and txt was a lambda decay, with λ = 0.95, and none for mm. in addition to these hyperparameters, the adam optimizer and batch size of 128 were used for all three models. ## results and discussion as seen from when comparing the unimodal baselines, it is clear that the im model performs much better with an auc-pr score of 0.962 compared to the txt baseline of 0.734. this could be a consequence of the image data being more homogenous in terms of how they represent protests. this also follows the reasoning above: that the texts contain a wider range of representations of protests. an aspect that would be interesting to further investigate is which characteristics in the data that are significant for the separate models when classifying protests, by carrying out an even more thorough data analysis. when inspecting some samples that the im model scores high on, see figure 2 , many of them contains concepts such as banners and placards, full-body humans, roads and cities as well as buildings. as for the text-model, some words that often occured in samples that recieved high scores include protest, traffic, roads, bodies of power (ie. government, police), bomb, students, injured, crowd. neither the list of visual concepts or words are exhaustive, but they could give an when it comes to the case of the multi-modal (mm) model we see two things. firstly, when comparing performance to the unimodal baselines it is clear that the mm model performs slightly worse in both cases. when testing on images, the difference in performance is ∼0.013, whereas for text ∼0.034. in contrast to our hypothesis, this indicates that training on both modalities does not provide the model with a broader understanding of the concept of protests, and consequently the performance on unimodal test sets is not improved. we speculate this is partly due to the fact that texts and images come from different source types (mainstream news vs social media), whereas clip has been trained to align text and image pairs from the same source. there is a possibility that the results would be different if the data used for testing was collected from a wider range of sources than the data used for training, since new data sources may represent protest in slightly different ways. what one does see however, is that when comparing the mm model to both the im and txt models in the cross-modal set up, the mm model performs noticeably better. this is an indication that the multi-modal model in fact learns a representation of protests that succesfully incorporates information from both modalities. ## related work our work is a contribution to the field of event detection, that is, identifying mentions of whether a certain event has occurred. early data-driven approaches to this task based on machine learning relied heavily on hand-designed lexical and syntactic features e.g. @xcite . however, since then approaches based on deep learning indicate better performance can be achieved using less feature engineering by training on "raw" (textual) data @xcite @xcite . specifically for the extraction and detection of socio-political events such as protests, some recent works have taken a pure visual approach. for example, @xcite demonstrated that a visual analysis can contribute protest related features that might be harder to extract from pure text, such as violence, crowd size as well as demographic composition. @xcite further investigated the ability to extract protest related information from images, where the ucla protest image dataset is presented along with experiments for the detection of protests and related attributes. previous works taking a multi-modal approach to socio-political event detection also exist. @xcite used a clustering method of textual as well as visual features to discover events in social media data. @xcite proposed a boosted multi-modal extension to lda for training a supervised event classification model. more recently, @xcite take a deep learning approach to the detection of collective action events based on text and potentially an image from social media posts in china. similarly to clip, they use a late-fusion dual encoder for the processing of text and image modalities. our work differs in that we investigate using nonparallel data, e.g. where protest texts and images are labeled and classified individually. we also differ in that we use data from different sources (images from social media and text from mainstream news), as well as using state-of-the-art pretrained visual-and-language representations. ## conclusions from the results and discussion carried out, we can conclude that the performance of a unimodal protest detection model trained on text can transfer almost fully to do zero-shot classification of protests in images. this means that a protest classifier trained on texts can be used directly on images without any further training or fine-tuning involved, and without significant decrease in performance. the benefit of this would naturally be that an image protest classifier can be put in use without the need of annotating any image data. on the other hand, we observe that the transfer from images to text implies a loss of performance while it is still significant compared to the random baseline. furthermore, the investigation shows that multi-modal training for protest detection can be used almost interchangeably to a unimodaly trained model, as performance does not differ substantially.