Late Fusion with Triplet Margin Objective for Multimodal Ideology Prediction and Analysis (2022)

Abstract: Prior work on ideology prediction has largely focused on single modalities, i.e., text or images. In this work, we introduce the task of multimodal ideology prediction, where a model predicts binary or five-point scale ideological leanings, given a text-image pair with political content. We first collect five new large-scale datasets with English documents and images along with their ideological leanings, covering news articles from a wide range of mainstream media in US and social media posts from Reddit and Twitter. We conduct in-depth analyses on news articles and reveal differences in image content and usage across the political spectrum. Furthermore, we perform extensive experiments and ablation studies, demonstrating the effectiveness of targeted pretraining objectives on different model components. Our best-performing model, a late-fusion architecture pretrained with a triplet objective over multimodal content, outperforms the state-of-the-art text-only model by almost 4% and a strong multimodal baseline with no pretraining by over 3%.

## introduction in an increasingly divided world rife with misinformation and hyperpartisanship, it is important to understand the perspectives and biases of the creators of media that we consume. media bias can manifest in many ways and has been analyzed from a variety of angles: the news may favor one side of a political issue @xcite , select certain events to report on @xcite @xcite or even distort or misrepresent facts @xcite . identifying a news article's underlying political slant is the task of ideology prediction, which has * equal contribution. focused largely on political texts like news articles and has been tackled with a variety of models, including bayesian approaches with attention @xcite , graph neural networks @xcite , and lstms and bert @xcite . however, past work focuses solely on news article's text; news articles contain other forms of non-verbal information in which the underlying ideology may be realized. consider figure 1a and figure 1b , two images from articles depicting the same news story, but by news sources with opposing ideologies (new york times and fox news, respectively). the underlying ideology of the news source may influence the choice of image: in figure 1a , michael flynn is depicted with a happy expression and surrounded by other figures, while in figure 1b , flynn bears a stern expression and is the sole figure. images are an integral part of news articles: over 56% of articles in allsides foot_0 include at least one image. images are often used to frame certain issues or influence the reader's opinion. for example, liberal websites devote more visual coverage of donald trump and also portray trump with more negative emotions compared with conservative websites @xcite . in addition, images of groups of immigrants, in contrast to individual immigrants, tend to decrease survey respondents' support for immigration @xcite . these findings naturally lead us to conduct a study of political images. in section 3.3, we present a thorough analysis of images, finding, inter alia, that (1) liberal sources tend to include more figures in an image, (2) conservative sources have a higher usage of collage images, and (3) faces are more likely to show negative or neutral emotion rather than positive. although modern american politics have centered around two polar opposites @xcite , 38% of us adults identify as politically independent and do not agree wholly with left or right ideologies. 2 ideology exists on a spectrum @xcite , and we wish to predict more fine-grained ideology than merely left or right. thus, we define multimodal ideology prediction in this work as predicting one of five ideological slants @xcite given both an article's text and cover image. to support this new task, we present several new large-coverage datasets of news articles and images across the ideological spectrum from various sources including allsides, reddit, twitter, and 11 independent news sources. we experiment with several early and late fusion architectures and evaluate several continued pretraining objectives to improve the image and text encoder separately as well as jointly. our technical contributions include a novel triplet margin loss over multimodal inputs, and the first systematic study on multimodal models for ideology prediction, which reveals several findings: (1) images are indeed helpful for ideology prediction, improving over a text-only model especially on right-leaning images; (2) late-fusion architectures perform better than early-fusion architectures; (3) ideology-driven pretraining on both the text and image encoders is beneficial; (4) finetuning with a joint triplet difference loss encourages the model to learn informative representations of ideology. code and datasets can be found at github.com/launchnlp/mulmodide. ## related work media bias/ideology on texts the study of media bias and ideology has a long history going as 2 https://www.pewresearch.org/fact-tank/2019/05/15/ facts-about-us-political-independents/ far back as @xcite . computationally, researchers have studied various approaches in classical machine learning as well as neural methods (e.g. @xcite @xcite . however, these works focus solely on text. there exist several resources of news articles across the political spectrum, compiled for the purpose of educating users on media bias (park et al., 2009, 2011; hamborg et al., 2017, 3 , 4 , 5 ). multimodal studies such as ours need annotated data for training and testing. thus, we collect several datasets containing both political text and images from various sources. ## data in this section, we describe several datasets collected in this work for pretraining and finetuning the proposed models. ## models armed with new diverse datasets of articles and images, we now propose several models, input encoding strategies, and pretraining regimens to tackle the challenges of multimodal ideology prediction. text encoder article image text on monday, president joe biden called for calm in the aftermath of a fatal police shooting of daunte wright, a 20-year-old black man, in minneapolis; this comes shortly after the minneapolis police chief said he believed the officer involved mistakenly used a gun instead of a taser. … cross-modal attn classifier ideology ## experiments we first perform preliminary experiments comparing single-modality models to determine whether the inclusion of images helps ideology prediction. then, we evaluate multimodal experimental setups, exhaustively selecting an image encoder, embedding combination methods, and pretraining methods; performing the continued pretraining; then finetuning the entire model on the task of ideology prediction. implementation details we implement the new models in pytorch, importing existing models from their authors' respective github pages. all models were trained for a maximum of 20 epochs with early-stopping patience of 4. for detailed hyperparameters for pretraining and finetuning and other specific implementation details, please refer to our code on the project page. ## results and analysis text-only and image-only models we first present unimodal experiments on allsides in multimodal models without pretraining next, we present multimodal model results without pretraining in ## conclusion this paper introduces the task of fine-grained multimodal ideology prediction, where a model predicts one of five ideologies leanings given a pair of text and an image. we collect five new largescale datasets of political images and present an in-depth characterization of these images, examining aspects such as facial features across ideologies. we experiment with a combination of stateof-the-art multimodal architectures and pretraining schemes, including a newly proposed triplet margin loss objectives. along with the release of our datasets, our experimental findings will inform the selection of models and training objectives in future work and spur future research in politics, ideology prediction, and other multimodal tasks.