Multimodal Argument Mining: A Case Study in Political Debates (2022)

Abstract: We propose a study on multimodal argument mining in the domain of political debates. We collate and extend existing corpora and provide an initial empirical study on multimodal architectures, with a special emphasis on input encoding methods. Our results provide interesting indications about future directions in this important domain.

## introduction argument mining (am) aims to extract arguments and their relations from natural language sources @xcite . performing am usually entails tackling one or more tasks like argumentative component detection and classification, link prediction, relation classification, or stance classification @xcite in a particular domain of interest. among the many areas and genres where am was investigated, the political domain allows for intuitive applications with the final aim of detecting fallacies, persuasiveness degree @xcite , truthfulness @xcite and coherence in the candidate's argumentation @xcite , or summarizing the candidate's positions @xcite . so far, most of am research has focused on textual inputs. political debates and speeches have been no exception. however, differently from other domains, this particular one is especially rich in audio input sources. this could be important, since the audio input, in addition to text, may leverage the exploitation of para-linguistic cues related to the argumentation process, improving the performance of argumentative component detection and other am tasks @xcite @xcite . to date, partly owing to the scarcity of non-textual corpora for am @xcite , only a couple of attempts have been made in this direction. con-versely, outside of am, in the broader area of natural language processing (nlp), multimodal deep learning (mmdl) is attracting growing interest, also owing to remarkable progress made in the field. current research in mmdl focuses on advanced input representations and fusion solutions. these include end-to-end architectures fully based on transfer learning for input representation @xcite and attention-based architectures for efficient input management @xcite @xcite . these noteworthy developments suggest that time is ripe to rethink multimodal am in light of the latest findings in multimodal nlp research. in spite of a wide availability of raw audio sources, processing and annotating good quality data can be very costly. to the best of our knowledge, the only two multimodal am corpora on political speeches are ukdebates @xcite , which addresses the task of claim detection, and m-arg @xcite , which focuses on argumentative relations between sentences. these are small-sized corpora where a handful of speakers debate in one or a few occasions over a year's time span. on the other hand, uselecdeb60to16 is a corpus curated by @xcite , where a significant number of us presidential candidates debate over a time span of several decades. however, it only contains annotated transcripts, with no link to the audio source. in an effort to push the envelope in multimodal am, with this work we release mm-uselecdeb60to16, an extended version of the uselecdeb60to16 corpus, where the text input is complemented by and aligned to the audio input. at the time of writing, this is the largest multi-modal resource for am in the domain of political debates, as well as the one with the largest number of speakers, and longest time span covered. these features make mm-uselecdeb60to16 a particularly challenging corpus, since para-linguistic cues are very much speaker-dependent @xcite and political communication, argumentation, and language have greatly evolved in such a long time span @xcite . alongside this new resource, we offer a preliminary but rigorous and reproducible experimental study of multimodal am in political debates. our benchmarks are all the relevant corpora available: ukdebates, m-arg, and mm-uselecdeb60to16. we build and compare architectures inspired to proposals from literature, in order to study the effect of changing the encoding of the audio input. in particular, we compare the more traditional featurebased audio encoding, with a more advanced input encoding technique that builds on recent findings in mmdl for nlp. our results indicate that the encoding of the audio input has a noticeable effect on performance, but they also suggest that a better fusion of textual and audio input encodings and more advanced architectural solutions might be needed in order to make progress in the more challenging tasks and corpora. the paper is structured as follows. in section 2 we overview related work in multimodal am and multimodal deep learning (mmdl), with a focus on architectures for text and audio processing. in section 3 we discuss corpora and in section 4 we define the am tasks addressed. section 5 presents the experimental setup and describes models, input encodings and training. section 6 discusses the results of our experimental study. section 7 concludes. in appendix we report all the information needed for reproducibility. the corpus and the code are publicly released. foot_0 ## related work there exists a strong connection between the argumentation process and the emotions felt by people involved in such a process @xcite . this observation motivated the hypothesis that paralinguistic elements encoded in the audio data are significant indicators that might aid identify arguments made in a debate. recent studies confirmed this hypothesis. in particular, in the domain of political debates, @xcite presented a case study in am from speech using a televised debate from the 2015 uk political elections. they built a first-of-a-kind political debate corpus by annotating arguments uttered by three prime ministerial candidates, and showed that audio features helped claim detection when used as input to a support vector machine (svm) classifier together with their textual transcript. more recently, @xcite built the m-arg corpus, which consists in 4,104 labelled pairs of sentences selected from debates of the 2020 us political elections. they experimented on this new corpus using a different multimodal input model. outside of political speeches, a corpus that couples transcript and audio of several debates was developed by @xcite . however, differently from the previous corpora, here non-political debates are carried out by paid actors on a set of controversial topics taken from the idebate web site. to the best of our knowledge, at least in political debates, multimodal am has not been further explored. reasons for that lie partly in the difficulty and heterogeneity of am tasks, partly in the scarcity of multimodal data for am, partly in the inherent challenges of multimodal deep learning (mmdl). one such challenge is in endowing models with the ability to digest and actually benefit from different, complementary modalities. in this respect, the works by @xcite , @xcite and @xcite could be viewed as proofs-of-concept of the potential of multimodality in am. they used mostly traditional methods for categorising data and encoding audio, such as svm classifiers and mfccs. like most other studies in am, they were carried out on a single corpus and a specific task. recent mmdl solutions suggest a number of promising directions for improvement. these include full transfer learning-based frameworks to alleviate the problem of multimodal data shortage @xcite @xcite and attention mechanisms to handle interactions among and between different modalities @xcite @xcite . for example, @xcite , a recent mmdl architecture, integrates pre-trained text and audio models via a dual self-attention mechanism. in our work, we examine the architectural designs presented in earlier studies @xcite and also suggest a multimodal architecture comparable to audibert, based on text and audio embedding taken from pre-trained models like glove @xcite , bert @xcite and wav2vec @xcite . ## data we experiment on three different debate corpora, designed to address four separate but strictly correlated argument mining tasks. ## methodology we consider four distinct classification tasks: an input sentence x is classified as containing an argument (arg), or not containing an argument (not-arg); each input is characterized by two modalities: the textual input x t and the audio input x a . to assess the impact of each modality, we consider three distinct input configurations: text-only (to), audio-only (ao), and text-audio (ta), where both modalities are given as input. ## experimental setup we define a reproducible and robust experimental setup to evaluate the contribution of each modality to am tasks, and to assess the impact of different input representations and classifiers. the limited amount of data, especially in ukdebates and m-arg, caused our setup to differ in several ways from previous studies. hence our results are not directly comparable with those published in the relevant literature. nonetheless, our setting includes all the classifiers used in such studies, in addition to more recent representation techniques. regarding ukdebates, lippi and torroni (2016a) address the cd task by experimenting on each politician individually. in particular, the authors evaluate a multimodal svm classifier via a 10-fold cross-validation routine. in contrast, we evaluate our models on all speaker sentences via a repeated 5-fold cross-validation routine. such a design choice was made to curb the high variance usually observed in a model's performance when neural models are trained with little data @xcite . we set the number of repetitions to 3. for the same reason, we evaluate our models on m-arg via a repeated 5-fold cross-validation routine. we set the number of repetitions to 3. our approach differs from the one proposed by @xcite , where the corpus is divided into train and validation splits. for mm-uselecdeb60to16 we follow the same experimental setup as in @xcite . despite the different number of samples, we keep the same train, validation, and test splits proposed for the original corpus. we define a repeated training and evaluation routine for model benchmark, setting the number of repetitions to 3. see appendix b for additional details on our experimental setting, number of samples and data splitting. ## results we report the classification results on each dataset. additionally, we perform an ablation study regarding the models trained in the ta configuration to evaluate the contribution of each input modality. ukdebates for each row, we report the best results in bold, second best results are underlined instead. m-arg ## conclusion political debates and speeches are an important domain where audio data is abundant. the automated argumentative analysis of such data could leverage a variety of innovative applications and open promising research avenues. yet, am research so far has mostly focused on textual transcripts. motivated by recent advances in mmdl and in an effort to push the envelope in multimodal am research, we release the largest-to-date multimodal am dataset. we thus run an empirical study on three multimodal am datasets differing from one another in many respects like size, topics, annotations, and speaker variety. to this end, we defined three architectures, inspired from literature baselines. our results indicate that embedding-based audio encodings have an edge over feature-based encodings. they also suggest that there is a significant margin for improvement, hence the need for different architectures to enable a tighter mutual interaction between input modalities. we speculate that current trends in mmdl, in particular attention-based methods for multimodal input fusion, should be investigated in this domain. we hope that our dataset will facilitate such endeavor. a remarkable result is the performance of the ao configuration, which in some cases is observed to be competitive with ta.