M -Arg: Multimodal Argument Mining Dataset for Political Debates with Audio and Transcripts (2021)

Abstract: Argumentation mining aims at extracting, analysing and modelling people’s arguments, but large, high-quality annotated datasets are limited, and no multimodal datasets exist for this task. In this paper, we present M-Arg, a multimodal argument mining dataset with a corpus of US 2020 presidential debates, annotated through crowd-sourced annotations. This dataset allows models to be trained to extract arguments from natural dialogue such as debates using information like the intonation and rhythm of the speaker. Our dataset contains 7 hours of annotated US presidential debates, 6527 utterances and 4104 relation labels, and we report results from different baseline models, namely a text-only model, an audio-only model and multimodal models that extract features from both text and audio. With accuracy reaching 0.86 in multimodal models, we find that audio features provide added value with respect to text-only models.

## introduction understanding and modelling argumentation is a recent and key challenge in natural language processing (nlp). most work addressing this task has focused on extracting arguments from argumentative essays @xcite , social networks like twitter @xcite or online reviews @xcite and discussions @xcite , and not much attention has been paid to mining arguments in natural dialogue. the two most common research questions consider how argumentative relations between units (e.g. support or attack) are annotated or how claims and/or premises are identified @xcite . we offer, to the best of our knowledge, the first multimodal argumentation mining dataset (m-arg) of political debates annotated for such argumentative relations of support and attack, using crowd-sourcing techniques. our contributions are: i) to provide a high quality annotated dataset of political debates with audio and time-stamped transcripts for multimodal argumentation mining; ii) to offer benchmark model results for the research community; and iii) a comparative analysis of the value that multi-modal models bring compared to text-only and audio-only models (section 5). the dataset is derived from a collection of us 2020 presidential debates. five debates were used with the principal speakers being donald trump, joe biden, mike pence and kamala harris, and a moderator ( ## related work much of the research in argumentation mining has been dedicated to the identification of argumentative discourse units (adus) like claims, major claims and premises. for instance, in a first iteration, @xcite annotated 90 while certain tasks in argument mining have been applied in other disciplines, interdisciplinary approaches are important for the impact of these methods to be fully realised. some research in political science has started to bridge the gap in tasks like identifying emotion rhetoric @xcite , gender and emotional expression in politics @xcite , emotional mining in political campaigns @xcite , lexicometrics of euromanifestos @xcite , and, from the ai perspective, ethos mining @xcite using hansard as a dataset. argument mining in political debate is, however, still largely to be explored, although @xcite provide an annotation of 2016 us presidential debates with argument types. @xcite have advocated for the use of crowd-sourced text analysis for political science, finding high levels of agreement and reproducibility between crowd-workers and experts. however, in subjective tasks like identifying support/attack relations, lower levels of agreement are expected. for instance, @xcite used amazon mechanical turk (amt) to annotate 8176 sentences with "for", "against" or "neutral", achieving about 66% of neutral cases and a cohen's κ of 0.70. @xcite also used amt to obtain 16429 labels of different types, including 1736 "relation" labels, defined in their case as "positive", "negative" or "no-argument", with κ = 0.51. these datasets focus exclusively on text, and, as far as we can tell, there is not much argument mining research using multiple modalities such as both text and audio, particularly focusing on identifying support or attack relations between adus. there are, however, some datasets that could be used by the community for this task. for instance, @xcite and orbach et al. (2020b) provide datasets of debate speeches with transcriptions that could help in the extraction of arguments. likewise, mirkin et al. (2020) and orbach et al. (2020a) come closer to argumentation mining research offering datasets of argumentative content and general-purpose rebuttal in speeches. also, @xcite use audio and transcripts of political debates to detect deception. other research explores emotion recognition or sentiment analysis using the iemocap dataset, which contains text, audio and video with emotion annotations @xcite . classic nlp models for relation classification have relied on bag of words (bow) approaches with common classifiers like random forests, support vector machines or naïve bayes @xcite , although more recently lstms and bi-lstms have been used with good results @xcite . some efforts are being devoted to the use of background knowledge or context. for instance, @xcite proposed bi-lstm encoders with self-attention, together with commonsense knowledge extraction. the use of both textual and audio features for the identification of argumentative relations, with approaches similar to those used in multimodal emotion recognition, seems to be mostly unexplored. argument mining of political debates can be seen as a long conversation text classification problem where context matters. unlike the well studied problem area of single and short utterance classification (e.g. 2-3 utterances), dialogue modelling and classification of longer conversations has received little attention to date @xcite . recent approaches to handle long sequence classification include augmented transformer models with information retrieval (ir) or summarisation models @xcite . we constrain ourselves in this paper to providing results for a set of short conversation classification baseline models as we want to focus on showing the value of using multimodal data. however, we expect recent advances in long conversation classification models to yield good results with our dataset in the future. ## methodology the original source of the m-arg dataset was available as audio tracks with transcripts from a kaggle competition. foot_0 this public-domain dataset was originally constructed by downloading audio from youtube and transcripts from rev, 2 as explained in the source metadata. the m-arg dataset with annotations, full transcripts and audio files, source code and model checkpoints for reproducibility is available online in our github repository. 3 ## conclusions and future work in this paper, we have presented a multimodal argumentation mining dataset (m-arg) for political debates based on a corpus of the us 2020 presidential debates with audio and transcripts. the dataset was annotated using crowd-sourcing techniques and we present descriptive statistics of the dataset itself, as well as of the annotations, with discussion of some interesting examples. as a baseline for future research, we evaluated the classification performance of audio-only, text-only and multimodal models. we found that the audio-only and multimodal models could perform with high levels of accuracy and f 1 , although they encountered problems classifying support and attack relations very efficiently. the text-only model performed similarly, but its accuracy in attack classifications was low due to the imbalance of the data. adding the audio to the text, however, in a multimodal model, helped increase the metrics of both support and attacks, although they still remained quite low. we believe these to be encouraging results, as improvements like reinforcement learning to tackle data imbalance, optimised extraction of audio features, addition of (cross-)attention layers, summarisation of the surrounding context or use of background knowledge databases, could greatly improve these performance metrics. moreover, the data can be filtered according to annotation agreement, the annotator's trust and self-confidence in the annotation, potentially training models with higher precision and/or recall, although with less data. one limitation of our dataset is that adus are defined in a very simple manner (by a period with tokeniser). adus might be full sentences on certain occasions, but they might encompass several sentences or simply a clause within one. further work to improve this dataset would include the identification of adus (without necessarily labelling them as claim or premise). likewise, even if a sentence contains a full adu, in natural dialogue it might not present as a clearly stated claim or premise, but might contain irony or rhetorical questions. as already discussed, whether a pair of sentences are showing support or attack is a somewhat sub-jective matter, and for that reason we obtain krippendorff's α = 0.43. one cannot expect crowdworkers to identify, or even easily understand, all types of arguments or what constitutes a fallacy; philosophers continue to disagree. yet with some instruction and information these annotations can better reflect real-world judgements about support and attack arguments. for certain applications, especially where including marginalised voices, ai systems will need to understand and detect how people argue, even if they do not follow the dictates of argumentation theory @xcite . we believe our dataset will be of interest for understanding what people think a proper argument is.