Hitting your MARQ : Multimodal AR gument Quality Assessment in Long Debate Video (2021)

Abstract: The combination of gestures, intonations, and textual content plays a key role in argument delivery. However, the current literature mostly considers textual content while assessing the quality of an argument, and it is limited to datasets containing short sequences (18-48 words). In this paper, we study argument quality assessment in a multimodal context, and experiment on DBATES, a publicly available dataset of long debate videos. First, we propose a set of interpretable debate centric features such as clarity, content variation, body movement cues, and pauses, inspired by theories of argumentation quality. Second, we design the Multimodal ARgument Quality assessor (MARQ) – a hierarchical neural network model that summarizes the multimodal signals on long sequences and enriches the multimodal embedding with debate centric features. Our proposed MARQ model achieves an accuracy of 81.91% on the argument quality prediction task and outperforms established baseline models with an error rate reduction of 22.7%. Through ablation studies, we demonstrate the importance of multimodal cues in modeling argument quality.

## introduction structured debates and discussions are the basis for expressing opposing opinions, and are a tool for convincing others to share that opinion. starting with a topic to argue, one can outline steps to reach a conclusion of why that topic is correct. this can take many forms in day-to-day life ranging from salesmen upselling a product or presidential debates, to people arguing whether to get vaccinated or to wear a mask. while the points of the argument may be valid, certain attributes such as clarity in the text, hand movements, and spoken style increase the effectiveness of the argument @xcite @xcite . these attributes increase the credibility of the speaker, and their ability to convince the listener (figure 1 ). measuring the quality of an argument given the language and other non-verbal features remains an elusive problem. although argument quality assessment is an established research area in nlp, assessment in a multimodal context is understudied. most of the previous work focused on argument quality prediction on short text sequences (22-48 words). however, often longer text sequences are needed to validate an argument on a certain topic. even when a model like bert @xcite can be trained to associate raw text with a quality metric @xcite , it can be difficult to interpret which features lead to the output score. such kind of interpretability is crucial to design a feedback system for people who want to improve their communication skills @xcite . in this paper, we study argument quality assessment in a multimodal context using dbates @xcite -the largest (@xmath0. we design interpretable debate-centric features (dcf) such as content variation, clarity, pauses, hand movement, emotional appeal, and so on based on theories of argument quality @xcite @xcite . moreover, we propose a hierarchical multimodal model named marq (multimodal argument quality assessor) to predict high vs. low quality arguments in long debate speeches (6 minutes recordings & 1500 words). sentence level rich unimodal embeddings are extracted from pretrained models (e.g universal sentence encoder @xcite , wav2vec2 @xcite ) to reduce long sequential dependency. a set of lstm encoders and a multihead self-attention layer are used to capture the interaction across the intra-modal, inter-modal, and dcf information. our main contributions are: dataset n modalities mean seq len ibm-rank-args 6.3k {l} 22.98 words ibm-rank-pairs 14k {l} 23.47 words ibm-rank-30k 30k {l} 18.22 words ukpconvarg1 11k {l} 48.3 words dtc 400 {l,a,v} 22.5 seconds dbates 716 {l,a,v} 6 minutes, 1500 words we choose dbates to study multimodal argument quality analysis, since it is the largest and only multimodal debate dataset that is publicly available for research. moreover, the dataset is collected from a competitive college debate competition and has been annotated by expert judges. the aver-age sequence length is around 1500 words which is significantly longer compared to other datasets. the long multimodal sequence is particularly challenging for neural models to comprehend, which is applicable to other multimodal tasks as well. ## debate centric features argument quality can be assessed in many different granularity, some of them are subjective and difficult to compute. here, we propose a set of computable and objective debate-centric features considering all of the language, acoustic and visual modalities. experiments in later sections show that these features can discriminate between high and low-quality arguments. ## marq neural model each data point in the dbates multimodal dataset can be represented as x @xmath1, a, v, dcf } where @xmath2, @xmath3, @xmath4. in addition to the debate-centric features (dcf ), we consider the raw text (l), acoustic (a) and visual (v ) information to model the summary of the video content. given these features our task is to predict whether the arguments presented in the debate are of high quality or not. each debate is around 6 minutes in length and the average text length is around 2000 words. to model this long multimodal sequence, we took the hierarchical approach in our marq model (figure 2 ). first, we extract the sentence level embeddings from the pre-trained models. then a system of lstm encoders is used to learn the temporal relations in the unimodal sentence embeddings. finally, the unimodal sentence embeddings go through a multi head self attention layer to create multimodal representation and enrich it with debate-centric features. ## experiments in this section, we discuss the baseline models and the hyperparameter settings that are used in the experiments. ## results & discussion we analyze the statistical significance of the debatecentric features and then compare our marq model with established multimodal baselines. ## related work the automatic assessment of argument quality @xcite has been receiving growing interest in the nlp community. identifying argument quality has applications in diverse domains, including but not limited to argument search @xcite , finding counter arguments @xcite , automated decision making @xcite , writing support @xcite and essay evaluation @xcite . @xcite proposed a taxonomy of dimensions for quantifying argument quality, where they summarized several high level dimensions behind the structure of good arguments such as clarity, coherence, effectiveness, emotional appeal, etc. however, the subjective nature of these dimensions makes the task of automatic argument quality scoring difficult. earlier research on automatic argument quality assessment focused on comparative pairwise approach, where the task is to identify higher quality argument from a given pair of arguments @xcite @xcite . recently, @xcite introduced straightforward point-wise argument quality metric that scales with the data size linearly. they introduced ibm-rank (6.3k text arguments) that was crowd sourced and then annotated with an individual quality score. following similar approach, @xcite proposed ibm-rank-30k -the largest dataset of argument quality score prediction in free text. both of them utilized bert @xcite based fine tuning for this task. the previous research and datasets ( recently, @xcite publicly released dbates -a dataset of debate videos (@xmath5. the authors performed logistic regression to show that beside text, other nonverbal features have correlation with the performance of a debate. the dbates dataset also presents a global challenge applicable to any multimodal assessment task -representing multimodal signals in a long video, which was not addressed by the authors. in this study, we use this dataset and make two major contributions -1) first to study multimodal argument quality assessment beyond logistic regression; 2) address a technical challenge of multimodal representation for long videos (6 minutes on average). ## conclusion in this paper, we presented a comprehensive study on multimodal argument quality assessment. the debate-centric features reveal interpretable patterns associated with the quality of argument and help improve the prediction performance. these features can easily be adapted to a working system with transparent, objective, repeatable feedback on assessing the quality of a speech and its arguments, and thus lead to equitable access to a training system for anyone wanting to become a good debater. we also proposed a hierarchical neural model (marq) to assess the quality of argument in a long video and showed the importance of having nonverbal cues through further ablation studies. although our work is limited to the only publicly available video dataset of debate, we hope it will inspire others to study the task of argument quality assessment in multimodal context, and develop new datasets and algorithms. the code and data described in this paper are publicly available at https://github.com/ matalvepu/marq