Augmenting pre-trained language models with audio feature embedding for argumentation mining in political debates (2023)

Abstract: The integration of multimodality in natural language processing (NLP) tasks seeks to exploit the complementary information contained in two or more modalities, such as text, audio and video. This paper investigates the integration of often under-researched audio features with text, using the task of argumentation mining (AM) as a case study. We take a previously reported dataset and present an audio-enhanced version (the Multimodal USElecDeb60To16 dataset). We report the performance of two text models based on BERT and GloVe embeddings, one audio model (based on CNN and Bi-LSTM) and multimodal combinations, on a dataset of 28,850 utterances. The results show that multimodal models do not outperform text-based models when using the full dataset. However, we show that audio features add value in fully supervised scenarios with limited data. We find that when data is scarce (e.g. with 10% of the original dataset) multimodal models yield improved performance, whereas text models based on BERT considerably decrease performance. Finally, we conduct a study with artificially generated voices and an ablation study to investigate the importance of different audio features in the audio models.

## introduction in recent years, there has been an increasing interest in multimodal classification, which refers to the task of automatically classifying an input based on multiple modalities or sources of information, such as text, images and audio @xcite . multimodal approaches are beneficial as they can reduce the subjectivity of classification with a single modality and improve the accuracy of the overall classification. however, finding the best representations (especially those that work well with other modalities), aligning and fusing them, and getting the models to co-learn are difficult challenges to overcome @xcite . a large body of literature has focused on the combination of image and text for applications like emotion recognition @xcite , fake news detection @xcite , image classification @xcite , or document image classification @xcite . much less attention has been paid to combining audio with text. audio can convey a variety of information about the pitch or intonation of the speaker that can indicate variance in emotional state as well as better identify modes of communication like sarcasm that have been difficult for models to detect. the integration of audio has successfully improved classification tasks like multimodal sentiment analysis and emotion recognition when compared with classic nlp models @xcite . in this paper, we focus on a less explored nlp area in terms of multimodality: argumentation mining (am). am is the computational study of arguments to develop models that can automatically identify, extract, and represent arguments in text or other forms of digital communication such as audio or video. am has traditionally focused on textual data such as news articles, blog posts, and online comments, but the advantages of using audio to detect arguments have not been extensively explored. in this work, we expand on an existing am dataset of us political debates (uselecdeb60to16 by @xcite ) by including audio. we test the performance of several multimodal am models in different variations of the same dataset, e.g., after balancing the labels and with fractional datasets. our contribution is three-fold: i) a new fully aligned audio dataset, expanding on an existing am dataset (section 3), adding balanced and fractional subsets for researchers to experiment with; ii) original multimodal benchmarking results for this dataset highlighting where audio feature embeddings add most value compared to text-only models (section 4); iii) analysis of audio features importance, including performance comparison of human and computerized voices (section 5) and an ablation study (section 6). ## related work multimodal approaches including audio have been mostly used for sentiment analysis or emotion recognition @xcite , often using the iemocap dataset, one of the oldest datasets that contains 12 hours of dialogue recordings with emotion labels in text, audio and video format @xcite . in recent years newer datasets have been released, such as the savee @xcite and ravdess databases @xcite , the meld dataset @xcite , and the cnu-mosei dataset @xcite . generally, audio-textual multimodal approaches contain separate pipelines for audio and text features, sometimes connected through attention layers. for instance, @xcite combined glove embeddings in a bidirectional long-short term memory (bi-lstm) array for text with a combination of a convolutional neural network (cnn) and a bi-lstm array for the audio. likewise, @xcite used glove embeddings and recurrent neural networks (rnn) for both audio and text, reaching accuracies of 71.8 % with the iemocap dataset. atmaja and akagi (2020) used either lstm or cnn (not at the same time) for the acoustic pipeline and lstm with fast-text and glove embeddings. ho et al. (2020) used a multi-level multi-head fusion attention using bidirectional encoder representations (bert) for the text representations, achieving improved accuracies in three different datasets. audio features in this domain have been generally embedded using low level descriptors (llds), such as mel-frequency cepstral coefficients (mfccs) @xcite . mfccs are computed from the mel spectrogram of the audio signal by performing a discrete cosine transform (dct) of its log to reduce its dimensionality in a way that is highly related to the raw signal, but approximating the human auditory system and often yielding higher classification performance @xcite . as llds do not contain global information about the utterance, high-level statistical functions (hsfs), such as mean, kurtosis and quadratic error, among many others, can also be used. @xcite compared the performance in speech emotion recognition of a hsf classifier based on a deep neural network (dnn), a lls classifier based on a recurrent neural network (rnn) and a raw-signal mel-spectrogram classifier based on a cnn, finding similar performance between the hsf and lld models, and a slightly lower performance for the model using the raw signal, showing the benefits of the low level representations. the use of rnn with llds has been shown to offer benefits by considering the temporal dimension of an utterance @xcite , but several researchers have started to use both cnns and rnns in combination to learn both temporal and local features in the frequency domain @xcite @xcite . whereas mfccs are the feature of choice for the great majority of applications, the list of remaining ldds are virtually endless, including the zero crossing rate, chroma vector, entropy of energy, hammarberg index, spectral slope, harmonic difference, among many others (atmaja and akagi, 2020). while some efforts have been made towards standardization of audio features @xcite , the choice is generally pragmatic and depends on the package used by the researcher, with opensmile toolkit @xcite , librosa @xcite and pyaudio-analysis @xcite being those most commonly chosen ones. on the other hand, am research has focused on a diverse set of applications using the text modality alone, from online interactions @xcite and tweets @xcite to argumentative essays @xcite and political debates @xcite . regarding multimodal am, @xcite presented a first step towards the use of audio features from speech to improve argument detection. in this paper, they used raw input signals, which were passed through a speech recognition api to obtain the text. then they used bag of words and bi-grams together with discrete hsf features from mfccs, namely minimum, maximum, average and standard deviation, to train a support vector machine in an argument classification task. the results were positive towards the addition of audio, although the performance was modest due to the small size of the dataset and the limitations of the text and audio representations. the only other work that considered multimodal aspects used the m-arg dataset @xcite . there, the authors analyzed argumentative relations in the 2016 us presidential debates using text and audio, building an argumentation mining pipeline based on bert embeddings for text and a combination of a bi-lstm and a cnn for the audio. although the dataset, annotated for "support" and "attack" between sentences, was rather small and heavily unbalanced towards the "neither" class, the authors reported a slight improvement when considering audio and text together in a multimodal model. surprisingly, audio features alone showed a better performance than the text-only model based on bert encodings, suggesting that in small datasets, when the performance of bert-based models suffers, audio features might provide a handy supplement to classify arguments. the effect of specific audio features on performance was not assessed. here, we build upon a previous dataset (us-elecdeb60to16) presented by @xcite , which contained english transcripts of the us presidential debates from 1960 to 2016 labelled with more than 29k annotations of argument components and their boundaries. we used the original videos from the debates to obtain aligned timestamps at the sentence level following the work of @xcite , thus enabling the task of multimodal am with a total of 28,850 aligned and annotated sentences. concurrently to the submission of our work, @xcite also presented and released a multimodal dataset, using the same videos and alignment process, with 26,791 sentences. both datasets are complementary, although our dataset is slightly larger as we did not drop any of the debate videos (see next section). @xcite compare datasets and architectures from the two previously mentioned works by @xcite and @xcite , as well as their new dataset, finding generally positive results to the addition of audio. in our work, we report an audio feature analysis, as well as the impact of using computerized voices, and we investigate the benefit of multimodal models on both balanced and fractional small data subsets. ## artificial voices it is not clear what the audio models are specifically looking at when they undergo training, as the features extracted are not always easy to interpret. one hypothesis is that they learn from the words being uttered, their pronunciation and associations thereof in a similar way to text-based models. however, it is also likely that these audio models are picking up intonation or pitch features that are possibly different when one person is making an argument. as a first step in investigating what the audio-based models are paying attention to, we ran our models using artificial voices, instead of the original voices from the presidential candidates. for this, we used the computer generated voices from the microsoft window's text to speech (tts) system. then, we used the text-to-speech conversion library pyttsx3 (v 2.9) for python to run each sentence through the microsoft tts system and generate utterances spoken by the so-called microsoft mark and microsoft zira, the male and female version of us voices. we set the speaking rate at 200 words per minute, but it could be interesting for future studies to observe the accuracy of the models when the speech rate is changed. likewise, each country package in microsoft windows has its own set of unique voices, even if they speak the same language, e.g., uk, india, south africa, and so on, so it could be interesting to check potential differences in accuracy with different accents. ## ablation study finally, to further understand how the different lld audio features might play a role in argument detection, we perform an ablation study with the audio model in which we eliminate one of the six features at a time and assess how the performance of the model changes. the results are in from the ablation study, we see that skipping features does not have a strong influence on the performance with the original full dataset. with the balanced dataset, the elimination of the spectral roll-off feature seems to have a strong effect, as it decreases its macro f-score to 0.402. a special mention to the mfccs is deserved. these are the most common ldds in the literature. eliminating this feature but keeping the rest does not affect the performance (f-score of 0.544) in the original dataset, and improves it in the balanced case to 0.612. as mfccs (and many of the other features) are commonly used to distinguish between voices based on their frequency and pitch, they could bias the model by considering information about the speaker, which is not necessarily relevant to the argument. this would also explain why the artificial voices performed better than the original dataset and why some of the simplest features, like spectral roll-off, have a big influence on performance. ## conclusion in this paper, we explored the possibilities of using audio to detect arguments with multimodal machine learning models in a dataset of us presidential debates that was annotated for arguments. we found that, generally, bert-based text-only models outperformed all models in the original dataset, but multimodal models can improve performance when the datasets are small and bert encodings start performing poorly, in both balanced and unbalanced versions of the data. multimodal models are therefore an alternative that could be used to improve classifications of arguments when data is scarce. to further investigate the reasons for these improvements, we ran audio-only models using artificially generated voices of male and female genders. although we did not find a significant difference in the performance with the artificial voices (only a slight preference toward female), we find that in the most difficult scenarios (balanced and small datasets), the models with artificial voices outperform those with the original audio from the debates. moreover, we perform an ablation study and we find that removing certain features like mfccs can improve the performance of the models. we recognize that these features are commonly used to distinguish between speakers, so irrelevant characteristics of speakers might be influencing the capacity of the models to accurately classify arguments. however, all these features are highly correlated with one another, so further work should investigate which features are more independent of the speakers themselves or if they can be normalized before being fed into the network. these results should be compared with artificially generated voices, which could have different accents or speech rates to understand how those features can influence the classification of arguments.