Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection (2021)

Abstract: In deployment, systems that use speech as input must make use of automated transcriptions. Yet, typically when these systems are evaluated, gold transcriptions are assumed. We explicitly examine the impact of transcription errors on the downstream performance of a multi-modal system on three related tasks from three datasets: emotion, sarcasm, and personality detection. We include three separate transcription tools and show that while all automated transcriptions propagate errors that substantially impact downstream performance, the open-source tools fair worse than the paid tool, though not always straightforwardly, and word error rates do not correlate well with downstream performance. We further find that the inclusion of audio features partially mitigates transcription errors, but that a naive usage of a multi-task setup does not.

## introduction with the large amount of available speech data, multimodal approaches to classic natural language processing tasks are becoming increasingly prevalent. many of these proposed systems, however, demonstrate their gains under the assumption of gold transcriptions (e.g., @xcite @xcite , a condition which is highly unrealistic in real-world scenarios. in practice, deployed systems will need to utilize an automatic speech recognition (asr) tool to obtain transcriptions. however, when selecting the best tool given the constraints of the use case, the results of an intrinsic evaluation such as the word error rate (wer) are not necessarily correlated with extrinsic performance on the downstream task of interest @xcite . this issue is exacerbated when considering that asr tools may perform quite differently across different domains @xcite . in this initial work, we explore how much transcription errors affect performance of a multimodal system on several related but distinct downstream tasks and domains. we compare two open-source and one paid transcription tools and evaluate on three multimodal english-language tasks: emotion, sarcasm, and personality detection. we explicitly compare intrinsic and extrinsic evaluations, and discuss the utility of wer as an indicator of taskperformance. our specific contributions are: ## related work multimodal language work frequently makes use of video, audio, and text modalities to examine emo-tion @xcite , sentiment @xcite , and personality @xcite . previous work in emotion recognition has used speech features alone @xcite , speech and text (atmaja and akagi, 2020), and speech, text, and video @xcite to make predictions. ghosal et al. (2020) use transformer-based models to establish a new state of the art on multiple multimodal datasets. recent work in personality detection has often examined the ocean traits (openness, conscientiousness, extraversion, agreeableness, and neuroticism; ponce-lópez et al., 2016) , using either apparent (as perceived by others) @xcite or self-reported traits @xcite . asr systems have also been emphasized in recent decades. some can be custom trained or used with pretrained models @xcite , while others are extensively trained but limited in their customization @xcite . @xcite examine the performance of different asr transcription tools on multiple asr datasets, providing a strong reference for wers; however, to the best of our knowledge, we are the first to compare this form of intrinsic performance to performance on the downstream tasks of emotion, personality, and sarcasm detection. ## approach when multimodal data including speech is used in a deployed system, the related text features generally come from automatic transcriptions of the speech itself. here, we compare how errors in these transcriptions affect the performance of a neural network model on three distinct tasks. further, we compare results from both a single-task and a multitask (mt) network composed of data from two datasets, a common strategy for mitigating issues with limited or flawed data @xcite . as we are not trying to define a new state of the art, we use a simple model for our experiments. this model takes as input text and audio features from meld and firstimpr (section 4), and feeds them through a late-fusion network (i.e., one which processes the modalities separately, then concatenates them before predicting). in the mt setting, the final layers are task specific, and the base layers share parameters. we generate transcriptions for each dataset using three separate asr systems (section 4.3) and find the intrinsic performance (wer foot_1 ; section 6.1) as well as the extrinsic performance for each (i.e., performance on the downstream tasks of interest; section 6). ## data for our experiments we use three datasets covering distinct tasks. for each data point, we extract acoustic features and obtain three different transcriptions, in addition to the dataset-provided gold transcriptions, which form the basis for our comparative study. dataset sizes are shown in ## models to evaluate the impact of transcription on performance, we create (a) baseline models that use only text or only audio features, plus (b) a multimodal model that uses audio and text. note that as our goal here is not to achieve a new state of the art, but rather to explore the impact of real-world options for transcriptions, we use straightforward models that are not particularly architecturally tuned. text-only baseline: the text baseline consists of a two-layer lstm @xcite . 300d glove embeddings trained on 42b words @xcite are concatenated with 30d trainable text embeddings and fed through the network. the 100d output vectors representing each utterance are then fed through a prediction layer with a cross-entropy loss function 8 . ## experiments and results we evaluate how the automated transcriptions from different asr tools affect performance on three distinct downstream tasks. we show the intrinsic performance of the tools (in terms of wers) and compare it to the extrinsic performance. show the difficulty of asr tools with these datasets' speech. ## conclusions we demonstrated that selecting the appropriate transcription service for a task is dependent upon the task and models to be used. paid transcriptions may result in better model performance, but differences in task performance may be much smaller than suggested by only considering wer. further, depending on the task, the noise introduced from automatic transcriptions may be mitigated by including additional input modalities, such as audio.