Investigating the Impact of ASR Errors on Spoken Implicit Discourse Relation Recognition (2022)

Abstract: We present an empirical study investigating the influence of automatic speech recognition (ASR) errors on the spoken implicit discourse relation recognition (IDRR) task. We construct a spoken dataset for this task based on the Penn Discourse Treebank 2.0. On this dataset, we conduct “Cascaded” experiments employing state-of-the-art ASR and text-based IDRR models and find that the ASR errors significantly decrease the IDRR performance. In addition, the “Cascaded” approach does remarkably better than an “End-to-End” one that directly predicts a relation label for each input argument speech pair.

## introduction discourse parsing is one of the key research areas in nlp @xcite . one important problem in discourse parsing is the implicit discourse relation recognition (idrr) task @xcite , which aims to identify the relation between two discourse arguments (e.g. clauses, sentences or paragraphs in the document) without explicit discourse connectives (e.g., but, and, because and the like). this idrr task has attracted many research works @xcite @xcite @xcite @xcite , and it is very useful for many downstream nlp tasks such as machine translation @xcite ), text summarization @xcite and question answering @xcite . implicit discourse relations also play essential roles in spoken language understanding tasks @xcite . thus, it is worth investigating the idrr task in spoken form. research works have been performed for idrr from the manual speech transcripts @xcite @xcite original argument 1: computer-generated videos help argument 2: the average american watches seven hours of tv a day transcript argument 1: computer generated vidio's health argument 2: the average american watches seven hours of tevia day in this paper, we present a study that investigates the influence of asr errors on the downstream spoken idrr task. as there is no public benchmark dataset for this spoken idrr task, we construct a dataset for this task based on the penn discourse treebank (pdtb) 2.0 @xcite . following previous works @xcite @xcite that construct spoken derivatives of text-based question answering and text-to-sql datasets, we use the google textto-speech system to produce a spoken variant of the pdtb 2.0 dataset. in our "cascaded" experiments combining state-of-the-art asr and textbased idrr models, we find that the asr errors significantly decrease the performance of the downstream idrr task. we also experiment with an "end-to-end" approach that directly predicts a relation label for each input argument speech pair, and find that the "end-to-end" obtains remarkably lower performances than the "cascaded". ## dataset construction this section presents the dataset construction process for our spoken idrr task. we construct our dataset in the spoken form based on the pdtb 2.0 dataset @xcite , which is one of the largest benchmark datasets used for idrr research. we employ the google text-to-speech system to generate spoken variants of the original text arguments from the pdtb 2.0 dataset. we thus obtain speech pairs and the gold relation label for each speech pair (i.e. the label of the original argument pair). we also employ the standard pdtb 2.0 data split @xcite that uses sections 2-20, 0-1 and 21-22 for training, validation and test, respectively. ## empirical approach on our spoken dataset, we compare two implicit discourse relation recognition approaches: cascaded vs. end-to-end. ## discussion the method of employing the google text-tospeech to generate spoken forms of the original text arguments in the pdtb 2.0 dataset produces an artificially generated dataset, thus not fully reflecting the error types of human speech. in addition, the original raw pdtb 2.0 dataset comes from the wall street journal (wsj) articles. so our dataset might not cover relevant real-world spoken genres. we unfortunately were unaware of the availability of the continuous speech recognition (csr) corpus that consists of human-read speech with texts from the wsj when conducting our study. 1 there might be an overlap between original texts from the pdtb 2.0 dataset and the csr corpus, thus the overlap might be used for further evaluation in future work. ## conclusion we have presented an empirical study investigating the influence of asr errors on the spoken idrr task. we construct a spoken derivative of the pdtb 2.0 dataset and conduct "cascaded" experiments employing state-of-the-art asr and textbased idrr models on this spoken dataset. we find that the asr errors significantly reduce the idrr performance. we also find that an "endto-end" approach that directly predicts a relation label for each input speech pair obtains remarkably lower performances than the "cascaded" one.