Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering (2021)

Abstract: Spoken question answering (SQA) requires fine-grained understanding of both spoken documents and questions for the optimal answer prediction. In this paper, we propose novel training schemes for spoken question answering with a self-supervised training stage and a contrastive representation learning stage. In the self-supervised stage, we propose three auxiliary self-supervised tasks, including utterance restoration, utterance insertion, and question discrimination, and jointly train the model to capture consistency and coherence among speech documents without any additional data or annotations. We then propose to learn noise-invariant utterance representations in a contrastive objective by adopting multiple augmentation strategies, including span deletion and span substitution. Besides, we design a Temporal-Alignment attention to semantically align the speech-text clues in the learned common space and benefit the SQA tasks. By this means, the training schemes can more effectively guide the generation model to predict more proper answers. Experimental results show that our model achieves state-of-the-art results on three SQA benchmarks. Our code will be publicly available after publication.

## introduction building an intelligent spoken question answering (sqa) system has attracted considerable attention from both academia and industry. in recent years, many significant improvements have achieved in speech processing and natural language processing (nlp) communities, such as multi-modal speech emotion recognition @xcite @xcite , spoken language understanding @xcite @xcite , and spoken question answering @xcite @xcite . among these * equal contribution. automatic speech recognition (asr) and text question answering (tqa) are two key components to build such a sqa system. the former module is used for transforming the speech sequences into text form, and the latter module trained on noisy asr transcriptions utilizes nlp techniques to give a concrete answer. however, utilizing existing stateof-the-art sqa systems to retrieval answers still remain formidable challenges, such as asr recognition errors. this is mainly because asr systems usually fail to recognize the speech, leading to word errors (e.g., "barcelona" to "bars alone"). to address these issues, most existing sqa methods are either text-based @xcite @xcite or fusion-based @xcite @xcite . one line of research examines internal vector representations both in speech and text domains @xcite , often using sub-word units for language modeling. another line of work @xcite investigates the transfer learning problem about how to leverage a large amount of speech and text data to improve the performance of sqa. however, some critical challenges remain, such as robustness, generalization, and data efficiency. different from previous methods @xcite @xcite , we move beyond leveraging dual nature of tqa and asr to mitigate recognition errors. in this paper, we focus not only on extracting the cross-modality information for joint spoken and textual understanding, but also on the training procedure that may take the most advantage of the given dataset. inspired by the recent advance in contrastive learning @xcite and recent breakthrough @xcite ., asr system vq-wav2vec bpe tokenizer speech-q text-q text-p audio-text input temporal-alignment attention position embedding segment embedding token embedding audio-align embedding self-supervised tasks answer prediction contrastive learning pre-trained language model (bert) (a) spoken question answering [cls] [sep] [sep] [cls] pre-trained language model (bert) (b) contrastive learning [cls] [sep] [sep] [sep] [mask] [mask] 2018; liu et al., 2019; rahman et al., 2020 during the self-supervised training stage, instead of building the complex spoken question answering model, we propose to learn a spoken question answering system based on pre-trained language models (plms) with several auxiliary selfsupervised tasks. in particular, we introduce three self-supervised tasks, including utterance restoration, utterance insertion, and question discrimination, and jointly train the model with these auxiliary tasks in a multi-task setting. on the one hand, these auxiliary tasks enable the model to capture sequential order within the given passage. on the other hand, they effectively learn cross-modality knowledge without any additional dataset or annotations to generate better representations for answer prediction. during the fine-tuning stage, along with the main qa loss, we incorporate the contrastive learning strategy to our framework in an auxiliary manner for the sqa tasks. specifically, we use multiple augmentation strategies, including span deletion and span substitution, to develop the capability of learning noise-invariant utterance representations. in addition, we propose a novel attention mechanism, termed temporal-alignment attention, to effectively learn cross-modal alignment between speech and text embedding spaces. by this mean, our proposed attention mechanism can encourage the training process to pay more attention to seman-tic relevance, consistency and coherency between speech and text in their contexts to provide better cross-modality representations for answer prediction. the overview of our framework is shown in figure 1 . we evaluate the proposed approach on the widely-used spoken question answering benchmark datasets -spoken-squad @xcite , spoken-coqa @xcite , and 2018 formosa grand challenge (fgc). experimental results show our proposed approach outperforms other state-of-the-art models when self-supervised training is preceded. moreover, evaluation results indicate our learning schema can also consistently bring further improvements to the performance of existing methods with contrastive learning. ## methods in this section, we first formalize the spoken question answering tasks. furthermore, we introduce the key components of our method with self-supervised contrastive representation learning. next, we describe the design of our proposed temporal-alignment attention mechanism. lastly, we discuss how to incorporate contrastive loss into our self-supervised training schema. ## experiments in this section, we conduct experiments to compare our proposed method with various baselines and state-of-the-art approaches. ## conclusions spoken question answering requires fine-grained understanding of both speech and text data. to this end, we propose a novel training scheme for spoken question answering. by carefully designing several auxiliary tasks, we incorporate the self-supervised contrastive learning framework to capture consistency and coherence within speech documents and text corpus without any additional data. we further propose a novel temporal-alignment strategy to align audio features and textual concepts by performing mutual attention over two modalities. our model achieves state-of-the-art performance on three sqa benchmark datasets. for future work, we will develop more effective auxiliary tasks to enhance the quality of answer prediction.