CTAL : Pre-training Cross-modal Transformer for Audio-and-Language Representations (2021)

Abstract: Existing audio-language task-specific predictive approaches focus on building complicated late-fusion mechanisms. However, these models are facing challenges of overfitting with limited labels and low model generalization abilities. In this paper, we present a Cross-modal Transformer for Audio-and-Language, i.e., CTAL, which aims to learn the intra-modality and inter-modality connections between audio and language through two proxy tasks on a large amount of audio-and-language pairs: masked language modeling and masked cross-modal acoustic modeling. After fine-tuning our pre-trained model on multiple downstream audio-and-language tasks, we observe significant improvements across various tasks, such as, emotion classification, sentiment analysis, and speaker verification. On this basis, we further propose a specially-designed fusion mechanism that can be used in fine-tuning phase, which allows our pre-trained model to achieve better performance. Lastly, we demonstrate detailed ablation studies to prove that both our novel cross-modality fusion component and audio-language pre-training methods significantly contribute to the promising results. The code and pre-trained models are available at https://github.com/tal-ai/CTAL_EMNLP2021 .

## introduction speech processing (sp) requires the understanding of a set of acoustic and language content, including phonemes, tones, words and semantic meanings. different from human, who benefit from selflearning through real-world experiences, current sp methods are more like narrow experts relying heavily on a large number of task-specific human annotations and a small change may cause the learning process to start all over again. in recent years, pre-training for single modality, such as language and audio signals, is widely explored due to its ease-of-use and competent generalization to various downstream tasks. * the corresponding author: zitao @xcite in the field of nlp, pre-trained models, such as bert @xcite , roberta @xcite , xlnet @xcite and gpt2 @xcite , share the same idea of first leveraging large-scale unlabeled corpus to perform contextualized language model pre-training then fine-tuning the model to adapt to downstream tasks, such as machine reading comprehension @xcite , question answering @xcite and natural language inference @xcite , etc. following the success of pre-trained models in nlp, bert-like models are also applied to sp community @xcite , where robust audio representations are learned through an audio-style self-supervised context prediction task. despite these influential unimodal methods, for tasks at the intersection of audio and language, such as speech emotion classification @xcite , speaker verification @xcite and sentiment analysis @xcite , large-scale pre-training for the modality-pair of audio and language is barely explored. the previous attempt is to train taskspecific experts upon the concatenation of language representations and audio representations in a late fusion manner @xcite @xcite @xcite @xcite , without any generic audio-and-language pre-training. these task-specific experts will suffer from the overfitting problem when trained with limited data. meanwhile, due to the heterogeneity across language and audio modalities, late fusion of high-level representations lacks surface-level cross-modal alignment and complementation during the pre-training phase. therefore, we propose ctal, a pre-training cross-modal transformer for audio-and-language representations, and has shown its strong performance on three established audio-and-language tasks: emotion classification @xcite , sentiment analysis @xcite and speaker verification @xcite . we propose multimodal transformer as our backbone model, which consists of two modules, a language stream encoding module which adapts word as input element, and a text-referred audio stream encoding module which accepts both frame-level melspectrograms and token-level output embeddings from the language stream encoding module as input elements. in order to learn both intra-and inter-modalities connections, we pre-train our model with two tasks: (1) masked language modeling (mlm); and (2) masked cross-modal acoustic modeling (mcam). different from unimodal pre-training, e.g., masked acoustic modeling in mockingjay @xcite , our cross-modal pre-training is able to reconstruct masked audio features from both intra-and inter-modalities information. on the basis of our pre-trained model, a regularization term based on feature orthogonality is introduced during the model fine-tuning stage, which is designed to ensure that features of different modalities provide information from different perspectives. please notice that this orthogonal regularization mechanism is general and not limited to audio-language tasks. the main contributions of our paper are listed as follows: ## our approach in this section, we first present our cross-modal pre-training framework ctal, including details of text and audio pre-processing and encoding modules for separate modalities. then we present our pre-training tasks. in the end, we propose a novel fusion mechanism which can be utilized in the finetuning stage. following conventions, we use bold upper case letters to represent matrices and bold lower case letters to represent vectors. ## conclusion in this work, we proposed ctal, a novel pretraining cross-modal transformer to learn effec-tive representations for audio-and-language tasks. it is pre-trained with two pre-training tasks on a large-scale dataset of audio-and-language pairs. extensive empirical analysis demonstrates that our pre-trained model improves various speech understanding performance significantly and achieves new sota results. besides, we show the effectiveness of different model components and the competent generalization capability via detailed ablation studies and analysis.