CCSRD : Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation (2023)

Abstract: Deep neural networks have demonstrated their capacity in extracting features from speech inputs. However, these features may include non-linguistic speech factors such as timbre and speaker identity, which are not directly related to translation. In this paper, we propose a content-centric speech representation disentanglement learning framework for speech translation, CCSRD, which decomposes speech representations into content representations and non-linguistic representations via representation disentanglement learning. CCSRD consists of a content encoder that encodes linguistic content information from the speech input, a non-content encoder that models non-linguistic speech features, and a disentanglement module that learns disentangled representations with a cyclic reconstructor, feature reconstructor and speaker classifier trained in a multi-task learning way. Experiments on the MuST-C benchmark dataset demonstrate that CCSRD achieves an average improvement of +0.9 BLEU in two settings across five translation directions over the baseline, outperforming state-of-the-art end-to-end speech translation models and cascaded models.

## introduction end-to-end (e2e) speech-to-text translation (st) aims to translate a source speech input into a target translation. compared with traditional cascaded st models, e2e st models avoid the issue of error propagation and exhibit low inference latency. recent works have made significant progress in e2e st, which enables it to even outperform traditional cascaded systems on language pairs like en-de and en-ru @xcite @xcite . despite the remarkable progress, e2e st is still confronted with challenges in learning desirable * *corresponding author speech representations. speech inputs to st encompass not only content information that is essential for translation but also non-linguistic factors such as pitch, timbre, prosody, speaker identity and so on. as a result, the speech encoder encodes not only the content information but also a range of non-linguistic speech elements. such non-linguistic factors may benefit translation, e.g., prosody as mentioned by @xcite . however, they may also introduce spurious correlations between speech inputs and target translations, undermining speech translation generalization. to mitigate this issue, we propose to explore speech representation disentanglement learning in the context of speech translation, which aims to separate content from non-linguistic factors (e.g., prosody, timbre). previous studies have consistently demonstrated the efficacy of representation disentanglement in improving model generalization @xcite @xcite . given the highly complex nature of speech features and the modality gap between text and speech, we argue that representation disentanglement could enable st to focus on content, reducing the negative influence from non-linguistic factors on speech translation modeling. specifically, we propose a content-centric speech representation disentanglement learning framework, termed as ccsrd, for end-to-end speech translation. we re-function the original encoder as the content encoder that encodes linguistic content information contained in the speech input. we introduce an additional encoder, noncontent encoder, to encode non-linguistic speech features. the decomposition of the speech input into content and non-linguistic factors for the two encoders is completed by a disentanglement module. the disentanglement module is trained in a multi-task learning way, which leverages three tasks: a cyclic reconstruction task to reduce the mutual information between content representation and non-content representation, a feature reconstruction task to ensure the retention of speech information, and a speaker classifier task to guide the training of the non-content encoder. additionally, we also employ a masking strategy to further improve disentanglement. it is noteworthy that our method does not require transcription to achieve representation disentanglement. therefore it can be used in scenarios that are short of transcription data or do not have transcription data at all. our work is hence significantly different from most previous works that heavily depend on transcription for speech translation. although disentangled speech representation learning is not new in the community of speech processing @xcite @xcite , to the best of our knowledge, this is the first attempt to learn disentangled speech representations for end-to-end speech translation. in a nutshell, our contributions are listed as follows. end-to-end speech translation to avoid error propagation in cascaded st and reduce inference latency, @xcite and @xcite propose end-to-end st that directly translates speech in the source language into text in the target language, without relying on the intermediate transcriptions. however, due to the inherent complexity and variation of speech signals and the scarcity of high-quality e2e st data, achieving satisfactory performance remains challenging. over the years, a variety of approaches have been proposed to address these issues, such as pre-training @xcite @xcite , multi-task learning @xcite @xcite , data augmentation @xcite , contrastive learning @xcite and knowledge distillation @xcite . most of these approaches focus on using the transcription data in speech data triplets to perform mt/asr tasks, pretrain model components and mitigate the modality gap between speech and text. significantly different from them, we attempt to improve translation quality by efficiently exploring the speech representation disentanglement to learn content-centric speech representations for st. representation disentanglement representation disentanglement refers to a learning paradigm in which models represent input signals through multiple separated dimensions or embeddings. therefore, it is always advantageous in obtaining representations that carry certain attributes or extract discriminative features. reconstruction based training @xcite @xcite is widely adopted in disentanglement learning and used to obtain disentangled representations. the application of representation disentanglement is extensive, including speech @xcite @xcite and natural language precessing @xcite . since speech often contains multiple factors, disentangled representation learning provides a way to extract different representations for different tasks like voice conversion @xcite , automatic speech recognition (chan and ghosh, 2022) and speaker recognition @xcite . our approaches are partially motivated by these efforts but are significantly different from them in two aspects. first, we make the first step to use representation disentanglement during the training stage of e2e st and propose ccsrd. second, we focus on the quality of the content representation for st, removing the additional encoder and disentanglement module during inference, while the previous works focus on the disentanglement of different speech factors for various speech tasks. disentanglement module non-content encoder content encoder translation decoder cnn wav2vec2.0 text embedding (a) overall model architecture (b) disentanglement module speaker classfier feature reconstruction cyclic reconstruction feature reconstruction speaker classifier + <@xmath0. masking speech encoder speaker id content feature predictor grl non-content feature predictor grl figure 1: diagram of the proposed content-centric speech representation disentanglement learning framework for e2e st. left: the overall architecture, where the text embedding layer is only used during the training under the mtl setting. the red arrow lines denote the data flow at inference. right: details of disentanglement module. it has three parallel tasks: cyclic reconstruction, speaker classifier and feature reconstruction. ## ccsrd in this section, we first introduce the overall model architecture of ccsrd and subsequently elaborate disentanglement learning, training and inference of ccsrd. ## conclusions in this work, we have presented ccsrd, an e2e st framework that explores speech representation disentanglement learning to capture content representations for st. experimental results validate the effectiveness of the proposed framework under both the st task setting and the multi-task setting. in-depth analyses demonstrate that ccsrd is capable of disentangling linguistic content from nonlinguistic speech factors. we would like to employ more speech factors to guide the training of the non-content encoder and explore the disentangled non-content representations for improving st in the future.