Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis (2023)

Abstract: Training a high performance end-to-end speech (E2E) processing model requires an enormous amount of labeled speech data, especially in the era of data-centric artificial intelligence. However, labeled speech data are usually scarcer and more expensive for collection, compared to textual data. We propose Latent Synthesis (LaSyn), an efficient textual data utilization framework for E2E speech processing models. We train a latent synthesizer to convert textual data into an intermediate latent representation of a pre-trained speech model. These pseudo acoustic representations of textual data augment acoustic data for model training. We evaluate LaSyn on low-resource automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. For ASR, LaSyn improves an E2E baseline trained on LibriSpeech train-clean-100, with relative word error rate reductions over 22.3% on different test sets. For SLU, LaSyn improves our E2E baseline by absolute 4.1% for intent classification accuracy and 3.8% for slot filling SLU-F1 on SLURP, and absolute 4.49% and 2.25% for exact match (EM) and EM-Tree accuracies on STOP respectively. With fewer parameters, the results of LaSyn are competitive to published state-of-the-art works. The results demonstrate the quality of the augmented training data.

## introduction in the data-centric artificial intelligence era, large quantity and high quality training data are essential for good performance of natural language processing (nlp) models including speech processing models. a conventional speech processing system is usually cascaded with an automatic speech recognition (asr) module and an nlp module. for example, in spoken language understanding (slu) which predicts semantic information from speech input, the system first transcribes input speech into text with asr, then pipes the text output to the natural language understanding (nlu) model for text analysis. an end-to-end (e2e) speech processing system leverages a single model which takes the input speech and performs spoken language processing tasks simultaneously. e2e models draw increasing attention due to less computational complexity and error propagation mitigation @xcite @xcite @xcite . however, a challenge of e2e model training is the collection of enormous annotated spoken data, which are significantly more expensive to collect compared with the text-only counterpart. in contrast, for a cascaded system, the asr module and nlp module are trained separately with paired speech-transcription data and annotated textual data respectively. separated types of data are usually more readily available and thus lower data collection costs. as the amount of high quality training data is critical for an e2e model, a strategy to alleviate the inadequate spoken data problem with more abundant textual data. two approaches have been proposed for utilizing textual data for e2e speech models in the literature. the first is modality conversion which utilizes a text-to-speech (tts) system to convert text into speech @xcite . the disadvantage is the requirement for a high-quality expressive tts system. another approach is unified representation learning for matching latent representations of speech and text with alignment losses @xcite . given the significant difference between speech and text, aligning the hidden latent space of the two modalities is challenging. we propose latent synthesis (lasyn), a method to utilize text-only data for e2e speech processing models. lasyn can be seen as an integration of the above two ideas. we train a latent synthesis model which synthesizes textual data into an intermediate latent representation of a pre-trained speech model. compared to modality conversion, speech latent representation contains fewer details and redundancy than the original speech signal, thus is easier to synthesize. compared to unified representation learning, instead of aligning two modalities of huge difference, lasyn learns to map the text into the latent representation of speech directly. we evaluate lasyn on low-resource asr and slu tasks. low-resource asr has gained big progress with the advancement of self-supervised speech pre-training @xcite @xcite . further performance improvement still relies on external language models @xcite . we show that lasyn allows an e2e asr model to utilize text-only data effectively without external language models, and outperforms asr models with external language models. we further evaluate lasyn on two publicly available datasets for slu tasks, namely slurp @xcite and spoken task oriented semantic parsing (stop) @xcite . lasyn achieves comparable performance to the state-of-the-art (sota) slu models but with significantly fewer model parameters. we summarize our contributions as follows: this paper is organized as follows. in the next section, we discuss related works of lasyn. in section 3, we discuss the model structure and training of lasyn. we present experimental setup and results in section 4, and ablation studies on slu tasks in section 5. finally, we conclude our work in section 6. ## related works in this section, we discuss the prior works of modality conversion and unified representation learning related to lasyn. laptev et al. (2020) shows that tts data augmentation improves asr performance in a low-resource setting. @xcite further shows that the diversity and quality of the tts system are important for asr data augmentation. @xcite demonstrates similar representations derived from synthesized speech help downstream asr tasks. @xcite confirms the effectiveness of speech synthesis for e2e slu models, either as a sole source of training data or as a form of data augmentation. @xcite utilizes artificially synthesized speech to adapt a slu model based on a recurrent neural network transducer. @xcite demonstrates the effectiveness of a multispeaker tts system under a low-resource slu setting. @xcite decouples the textto-semantic and semantic-to-acoustic tasks to realize a multi-speaker text-to-speech system. lasyn generates pseudo acoustic representations from text without requiring a vocoder for speech waveform generation. ## conclusion we present lasyn, a framework which enables efficient textual data utilization for e2e speech processing. by converting text into pseudo acoustic latent representation with a latent synthesizer, crossmodality knowledge transfer from textual data to e2e speech processing models is achieved. for the low-resource asr task with librispeech, lasyn achieves relative wer reduction from 22.3% to 40.5% on test sets, compared to our e2e baseline with the same model structure. the results are competitive to published works which utilize textual data through external language models. for slu tasks, lasyn improves over our e2e baseline by absolute 4.1% and 3.8% for for ic accuracy and sf slu-f1 on slurp, and absolute 4.49% and 2.25% of em and em-tree accuracies on stop. the results are competitive to published sota works with much fewer model parameters. future improvement of latent synthesizer should further bridge the gap between speech and textual modality, which we leave as next step.