CIF - PT : Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training (2023)

Abstract: Speech or text representation generated by pre-trained models contains modal-specific information that could be combined for benefiting spoken language understanding (SLU) tasks. In this work, we propose a novel pre-training paradigm termed Continuous Integrate-and-Fire Pre-Training (CIF-PT). It relies on a simple but effective frame-to-token alignment: continuous integrate-and-fire (CIF) to bridge the representations between speech and text. It jointly performs speech-to-text training and language model distillation through CIF as the pre-training (PT). Evaluated on SLU benchmark SLURP dataset, CIF-PT outperforms the state-of-the-art model by 1.94% of accuracy and 2.71% of SLU-F1 on the tasks of intent classification and slot filling, respectively. We also observe the cross-modal representation extracted by CIF-PT obtains better performance than other neural interfaces for the tasks of SLU, including the dominant speech representation learned from self-supervised pre-training.

## introduction spoken language understanding (slu) plays a key role in speech interaction systems such as spoken dialogue systems, voice assistants, automated calling robots, etc. it focuses on extracting key information and making predictions from audio signals of human speech @xcite . traditional methods decompose slu into two cascading tasks: automated speech recognition (asr) and natural language understanding (nlu), where audio signals are first transcribed into texts, and then processed by a text-based language understanding model. in the cascading scheme, the errors of asr module will be accumulated in the nlu module and degrade the final performance. moreover, predicted text of asr module may not be the ideal interface for the language understanding task. for example, acoustic information such as intonation and pitch that may be helpful for understanding tasks are lost after asr. to tackle the problems above, resent researches employ endto-end approaches for slu @xcite @xcite , where the language understanding is directly performed from audio signals without explicitly utilizing predicted text of asr. for text-based language understanding tasks, pre-trained language models such as bert @xcite , roberta @xcite and gpt @xcite have achieved remarkable success. these models utilize self-supervised pre-training on large-scale unlabeled corpora to learn contextual representations in token or sentence level with rich syntactic and semantic knowledge @xcite , which significantly benefit downstream tasks such as nlu during fine-tuning. this self-supervised pre-training fashion has been extended into the representative learning on speech. researches such as wav2vec @xcite , hubert @xcite and data2vec @xcite focus on learning better frame-level contextual representations using unlabeled speech data, to improve the performance of asr as well as other speech processing tasks. for end-to-end slu, these self-supervised speech models have been proven to be powerful backbones on learning semantic representations @xcite . the self-supervised pre-training methods for speech mainly focus on leveraging speech data to model acoustic information @xcite on the frame level, while pre-trained language models work on higher token or sentence levels to encode linguistic knowledge @xcite . these two kinds of representation could be combined for better benefiting downstream tasks such as slu. the combination of speech and text representations can be performed by jointly pre-training on data of the two modalites @xcite , or distillating one pre-trained representations into another @xcite . in either way the framelevel speech representation needs to be aligned with the token-level textual representation. frame-totoken alignment methods such as forced alignment has been applied to speech-text joint pre-training @xcite . however, these alignment methods mainly rely on external models or rules, and can only generate hard alignment mapping that can not be updated in end-to-end training. on the other hand, aligning frames and tokens through cross-attention @xcite suffers from high complexity and lack of token timestamps that synchronized to frames. the frame-to-token alignment also plays a critical role in asr systems. various works, such as connectionist temporal classification (ctc) @xcite , listen, attend and spell (las) @xcite , rnn transducer (rnn-t) @xcite and continuous integrate-and-fire (cif) @xcite , focus on bringing effective alignment methods for better speech recognition performance. among these works, the cif alignment, which explicitly aggregates framelevel speech representations into token-level, is adopted in our work to combine with text representation. specifically, we propose a novel pretraining paradigm: continuous integrate-and-fire pre-training (cif-pt) for end-to-end slu. two pre-training tasks are included in cif-pt: the first task is speech-to-text modeling @xcite with cif alignment. in this work, asr task that transcribes speech to text is applied. the second task is language model distillation (lmd). since the integrated speech representation by cif is at token-level, token-level distillation from a pretrained language model can be performed to inject text-based linguistic knowledge into the representation. through the joint pre-training of the two tasks, cif-pt is able to generate representations with information from both speech and text modalites. we examine our cif-pt methods in downstream slu tasks including intent classification and slot filling. on slu benchmark slurp @xcite dataset, the end-to-end slu model with cif-pt outperforms the state-of-the-art model by 1.94% of accuracy and 2.71% of slu-f1 on the tasks of intent classification and slot filling, respectively. the cross-modal representation extracted by cif-pt also shows its competitiveness in comparison of other neural interfaces @xcite utilized in slu. the obtained results and a series of experiments including ablation study and the pre-training on out-of-domain data demonstrate the effectiveness and generalization of cif-pt. cross-modal pre-training for slu in order to exploit information from speech and text for slu, jointly pre-training on both of speech and text data has been proposed. @xcite extends the masked language model (mlm) pre-training from bert into the mixture of audio and text data. in splat @xcite , a speech module and a language module are jointly pre-trained with token-level and sentencelevel alignment. another branch of researches focus on knowledge distillation from pre-trained language model into pre-trained speech encoder. @xcite utilize bert as a teacher to perform sentence-level knowledge distillation at the pre-training stage and target-specific distillation during fine-tuning. @xcite introduce cross-attention between text and speech and perform distillation on the attention heads for knowledge transfering. frame-to-token alignment in slu in speech-bert @xcite , forced alignment based on external asr engine is used to train the initial phonetic-semantic joint embedding. @xcite adopt a heuristic alignment approach in splat, where alignment scores is computed by the cosine similarity between the output embeddings of the pre-trained speech and text models. the cross-attention alignment is introduced in @xcite to capture the interactions between text tokens and speech frames. for speecht5, since the pre-training does not strictly rely on audio-text pair data, @xcite adopt shared codebook for speech and text representation and a diversity loss to encourage the alignment in latent space. ## method in this section, we present the architecture of our proposed continuous integrate-and-fire pre-training (cif-pt) method for slu. as shown in figure 1 , our end-to-end slu models go through two stages: cif-pt and slu training. during cif-pt, we employ two pre-training tasks: asr training with cif alignment and tokenlevel language model distillation (lmd). these two tasks help the model learn contextual representation of the speech features aligned to the tokens with high level linguistic knowledge. after cif-pt, the pre-trained parameters including the speech encoder and cif part are used for downstream slu tasks such as intent classification and slot filling. ## conclusion in this work, we propose a new pre-training paradigm: continuous integrate-and-fire pre-training (cif-pt) for end-to-end slu. cif serves as a bridge connecting speech and text modality: on the one hand, it integrates speech representation into token-level through its frame-to-token alignment ability learned from asr pre-training task. on the other hand, it support one-to-one transfer of the textual knowledge into the integrated tokenlevel speech representation via the pre-training of language model distillation. after cif-pt, we obtain a cross-model representation that is used as neural interface into down-streaming slu tasks. evaluated on the largest slu benchmark of slurp, cif-pt creates new state-of-the-art result on both of ic and sf tasks. we further validate the effectiveness and generalization of cif-pt by a series of experiments including ablation study and the pre-training on out-of-domain data. we also observe the cross-modal representation extracted by cif-pt shows its competitiveness in comparison with other neural interfaces on slu. we believe that cif-pt has the potential to better encode long-form speech content (e.g. spoken paragraph) through its language model distillation, and will explore to combine it with llm methods like chatgpt to further empower spoken language understanding (slu) systems.