Learning Spoken Language Representations with Neural Lattice Language Modeling (2020)

Abstract: Pre-trained language models have achieved huge improvement on many NLP tasks. However, these methods are usually designed for written text, so they do not consider the properties of spoken language. Therefore, this paper aims at generalizing the idea of language model pre-training to lattices generated by recognition systems. We propose a framework that trains neural lattice language models to provide contextualized representations for spoken language understanding tasks. The proposed two-stage pre-training approach reduces the demands of speech data and has better efficiency. Experiments on intent detection and dialogue act recognition datasets demonstrate that our proposed method consistently outperforms strong baselines when evaluated on spoken inputs. The code is available at https://github.com/MiuLab/Lattice-ELMo .

## introduction the task of spoken language understanding (slu) aims at extracting useful information from spoken utterances. typically, slu can be decomposed with a two-stage method: 1) an accurate automatic speech recognition (asr) system transcribes the input speech into texts, and then 2) language understanding techniques are applied to the transcribed texts. these two modules can be developed separately, so most prior work developed the backend language understanding systems based on manual transcripts @xcite @xcite . despite the simplicity of the two-stage method, prior work showed that a tighter integration between two components can lead to better performance. researchers have extended the asr 1-best results to n-best lists or word confusion networks in order to preserve the ambiguity of the transcripts. ( @xcite @xcite @xcite . another line of research focused on using lattices produced by asr systems. lattices are directed acyclic graphs (dags) that represent multiple recognition hypotheses. an example of asr lattice is shown in figure 1 . @xcite introduced latticernn, a variant of recurrent neural networks (rnns) that generalize rnns to latticestructured inputs in order to improve slu. zhang and yang (2018) proposed a similar idea for chinese name entity recognition. sperber et al. (2019); xiao et al. (2019); zhang et al. (2019) proposed extensions to enable the transformer model @xcite to consume lattice inputs for machine translation. @xcite proposed to adapt the transformer model originally pre-trained on written texts to consume lattices in order to improve slu performance. @xcite also found that utilizing lattices that represent multiple granularities of sentences can improve language modeling. with recent introduction of large pre-trained language models (lms) such as elmo @xcite , gpt @xcite and bert @xcite , we have observed huge improvements on natural language understanding tasks. these models are pre-trained on large amount of written texts so that they provide the downstream tasks with high-quality representations. however, applying these models to the spoken scenarios poses several discrepancies between the pre-training task and the target task, such as the domain mismatch between written texts and spoken utterances with asr errors. it has been shown that fine-tuning the pre-trained language models on the data from the target tasks can mitigate the domain mismatch problem @xcite . @xcite focused on pre-training a language model specifically for spoken content with huge amount of automatic transcripts, which requires a large collection of indomain speech. in this paper, we propose a novel spoken language representation learning framework, which focuses on learning contextualized representations of lattices based on our proposed lattice language modeling objective. the proposed framework consists of two stages of lm pre-training to reduce the demands for lattice data. we conduct experiments on benchmark datasets for spoken language understanding, including intent classification and dialogue act recognition. the proposed method consistently achieves superior performance, with relative error reduction ranging from 3% to 42% compare to pre-trained sequential lm. ## neural lattice language model the two-stage framework that learns contextualized representations for spoken language is proposed and detailed below. ## experiments in order to evaluate the quality of the pre-trained lattice lm, we conduct the experiments for two common tasks in spoken language understanding. ## conclusion in this paper, we propose a spoken language representation learning framework that learns contextualized representation of lattices. we introduce the lattice language modeling objective and a two-stage pre-training method that efficiently trains a neural lattice language model to provide the downstream tasks with contextualized lattice representations. the experiments show that our proposed framework is capable of providing high-quality representations of lattices, yielding consistent improvement on slu tasks.