Can We Train a Language Model Inside an End-to-End ASR Model? - Investigating Effective Implicit Language Modeling (2022)

Abstract: Language models (LM) have played crucial roles in automatic speech recognition (ASR) to enhance end-to-end (E2E) ASR systems’ performance. There are two categories of approaches: finding better ways to integrate LMs into ASR systems and adapting on LMs to the task domain. This article will start with a reflection of interpolation-based integration methods of E2E ASR’s scores and LM’s scores. Then we will focus on LM augmentation approaches based on the noisy channel model, which is intrigued by insights obtained from the above reflection. The experiments show that we can enhance an ASR E2E model based on encoder-decoder architecture by pre-training the decoder with text data. This implies the decoder of an E2E model can be treated as an LM and reveals the possibility of enhancing the E2E model without an external LM. Based on those ideas, we proposed the implicit language model canceling method and then did more discussion about the decoder part of an E2E ASR model. The experimental results on the TED-LIUM2 dataset show that our approach achieves a 3.4% relative WER reduction compared with the baseline system, and more analytic experiments provide concrete experimental supports for our assumption.

## introduction in the 1980s, a significant step was achieved by introducing the acoustic model (am) and language model (lm) into asr framework. from that time, the methodology of asr shifted from the more intuitive template-based approach (a straightforward pattern recognition paradigm) towards a more rigorous statistical modeling framework @xcite . moreover, those two concepts of am and lm became the foundation of asr that we are familiar with nowadays. relying solely on acoustic observations proved to be insufficient to achieve human-like performance. with the rapid development of deep learning techniques, many powerful neural network-based systems were invented in the new century. among them, various end-to-end (e2e) systems become prevail @xcite @xcite @xcite , which are benefited from sufficient computing power and data sets. from this stage, e2e asr becomes the mainstream of modern asr techniques. we have emphasized the importance of lms in asr, but how an independent lm can be utilized in an e2e system? the answers are lm integration, and lm adaptation @xcite @xcite . lm integration increases accuracies of e2e asr systems in practical indeed. however, intuitively, if an e2e asr model is powerful enough, there is no need for an extra lm. so, the question becomes how an lm can benefit an e2e asr system. to be more specific, we need to figure out what happens when we try to integrate the e2e asr model with an lm and how to adapt an lm to an asr domain. furthermore, can we reveal the capability of language modeling in an e2e asr model? in this paper, we try to answer those questions theoretically and experimentally. firstly, we analyzed shallow fusion of lm integration mathematically using lm adaptation framework @xcite . then, we proposed an implicit lm canceling method to fully control the language modeling functionality of an e2e asr system. finally, we discussed the feasibility that a decoder of an e2e asr model could be treated as an lm by experiments. to the best of our knowledge, we are the first to analyze the language modeling functionality of the decoder part in an e2e asr model. the rest of this paper is structured as follows. section 2 discusses the most common lm integration approach (shallow fusion) to explore its essence from the perspective of probability models. then we try to figure out a way to compose lm integration and lm adaptation tasks into a single method in section 3 and 4. in section 5, we analyze the result and reveal crucial insights about a decoder's characteristics in an e2e asr system from several experiments. we conclude the paper in section 6. ## implicit lm canceling method inspired by the density ratio approach, we propose to restructure the shallow fusion of eq.1 in a more general way: ## the implicit lm of an e2e asr system we can check the assumption that the decoder of an e2e model can be treated as an lm from two aspects: the decoder of this e2e model should potentially be an lm because it is an auto-regression model, just like a normal language model, which receives a token sequence and outputs the next token. it has transformer layers for memorizing information from the training set. and from fig. 1 the multi-head attention layers which receive a word sequence and hidden states from the encoder are built relatively independent. so, we can assume the layers which did not receive hidden states directly from the encoder would be dominated by outputs (subword). those characteristics above fulfill the first aspect. we can check the second statement by sampling token sequences in an auto-regression manner or calculate its perplexity about an lm's behaviors. moreover, the above clarification leads to our new proposal: the decoder of an e2e model can be treated as an lm and be pre-trained on text data before e2e training to improve the e2e model. we will validate this in the following experiments.