Residual Learning of Neural Text Generation with n-gram Language Model (2022)

Abstract: N-gram language models (LM) has been largely superseded by neural LMs as the latter exhibits better performance. However, we find that n-gram models can achieve satisfactory performance on a large proportion of testing cases, indicating they have already captured abundant knowledge of the language with relatively low computational cost. With this observation, we propose to learn a neural LM that fits the residual between an n-gram LM and the real-data distribution. The combination of n-gram LMs and neural LMs not only allows the neural part to focus on deeper understanding of the language, but also provides a flexible way to customize a LM by switching the underlying n-gram model without changing the neural model. Experimental results on three typical language tasks (i.e., language modeling, machine translation, and summarization) demonstrate that our approach attains additional performance gains over popular standalone neural models consistently. We also show that our approach allows for effective domain adaptation by simply switching to a domain-specific n-gram model, without any extra training.

## introduction n -gram language model (lm) was widely adopted in a broad range of natural language processing (nlp) applications, such as input method @xcite , statistical machine translation @xcite , and audio speech recognition @xcite . however, with the development of deep learning, neural lms have gradually taken the place of n-gram lms and became the new standard in recent literature @xcite @xcite . one critical reason is the superior performance of neural lms, e.g., the gpt-2 model @xcite can generate text near the human level, outperforming n-gram lms by large margins. ppl 1 10 100 index of bins 1 2 3 4 5 5-gram gpt-2 5.83224260940605 47.2420739963172 117.796423416745 203.440992166668 775.02 7.16233646523529 26.0673702595165 44.8518225772504 52.8395811179091 428.4 despite that neural lms have surpassed n-gram models at the macro level, we find that n-gram lms are still attractive: they are able to achieve satisfactory performance on a large proportion of testing cases at a much lower cost than neural lms. as observed in figure 1 , our preliminary experiments show that the performance of 5-gram lm is close to the gpt-2 model trained from scratch on 3 out of 5 bins ( 1, 2, and 5). moreover, the performance of 5-gram on the first bin is slightly better than gpt-2. because training a neural lm is much more expensive, spending effort on learning the knowledge that can be cheaply captured by n-gram seems a waste. inspired by the above observation, we propose to learn a neural lm that focuses on the information gap that has not been captured by an n-gram model: f := g -q, where g and q are the real-data distribution and the n-gram prediction distribution respectively, which is in a similar spirit to residual learning @xcite . more concretely, we combine the logits (the unnormalized probabil-ity scores before softmax layer) of a neural model and those derived from an n-gram model. the joint neuro-symbolic system at least brings two appealing characteristics. first, since the neural model stands on the shoulders of the shallow n-gram lm, it can concentrate on deeper understanding. second, the underlying n-gram lm can be purposefully switched without changing the neural model, which offers great flexibility in tackling scenarios such as domain adaptation. that is, we can adapt the model to a specific domain by changing the underlying n-gram lm in a plug-and-play manner, without changing any parameters of the neural model. we conduct extensive experiments to evaluate the proposed approach. experiments on the standard benchmarks of three typical language tasks, including language modeling, machine translation, and summarization, show that our approach can improve the performance of recent state-of-theart neural models consistently and considerably. for example, our approach outperforms popular baseline models by at least 0.7 ppl scores on the wikitext-103 dataset for language modeling, 0.65 bleu scores on average on iwslt datasets for machine translation, and 0.36 rouge-l scores on the cnn/dailymail dataset for summarization. moreover, on the language modeling task, when switching the underlying n-gram lm to a particular domain-specific one (e.g., it, koran, law, medical, and subtitles) in a plug-and-play manner, our model can reduce the ppl by 5.4 points on average without any domain-specific training of the neural part. remarkably, the performance of our approach is even close to fine-tuning the whole model on domain-specific corpora. our contributions are three-fold: language model the n-gram language model (lm) has been widely used in lots of applications of natural language processing (nlp) since a long time ago @xcite . the emergence of advanced smoothing technologies makes the n-gram model able to provide a better estimation of human languages @xcite @xcite . in statistical machine translation @xcite and automatic speech recognition @xcite , the decoder-side n-gram model is critical to estimate the quality of generated candidates. in recent literature on input methods, the n-gram lm is still the most popular choice for providing word suggestions @xcite , because of its low cost and low latency. however, with the development of deep neural networks, the macro-level performance of neural lm has surpassed that of n-gram lm by a large margin. comparing with the n-gram lm, one big advantage of the neural lm basing on recurrent neural network @xcite and attention neural network @xcite is their ability to modeling long-distance dependencies @xcite . the success of neural lm can also be observed in the big improvement achieved in lots of downstream tasks, e.g., text generation @xcite @xcite @xcite , machine translation @xcite @xcite and summarization @xcite @xcite . although neural lm has outperformed n-gram lm at the macro level, we find that n-gram lm can achieve satisfactory performance on a large portion of testing cases. since the training cost of neural lm is much more expensive and the model capacity is fixed, we hypothesize that it is not necessary to train the neural lm to learn the knowledge that can be captured by n-gram lm at a much lower cost. therefore, we propose a residual learning method to let the neural lm learn the gap of knowledge that has not been captured by n-gram lm. residual learning residual learning is a useful technique for lots of neural networks in computer vision (cv) and natural language processing (nlp). @xcite propose deep residual learning to alleviate the training difficulties of deep models, which has been the backbone of lots of tasks in cv. in nlp, @xcite and @xcite use the residual learning technique to train deep recurrent neural networks for text generation. different from previous works that conduct residual learning over different layers, @xcite propose to aggregate the information of historical predictions using residual learning. in @xcite , they use the residual learning to propagate attention scores across different layers of the transformer-based model. most of these works conduct residual learning over homogeneous model structures, e.g., stacked identical layers of the same model. in our work, we use residual learning to combine the neural and symbolic models, i.e., learn a neural lm that approximates the information that has not been captured by the n-gram model. ## background models that estimate the probabilities of sequences of words are called language models (lm) @xcite . let @xmath0, x 2 , ..., x l } be a sequence of words with length l. the probability of p (x) can be formalized according to the chain rule of probability: in this section we will briefly introduce two kinds of language models, the n-gram and neural language models, to compute the probability in eq. ( 1 ). ## experiments in our work, we consider three kinds of natural language generation tasks: language modeling, machine translation, and summarization. for the language modeling task, we first evaluate the performance of our approach on the standard setting of the language modeling task. then we turn to a domain adaptation setting. ## conclusion and future work this work aims to learn a neural lm that approximates the information that has not been captured by n-gram lm. to achieve this goal, we propose a residual learning approach to force the two neural and symbolic models, i.e., the neural lm and n-gram lm, to learn complementary information. we conduct extensive experiments to evaluate the performance of the proposed approach. in our experiments, we find that our neuro-symbolic system can not only improve the performance of recent state-of-the-art neural models consistently and considerable on three typical language tasks (including language modeling, machine translation, and summarization) but also exhibits a good plug-and-play property on the multi-domain language modeling task. the n-gram lm has lots of attractive properties that we have not explored in this work. first, the n-gram model has good interpretability. the behavior of n-gram lm is easier to understand than the weights of neurons from the perspective of humans. in the future, we want to leverage the property of the n-gram model to better understand the decision-making process of the neural lm. second, controlling the system predictions through the n-gram model may have a big potential. as observed in our multi-domain experiments, we are able to customize an lm by switching the underlying n-gram model without changing the neural part. it is also interesting to explore how to control the model output at a fine-grained level using the n-gram lm.