The Role of n -gram Smoothing in the Age of Neural Networks (2024)

Abstract: For nearly three decades, language models derived from the n -gram assumption held the state of the art on the task. The key to their success lay in the application of various smoothing techniques that served to combat overfitting. However, when neural language models toppled n -gram models as the best performers, n -gram smoothing techniques became less relevant. Indeed, it would hardly be an understatement to suggest that the line of inquiry into n -gram smoothing techniques became dormant. This paper re-opens the role classical n -gram smoothing techniques may play in the age of neural language models. First, we draw a formal equivalence between label smoothing, a popular regularization technique for neural language models, and add- \lambda smoothing. Second, we derive a generalized framework for converting any n -gram smoothing technique into a regularizer compatible with neural language models. Our empirical results find that our novel regularizers are comparable to and, indeed, sometimes outperform label smoothing on language modeling and machine translation.

## introduction let σ be an alphabet. @xcite a language model is a probability distribution p over σ * , the set of all strings @xmath0. a fundamental task in natural language processing (nlp) is to estimate a language model-often from a parametric family-that places a high probability on held-out, humangenerated text. a common design choice is to construct a locally normalized language model, i.e., one which factorizes autoregressively foot_1 as p(x) = p(eos | x) with maximum-likelihood estimation (mle), a language model q θ is trained to match p d , the empirical distribution induced by a dataset d. however, we can also modify (smooth) p d into pn d and train a language model qθ on pn d . we show that the latter can be thought of as training qθ with a regularized maximum-likelihood objective. for years, the best parametric families of language models for this estimation task applied the n-gram assumption, detailed below. assumption 1.1 (n-gram assumption). a language model obeys the n-gram assumption if the following conditional independence holds assumption 1.1 was, historically, considered a practically effective manner to fight the curse of dimensionality, despite its inability to attend to contexts longer than n-1 words and thus capture long-range dependencies. @xcite the maximum-likelihood estimator of a model q n mle under the n-gram assumption is straightforward to derive. @xcite indeed, one can express it simply using counts of sub-string occurrences: in the context of nlp, add-λ smoothing has often served a pedagogical purpose and is thus commonly taught, but its efficacy in practice was considered limited (eisner, 2023). however, researchers developed more sophisticated related smoothing methods (jelinek, 1980; katz, 1987; ney et al., 1995) that were useful in practice. ney et al. (1995), for example, was long considered the best available method (chen and goodman, 1999) and its popularity inspired a principled bayesian interpretation (teh, 2006). interestingly though, add-λ smoothing still has a place in today's literature; as we prove in §2, it is identical to a regularization technique called label smoothing, common in the training of (conditional) neural language models (pereyra et al., 2017; meister et al., 2020), and particularly common in machine translation (costa-jussà et al., 2022). in this context, we present the primary theoretical research question addressed in our paper. if add-λ smoothing is equivalent to label smoothingan effective regularization technique for training today's neural language models-can we reverseengineer further regularization methods starting from other n-gram smoothing methods? given that add-λ smoothing was not known to perform well compared to other smoothing techniques in the context of n-gram language models, it is natural to suspect that reverse-engineered regularizers based on empirically more successful smoothing techniques may indeed outperform label smoothing. to this end, we derive a straightforward relationship between training on a smoothed empirical distribution and additive regularization methods, e.g., entropy regularization. using that relationship, we further show that any smoothing method can be reformulated as an additive regularization of the standard maximum-likelihood objective, and, hence, we provide an explicit way to construct the regularizer that can be applied to the training of any (neural) language model, helping connect classical smoothing methods and modern language models by introducing a way to incorporate smoothing techniques into the training of neural language models. we complement our theoretical analysis by empirically verifying the validity of our proposed methods on two small-scale datasets and observe that some regularizers based on more complex n-gram smoothing techniques do indeed perform better than label smoothing on language modeling and machine translation. ## label smoothing and add-λ smoothing in this section, we derive a formal relationship between the add-λ smoothing applied to n-gram models and label smoothing. the relationship is based on a regularizer that, when applied during mle, simulates add-λ smoothing exactly. additionally, we contend that, in this sense, label smoothing generalizes add-λ smoothing from a method applicable only to n-gram language models to one that can be used with neural language models. @xcite to the best of the authors' knowledge, this derivation and the relationship it exposes are novel. we re-use this meta paradigm in §4 to derive novel regularizers, which we then compare experimentally to label smoothing. ## a generalized framework in §2, we evince a connection between add-λ smoothing and regularization of the maximumlikelihood objective. however, the derivation we formalize (cf. app. a.1.2) is tedious and long. moreover, it exploits several specific properties of add-λ smoothing. performing such a derivation for each smoothing technique individually would be laborious and further, it would hinder building intuitions about the relationships between different methods. luckily, we can introduce a more general framework. specifically, in this section, we propose a framework that allows us to formulate equivalent regularizers for any smoothing technique and apply them to the training of neural language models. ## conclusion in this work, we re-imagine the application of classical n-gram smoothing techniques in the context of modern neural nlp models. for several of these historic methods, we derive equivalent, differentiable regularizers that can be added to neural models' training objectives. we present these results within a generalized framework that allows for insights about the smoothing methods themselves and their relationships to each other. we apply these smoothing methods in the training of neural language models and machine translation models. we find that our smoothing-based regularizers outperform label smoothing and standard mle in language modeling, while some methods also achieve competitive results with label smoothing for machine translation.