In the face of sparsity , statistical models are often interpolated with lower order ( backoff ) models , particularly in Language Modeling . In this paper , we argue that there is a relation between the higher order and the backoff model that must be satisfied in order for the interpolation to be effective . We show that in n-gram models , the relation is trivially held , but in models that allow arbitrary clustering of context ( such as decision tree models ) , this relation is generally not satisfied . Based on this insight , we also propose a generalization of linear interpolation which significantly improves the performance of a decision tree language model . A prominent use case for Language Models ( LMs ) in NLP applications such as Automatic Speech Recognition ( ASR ) and Machine Translation ( MT ) is selection of the most fluent word sequence among multiple hypotheses . Statistical LMs formulate the problem as the computation of the model 's probability to generate the word sequence w 1 w 2 . . . w m ≡ w m 1 , assuming that higher probability corresponds to more fluent hypotheses . LMs are often represented in the following generative form : p(w m 1 ) = m i=1 p(w i |w i-1 1 ) In the following discussion , we will refer to the function p(w i |w i-1 1 ) as a language model . Note the context space for this function , w i-1 1 is arbitrarily long , necessitating some independence assumption , which usually consists of reducing the relevant context to n -1 immediately preceding tokens : p(w i |w i-1 1 ) ≈ p(w i |w i-1 i-n+1 ) These distributions are typically estimated from observed counts of n-grams w i i-n+1 in the training data . The context space is still far too large ; therefore , the models are recursively smoothed using lower order distributions . For instance , in a widely used n-gram LM , the probabilities are estimated as follows : p(w i |w i-1 i-n+1 ) = ρ(w i |w i-1 i-n+1 ) + ( 1 ) γ(w i-1 i-n+1 ) • p(w i |w i-1 i-n+2 ) where ρ is a discounted probability 1 . In addition to n-gram models , there are many other ways to estimate probability distributions p(w i |w i-1 i-n+1 ) ; in this work , we are particularly interested in models involving decision trees ( DTs ) . As in n-gram models , DT models also often utilize interpolation with lower order models ; however , there are issues concerning the interpolation which arise from the fact that decision trees permit arbitrary clustering of context , and these issues are the main subject of this paper . The main contribution of this paper is the insight that in the standard recursive backoff there is an implied relation between the backoff and the higher order models , which is essential for adequate performance . When this relation is not satisfied other interpolation methods should be employed ; hence , we propose a generalization of linear interpolation that significantly outperforms the standard form in such a scenario .

Challenge: They show that current statistical language models assume a relation between the backoff and the higher models although it is not always guaranteed.
Approach: They propose a generalization of linear interpolation for cases when the required relation by current methods is not satisfied.
Outcome: The proposed method significantly improves the performance of a decision tree language model in concerned cases.