When a trigram backoff language model is created from a large body of text, mgrams and bigrams that occur few times in the training t a t are often excluded from the model in order to de-Crease the model size. Generally, the elimination of n-grams with very low counts is believed to not significantly affect model performance. This project investigates the degradation of a trigram backoff model's perplexity and word error rates as bigram and trigram cutoffs are increased. The advantage of reduction in model size is compared to the increase in word e m r rate and perplexity scores. More importantly, this project also investigates altemative ways of excluding bigrams and trigrams from a backoff language model, using criteria other than the number of times an n-gram occurs in the training text. Specifically, a difference method has been investigated where the difference in the logs of the original and backed off Uigram and bigram probabilities is used as a basis for n-gram exclusion from the model. We show that excluding mgrams and bigrams based on a weighted vmion of this difference method results in better perplexity and word error rate performance than excluding trigrams and bigrams based on counts alone. Cumnt collections of text for statistical language model rraining arc making the sparse training data problem less serious for certain domains. such as ARPA's Wall Street Journal corpus, which is part of the 305 million word Nonh American Business News collection. The more training text that is used for language model creation, the more unique word sequences are encountered that must be stored in the model. Thus, as training text size increases, language model size necessarily increases, which can lead to models that are too unwieldy and memory-demanding to be of practical use. This overabundance of training data will allow us, or more comctly force us, to be selective in choosing the training data that we use to create our models. We explore two methods of training u t pruning that allow for compact and efficient creation of trigram backoff lanl guage models. The effects of the original amount of training data on a scalddown model is also investigated. The backoff language model was developed by Kaa [2] to address the problems associated with sparse training data. Small counts result in unreliable estimates. The backoff model handles this type of sampling error by discounting the probability of low count even& and distributing the freed probability mass among unseen events. As the amount of training text used to create the backoff model increases, the number of unique mgrams and bi,orams increases. The language model will necessarily takes up more memory in order to store the additional information from the training text At some point, the model's memory requirements will exceed any practical system capacity. Therefore, we can either limit the amount of training data we use to develop the model, or take from a large amount of training text that portion which leads to the most reliable word predictions. Word sequences that occur the fewest number of times in a training text can lead to unreliable predictions. This idea has led to the popular cutoff method of training text reduction, where only information about the most frequently occurring bigrams and trigrams is included in the language model. This method will be explored in depth in Section 3.1. However, we also need to consider the word sequences for which the model would not make a good prediction if they were eliminated from the model. This idea has led to the development of the weighted difference method of training text Eduction, which will be introduced in Section 3.2. The cutoff method of training text pruning excludes from the Ianguage model those bigrams and trigrams that occur infrequently. The motivation for this method lies in the argument that there is not much difference between a mgxam or bigram occurring once in a t a t of millions of words and it not occuring at all. Just by excluding those n-grams with a count of one from a model, a significant savings in memory can be achieved, In a typical training text, roughly 80% of unique aigram sequences occur only once. This idea can be extended fuaher to bigrams and trigrams that occur any number of times. We can designate a trigram cutoff and a bigram cutoff, and 232
REF showed that this method is more effective than the traditional cut-off method.
Scalable backoff language models
Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96
Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96
Computer Science