This paper investigates model merging , a technique for deriving Markov models from text or speech corpora . Models are derived by starting with a large and specific model and by successively combining states to build smaller and more general models . We present methods to reduce the time complexity of the algorithm and report on experiments on deriving language models for a speech recognition task . The experiments show the advantage of model merging over the standard bigram approach . The merged model assigns a lower perplexity to the test set and uses considerably fewer states . Hidden Markov Models are commonly used for statistical language models , e.g. in part-of-speech tagging and speech recognition ( Rabiner , 1989 ) . The models need a large set of parameters which are induced from a ( text- ) corpus . The parameters should be optimal in the sense that the resulting models assign high probabilities to seen training data as well as new data that arises in an application . There are several methods to estimate model parameters . The first one is to use each word ( type ) as a state and estimate the transition probabilities between two or three words by using the relative frequencies of a corpus . This method is commonly used in speech recognition and known as word-bigram or word-trigram model . The relative frequencies have to be smoothed to handle the sparse data problem and to avoid zero probabilities . The second method is a variation of the first method . Words are automatically grouped , e.g. by similarity of distribution in the corpus ( Pereira et al . , 1993 ) . The relative frequencies of pairs or triples of groups ( categories , clusters ) are used as model parameters , each group is represented by a state in the model . The second method has the advantage of drastically reducing the number of model parameters and thereby reducing the sparse data problem ; there is more data per group than per word , thus estimates are more precise . The third method uses manually defined categories . They are linguistically motivated and usually called parts-of-speech . An important difference to the second method with automatically derived categories is that with the manual definition a word can belong to more than one category . A corpus is ( manually ) tagged with the categories and transition probabilities between two or three categories are estimated from their relative frequencies . This method is commonly used for part-of-speech tagging ( Church , 1988 ) . The fourth method is a variation of the third method and is also used for part-of-speech tagging . This method does not need a pre-annotated corpus for parameter estimation . Instead it uses a lexicon stating the possible parts-of-speech for each word , a raw text corpus , and an initial bias for the transition and output probabilities . The parameters are estimated by using the Baum-Welch algorithm ( Baum et al . , 1970 ) . The accuracy of the derived model depends heavily on the initial bias , but with a good choice results are comparable to those of method three ( Cutting et al . , 1992 ) . This paper investigates a fifth method for estimating natural language models , combining the advantages of the methods mentioned above . It is suitable for both speech recognition and partof-speech tagging , has the advantage of automatically deriving word categories from a corpus and is capable of recognizing the fact that a word belongs to more than one category . Unlike other techniques it not only induces transition and output probabilities , but also the model topology , i.e. , the number of states , and for each state the outputs that have a non-zero probability . The method is called model merging and was introduced by ( Omohundro , 1992 ) . The rest of the paper is structured as follows . We first give a short introduction to Markov mo-dels and present the model merging technique . Then , techniques for reducing the time complexity are presented and we report two experiments using these techniques . We investigated model merging , a technique to induce Markov models from corpora .. The original procedure is improved by introducing constraints and a different initial model . The procedures are shown to be applicable to a transliterated speech corpus . The derived models assign lower perplexities to test data than the standard bigram model derived from the same training corpus . Additionally , the merged model was much smaller than the bigram model . The experiments revealed a feature of model merging that allows for improvement of the method 's time complexity . There is a large initial part of merges that do not change the model 's perplexity w.r.t , the test part , and that do not influence the final optimal model . The time needed to derive a model is drastically reduced by abbreviating these initial merges . Instead of starting with the trivial model , one can start with a smaller , easy-to-produce model , but one has to ensure that its size is still larger than the optimal model .

Challenge: Statistical language models based on the Hidden Markov Model need a large set of parameters that is optimal to assign high probabilities to seen data.
Approach: They propose to use model merging to combine the existing methods by starting with a large and specific model with smaller and more general models.
Outcome: The proposed approach has the advantage over the bigram approach which is assigning a lower perplexity with fewer states for a speech recognition task.