We present a technique which complements Hidden Markov Models by incorporating some lexicalized states representing syntactically uncommon words . ' Our approach examines the distribution of transitions , selects the uncommon words , and makes lexicalized states for the words . We perfor'med a part-of-speech tagging experiment on the Brown corpus to evaluate the resultant language model and discovered that this technique improved the tagging accuracy by 0.21 % at the 95 % level of confidence . Hidden Markov ' Models are widely used for statistical language modelling in various fields , e.g. , part-of-speech tagging or speech recognition ( Rabiner and Juang , 1986 ) . The models are based on Markov assumptions , which make it possible to view the language prediction as a Markov process . ' In general , we make the firstorder Markov ass'umptions that the current tag is only dependant on the previous tag and that the current word is only dependant on the current tag . These are very ' strong ' assumptions , so that the first-order Hidden Markov Models have the advantage of drastically reducing the number of its parameters . On the other hand , the assumptions restrict the model from utilizing enough constraints provided by the local context and the resultant model consults only a single category ' as the contex . A lot of effort has been devoted in the past to make up for the insufficient contextual information of the first-order probabilistic model . The second order Hidden Markov Models with " The research underlying this paper was supported t ) 3 " research grants fl'om Korea Science and Engineering Foundation . appropriate smoothing techniques show better performance than the first order models and is considered a state-of-the-art technique ( Merialdo , 1994 ; Brants , 1996 ) . The complexity of the model is however relatively very high considering the small improvement of the performance . Garside describes IDIOMTAG ( Garside et al . , 1987 ) which is a component of a part-ofspeech tagging system named CLAWS . ID-IOMTAG serves as a front-end to the tagger and modifies some initially assigned tags in order to reduce the amount of ambiguity to be dealt with by the tagger . IDIOMTAG can look at any combination of words and tags , with or without intervening words . By using the IDIOMTAG , CLAWS system improved tagging accuracy from 94 % to 96 - 97 % . However , the manual-intensive process of producing idiom tags is very expensive although IDIOMTAG proved fruitful . Kupiec ( Kupiec , 1992 ) describes a technique of augmenting the Hidden Markov Models for part-of-speech tagging by the use of networks . Besides the original states representing each part-of-speech , the network contains additional states to reduce the noun / adjective confusion , and to extend the context for predicting past participles from preceding auxiliary verbs when they are separated by adverbs . By using these additional states , the tagging system improved the accuracy from 95.7 % to 96.0 % . However , the additional context is chosen by analyzing the tagging errors manually . An automatic refining technique for Hidden Markov Models has been proposed by Brants ( Brants , 1996 ) . It starts with some initial first order Markov Model . Some states of the model are selected to be split or merged to take into account their predecessors . As a result , each of new states represents a extended context . With this technique , Brants reported a performance cquivalent to the second order Hidden Markov Models . In this paper , we present an automatic refining technique for statistical language models . First , we examine the distribution of transitions of lexicalized categories . Next , we break out the uncommon ones from their categories and make new states for them . All processes are automated and the user has only to determine the extent of the breaking-out . In this paper , we present a method for complementing the Hidden Markov Models . With this method , we lexicalize the Hidden Markov Model seletively and automatically by examining the transition distribution of each state relating to certain words . Experimental results showed that the selective lexicalization improved the tagging accurary from about 95.79 % to about 96.00 % . Using normal tests for statistical significance we found that the improvement is significant at the 95 % level of confidence . Tile cost for this imt~rovenmnt is minimal . The resulting network contains 210 additional lexicalized states which are found automatically . Moreover , the lexicalization will not decrease the tagging speed 2 , because the lexicalized states and their corresponding original states are exclusive in our lexicalized network , and thus the rate of ambiguity is not increased even if the lexicalized states are included . Our approach leaves much room for improvement . We have so far considered only the outgoing transitions from the target states . As a result , we have discriminated only the words with right-associativity . We could also discriminate the words with left-associativity by examining the incoming transitions to the state . Furthermore , we could extend the context by using the second-order context as represented in Figure l(c ) . We believe that the same technique presented in this paper could be applied to the proposed extensions .

Challenge: Hidden Markov Models used for language modeling are limited by the Markov assumptions and existing workarounds either make models complex or require expensive manual efforts.
Approach: They propose to complement Hidden Markov Models by first examining the transition distribution and breaking out new lexicalized states for uncommon words.
Outcome: The proposed model improves on a part-of-speech tagging task from Brown corpus by finding 210 lexicalized states automatically while keeping the additional cost minimal.