Morphological disambiguation proceeds in 2 stages : ( 1 ) an analyzer provides all possible analyses for a given token and ( 2 ) a stochastic disambiguation module picks the most likely analysis in context . When the analyzer does not recognize a given token , we hit the problem of unknowns . In large scale corpora , unknowns appear at a rate of 5 to 10 % ( depending on the genre and the maturity of the lexicon ) . We address the task of computing the distribution p(t|w ) for unknown words for full morphological disambiguation in Hebrew . We introduce a novel algorithm that is language independent : it exploits a maximum entropy letters model trained over the known words observed in the corpus and the distribution of the unknown words in known tag contexts , through iterative approximation . The algorithm achieves 30 % error reduction on disambiguation of unknown words over a competitive baseline ( to a level of 70 % accurate full disambiguation of unknown words ) . We have also verified that taking advantage of a strong language-specific model of morphological patterns provides the same level of disambiguation . The algorithm we have developed exploits distributional information latent in a wide-coverage lexicon and large quantities of unlabeled data . The term unknowns denotes tokens in a text that can not be resolved in a given lexicon . For the task of full morphological analysis , the lexicon must provide all possible morphological analyses for any given token . In this case , unknown tokens can be categorized into two classes of missing information : unknown tokens are not recognized at all by the lexicon , and unknown analyses , where the set of analyses for a lexeme does not contain the correct analysis for a given token . Despite efforts on improving the underlying lexicon , unknowns typically represent 5 % to 10 % of the number of tokens in large-scale corpora . The alternative to continuously investing manual effort in improving the lexicon is to design methods to learn possible analyses for unknowns from observable features : their letter structure and their context . In this paper , we investigate the characteristics of Hebrew unknowns for full morphological analysis , and propose a new method for handling such unavoidable lack of information . Our method generates a distribution of possible analyses for unknowns . In our evaluation , these learned distributions include the correct analysis for unknown words in 85 % of the cases , contributing an error reduction of over 30 % over a competitive baseline for the overall task of full morphological analysis in Hebrew . The task of a morphological analyzer is to produce all possible analyses for a given token . In Hebrew , the analysis for each token is of the form lexeme-and-features 1 : lemma , affixes , lexical cate-gory ( POS ) , and a set of inflection properties ( according to the POS ) -gender , number , person , status and tense . In this work , we refer to the morphological analyzer of MILA -the Knowledge Center for Processing Hebrew 2 ( hereafter KC analyzer ) . It is a synthetic analyzer , composed of two data resources -a lexicon of about 2,400 lexemes , and a set of generation rules ( see ( Adler , 2007 , Section 4.2 ) ) . In addition , we use an unlabeled text corpus , composed of stories taken from three Hebrew daily news papers ( Aruts 7 , Haaretz , The Marker ) , of 42 M tokens . We observed 3,561 different composite tags ( e.g. , noun-sing-fem-prepPrefix : be ) over this corpus . These 3,561 tags form the large tagset over which we train our learner . On the one hand , this tagset is much larger than the largest tagset used in English ( from 17 tags in most unsupervised POS tagging experiments , to the 46 tags of the WSJ corpus and the about 150 tags of the LOB corpus ) . On the other hand , our tagset is intrinsically factored as a set of dependent sub-features , which we explicitly represent . The task we address in this paper is morphological disambiguation : given a sentence , obtain the list of all possible analyses for each word from the analyzer , and disambiguate each word in context . On average , each token in the 42 M corpus is given 2.7 possible analyses by the analyzer ( much higher than the average 1.41 POS tag ambiguity reported in English ( Dermatas and Kokkinakis , 1995 ) ) . In previous work , we report disambiguation rates of 89 % for full morphological disambiguation ( using an unsupervised EM-HMM model ) and 92.5 % for part of speech and segmentation ( without assigning all the inflectional features of the words ) . In order to estimate the importance of unknowns in Hebrew , we analyze tokens in several aspects : ( 1 ) the number of unknown tokens , as observed on the corpus of 42 M tokens ; ( 2 ) a manual classification of a sample of 10 K unknown token types out of the 200 K unknown types identified in the corpus ; ( 3 ) the number of unknown analyses , based on an annotated corpus of 200 K tokens , and their classification . About 4.5 % of the 42 M token instances in the Buckwalter 's Arabic analyzer ( 2004 ) , which looks for any legal combination of prefix-stem-suffix , but does not provide full morphological features such as gender , number , case etc . 2 http://mila.cs.technion.ac.il.html training corpus were unknown tokens ( 45 % of the 450 K token types ) . For less edited text , such as random text sampled from the Web , the percentage is much higher -about 7.5 % . In order to classify these unknown tokens , we sampled 10 K unknown token types and examined them manually . The classification of these tokens with their distribution is shown in Table 1 3 . As can be seen , there are two main classes of unknown token types : Neologisms ( 32 % ) and Proper nouns ( 48 % ) , which cover about 80 % of the unknown token instances . The POS distribution of the unknown tokens of our annotated corpus is shown in Table 2 . As expected , most unknowns are open class words : proper names , nouns or adjectives . Regarding unknown analyses , in our annotated corpus , we found 3 % of the 100 K token instances were missing the correct analysis in the lexicon ( 3.65 % of the token types ) . The POS distribution of the unknown analyses is listed in Table 2 . The high rate of unknown analyses for prepositions at about 3 % is a specific phenomenon in Hebrew , where prepositions are often prefixes agglutinated to the first word of the noun phrase they head . We observe the very low rate of unknown verbs ( 2 % ) -which are well marked morphologically in Hebrew , and where the rate of neologism introduction seems quite low . This evidence illustrates the need for resolution of unknowns : The naive policy of selecting ' proper name ' for all unknowns will cover only half of the errors caused by unknown tokens , i.e. , 30 % of the whole unknown tokens and analyses . The other 70 % of the unknowns ( 5.3 % of the words in the text in our experiments ) will be assigned a wrong tag . As a result of this observation , our strategy is to focus on full morphological analysis for unknown tokens and apply a proper name classifier for unknown analyses and unknown tokens . In this paper , we investigate various methods for achieving full morphological analysis distribution for unknown tokens . The methods are not based on an annotated corpus , nor on hand-crafted rules , but instead exploit the distribution of words in an available lexicon and the letter similarity of the unknown words with known words . We have addressed the task of computing the distribution p(t|w ) for unknown words for full morphological disambiguation in Hebrew . The algorithm we have proposed is language independent : it exploits a maximum entropy letters model trained over the known words observed in the corpus and the distribution of the unknown words in known tag contexts , through iterative approximation . The algorithm achieves 30 % error reduction on disambiguation of unknown words over a competitive baseline ( to a level of 70 % accurate full disambiguation of unknown words ) . We have also verified that taking advantage of a strong language-specific model of morphological patterns provides the same level of disambiguation . The algorithm we have developed exploits distributional information latent in a wide-coverage lexicon and large quantities of unlabeled data . We observe that the task of analyzing unknown tokens for POS in Hebrew remains challenging when compared with English ( 78 % vs. 85 % ) . We hypothesize this is due to the highly ambiguous pattern of prefixation that occurs widely in Hebrew and are currently investigating syntagmatic models that exploit the specific nature of agglutinated prefixes in Hebrew .

Challenge: In a large corpus, 5 to 10% of tokens are unknown to analyzers used for morphological analysis and cannot be resolved in a given lexicon.
Approach: They propose a language independen algorithm which exploits a maximum entropy letters model trained over known words and gives distributions of unknowns by iterative approximation.
Outcome: By evaluations in Hebrew, the proposed model achieves a 30% error reduction on unknown tokens over a baseline and is comparable to a language-specific model.