A Novel Challenge Set for H ebrew Morphological Disambiguation and Diacritics Restoration (2020)

Abstract: One of the primary tasks of morphological parsers is the disambiguation of homographs. Particularly difficult are cases of unbalanced ambiguity, where one of the possible analyses is far more frequent than the others. In such cases, there may not exist sufficient examples of the minority analyses in order to properly evaluate performance, nor to train effective classifiers. In this paper we address the issue of unbalanced morphological ambiguities in Hebrew. We offer a challenge set for Hebrew homographs — the first of its kind — containing substantial attestation of each analysis of 21 Hebrew homographs. We show that the current SOTA of Hebrew disambiguation performs poorly on cases of unbalanced ambiguity. Leveraging our new dataset, we achieve a new state-of-the-art for all 21 words, improving the overall average F1 score from 0.67 to 0.95. Our resulting annotated datasets are made publicly available for further research.

## introduction it is a known phenomenon that the distribution of linguistic units, or words, in a language follows a zipf law distribution @xcite , wherein a relatively small number of words appear frequently, and a much larger number of items appear in a long tail of words, as rare events @xcite . significantly, this also applies to the distribution of analyses of a given homograph. take for instance the simple pos-tag ambiguity in english between noun and verb @xcite . the word "fair" can be used as an adjective ("a fair price") or as a noun ("she went to the fair"). yet, the distribution of these two analyses is certainly not fair; the adjectival usage is far more frequent than the nominal usage (e.g., in @xcite the latter is six times more frequent than the former). we will call such cases "unbalanced homographs". cases of unbalanced homographs pose a formidable challenge for automated morphological parsers and segmenters. in tagged training corpora, the frequent option will naturally dominate the overwhelming majority of the occurrences. if the training corpus is not sufficiently large, then the sparsity of the minority analysis will prevent generalization by machine-learning models. by the same token, it can be difficult to evaluate the performance of tagging systems regarding unbalanced homographs, because the sparsity of the minority analysis prevents computation of adequate scoring. the empirical consequences of unbalanced homographs are magnified in morphologically rich languages (mrls), including many semitic languages, where distinct morphemes are often affixed to the word itself, resulting in additional ambiguity @xcite . furthermore, in many semitic mrls, the letters are almost entirely consonantal, omitting vowels. this results in a particularly high number of homographs, each with a different pronunciation and meaning. in this paper, we focus upon unbalanced homographs in hebrew, a highly ambiguous mrl in which vowels are generally omitted @xcite . take for example the hebrew word ‫.מדינה‬ this frequent word is generally read as a single nominal morpheme, ‫ה‬ ‫י‬ ‫ְד‬ ‫,מ‬ meaning "country". however, it can also be read as ‫הּ‬ ‫י‬ ‫ִדּ‬ ‫,מ‬ "from the law/judgment of her", wherein the initial and final letters both serve as distinct morphemes. this last usage is far less common, and, in an overall distribution, it would be relegated to the long tail, with very few attestations in any given corpus. hebrew is a low-resource language, and as such, the problem of unbalanced homographs is particularly acute. existing tagged corpora of hebrew are of limited size, and in most cases of unbalanced homographs, the corpora do not provide sufficient examples to evaluate performance regarding minority analyses, nor to train an effective classifier. here, we propose to overcome this difficulty by means of a challenge set: a group of specialized training sets which each focus upon one particular homograph, offering substantial attestations of the competing analysis. designing such contrast sets that expose particularly hard unbalanced cases was recently proposed as a complementary evaluation effort for a range of nlp tasks by @xcite . notably, all tasks therein focus exclusively on english, and do not make any reference to morphology. another, particularly successful, instance of this approach is the noun/verb challenge set for english built by @xcite . yet, heretofore, no challenge sets have been built to address cases of unbalanced homographs in hebrew. in order to fill this lacuna, we built a challenge set for 12 frequent cases of unbalanced hebrew homographs. each of these words admits of two possible analyses, each with its own diacritization and interpretation. 1 for each of the possible analyses, we gather 400-2,500 sentences exemplifying such usage, from a varied corpus consisting of news, books, and wikipedia. furthermore, in order to highlight the particular problem regarding unbalanced homographs, we add an additional 9 cases of balanced homographs, for contrast and comparison. all in all, the corpus contains over 56k sentences. 2 ## description of the corpus in in choosing our 21 homographs, we first assembled a list of the most frequent homographs in the hebrew language. for the simplicity of this initial proof of concept, we constrained our list to homographs with only two primary analyses. we also constrained our list to cases where the two analyses represent different lexemes, skipping over cases in which the difference is only one of inflection. further, some cases were filtered out due to data sparsity. finally, we also included a number of less frequent homographs, to allow for a comparison between frequent and infrequent homographs. in order to gather sentences for the contrast sets, we first sampled 5000 sentences for each target word, and sent them to student taggers. for balanced homographs, with ratios of 1:3 or less, this process handily provided a sufficiently large number of sentences for each of the two analyses. however, regarding cases of unbalanced homographs, wherein the naturally occurring ratio of the minority analysis can be 30:1 or even 129:1, this initial corpus was far from adequate. we used two methods to identify additional candidate sentences: (1) we ran texts through an automated hebrew diacritizer @xcite and took the cases where the word was diacritized as the minority analysis. (2) where relevant, we leveraged optional hebrew orthographic variations which indicate that a given word is intended in one specific way. these candidate sentences were then sent to student taggers to confirm that the minority analysis was in fact intended. our student taggers tagged approximately 300 sentences per hour. evaluation of their work revealed that they averaged an accuracy of 98 percent. in order to overcome this margin of error, we employed a hebrew-language expert who proofread the resulting contrast sets. in our final corpus, each analysis of each homograph is attested in at least 400 sentences, and usually in 800-2.5k sentences (full details in appendix one issue we encountered when collecting naturally-occurring hebrew sentences is that a small number of specific word-neighbors and collocations tend to dominate the examples. as an example: the word ‫אפשר‬ can be vocalized as ‫ר‬ ‫ְשׁ‬ ‫ֶפ‬ ‫א‬ ("possible", the majority case), or ‫ר‬ ‫ְשׁ‬ ‫ִפ‬ ‫א‬ ("he allowed"). however, over one third of the naturally occurring cases of the majority case boil down to some 90 frequently-occurring collocations, such as ‫אפשר‬ ‫אי‬ ("impossible") or ‫אפשר‬ ‫הא‬ ("is it possible?"). as such, a machine-learning model would overfit to those specific collocations, rather than learning more generic overarching patterns of option 1 option 2 # form word (translation) morphology count / 1m word (translation) morphology count / 1m ratio 1 ‫את‬ ([accusative]) ‫ֶת‬ ‫א‬ acc 18164 (you) ְ ‫ַתּ‬ ‫א‬ pronoun [f,s,2] 275 66:1 2 ‫אתה‬ (you) ‫ָה‬ ‫ַתּ‬ ‫א‬ pronoun [m,s,2] 1430 (with her) ‫ָהּ‬ ‫ִתּ‬ ‫א‬ prep+suf pron [f,s,3] 26 55:1 3 ‫אתכ‬ (you) ֶ ‫ְכ‬ ‫ֶת‬ ‫א‬ acc+suf pron [m,p,2] 70 (with you) ֶ ‫ְכ‬ ‫ִתּ‬ ‫א‬ prep+suf pron [m,p,2] 7 10:1 4 ‫את‬ (you) ֶ ‫ַתּ‬ ‫א‬ pronoun [m,p,2] 324 (with them) ָ ‫ִתּ‬ ‫א‬ prep+suf pron [m,p,3] 34 10:1 5 ‫ברכת‬ (blessing) ‫ַת‬ ‫כּ‬ ‫ִר‬ ‫בּ‬ noun [cons,f,s] 25 (pool) ‫ַת‬ ‫כ‬ ‫ְר‬ ‫בּ‬ noun [cons,f,s] 0.8 30:1 6 ‫הרי‬ (indeed) ‫י‬ ‫ֲר‬ ‫ה‬ conj / intj 418 (mountains) ‫י‬ ‫ָר‬ ‫ה‬ noun [cons,m,p] 12 33:1 7 ‫יאמר‬ (he will say) ‫ַר‬ ‫אמ‬ ‫י‬ verb [m,s,3,future] 18 (will be said) ‫ֵר‬ ‫ָמ‬ ‫א‬ ‫י‬ verb [m,s,3,future] 0.4 43:1 8 ‫מסכת‬ (tractate) ‫ֶת‬ ‫ֶכ‬ ‫ַסּ‬ ‫מ‬ noun [abs/cons,f,s] 54 (mask) ‫ַת‬ ‫ֵכ‬ ‫ַסּ‬ ‫מ‬ noun [cons,f,s] 1 43:1 9 ‫ע‬ (with) ִ ‫ע‬ preposition 4240 (nation) ַ ‫ע‬ noun [abs/cons,m,s] 286 14:1 10 ‫פניה‬ (her face) ָ ‫יה‬ ‫ָנ‬ ‫פּ‬ noun [f,m,p,@xmath0,s,3] 55 (application) ‫ה‬ ‫יּ‬ ‫ְנ‬ ‫פּ‬ noun [f,s] 2 33:1 11 ‫פרשו‬ (they left) ‫שׁוּ‬ ‫ָר‬ ‫פּ‬ verb [mf,p,3,past] 6 (they interpreted) ‫שׁוּ‬ ‫ֵר‬ ‫פּ‬ verb [mf,p,3,past] 0.4 15:1 12 ‫שלישית‬ (third) ‫ית‬ ‫ִישׁ‬ ‫ל‬ ‫שׁ‬ cardinal [f,s] 107 (trio) ‫ת‬ ‫יּ‬ ‫ִישׁ‬ ‫ל‬ ‫שׁ‬ noun [cons,f,s] 0.8 129:1 13 ‫אחר‬ (different) ‫ֵר‬ ‫ַח‬ ‫א‬ adj [m,s] 474 (after) ‫ַר‬ ‫ַח‬ ‫א‬ preposition 387 1:1 14 ‫בניה‬ (her sons) ָ ‫יה‬ ‫ָנ‬ ‫בּ‬ noun [m,p,@xmath1,s,3] 8 (building) ‫ה‬ ‫יּ‬ ‫ְנ‬ ‫בּ‬ noun [f,s] 5 1.5:1 15 ‫חזרה‬ (returning) ‫ה‬ ‫ר‬ ‫ֲז‬ ‫ח‬ noun [f,s] 62 (she returned) ‫ה‬ ‫ר‬ ‫ָז‬ ‫ח‬ verb [f,s,3,past] 55 1:1 16 ‫ידע‬ (he knew) ‫ע‬ ‫ד‬ ‫י‬ verb [m,s,3,past] 88 (knowledge) ‫ע‬ ‫ד‬ ‫י‬ noun [abs/cons,m,s] 55 1.5:1 17 ‫כשר‬ (as minister) ‫ר‬ ‫ְשׂ‬ ‫כּ‬ prep+noun [abs/cons,m,s] 35 (kosher) ‫ר‬ ‫ָשׁ‬ ‫כּ‬ adj [m,s] / propn [mf,s] 14 2.5:1 18 ‫כתב‬ (he wrote) ‫ַב‬ ‫ָת‬ ‫כּ‬ verb [m,s,3,past] 252 (writing) ‫ַב‬ ‫ְת‬ ‫כּ‬ noun [cons,m,s] 103 2.5:1 19 ‫מבי‬ (understands) ‫ִי‬ ‫ֵב‬ ‫מ‬ participle [m,s] 174 (from amongst) ‫ֵי‬ ‫ִבּ‬ ‫מ‬ preposition 98 2:1 20 ‫ספריה‬ (her books) ָ ‫יה‬ ‫ָר‬ ‫ְפ‬ ‫ס‬ noun [m,p,@xmath2,s,3] 13 (library) ‫ה‬ ‫יּ‬ ‫ְר‬ ‫ִפ‬ ‫ס‬ noun [f,s] 4 2.5:1 21 ‫עמנו‬ (our nation) ‫ֵנוּ‬ ‫ַמּ‬ ‫ע‬ noun [m,s,@xmath3,p,1] 23 (with us) ‫ָנוּ‬ ‫ִמּ‬ ‫ע‬ prep+suf pron [mf,p,1] 12 2:1 the word usage. therefore, we constrained our data collection such that there may be no more than 20 cases of any given word-neighbor combination. 4 3 experiments we first use our challenge set to evaluate current state-of-the-art performance on the morphological disambiguation of hebrew homographs. the best existing tool for hebrew morphological disambiguation is yap: yet another parser @xcite . we run all 56,000+ sentences from our challenge set through yap. due to the unbalanced natural distribution of the possible analyses in many of the cases, we compute recall and precision results separately for each analysis, and we then compute a macro-averaged f1 score. next, we use our challenge set to train classifiers for each of the homographs in our corpus. we implement 2-layer mlps using the dynet framework @xcite . as input, we feed the mlp an encoding h(w i ), a representation of the context of the target word within the sentence. the target word itself is masked and not included in the input. the output of the mlp is a probabilistic choice of either class 1 or class 2, where each class represents one of the two possible diacritization options. we applied two methods to represent the surrounding context in the mlp input. the first is encoding the three neighboring words on both sides 4 our challenge set is available for use in future research. we run each contrast set using each of our three methods for encoding the neighboring words. we evaluate the results using 10-fold cross validation. ## results and analysis in yap does considerably better here: all f1 scores are above .5, and four of the cases are above .8. the weakest cases are those in which yap has to differentiate between an unsegmented noun and a case of a noun plus possessive suffix (cases 14,20). in both of these cases, yap scores an f1 of approximately .56 (which, interestingly, is precisely on par with the analogous unbalanced case [10]). in in ## related work many recent papers have proposed global or unsupervised methods for homograph disambiguation in english (e.g. @xcite ; @xcite ; @xcite ). while such methods have obvious advantages, they have limited applicability to hebrew. as noted, in hebrew the majority of the words are ambiguous, including the core building blocks of the language; without these anchors, global approaches tend to result in poor performance regarding unbalanced homographs. the problem of hebrew diacritization is analogous to that of arabic diacritization; arabic, like hebrew, is a morphologically-rich language written without diacritics, resulting in high ambiguity. many recent studies have proposed machinelearning approaches for the prediction of arabic diacritics across a given text (e.g. @xcite ; @xcite ; neme and paumier (2019); @xcite ; @xcite . however, these studies all perform evaluations on standard arabic textual datasets, and do not evaluate accuracy regarding minority options of unbalanced homographs. we believe that these models would likely benefit from specialized challenge sets of the sort presented here to overcome the specific hurdle of unbalanced homographs. ## conclusion due to high morphological ambiguity, as well as the lack of diacritics, semitic languages pose a particularly difficult disambiguation task, especially when it comes to unbalanced homographs. for such cases, specialized contrast sets are needed, both in order to evaluate performance of existing tools, as well as in order to train effective classifiers. in this paper, we construct a new challenge set for hebrew disambiguation, offering comprehensive contrast sets for 21 frequent hebrew homographs. these contrast sets empirically demonstrate the limitations of reported sota results when it comes to unbalanced homographs; a model may report a sota for a benchmark, yet fail miserably on real world rare-but-important cases. our new corpus will allow hebrew nlp researchers to test their models in an entirely new fashion, evaluating the ability of the models to predict minorityhomograph analyses, as opposed to existing hebrew benchmarks which tend to represent the language in terms of its majority usage. furthermore, our corpus will allow researchers to train their own classifiers and leverage them within a pipeline architecture. we envision the classifiers positioned at the beginning of the pipeline, disambiguating frequent forms from the get-go, and yielding improvement down the line, ultimately improving results for downstream tasks (e.g. nmt). indeed, as we have demonstrated, neural classifiers trained on our contrast sets handily achieve a new sota for all of the homographs in the corpus.