N akdan: Professional H ebrew Diacritizer (2020)

Abstract: We present a system for automatic diacritization of Hebrew Text. The system combines modern neural models with carefully curated declarative linguistic knowledge and comprehensive manually constructed tables and dictionaries. Besides providing state of the art diacritization accuracy, the system also supports an interface for manual editing and correction of the automatic output, and has several features which make it particularly useful for preparation of scientific editions of historical Hebrew texts. The system supports Modern Hebrew, Rabbinic Hebrew and Poetic Hebrew. The system is freely accessible for all use at http://nakdanpro.dicta.org.il

## introduction we present a web-based system for diacritization of hebrew text, which caters to both casual and expert users. the diacritization engine driving the system combines manually curated linguistic resources with modern machine learning models. diacritization in hebrew writing, the letters are almost entirely consonantal; the vowels are indicated by diacritic marks, generally positioned underneath the letters. however, in most cases, printed hebrew omits the diacritic marks and includes only the letters, resulting in a highly ambiguous text, in which any given non-diacritized word can represent a host of different hebrew words, each with a different meaning and pronunciation. for example, the form ‫בצל‬ can be diacritized as ‫ָל‬ ‫ָצ‬ ‫בּ‬ (noun, "onion"), ‫ֵל‬ ‫ְצ‬ ‫בּ‬ (prefix+noun, "in a shadow"), ‫ֵל‬ ‫ַצּ‬ ‫בּ‬ (prefix+definitive+noun, "in the shadow") and others. the task of diacritization is thus a task of disambiguation: choosing from among the valid word possibilities for each nondiacritized word, and then adding in the diacritic marks accordingly. the multiple possibilities for diacritizing any given word often represent different morphological possibilities. thus, to an extent, choosing the correct diacritization entails morphological disambiguation; conversely, prior morphological disambiguation greatly reduces the total possible diacritization possibilities.we provide further details in §2. hybrid neural and rule-based approach our approach, described in §3, uses several bi-lstmbased deep-learning modules for disambiguating the correct diacritization in context. however, it is also supplemented by comprehensive inflection tables and lexicons, when appropriate. web interface we provide a web interface for the user to input a text for diacritization and refine the resulting diacritized text (figure 1 ). our system parses the text and automatically adds diacritics throughout. afterward, the user can proofread the text in the interface. for each word, all alternate diacritization possibilities are provided for immediate selection, ordered according to their predicted probability. keyboard shortcuts allow efficient navigation of the text and fast selection of alternate options. users can choose to see morphological analyses for each of the diacritization options, to assist in distinguishing between options. ## the hebrew diacritics system the diacritics system of modern hebrew marks vowels and gemination, and includes 12 primary diacritic symbols: additionally, a dot in the middle of a letter indicates gemination. for the case of the 'shin' letter, an upper dot distinguishes between pronunciation as 's' or as 'sh'. diacritized hebrew aims to position a diacritic on every single letter of the word, with the exception of final letters and matres lectionis. ambiguity in our tests, knowing the correct diacritics reduces the full-morphological-analysis ambiguity from 9.1 to 2.4 average analyses per word form, while knowing the full-morphologicalanalysis reduces the diacritization ambiguity from 6.2 to 1.4 average options per word form. note that these numbers reflect fine-grained morphological tagging. if we utilize coarse-grained tagging, sufficing with the part of speech for each word, then knowing the correct diacritization reduces the average morphological ambiguity from 3.2 options to 1.97, while knowing the correct pos tag reduces the average diacritization ambiguity from 6.2 options to 2.75. thus, the need for an automated diacritization utility is particularly crucial in order to properly disambiguate a hebrew text. ## approach recent trends in nlp suggest moving towards machine-learned models that automatically learn to extract the regularities in the data. such approaches have also been applied to diacritization of arabic (belinkov and glass, 2015; rashwan et al., 2015; abandah et al., 2015; mubarak et al., 2019). however, while these generally provide very strong results, they also often make mistakes that contradict our prior knowledge of the linguistic system. while the machine-learned models generalize very well and can learn to perform tasks in which humans cannot articulate the underlying regularities, there are also many cases that language-experts can articulate precisely, and these tend to correlate with the cases that the learned models fail on. we therefore take a hybrid approach. similar to traditional diacritization systems (choueka and neeman, 1995), we use our explicit knowledge about the language and the diacritization system whenever we can. however, we also supplement our knowledge with learned model predictions for the challenging cases for which we cannot articulate the rules and regularities: selecting the appropriate diacritization in context, and providing diacritization for out-of-vocabulary words. this methodology departs from recent diacritization works that rely on hmm and neural-network methods (gal, 2002; belinkov and glass, 2015), while ignoring forms of explicit linguistic knowledge. we use such a combination of machine-learned and human-specified knowledge in all the components of the system, either by supplementing the predictor with manually constructed options, or by filtering its output space. of course, a prerequisite for an effective machine-learned system is high-quality training data. our system is trained on a collection of 1,5m diacritized tokens which we annotated in-house. ## high-quality data sources we make use of the following language resources and corpora, which we collected. language resources our main resource is a high-coverage and accurate lexicon of hebrew word forms, their diacritization and their corresponding morphological analyses. employing a staff of language experts, we began by assembling a list of all nouns, adjectives and verbal roots in the hebrew language. this list includes 50k lexemes altogether (10k roots, 30,5k nouns, and 9,5k adjectives). we then built comprehensive inflection tables to generate all possible inflected forms from each of these lexemes, including all valid combinations of possessive and accusative suffixes, with full diacritization. altogether, this process generated some 5,5 million inflected forms (3,8m verbal forms; 1,3m nominal forms; and 460k adjectival forms). we also added 1,7k adverbs, and another 4,5k function words (conjunctions, prepositions, existentials, quantifiers, etc., including all possible suffix combinations). finally, we collected a set of 17,5k frequent proper nouns (countries and major cities; heads of state and other notable people; and frequently-mentioned companies and organizations), and our language experts diacritized these as well. these tables suffice for modern hebrew; however, in historical hebrew texts, we often find aramaic terms interspersed within the hebrew. therefore, we also built a similarly comprehensive and diacritized wordlist for babylonian aramaic. our aramaic wordlist contains 750k verbal inflected forms; 200k nominals; 1,5k adjectives; and another 2k adverbs and function words. we additionally assembled an exhaustive list of nondiacritized hebrew names of persons and locations (including collections of both street names and city names). annotated corpora for morphological tagging, we make use of a corpus of 200k tokens of modern hebrew, composed of hebrew fiction, news, wikipedia, and blogs. these tokens were manually annotated with fine-grained morphological information according to the scheme of (elhadad et al., 2005). additionally, as noted, we anno-tated a 1,5m word diacritized modern hebrew corpus, consisting of hebrew prose (both fiction and non-fiction), newspapers (both news and op-ed), wikipedia, blogs (including many female-dominant blogs, to ensure coverage of feminine word forms), law protocols, parliament proceedings, tv transcripts, academic texts, and biographical sketches. we have similarly collected and annotated corpora of historical hebrew, consisting of jewish legal writings and commentaries from the 3rd-12th centuries: 110k words with fine-grained morphological tagging, and 2m words with diacritization. finally, regarding poetic hebrew, we collected and annotated a corpus of 1,3m words, containing hebrew poetry from both medieval and modern periods. the undiacritized base texts were collected largely through partnerships with cooperating organizations in israel; the morphological tagging and diacritization was done primarily in-house by our hebrew language experts. ## system architecture on a high level, our system works in the following stages, which we will elaborate on below. each stage combines engineered linguistic information and a trained neural model. 1. pos-tagging and morphological disambiguation. part-of-speech tagging and morphological disambiguation as diacritic marks closely interact with the morphological analysis and part-of-speech (pos) of the token, we first perform pos-tagging and morphological disambiguation, using a twostage process. in the first stage, each word is assigned its core part-of-speech, and in the second stage it is enriched with additional morphological properties, where the set of considered morphological properties is determined based on the coarsegrained pos (e.g., nouns take gender, number and definiteness, while verbs do not take definiteness but do take tense and person). @xcite training is performed on our annotated corpus of 200k tokens. the resulting tagger has an accuracy of 92% for the coarse-grained part-of-speech, and 79% for full morphological disambiguation. @xcite both taggers are 2-layer bi-lstm transducers (goldberg, 2017), where the first stage coarsegrained tagger maps each token w i to a coarse pos-tag t i , while the second stage morphological tagger adds additional morphological properties m 1 i , ..., m k i . each bi-lstm takes as inputs vectors x 1 , ..., x n corresponding to tokens w 1 , ..., w n and produces vectors h(x 1 ), ..., h(x n ). these vectors are then fed into multi-layer perceptrons (mlp) for predicting the pos-tags and morphological properties, where each property is predicted by a different mlp: the set of mlps m k i for a word is determined based on its predicted coarse-grained pos-tag. in the coarse-grained tagger, each token w i is mapped to an input vector x i which encodes character level information, distributional word-level information, possible morphological analyses of w i , 3 and lexicon-based features of w i . specifically, x i is a concatenation of: (a) for a word w i made of characters c w i 1 , ..., c w i m the sum of bi-lstm states j h(c w i j ) from a char-level bi-lstm that runs over the entire sentence; (b) bi-lstm state at w i , for a word-level bi-lstm that runs on pretrained word2vec vectors for all of the words in the sentence; (c) a vector representing the possible fine-grained morphological analyses for the word, 4 according to our wide-coverage lexicon; (d) bits interrogative, interj, quantifier, existential, modal, prefix, participle, copula, titular, shel prep, and the following morphological properties: gender, number, person, construct/absolute, suffix (possessive / accusative / pronominal). ## conclusion we are pleased to release our hebrew diacritization system for free unrestricted use. it is powered by a combination of advanced machine learning and manually curated linguistic resources, and thus succeeds in setting a new state of the art for hebrew diacritization. we have released also our diacritized test corpora for benchmarking.