Restoring H ebrew Diacritics Without a Dictionary (2022)

Abstract: We demonstrate that it is feasible to accurately diacritize Hebrew script without any human-curated resources other than plain diacritized text. We present Nakdimon, a two-layer character-level LSTM, that performs on par with much more complicated curation-dependent systems, across a diverse array of modern Hebrew sources. The model is accompanied by a training set and a test set, collected from diverse sources.

## introduction the vast majority of modern hebrew texts are written in a letter-only version of the hebrew script, one which omits the diacritics present in the full diacritized, or dotted variant. 1 since most vowels are encoded via diacritics, the pronunciation of words in the text is left underspecified, and a considerable mass of tokens becomes ambiguous. this ambiguity forces readers and learners to infer the intended reading using syntactic and semantic context, as well as common sense @xcite . in nlp systems, recovering such signals is difficult, and indeed their performance on hebrew tasks is adversely affected by the presence of undotted text @xcite @xcite . as an example, the sentence in one possible way to overcome this problem is by adding diacritics to undotted text, or dotting, implemented using data-driven algorithms trained on dotted text. obtaining such data is not trivial, even given correct pronunciation: the standard tiberian diacritic system contains several sets of identicallyvocalized forms, so while most hebrew speakers easily read dotted text, they are unable to produce it. moreover, the process of manually adding diacritics in either handwritten script or through digital input devices is mechanically cumbersome. thus, the overwhelming majority of modern hebrew text is undotted, and manually dotting it requires expertise. the resulting scarcity of available dotted text in modern hebrew contrasts with biblical and rabbinical texts which, while dotted, manifest a very different language register. this state of affairs allows individuals and companies to offer dotting as paid services, either by experts or automatically, e.g. the morfix engine by melingo. 2 such usage practices also force a disconnect in the nlp pipeline, requiring an api call into an external service whose parameters cannot be updated. existing computational approaches to dotting are manifested as complex, multi-resourced systems which perform morphological analysis on the undotted text and look undotted words up in handcrafted dictionaries as part of the dotting process. dicta's nakdan @xcite , the current state-of-the-art, applies such methods in addition to applying multiple neural networks over different levels of the text, requiring manual annotation not only for dotting but also for morphology. among the resources it uses are a diacritized corpus of 3m tokens and a pos-tagged corpus of 300k tokens. training the model takes several weeks. 3 in this work, we set out to simplify the dotting task as much as possible to standard modules. we introduce a large corpus of semi-automatically dotted hebrew, collected from various sources, and use it to train an rnn-based model. our system, nakdimon, accepts the undotted character sequence as its input, consults no external resources or lexical components, and produces diacritics for each character, resulting in dotted text whose quality is comparable to that of the commercial morfix, on both character-level and word-level accuracy. our model is easy to integrate within larger systems that perform end-to-end hebrew processing tasks, as opposed to the existing proprietary dotters. to our knowledge, this is the first attempt at a "light" model for hebrew dotting since early hmm-based systems @xcite . we introduce a novel test set for modern hebrew dotting, derived from larger and more diverse sources than existing datasets. in experiments over our dataset, we show that our system is particularly useful in the main use case of modern dotting, which is to convey the desired pronunciation to a reader, and that the errors it makes should be more easily detectable by non-professionals than dicta's. foot_3 2 task and datasets ## experiments we compare the performance of nakdimon on our new test set ( §2.3) against dicta, foot_8 snopi, foot_9 and morfix @xcite . as well as a ma-jority baseline which returns the most common dotting for each word seen in our full training set. metrics we report four metrics: decision accuracy (dec) is computed over the entire set of individual possible decisions: dagesh/mappiq for letters that allow it, sin/shin dot for the letter ‫,ש‬ and all other diacritics for letters that allow them; character accuracy (cha) is the portion of characters in the text that end up in their intended final form (which may combine two or three decisions, e.g. dagesh + vowel); word accuracy (wor) is the portion of words with no mistakes; and vocalization we note the substantial improvement our model achieves on the voc metric compared to the wor metric: 18.43% of word-level errors are attributable to vocalization-agnostic dotting, compared to 13.80% for dicta and 10.41% for snopi (but 20.91% for morfix). considering that the central use case for dotting modern hebrew text is to facilitate pronunciation to learners and for reading, and that undotted homograph ambiguity typically comes with pronunciation differences, we believe this measure to be no less important than wor. results on dicta's test set @xcite ) are presented in appendix c. ## related work existing work on diacritizing hebrew is not common, and all efforts build on word-level features. @xcite trains an hmm on a vocalized and morphologically-tagged portion of the hebrew bible containing 30,743 words, and evaluates the result on a test set containing 2,852 words, achieving 81% wor accuracy. note that biblical hebrew is very different from modern hebrew in both vocabulary, grammatical structure, and diacritization, and also has many words with unique diacritization. in our system, we exclude the bible altogether from the training set, as its inclusion actively hurts performance on the validation set, which consists of modern hebrew. tomer (2012) designs a diacritization system for hebrew verbs consisting of a combination of a verb inflection system, a syllable boundary detector, and an svm model for classifying verb inflection paradigms. the focus on verbs in a type-level setup makes this work incomparable to ours or to others in this survey. in arabic, diacritization serves a comparable purpose to that in hebrew, but not exclusively: most diacritic marks differentiate consonantal phonemes from each other, e.g. /b/ vs. /t/ (which only the sin/shin dot does in hebrew), whereas vocalization marks are in a one-to-one relationship with their phonetic realizations, e.g. only the fatha as in /ba/ encodes the /a/ vowel. dictionary-less arabic diacritization has been attempted using a 3-layer bi-lstm @xcite . @xcite use a bi-lstm where characters are assigned either one or more diacritic symbols. our system differs from theirs by virtue of separating the diacritization categories. @xcite tackled arabic diacritization as a sequence-to-sequence problem, tasking the model with reproducing the characters as well as the marks. @xcite have made the case against rnn-only systems, arguing for the importance of morphological analyzers in arabic nlp systems. we concede that well-curated systems may perform better than uncurated ones, particularly on low-resource languages such as hebrew, but we note that they are difficult to train for individual use cases and are burdensome to incorporate within larger systems. diacritics restoration in latin-based scripts, applicable mostly to european languages, forms a substantially different problem from the one in hebrew given the highly lexicalized nature of diacritic usage in these languages and the very low rate of characters requiring diacritics. the state-of-theart systems in such languages employ transformer models in a sequence-to-sequence setup @xcite , supplanting character-rnn sequence prediction architectures reminiscent of ours @xcite . indeed, the authors of this latter work note the only non-european in their dataset, vietnamese, as a special outlier. ## conclusion learning directly from plain diacritized text can go a long way, even with relatively limited resources. nakdimon demonstrates that a simple architecture for diacritizing hebrew text as a sequence tagging problem can achieve performance on par with much more complex systems. we also introduce and release a corpus of dotted hebrew text, as well as a source-balanced test set. in the future, we wish to evaluate the utility of dotting as a feature for downstream tasks such as question answering, machine translation, and speech generation, taking advantage of the fact that our simplified model can be easily integrated in an end-to-end hebrew processing system.