NLP in the DH pipeline: Transfer-learning to a Chronolect (2021)

Abstract: A big unknown in Digital Humanities (DH) projects that seek to analyze previously untouched corpora is the question of how to adapt existing Natural Language Processing (NLP) resources to the specific nature of the target corpus. In this paper, we study the case of Emergent Modern Hebrew (EMH), an under-resourced chronolect of the Hebrew language. The resource we seek to adapt, a diacritizer, exists for both earlier and later chronolects of the language. Given a small annotated corpus of our target chronolect, we demonstrate that applying transfer-learning from either of the chronolects is preferable to training a new model from scratch. Furthermore, we consider just how much annotated data is necessary. For our task, we find that even a minimal corpus of 50K tokens provides a noticeable gain in accuracy. At the same time, we also evaluate accuracy at three additional increments, in order to quantify the gains that can be expected by investing in a larger annotated corpus.

## introduction digital humanities (dh) projects that deal with understudied languages, including many historical languages, often face a "throughput bottleneck" due to the lack of natural language processing (nlp) resources trained for the specific language variety they document. in many cases, there is a closely related language, typically a modern standard variety, for which there exist large digital corpora, goldsets manually-annotated for various features, and nlp tools (e.g., morphological analyzers, syntactic parsers). it is an open question to what extent it is useful to adapt tools tailored for one chronolect (a language used in a particular point in time), in order to build resources for a closely related yet different chronolect @xcite on spelling normalization for english chronolects; @xcite for early modern swedish spelling, tagging, and parsing; pettersson and nivre 2015 for verb phrase extraction in early modern swedish; @xcite for morphological tagging of old czech). the researcher is faced in such a case with questions such as: how much data is needed for training? is it better to fine-tune existing nlp models, which were trained to fit a different chronolect, or instead to train new models only on the chronolect of interest? these are the questions that arose in the framework of a dh project "the jerusalem corpus of emergent modern hebrew" (jemh; rubinstein 2019), which aims to digitize and enable linguistic analysis of multi-genre archival material in emergent modern hebrew (emh; hebrew of the early 20th century). diacritization of this corpus is a critical step both in order to increase its accessibility (in particular because the diacritics are a necessary prerequisite for the application of text-to-speech algorithms), and also for downstream nlp tasks such as intelligent search, relation extraction, and linguistic analysis of the historical materials. although emh is a severely undersourced hebrew chronolect, diacritization models do exist for other hebrew chronolects. we thus aim to answer the following questions: how much diacritized emh data must be assembled, and how can we best leverage existing models from other hebrew chronolects? we manually add annotations of diacritics to a representative sample of the jemh corpus and use this new goldset to train a neuralnet diacritizer for emh. the resulting diacritizer is the first nlp tool developed for this historical langauge variety. it can be incorporated into the pipeline of any dh project dealing with emh, providing valuable disambiguations of the text. we compare two approaches to the creation of the diacritizer for emh. the indirect approach uses the emh goldset to fine-tune existing dia-critizers on top of two well-resourced chronolects of hebrew which are closely-related to our target chronolect (one variety predates emh and one is the present-day standard variety). in the direct route, we train a new model for emh using just our (small) annotated goldset. we provide, for each approach, an estimate of the lower bound on the amount of training data needed to reach acceptable performance, discussing the implications of our findings for planning dh projects. ## linguistic challenges the use of hebrew as a spoken language only began around the turn of the 20th century, and the norms of modern hebrew -lexical, morphological, syntactic, and orthographical -only crystallized during the decades afterward @xcite @xcite @xcite @xcite . however, knowledge about previous stages of the language was accessible to learners through canonized texts which formed the basis of jewish education throughout history @xcite . therefore, emh constitutes a chronolect that is considerably distant from modern hebrew. first of all, on the lexical plane, neologisms for many everyday objects such as "kitchen" or "newspapers" were only just being invented for the first time, and were often referred to by multiword circumlocutions instead (e.g., for kitchen: ‫וּל‬ ִ ‫ַבּ‬ ‫ה‬ ‫ֵית‬ ‫בּ‬ bet habišul, lit. 'house of cooking', rather than the modern word ‫ָח‬ ‫ְבּ‬ ‫ִט‬ ‫מ‬ mitbaè, or, for newspaper, ‫ִי‬ ‫ִתּ‬ ‫ע‬ ‫ָב‬ ‫תּ‬ ִ ‫מ‬ mixtav qiti, lit. 'periodical letter', rather than ‫ִתּוֹ‬ ‫ע‬ qiton). on the morphological plane, we find much use of nominal patterns now completely obsolete (e.g., ‫ה‬ ‫ָר‬ ‫א‬ ‫ְפ‬ ‫מ‬ mefoparah 'magificent', rather than ‫ת‬ ‫ֶר‬ ‫א‬ ‫ְפ‬ ‫מ‬ mefoperet, or ‫ֶת‬ ‫ֶל‬ ‫ְח‬ ‫ַת‬ ‫מ‬ matèelet 'start (participle, fem.)' rather than ‫ָה‬ ‫ִיל‬ ‫ְח‬ ‫ַת‬ ‫מ‬ matèilah). finally, and most significantly, plene orthographynow normative in modern hebrew -had not yet been embraced, causing ambiguities to abound. as has been noted, modern hebrew by itself is already highly ambiguous morphologically @xcite ; however, without the norms of plene spelling, the ambiguity is amplified considerably. for example, in emh ‫חדשׁי‬ èdšim may be analyzed as one of three words: ‫י‬ ‫שׁ‬ ‫ֲד‬ ‫ח‬ èadašim 'new (pl.)', ‫י‬ ‫שׁ‬ ‫ֳד‬ ‫ח‬ èodašim 'months', or ‫י‬ ‫שׁ‬ ‫ָד‬ ‫ח‬ èodšayim 'two months', whereas in modern hebrew, each is represented by a distinct unambiguous spelling, via the addition of matres lectionis: resolving these challenges is crucial for natural lanuage processing of the formative and highly influential emh corpus of hebrew. adding diacritics can dramatically reduce ambiguity and is the tool we sought to develop. ## diacritizaton modern hebrew (similar to other semitic languages such as arabic and syriac) is generally written and published with many of the vowels omitted. in emh, even fewer vocalizations are marked. diacritization refers to the specification of all vowels as part of the written word. the set of diacritics generally used in hebrew is termed "tiberian diacritization", and consists of a set of a dozen essential marks placed below, within, or above the characters @xcite . letters can be optionally geminated (marked with a dot in the center of the letter), leading to a total of 24 possible diacritic permutations for each letter. a given non-diacritized hebrew word generally admits to multiple possible diacritizations, each representing a different semantic and morphological analysis of the word. thus, diacritization cannot be automated via a simple lookup table; rather, it is necessary to use contextual information to choose from among the multiple analyses. several machine-learning systems have been developed to perform this task @xcite @xcite . the current state-of-the-art for modern hebrew is the lstmbased diacritizer developed by dicta @xcite , as per external evaluations performed by @xcite . however, as noted, the emh chronolect presents a new set of challenges, and, indeed, automated diacritization systems for modern hebrew falter on emh. we therefore set out to determine how large a corpus would be necessary to train the same kind of lstm specifically for emh, and to determine whether it would be beneficial, alternatively, to attempt a transfer-learning architecture based upon the pre-trained modern hebrew lstm model. in evaluating the accuracy of the resulting model, we separately consider two approaches: (1) we test the ability of the lstm model to predict the correct vocalization without any constraints. for each word, we run a beam search across the top eight vocalization predictions for each letter, and we take the top scoring beam. (2) we test the ability of the lstm model to choose the correct vocalization option from a set of options provided by a wordlist. 4 thus, as opposed to the previous approach, here we constrain the lstm to known valid vocalization options. for each word, we calculate the lstm finally, we examine the effects of doing the transfer-learning from an earlier chronolect, rather than from a later chronolect. in the previous experiments, we took a pre-trained model from modern hebrew, and we fine-tuning it for the emh ## discussion the results of these experiments lead us to the following observations: first of all, in all cases, transfer-learning from a different hebrew chronolect is better than training from scratch. this is true whether the source chronolect is later or earlier. at the same time, finetuning on top of a later chronolect was clearly superior to fine-tuning on top of an earlier chronolect, likely because a later chronolect will still have remnants of the earlier chronolect, while the same cannot be said of an earlier choronolect. additionally, in all cases, the wordlist filter improved scores considerably. without the wordlist filter, the lstm is free to choose any diacritization sequence at all, and in many cases ends up predicting a sequence that is not a valid word. although many diacritization patterns are limited to certain letter configurations, and we expect the lstm to learn these patterns, in other cases the choice of one pattern or another is fairly arbitrary, and just indicates natural development stages of the language. thus, the wordlist provides the necessary knowledge to ensure that the predicted diacritics are indeed a pattern known to apply to the given word. regarding the question of how much training data is necessary: of course, as expected, the larger the training corpus, the more accurate the result. however, the tables indicate just how much of an improvement we can expect with every additional 20,000-25,000 tokens. as we see, there is generally an increase of somewhere between half a percent and one and a half percent from stage to stage. presumably, if we were to continue to enlarge our training corpus, the accuracy would continue to rise, until it reached a point of diminishing returns. importantly, although the full 120k training corpus yields best results, even a 50k corpus succeeds in providing a palpable improvement when finetuning on top of other chronolects. this suggests that investing in a small annotated corpus is a viable route for dh projects that seek to adapt nlp tools from neighboring chronolects. finally, the successful effects of the transferlearning are limited to texts whose genre is represented within the training corpus. in contrast, when we tested our fine-tuned models on the outof-domain ephemera corpus, we found that the transfer-learning actually lowered the accuracy of the model. this result may be related to the particular languages we tested, however, showing that the language of the emh ephemera is closer to the present-day language than other emh materials. ## conclusion this paper examines a central concern in dh: the question of how much data is needed to train viable nlp tools for under-resourced chronolects. we addressed these questions for the specific case of a historical dialect of hebrew, showing an advantage for fine-tuning a diacritization model using a small annotated corpus (i.e., transfer-learning) over direct training of a new model for the chronolect of interest. our findings have implications for any project that aims to adapt an nlp algorithm to an alternate chronolect. given access to existing nlp tools, it is likely that producing even a small annotated cor-pus of historical materials from the relevant time period will result in substantial gains.