Semi-supervised Contextual Historical Text Normalization (2020)

Abstract: Historical text normalization, the task of mapping historical word forms to their modern counterparts, has recently attracted a lot of interest (Bollmann, 2019; Tang et al., 2018; Lusetti et al., 2018; Bollmann et al., 2018;Robertson and Goldwater, 2018; Bollmannet al., 2017; Korchagina, 2017). Yet, virtually all approaches suffer from the two limitations: 1) They consider a fully supervised setup, often with impractically large manually normalized datasets; 2) Normalization happens on words in isolation. By utilizing a simple generative normalization model and obtaining powerful contextualization from the target-side language model, we train accurate models with unlabeled historical data. In realistic training scenarios, our approach often leads to reduction in manually normalized data at the same accuracy levels.

## introduction text normalization is the task of mapping texts written in some non-standard variety of language l (a dialect or an earlier diachronic form) to some standardized form, typically the official modern standard variety of l ( text can to a large extent be normalized by replacing non-standard words with their standard counterparts. because of this often-made assumption, this task is also known as "lexical" or "spelling normalization" @xcite . there has been a lot of interest in historical and dialectal text normalization over the past years. earlier works attempt type-level normalization by way of search for standardized words @xcite . more recently, the dominant approach casts the problem as probabilistic type-level character transduction. most commonly, a fully-supervised machine translation system transduces words in isolation @xcite . the use of context is limited to employing a target-side language model for an improved, contextualized decoding @xcite @xcite . in this paper, we develop simple approaches to semi-supervised contextualized text normalization. on the example of historical text normalization, we show that one can reduce the amount of supervision by leveraging unlabeled historical text and utilizing context at training. our methods build on familiar techniques for semisupervised learning such as generative modeling and expectation-maximization and unify previous work (search, noisy channel, contextualized decoding, neural character-level transduction) in a simple setup. we experimentally validate the strength of our models on a suite of historical datasets. in addition to the token-level supervision scenario, we show benefits of a more economic supervision by a word-type normalization dictionary. another line of work follows a principled probabilistic solution: a noisy channel model @xcite , which consists of a channel p(x | y) and a language model p(y) @xcite @xcite @xcite . the channel model operates at the character level and takes the form of either a character alignment model @xcite or a weighted finite-state transducer (wfst, @xcite . channel model parameters are estimated from a manually normalized corpus. the language model is often trained on external targetside data. some works perform normalization of words in context. @xcite and @xcite decode sequences of historical word tokens by combining a character-level channel with a word-level language model p(y 1:m ). @xcite learn a character alignment model directly over untokenized segments of historical texts. numerous neural approaches to text normalization @xcite @xcite @xcite learn a discriminative model p(y | x)parameterized with some generic encoder-decoder neural network-that performs the traditional character-level transduction of isolated words. the models are trained in a supervised fashion on a lot of manually labeled data. for example, @xcite train on tens of thousands of labeled pairs, including for varieties that share more than 75% of their vocabularies. except @xcite , who use a target-side language model to rerank base model hypotheses in context, no other approach in this group uses context in any way. ## the role of context if non-standard language exhibits normalization ambiguity, one would expect contextualization to reduce it. for example, historical german "desz" in the ridges corpus @xcite normalizes to three modern word types: "das", "des" (various forms of the definite article), and "dessen" (relative pronoun whose). knowing the context (e.g. whether the historical word occurs clause-initially or before a neuter noun) would help normalize "desz" correctly. as suggested by @xcite , the accuracy of the oracle that normalizes words in isolation by always selecting their most frequent normalization upperbounds the accuracy of non-contextual systems. many historical normalization corpora do not have high normalization ambiguity ( even if context may sometimes not be necessary for adequately solving the task in a fully supervised manner, we would expect contextualization to lead to more accurate unsupervised and semi-supervised generative models. ## experiments for our experiments, we use eight datasets compiled by various researchers @xcite @xcite from historical corpora (tables 2 and 3 ). 1 seven languages are indo-european: germanic (english, german, icelandic, and swedish), romance (portuguese and spanish), and slavic (slovene). additionally, we experiment with hungarian, a finno-ugric language. from the slovene dataset, we only use the collection of the older and more challenging texts in the bohorič alphabet. the data are of different genres (letters, religious and scientific writings). the earliest texts are in 14th-c. middle english. in many datasets, the proportion of identity normalizations is substantial. the smallest word overlap is in the hungarian data (18%), the largest is in english (75%). all corpora are tokenized and aligned at the segment and token level. for some datasets, either segments do not coincide with grammatical sentences, or the data have no segment boundaries at all (e.g. hungarian or icelandic). in such cases, to make input amenable to training with context, we resort to sentence splitting on punctuation marks. algorithm 2 map decoding or reranking ( §4.6) input: non-standard word sequence x1:m, number of modern candidates c to generate, number of sentence hypotheses k to generate. we also split very long segments to ensure the maximum segment length of fifty words. token alignment is largely one-to-one, with rare exceptions. clitization and set phrases (e.g. german "mu u" → "musst du" (you must), "aller handt" → "allerhand" (every kind of )) are common causes for many-to-one alignments, which our models fail to capture properly. ## discussion the semi-supervised contextualized approach results in consistent improvements for most languages and labeled data sizes (tables 4 and 5 ). unsupervised case. remarkably, with no labeled training data (and only a 500-token labeled development set), the best configuration achieves 88.4% of the top scores reported for fully supervised models ( effects of unlabeled dataset size. we typically see strong performance for languages where the unlabeled dataset u is large (≈ official training and development sets together, effects of the modern corpus and preprocessing. the size and coverage of the wikipedia dump ( normalization ambiguity. we would expect languages with higher normalization ambiguity to profit from contextualization @xcite . german, portuguese, and spanish gain even in the most competitive semi-supervised 5k condition, consistent with the amount of ambiguity they exhibit ( type-level normalization dictionary. we observe gains equivalent to using a token-level dataset of at least double the dictionary size ( shortcomings of the approach. the general problem of our approach, as well as most approaches that we build on, is reliance on gold tokenization. overall, we have faced minor issues with tokenization (one notable example is swedish where 0.6% of the target-side test data are words with a colon for which we fail to retrieve candidates from wikipedia). tokenization remains a challenge for normalization of unpreprocessed non-standard data @xcite . (bottom) semi-supervised conditions. comparisons are between neural hmm (+direct+rerank, green) and csmt @xcite . ambiguity (=whether a historical word normalizes into more than one standard word type) is computed on the official training data. ## future work clearly, one can simultaneously use both methods of candidate generation ( §4.5). we leave it for future work to verify whether this leads to an improved performance. computing the posterior p(y | x) in both generative models is hard, which is why we are forced to reduce the number of admissible candidates y and, in the case of the rnn lm-based model, approximate the posterior with maximization. this problem can be addressed in a principled way by using variational inference @xcite , a framework for approximate inference that deals with intractable distributions. we leave it for future work to validate its effectiveness for this problem. as noted earlier, it is a simplification to assume that non-standard text is tokenized. being able to normalize across token boundaries (by merging multiple non-standard tokens or splitting one into multiple standardized ones) is crucial for tackling real-world text normalization tasks and related problems such as optical character recognition error correction. an appealing direction for future work would be developing a joint model for text tokenization and normalization. one family of latent-variable models that would be suitable for this task are segmental recurrent neural networks (srnns, @xcite . srnns explicitly model input segmentation and have been successfully applied to online handwriting recognition, chinese word segmentation, joint word segmentation and part-of-speech tagging @xcite . ## conclusion this paper proposes semi-supervised contextual normalization of non-standard text. we focus on historical data, which has gained attention in the digital humanities community over the past years. we develop simple contextualized generative neural models that we train with expectationmaximization. by leveraging unlabeled data and accessing context at training time, we train accurate models with fewer manually normalized training samples. no labeled training data are necessary to achieve 88.4% of the best published performance that uses full training sets. strong gains are observed for most of the considered languages across realistic low-resource settings (up to 5k labeled training tokens). the techniques developed here readily apply to other types of normalization data (e.g. informal, dialectal). we will make our implementation publicly available. 2 modern candidates for frequent historical words. using the smallest supervised development set (500 tokens)-required by all our experimentswe compute coverage over all languages, and set the log base for squeezing the frequency ratio to 200. finally, a penalty for rare modern forms based on their frequency is added. the score for reranking the candidate list is: s h,@xmath4, m)+ max(log 200 (f h,m ), 0) + 1 #(m) . neural channel training. in every m-step (the maximization of eq. 4), we start training from the previous em iteration's best parameters θ (t-1) and train for 25 epochs with 15 epochs of patience. we optimize the parameters with adadelta (zeiler, 2012) using mini-batches of size 20. we do not update on candidates whose posterior probability is @xmath5.01 • λ -1 . if the development set scores (x,y)∈ dev p θ (x | y) do not increase compared to the previous em iteration, we restart training from randomly initialized parameters and using the type-level posterior probability from the best generative model found so far. that model, decoded using map decoding, has so far produced the highest normalization accuracy on the development set. in the semi-supervised scenario, we initially pretrain the channel for 50 epochs and 15 epochs of patience using mini-batches of size 1, as suggested by @xcite . sentence-wise reranker.