Using Language Models to Improve Rule-based Linguistic Annotation of Modern Historical J apanese Corpora (2022)

Abstract: Annotation of unlabeled textual corpora with linguistic metadata is a fundamental technology in many scholarly workflows in the digital humanities (DH). Pretrained natural language processing pipelines offer tokenization, tagging, and dependency parsing of raw text simultaneously using an annotation scheme like Universal Dependencies (UD). However, the accuracy of these UD tools remains unknown for historical texts and current methods lack mechanisms that enable helpful evaluations by domain experts. To address both points for the case of Modern Historical Japanese text, this paper proposes the use of unsupervised domain adaptation methods to develop a domain-adapted language model (LM) that can flag instances of inaccurate UD output from a pretrained LM and the use of these instances to form rules that, when applied, improves pretrained annotation accuracy. To test the efficacy of the proposed approach, the paper evaluates the domain-adapted LM against three baselines that are not adapted to the historical domain. The experiments conducted demonstrate that the domain-adapted LM improves UD annotation in the Modern Historical Japanese domain and that rules produced using this LM are best indicative of characteristics of the domain in terms of out-of-vocabulary rate and candidate normalized form discovery for “difficult” bigram terms.

## introduction annotating unlabeled corpora with linguistic metadata is a fundamental task in many scholarly workflows in the digital humanities (dh) @xcite . these can benefit from the application of "off-the-shelf", or pretrained, natural language processing (nlp) pipelines that supply tokenization, tagging, and dependency parsing simultaneously using an annotation scheme like universal dependencies (ud) @xcite . for these tools to warrant any integration, the annotations produced must be accurate for the target corpus under study and enable helpful evaluation by domain experts. current efforts have prioritized the former as the accuracy of pretrained ud tools remains unknown for historical texts sampled from domains that are different from pretraining domains @xcite . in applications to historical text in east asian languages like chinese and japanese that can be written without specifying word boundaries, substantial errors in word segmentation can result in degraded accuracy of resulting dependency parsings @xcite . the insufficiency of the vocabulary used by pretrained ud tools can be a major source of errors in word segmentation. consequently, a straightforward application is likely not possible without extensive manual revision by domain experts and the process can be prohibitive when corpus size is large @xcite . recent work in nlp has addressed the accuracy of sequence labeling tasks (e.g., word segmentation, part of speech annotation, named entity recognition) for historical materials through unsupervised finetuning of pretrained contextualized word embeddings using transformer-based language models (lms) @xcite . these finetuned lms are capable of capturing useful high-level features about the domain without requiring any labels from the target corpus. the improvements are encouraging, yet, the usefulness of a transformer-based method for a domain expert remains unknown. the lack of supervision in these methods means that direct mechanisms for finetuning the lm other than through the training data used are beyond the control of a domain expert. put another way, the explicit representations of domain knowledge that occur during manual revision of pretrained output is not possible with current nlp for adapting pretrained lms to historical corpora. in prior work, @xcite pro-posed a rule-based expert system for the case of modern historical japanese corpora that generates accurate ud annotations using a set of handcrafted rules. the rules compose a two-parted workflow that (1) normalizes non-standard lexical variants to a more canonical form so that the normalized text can receive a more accurate parsing by a pretrained tool, and (2) an assignment step where the updated ud is then linked to word forms from the original text. figure 1 shows an overview. the use of rules allows for immediate improvement in ud annotation for out-of-vocabulary terms by parsing dependencies using substituted terms that are present in the vocabulary, and then restoring the historical forms in the parsed sentence. moreover, each rule forms a direct application of domain knowledge and supports human comprehension. however, because rule generation cannot proceed without manual review, the workflow can be a time intensive undertaking when the number of rules needed to achieve improved accuracy is not known. this paper aims to enrich the rule-based expert system by incorporating it into a workflow that is usefully guided by a pretrained lm adapted to the domain of modern historical japanese. the contributions of the proposed workflow are as follows: the proposed method offers improved ud accuracy for modern historical japanese corpora while also enabling useful evaluations by a domain expert. this can serve as welcome news to dh scholars who would like to make more frequent use of transformer-based nlp methods in their scholarship. figure 1 : overview of the rule-based expert system. in this example, the rule '假令へ'→ 'たとえ' is applied to the sentence '否假令へ貸倒れとならずとするも、'. after string replacement of the left-hand side with the right-hand side, the normalized sentence is submitted to a pretrained ud tool for annotation. the updated ud returned is then aligned with historical word forms (i.e., '假令へ') from the source text. universal dependencies (ud) is a community effort for cross-linguistic annotation of grammar (parts of speech, morphological information, and syntactic dependencies) @xcite . recent methods have been put forward for generating ud annotations using transformer-based language models, e.g., bert and electra, as a backbone @xcite . these are also available for generating ud from japanese text input. @xcite . however, the transfer of adaptive bert models when using these methods for the case of historical textand specifically historical japanese text -remains to be evaluated thoroughly. the need for helpful evaluations of nlp-based tools is gaining traction in the dh community @xcite . in the case of historical japanese, @xcite trains a combination of udpipe and crf++ for sequence labeling using training data made available through the labor-intensive corrections done by domain experts. to the best of our knowledge, this paper is the first to address the applicability of already pretrained tools through domain adaptation methods and the use of a pretrained neural architecture as a mechanism for enhancing an expert system that generates ud annotations for the case of modern historical japanese corpora. ## evaluation a prerequisite to using a domain-adapted lm to enrich the rule-based expert system is that it must first produce accurate ud in the target domain. because taiyo is unlabeled, we evaluate against ground truth labels from similar corpora in the target domain using the ud-japanese-modern treebank. moreover, no canonical train-test split exists for taiyo so we randomly partition the documents in a 75%/25% scheme (2121/707 documents) where the training set is used for domain tuning and the testing set for development of the rule set. contexts with misclassified bigram terms sentences domain-adapted lm pretrained lm normalized form ranking ud consistency verification mlm prediction context masking historical word forms candidate normalized forms figure 4: flow diagram illustrating steps for rule generation using the domain-adapted lm. the goal of the module is to develop candidate normalized forms that can be used as substitutions for historical word forms. the mapping from a historical word form to some candidate normalized form composes one potential rule. ## results the results we report in this section are designed to answer the questions: (1) does the use of domain adaptation bring an improvement in ud annotation for the modern historical japanese domain, (2) when compared to baseline systems, do the flagged instances raised by adapt-esupar suggest unique characteristics about the target corpus, and (3) does the application of adapt-esupar bring the best discovery rate for potential rules in the target domain? ## conclusion this paper demonstrates the use of domain adaptation methods for bringing improved ud annotation in the modern historical japanese domain. it incorporates a domain-adapted lm into a workflow designed to enable evaluation by domain experts. features salient to this workflow: the domain-adapted lm is deployed to flag instances of incorrect pretrained ud output and these are then used to form rules that, when applied, improve annotation accuracy in these contexts. the rules can be used to supplement and enrich a rule-based expert system. our experiments indicate that domain adaptation is a necessary step to enable flagging of incorrect ud output and generation of candidate normalized forms that can be used to build a rule set. however, we find that the choice of source domain data used for domain adaptation is significant, especially when there exists considerable similarity between data sampled from the source and target domains. to best maximize this transfer, the source data sampled should minimize similarity with the target domain. we are interested in exploiting associations between degree of dissimilarity in domains and margin of improvement brought by transfer learning in the context of historical texts. future work will also do well to explore metrics other than vocabulary overlap for quantifying this similarity. we hope to have the proposed workflow reviewed and evaluated by domain experts in the future, and that this work can help pave the path toward greater adoption of pretrained neural architectures into scholarly workflows in dh.