Cross-domain Analysis on J apanese Legal Pretrained Language Models (2022)

Abstract: This paper investigates the pretrained language model (PLM) specialised in the Japanese legal domain. We create PLMs using different pretraining strategies and investigate their performance across multiple domains. Our findings are (i) the PLM built with general domain data can be improved by further pretraining with domain-specific data, (ii) domain-specific PLMs can learn domain-specific and general word meanings simultaneously and can distinguish them, (iii) domain-specific PLMs work better on its target domain; still, the PLMs retain the information learnt in the original PLM even after being further pretrained with domain-specific data, (iv) the PLMs sequentially pretrained with corpora of different domains show high performance for the later learnt domains.

## introduction transformer-based pretrained language models (plms) such as bert @xcite and its successors @xcite @xcite achieved solid performance in various nlp tasks for a generic domain @xcite . following their success, domain-specific plms have been proposed for science @xcite , medical @xcite , financial @xcite , and legal @xcite domains. these domain-specific plms are pretrained solely with the target domain corpora, or with both the generic and target domain corpora. the latter is a good option when the domain corpus size is limited. @xcite empirically proved that further pretraining a generic plm using domainspecific corpora provided benefits; @xcite confirmed this claim for the legal domain. however, previous studies do not care the performance of the domain-adapted plms for a generic domain. the domain adaptation might degrade the model performance for a generic domain. the domain-adapted plm should perform well in both the target domain and the domain in general. this requirement is essential for the legal domain, where the legal argumentation includes evidence descriptions cited from non-legal text such as web pages, books and sns posts. the requirement is related to catastrophic forgetting. @xcite recently showed that more steps and data for pretraining make a model robust against catastrophic forgetting. however, their findings are primarily in computer vision, and their experiments with plms are still preliminary. they focus on sequential fine-tuning of various size plms pretrained with a single domain corpus. on the other hand, we focus on pretraining plms with different domains through evaluation using corpora from 13 domains, including domains exclusive of training data. also, compared with english, there are few findings in domain adaptation strategies of japanese plms, despite several japanese plms available for the generic (nict, 2020; tohoku nlp group, 2021; nlp-waseda, 2021), financial @xcite and medical @xcite domains. further, despite its significance, no plm study exists in the japanese legal domain. in the recent coliee workshop, a competition on legal information extraction and entailment tasks, including the japanese language, most high-scoring approaches utilise bert-like plms @xcite trained on japanese wikipedia text. although there is an expectation that plms trained with japanese legal corpora improve their performance, the insufficient size of publicly available corpora does not allow it. further pretraining a generic plm with available legal corpora is one of the promising adaptation strategies. against this backdrop, particularly considering the above-mentioned legal-domain peculiarity that both domain-specific and generic meanings are equally important, this paper reports the first comprehensive study on plm adaptation strategies in the japanese legal domain and their performance across different domains through intrinsic evaluation. ## research questions chalkidis et al. (2020) adopted two strategies for pretraining domain-specific plms: further pretraining (fp) an existing plm with the domain corpus and pretraining a domain-specific plm with the domain corpus from scratch (sc). comparing these two strategies, we investigate the cross-domain performance of domain-specific plms, specialised in the japanese legal domain. we set up the following research questions. rq1: is the fp/sc learning strategy effective and which is more effective? rq2: can the domain-adapted plm learn the domain-specific meaning and distinguish it from the meaning of general usage? rq3: does the plm performance change across the domain? rq4: what is the best order of training data domains for pretraining? base plm we use the bert-base (wwm version) checkpoint by @xcite , which is pretrained with the japanese wikipedia dataset foot_2 . ## conclusion this paper presents an empirical study of the pretrained language model specialised in the japanese legal domain. our findings are (i) the plm built with general domain data can be improved by further pretraining with domain-specific data, (ii) domain-specific plms can learn domain-specific and general word meanings simultaneously and can distinguish them, (iii) domainas we compared the plm performance across different domains, we adopted intrinsic evaluation with domain-neutral tasks, mlm and nsp. as gururangan et al. (2020) did, our future plan includes conducting extrinsic evaluation using downstream tasks like jglue @xcite . according to our annotation, 200 out of 1088 sentences mean "malicious" in jd (test). even in jd, which is a corpus of legal domain, "akui" does not always mean "knowing a fact" but also means "malicious". for example, a legal argumentation includes evidence descriptions cited from non-legal text such as web pages, books and sns posts. moreover, both meanings can simultaneously appear in a single document. thus, source of documents does not necessarily suggest which meaning "akui" has.