A J apanese Masked Language Model for Academic Domain (2022)

Abstract: We release a pretrained Japanese masked language model for an academic domain. Pretrained masked language models have recently improved the performance of various natural language processing applications. In domains such as medical and academic, which include a lot of technical terms, domain-specific pretraining is effective. While domain-specific masked language models for medical and SNS domains are widely used in Japanese, along with domain-independent ones, pretrained models specific to the academic domain are not publicly available. In this study, we pretrained a RoBERTa-based Japanese masked language model on paper abstracts from the academic database CiNii Articles. Experimental results on Japanese text classification in the academic domain revealed the effectiveness of the proposed model over existing pretrained models.

## introduction academic papers in various fields and languages are accumulating daily on the web. for example, more than 76k papers in the field of natural language processing (nlp) are currently available on the acl anthology. 1 since the cost for humans to exhaustively learn from these large numbers of academic papers is immeasurable, scholarly document processing by nlp @xcite @xcite is promising. in nlp based on deep learning, which is currently the mainstream, supervised learning with a large-scale labeled corpus is effective. however, in domains where technical terms are frequently used, such as in academic fields, hiring professional annotators is very expensive. therefore, the lowresource problem is a serious issue in various languages, domains, and tasks. in recent nlp, finetuning of pretrained masked language models on large-scale raw corpora, such 1 https://aclanthology.org/ as bert @xcite and roberta @xcite , has been widely employed to address the low-resource problem. especially in domains such as medical @xcite and academic @xcite , the effectiveness of domain-specific pretraining has been reported. similar to these previous studies in english, domain-specific masked language models for medical @xcite and sns foot_0 are widely used in japanese, along with domainindependent masked language models. 3,4 however, there are no publicly available pretrained japanese models that are specific to the academic domain. in this study, we pretrained a roberta-based japanese masked language model @xcite using 6.28m sentences of paper abstracts from a scholarly article database cinii articles foot_3 to improve the performance of scholarly document processing in japanese. experimental results on japanese text classification in the academic domain revealed the effectiveness of the proposed model, which is specific to the academic domain, compared to the domain-independent masked language models. our model (academic roberta) will be available on github foot_4 when this paper is published. ## related work finetuning of pretrained transformer @xcite achieves excellent performance on many nlp tasks @xcite . bert @xcite , a typical pretraining model, trains the transformer encoder by multi-task learning of masked language modeling and next sentence pre-diction. roberta @xcite outperforms bert by pretraining only masked language modeling, through dynamic masking and increasing batch size and number of training steps. this study conducts powerful roberta-based pre-training to develop a japanese masked language model specific to the academic domain. the effectiveness of domain-specific pretraining to address technical terms and style-specific expressions has been reported. in english, domainspecific masked language models are publicly available for various domains, including medical @xcite , academic @xcite , and sns @xcite . domain-specific masked language models have also been developed in japanese, such as uth-bert @xcite for the medical domain and hottosns-bert 2 for the sns domain. however, no pretraining japanese model specific to the academic domain has been released. ## methods to improve the performance of scholarly document processing in japanese, we release a japanese masked language model specific to the academic domain. first, in section 3.1, we create a japanese corpus consisting of paper abstracts. then, in section 3.2, we use this corpus to conduct pretraining based on roberta @xcite . ## evaluation to evaluate the effectiveness of our masked language model (academic roberta) specific to the academic domain, we empirically compare our model with existing domain-independent masked language models through experiments on japanese text classification in the academic domain. ## conclusion in this study, we released academic roberta, a japanese masked language model specific to the academic domain, pretrained on abstracts of academic papers included in cinii articles. experimental results on japanese text classification in the academic domain revealed that our model consistently outperforms existing domain-independent masked language models. detailed analysis confirmed the effectiveness of domain-specific pretraining, as many domain-specific expressions were included in the vocabulary and the accuracy of text classification improved significantly for more detailed categories requiring more expertise. our future work includes making japanese text generation models such as gpt-2 @xcite and bart @xcite specific to the academic domain. these models could contribute to summarization and grammatical error correction in the academic domain.