Domain-Specific J apanese ELECTRA Model Using a Small Corpus (2021)

Abstract: Recently, domain shift, which affects accuracy due to differences in data between source and target domains, has become a serious issue when using machine learning methods to solve natural language processing tasks. With additional pretraining and fine-tuning using a target domain corpus, pretraining models such as BERT (Bidirectional Encoder Representations from Transformers) can address this issue. However, the additional pretraining of the BERT model is difficult because it requires significant computing resources. The efficiently learning an encoder that classifies token replacements accurately (ELECTRA) pretraining model replaces the BERT pretraining method’s masked language modeling with a method called replaced token detection, which improves the computational efficiency and allows the additional pretraining of the model to a practical extent. Herein, we propose a method for addressing the computational efficiency of pretraining models in domain shift by constructing an ELECTRA pretraining model on a Japanese dataset and additional pretraining this model in a downstream task using a corpus from the target domain. We constructed a pretraining model for ELECTRA in Japanese and conducted experiments on a document classification task using data from Japanese news articles. Results show that even a model smaller than the pretrained model performs equally well.

## introduction a domain shift problem occurs when using machine learning to solve natural language processing tasks, where the training data (source) domain is differs from the test data (target) domain. because downstream tasks fine-tune a model, pretrained models such as bert (devlin et al., 1 bidirectional encoder representations from transformers 2019) can deal with the domain shift problem. to improve accuracy, the additional pretraining of bert using the target domain corpus and finetuning of the additional pretrained models have been used recently. however, the additional pretraining of bert requires considerable computing resources and therefore cannot be performed easily. furthermore, a large corpus of the target domain to be used is required, but this corpus is often unavailable in reality. in this paper, we attempted to address the computational efficiency of pretraining in domain shifting using efficiently learning an encoder that classifies token replacements accurately (electra) @xcite . electra is a pretraining model that uses replaced token detection (rtd) to replace the masked language modeling (mlm) used in bert. in mlm, the model is trained by estimating the @xcite and replacing some words in the input sentence with @xcite . however, only 15% of the words in bert are replaced by @xcite , which is computationally inefficient: thus, rtd is an improvement on bert. rtd provides two models: generative and discriminative, which are based on the idea of generative adversarial network (gan) @xcite . the discriminator is pretrained by deciding if it replaces each token generated by the generator. rtd is computationally more efficient because it can handle all tokens in training, and the elec-tra model using rtd performs better than the bert model of the same size. in this study, we first built a general small-scale electra model. thereafter, we constructed a domain-specific electra model by additional pretraining it using a small corpus of the target domain. although the constructed domain-specific electra model is smaller than bert-base, we confirmed that it outperforms bert-base in the target domain for a document classification task. ## method the corpus used to train pretraining models can be biased @xcite . we can use a pretraining model unique to the domain that the actual system is targeting to increase accuracy in that domain if we construct one. however, most of the current pretraining methods require considerable computational resources to be effective. in most cases, increasing the amount of computation needed for pretraining, would increase the accuracy of the downstream task; however, when conducting pretraining, considering the accuracy of the downstream task as well as the computational performance is important. therefore, herein, we use electra, that uses rtd, a computationally efficient pretraining method. electra models of various sizes evaluated the performance of the downstream tasks considering computational complexity. experiments on glue @xcite , a benchmark for natural language understanding, and squad @xcite , a benchmark for question answering techniques, were performed. for the same amount of computation, the electra model outperforms other state-of-the-art natural language processing models. for example, it performs roberta and xlnet with less than 25% of the computational effort. for further efficiency, electra-small, which can be trained in four days on a single gpu, performs better than gpt and requires only 1/30 of the computation. building a pretraining model using electra, which employs rtd, a more computationally efficient pretraining method, and performing additional pretraining on the corpus in the target domain, we can build a model with accuracy comparable to existing pretraining models for document classification tasks with fewer computational resources and less training time; see figure 1 for an overview. ## conclusion in this paper, focusing on the computational efficiency of the electra model and its good performance at scale, we constructed a domain-specific pretrained electra model by additional pretraining it using a small corpus of the target domain. although the constructed electra model is smaller than the tohoku-bert model to be compared, it achieved higher performance than tohoku-bert in document classification in the target domain. in future work, we would like to construct a pretraining model of the same size to compare the performance of the two models more rigorously.