Pretraining Language- and Domain-Specific BERT on Automatically Translated Text (2023)

Abstract: Domain-specific pretrained language models such as SciBERT are effective for various tasks involving text in specific domains. However, pretraining BERT requires a large-scale language resource, which is not necessarily available in fine-grained domains, especially in non-English languages. In this study, we focus on a setting with no available domain-specific text for pretraining. To this end, we propose a simple framework that trains a BERT on text in the target language automatically translated from a resource-rich language, e.g., English. In this paper, we particularly focus on the materials science domain in Japanese. Our experiments pertain to the task of entity and relation extraction for this domain and language. The experiments demonstrate that the various models pretrained on translated texts consistently perform better than the general BERT in terms of F1 scores although the domain-specific BERTs do not use any human-authored domain-specific text. These results imply that BERTs for various low-resource domains can be successfully trained on texts automatically translated from resource-rich languages.

## introduction domain-specific pretrained language models (lms), such as scibert @xcite , are known to perform better on many downstream tasks with texts in the specific domain, such as named entity recognition in biomedical @xcite and relation extraction in chemical domains @xcite . this trend has motivated researchers to release many domain-specific lms for resource-rich domains and languages, specifically in medicine @xcite , biomedicine @xcite , finance @xcite , and materials science @xcite . many of the domain-specific lms have been trained on corpora consisting of academic papers or articles, which are usually open to the public. however, such open corpora are often not available in non-english languages. meanwhile, there are a lot of documents that are not open such as internal corporate documents also in non-english languages, which still need to be processed with pretrained lms. we focus on a novel setup for pretraining domain-specific berts without the use of humanauthored domain-specific text. as a solution to the problem, we pretrain lms on domain-specific text automatically translated from a resource-rich language, i.e., english. as shown in figure 1 , journal papers are automatically translated from english to the target language, e.g., japanese in this paper, then used in bert pretraining in different configurations with or without general texts, e.g., japanese wikipedia, to investigate the viability on domain-specific japanese text. although this is a very simple approach with wide applicability to various domains and languages, the following two questions still need to be answered: 1) is the use of translated text effective in various strategies for pretraining bert? and 2) does the vocabulary induced from the domain-specific corpus improve performance? we evaluate our pretrained bert models on named entity extraction and relation extraction for the materials science domain in japanese due to its high demand. the results empirically show that all the models trained on the translated text consistently achieve better performance than the model trained only on the general text, despite the fact that noise may exist in translation @xcite . in addition, we found that the domainspecific vocabularies are effective when bert is pretrained on a mixture of the two corpora. our contributions are: 1) we propose a new setup for pretraining domain-specific berts without any human-authored domain-specific text in the target language, 2) we show the effectiveness of the use of translated text for various pretraining strategies, 3) we release the japanese bert specific to the domain of materials science and a web-based application of information extractors where even non-nlp experts can benefit from our bert foot_0 . ## related work different types of llms have different architectures. for example, bert @xcite has only an encoder, @xcite adopts decoder-only model, and bart @xcite adopts an encoder-decoder architecture. although gpt-like models are more actively studied recently, we focus on bert because it is still fundamental to many entity and relation extractors in many domains @xcite . various pretraining methods for bert have been proposed. the original bert uses only the general text @xcite . scibert is trained on domain-specific text @xcite . others adapt an lm, pretrained on the general domain, to specific domains by continuing the pretraining on domain-specific text @xcite @xcite . multilingual bert (mbert) @xcite is trained on a mixture of multiple corpora written in different languages. a domain-specific bert can also be trained on a mixture of a general and a domain-specific corpus. the methods above use different vocabularies consisting of only general domain tokens @xcite , only domain-specific tokens @xcite , or tokens extracted from the union of the two @xcite . we examine the impact of different combinations of data usages and vocabularies. our approach is partly inspired by data augmentation techniques that benefit from machine translation @xcite , whereby labelled data were augmented for reading comprehension @xcite , fake news detection @xcite and other tasks. unlike those approaches, our focus is on augmenting unlabelled data for pretraining, which has not been well explored, compared with augmenting labelled data for finetuning. ## methodology we show details about our collection of translated domain-specific texts, the data usage and vocabulary for pretraining. ## experiments we explain tasks, models, and datasets used for evaluating the proposed berts. ## results tables 3 and 4 show the respective scores for the two different settings: "glass transition temperature" and "elasticity". the span-based macro-precision, recall and f-score, which are commonly used, e.g., in @xcite , are adopted as evaluation metrics. from top to bottom for both tables, we show the performances of the baseline (model i) and nine proposed models (model ii to x). the proposed models are divided into three categories based on pretraining methods: 1) transtrain, 2) mixtrain, and 3) pipelinetrain. for evaluating the relation extractors, we report performances on two settings; whether we use gold entities as input @xcite or not in evaluation. ## conclusion we showed that translated texts are beneficial for pretraining domain-specific berts in a lowresource language despite occasional translationese @xcite . our approach can be applied to other languages and domains in which large-scale corpora are hard to obtain. in future work, our approach will be investigated on other pretrained models, e.g., gpt or bart, as well as other domains and languages. we leave investigations on the correlation between the translation qualities and downstream task performance as a future direction.