Is Domain Adaptation Worth Your Investment? Comparing BERT and F in BERT on Financial Tasks (2021)

Abstract: With the recent rise in popularity of Transformer models in Natural Language Processing, research efforts have been dedicated to the development of domain-adapted versions of BERT-like architectures. In this study, we focus on FinBERT, a Transformer model trained on text from the financial domain. By comparing its performances with the original BERT on a wide variety of financial text processing tasks, we found continual pretraining from the original model to be the more beneficial option. Domain-specific pretraining from scratch, conversely, seems to be less effective.

## introduction the transformer architectures have taken the field of natural language processing (nlp) by storm, leading to remarkable performance leaps in several tasks @xcite . the first-generation transformers were mainly trained on general corpora, such as wikipedia or common crawl. however, considering domain adaptations, many researchers have later injected domain-specific knowledge in such architectures, leading to the publication of transformers trained on different types of in-domain text, e.g., scientific articles @xcite , biomedical text @xcite , clinical notes @xcite , and patent corpora @xcite . since language technologies have seen increasingly frequent use in accounting and finance @xcite , it is not surprising that several attempts have been made to adapt transformers to the financial domain @xcite @xcite . in this study, we test the finbert model by @xcite on a variety of tasks in the field of financial nlp, including sentiment analysis, causality detection, numeral understanding, and numeral attachment, and we study the impact of different types of pretraining on the system performance. we obtained the best results with a fin-bert model with pretraining continuing from the original bert and with the same general-domain vocabulary, while a model trained anew on financial corpora and with a domain-adapted vocabulary performed similarly to bert base. ## related work although financial nlp is a relatively recent field, it already has an active research community, which has regularly introduced new shared tasks and benchmarks in recent years, e.g., sentence boundary detection in financial documents @xcite @xcite , hypernymy detection (el @xcite , document causality detection @xcite , document structure extraction @xcite , and document summarization @xcite . given the success of transformer models in general-domain nlp, it is not surprising that they are also a popular choice for many systems competing in financial tasks @xcite . to adapt the original bert to sentiment analysis in the financial domain, araci (2019) was the first to propose a finbert model by further pretraining bert base on the financial subset of the reuters trc2 corpus. the evaluation, carried out on the financial phrase bank @xcite and the fiqa sentiment scoring dataset @xcite , demonstrated that finbert largely outperformed all the lstm-based baselines and was slightly better than the original model. the second finbert model, introduced by @xcite , followed two different training strategies. the first version (finbert base-vocab) was further pretrained from a bert base checkpoint on three financial corpora (i.e., the corporate reports 10-k & 10-q from the securities exchange commission, foot_0 the earnings call transcripts from the seeking alpha website, foot_1 and the analyst reports from the investext database), and the second (finbert finvocab) was trained afresh on the same three corpora but with new vocabulary specific to the financial domain, not inheriting it from the original bert. they evaluated the models on the same sentiment analysis datasets, in conjunction with the opinion mining data from @xcite , and reported improved performance over bert base, especially when using the finbert model with the domain-adapted vocabulary. in this study, we chose to test the finbert system used in @xcite , which has two publicly available versions, in order to directly compare the impact of the two different domain adaptation strategies and to evaluate them on more semantic tasks. the previous studies @xcite focused their evaluation exclusively on sentiment analysis. however, sentiment analysis is a general task that is not necessarily ideal for observing the advantages of domain adaptation because the expressions of sentiment might not reflect the in-domain language. for example, in the biomedical domain, several tasks have recently been shown to benefit from training from scratch on an in-domain text and from a domainspecific vocabulary @xcite . therefore, besides sentiment analysis, we decided to evaluate our models on three semantic tasks that are more specific to the financial domain: document causality detection @xcite , numeral understanding @xcite , and numeral attachment @xcite . ## results and discussion the full results are shown in ## conclusions in this paper, we compared the original bert model with the financially adapted models by @xcite . domain adaptation was generally confirmed to be beneficial and, unlike what has been recently observed in the biomedical domain @xcite , the model benefiting from continuous pretraining from bert base showed more consistent improvements across tasks and datasets. this suggests that the models take advantage from exposure to financial text, but the tasks do not necessarily require a specialized vocabulary. on the negative side, fluctations in the results confirmed that there is some degree of instability in the fine-tuning of bert-like models on relatively small datasets @xcite . in our future work, we plan to investigate also the contextualized embeddings produced by the domain-adapted transformers. word embeddings have been used in tasks with important applications in the financial domain, such as the identification of semantic relations @xcite , which is useful for building domain ontologies @xcite @xcite , and the unsupervised detection of semantic changes in diachronic data, e.g., annual reports of traded companies @xcite @xcite . in this perspective, a promising research direction would be to analyze how different domain adaptation strategies affect the quality of the embedding representations.