Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models (2023)

Abstract: Over the past few years, various domain-specific pretrained language models (PLMs) have been proposed and have outperformed general-domain PLMs in specialized areas such as biomedical, scientific, and clinical domains. In addition, financial PLMs have been studied because of the high economic impact of financial data analysis. However, we found that financial PLMs were not pretrained on sufficiently diverse financial data. This lack of diverse training data leads to a subpar generalization performance, resulting in general-purpose PLMs, including BERT, often outperforming financial PLMs on many downstream tasks. To address this issue, we collected a broad range of financial corpus and trained the Financial Language Model (FiLM) on these diverse datasets. Our experimental results confirm that FiLM outperforms not only existing financial PLMs but also general domain PLMs. Furthermore, we provide empirical evidence that this improvement can be achieved even for unseen corpus groups.

## introduction pretrained language models (plms) have been successfully employed in various natural language processing tasks. this trend has been extended to further pretraining of plms on domain-specific corpus in various fields such as biomedical @xcite , scientific @xcite , and clinical @xcite domains. in financial domain, @xcite ; @xcite ; @xcite ; @xcite proposed domain-specific plms based on bert @xcite model. however, each study tested a small number of tasks, resulting in inadequate validation. we conduct extensive experiments to show that the generalization performance of existing financial plms is unsatisfactory. even general domain plms such as bert and roberta (liu * major in bio artificial intelligence et al., 2019) outperformed the financial plms on financial tasks. to investigate the reasons for the low generalization performance, we categorize the financial corpus into five groups and examine their use in existing financial plms. this reveals that most exexisting plms are pretrained on a corpus consisting of only two of these corpus groups. this indicates that the pretraining data for financial plms lacks diversity, which might cause their low generalization performance across various financial tasks. motivated by this observation, we collect a broad range of corpus and analyze the impact of corpus diversity on the performance of language models. our investigations demonstrate that as a corpus becomes more diverse, the model's performance improves. incorporating a diverse corpus, even with fewer tokens, yields a better generalization performance than relying on numerous tokens from a nondiverse corpus. furthermore, when using diverse corpora, the model exhibits robust generalization performance on unseen tasks. we train our financial language model (film) on a corpus with diverse documents and evaluate it on six financial tasks, including recently introduced works @xcite @xcite . our experimental results show that film outperforms not only existing financial plms but also general domain plms on most financial tasks. in addition, we achieve an improvement of performance while reducing the number of tokens trained and energy consumption. to the best of our knowledge, film is the first model to surpass roberta in financial domain. we make our model and code publicly available on github foot_0 and huggingface hub foot_1 for continuous advancement in financial domain. ## proposed method first, the datasets used to train the model are introduced. second, we describe the method for preprocessing the datasets. finally, we describe film training procedure. ## financial tasks for evaluation we evaluate the performance of financial plms on the following six financial nlp tasks: figure 1b shows the embeddings of the sentences sampled from the financial tasks. the colors of each point distinguish the six tasks and the gray points represent the sentences from the pretraining data. finer and finqa are located on the top side because both tasks use sec filings to create the dataset. meanwhile, the headline task is located further away from the other tasks due to its focus on the gold commodity. in addition, the sentences in fpb, ner, and fomc form their own clusters, separate from the other datasets. as observed, each financial nlp task has unique aims and distinctive textual features. this implies that the performance of language model has to be evaluated across a broad range of tasks. for more detailed information about financial tasks, please refer to appendix b.