When FLUE Meets FLANG : Benchmarks and Large Pretrained Language Model for Financial Domain (2022)

Abstract: Pre-trained language models have shown impressive performance on a variety of tasks and domains. Previous research on financial language models usually employs a generic training scheme to train standard model architectures, without completely leveraging the richness of the financial data. We propose a novel domain specific Financial LANGuage model (FLANG) which uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective. Additionally, the evaluation benchmarks in the field have been limited. To this end, we contribute the Financial Language Understanding Evaluation (FLUE), an open-source comprehensive suite of benchmarks for the financial domain. These include new benchmarks across 5 NLP tasks in financial domain as well as common benchmarks used in the previous research. Experiments on these benchmarks suggest that our model outperforms those in prior literature on a variety of NLP tasks. Our models, code and benchmark data will be made publicly available on Github and Huggingface.

## introduction efficient financial markets incorporate all price relevant information available to investors at that point of time. unstructured data, such as textual data, help complement structured data traditionally used by investors. for example, in addition to quantitative data such as firm's financial performance, the tone and sentiment of firms' financial reports, earnings calls and social media posts can also influence the stock price movement @xcite email ids of the authors: {rajsanjayshah, kunalchawla, deidnani, ashah482, wendi.du, schava6, jchen896}@gatech.edu, natraj.raman@jpmorgan.com, charese.h.smiley@jpmchase.com, diyiy@cs.stanford.edu * these authors contributed equally to this work 1 the website can be found at https://saltnlp.github.io/flang/ . all the flang models are available on the huggingface salt-nlp site. 2020). we aim to capture these textual features with the help of pre-trained deep learning models, which have shown superior performance in a variety of natural language processing (nlp) tasks @xcite @xcite . however, the language used in finance and economics is likely to be different from the language of common usage. a statement like "the crude oil prices are going up" has a negative sentiment for the financial markets, but it does not contain traditionally negative words such as danger, hate, fear, etc. @xcite . therefore, it is necessary to develop a domain-specific language model training methodology that improves the performance in the downstream nlp tasks like managers' sentiment analysis and financial news classification. previous research, for example, @xcite ; @xcite have pre-trained the state-ofthe-art language models like bert @xcite with financial documents, but suffer from two major limitations. first, financial domain knowledge and adaptation are not utilized in the pre-training process. we argue that the financial terminologies play a critical role in understanding the language used in financial markets, and expect a performance improvement after incorporating the financial domain knowledge into the pre-training process. second, the lack of different evaluation benchmarks limit the test the language models' performance in finance-related tasks. in this work, we propose a simple yet effective language model pre-training methodology with preferential token masking and prediction of phrases. this helps capture the fact that many financial terms are actually multi-token phrases, such as margin call and break-even analysis. we contribute and make public two language models trained using this technique. financial language model (flang-bert) is based on bert-base architecture @xcite , which has a relatively figure 1 : architecture of our model. we use finance specific datasets and general english datasets (wikpedia and bookscorpus) for training the model. we follow the training strategy of electra @xcite with span boundary task which first predicts masked tokens using language model and then uses a discriminator to assess if a token is original or replaced. the generator and discriminator are trained end-to-end, and both words and phrases from financial vocabulary are used for masking. the final discriminator is then fine-tuned on individual tasks on our contributed benchmark suite, financial language understanding evaluation (flue). note that our method is not specific to electra and can be generalized to other models. small memory footprint and inference time. it also enables comparison with previous works, most of which are based on bert. we also contribute flang-electra, our best performing model, based on the electra-base architecture @xcite , where we introduce a span boundary objective on the electra generator pre-training task to learn robust financial multi-word representations while masking contiguous spans of text. we show that flang-bert outperforms all previous works in nearly all our benchmarks, and flang-electra further improves the performance giving two new state-of-the-art models. our training methodology can be extended to other domains that would benefit from domain adaptation. financial domain benchmarks are critical to evaluate the newly developed financial language models. inspired by glue @xcite , a set of comprehensive benchmarks across multiple nlp tasks, we construct financial language understanding evaluation (flue) benchmarks. flue consists of 5 financial domain tasks: financial sentiment analysis, news headline classification, name entity recognition, structure boundary detection, and question answering. we intend for this benchmark suite to be a standard for evaluation of natural language tasks in financial domain, subject to appropriate license and privacy considerations. all proposed benchmarks will be made publicly available on github and huggingface. our contributions are as follows: ## model for flang-bert, we add financial word and phrase masking, while for flang-electra, we also add a span boundary objective. the addition of financial word and phrasal masking is model agnostic and can be used for any model with a generator. ## conclusion we contribute two language models in the finance domain, which use domain-specific word and phrase masking as a pre-training objective. additionally, we contribute a comprehensive suite of benchmarks in finance domain across 5 natural language tasks, including new benchmarks using public sources. our language model outperforms previous language models on all the benchmarks. we will release our models, code and benchmark data on acceptance. we also note that our method is not specific to finance and can be used for any domain-specific language model training.