F in BART : A Pre-trained Seq2seq Language Model for C hinese Financial Tasks (2023)

Abstract: “Pretrained language models are making a more profound impact on our lives than ever before. They exhibit promising performance on a variety of general domain Natural Language Process-ing (NLP) tasks. However, few work focuses on Chinese financial NLP tasks, which comprisea significant portion of social communication. To this end, we propose FinBART, a pretrainedseq2seq language model for Chinese financial communication tasks. Experiments show thatFinBART outperforms baseline models on a series of downstream tasks including text classifica-tion, sequence labeling and text generation. We further pretrain the model on customer servicecorpora, and results show that our model outperforms baseline models and achieves promisingperformance on various real world customer service text mining tasks.”

## introduction from making investment decisions to managing personal finances, financial text data is crucial in helping individuals and organizations stay up-to-date with the latest financial trends. with the development of information technology, the volume of available financial information looms so large that the use of machine learning methods to facilitate financial text processing becomes increasingly important. however, financial text data differs from general domain data as it contains a large number of technical terms and financial concepts, which require abundant expert knowledge to analyze and annotate. as a result, the application of powerful deep learning technology in financial text processing tasks is limited. in this work, we apply the fast-developing pretraining techniques to chinese financial domain. although a number of works attempt to adapt transformer-based @xcite pretrained language models to financial text @xcite @xcite , few studies focus on the application on chinese financial text data. what's worse, these works adopt transformer encoder architecture, leading to limited ability in modeling the dependency between future tokens and poor performance in text generation tasks. to this end, we propose finbart, which is an encoder-decoder transformer language model pretrained on chinese financial corpora. fin-bart extracts bi-directional semantic information with its encoder, and generates text with the decoder part. in this way, finbart is capable of tackling both understanding and generation tasks in chinese financial domain. in real world scenarios of the financial industry, customer service is mainly conducted through text interaction. therefore, there is a great demand for text analysis tools. customer service corpora differ from financial ones for their sample text is highly colloquial. although finbart learns professional knowledge from the financial field, it cannot handle customer service tasks as well as financial ones without adaptation. to this end, we further pretrain finbart model with customer service corpora, which are comprised of customers' queries about financial services. further pretraining on customer service corpora adapts finbart to text interaction scenarios, enabling finbart to capture accurate semantic information of customer queries. in summary, the main contributions of this work are listed as follows: ## related works unsupervised pretraining technique is regarded as one of the most significant breakthroughs in recent natural language processing (nlp) research. taking advantage of unsupervised pretraining methods, pretrained language models (plms) achieve remarkable performance on a variety of natural language understanding and generation tasks. decoder-only language models, represented by gpt @xcite , possess a remarkable capability for text generation, but lack language understanding ability because they model semantic information in a causual way. encoder-only language models like bert @xcite , electra @xcite and roberta @xcite are proposed to introduce bidirenctional semantic information to obtain more representative word embeddings for downstream tasks. however, encoder-only models cannot model the dependency between [mask] tokens and therefore suffer from limited generation capability. encoderdecoder transformer models are more versatile because they extract bidirenctional semantic information with the encoder part and generate text smoothly with their decoders. representative encoder-decoder models are bart @xcite and t5 @xcite , which adopt different pretraining objectives to gain general language modeling ability. although it is not far from trivial to adapt language models pretrained on general corpus to domainspecific downstream tasks, their performance may be limited because of the lack of professional knowledge. this problem looms larger for domains involving large amounts of technical terms and concepts like finance. to this end, researchers propose to pretrain language models on domain-specific corpus to endow models with professional skills for downstream tasks. biobert @xcite collects large biomedical corpora and pretrains a transformer encoder model to produce contextualized word representation for biomedical text. scibert @xcite trains a bert model for scientific nlp tasks with scientific publication corpora. in financial industry, researchers also seek to train domain-specifc language models to facilitate financial text meterials processing. finbert @xcite designs a series of pretraining objectives for english financial text to train the model more effectively. mengzi-bert-fin @xcite further trains chinese bert model to adapt to chinese financial text. however, these works are mainly designed for engligsh natural lnaguage understanding tasks and lack the ability to tackle chinese nlp tasks involving generation. our work fills this gap by pretraining a seq2seq transformer-based language model on chinese financial corpora. ## methods in this section, we introduce the pretraining procudure of the proposed finbart and finbart-cs. we choose transformer encoder-decoder architecture as the backbone, reconciling language understanding and generation ability. we use financial corpora and customer service corpora for model pretraining, and then finetune finbart and finbart-cs in their domain-specific downstream tasks, respectively. the overall data collecting, model pretraining and downstream task adapting pipeline is shown in figure 1 . ## experimetns in this section, we introduce the experimental settings and resources used in the pre-training and downstream task adaptation stages of finbart and finbart-cs. ## conclusions in this work, we propose finbart and finbart-cs, which are chinese encoder-decoder language models pretrained on chinese financial corpora and customer service corpora, respectively. the proposed models fill the blank of chinese seq2seq language model for financial and customer service nlp tasks. experiments on a series of chinese financial nlp tasks and customer service nlp tasks indicate the promising performance of the proposed models. we also conduct analysis experiments to show the rationality of pretraining hyper parameter selection and the effectiveness of the proposed pretraining objective for finbart-cs.