M ulti F in: A Dataset for Multilingual Financial NLP (2023)

Abstract: Financial information is generated and distributed across the world, resulting in a vast amount of domain-specific multilingual data. Multilingual models adapted to the financial domain would ease deployment when an organization needs to work with multiple languages on a regular basis. For the development and evaluation of such models, there is a need for multilingual financial language processing datasets. We describe MultiFin – a publicly available financial dataset consisting of real-world article headlines covering 15 languages across different writing systems and language families. The dataset consists of hierarchical label structure providing two classification tasks: multi-label and multi-class. We develop our annotation schema based on a real-world application and annotate our dataset using both ‘label by native-speaker’ and ‘translate-then-label’ approaches. The evaluation of several popular multilingual models, e.g., mBERT, XLM-R, and mT5, show that although decent accuracy can be achieved in high-resource languages, there is substantial room for improvement in low-resource languages.

## introduction natural language processing technology has substantially improved in recent years due to the generalpurpose transformer model @xcite , large-scale self-supervised training from unlabelled corpora @xcite , and the scaling of both of these to increasingly large datasets and models @xcite . nevertheless, there are still benefits to having domain-specific models @xcite , especially when working with clinical @xcite or financial text @xcite . the domain of financial text is particularly interesting for multilingual nlp, given that it is produced across the world @xcite . the text often includes invoices, transactions, accounting data, tax policies, and stock market information, inter-alia, and there is an emerging effort to create monolingual financial berts (fin-berts) to process financial text @xcite @xcite . however, the handling of financial text by multinational companies is inherently multilingual, therefore, there is is a need for datasets to evaluate how well models can process multilingual financial text. to this end, we introduce the multifin dataset, a publicly available financial dataset consisting of real-world financial article headlines in 15 languages (see examples in we benchmark four large-scale pretrained language models (sentencebert, mbert, xlm-r, and mt5) and find that the benefits of large-scale pretraining also apply to financial text. xlm-r is clearly the best performing model in all of our experiments, however, there is a subsantial gap in performance between high-and low-resource languages in multifin. moreover, a simple lstm initialized with fasttext word embeddings gives surprisingly competitive performance in several experiments. overall, we find the financial domain can benefit from multilingual nlp, and future work should focus on domain adaptive efforts and improving models' capacity to generalize to low-resource languages. • ## existing datasets for financial nlp financial nlp is an emerging area of nlp. researchers and practitioners have a keen interest in processing natural language for different downstream tasks in the financial domain, such as text mining in accounting @xcite , financial transactions (jørgensen and igel, 2021), sentiment analysis @xcite , and text classification @xcite . also, financial economics research shows that news articles and media can be used to forecast firm performance @xcite , predict stock market volatility (glasserman and mamaysky, 2019) and predict market return @xcite . moreover, @xcite show that textual transcripts in combination with audio recordings of company earnings conference calls can be used to predict stock price volatility. there is a large variety of downstream nlp tasks in the financial domain. however, most work within the community is carried out in a monolingual english setting, where the focus is on adapting successful generic monolingual models to the financial domain @xcite @xcite . only a little work on multilingual domain-adapted models has been investigated @xcite . since the finan-cial environment is indeed multilingual, further progression is conditioned on the availability of multilingual resources to develop new methods for multilingual nlp in the financial domain. ## the multifin dataset the multifin dataset is a multilingual corpus, consisting of real-world article headlines covering 15 languages. we annotate the corpus using hierarchical label structure, providing two classification tasks: multi-class and multi-label classification. ## experiments and results we employ popular pre-trained multilingual models foot_0 and test their effectiveness under different experimental setups. for experimentation, we will only focus on the low-level multi-label task, and high-level results are reported in the appendix, ## analysis and discussion our experiments suggest that although decent accuracy can be achieved for high-resource languages, there is substantial room for improvement in achieving better performance on the multilingual financial dataset. in this section, we present a detailed analysis of the results and investigate some of the findings to identify possible modelling improvements and look into the different dimensions of our dataset. ## conclusion we proposed multifin, a dataset for the evaluation of multilingual financial nlp models. the main aim is to advance multilingual nlp in the financial domain so it is better suited for new development and evaluation of domain-specific models. multifin is a diverse dataset with 10,000 examples, covering 15 languages, including different language families and writing systems. we benchmark a collection of standard multilingual language models on multifin and find that although these models often achieve good performance in high-resource languages, there is a substantial gap in performance between high-and lower-resource languages. the per-language analysis uncovered that most of the benchmarked models do not facilitate a good transfer across the evaluated languages, and for specific languages, indicate a strong need for improving the models' capacity 5 artetxe et al. (2020) found that improvements of a translation baseline in a cross-lingual nli task do not stem from overcoming the cross-lingual gap, but from the fact that translation of the training data introduces alterations which improve generalization to a translated test set. it is possible that in our experiments, the performance of the monolingual model generalizing from translated training data to translated test data is impacted by similar mechanisms. to generalize. the multilingual mdapt model presented overall better generalization, particularly to low-resource languages, indicating that focusing on multilingual domain-specific methods is a promising direction for future work in financial nlp. future work includes extending the dataset to include more examples across more languages so better understand the limits of multilingual financial text processing. we are also exploring including the entire document, as opposed to only the headline, but this would depend on high-quality long document processing models @xcite . we hope to motivate and inspire collective work on multilingual nlp in the financial domain.