F in T ral: A Family of GPT -4 Level Multimodal Financial Large Language Models (2024)

Abstract: We introduce FinTral, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We enhance FinTral with domain-specific pretraining, instruction fine-tuning, and RLAIF training by exploiting a large collection of textual and visual datasets we curate for this work. We also introduce an extensive benchmark featuring nine tasks and 25 datasets for evaluation, including hallucinations in the financial domain. Our FinTral model trained with direct preference optimization employing advanced Tools and Retrieval methods, dubbed FinTral-DPO-T&R, demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine tasks, marking a significant advancement in AI-driven financial technology. We also demonstrate that FinTral has the potential to excel in real-time analysis and decision-making in diverse financial contexts.

## introduction natural language processing (nlp) plays a key role in financial document analysis, interpretation, and utilization. in recent years, a wide range of applications incorporating advances in nlp have emerged. these include sentiment analysis of financial news, event extraction from financial documents, and the generation and summarization of financial reports @xcite @xcite . these developments have uncovered the potential for unstructured data for datadriven financial decision-making and the transformation of financial documents into actionable insights and market intelligence. applying nlp in finance, however, is challenging because financial documents often include dense numerical information and domain-specific jargon requiring advanced numerical processing and reasoning capabilities @xcite . this means that financial nlp models need extensive domain knowledge before they can capture the nuanced implications of accounting and financial measures, economic indicators, and market trends. this is also compounded by the rapid pace of financial markets, where real-time analysis is crucial but challenging to achieve @xcite . similar to other domains, large language models (llms) are starting to disrupt financial document understanding @xcite but can also suffer from the same issues as transitional approaches. llms are also prone to hallucination, reducing their usability in financial decision-making @xcite . financial documents can also involve various types of visual content, which require models with multimodal abilities. to meet these challenges, we introduce a groundbreaking llm specialising in the financial domain. our model, dubbed fintral, is designed to overcome hurdles of the financial domain through a multimodal approach that integrates textual, numerical, tabular, and visual data processing for comprehensive document understanding. we train our model off mistral-7b @xcite on a sizeable domain-specific dataset and instruction-tune it for the financial domain using extensive instruction data. we then carefully align it with gpt-4 generated responses leveraging the recently introduced direct policy optimization (dpo) method @xcite . in order to evaluate fintral, we introduce an extensive benchmark of eight different tasks based on 25 different datasets. our model outperforms all other models of comparable size and, in spite of its much smaller size, performs on par with gpt-4. to summarize, we offer the following contributions: (1) we introduce fintral a cutting-edge multimodal llm specialized in financial data, and finset, an extensive financial llm training and evaluation benchmark. finset is the largest financial evaluation benchmark and the only one that measures model hallucinations, encompassing nine tasks across 25 datasets. (2) fintrals further instruction-finetuned and carefully aligned using the dpo objective, using ai feedback data, resulting in fintraldpo. (3) we have also endowed fintral with vision capabilities, extending it to fintralvl, which employs the clip @xcite vision encoder. for enhanced performance, we developed a version that utilizes tools and retrieval, fintraldpo-t&r. (4) fintraldpo demonstrates exceptional zero-shot capabilities, outperforming chatgpt (openai, 2023a) in all tasks. moreover, our best model, fintraldpo-t&r, surpasses @xcite in five of eight text-based tasks. the rest of this paper is organized as follows: in section 2, we review related work with a particular emphasis on financial llms, their applications and challenges. section 3 outlines how we built our benchmark dataset: finset. we present our approach to model pretraining, instruction tuning, and prompting strategies, and subsequently introduce fintral models in section 4. in section 5, we present our experiments and comprehensively analyse our models. we discuss our results in section 6 and conclude in section 7. finally, we discuss limitations in section 8 and provide an ethics statement in section 9. financial llms advancements in financial models began with finbert @xcite . recently, models like bloomberggpt @xcite , pixiu @xcite , instruct-fingpt @xcite , and gpt-finre (rajpoot and parikh, 2023) are notable contributions. other innovations include introduction of multimodal capabilities (finvis-gpt @xcite ), enhancement of investment strategies (gpt-investar @xcite , investlm @xcite ), and efforts to address challenges such as economic sentiment analysis and hallucination in information extraction @xcite . finlmeval @xcite and disc-finllm @xcite focus on evaluation and model performance in monetary scenarios. other work, such as chu et al. (2023), emphasizes sophisticated data preprocessing for better handling of financial tasks. appendix a provides a further discussion of the nlp and llms literature in finance. pretraining dataset c4 press news us sec fillings reddit yahoo finance finnhub instruction tuning text instruction data multimodal instruction data financial ai feedback data downstream tasks sentiment analysis number understanding chart understanding firm disclosures financial hallucination credit scoring ner text summarisation stock prediction ## finset we develop comprehensive and diverse datasets to build fintral. we first describe our raw datasets rich in domain-specific tokens, setting a solid foundation for model training, then our instruction finetuning and ai-driven feedback datasets. subsequently, we present a multi-modal financial dataset to facilitate a nuanced approach to data interpretation. finally, we introduce an extensive set of evaluation benchmark datasets tailored to test the model's performance across diverse financial tasks. ## fintral we use mistral-7b-v0.1 @xcite as our base model for further development, due to its strong performance and employment of a bpe tokenizer that segments numbers into single digits, which is suitable for numerical tasks. ## experiments we conducted multiple experiments to illustrate the efficacy of the methods described in section 4. we evaluated our model on the downstream tasks described in section 3.5. symbols in the following tables indicate the types of models: ♣, ♠, ♢, ♡, ⋆ and, ■ represent the pre-trained model, the finetuned model, the instruction fine-tuned model, the rl-tuned models, tools, and retrieval, respectively. we then performed a hallucination index accuracy check to assess how well our model mitigates one of the biggest challenges for llms. we introduce three versions of our model. firstly, fintral-inst is our instruction-fine-tuned model obtained by fine-tuning our pre-trained model. note that we do not assess the performance of the pre-trained model as it serves as an intermediate step to the instruction fine-tuning model. secondly, we introduce fintral-dpo, which has been further trained based on fintral-inst utilizing reinforcement learning using ai feedback with the ddpo objective. then, we introduce our fintral-dpo-t&r, which combines the fintral-dpo with tools and retrieval. we also compare performance of our models to nine other baselines llms. these are llama-2 (touvron et al., 2023), mistral @xcite , three versions of finma @xcite , vicuna @xcite , chatgpt (openai, 2023a), gpt-4 (openai, 2023c). ## conclusion we presented fintral an advanced multimodal financial language model with remarkable capabilities. key advancements include integrating textual, numerical, and visual data, a training pipeline with various finetuning capabilities, and employment of tools and retrieval mechanisms. the model effectively addresses challenges like financial hallucination, evidenced by high performance in various financial tasks compared to baseline models. the achievements of fintral hold a great potential for financial models of a moderate size (e.g., 7b).