Financial Language Model Evaluation ( FL a ME ) (2025)

Abstract: Language Models (LMs) have demonstrated impressive capabilities with core Natural Language Processing (NLP) tasks. The effectiveness of LMs for highly specialized knowledge-intensive tasks in finance remains difficult to assess due to major gaps in the methodologies of existing evaluation frameworks, which have caused an erroneous belief in a far lower bound of LMs’ performance on common Finance NLP (FinNLP) tasks. To demonstrate the potential of LMs for these FinNLP tasks, we present the first holistic benchmarking suite for Financial Language Model Evaluation (FLaME). We are the first research paper to comprehensively study LMs against ‘reasoning-reinforced’ LMs, with an empirical study of 23 foundation LMs over 20 core NLP tasks in finance. We open-source our framework software along with all data and results.

## introduction benchmarks and datasets are the foundation for artificial intelligence (ai) research. how the research community collectively defines 'success' directly shapes researchers' priorities and goals @xcite . benchmarks enable the wider research community to understand and track progress in ai development @xcite . recent developments enabling the general commercial availability of foundation language models (lms) @xcite (e.g., @xcite , claude (anthropic), gemini @xcite , etc. have fueled widespread interest in tracking their progress @xcite . the widespread availability of lms has spurred research into ai capabilities for highly specialized and knowledge-intensive domains, such as figure 1 : technical overview: flame uses a unified inference hub, providing a single, model-agnostic api across three deployment modes: (i) proprietary cloud apis (e.g., claude 4, gemini 2.5 pro), (ii) cloud-hosted open-weight models (e.g., olmo 2, qwen 2.5) served by either cloud providers (togetherai, huggingface), and (iii) fully local inference backends (e.g., vllm, ollama). this modular software design abstracts deployment complexity, enabling rapid experimentation and comprehensive benchmarking across a diverse range of language models, significantly simplifying evaluation workflows to promote replication and transparency in finnlp research. ## methodology we present our methodology for holistic financial language model evaluation. flame is the first holistic benchmark suite for core nlp tasks in finance. this methodology enables researchers to evaluate the fundamental abilities of foundation models systematically. ## conclusion we present flame, a robust evaluation framework and open-source software package for conducting holistic evaluation of language models for finance. flame provides standardized multimetric evaluation for finance-specific datasets and evaluation methods. this framework provides a valuable foundation for building, testing, and advancing high-performance nlp models tailored to the unique challenges of financial language understanding. we believe that the adoption of a collaborative evaluation framework like flame will be used by researchers to easily conduct holistic evaluations of any generally available lm for core finnlp tasks. our evaluation underscores the complex landscape of finnlp. our key insights are as follows: