A veni B ench: Accessible and Versatile Evaluation of Finance Intelligence (2025)

Abstract: Over the last few years, there has been great interest in applying large language models (LLMs) to problems in the finance industry, and the field needs a robust LLM benchmark to support this work. Current financial LLM benchmarks contain simple tasks which are not representative of real use cases and have test sets with licences that do not allow commercial use. In response, we release AveniBench, a permissively licensed benchmark that tests a group of six key finance-related skills: tabular reasoning, numerical reasoning, question answering, long context modelling, summarisation and dialogue. We refactor the test sets to ensure that metrics are comparable, providing a unified framework. Furthermore, AveniBench introduces two task difficulty modes, easy and hard, enabling scalable evaluation based on real-world deployment needs. We use our benchmark to evaluate a diverse set of 20 widely used LLMs, from small open-weight models to proprietary systems like GPT-4. This evaluation initiates our public leaderboard, providing valuable insights for future academic research and commercial development.

## introduction large language models (llms) have the potential to automate and enhance labour-intensive processes across a wide range of industries. finance, as a service industry, is a key sector where llms can have a significant impact, due to its large user base (e.g. commercial banking), opportunities for profitability (e.g. investment decisions), and stringent regulatory requirements (e.g. privacy and fairness). due to the complicated nature of many financial tasks, and the high risks associated with making errors, llms developed for the finance domain must be rigorously evaluated prior to deployment. to support this, a number of benchmarks have been proposed, including finben @xcite , flue 1 https://huggingface.co/aveni-ai figure 1 : overview of current capabilities of llms on avenibench. we pick a representative language model for each group/type. see more fine-grained analysis in we find that whilst many existing benchmarks provide good coverage of financial natural language processing (finnlp) tasks, they are limited in their usefulness for evaluating real-world commercial llm systems. specifically, these benchmarks 1) typically adopt a wide range of multiple pre-existing nlp and machine learning datasets with little thought as to their suitability for llms (e.g. named entity recognition or sentiment analysis); 2) provide limited insight into the difficulty of tasks or examples; 3) have inconsistent score ranges across diverse test sets; and 4) often include data under restrictive licences making them unfit for commercial purposes, which undermines their value as financial llms are going to be heavily used by industry @xcite . in this paper, we directly address each of these limitations by re-examining existing financial test sets, making appropriate modifications, and filtering out those with a restrictive licence. our contributions are as follows: finben @xcite is the most extensive among the existing benchmarks in the finnlp domain. it contains an impressive number of 36 tasks; however, we note that (1) not all of them are suited or formatted for llms and (2) the majority of the tasks are released with non-permissive licences. moreover, due to the extensive number of tasks covered, the datasets have been adapted for llms but not revisited on an individual task basis. other comprehensive finance benchmarks include flue @xcite which contains classification, information extraction, and question answering in the finance domain, as well as bizbench @xcite which includes program synthesis to test reasoning in business and finance scenarios. some other efforts target more specialised capabilities. finbench @xcite focuses on financial risks: credit card default, loan default, credit card fraud, and customer churntasks which involve processing large amounts of numerical data but little text data, and we argue are not well suited to llms. insightbench @xcite evaluates llm agents' data analytics in various business use cases. ucfe @xcite is a multi-turn finance dialogue benchmark covering 17 task types, which is tailored to four distinct user groups: analysts, financial professionals, regulatory professionals, and the general public. there is also a range of emerging datasets that focus on tabular data and mathematical reasoning -tasks that we also include in our benchmark. in particular, we highlight financemath @xcite a knowledge-intensive financial math reasoning qa dataset, and tablebench @xcite a tabular qa dataset, for which financial reports make up a third of the data. ## leaderboard we present the evaluation results on avenibench in we benchmark 18 open-weight base llms and include gpt-4o and gpt-4o-mini for reference. gpt models are instruction-tuned, so we require a direct answer via a system prompt. for a detailed list of evaluated models, see ## conclusion and future work in summary, we scrutinised existing finnlp test sets, modified and adapted data, tasks, and metrics, and finally presented a permissive avenibench. to ensure that it continues to be useful to the community, we aim to regularly review, adjust (if necessary), and incorporate new tests as they become available. we plan to ingest avenibench into lm-eval-harness to facilitate public contributions that could extend the leaderboard to support missing multilingual and multi-modal evaluations.