SECQUE : A Benchmark for Evaluating Real-World Financial Analysis Capabilities (2025)

Abstract: We introduce SECQUE, a comprehensive benchmark for evaluating large language models (LLMs) in financial analysis tasks. SECQUE comprises 565 expert-written questions covering SEC filings analysis across four key categories: comparison analysis, ratio calculation, risk assessment, and financial insight generation. To assess model performance, we develop SECQUE-Judge, an evaluation mechanism leveraging multiple LLM-based judges, which demonstrates strong alignment with human evaluations. Additionally, we provide an extensive analysis of various models’ performance on our benchmark. By making SECQUE publicly available (https://huggingface.co/datasets/nogabenyoash/SecQue), we aim to facilitate further research and advancements in financial AI.

## introduction recent advances in large language models (llms) have demonstrated their potential across diverse domains, including law @xcite , medicine @xcite , and finance @xcite . however, as these models are increasingly adopted for specialized applications, the need for domainspecific evaluation has become more pressing. while general-purpose benchmarks assess a wide range of capabilities, they often fail to capture the nuances and challenges inherent in domain-specific tasks @xcite . while domain-specific evaluation is challenging across many fields, the financial domain presents unique challenges in assessing llm capabilities. financial analysts routinely analyze complex datasets, extract meaningful insights from textual and numerical data, and answer high-stakes questions about companies, industries, and market trends. these tasks require models to excel in financial reasoning, numerical computation, and the synthesis of information from lengthy, multiformat documents. yet, many existing benchmarks for financial llms often focus on isolated downstream tasks, such as sentiment analysis or named entity recognition, and do not adequately reflect the breadth of questions analysts face in real-world scenarios @xcite @xcite . to address this gap, we introduce secque, a benchmark specifically designed to evaluate llms on the types of questions financial analysts pose while analyzing sec foot_0 filings. secque includes questions spanning four key categories: comparison and trend analysis, ratio analysis, risk factors, and analyst insights, thus representing essential components of financial analysis. for each question, we present a ground truth answer and variations of the supporting data from the sec filings, representing different textual pre-processing methods. the benchmark consists of 565 questions curated to challenge models' abilities to comprehend, reason, and synthesize information within the context of corporate filings. our benchmark offers several key advantages. first, secque is designed to reflect real-world financial tasks, moving beyond basic text processing to assess reasoning over long unstructured data. second, it emphasizes long-context questions, requiring models to extract relevant information from complex and detailed inputs, such as financial tables with varied structures. third, secque addresses limitations identified in financebench @xcite by introducing cross-company comparisons and high-difficulty questions. additionally, following @xcite , to address this difficulty, we present secque-judge that, following @xcite , leverages multiple llm judges evaluations. we perform a thorough investigation of secque-judge and demonstrate its alignment with human evaluation. using our validated secque-judge, we have performed a thorough analysis of secque. finally, we conduct an ablation study to examine how different configurations, such as prompt choice, affect the results. ## secque benchmark the secque benchmark was developed as a tool to evaluate the performance of large language models (llms) specializing in the financial domain in real-world financial scenarios. our evaluation focuses on key use cases where llms could significantly impact the work of financial professionals in general, and financial analysts in particular. financial analysts rely on diverse documents in their work, and we focused on the primary publicly available financial reports 3 : 10-k and 10-q sec filings. a 10-k is a company's annual financial report filed with the sec, while a 10-q is a quarterly update on its financial performance. these documents include textual and tabular data about publicly traded companies, covering sections such as risk factors, income statements, balance sheets, and cash flow statements. ## evaluating judge performance manual evaluation of the entire benchmark is impractical, therefore, we have implemented secque-judge, an automated comparison for various model outputs with the secque ground truth answers (denoted as ⟨ỹ, y⟩, respectively). in this section we describe our secque-judge implementation and verify alignment with human evaluation. ## related work recent advances in large language models (llms) have spurred considerable research in domainspecific benchmarks and evaluation frameworks, particularly in finance. in this section, we briefly review work on financial benchmarks and the use of llms for evaluation. financial benchmarks and datasets a variety of benchmarks have been introduced to assess llm performance on financial tasks. comprehensive evaluation frameworks such as finben @xcite , pixiu @xcite , and bbt-fin @xcite aggregate diverse tasks to measure general financial skills. other datasets target specialized skills: fineval @xcite focuses on textbook-based financial knowledge, superclue-fin @xcite decomposes real-world financial tasks into fine-grained subtasks, and findabench @xcite emphasizes financial analysis and reasoning. in parallel, several financial qa datasets have been proposed. early efforts include fiqa @xcite for sentiment analysis and opinionated qa, while finqa @xcite and its conversational extension convfinqa @xcite offer more realistic, multi-turn interactions. datasets such as tat-qa @xcite incorporate numerical reasoning over tabular and textual data from financial reports. despite these efforts, many of the existing benchmarks do not fully capture the retrieval, analysis and reasoning challenges inherent to day-to-day financial analysis @xcite , which are necessary for real-world financial work. evaluation paradigms: llm-as-a-judge traditional benchmark evaluation has evolved with the emergence of llms. beyond standard multiplechoice or completion tasks where easy evaluation is possible, recent approaches leverage llms (notably gpt-4 @xcite ) as automated judges for assessing generation quality. for example, li et al. @xcite and zheng et al. @xcite have demonstrated the effectiveness of using llms to score answers in openended question setups, while @xcite employed majority voting from multiple judges. @xcite and others have conducted extensive studies around the alignment of llm evaluators with human annotators, yet a single optimal setup has not been identified, prompting the need for further case-by-case optimization. ## conclusions we have presented secque, a comprehensive benchmark for evaluating llms in financial analysis tasks. our results demonstrate that while leading models show promising capabilities in financial analysis, significant challenges remain, particularly in complex reasoning tasks and analyst insights generation. the benchmark reveals important differences in model performance across question types and highlights the critical role of configurations in evaluation results. these findings provide valuable guidance for future development of financial llms and evaluation frameworks.