F in NLP - FNP - LLMF in L egal-2025 Shared Task: Regulations Challenge (2025)

Abstract: Financial large language models (FinLLMs) have been applied to various tasks in business, finance, accounting, and auditing. Complex financial regulations and standards are critical to financial services, which LLMs must comply with. However, FinLLMs’ performance in understanding and interpreting financial regulations has rarely been studied. Therefore, we organize the Regulations Challenge, a shared task at COLING FinNLP-FNP-LLMFinLegal-2025. It encourages the academic community to explore the strengths and limitations of popular LLMs. We create 9 novel tasks and corresponding question sets. In this paper, we provide an overview of these tasks and summarize participants’ approaches and results. We aim to raise awareness of FinLLMs’ professional capability in financial regulations and industry standards.

## introduction the financial industry follows strict regulations and industry standards to ensure market integrity, protect investor interests, and mitigate systemic risk @xcite . large language models (llms) with remarkable capabilities in understanding and generating texts are promising tools to process and interpret financial regulations, with a rapidly growing number of llms available on hugging face hub @xcite . however, financial regulations and industry standards present unique challenges to the professional readiness of financial llms (finllms). the complex regulatory framework and overlapping jurisdictions, such as the fragmented dual federal-state framework in the u.s., make the compliance process challenging @xcite . financial regulation requires processing multimodal data (yanglet and deng, 2024), including, but not limited to, legal texts, financial statements, 1 website: https://coling2025regulations.thefin. ai/home mathematical formulas, tables, figures, and charts. moreover, llms face issues with misinformation and hallucinations, where they generate inaccurate or seemingly plausible but fabricated information @xcite . such hallucinations or misinformation are unacceptable in deployment and can lead to regulatory violations, substantial monetary losses, and erosion of trust between companies and their customers @xcite . to evaluate llms' capabilities in financial regulations, we organize the regulations challenge, a shared task at coling finnlp-fnp-llmfinlegal-2025. it aims to challenge the academic community to explore the strengths and limitations of llms in financial regulations and industry standards. we designed 9 novel tasks to evaluate llms in 5 areas: information retrieval, passing certificates, the common domain model (cdm), the model openness framework (mof), and extensible business reporting language (xbrl) analytics. for each task, we create a question set from diverse documents, such as regulatory filings and official documentation. the remainder of this report is organized as follows. section 2 describes the tasks and question sets. section 3 discusses the participants' methods. section 4 discusses their results. section 5 concludes and recommends future research directions. ## task and dataset in this section, we present our nine novel tasks and the corresponding question sets. ## participants there were 25 teams registered for the regulations challenge, out of which 6 teams submitted their full solutions. we specify three baseline models: llama 3.1-8b (meta ai, 2024a), gpt-4o @xcite , and mistral large 2 @xcite . gpt-4o and mistral large 2 are selected for their strong performance, while llama 3.1-8b is chosen because its model size is manageable for participants. some teams' methods are as follows: ## conclusion and future work in the regulations challenge, we created nine novel tasks and corresponding question sets to assess llms' ability to understand and interpret financial regulations and industry standards, and also llms' understanding of financial products and markets. through it, we encouraged the academic community to identify the strengths and limitations of llms in financial regulations and gain insights into their professional readiness. we will organize follow-up challenges on financial regulations. the question sets and evaluation results will be merged back to the open finllm leaderboard on hugging face @xcite . to better showcase use cases, we will provide demos by leveraging fingpt search agent @xcite .