I ntelli C hain Stars at the Regulations Challenge Task: A Large Language Model for Financial Regulation (2025)

Abstract: We present our approach to the COLING-2025 Regulations Challenge, which evaluates large language models (LLMs) on nine regulatory tasks, such as abbreviation recognition and financial data extraction. To address challenges like domain-specific terminologies and dynamic regulatory contexts, we developed a robust data construction pipeline, integrating proprietary Chinese regulatory data, Fin-GPT datasets, and financial Q&A data. The pipeline applied, but was not limited to, language filtering, semantic screening, and deduplication, resulting in a 30,000-example dataset combining financial regulations and general financial data. Using this dataset, we fine-tuned Llama 3.2-3B-Instruct to create Reg-LLaMA, a specialized model that outperformed baselines on the Regulations Challenge and PIXIU datasets. These results demonstrate the effectiveness of domain-specific data construction in advancing LLMs for regulatory tasks, paving the way for reliable and interpretable AI in regulated industries.

## introduction the coling-2025 regulations challenge @xcite is a benchmark designed to evaluate the capabilities of large language models (llms) in processing and responding to regulatory texts. the competition consists of 9 distinct tasks, ranging from abbreviation recognition to advanced financial data extraction and licensing requirements under specific frameworks. each task is structured to assess an llm's ability to interpret, analyze, and generate precise outputs based on complex regulatory information. the tasks are designed with standardized templates that ensure consistency in input and output formats, reflecting real-world regulatory use cases. despite their immense potential, existing llms face significant challenges in the regulatory domain, such as: to address these challenges, we developed a comprehensive data construction pipeline to curate a high-quality dataset tailored to financial regulations. this pipeline integrates key steps such as language filtering, regular expression matching, semantic screening using financial domain embeddings, and optimization of data quality through perplexity-based filtering @xcite and deduplication @xcite . additionally, privacy-sensitive content was removed to ensure compliance with security standards. these processes allowed us to construct a dataset of 30,000 examples, balancing domain-specific regulatory data and general financial datasets to enhance model robustness and task alignment. through this pipeline, we constructed a highquality instruction dataset comprising 30,000 examples, including 10,000 financial regulation datasets and 20,000 general finance datasets, as detailed in our experimental results validate the effectiveness of this approach. on three distinct frameworks, reg-llama outperformed peer models in tasks requiring nuanced understanding of financial regulations. these results demonstrate its superior capability in handling complex, domain-specific queries. in summary, our contributions include: by tackling the core difficulties of regulatory text comprehension, this work paves the way for more reliable and interpretable ai systems in regulated industries. ## related work large language models in financial regulation. large language models (llms) such as gpt-4 @xcite , llama-3.2 @xcite , and mistral-large-2 @xcite have demonstrated exceptional capabilities in various natural language processing tasks, including question answering, summary generation, and text creation @xcite . these models are increasingly utilized for financial regulatory tasks, but they face challenges in understanding complex domain-specific terminologies, adapting to rapidly changing regulatory frameworks, and ensuring the interpretability and ethical compliance of outputs @xcite . models like finbert @xcite and fingpt @xcite attempt to address these issues by fine-tuning on specialized financial datasets, showcasing improved performance and robustness in handling regulatory tasks @xcite . additionally, initiatives such as the coling-2025 regulations challenge emphasize the importance of assessing llms' capabilities in regulatory scenarios, providing valuable benchmarks that identify gaps and drive innovation. datasets and competitions in financial regulation. the complexity and dynamic nature of the financial regulatory domain necessitate highquality and up-to-date datasets. research has highlighted the need for integrating knowledge retrieval mechanisms and domain-specific fine-tuning to enhance model performance in regulatory tasks. competitions like the coling-2025 regulations challenge play a pivotal role by providing benchmark datasets and evaluation metrics that promote advancements in compliance automation and question answering. these benchmarks not only improve model evaluation but also reduce the reliance on costly manual annotations by encouraging automated solutions @xcite . for instance, finqa @xcite introduces a high-quality dataset crafted by financial experts, which emphasizes the importance of integrating complex numerical reasoning and domain-specific knowledge to enhance the performance of regulatory systems. ## reg-llama: datasets this section details the reg-llama instruction dataset, including the the data collection and a complete pipeline for the data reconstruction. ## reg-llama: evaluation in this study, we employed bertscore, a scoring system based on bert embeddings, to evaluate the performance of text generation models. by comparing the cosine similarity of embedding vectors between generated texts and reference texts, bertscore effectively measures the semantic similarity of the texts. we configured the bert-baseuncased model and enabled baseline calibration to ensure comparability of the scores. the entire evaluation process included data loading, performance calculation, and result aggregation. ultimately, by analyzing the distribution and average values of bertscore, we comprehensively assessed the adaptability and generative capabilities of the model in different contexts. these detailed evaluation results will provide solid data support for further comparisons and analyses of the model, and will be elaborately presented in the results section of the research paper. to further investigate the adaptability and generative capabilities of the model, we conducted evaluations under three specific datasets and frameworks: e1: validation, e2: regulation, and e3: financial. these frameworks represent diverse application scenarios and are carefully selected to challenge the model's performance across varying contexts. to enhance evaluation efficiency, we randomly sampled 50 entries from each task in e2 and e3 datasets (using the full dataset when fewer than 50 entries were available) as the basis of our evaluation framework. ## conclusion in this work for the coling-2025 regulations challenge, we focused on enhancing the ability of large language models (llms) to handle the complexities of financial regulations. by leveraging llama 3.2-3b-instruct as the base model, we balanced performance and efficiency through a robust data construction pipeline. this pipeline integrated translated proprietary chinese regulatory datasets, fin-gpt public datasets, and internal financial q&a data, resulting in a high-quality, domain-specific dataset. fine-tuning with lora further optimized the model for interpretability and accuracy in regulatory tasks. our results demonstrate that targeted data augmentation and domain-specific optimization significantly improve llm performance in understanding and applying financial regulations. by addressing challenges such as regulatory complexity, evolving standards, and specialized terminology, this work establishes a foundation for advancing llms in regulated industries. these findings contribute to the development of more reliable and interpretable ai-driven solutions for regulatory compliance and financial decision-making. input: expand the following acronym into its full form: eea. answer: