A Scalable Framework for Legal Text Understanding in Regulatory and Financial Contexts. (2025)

Abstract: This study presents a comprehensive approach to developing a domain-specific large language model (LLM) for regulatory and financial text interpretation. A specialized corpus was constructed through large-scale scraping of financial and regulatory documents across domains such as compliance, licensing, and financial reporting. The data was preprocessed using GPT-4o-mini with prompt engineering to retain critical information and remove noise. We further pre-trained a LLaMA-3.1-8B model on the curated corpus and fine-tuned it using an instruction dataset covering nine tasks from the Coling 2025 Regulations Challenge, including acronym expansion, regulatory question-answering, and XBRL-based financial analytics, employing QLoRA to reduce memory requirements. The model exhibits a slight improvement from baseline answering complex regulatory questions (detailed QA) and expanding acronyms. This study demonstrates the potential of domain-specific LLMs in regulatory text interpretation and lays the groundwork for future research in specialized NLP evaluation methodologies.

## introduction the rapid growth and increasing complexity of regulatory and financial documentation have created a pressing need for tools capable of extracting, analyzing, and responding to nuanced queries with precision and contextual relevance. while large language models (llms) have demonstrated exceptional capabilities in natural language understanding, their general-purpose nature often renders them inadequate for domain-specific applications. addressing this gap, we present a domain-specific llm designed specifically for regulatory and financial texts, equipped to tackle diverse and intricate tasks with slightly enhanced accuracy and contextual awareness. to develop this model, we began by constructing a comprehensive corpus sourced from publicly available financial and regulatory documents including legal statutes, compliance guidelines, and financial reports. recognizing the inherent noisiness of web-sourced data, we implemented a preprocessing pipeline. we first used a subset of our data and trained a tf-idf model, which we used to score documents to ignore very noisy entries. then, using prompt engineering with gpt-4o-mini, we refined the corpus by filtering out irrelevant content while retaining key information critical for downstream tasks. this preprocessing approach allowed us to create our dataset, tailored to the unique demands of regulatory language modeling. the core of our methodology involved finetuning the llama-3.1-8b model, presented in @xcite , leveraging its capabilities as a foundational llm. notably, the model presents a strategic balance between computational efficiency and performance. while many stateof-the-art large language models require extensive computational resources-often demanding highend gpu clusters or cloud computing infrastructure-the llama-3.1-8b model offers a more accessible alternative. with a modest parameter count (for modern llm standards) of 8 billion, the model strikes a balance between computational complexity and inferential capabilities. this design allows for potential local deployment on high-range computational hardware, such as workstations with 32-64 gb of ram and modern consumer grade gpus. however, it is crucial to acknowledge the tradeoffs: while the reduced infrastructural footprint enables broader accessibility, it may inherently limit the model's capacity to match the absolute performance of larger, more computationally intensive models. to optimize computational efficiency and scalability we employed quantized low-rank adap-tation (qlora), a parameter-efficient fine-tuning technique. qlora allowed for substantial memory savings while maintaining model performance @xcite . however, the lack of standardized evaluation benchmarks for regulatory nlp tasks posed a significant challenge, leading us to rely on qualitative analyses and comparisons with the base llama-8b model to assess improvements. these qualitative assessments demonstrated notable gains in task performance, particularly in named entity recognition and question-answering. this paper details the methodologies and challenges encountered in developing this domainspecific regulatory language model. by combining advanced preprocessing techniques with taskspecific fine-tuning strategies, our work highlights the potential of tailored llms in addressing the unique challenges of regulatory text interpretation and establishes a foundation for future research in this critical area. all code, prompts and implementation details can be found in this repository: https://github.com/smartinez1/ coling-2025-regulations-challenge_ uniandes ## related work in the evolving landscape of large language models, their application to specialized domains such as regulatory and financial text analysis has gained significant attention. these domains present unique challenges due to the complexity and specificity of the language used, which often surpasses the capabilities of general-purpose models. tailored approaches are thus essential to effectively address the specific challenges of these domains. li et al. @xcite developed the legalqa dataset, enhancing llm performance through retrieval-augmented generation (rag) with expertcurated question-answer pairs. while this dataset performs well in legal question-answering, it falls short in covering the diverse tasks addressed in our study, such as named entity recognition (ner) and xbrl analytics. the regulations challenge at coling 2025, led by wang et al. @xcite , provides a benchmark to assess the readiness of llms for financial regulations. their framework, which includes nine tasks and corresponding datasets, is a valuable tool for evaluating llm performance in legal and financial contexts. while our tasks dif-fer, their methodology has greatly influenced ours, emphasizing the need for thorough evaluation. mavi et al.'s work @xcite on retrieval techniques for semi-structured domains aligns with our data preprocessing efforts, where we use frequency-based methods to curate highquality datasets. similarly, pipitone's legalbench-rag (pipitone and alami, 2024) supports retrieval techniques, ensuring scalability and adaptability across regulatory contexts. our approach uses tf-idf for document retrieval, aligning with the emphasis on precise retrieval in specialized domains, while differing on the specific tasks and data used. dahan and wu's studies @xcite emphasize the critical need to mitigate hallucination and ensure data reliability, particularly when guiding non-expert users. in our model, these insights are incorporated through taskspecific prompt design, which should enhance the model's practical utility by ensuring reliable and accurate responses. our study addresses gaps in previous research by developing a domain-specific llm that integrates frequentist preprocessing with task-specific finetuning. this method shows promising results in managing cross-domain tasks and complex financial data, providing a robust alternative for tackling the challenges of regulatory and financial text analysis. ## dataset creation the challenge tasks aim to assess the ability of large language models (llms) to generate accurate responses to questions related to regulatory texts, focusing on their performance across the following nine specific areas: ## instruction dataset generation in this section, we present the methodology for creating an instruction dataset specifically designed to optimize large language models (llms) for understanding regulatory and financial texts. this dataset aims to improve model performance in specialized tasks such as abbreviation expansion, question-answering, and named entity recognition. ## training methodology the training process was conducted on an nvidia a40 gpu. the base model employed was the 8billion-parameter llama (llama 3.1). additionally, the associated tokenizer was modified to include a custom padding token, ensuring consistent input formatting throughout the training process. the dataset was randomized and divided into training and validation subsets, with 95% of the batches allocated for training and the remaining 5% for validation. ## fine-tuning the model with instruction data we developed a two-stage fine-tuning approach driven by the differing context window requirements across tasks, with named entity recognition (ner) posing unique computational and contextual challenges. while most instruction-based tasks could be effectively handled within a standard 128-token context, ner requires a much larger context window to capture complex interdependencies and long-range relationships in the text. we designed a two-part fine-tuning strategy to address these contextual differences: ## results in this section, we present the findings of our study on the performance of our domain-specific large language model for regulatory and financial text understanding. we compare our model's performance with baseline models including gpt-4o, llama 3.1 8b, and mistral large 2. the full leaderboard results can be found at https:// coling2025regulations.thefin.ai/winners . ## conclusions and future work our research presents an approach to developing a domain-specific large language model (llm) for regulatory and financial text interpretation, addressing the critical challenges of extracting and analyzing complex regulatory documents. by constructing a data collection and preprocessing pipeline, we successfully created a corpus of 2,286 diverse regulatory documents. the methodology integrated recursive web scraping, tf-idf-based relevance scoring, and text cleaning techniques using gpt-4o-mini, demonstrating a novel approach to building domain-specific training datasets. the two-stage fine-tuning strategy utilizing llama-3.1-8b revealed both the potential and limitations of our domain-specific model. while achieving notable strengths in question-answering tasks, the model also exposed critical areas for future improvement, particularly in named entity recognition and xbrl analytics. these insights not only highlight the complexities of developing specialized language models for regulatory domains but also provide a clear roadmap for future research, emphasizing the need for more sophisticated approaches to capturing the nuanced language of financial and regulatory texts. our study contributes to the field of domainspecific natural language processing by demonstrating the feasibility and challenges of creating targeted large language models. by providing a transparent methodology, we offer researchers and practitioners a valuable framework for developing more accurate and contextually aware language models in specialized domains, ultimately advancing the capability of ai to understand and process complex regulatory information. due to time and resource constraints, we were unable to conduct comprehensive expert validation. inspired by the work of chen et al. @xcite , we propose developing a novel methodology to create discriminative small language models specifically designed for autonomous data quality assessment, in close collaboration with domain ex-perts. drawing from their "honest ai" approach, we aim to develop a collaborative framework where specialized small language models (e.g., bert) are trained with data curated by legal or financial experts. these models will be co-designed to validate data, acknowledge limitations, and provide transparent insights. by integrating expert knowledge throughout the model development process, we can create a scalable and efficient approach to data validation that combines the strengths of ai and human expertise. the proposed system would: