Audit- FT at the Regulations Challenge Task: An Open-Source Large Language Model for Audit (2025)

Abstract: Intelligent auditing represents a crucial advancement in modern audit practices, enhancing both the quality and efficiency of audits within the realm of artificial intelligence. With the rise of large language model (LLM), there is enormous potential for intelligent models to contribute to audit domain. However, general LLMs applied in audit domain face the challenges of lacking specialized knowledge and the presence of data biases. To overcome these challenges, this study introduces AuditWen, an open-source audit LLM by fine-tuning Qwen with constructing instruction data from audit domain. We first outline the application scenarios for LLMs in the audit and extract requirements that shape the development of LLMs tailored for audit purposes. We then propose an audit LLM, called AuditWen, by fine-tuning Qwen with constructing 30k instruction dataset from 15 audit tasks and 3 layers. In evaluation stage, we proposed a benchmark with 5k instructions that covers a set of critical audit tasks derived from the application scenarios. With the benchmark, we compare AuditWen with other existing LLMs from information extraction, question answering and document generation. The experimental results demonstrate superior performance of AuditWen both in question understanding and answer generation, making it an immediately valuable tool for audit.

## introduction audit is an independent economic supervision activity conducted by governmental agencies or a special organ in accordance with the law to conduct pre-and-post-event reviews of major projects and financial revenues and expenditures of financial institutions or enterprises. in recent years, with the development of big data, the data foundation and audit methodology of national audit are also undergoing changes @xcite . the audit methodology is transitioning from big data audit to intelligent audit @xcite , aiming at recommending or selecting the optimal strategy for audit decision-making through the extensive integration of machine learning, deep learning, and other information technologies. with the emergence of chatgpt foot_0 , large language models (llms) @xcite have attracted much attention from researchers. its smooth natural dialogue and document generation capabilities have rendered it widely used in various fields, such as in financial @xcite , medical @xcite , legal @xcite and so on. a large language model is a deep learning model with a very high number of parameters and computational power that can automatically learn the syntax, semantics, and context of input natural language and can generate text of corresponding to it. as a powerful artificial intelligence technology, large language model possess a strong capacity for understanding and generating natural language and can provide innovative solutions for the audit. however, the current general llms commonly encounter issues like a deficiency in domain-specific knowledge and the existence of data bias. similar to their application in other domain-oriented tasks, llms face challenges when directly applied to auditing, including difficulties in understanding input issues clearly and providing accurate responses to fact-based tasks, a phenomenon known as hallucination @xcite . moreover, auditors argue that intelligent auditing with llms should prioritize collaboration between individuals and the model to jointly accomplish complex audit tasks @xcite . this demand necessitates that llms not only comprehend concepts, entities, and knowledge within the audit domain, but also master the fundamental processes of audit work to assist auditors in achieving high-quality results. llms excel in context memory, knowledge retrieval, and text generation, thereby offering unique advantages in this regard. therefore, it is essential to train a llm specifically for the audit domain, aligning with the actual requirements and raw data of auditing practices. by refining and tailoring llm tasks to align with auditing requirements, the audit-focused llm should grasp the terminology, concepts, and regulations of auditing, ultimately delivering more precise and dependable results, especially for the complicated audit tasks. guided by the practical applications of national audit, this study aims to identify potential uses of llm in the audit domain, collect high-quality audit-relevant raw texts and further construct an instruction dataset to build a large language model tailored for audit by fine-tuning a state-of-the-art llm. this model is referred to as auditwen. the contributions of this study are as follows: ## ait: audit instruction dataset and tuning in this section, we initially outline the tasks of audit llm based on the application scenarios of audit. then we collect source data and design relevant instruction dataset and evaluation benchmark for audit llm. at last, we build auditwen by finetuning qwen @xcite with ait. ## conclusion and discussion in this study, we presented auditwen, the first audit-oriented open-source large language model. along with the model, we also release the fine-tune model auditwen and the evaluation benchmark dataset. drawing from the discussion on application scenarios of llm in audit, we have identified various audit tasks. subsequently, we gather and construct a large-scale audit instruction dataset to fine-tune a domain-specific large language model tailored for audit tasks. the extensive evaluation results on the proposed benchmark dataset demonstrated the effectiveness of the auditwen. nevertheless, while acknowledging the positive contribution of this study, we also recognize the following limitations. resource constraints. due to time constraints, the scale of dataset for finetuning auditwen is limited, which may not support for fine-tuning model with larger scales. model and training constrains. we only presented the auditwen models with 7b parameters. due to computational and resource constraints, auditwen models with 14b or 30b have not been released so far. for the further work, more relevant source texts about audit cases and statute will be collected and more elaborate tasks such as audit-issue phrase extraction from clause of statute will be constructed. based on these dataset and tasks, we devote to train a larger-scale of audit-oriented llm. level task name sub-task name #train/val./test annotation sentence level audit ner 4091/1022/1424 human annotation relation classification 817/232/117 human annotation phrase classification audit entity cla. (aec) -/-/1578 human annotation audit-issue phrase cla. (aic) 1210/344/166 legal name cla. (lnc) 1463/418/218 paragraph level definition of audit entity 1756/500/19 extract from raw text audit-legal relevant question 15774/112/505 generated by gpt-4 audit issue audit issue summary (ais) 253/71/36 extract from raw text audit issue describe (aid) 202/56/29 legal recommendation (lr) 1567/445/224 other-audit relevant question audit procedures and material (apm) 671/190/96 extract from raw text audit type and objectives (ato) 609/171/87 other question (oq) 903/257/129 documents level audit case analysis 544/151/77 extract from raw text audit doc. generation 48/11/6 total 29908/3980/4941 relation between an audit item and the corresponding audit unit legal name content of item qa generated by gpt-4