A udit W en: An Open-Source Large Language Model for Audit (2024)

Abstract: “Intelligent auditing represents a crucial advancement in modern audit practices, enhancing boththe quality and efficiency of audits within the realm of artificial intelligence. With the rise oflarge language model (LLM), there is enormous potential for intelligent models to contribute toaudit domain. However, general LLMs applied in audit domain face the challenges of lackingspecialized knowledge and the presence of data biases. To overcome these challenges, this studyintroduces AuditWen, an open-source audit LLM by fine-tuning Qwen with constructing instruc-tion data from audit domain. We first outline the application scenarios for LLMs in the audit andextract requirements that shape the development of LLMs tailored for audit purposes. We thenpropose an audit LLM, called AuditWen, by fine-tuning Qwen with constructing 30k instructiondataset from 15 audit tasks and 3 layers. In evaluation stage, we proposed a benchmark with 5kinstructions that covers a set of critical audit tasks derived from the application scenarios. Withthe benchmark, we compare AuditWen with other existing LLMs from information extraction,question answering and document generation. The experimental results demonstrate superiorperformance of AuditWen both in question understanding and answer generation, making it animmediately valuable tool for audit.Keyword AuditWen, LLM, instruction dataset, fine-tuning, benchmarkIntroduction”

## introduction audit is an independent economic supervision activity conducted by governmental agencies or a special organ in accordance with the law to conduct pre-and-post-event reviews of major projects and financial revenues and expenditures of financial institutions or enterprises. in recent years, with the development of big data, the data foundation and audit methodology of national audit are also undergoing changes @xcite . the audit methodology is transitioning from big data audit to intelligent audit @xcite , aiming at recommending or selecting the optimal strategy for audit decision-making through the extensive integration of machine learning, deep learning, and other information technologies. with the emergence of chatgpt 0 , large language models (llms) @xcite have attracted much attention from researchers. its smooth natural dialogue and document generation capabilities have rendered it widely used in various fields, such as in financial @xcite , medical @xcite , legal @xcite and so on. a large language model is a deep learning model with a very high number of parameters and computational power that can automatically learn the syntax, semantics, and context of input natural language and can generate text of corresponding to it. as a powerful artificial intelligence technology, large language model possess a strong capacity for understanding and generating natural language and can provide innovative solutions for the audit. however, the current general llms commonly encounter issues like a deficiency in domain-specific knowledge and the existence of data bias. similar to their application in other domain-oriented tasks, llms face challenges when directly applied to auditing, including difficulties in understanding input issues clearly and providing accurate responses to fact-based tasks, a phenomenon known as hallucination @xcite . moreover, auditors argue that intelligent auditing with llms should prioritize collaboration between individuals and the model to jointly accomplish complex audit tasks @xcite . this demand necessitates that llms not only comprehend concepts, entities, and knowledge within the audit domain, but also master the fundamental processes of audit work to assist auditors in achieving high-quality results. llms excel in context memory, knowledge retrieval, and text generation, thereby offering unique advantages in this regard. therefore, it is essential to train a llm specifically for the audit domain, aligning with the actual requirements and raw data of auditing practices. by refining and tailoring llm tasks to align with auditing requirements, the audit-focused llm should grasp the terminology, concepts, and regulations of auditing, ultimately delivering more precise and dependable results, especially for the complicated audit tasks. guided by the practical applications of national audit, this study aims to identify potential uses of llm in the audit domain, collect high-quality audit-relevant raw texts and further construct an instruction dataset to build a large language model tailored for audit by fine-tuning a state-of-the-art llm. this model is referred to as auditwen. the contributions of this study are as follows: currently, there is no established benchmark for evaluating llms in the field of audit. according to the audit service requirements, this study designs 15 different llm tasks across 3 layers, constructs the corresponding instruction datasets, and release multi-dimensional evaluation results for both existing mainstream llms and our fine-tuned audit-specific llm, auditwen. ## ait: audit instruction dataset and tuning in this section, we initially outline the tasks of audit llm based on the application scenarios of audit. then we collect source data and design relevant instruction dataset and evaluation benchmark for audit llm. at last, we build auditwen by fine-tuning qwen @xcite with ait. ## conclusion and discussion in this study, we presented auditwen, the first audit-oriented open-source large language model. along with the model, we also release the fine-tune model auditwen and the evaluation benchmark dataset. drawing from the discussion on application scenarios of llm in audit, we have identified various audit tasks. subsequently, we gather and construct a large-scale audit instruction dataset to fine-tune a domainspecific large language model tailored for audit tasks. the extensive evaluation results on the proposed benchmark dataset demonstrated the effectiveness of the auditwen. nevertheless, while acknowledging the positive contribution of this study, we also recognize the following limitations. resource constraints. due to time constraints, the scale of dataset for fine-tuning auditwen is limited, which may not support for fine-tuning model with larger scales. model and training constrains. we only presented the auditwen models with 7b parameters. due to computational and resource constraints, auditwen models with 14b or 30b have not been released so far. for the further work, more relevant source texts about audit cases and statute will be collected and more elaborate tasks such as audit-issue phrase extraction from clause of statute will be constructed. based on these dataset and tasks, we devote to train a larger-scale of audit-oriented llm.