Evaluating Financial Literacy of Large Language Models through Domain Specific Languages for Plain Text Accounting (2025)

Abstract: Large language models (LLMs) have proven highly effective for a wide range of tasks, including code generation. Recently, advancements in their capabilities have shown promise in areas like mathematical reasoning, chain-of-thought processes and self-reflection. However, their effectiveness in domains requiring nuanced understanding of financial contexts, such as accounting, remains unclear. In this study, we evaluate how well LLMs perform in generating code for domain-specific languages (DSLs) in accounting, using Beancount as a case study. We create a set of tasks based on common financial ratios, to evaluate the numeracy and financial literacy of LLMs. Our findings reveal that while LLMs are state-of-the art in generative tasks, they struggle severely with accounting, often producing inaccurate calculations and misinterpreting financial scenarios. We characterize these shortcomings through a comprehensive evaluation, shedding light on the limitations of LLMs in understanding and handling money-related tasks.

## introduction in recent years, natural language processing methods and transformer models have seen significant improvements in text, language and coding related tasks. especially the release of the pre-trained large language model gpt-3 @xcite and its derivative chatgpt @xcite to the general public have generated considerable public interest. large language models (llms) have the ability to understand and generate text in a wide spectrum of disciplines and tasks, and are also able to generate code. these properties are leveraged across several industries to automate e.g., customer support and content creation. although these models have had a significant impact, they are not sufficiently studied in the accounting practice, despite their potential to enable process automation. accounting in digital business practice. in contemporary business, enterprise resource planning (erp) systems are a commonplace phenomenon, providing the foundation for a multitude of organisational functions and decision-making processes. erp software typically supports a plethora of business dimensions, including human resource management or supply chain management among others. these systems provide a centralized platform for managing operations, offering features like analytics and automation to improve production and decision-making. accounting plays a pivotal role in the functioning of erps, serving as a fundamental pillar upon which various business segments are constructed. it is responsible for ensuring the accuracy of financial data and the monitoring of budgetary allocations throughout the whole economic activity of a company. however, erp systems can be rigid and complex, requiring human training and often leading to a bottleneck in user interaction, hindering efficiency and accessibility. in this work we aim to evaluate whether open-weight llms can accurately and efficiently perform accounting tasks using plain text accounting domain specific languages (dsls). we investigate their ability to understand financial ratios, by generating accounting scenarios that affect them, e.g., selling a property to increase the current ratio (cur). llms for plain text accounting. we find in the dsls of plain text accounting (pta) the ideal target for llms to interface with financial transactional data. pta is an accounting paradigm to record transactions in a human readable format with dsls like ledger @xcite , hledger @xcite or beancount @xcite . these languages are strictly compiled and incorporate double-entry accounting principles that can partially categorize the error classes of transactions generated by llms. we create two tasks for llms to generate scenarios motivated by the semantics of common financial ratios (e.g., current ratio). these quantities are generally used by financial practitioners to assess the economic state of a company and are a good proxy for financial literacy. furthermore, with these scenarios, we also explore the capabilities of llms to generate corresponding transactions using a dsl, which we evaluate with a compiler. we subsample and thoroughly examine the results of the generation with the help of experts in the field. in our evaluation llms generally show significant problems regarding financial literacy and numeracy by extension. we characterize these essential deficiencies through six financial scenario error classes, and six transaction error classes. our contributions can be summarized as follows: we provide all data to replicate our experiments including our prompts, data and methodology. foot_0 ## related work language models for code generation. in addition to common language-related tasks, llms are also applied in the field of code generation. there are three types of transformer models: encoder-only, encoder-decoder, and decoder-only. code2vec @xcite is one of the first language models to attempt to understand code by representing code snippets as embeddings. encoderonly models include codebert @xcite and cubert @xcite , which are pretrained bert models @xcite and are typically utilized in search or classification tasks. encoder-decoder models, such as alphacode @xcite and codet5+ @xcite , are instrumental for source code summarization, text-to-code and code editing @xcite . recently, decoder-only transformer models, such as codex @xcite , codegeex @xcite , starcoder @xcite and wizardcoder @xcite , comprehend the state-of-the-art in generating code from natural language descriptions. in our work, we use llms as generators of accounting dsls that can be inherently evaluated via compilation. large language models on accounting tasks. llms are leveraged to perform accounting tasks, such as auditing @xcite @xcite and analyzing financial statements @xcite . eulerich and wood (2023); emett et al. (2023) show that chatgpt can help in open generation tasks such as audit report writing. kim et al. (2024) examine the ability of llms (namely gpt-4) to analyze financial statements. their findings suggest that gpt-4 and human analysts complement each other and chain-of-thought prompting (cot) @xcite leads to significantly better results. @xcite also make use of cot prompting for co-piloted auditing and present financial ratio analysis, post-implementation review and journal entry testing as examples. we leverage cot as a framework to direct and enhance the output of llms for our accounting scenario tasks. plain text accounting tools. in contrast to common erp systems, plain text accounting is based on human-readable text files, which facilitates access and editing of transactions by both humans and machines. among the most popular tools for plain text accounting are ledger @xcite , hledger @xcite and beancount @xcite . beancount offers features that are tailored to domains such as trading and investing and provides the most customization. the corresponding compiler is highly pessimistic and follows a strict approach assuming an unreliable user. hence, we use beancount as a target for the evaluation of our experiments. ## evaluation results transactions. we report the distribution of the transaction error classes across the 300 samples in codellama and codeqwen 1.5 fail to generate any transactions, which is expected considering their poor performance on generating scenarios. in contrast, mistral, llama 3 and qwen 2 successfully generate transactions, albeit with varying error rates. the qwen 2 model mainly generates transactions that do not balance (61.67%). of the transactions generated by mistral, 28.33% compile successfully but show inconsistencies with the scenarios they are intended to represent. this class is less pronounced in the transactions generated by llama 3 (11.67%) and qwen 2 (16.67%). the model that exhibits the best performance is qwen 2, with 16.67% of transactions being compiled and correct. llama 3 and mistral achieve an accuracy of 15% and 10% respectively. overall, qwen 2 shows the highest accuracy, but generally, all models demonstrate significant limitations in generating correct transactions. ## discussion task generalization from context. we observe significant limitations in the ability of the chosen language models to generate accurate financial scenarios and transactions. the llms that are specialized in code generation performed significantly worse than the general models. in fact, they do not generate a single correct scenario. we argue that this is due to a high sensitivity of the model response to the prompt structure. these models consistently produce the string "processed -waiting for next input." after receiving the task prompt, resulting in no viable scenarios or transactions being generated. while this problem could potentially be mitigated with different prompting strategies, we do not explore this further and leave it as future work. among the scenarios generated by the general purpose models, nearly half are incomplete, often due to unspecific account descriptions. only qwen 2 generates scenarios with an error rate in the accounts of less than 50%. this result is problematic because accounting is typically a field in which accuracy is of paramount importance. ## conclusion we evaluate the capabilities of open-weight large language models in generating meaningful accounting scenarios and code for plain text accounting with domain specific languages. through a comprehensive evaluation of two novel tasks we gauge the domain knowledge and financial numeracy of these models. we highlight that the models show very poor performance. in a human expert evaluation we find that only for 2.3% scenario-transaction generations, llms succeed at our tasks, with most of them stemming from a single model (llama 3). these results raise significant concerns about the practical applicability of llms for code generation using domain-specific languages for accounting. our results show that even successfully compiled (balanced) transactions can be flawed (e.g. hallucinated effects), severely propagating errors in an automated evaluation and assessment of results. although we evaluate state-of-the-art prompt engineering techniques, these seem to be limited towards steering llms to a useful generation of transactions. this is worsened by the time-intensive nature of both inference and scenario human evaluation, which further complicates the search for a "golden prompt".