Translating Tax Law to Code with LLM s: A Benchmark and Evaluation Framework (2025)

Abstract: Catala is a domain-specific programming language for tax law, meant to facilitate the translation of legal text into executable computer code, thanks to a syntax close to that of legal language and reasoning. Legal statutes paired with their Catala translation have been published online periodically, but manual translation remains labor-intensive. In this work, we develop a benchmark for the evaluation of Catala code generation from legal text, including a training set to fine-tune Large Language Models. To assess the quality of the generated code, we introduce an evaluation framework extending current metrics for code generation. Our experiments with few-shot learning, as well as fine-tuned models, suggest the feasibility of automating legal code generation, and contrast with prior attempts to translate legal language into a formal representation.

## introduction many tax agencies across the world have a legal duty to compute income tax owed, on the basis of the statements provided by taxpayers @xcite . in other jurisdictions, the burden of this complex task is on the taxpayer. since the 1990s, the french tax administration has maintained an expert system to calculate taxes and social benefits. this expert system must be periodically updated to follow the evolution of tax law, a process hampered by the limitations of the current programming paradigm. the catala programming language @xcite was designed to address these limitations: first, by providing a domainspecific language better aligned with the syntax of legal language and reasoning, and second, by encouraging collaboration between lawyers and computer scientists using pair programming. a consid-{ "input": "4 a compter du 1er janvier 2022, pour l'application du 5 de l'article d. 823-17 (...) pas celui des al.", "metadata": "declaration champ d'application calculaidepersonnaliseelogementlocatif: entree loyer_principal contenu argent (...) --mayotte", "output": "champ d'application calculaidepersonnaliseelogementlocatif sous condition @xmath0: exception metropole (...) 8 181 eur", "generated_output": "champ d'application calculaidepersonnaliseelogementlocatif sous condition @xmath1: exception metropole (...) 8 181 eur" } figure 1 : extracts of one sample from our dataset, with its input, metadata and reference output. we also show an output generated by qwen2.5-coder-32b-instruct. a translation into english can be found in figure 5 . how to translate legal language into executable computer code is an open research question @xcite @xcite , which can be traced back to initial efforts at representing parts of legislation with tools from expert systems @xcite . it is also of practical significance for tax agencies and taxpayers, as mentioned above. a significant challenge is the substantial human effort required for translation: each section of tax law takes hours to convert into code, the volume of existing laws is immense -e.g. the french tax code spans approximately 3,500 pages -and frequent amendments necessitate continuous up-dates and translations. in addition, the structure of laws is not strictly linear. for instance, some sections modify or override provisions stated in earlier parts. this requires careful management of dependencies between provisions to ensure a consistent and faithful implementation of the legal text. this law-to-code translation task is related to that of semantic parsing of legal language @xcite @xcite . so far, results have been mostly negative, for two main reasons. there is a stark contrast between the language semantic parsers are made for, and legal language. further, there is no large collection of legal text annotated for semantic parsing. catala code forces the programmer to commit to one interpretation, which prohibits alternative interpretations of the same legal text, a feature that would be necessary for a general semantic representation of legal language. but it trades the ability to represent multiple interpretations for the ability to thoroughly represent one interpretation, catching inconsistencies and gaps @xcite , and enabling automated legal reasoning. we report results on par with code generation for other programming languages, making this a positive result in semantic parsing for legal language. our main contributions are: ## related work meaning representations semantic parsing aims at faithfully representing the meaning of language and is a long-standing nlp task -see for example @xcite for a comprehensive review. first-order logic is sufficient to model legal reasoning, as long as humans provide values for ambiguous or vague predicates, as was done in @xcite . but formalisms for semantic parsing generally aim for close syntactic alignment between input and output, as can be found in abstract meaning representation @xcite and universal decompositional semantics @xcite . semantic parsing of legal language has been shown to be a major challenge @xcite @xcite . in particular, sentence length and logical connectives are a problem @xcite . alignment between legal language and formal representation is hard to achieve, even if some formalisms achieve moderate correspondence. legal expert systems while first-order-logicbased frameworks such as prolog are sufficient to represent the logic of laws and regulations, legal language has a specific way of expressing logic, for instance through defeasible logic @xcite . this has prompted the creation of semantic formalisms to represent legal rules. proleg @xcite is an extension of prolog designed to represent japanese law. in particular, it has been augmented with a feature to visualize reasoning traces, to identify bugs in the formalization or issues in a legal text @xcite . there have been attempts to generate proleg from legal language, with promising results on narrow scopes @xcite . openfisca is a software package aimed at representing financial law. so far, it has been developed and published open-source, foot_1 and has been used to model specific aspects of law in scientific publications @xcite . logical english (kowalski and datoo, 2022) is a simplified version of the english language, which may be easily mapped to first-order logic. in that respect, it is close to a controlled natural language @xcite . code generation existing models can generate code in a variety of programming languages, and at varying levels of granularity @xcite . in particular, github repositories are a source of data to train llms on code. codex @xcite is a gpt-3 model fine-tuned on code from github. similarly, deepseek-coder-v2 was fine-tuned from deepseek-v2 (deepseek-ai et al., 2024), and codellama from llama 2 (rozière @xcite . in contrast, starcoder models were trained on code only @xcite . llms trained on code are generally proficient on widelyused languages such as python. catala is a lowressource language. to the best of our knowledge, the only existing ressource is the github repository we used in this paper. querying the tool "am i in the stack?" foot_2 for "catalalang" showed that stack v2.0.1 and v1.2 @xcite contain the repositories catalalang/catala and catalalang/catala-website. the former holds the compiler for catala, in ocaml. the latter is the source code for http://catala-lang.paris. inria.fr/ . this means starcoder models have seen a trace amount of catala code, in the form of snippets written on the catala website. code generation with llms may leverage controlled languages and constrained decoding @xcite . as a first step, we turn to efficient methods for fine-tuning llms: low-rank parameter adaptation @xcite and its quantized versions @xcite . evaluation metrics benchmarks for code generation generally pair natural-language instructions with reference, expected code output. this makes it possible to evaluate code generation as a machine-translation task. borrowing from the bleu score @xcite , ren et al. ## dataset the publicly available catala code repository on github 4 contains examples of legal texts translated into catala by computer scientists and lawyers. topics include housing benefits (aides logement), family allowances (allocations familiales), the monthly basis for family benefits (base mensuelle allocations familiales), inheritance law (droit successions), and income tax (impôt sur le revenu). we extracted and structured the data into json format. each sample in our dataset corresponds to a single provision in a legal statute, structured as follows (see figure 1 ): the dataset was randomly split into 70% training, 15% validation and 15% test. since samples come from diverse legal contexts and are shuffled before splitting, the training, validation and test sets share similar statistical properties. the dataset has 416 training, 86 validation and 89 test samples, with varying input and metadata lengths. this can be challenging, as our 4096-token context window may not capture all information. using the tokenizer of our best-performing model (qwen-2.5-coder-32b-instruct), and concatenating input, output and metadata, this 4096-token window covers 97% of train, 95% of validation, and 93% of test. for comparability across models, we do not exclude samples in our experiments, instead truncating the input as needed. the size of the resulting dataset is comparable to other specialized code generation datasets @xcite . figure 2 shows more details about the length of inputs and outputs. ## metrics we use multiple metrics, each analyzing the code from a different perspective. our approach considers lexical similarity, syntactic correctness, and structural validity. the evaluation framework includes 5 metrics: (1) chrf, character-based similarity between reference and generated code, (2) bertscore: semantic similarity using text embedding models, (3) tree edit distance (ted): structural similarity of syntax trees, (4) valid syntax (vs): checks if the generated code is syntactically correct, and (5) codebleu (ren et al., 2020). ## experiments our primary goal in this experimental evaluation is to assess the effectiveness of different llms in translating legal text into catala code. code generation can be approached as either an autoregressive task or a translation task, with llms representing the current frontier in this domain. these two interpretations correspond to different model architectures: decoder-only models, which generate code token-by-token in an autoregressive manner, and encoder-decoder models, which process input and output as a sequence-to-sequence task. we focus on decoder-only models, as they are the most common architecture used when working with textto-code generation. ## discussion ## conclusion in this paper, we have introduced a benchmark and metrics for translating legal text to computerexecutable code, starting from open-source catala code. we further experiment with llms in fewshot learning and fine-tuning settings. the performance we report is in line with comparable experiments on low-resource programming languages @xcite . our results contrast with prior attempts at semantic parsing of legal language, as we reach non-trivial performance. at present, the model takes as input the legal text and its associated metadata, guiding the generation of the corresponding catala code. in future iterations, we aim to (1) train and evaluate the model on generating both output code and metadata directly from legal text, (2) translate entire documents at once, (3) include unit tests in the evaluation and (4) estimate quantitatively how an automated translation system can assist pair programmers.