Learning to Solve Domain-Specific Calculation Problems with Knowledge-Intensive Programs Generator (2025)

Abstract: Domain Large Language Models (LLMs) are developed for domain-specific tasks based on general LLMs. But it still requires professional knowledge to facilitate the expertise for some domain-specific tasks. In this paper, we investigate into knowledge-intensive calculation problems. We find that the math problems to be challenging for LLMs, when involving complex domain-specific rules and knowledge documents, rather than simple formulations of terminologies. Therefore, we propose a pipeline to solve the domain-specific calculation problems with Knowledge-Intensive Programs Generator more effectively, named as KIPG. It generates knowledge-intensive programs according to the domain-specific documents. For each query, key variables are extracted, then outcomes which are dependent on domain knowledge are calculated with the programs. By iterative preference alignment, the code generator learns to improve the logic consistency with the domain knowledge. Taking legal domain as an example, we have conducted experiments to prove the effectiveness of our pipeline, and extensive analysis on the modules. We also find that the code generator is also adaptable to other domains, without training on the new knowledge.

## introduction large language models (llms) @xcite @xcite have exhibited outstanding results on several tasks, covering both language understanding and generation, even without additional training. based on which, domain-specific llms @xcite @xcite are developed via techniques such as continual pre-training and supervised fine-tuning (sft). malus law is expressed as 𝐼 𝜃 = 𝐼! 𝑐𝑜𝑠 " 𝜃, where 𝐼 𝜃 is the intensity of the polarized light after passing through a polarizer, 𝐼! is the initial intensity, and 𝜃 is the angle between the light's initial polarization direction and the axis of the polarizer. the speed of sound in air is about 340 m/s at standard atmospheric pressure. how long does it take for sound to travel from a speaker to a listener who are 100 meters apart? article 7 of the supreme people's court's i nterpretation on several issues concernin g... if the victim's injury leads to disability and continuous loss of work... for victims with a fixed income... for victims without a fixed income…; if the victim cannot provide evidence of their average income status ove r the past three years... ## dataset construction for experimental analysis, we construct a dataset in legal domain to evaluate the calculation ability of llms given domain-specific knowledge describing complex instructions. we also collect various knowledge to build another dataset in medical domain as a testset, to assess the generalization. details of the dataset construction are described in appendix e. knowledge selection to improve the challenge of the task, we focus on practical domain knowledge, which mostly contains detailed instructions for different cases. it is non-trivial for llms to strictly follow the complex knowledge document to perform calculation given queries, especially considering that it requires domain knowledge to identify the corresponding conditions. for legal domain, we have engaged professional lawyers with expertise in chinese law to identify satisfactory law articles that involve mathematical computations. for medical domain, we search for medication instructions and indicators involving computations from the internet to ensure the correctness of the documents. some statistics are provided in query and responses the queries should ask a specific question and expect a single number as the answer. the queries are forbidden to provide the clues to the required domain-specific knowledge. for the responses, the annotators are prompted to write the reasoning chains as detailed as possible, which help to explain and check the results. a single numeric answer is expected at the end for each query. for evaluation, we extract the numeric answer with regex and calculate the accuracy. ## method in this section, we introduce the framework of kipg, as illustrated in figure 3 . ## experiments we have conducted extensive experiments comparing kipg with several competitive baselines, including vanilla llms @xcite , cot @xcite , @xcite , algebraic @xcite , incontext-learning @xcite , sciagent @xcite , @xcite , rci @xcite , eedp @xcite , and qwen-max foot_0 with 200b parameters. we also try training techniques with different training datasets. detailed baseline descriptions can be found in appendix a. we train the models on the types of "compensation", "tax" and "other fees", while the calculation problems of the "penalties" and "traffic violations" are served as out-of-distribution test questions. we mainly report the accuracy in our ex- ## conclusion in this paper, we present the task of domain-specific calculation involving knowledge of complex rules and conditions, and the usage of knowledgeintensive programs. additionally, we also intro-duce the framework named kipg to solve the calculation task. we construct datasets in legal and medical domains with human labor and llms in a carefully designed pipeline. empirical results illustrate the superiority of our method from multiple perspectives. kipg outperforms other baselines with different base llms and different scales, in both chinese and english.