C hat GLM -Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline (2024)

Abstract: Large language models (LLMs) have shown excellent mastering of human language but still struggle in real-world applications that require mathematical problem-solving. While many strategies and datasets to enhance LLMs’ mathematics are developed, it remains a challenge to simultaneously maintain and improve both language and mathematical capabilities in deployed LLM systems. In this work, we tailor the Self-Critique pipeline, which addresses the challenge in the feedback learning stage of LLM alignment. We first train a general Math-Critique model from the LLM itself to provide feedback signals. Then, we sequentially employ rejective fine-tuning and direct preference optimization over the LLM’s own generations for data collection. Based on ChatGLM3-32B, we conduct experiments on both academic and our newly created challenging dataset, MathUserEval. Results show that our pipeline significantly enhances the LLM’s mathematical problem-solving while still improving its language ability, outperforming LLMs that could be two times larger. Related techniques have been deployed to ChatGLM, an online serving LLM. Related evaluation datasets and scripts are released at https://github.com/THUDM/ChatGLM-Math .

## introduction large language models (llms) @xcite @xcite @xcite @xcite have garnered widespread attention for their remarkable proficiency in various linguistic tasks such as text summarization @xcite @xcite , question answering @xcite ; kwiatkowski † yifan and xiao contributed equally. emails: xu-yf23@mails.tsinghua.edu.cn,shawliu9@gmail.com * corresponding author. ## math-critique: a general critic for math definition. we propose math-critique, an evaluation model inspired by large models used for assessment @xcite . it scores mathematical responses based on questions and reference answers, providing explanatory analysis and a score from 1 to 10. unlike traditional reward models, math-critique enhances judgment accuracy by incorporating reference answers and explanatory analysis inspired by thought chains. responses are categorized into four types: entirely incorrect (1-2), partially correct methodology with errors (3-5), accurate conclusion with flawed methodology (6-8), and wholly correct (9-10). math-critique can be defined as: q is the question, r is the reference answer, and a is the evaluated answer. we used two evalua-tion methods: average score evaluation, computing the mean critique scores, and hard-split evaluation, classifying answers as passing or failing based on a correctness threshold and then calculating the proportion of correct answers. ## the self-critique pipeline overview. based on the construction method of math-critique, this section introduces the self-critique pipeline. this pipeline is a weakly supervised iterative training method for enhancing mathematical abilities originating from a single model. initially, we train a math-critique model using the base model and concurrently train a basic chat model using the fundamental sft dataset. subsequently, we employ the math-critique model to supervise the fine-tuning of the chat model through rejection sampling. the outcome of this step can serve as a new base model to update both the math-critique model and the rejection sampling supervised fine-tuning model. building upon these steps, our final action involves utilizing the latest math-critique model to sample contrast data and then proceeding with dpo training. in the following formula, we use mc to represent mathcritique. in these steps, the data construction for the mathcritique-base involves a small amount of manual annotation. however, this batch of annotations is a one-time effort, as only this batch of annotated data instruction: act as a judge … question: a has three eigenvalues 0, 3, and 5, … reference:… @xmath0. model answer: …@xmath1,where p is any real number response: analysis: the ai assistant's answer… score: 9 (1) train math-critique (2) train critique-rft model pseudo tag & human label questions references question and references are sampled from our math dataset. generate the pseudo tag from critiquellm and manually label it. the data is used to train the math-critique model from the base model. the score in response refers to the quality of the model answer. ans. 1 ans. 2 … ans. n sample multiple candidate answers from our model. questions 10 8 … 2 answers are scored by math-critique model. critique-rft training set question response select better model responses for training our rejected sampling supervised fine-tuning model (critique-rft model). ans. 1 ans. 2 … ans. n questions sample questions were incorrectly answered during step (2) from our critique-rft model. ans. 1 9 … … ans. n 2 chosen rejected answers are reranked by the math-critique model. best and worst answers are labeled as chosen and rejected answers. critique-dpo training set question response-chosen response-rejected select chosen and rejected answers for training critique-dpo model. ## mathusereval: benchmarking llms' math reasoning in application mathusereval is a test set designed for realuse scenarios, addressing user concerns and more challenging mathematical problems. some data originates from university examination questions, while others come from simulated dialogues. in the latter, annotators posed math-related questions using large models based on their daily experiences and observations. based on the distribution of the collected data, we divided the test set into two main categories, elementary and advanced, and eight sub-categories. given that calculate applications are less challenging and closely aligned with the scope of previous public datasets, we selected fewer questions from this category. the quantity of questions in each of these categories is as shown in we offer two evaluation methods: gpt-4-1106-preview @xcite @xcite evaluation and math-critique evaluation. the former adopts the alignbench @xcite evaluation method for a more accurate, fair, and accessible approach; the latter uses the math-critique method described earlier. ## experiment 6.1 data collection the primary sources of our data collection include public datasets and publicly available middle school and university examination questions. we selected english data prompts from the gsm8k and solution: the length of the first curve of the track: 3.14 × 72.6 = 227.964 (meters), the length of the second curve of the track: 3.14 × (72.6 ÷ 2 + 1.25) × 2 = 235.814 (meters); the difference between the two curves: 235.814 -227.964 = 7.85 (meters); ## conclusion in this paper, we present math-critique, a method for evaluating mathematical problem correctness, and self-critique, aimed at enhancing language models' mathematical skills without supervision. our experiments in english and chinese show a 32b parameter model achieving state-of-the-art results among open-source models, surpassing proprietary models like gpt-4-0613 on the math-usereval test set. this approach was key in developing glm-4, boosting performance on datasets such as mathusereval, gsm8k, and the hun-limitations graphic thinking and drawing abilities. our purely linguistic model struggles with tasks requiring drawing or understanding images. for instance, it correctly lists, but cannot accurately draw, a topology of numbers requiring divisors in the hungary test. integrating multimodal input and output components could address these issues, which is a direction we plan to explore. precision calculation capability. the model may deviate up to 5% for problems needing multiplication, division, or exponentiation of numbers with three or more decimal places, similar to gpt-4 without an integrated code interpreter. this fundamental limitation might not be resolved by merely increasing the model size. utilizing external computation tools or code interpreters can solve this problem, though our focus here remains on pure language models. we aim to tackle these precision issues in future work. pei ke, bosi wen, zhuoer feng, xiao liu, xuanyu lei, jiale cheng, shengyuan wang, aohan zeng, yuxiao dong, hongning wang, et al. 2023. critiquellm: scaling llm-as-critic for effective and explainable evaluation of large language model generation. arxiv preprint arxiv:2311.18702. rik koncel-kedziorski, subhro roy, aida amini, nate kushman, and hannaneh hajishirzi. 2016. mawps: a math word problem repository. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152-1157. vivek kumar, rishabh maheshwary, and vikram pudi. 2021. adversarial examples for evaluating math word problem solvers. in findings of the association for computational linguistics: emnlp 2021, pages 2705-2712. tom kwiatkowski, jennimaria palomaki, olivia redfield, michael collins, ankur parikh, chris alberti, danielle epstein, illia polosukhin, jacob devlin, kenton lee, et al. 2019. natural questions: a benchmark for question answering research. transactions of the association for computational linguistics, 7:453-466. harrison lee, samrat phatale, hassan mansoor, thomas mesnard, johan ferret, kellie lu, colton bishop, ethan hall, victor carbune, abhinav rastogi, and sushant prakash. 2023. rlaif: scaling reinforcement learning from human feedback with ai feedback. yudong li, yuqing zhang, zhe zhao, linlin shen, weijie liu, weiquan mao, and hui zhang. 2022. csl: a large-scale chinese scientific literature dataset. in proceedings of the 29th international conference on computational linguistics, pages 3917-3923. international committee on computational linguistics. hunter lightman, vineet kosaraju, yura burda, harri edwards, bowen baker, teddy lee, jan leike, john schulman, ilya sutskever, and karl cobbe. 2023. let's verify step by step. wang ling, dani yogatama, chris dyer, and phil blunsom. 2017. program induction by rationale generation: learning to solve and explain algebraic word problems. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 158-167. xiao liu, xuanyu lei, shengyuan wang, yue huang, zhuoer feng, bosi wen, jiale cheng, pei ke, yifan xu, weng lam tam, et al. 2023a. alignbench: benchmarking chinese alignment of large language models. arxiv preprint arxiv:2311.18743. first, we can rewrite 1/x + x 3 as x -1 + x 3 , and then use the derivation principles of power functions and polynomial functions to find its original function. specifically, we get: the ai assistant's answer is very accurate and detailed, and it correctly explains how to solve the original function of a given function. the ai assistant first integrates the two parts of the function separately and then adds their results to obtain the original function. therefore, i would give it a high score. rating:[[9]] given the three eigenvalues of the third-order matrix a are 0, 3, 5, the corresponding eigenvectors are u, v, w. to find the general solution to a system of linear equations @xmath2, we can exploit the properties of eigenvalues and eigenvectors. first of all, we know @xmath3. therefore, v and w are the eigenvectors corresponding to the eigenvalues 3 and 5 respectively. the system of linear equations @xmath4. we can write x as @xmath5, where p, q, r are the undetermined coefficients. substituting x into the @xmath6, we get: where p is any real number. so, the general solution is in the form: hereu is the eigenvector corresponding to the eigenvalue 0, v and w are the eigenvectors corresponding to the eigenvalues 3 and 5 respectively. p is any real number and can take on any value. the ai assistant's answer is very detailed. it first explains how to use the properties of eigenvalues and eigenvectors to solve this problem, and then goes through the detailed steps, including rewriting the equation, writing x in the form of undetermined coefficients, and then converting these the coefficients are substituted into the equation and finally the coefficients are solved. this answer is not only accurate but also in-depth, providing detailed explanations and steps so the reader can understand how to solve the problem. compared with the reference answers, the ai assistant's answers are more detailed and in-depth, so i think the quality of the ai assistant's answers is very high. rating: @xcite ]. rewrite the linear equation 4x-3y=12 into the expression of y: 4x-3y=12 -3y = -4x + 12 @xmath7, @xmath8; when @xmath9, @xmath10; when @xmath11, @xmath12; when @xmath13, @xmath14. therefore, when x is 0, 1, 2, and 3 respectively, the values of y are -4, -8/3, -4/3, and 0 respectively.