C onic10 K : A Challenging Math Problem Understanding and Reasoning Dataset (2023)

Abstract: Mathematical understanding and reasoning are crucial tasks for assessing the capabilities of artificial intelligence (AI). However, existing benchmarks either require just a few steps of reasoning, or only contain a small amount of data in one specific topic, making it hard to analyse AI’s behaviour with reference to different problems within a specific topic in detail. In this work, we propose Conic10K, a challenging math problem dataset on conic sections in Chinese senior high school education. Our dataset contains various problems with different reasoning depths, while only the knowledge from conic sections is required. Since the dataset only involves a narrow range of knowledge, it is easy to separately analyse the knowledge a model possesses and the reasoning ability it has. For each problem, we provide a high-quality formal representation, the reasoning steps, and the final solution. Experiments show that existing large language models, including GPT-4, exhibit weak performance on complex reasoning. We hope that our findings could inspire more advanced techniques for precise natural language understanding and reasoning. Our dataset and codes are available at https://github.com/whyNLP/Conic10K.

## introduction mathematical understanding and reasoning ability is an important component of human intelligence. such an ability is the foundation of data analysis, financial applications and scientific research. though there have been lots of studies @xcite , mathematical reasoning are far from being solved by existing methods @xcite , even with symbolic reasoners @xcite and large language models (llms) @xcite . to evaluate and analyse the mathematical ability, various datasets and benchmarks have been proposed in recent years @xcite @xcite . however, these datasets or benchmarks often suffer from the following problems: (1) the problems can be solved with only a few reasoning steps, so language models may rely on shallow heuristics to achieve high performance @xcite ; (2) the dataset covers a wide range of topics and hence there is only a small amount of data for each topic, which makes it hard to distinguish whether the model fails because of a lack of background information, or due to weak reasoning ability. to address the above issues, we propose conic10k, an open-ended math problem dataset on conic sections in chinese senior high school education. this dataset contains 10,861 carefully annotated problems, each one has a formal representation, the corresponding text spans, the answer, and natural language rationales. figure 1 shows an example problem in our dataset. to evaluate the mathematical understanding and reasoning ability, we perform two different tasks on existing llms: semantic parsing and mathematical question answering (mathqa). semantic parsing assesses a language model's ability to understand mathematics. the model is required to translate the math problem in natural language into its formal meaning representations. mathqa jointly evaluates the language model's ability of mathematical understanding and reasoning. the model needs to gen-erate the answers to questions. since the topic of conic10k is restricted to conic sections, the knowledge required to solve different problems is the same, while the only difference is the difficulty in reasoning. therefore, if the model is able to solve simple problems but not hard ones, we are assured that the failure lies in the lack of ability in mathematical reasoning. our experiments show that current models obtain good performance in semantic parsing. however, in mathqa, these models are far from being satisfactory. when performing zero-shot chain of thought (cot) @xcite prompting, the best model gpt-4 (openai, 2023) can only achieve 15.5% accuracy using human evaluation. when finetuning is further applied, the best model chatglm-6b @xcite still obtains a poor accuracy of 22.5% under human evaluation. when we translate the problems into english and apply zero-shot cot to reason in english, the accuracy of gpt-4 is 26.0%, which is still far below the performance of human experts at 57.5% with a 3-minute time limit for each problem. this shows that the poor performance is not due to the language being used but to a deficiency in reasoning ability. therefore, we believe the mathematical reasoning ability of language models is still limited despite their huge success in natural language understanding. we conclude our contributions as follows: 1) we propose conic10k, a challenging math problem dataset on conic sections in chinese senior high school education, with high-quality annotations of formal representations; 2) we perform experiments to inspect the mathematical understanding and reasoning ability of llms separately; 3) we give detailed analysis on the model behaviour and conduct comprehensive case studies. we hope that our work could help the community to better analyse llms in mathematical understanding and reasoning and inspire more advanced techniques to enhance the mathematical reasoning ability of llms. ## related work there has been a wide range of datasets on math problems in the literature. mathqa @xcite and gsm8k @xcite are math word problem datasets. they focus on open-domain understanding, where the objective is to extract a single equation based on the information about quantities in the problem, rather than mathematical reasoning. similarly, math23k @xcite and ape210k @xcite are popular datasets about chinese math word problems with open-domain scenarios and simple reasoning steps. geometry3k @xcite is a geometry problem-solving dataset that provides formal representations, but the dataset size is small and the problems do not require complex reasoning. aqua @xcite , numglue @xcite and lila @xcite are large-scale datasets of various math problems. they have been used as benchmarks in solving math word problems and mathematical reasoning tasks, but we find that these datasets require only a few reasoning steps. math @xcite is the one with the longest reasoning steps among these datasets. it has been used as a standard benchmark in recent work of llms @xcite . however, while it covers a wide range of problems, it contains limited data in each specific topic, making it hard to analyse the model behavior in detail with reference to one topic. it also does not provide any formal representations. our proposed conic10k contains problems of long reasoning steps using closed-domain knowledge and has highquality annotations with formal representations. a detailed comparison between the aforementioned datasets and conic10k is shown in ## experiments this section describes our experiments to evaluate the mathematical understanding and reasoning abilities of various models. ## results and discussions in this section, we introduce and explain the results of the experiments. the main results of semantic parsing and mathqa are shown in ## conclusion we present conic10k, a math problem understanding and reasoning benchmark. it provides problems that require complex reasoning, while only involving knowledge about conic sections in chinese senior high school education. we test popular llms on both semantic parsing and math question answering, inspecting model performance and behaviours. results show that existing llms, including gpt-4, have poor performance in mathematical reasoning, while most models could achieve good performance in mathematical understanding (but not perfect yet). we analyse the model predictions in detail and find llms tend to hallucinate in reasoning, often fail to find the shortcuts solution, and may lack the knowledge to solve problems. we hope our dataset, conic10k, can help to discover the weaknesses of llms in mathematical understanding and reasoning and inspire more advanced techniques to enhance the mathematical reasoning ability of llms.