G eo E val: Benchmark for Evaluating LLM s and Multi-Modal Models on Geometry Problem-Solving (2024)

Abstract: Recent advancements in large language models (LLMs) and multi-modal models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2,000 problems, a 750 problems subset focusing on backward reasoning, an augmented sub- set of 2,000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs in solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67% accuracy rate on the main subset but only a 6.00% accuracy on the hard subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities.

## introduction geometry math problems are a key component in assessing the mathematical reasoning skills of k12 students, serving as a critical benchmark for evaluating educational outcomes @xcite . the complexity of solving these problems stems from the requirement to interpret both textual and visual information, in addition to applying mathematical reasoning skills. this complexity has made geometry problem-solving a key area of interest for researchers aiming to evaluate the capabilities of ai models in this domain @xcite @xcite @xcite . in recent years, several datasets, such as geom-etry3k @xcite , pgps9k @xcite , and geomverse @xcite , have been developed to test the proficiency of ai models in solving geometry math problems. yet, these datasets often lack a standardized format and sufficient diversity, complicating the assessment of models' genuine proficiency in geometry problemsolving. furthermore, these datasets typically focus on one type of geometry problem, such as flat geometry, overlooking other crucial areas like solid geometry. this oversight limits the ability to conduct a thorough evaluation across the full spectrum of geometry problems. simultaneously, advancements in large language models (llms) and multi-modal models (mms) have demonstrated significant potential in handling complex reasoning tasks @xcite @xcite . this potential has raised considerable interest in testing these advanced models across a variety of tasks, such as math word problem solving @xcite and physical problem solving @xcite . despite this interest, specific research on evaluating these models' effectiveness in geometry problem-solving remains scarce. therefore, it is critical to develop a new, comprehensive benchmark that can effectively assess llms and mms in geometry problem-solving, especially considering the potential exposure of existing public datasets during model training @xcite . comparing the performance of current llms and mms on such a benchmark is essential, as it could yield valuable insights that further the development of models capable of tackling complex reasoning tasks. to prompt research towards assessing llms' and mms' proficiency in geometry math problemsolving, we introduce the geoeval benchmark, a comprehensive collection specifically designed for this task. geoeval features its comprehensive variety, sourced from seven public datasets and formatted uniformly to encompass a wide range of geometric shapes. it includes varied problems, covering flat, solid, and analytic geometry to challenge models comprehensively. geoeval supports dual inputs, accommodating both geometric diagrams and textual problem statements, making it suitable for evaluating both llms and mms. to counter the potential overfitting to previously seen datasets, geoeval introduces diverse challenges through backward reasoning, augmented, and hard subsets, each designed to test different aspects of models' geometry problem-solving abilities. additionally, geoeval is annotated with complexity ratings, allowing for a fine-grained analysis of model performance across various difficulty levels, thus providing a robust framework for advancing ai capabilities in understanding and solving geometry math problems. examples of geometry problems from our geoeval can be found in figure 1 . in this paper, we conduct extensive experiments using the geoeval benchmark to evaluate the proficiency of ten llms and mms in solving geometry problems. this includes three llms: codegen2-16b @xcite , gpt-3.5 (openai, 2022), and gpt-4 (openai, 2023); two llms specialized in mathematics: wizardmath-70b and wizardmath-7b-v1.1 @xcite ; and five mms: llava-7b-v1.5 @xcite , qwen-vl @xcite , mplug-owl2 @xcite , instructblip @xcite , and gpt-4v (ope-nai, 2023). the findings reveal that geoeval forms a challenging benchmark, with both llms and mms struggling to resolve its complexities effec-tively. notably, our results indicate that: 1 models pre-trained on mathematical corpora, such as the wizardmath models, deliver superior performance across various geoeval subsets (section 4.3.1), establishing new benchmarks in the field. 2 one advantage of these models is that they implicitly encompass the required mathematical knowledge demanded to solve geometry math problems (section 4.6). 3 however, we also find that though pre-training on a mathematical corpus is crucial for solving geometry math problems, it may not be enough (section 4.3.4). 4 additionally, we observe that gpt series models exhibit enhanced problem-solving efficiency when tackling geometry questions that they have previously rephrased (section 4.3.4). 5 further analyses underscore the value of incorporating descriptions of geometric diagrams, which significantly aids llms in understanding and solving geometry problems (section 4.5). 6 finally, our experiments show that the performance of both llms and mms declines as the problem length and complexity of the problem increases (section 4.7). through the geoeval benchmark, we believe this research provides the first comprehensive quantitative assessment of the latest llms and mms in the domain of geometry problem-solving. ## related work numerous benchmarks have been developed to assess the capabilities of llms in the the geometry problem-solving task. however, these benchmarks face limitations, such as restricted access, like @xcite and geoshader @xcite datasets, or insufficient scale, as seen with geos++ @xcite . although recent efforts have introduced new benchmarks like geometry3k @xcite , unigeo @xcite , and pgps9k @xcite , they still fall short in offering a uniform format and embracing a wide range of problem types. in response, we introduce the comprehensive and challenging geoeval benchmark, aiming to advance the evaluation of geometry problem-solving abilities. recently, llms @xcite @xcite and mms @xcite @xcite have achieved impressive results on complex tasks, attracting research into their performance across specialized tasks. previous work like mathvista @xcite have concentrated on scientific domains, likewise seed @xcite ) explores models' understanding of temporal and spatial relationships. despite these advancements, there remains a gap in the examination of models' ability to solve geometry math problems. through the geoeval benchmark, we aim to fill this gap by offering a detailed assessment of both llms' and mms' abilities to tackle a variety of geometry math challenges. ## geoeval dataset the geoeval benchmark is structured into four subsets: geoeval-2000, comprising 2,000 problems; geoeval-backward, with 750 problems; geoevalaug, containing 2,000 problems; and geoeval-hard, including 300 problems. the subsequent sections will detail the collection process for each subset, followed by an explanation of the unique features of the geoeval benchmark. 2 ## conclusion in this study, we present geoeval, a benchmark developed to assess the geometry problem-solving capabilities of large language models (llms) and multi-modal models (mms). geoeval comprises four distinct subsets, each designed to facilitate a thorough evaluation. through our assessment of ten cutting-edge llms and mms using the geoeval benchmark, we underscore the critical role of mathematical corpus pre-training for effective geometry problem resolution. this is exemplified by the wizardmath model's leading performance on the geoeval-2000 subset, achieving an accuracy of 55.67%. however, the wizardmath model's challenges with the geoeval-hard subset suggest a need for enhanced reasoning skills. additionally, our analysis reveals that gpt-series models exhibit improved performance on geometry problems they have rephrased, pointing to the potential benefits of self-rephrasing in problem-solving.