G eo GPT 4 V : Towards Geometric Multi-modal Large Language Models with Geometric Image Generation (2024)

Abstract: Large language models have seen widespread adoption in math problem-solving, yet for geometry problems, which often necessitate visual aids even for humans, the most advanced multi-modal models still struggle to effectively utilize image information. High-quality data is crucial for enhancing the geometric capabilities of multi-modal models, yet existing open-source datasets and related efforts are either too challenging for direct model learning or suffer from misalignment between text and images. To overcome this issue, we introduce a novel pipeline that leverages GPT-4 and GPT-4V to generate relatively basic geometry problems with aligned text and images, facilitating model learning. We have produced a dataset of 4.9K geometry problems and combined it with 19K open-source data to form our GeoGPT4V dataset. Experimental results demonstrate that the GeoGPT4V dataset significantly improves the geometry performance of various models on the MathVista and MathVision benchmarks. The code is available at https://anonymous.4open.science/r/GeoGPT4V-08B2.

## introduction with large language models (llms) demonstrating formidable performance, their application in solving mathematical problems has become an increasingly popular trend @xcite @xcite . prior research has indicated that humans encounter a significant reduction in accuracy when resolving geometric problems devoid of visual aids @xcite . thus, the integration of visual information from images is imperative for accurately solving of such mathematical problems, necessitating the visual perception capabilities of multimodal large language models (mllms). however, even the best batch of mllms available now (such as gpt-4v (openai, 2023b), gemini @xcite ) still lag significantly behind human performance @xcite . therefore, researchers are eagerly exploring methods to enhance the geometric capabilities of mllms. to enhance the geometric capabilities of mllms, an important step is to construct corresponding high-quality data @xcite @xcite . nevertheless, current data often suffer from two main issues. on the one hand, most open-source datasets are quite challenging, making it difficult for models to directly learn geometric capabilities from them @xcite . for instance, the uni-geo @xcite dataset consists of problems extracted from high school textbooks, but the models have not been exposed to the corresponding foundational knowledge. on the other hand, current data augmentation techniques @xcite , using chatgpt-3.5 to adjust numerical values in the text, fail to harmonize these changes with the corresponding values in images. consequently, mismatches between the altered text and images can bewilder the model and impede its learning process @xcite . in this paper, we address the aforementioned issues by introducing a straightforward and efficient pipeline for generating geometric problem data. our objectives are two-fold: @xcite to create geometric problems that facilitate the model's acquisition of basic geometric concepts, and @xcite to ensure that the image and the text of the generated geometric problems are well-aligned. in detail, we first employ gpt-4v to create a collection of simplified geometric problems based on open-source datasets. subsequently, we harness the capabilities of @xcite to generate k individ-ual pieces of wolfram foot_1 code for each geometric problem previously crafted. the code is then executed to produce k distinct geometric images. finally, gpt-4v is employed to score these images, allowing us to select the best one that optimally aligns with the associated textual descriptions. through the above pipeline, we generate a dataset comprising 4.9k geometric problems characterized by simplicity and image-text matching. we then mix our generated problems with 19k problems from open-source datasets to formulate a dataset with various difficulty levels, named geogpt4v. we have conducted comprehensive experiments on the geometry problem subset of mathvista @xcite and mathvision @xcite datasets, two commonly used datasets for multi-modal math. our experimental results show that models of various sizes and types can achieve significant improvements in geometric capabilities after training with our dataset (achieving 58.2% and 33.8% relative improvement for llava-1.5-7b @xcite and sharegpt4v-7b @xcite , respectively, on geometry problem solving (gps) minitest split of mathvista), which validates the effectiveness of our approach. in conclusion, the contributions of this paper are summarized as follows: ## related work in this section, we delve into related studies from two perspectives: multi-modal large language models and mathematical problem solving. ## method in this section, we will elaborate on the pipeline we have constructed. an overview of our pipeline is depicted in figure 1 . specifically, our process includes: (1) generating new question-answer pairs (section §3.1), ( 2 ) producing corresponding geometric images (section §3.2), and (3) scoring and filtering based on the image-text matching degree (section §3.3). formally, the original data from the open-source datasets can be represented as @xmath0, a, i}, where q represents the question, a represents the answer, and i represents the image. ## experiment in this section, we conduct experiments to answer the following research questions (rq): • rq1: can geogpt4v dataset improve geometric capabilities of different models? ## conclusion in this study, we propose a novel pipeline to enhance the geometric capabilities of mllms. we have proposed data generation methods for multimodal geometric tasks involving problem simplification and the generation of images that match newly generated text. specifically, we use gpt4v and gpt4 to generate sub-problems or lead-up problems for given geometric tasks, along with the corresponding wolfram code that can be executed to generate geometric images. based on the pipeline, we have generated 4.9k simplified and image-text matching geometric problems. we mix our generated data with 19k open-source data to formulate a dataset with various difficulty levels, named geogpt4v. after training on the ge-ogpt4v dataset, various models have improved geometric scores on both mathvista and mathvision benchmarks. the extensive experimental results demonstrate the effectiveness of the geogpt4v dataset. we have open-sourced the geogpt4v dataset and the checkpoints of models trained on the geogpt4v dataset, with the aim of fostering the community's growth.