U ni M ath: A Foundational and Multimodal Mathematical Reasoner (2023)

Abstract: While significant progress has been made in natural language processing (NLP), existing methods exhibit limitations in effectively interpreting and processing diverse mathematical modalities. Therefore, we introduce UniMath, a versatile and unified system designed for multimodal mathematical reasoning tasks. Tackling complex problem-solving in arithmetic, geometry, and table-based math, UniMath utilizes a fine-tuned T5 model augmented with a variational autoencoder (VAE)-based image tokenizer. By jointly training and evaluating the model on three diverse datasets - SVAMP, GeoQA, and TableMWP, UniMath achieves state-of-the-art performance. The model’s generalization ability is further demonstrated via fine-tuning on two additional datasets, MathQA and Geo-Proving. Through comprehensive evaluations, we showcase that joint training across diverse math tasks improves overall model performance and enhances its ability to generalize across different mathematical reasoning tasks. This pioneering approach provides a blueprint and inspires further efforts on unified mathematical reasoning with deep learning systems.

## introduction mathematical reasoning, an essential aspect of human intelligence, plays a pivotal role in our daily lives and decision-making processes @xcite . despite the significant progress in natural language processing (nlp) and research, the development of a robust nlp system capable of handling multimodal input and accommodating diverse downstream tasks remains an under-explored challenge. mathematical reasoning naturally involves a wide range of tasks and modalities, reflecting the complexity and adaptability of human thinking in this area. one common task used to test mathematical reasoning skills is solving math word problems (mwp), which necessitates comprehension of tex-tual information and execution of symbolic reasoning @xcite . another essential aspect of mathematical reasoning is geometry problem-solving, which demands the understanding of visual context and reasoning on spatial relations @xcite @xcite . furthermore, tablebased math problem-solving presents a unique challenge as it requires processing heterogeneous and structured table content to extract relevant information for problem-solving @xcite . existing work often resorts to task-specific models, each fine-tuned for a specific modality. while these models perform efficiently in their specific domains, they struggle to generalize across different modalities, which is an essential capability for advanced ai systems. observing this limitation, we propose a shift towards a unified model, applicable across all tasks and grounded in formal symbolic language generation. our proposed model aims to address a wide range of tasks in the mathematical reasoning domain with a single adaptable model, improving its capacity for advanced reasoning similar to human cognition. in the pursuit of multimodal reasoning, existing approaches can be broadly categorized into two types: 1) @xcite leveraging pre-trained image captioning models to convert images into textual descriptions, which are then combined with the text content, and 2) chen et al. (2021a, 2022a ) employing pre-trained image feature extractors, such as resnet @xcite , to obtain latent representations of images and concatenate them with the latent features of the text component. however, these approaches exhibit limitations when applied to mathematical reasoning tasks. both image captioning and resnet models are primarily designed to comprehend and describe real-world images rather than mathematical diagrams containing geometric shapes. as a result, these models table tokenizer vq-vae reconstruction geometry problem solving table problem solving math word problem solving find the mrs. south row. find the numbers in this row for boys and girls. answer: 14 -12 = 2 answer: 732 + 682 = 1414 girls ∵ab is the diameter of ⊙o, chord cd⊥ab, ∴@xmath0, ∵∠@xmath1,∴∠b @xmath2: there are 732 girls and 761 boys. 682 more girls and 8 more boys joined the school. how many girls are there in the school now? may not be suitable for accurately interpreting geometry images in a mathematical context. another challenge in the development of a unified model for mathematical reasoning is the issue of data management, including data collection and tokenization. to achieve unified reasoning, we need to process geometric diagrams, natural languages, and symbolic representations. handling these various data modalities, which involve tasks such as pre-processing and tokenization, emerges as a significant question of exploration. ensuring an efficient and coherent process to unify these divergent data types is critical to the successful implementation and performance of a singular model for mathematical reasoning. in this paper, we present unimath, a unified mathematical reasoner capable of processing multimodal input, including math word problems (mwp), geometry problems, and table-based mwp. we achieve this by fine-tuning a t5 (chung et al., 2022) model and augmenting its vocabulary with tokenized image representations generated by trainable vq-vae @xcite . we jointly train and evaluate our model on three held-in datasets, i.e., svamp (patel et al., 2021), geoqa @xcite and tablemwp @xcite . also, we finetune our unimath towards two held-out datasets mathqa @xcite and geo-proving @xcite to demonstrate its potential on external tasks. furthermore, we conduct a comprehensive evaluation of different data tokenization methods and examine the impact of chainof-thought explanations in training such a multimodal reasoner. we believe that our work serves as a pioneering study in the development of unified mathematical reasoning systems, inspiring future researchers to devise more advanced approaches in this domain. ## our approach the architecture of our proposed model can be found in fig. 1 , a detailed description of our framework is located in the appendix. we build unimath on a t5 backbone (also extendable to any encoderdecoder llms). for the mwp solving and table mwp solving task, we train the model with textual problem description as input and the answer as output. for tables, we follow the approach in @xcite to transform tables into texts. also, we separate the explanation and answer of the tablemwp dataset into two targets during training controlled by different prefixes. for geometry problems, we integrate an external vector quantized variational autoencoders (vq-vae) as our image encoder to transform image patches to new tokens and concatenate them with the textual tokens as the input. this framework is flexible to deal with text-only input and text-image mixed input. also, the image tokenizer employed in our approach is flexible and compatible with various backbone models, allowing for the integration of more advanced geometry feature extractors when they become available. empirically, we use two 2-layer resblocks as our encoder and decoder in vq-vae. our unimath is trained by simply adding the vae reconstruction loss and the cross-entropy loss of our pre-trained language model backbone. during testing, the decoder part of vq-vae is discarded. ## related work math word problem solving: math word problem solving is a prevalent evaluation method for nlp models @xcite @xcite . early solutions relied on statistical and rule-based parsing @xcite , and have evolved into seq2seq-based neural networks @xcite @xcite and llm-enhanced solvers @xcite . geometry problem solving: geometry problem solving often necessitates the understanding of visual diagrams and textual descriptions. state-ofthe-art models and datasets, namely geometry3k @xcite , geoqa @xcite , geoqa+ @xcite , and unigeo @xcite , are aiming to enhance both performance and explainability. structural problem solving: structural math problems, especially those entailing tables, require an intricate blend of interpretation and reasoning. for example, tapas @xcite introduced a new way of parsing tabular data using pre-training on a large corpus of tables. @xcite introduced a new dataset and employed reinforcement learning to select in-context examples. for a more detailed summary of mathematical problem solving, please refer to the survey @xcite . multimodal foundation models: pioneering works such as beit @xcite and uniperceiver @xcite have been investigating pre-training strategies for handling multimodal data. further, all-in-one transformer @xcite and ofa @xcite have shown promise in providing a unified approach to handling diverse modalities. ml-mfsl @xcite introduced meta-learning that enables flexible adaptation across vision and language tasks. ## conclusion in this work, we introduce unimath, a unified mathematical reasoner, aiming to address diverse and multi-modal mathematical tasks. our extensive experiments demonstrate its superior performance and generalizability across various mathematical modalities, outperforming many task-specific baselines. moreover, we share key insights on data preprocessing and tokenization techniques to achieve our unification goal. we further discuss the impact of chain-of-thought explanations in the training process of such a unified reasoner. we hope that this work will encourage further exploration in nlp, and contribute to the broader goal of achieving human-like mathematical reasoning in ai systems.