An Augmented Benchmark Dataset for Geometric Question Answering through Dual Parallel Text Encoding (2022)

Abstract: Automatic math problem solving has attracted much attention of NLP researchers recently. However, most of the works focus on the solving of Math Word Problems (MWPs). In this paper, we study on the Geometric Problem Solving based on neural networks. Solving geometric problems requires the integration of text and diagram information as well as the knowledge of the relevant theorems. The lack of high-quality datasets and efficient neural geometric solvers impedes the development of automatic geometric problems solving. Based on GeoQA, we newly annotate 2,518 geometric problems with richer types and greater difficulty to form an augmented benchmark dataset GeoQA+, containing 6,027 problems in training set and 7,528 totally. We further perform data augmentation method to expand the training set to 12,054. Besides, we design a Dual Parallel text Encoder DPE to efficiently encode long and medium-length problem text. The experimental results validate the effectiveness of GeoQA+ and DPE module, and the accuracy of automatic geometric problem solving is improved to 66.09%.

## introduction in recent years, with the continuous development of deep learning technology in nlp, more and more math problem solvers have been developed. however, most of these works focus on solving arithmetic and algebra problems @xcite @xcite . there are few systems for geometric problem solving, especially those based on the method of the neural networks. the solving of geometric problems requires a combination of text and diagram information, and therefore the study of it also helps to promote the development of cross-modal problemsolving. as shown in figure 1 , a typical geometry problem mainly consists of textual descriptions and geometric diagrams. there are three steps to solve this problem. first, the text and diagram information are encoded separately. second, the solver needs to understand the semantics of the text and diagram information simultaneously. third, in order to solve the problem, we may need to combine the information of text-diagram with relevant theorem knowledge. for example, the problem in figure 1 use the theorem of complementary adjacent angles of a parallelogram. though some previous methods attempt to solve geometric problems automatically, the performance of their solving system is far away from satisfactory @xcite @xcite . they highly relied on limited handcraft rules and were only validated on small-scale datasets, making it hard to generalize to more complex and real-world cases @xcite . in this case, we mainly focus on building an efficient solving system based on neural networks. to resolve the mentioned issues, @xcite proposed a geometric problems dataset geoqa which contains 5,010 geometric problems and the first neural network-based geometric problems solving system ngs. however, we believe that there are some limits to this work. first, we think that the problem type in geoqa is not rich enough, and it only contains angle and length problems as well as a very small number of other types. second, we think that the problems in geoqa are not difficult enough, and the average solving step of the problems in geoqa is only 1.96. third, the geometric problems solver ngs can not effectively solve the problems with long text for the lack of text feature extraction capability. inspired by the exiting works @xcite @xcite , to refresh the research on geometric problem solving and further promote the development of cross-modal numerical reasoning, we newly annotate 2,518 geometric problems containing 636 area-type problems that are not included in geoqa. the problems we collect are more difficult. the average solving step of our problems is 2.61, which compares to 1.96 of geoqa. we add our new dataset to the training set of geoqa to build a new dataset named geoqa+ and it contains 7,528 problems in total and 6,027 for training. to the best of our knowledge, geoqa+ is the largest benchmark dataset for geometry problem solving at present and it improves the overall difficulty and diversity of the original dataset. we further perform data augmentation method on geoqa+, which expand the data size to 12,054 to obtain more diverse data. as for the model, we design a dual parallel encoder dpe that consists of roberta @xcite and a bi-lstm @xcite to address the limit of ngs. our dpe module encodes long and medium-length problem text effectively, and we name this new geometric problems solver as dpe-ngs. we conduct a series of experiments and the experimental results indicate that the geoqa+ dataset and our dpe-ngs model show the superiority over the state-of-the-art results. in summary, our contributions are three-fold: wong et al. (2007) designed the first automatic solver lim-g for geometric problems, but this method was only based on text information to solve the problems. subsequently, seo et al. (2014, 2015) constructed the first automatic problems solver that combines text and diagram information with nlp methods and computer vision technology (ocr). however, this method relies too much on handcrafted rules, and it was only verified on the data set with 185 problems. to improve geos, sachan and xing (2017) replaced these handcraft constraints with geometry axiomatic knowledge in the form of hornclause rules, but their dataset and code are not released. lu et al. (2021) proposed inter-gps which achieved higher accuracy than all previous geometric problem solvers based on rule-based methods. and their dataset geometry3k contains 3,002 geometric problems. but inter-gps was still designed based on the rule-based method and geometry3k is not suitable for training neural network-based solvers because of the complex annotating work. aiming to improve the performance and interpretability of existing models, chen et al. (2021) proposed a geometric problems dataset geoqa, and they proposed the first geometric problems solver based on neural networks named ngs. while the geoqa dataset is not difficult and diverse enough, and the feature extraction ability of ngs is also not good enough. to improve the limits of existing works and promote the development of automatic geometric problem solving, we effectively expand geoqa dataset and propose dpe-ngs model. multimodal reasoning visual question answering is a typical multimodal problem. the solving of this kind of problem often requires the model to have a certain reasoning ability @xcite . on this basis, some methods propose an implicit reasoning framework to jointly encode multimodal information @xcite . however, geometric problem solving is more logical and deductive, and the solving process requires additional knowledge of theorems, so these visual problem answering models are not directly applicable to geometric problem solving. pre-training model in nlp pre-training models have greatly advanced the development of nlp @xcite . and it has also been applied in the automatic solving of mwps @xcite . however, it has not been applied to the automatic solving of geometric problems since the lack of dataset. our experimental results show that the introduction of pretraining models facilitates the possibility of solving geometric problems based on our newly annotate dataset. text data augmentation text data augmentation methods has been widely used in nlp, like eda @xcite and back translation @xcite ) method. we also perform the backtranslated method on geoqa+. we first translate the original data into minor languages and then re-translate the results into the original language. data back translation enhances the diversity of data. ## geoqa+ dataset the original geoqa dataset contains 5,010 geometric problems, 3,509 for training, 746 for validation and 755 for test. we newly annotate 2,518 geometric problems and add them to geoqa's training set to form a new dataset geoqa+ which contains 6,027 geometric problems in training set and 7,528 in total. ## models to improve the limit of ngs, we redesign the text encoder module. we refer to this improved geometric problems solver as dpe-ngs, and the overall structure of dpe-ngs is shown in figure 2 . ## conclusion in this work, we newly annotate 2,518 geometric problems which are more difficult and with richer problem types to expand geoqa and form a new benchmark dataset geoqa+, the largest geometric problem dataset at present. moreover, we propose a new text-encode method(dpe) to improve the limits of ngs. the experimental results show that both geoqa+ and dpe-ngs have contributed to the accuracy improvement, and we have improved the baseline accuracy in automatic geometry problem solving from 60.7% to 66.09%. in the future, we will focus on the understanding of problem diagram by enhancing the ability of diagram features extraction as well as the representation of diagram information.