LANS : A Layout-Aware Neural Solver for Plane Geometry Problem (2024)

Abstract: Geometry problem solving (GPS) is a challenging mathematical reasoning task requiring multi-modal understanding, fusion, and reasoning. Existing neural solvers take GPS as a vision-language task but are short in the representation of geometry diagrams that carry rich and complex layout information. In this paper, we propose a layout-aware neural solver named LANS, integrated with two new modules: multimodal layout-aware pre-trained language module (MLA-PLM) and layout-aware fusion attention (LA-FA). MLA-PLM adopts structural-semantic pre-training (SSP) to implement global relationship modeling, and point-match pre-training (PMP) to achieve alignment between visual points and textual points. LA-FA employs a layout-aware attention mask to realize point-guided cross-modal fusion for further boosting layout awareness of LANS. Extensive experiments on datasets Geometry3K and PGPS9K validate the effectiveness of the layout-aware modules and superior problem-solving performance of our LANS solver, over existing symbolic and neural solvers. We have made our code and data publicly available.

## introduction automatic geometry problem solving (gps) is a long-standing and challenging research topic in both computer vision and natural language processing communities @xcite @xcite . each geometry problem consists of a geometry diagram and a textual problem in different modal forms, complementing each other. gps necessitates comprehensive mathematical reasoning and multi-modal understanding, making it a pivotal testbed for evaluating the high-level multimodal reasoning ability of artificial intelligence. past research works of gps were mainly focused on symbolic solvers @xcite @xcite , which are criticized in respect of complex rules and poor adaptability. with the development of deep learning, neural solvers @xcite @xcite , treating gps as a special vision-language reasoning task, have attracted dominant attention recently. layout information is typically defined as positional coordinates of elements such as text, paragraphs, tables, and figures within images @xcite . supplying layout details for elements in document images facilitates parsing reading sequences, executing information extraction, and enhancing document comprehension @xcite @xcite . in the layout of geometric diagram, the coordinate positions of geometric points and symbols play a crucial role in understanding the elements within geometric diagrams. for example, the coordinate positions of geometric symbols "a" as shown in figure 1 , determine which geo-1 2596 metric points are named a, while the coordinate position of the non-geometric symbol "120 • " determines the numerical assignment of ̸ abd instead of other angle. despite considerable efforts devoted to constructing proficiently crafted representations for geometric diagrams, the explicit fusion of positional information into geometric diagrams remains unexplored. existing neural solvers have adopted different diagram representation schemes, such as feature maps @xcite @xcite , image patches @xcite and textual clauses @xcite . for methods based on image patches and feature maps, several representative geometric problem solvers have employed extensive pre-training strategies, such as jigsaw location prediction @xcite , geometry elements prediction @xcite , masked image modeling @xcite , and character alignment @xcite , to bridge the gap between geometric and natural scene images @xcite @xcite . although rough image pre-training methods have achieved some effectiveness, they often fail to capture finer-grained details. conversely, methods based on text clauses extract the crucial structural and semantic information of geometric problems in the form of clauses. currently, clausebased approaches yield superior inference results through clause-based deductive reasoning @xcite or clause pre-training @xcite . we attribute this to the structured nature of clauses, which makes them more adept at capturing structural information in geometric problems. for example, the structural clause "line b c d" describes a structural relationship that points "b", "c" and "d" lie on one line in order. the semantic clause "m @xmath0. although the textual clauses are capable of capturing the primary layout relationships within the images, they lose significant spatial information during the conversion process of diagram parsing @xcite . they cannot distinguish the geometry diagram (a) and (b) displayed in figure 1 because of loss of position information. for example, " ̸ cba" in figure 1(a ) and (b) need the spatial relationship to determine whether it is acute or obtuse. the lack of positional indicators for geometry elements (such as "a," "b," etc.) makes it challenging for neural solvers based ## method before presenting the neural solver model, we first describe the formal definition of gps task here. given a geometry problem p including a geometry diagram d and a textual problem t prob , the goal is to solve the problem by applying geometric knowledge and obtaining the solution steps s, formulated as @xmath1, t prob } ⇒ s. then solution steps are verified in the form of fill-in-the-blank, multiple-choice, or logical reasons. ## conclusion we propose a layout-aware neural solver lans to understand complex layouts of plane geometry diagrams. benefiting from the multimodal layoutaware pre-training, lans is endowed with abilities of global relationship cognition and cross-modal point alignment. thanks to layout-aware fusion attention, lans further improves cross-modal fusion directed by point positions. the experimental results demonstrate the superiority of lans enhanced with layout awareness.