Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning (2023)

Abstract: Spatial reasoning over text is challenging as the models not only need to extract the direct spatial information from the text but also reason over those and infer implicit spatial relations. Recent studies highlight the struggles even large language models encounter when it comes to performing spatial reasoning over text. In this paper, we explore the potential benefits of disentangling the processes of information extraction and reasoning in models to address this challenge. To explore this, we design various models that disentangle extraction and reasoning(either symbolic or neural) and compare them with state-of-the-art(SOTA) baselines with no explicit design for these parts. Our experimental results consistently demonstrate the efficacy of disentangling, showcasing its ability to enhance models’ generalizability within realistic data domains.

## introduction despite the high performance of recent pretrained language models on question-answering (qa) tasks, solving questions that require multi-hop reasoning is still challenging @xcite . in this paper, we focus on spatial reasoning over text which can be described as inferring the implicit foot_0 spatial relations from explicit relations foot_1 described in the text. spatial reasoning plays a crucial role in diverse domains, including language grounding @xcite , navigation @xcite , and human-robot interaction @xcite . by studying this task, we can analyze both the reading comprehension and logical reasoning capabilities of models. previous work has investigated the use of general end-to-end deep neural models such as pretrained language models (plm) @xcite in coreference (a grey car, the car) reasoning extraction extraction figure 1 : an example of steps of spatial reasoning on resq dataset. we begin by searching for the plants from the question triplet within the text, enabling us to extract explicit triplets (1,2). next, we apply rules such as inverse to deduce implicit triplets (3,4,5). then, utilizing triplets 5 and 6 we determine the final answer, 'yes'. ntpp: non-tangential proper part ( as a solution for better multi-hop reasoning, recent research has investigated the impact of using fine-grained information extraction modules such as named entity recognition (ner) @xcite , gated entity/relation @xcite or semantic role labels (srl) @xcite on the performance of models. on a different thread, cognitive studies @xcite show when the given information is shorter, humans also find spatial abstraction and use spatial rules to infer implicit information. figure 1 shows an example of such extractions. building upon these findings, we aim to address the limitations of end-to-end models and capitalize on the advantages of fine-grained information extraction in solving sqa. thus, we propose models which disentangle the language understanding and spatial reasoning computations as two separate components. specifically, we first design a pipeline model that includes trained neural modules for extracting direct fine-grained spatial information from the text and performing symbolic spatial reasoning over them. the second model is simply an end-to-end plm that uses annotations used in extraction modules of pipeline model in the format of extra qa supervision. this model aims to demonstrate the advantages of using separate extraction modules compared to a qa-based approach while utilizing the same amount of supervision. ultimately, the third model is an end-to-end plm-based model on relation extraction tasks that has explicit latent layers to disentangle the extraction and reasoning inside the model. this model incorporates a neural spatial reasoner, which is trained to identify all spatial relations between each pair of entities. we evaluate the proposed models on multiple sqa datasets, demonstrating the effectiveness of the disentangling extraction and reasoning approach in controlled and realistic environments. our pipeline outperforms existing sota models by a significant margin on benchmarks with a controlled environment (toy tasks) while utilizing the same or fewer training data. however, in realworld scenarios with higher ambiguity of natural language for extraction and more rules to cover, our end-to-end model with explicit layers for extraction and reasoning performs better. these results show that disentangling extraction and reasoning benefits deterministic spatial reasoning and improves generalization in realistic domains despite the coverage limitations and sensitivity to noises in symbolic reasoning. these findings highlight the potential of leveraging language models for information extraction tasks and emphasize the importance of explicit reasoning modules rather than solely depending on black-box neural models for reasoning. ## proposed models to understand the effectiveness of disentangling the extraction and reasoning modules, we provide three groups of models. the first model is a pipeline of extraction and symbolic reasoning ( §3.1), the second model is an end-to-end plm that uses the same spatial information supervision but in a qa format ( §3.2), and the third model is an end-toend neural model with explicit layers of extraction and reasoning ( §3.3). we elaborate each of these models in the subsequent sections. task the target task is spatial question answering (sqa), which assesses models' ability to comprehend spatial language and reason over it. each example includes a textual story describing entities and their spatial relations, along with questions asking an implicit relation between entities (e.g., figure 1 ). sqa benchmarks provide two types of questions: yn (yes/no) queries about the existence of a relation between two groups of entities, and fr (find relation) seeks to identify all possible (direct/indirect) relations between them. the answer to these questions is chosen from a provided candidate list. for instance, the candidate list for fr questions can be a sublist of all relation types in ## results and discussion here, we discuss the influence of disentangling extraction and reasoning manifested in pistaq and sreqa models compared to various end-to-end models with no explicit design for these modules, such as bert, bert-eq, and gpt3.5. since the performance of extraction modules, spatial role labeling (sprl) and coreference resolution (coref), directly contribute to the final accuracy of the designed models, we have evaluated these modules and reported the results in ## conclusion and future works we investigate the benefits of disentangling the processes of extracting spatial information and reasoning over them. to this end, we devised a series of experiments utilizing plms for spatial information extraction coupled with a symbolic reasoner for inferring indirect relations. the outcomes of our experiments provide noteworthy insights: (1) our observations in controlled experimental conditions demonstrate that disentangling extraction and symbolic reasoning, compared to plms, enhances the models' reasoning capabilities, even with comparable or reduced supervision. (2) despite the acknowledged fragility of symbolic reasoning in real-world domains, our experiments highlight that employing explicit extraction layers and utilizing the same symbolic reasoner in data preprocessing enhances the reasoning capabilities of models. (3) despite the limitations of llms in spatial reasoning, harnessing their potential for information extraction within a disentangled structure of extraction and reasoning can yield significant benefits. all of these results emphasize the advantage of disentangling the extraction and reasoning in spatial language understanding. in future research, an intriguing direction is incorporating spatial commonsense knowledge using llms as an extraction module in the pipeline of extraction and reasoning. additionally, the model's applicability extends beyond spatial reasoning, making it suitable for various reasoning tasks involving logical rules, such as temporal or arithmetic reasoning. however, this methodology can be readily extended to other reasoning tasks that involve a limited set of logical rules, which can be implemented using logic programming techniques. the extraction modules provided in this paper are task-specific and do not perform well on other domains, but they can be fine-tuned on other tasks easily. using llm in the extraction phase can also deal with this issue. also, using msprl annotation on resq(which this data is provided on) decreases the performance of our models. this annotation does not contain the whole existing relations in the context. the evaluation of the reasoning module is based on the existing datasets. however, we cannot guarantee that they cover all possible combinations between spatial rules and relation types. many questions in resq need spatial commonsense to be answered. as a result, due to the limitation of our symbolic spatial reasoner, the performance of the pipeline model is much lower than what we expected. due to the high cost of training gpt3.5 on large synthetic data, we cannot fine-tune the whole gpt3.5 and only provide the gpt3.5 with f ew_shot learning on small human-generated benchmarks. also, due to the limited access, we can only test palm2 and gpt4 on a few examples.