SPARTQA : A Textual Question Answering Benchmark for Spatial Reasoning (2021)

Abstract: This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs’ capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.

## introduction spatial reasoning is a cognitive process based on the construction of mental representations for spatial objects, relations, and transformations @xcite , which is necessary for many natural language understanding (nlu) tasks such as natural language navigation @xcite @xcite , human-machine interaction @xcite , dialogue systems @xcite , and clinical analysis @xcite . modern language models (lm), e.g., bert @xcite , albert @xcite , and xlnet @xcite have seen great successes in natural language processing (nlp). however, there has been limited investigation into spatial reasoning capabilities of lms. to the best of our knowledge, babi @xcite (fig 9 ) is the only dataset with direct textual spatial question answering (qa) (task 17), but it is synthetic and overly simplified: (1) the underlying scenes are spatially simple, with only three objects and relations only in four directions. (2) the stories for these scenes are two short, templated sentences, each describing a single relation between two objects. (3) the questions typically require up to two-steps reasoning due to the simplicity of those stories. to address these issues, this paper proposes a new dataset, spartqa foot_0 (see fig. 1 ). specifically, (1) spartqa is built on nlvr's @xcite images containing more objects with richer spatial structures (fig. 1b ). (2) spartqa's stories are more natural, have more sentences, and richer in spatial relations in each sentence. (3) spartqa's questions require deeper reasoning and have four types: find relation (fr), find blocks (fb), choose object (co), and yes/no (yn), which allows for more fine-grained analysis of models' capabilities. we showed annotators random images from nlvr, and instructed them to describe objects and relationships not exhaustively at the cost of naturalness (sec. 3). in total, we obtained 1.1k unique qa pair annotations on spatial reasoning, evenly distributed among the aforementioned types. similar to babi, we keep this dataset in relatively small scale and suggest to use as little training data as possible. experiments show that modern lms (e.g., bert) do not perform well in this low-resource setting. this paper thus proposes a way to obtain distant supervision signals for spatial reasoning (sec. 4). as spatial relationships are rarely mentioned in existing corpora, we take advantage of the fact that spatial language is grounded to the geometry of visual scenes. we are able to automatically generate stories for nlvr images @xcite via our newly designed context free grammars (cfg) and context-sensitive rules. in the process of story generation, we store the information about all ob-questions: fb: which block(s) has a medium thing that is below a black square? a, b, c fb: which block(s) doesn't have any blue square that is to the left of a medium square? a, b fr: what is the relation between the medium black square which is in block c and the medium square that is below a medium black square that is touching the bottom edge of a block? left co: which object is above a medium black square? the medium black square which is in block c or medium black square number two? medium black square number two yn: is there a square that is below medium square number two above all medium black squares that are touching the bottom edge of a block? yes jects and relationships, such that qa pairs can also be generated automatically. in contrast to babi, we use various spatial rules to infer new relationships in these qa pairs, which requires more complex reasoning capabilities. hereafter, we call this automatically-generated dataset spartqa-auto, and the human-annotated one spartqa-human. experiments show that, by further pretraining on spartqa-auto, we improve lms' performance on spartqa-human by a large margin. 2 the spatially-improved lms also show stronger performance on two external qa datasets, babi and boolq @xcite : bert further pretrained on spartqa-auto only requires half of the training data to achieve 99% accuracy on babi as compared to the original bert; on boolq's development set, this model shows better performance than bert, with 2.3% relative error reduction. 3 2 further pretraining lms has become a common practice and baseline method for transferring knowledge between tasks @xcite . we leave more advanced methods for future work. our contributions can be summarized as follows. first, we propose the first human-curated benchmark, spartqa-human, for spatial reasoning with richer spatial phenomena than the prior synthetic dataset babi (task 17). second, we exploit the scene structure of images and design novel cfgs and spatial reasoning rules to automatically generate data (i.e., spartqa-auto) to obtain distant supervision signals for spatial reasoning over text. third, spartqa-auto proves to be a rich source of spatial knowledge that improved the performance of lms on spartqa-human as well as on different data domains such as babi and boolq. ## related work question answering is a useful format to evaluate machines' capability of reading comprehension @xcite and many recent works have been implementing this strategy to test machines' understanding of linguistic formalisms: 2020 ). an important advantage of qa is using natural language to annotate natural language, thus having the flexibility to get annotations on complex phenomena such as spatial reasoning. however, spatial reasoning phenomena have been covered minimally in the existing works. to the best of our knowledge, task 17 of the babi project @xcite is the only qa dataset focused on textual spatial reasoning (examples in appendix f). however, babi is synthetic and does not reflect the complexity of the spatial reasoning in natural language. solving task 17 of babi typically does not require sophisticated reasoning, which is an important capability emphasized by more recent works (e.g., dua et al. (2019 2020) ). spatial reasoning is arguably more prominent in multi-modal qa benchmarks, e.g., nlvr @xcite , vqa @xcite , gqa (hudson and manning, 2019), clevr @xcite . however, those spatial reasoning phenomena are mostly expressed naturally through images, while this paper focuses on studying spatial reasoning on natural language. some other works on visual-spatial reasoning are based on geographical information inside maps and diagrams @xcite and navigational instructions @xcite . as another approach to evaluate spatial reasoning capabilities of models, a dataset proposed in @xcite generates a synthetic training set of spatial sentences and evaluates the models' ability to generate spatial facts and sentences containing composition and decomposition of relations on grounded objects. ## spartqa-human to mitigate the aforementioned problems of task 17 of babi, i.e., simple scenes, stories, and questions, we describe the data annotation process of spartqa-human, and explain how those problems were addressed in this section. first, we randomly selected a subset of nlvr images, each of which has three blocks containing multiple objects (see fig 1b ). the scenes shown by these images are more complicated than those described by babi because (1) there are more objects in nlvr images; (2) the spatial relationships in nlvr are not limited to just four relative directions as objects are placed arbitrarily within blocks. second, two student volunteers produced textual description of those objects and their corresponding spatial relationships based on these images. since the blocks are always horizontally aligned in each nlvr image, to allow for more flexibility, annotators could also rearrange these blocks (see fig. 1a ). relationships between objects within the same block can take the forms of relative direction (e.g., left or above), qualitative distance (e.g., near or far), and topological relationship (e.g., touching or containing). however, we instructed the annotators not to describe all objects and relationships, (1) to avoid unnecessarily verbose stories, and (2) to intentionally miss some information to enable more complex reasoning later. therefore, annotators describe only a random subset of blocks, objects, and relationships. to query more interesting phenomena, annotators were then encouraged to write questions requiring detecting relations and reasoning over them using multiple spatial rules. a spatial rule can be one of the transitivity (@xmath0, @xmath1, symmetry (@xmath2, converse ((a, r, b) ⇒ (b, reverse(r), a)), inclusion (obj1 in a), and exclusion (obj1 not in b) rules. there are four types of questions (q-type). (1) fr: find relation between two objects. (2) fb: find the block that contains certain object(s). (3) co: choose between two objects mentioned in the question that meets certain criteria. (4) yn: a yes/no question that tests if a claim on spatial relationship holds. fb, fr, and co questions are formulated as multiple-choice questions 4 and receive a list of candidate answers, and yn questions' answer is choosing from yes, no, or "dk" (do not know). the "dk" option is due to the open-world assumption of the stories, where if something is not described in the text, it is not considered as false (see fig. 2 ). finally, annotators were able to create 1.1k qa pairs on spatial reasoning on the generated descriptions, distributed among the aforementioned types. we intentionally keep this data in a relatively small scale due to two reasons. first, there has been some consensus in our community that modern systems, given their sufficiently large model capacities, can easily find shortcuts and overfit a dataset if provided with a large training data @xcite . second, collecting spatial reasoning qas is very costly: the two annotators spent 45-60 mins on average to create a single story with 8-16 qa pairs. we estimate that spartqa-human costed about 100 human hours in total. the expert performance on 100 examples of spartqa-human's test set measured by their accuracy of answering the questions is 92% across four q-types on average, indicating its high quality. ## distant supervision: spartqa-auto since human annotations are costly, it is important to investigate ways to generate distant supervision signals for spatial reasoning. however, unlike conventional distant supervision approaches (e.g., mintz et al. (2009); zeng et al. (2015); zhou et al. (2020) ) where distant supervision data can be selected from large corpora by implementing specialized filtering rules, spatial reasoning does not appear often in existing corpora. therefore, similar to spartqa-human, we take advantage of the ground truth of nlvr images, design cfgs to generate stories, and use spatial reasoning rules to ask and answer spatial reasoning questions. this automatically generated data is called spartqa-auto, and below we describe its generation process in detail. ## models for spatial reasoning over language this section describes the model architectures on different q-types: fr, yn, fb, and co. all q-types can be cast into a sequence classification task, and the three transformer-based lms tested in this paper, bert @xcite , albert @xcite , and xlnet @xcite , can all handle this type of tasks by classifying the representation of [cls], a special token prepended to each target sequence (see appendix e). depending on the q-type, the input sequence and how we do inference may be different. fr and yn both have a predefined label set as candidate answers, and their input sequences are both the concatenation of a story and a question. while the answer to a yn question is a single label chosen from yes, no, and dk, fr questions can have multiple correct answers. therefore, we treat each candidate answer to fr as an independent binary classification problem, and take the union as the final answer. as for yn, we choose the label with the highest confidence (fig 8b ). as the candidate answers to fb and co are not fixed and depend on each story and its question the input sequences to these q-types are concatenated with each candidate answer. since the defined yn and fr model has moderately less accurate results on fb and co q-types, we add a lstm @xcite layer to improve it. hence, to find the final answer, we run the model with each candidate answer and then apply an lstm layer on top of all token representations. then, we use the last vector of the lstm outputs for classification (fig 8a ). the final answers are selected based on eq. ( 1 ). ## experiments as fine-tuning lms has become a common baseline approach to knowledge transfer from a source dataset to a target task, including but not limited to @xcite ; @xcite ; @xcite , we study the capability of spatial reasoning of modern lms, specifically bert, albert, and xlnet, after fine-tuning them on spartqa-auto. this fine-tuning process is also known as further pretraining, to distinguish with the finetuning process on one's target task. it is an open problem to find out better transfer learning techniques than simple further pretraining, as suggested in @xcite ; @xcite , which is beyond the scope of this work. all experiments use the models proposed in sec. 5. we use adamw @xcite with 2 × 10 - foot_3 learning rate and focal loss @xcite with γ = 2 for training all the models. 6 ## conclusion spatial reasoning is an important problem in natural language understanding. we propose the first human-created qa benchmark on spatial reasoning, and experiments show that state-of-the-art pretrained language models (lm) do not have the capability to solve this task given limited training data, while humans can solve those spatial reasoning questions reliably. to improve lms' capability on this task, we propose to use hand-crafted grammar and spatial reasoning rules to automatically generate a large corpus of spatial descriptions and corresponding question-answer annotations; further pretraining lms on this distant supervision dataset significantly enhances their spatial language understanding and reasoning. we also show that a spatially-improved lm can have better results on two extrinsic datasets (babi and boolq).