Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning (2022)

Abstract: Recent research shows synthetic data as a source of supervision helps pretrained language models (PLM) transfer learning to new target tasks/domains. However, this idea is less explored for spatial language. We provide two new data resources on multiple spatial language processing tasks. The first dataset is synthesized for transfer learning on spatial question answering (SQA) and spatial role labeling (SpRL). Compared to previous SQA datasets, we include a larger variety of spatial relation types and spatial expressions. Our data generation process is easily extendable with new spatial expression lexicons. The second one is a real-world SQA dataset with human-generated questions built on an existing corpus with SPRL annotations. This dataset can be used to evaluate spatial language processing models in realistic situations. We show pretraining with automatically generated data significantly improves the SOTA results on several SQA and SPRL benchmarks, particularly when the training data in the target domain is small.

## introduction understanding spatial language is important in many applications such as navigation @xcite @xcite , medical domain @xcite @xcite , and robotics @xcite . however, few benchmarks have directly focused on comprehending the spatial semantics of the text. moreover, the existing datasets are either synthetic @xcite @xcite or at small scale @xcite . the synthetic datasets often focus on specific types of relations with a small coverage of spatial semantics needed for spatial language understanding in various domains. figure 2 indicates the coverage of sixteen spatial relation types (in q: is the yellow apple to the west of the yellow watermelon? yes three boxes called one, two and three exist in an image. box one contains a big yellow melon and a small orange watermelon. box two has a small yellow apple. a small orange apple is inside and touching this box. box one is in box three. box two is to the south of, far from and to the west of box three. a small yellow watermelon is inside box three. q: where is box two relative to the yellow watermelon? left, below, far (a) spartun -a synthetic large dataset provided as a source of supervision. a grey car is parking in front of a grey house with brown window frames and plants on the balcony. q: are the plants in front of the car? no q: are the plants in the house? yes (b) resq -a human-generated dataset for probing the models on realistic sqa figure 1 : two new datasets on sqa in this work, we build a new synthetic dataset on sqa, called spartun foot_0 (fig 1a ) to provide a source of supervision with broad coverage of spatial relation types and expressions foot_1 . to generate spartun, we follow the idea of spartqa @xcite benchmark and generate scene graphs from a set of images. the edges in this graph yield a set of triplets such as above(blue circle, red triangle), which are used to generate a scene description (i.e., a story). in spartun, we map the spatial relation types in triplets (e.g., above) to a variety of spatial language expressions (e.g., over, north, above) to enable the transfer learning for various data domains 3 . we also build a logical spatial reasoner to compute all possible direct and indirect spatial relations between graph nodes. then, the questions of this dataset are selected from the indirect relations. to evaluate the effectiveness of spartun in transfer learning, we created another dataset named resq 4 (fig 1b ). this dataset is built on msprl @xcite corpus while we added human-generated spatial questions and metaphoric usages and implicit meaning are not covered in this work. 3 the full list of spatial expressions used in this dataset and the dataset generation code are provided in https://github. com/hlr/spartun . the contributions of this paper can be summarized as: (1) we build a new synthetic dataset to serve as a source of supervision and transfer learning for spatial language understanding tasks with broad coverage of spatial relation types and expressions (which is easily extendable); (2) we provide a human-generated dataset to evaluate the performance of transfer learning on real-world spatial question answering; (3) we evaluate the transferability of the models pretrained with spartun on multiple sqa and sprl benchmarks and show significant improvements in sota results. ## related research requiring large amounts of annotated data is a wellknown issue in training complex deep neural mod-els @xcite that is extended to spatial language processing tasks. in our study, we noticed that all available large datasets on sqa task including babi @xcite , spartqa-auto @xcite , and stepgame @xcite are, all, synthetic. babi is a simple dataset that covers a limited set of relation types, spatial rules, and vocabulary. stepgame focuses on a few relation types but with more relation expressions for each and considers multiple reasoning steps. spartqa-auto, comparatively, contains more relation types and needs complex multi-hop spatial reasoning. however, it contains a single linguistic spatial expression for each relation type. all of these datasets are created based on controlled toy settings and are not comparable with real-world spatial problems in the sense of realistic language complexity and coverage of all possible relation types. spartqa-human @xcite is a human-generated version of spartqa-auto with more spatial expressions. however, this dataset is provided for probing purposes and has a small training set that is not sufficient for effectively training deep models. for the sprl task, msprl and spaceeval (semeval-2015 task 8) @xcite are two available datasets with spatial roles and relation annotations. these are small-scale datasets for studying the sprl problem. from the previous works which tried transfer learning on sprl task, @xcite only used it on word embedding of their sprl model, and @xcite used plm without any specifically designed dataset for further pretraining. these issues motivated us to create spartun for further pretraining and transfer learning for sqa and sprl. transfer learning has been used effectively in different nlp tasks to further fine-tune the plms @xcite @xcite . besides transfer learning, several other approaches are used to tackle the lack of training data in various nlp areas, such as providing techniques to label the unlabeled data @xcite , using semi-supervised models @xcite or data augmentation with synthetic data @xcite . however, transfer learning is a simple way of using synthetic data as an extra source of supervision at no annotation cost. compared to the augmentation methods, the data in the transfer learning only needs to be close to the target task/domain @xcite and not necessarily the same. mirzaee et al. is the first work that considers transfer learning for sqa. it shows that training models on synthetic data and finetuning with small human-generated data results in a better performance of plms. however, their coverage of spatial relations and expressions is insufficient for effective transfer learning to realistic domains. using logical reasoning for building datasets that need complex reasoning for question answering is used before in building qa datasets @xcite . more recent efforts even use the path of reasoning and train models to follow that @xcite . however, there are no previous works to model spatial reasoning as we do here with the broad coverage of spatial logic. ## transfer learning for spatial language understanding to evaluate transfer learning on spatial language understanding, we select two main tasks, spatial question answering (sqa) and spatial role labeling (sprl). given the popularity of plms in transfer learning @xcite @xcite , we design plm-based models for this evaluation. in the rest of this section, we describe each task and model in detail. ## spartun: dataset construction to provide a source of supervision for spatial language understanding tasks, we generate a synthetic dataset with sqa format that contains sprl annotation of sentences. we build this dataset by expanding spartqa in multiple aspects. the following additional features are considered in creating spartun: f1) a broad coverage of various types of spatial relations and including rules of reasoning over their combinations (e.g. ntpp(a, b), left(b, c) → left(a, c) ) in various domains. f2) a broad coverage of spatial language expressions and utterances used in various domains. f3) including extra annotations such as the supporting facts and number of reasoning steps for sqa to be used in complex modeling. in the rest of this section, we describe the details of creating spartun and the way we support the above mentioned features. figure 4 depicts spartun data construction flow. ## experimental results the focus of this paper is to provide a generic source of supervision for spatial language understanding tasks rather than proposing new techniques or architectures. therefore, in the experiments, we analyze the impact of spartun on sqa and sprl using the plm-based models described in section 3. in all experiments, we compare the performance of models fine-tuned with the target datasets with and without further pretraining on synthetic supervision (synsup). all codes are publicly available 11 . the details of experimental settings and hyperparameters of datasets are provided in the appendix. ## conclusion and future work we created a new synthetic dataset as a source of supervision for transfer learning for spatial question answering (sqa) and spatial role labeling (sprl) tasks. we show that expanding the coverage of relation types and combinations and spatial language expressions can provide a more robust source of supervision for pretraining and transfer learning. as a result, this data improves the models' performance in many experimental scenarios on both tasks when tested on various evaluation benchmarks. this data includes rules of spatial reasoning and the chain of logical reasoning for answering the questions that can be used for further research in the future. moreover, we provide a human-generated dataset on a realistic sqa task that can be used to evaluate the models and methods for spatial language understanding related tasks in real-world problems. this data is an extension of a previous benchmark on sprl task with spatial semantic annotations. as a result, this dataset contains annotations for both sprl and sqa tasks. in future work, we plan to investigate explicit spatial reasoning over text by neuro-symbolic models. moreover, using our methodology to generate synthetic spatial corpus in other languages or for other types of reasoning, such as temporal reasoning, is an exciting direction for future research.