S pace NLI : Evaluating the Consistency of Predicting Inferences In Space (2023)

Abstract: While many natural language inference (NLI) datasets target certain semantic phenomena, e.g., negation, tense & aspect, monotonicity, and presupposition, to the best of our knowledge, there is no NLI dataset that involves diverse types of spatial expressions and reasoning. We fill this gap by semi-automatically creating an NLI dataset for spatial reasoning, called SpaceNLI. The data samples are automatically generated from a curated set of reasoning patterns (see Figure 1), where the patterns are annotated with inference labels by experts. We test several SOTA NLI systems on SpaceNLI to gauge the complexity of the dataset and the system’s capacity for spatial reasoning. Moreover, we introduce a Pattern Accuracy and argue that it is a more reliable and stricter measure than the accuracy for evaluating a system’s performance on pattern-based generated data samples. Based on the evaluation results we find that the systems obtain moderate results on the spatial NLI problems but lack consistency per inference pattern. The results also reveal that non-projective spatial inferences (especially due to the “between” preposition) are the most challenging ones.

## introduction natural language inference (nli) is a popular task that evaluates nlp systems on text reasoning skills. in the task, a system has to predict an inference relation from a premise text to a hypothesis sentence/phrase. usually, the task is three-or two-way classification, depending on whether in the inference labels of entailment, neutral, and contradiction, the latter two are merged into non-entailment. the task is intended for evaluation of nlp systems on reasoning, however, the systems with competitive results on nli benchmarks are often exploiting dataset biases @xcite @xcite alia) and their performance suffers from out-ofdistribution nli sample problems @xcite . to better evaluate the reasoning skills of nli systems, a series of works have been (semi-)automatically or manually creating nli datasets that specialize in certain semantic phenomena. while some of these datasets come with a training part, most of them are intended solely for evaluation. for example, several datasets have been dedicated to monotonicity reasoning @xcite , negation was targeted by @xcite , the dataset by @xcite focuses on temporal and aspectual inferences, @xcite semi-automatically generated nli problems for implicatures and presuppositions. there are also nli datasets that cover several semantic phenomena, having a separate section for each of the phenomena @xcite . while spatial reasoning has been included in several multi-modal qa datasets @xcite @xcite and in a couple of text-based qa datasets @xcite , to the best of our knowledge, no nli dataset has specifically covered it. 2 this paper fills the gap by semi-automatically creating an nli dataset for spatial inferences. first, we collected a diverse set of nli problems inspired by the inference examples found in the literature on spatial semantics. second, the nli problems were manually converted into nli patterns (see figure 1 ), and finally, we automatically generated a large number of nli problems from the patterns. the paper makes two main contributions: c1. spacenli: the spatial nli dataset with diverse types of spatial inferences; the inference labels of the generated problems are highly faithful (97%) to the labels of the corresponding original patterns. c2. pattern accuracy and its curve: they measure systems' performance on patterns and the consistency of predictions on samples from the same patterns. the conducted experiments answer the following research questions: q1. how much spatial reasoning current sota nli systems are capable of? a1. we found out that the sota nli systems have problems with fine-grained spatial inferences. their performance drops at least by 24% compared to their results on common nli datasets. moreover, their consistency in predictions is sensitive to irrelevant lexical substitutions. q2. what types of spatial inference problems are easy or challenging for the sota nli systems? a2. the results showed that the non-projective spatial relations are most challenging for the models. this was mainly due to difficulty associated with "between" and its frequent occurrence in the evaluation dataset. ## analysis to find out what type of inferences the models find challenging, we analyze the models' performance per inference type. figure 5 shows the sampleand pattern-based accuracy scores of the models per spatial inference types as defined in §2.2. the model ranking based on the sample accuracy varies across the inference types. for instance, the best model, debertav3-l#2, remains at the top of the rankings for all inference types with quite a margin except for the projective type. on average, non-projective spatial inferences are the most challenging for the models. the easiest of the types is argument orientation, the type that is closest to the pp attachment task. for the other inference types, projective inferences are harder than directional ones. the apparent distinction in the scores between the inference types is also preserved for the pa 0.95 score (shown with the dark bars in figure 5 ). the fine-grained analysis additionally shows that the best model, debertav3-l#2, suffers least in terms of consistency on the projective inferences while its performance on this inference type is not among the best. directional non-proj. projective arg. orient spatial inference type 20 30 40 50 60 70 80 accuracy: sample (light) & pattern.95 (dark) deberta-l debertav3-l#1 debertav3-l#2 albert-xxlv2 roberta-l bart-l xlnet-l-cased figure 5: sample-based (in light shades) and pa 0.95 (in dark shades) accuracy scores of the models per spatial inference type. based on the results in figure 5 , the nonprojective nli patterns and samples are the most challenging for the sota models. when looking closer at the set of non-projective problems, it turns out that it contains a high number of problems (46%) with the spatial expression "between" (as shown in we further zoom in on the nli patterns and measure a model's probabilistic predictions for the patterns. namely, following @xcite , we measure a model's confidence and variability. originally the dataset cartography @xcite was used to analyze the training dynamics of a model across the epochs and identify training samples that are easy or difficult for learning. in contrast, we use dataset cartography for analyzing evaluation dynamics across patterns and identifying easy and hard ones. 10 figure 4 illustrates the pattern-based evaluation dynamics of roberta-l @xcite , an average model based on the evaluations. for instance, nli pattern (102f) happens to have one of the most variable samples according to the model predictions: the mean and the standard deviation of the probabilities the model assigns to the entailment class of the samples of (102f) are 0.45 and 0.35, respectively. with its pattern-based construction and restricted vocabulary, spacenli comes close to the imp-pres @xcite and hans @xcite datasets. unlike these datasets, spacenli involves multiple-premised problems and puts more emphasis on satisfying selection restrictions to prevent nonsensical sentences. based on the nature of nli problems, spacenli resembles fracas @xcite as both contain inference problems often found in textbooks on formal semantics. unlike fracas, the inference labels of patterns in spacenli are quite balanced and the number of spatial nli patterns is twice the size of the largest section in fracas. there have been attempts to identify semantic phenomena in existing nli datasets, including aspects of spatial reasoning. by looking up certain keywords, @xcite automatically detect nli problems in multinli @xcite that might contain spatial expressions. they create a mutated sample from the original nli problem by negating the sentence with the potential spatial expression. @xcite annotate multinli problems based on the semantic aspects required by the inference label. their taxonomic categories include the spatial subcategory, grouped with the relational, temporal, causal, and co-reference subcategories. the problems in spacenli are substantially diverse from a semantic perspective than the multinli problems that were identified by @xcite and @xcite . the multinli dataset is crowd-elicited and doesn't have problems with sufficient depth in spatial reasoning. ## conclusion to the best of our knowledge, we have created the first spatial inference dataset that involves diverse spatial inference types. the structure and the evaluation protocol are unique as we focus on performance on the nli patterns and consistency across the samples in the pattern, instead of focusing on mere quantitative accuracy based on the nli problems/samples. the evaluation protocol tests models whether they can consistently recognize inference patterns while generalizing over irrelevant lexical substitutions. the more consistent a model is in its predictions, the less unexpected its behavior becomes. the sota nli models show moderate generalization capacity on spatial problems. while the top-performing model gets the highest overall accuracy, it is ranked third when it comes to the consistency of predictions inside the patterns: predicting at least 95% of the samples per pattern. the introduced pattern accuracy (pa) curves provide a more fine-grained distinction between the models: the models with comparable standard accuracy scores might substantially differ in the consistency of their predictions. overall the performance of models drops ca. 10% when raising the consistency threshold to 95%. this illustrates that the predictions of the sota models are sensitive to lexical replacements that have no effect on the semantics of the inference. the evaluation results revealed that the most challenging inference type is associated with nonprojective locatives mainly due to the complex semantics of "between" while the argument orientation type is the easiest. the latter is somewhat expected as the problems in the argument orientation type are close to the task of pp attachment which llms are expected to be good at.