P ro SP er: Probing Human and Neural Network Language Model Understanding of Spatial Perspective (2021)

Abstract: Understanding perspectival language is important for applications like dialogue systems and human-robot interaction. We propose a probe task that explores how well language models understand spatial perspective. We present a dataset for evaluating perspective inference in English, ProSPer, and use it to explore how humans and Transformer-based language models infer perspective. Although the best bidirectional model performs similarly to humans, they display different strengths: humans outperform neural networks in conversational contexts, while RoBERTa excels at written genres.

## introduction point-of-view, or perspective, affects many aspects of language. this paper presents prosper, a dataset for probing how humans and neural language models track spatial perspective in text. in recent years, neural network language understanding has been probed in a variety of syntactic and semantic tasks. 1 we propose a probe task for one of the most complex aspects of language: relative spatial language, or spatial perspective. we measure the ability to infer spatial perspective using a come/go prediction task: infer a missing motion verb from a passage of text (figure 1 ). this task combines aspects of previous probe tasks (long-distance dependencies, co-reference resolution), but also poses new challenges: (1) ranking the importance of individuals in a discourse, (2) reasoning over ambiguity, and (3) inferring spatial relations. this makes it challenging for any language user: as our behavioral data shows, human performance is not perfect. however, the task may be particularly hard for language models since 1 including subject-verb agreement @xcite @xcite , question formation @xcite , filler-gap dependencies @xcite , anaphora @xcite , category membership @xcite , and negative polarity items @xcite . rick changed the subject. "i heard that you were having some furniture delivered this afternoon," he said to aunt emily. "i thought i'd by and see if you needed any help." this paper explores human and neural network language model understanding of perspective. in sections 2-3, we motivate and present our task and dataset. in section 4, we measure human performance on prosper and find accuracy rates of 77-88%. in section 5, we evaluate pre-trained neural language models and show that the bert family @xcite achieves human-like accuracy. in section 6, we explore differences between human and model behavior. drawing on psycholinguistic work on perspective @xcite , we outline three perspective inference strategies. our evidence supports a genre frequency bias: humans perform best in conversation-like contexts, while roberta excels in written genres, reflecting the language each encounters most during learning. this paper contributes to the understanding of both neural network and human language capabilities. from a cognitive science perspective, our findings contribute to two open debates: the role of grounded information in language acquisition (section 5) and the existence of cognitive biases in perspective inference @xcite . from an applied perspective, our results motivate greater use of conversational data in applications where perspectival language is important, such as in navigation, story generation, and human-robot interaction. ## probing perspective inference the perspectival motion verb prediction task is a compelling probe task because it is simple in design, but high in linguistic complexity. ## prosper: probing spatial perspective we present a new corpus for probing human and neural network ability to infer spatial perspective: prosper. 3 . the prosper dataset consists of two parts: the automatic subset, a large set of examples extracted by string-matching, and the annotated subset, a smaller set for fine-grained analysis. the annotated items were hand-selected for linguistic diversity and annotated with linguistic features. ## human performance we measured human performance on a representative subset of prosper using a forced choice task similar to bidirectional language modeling. 5 human format: and they worked with her and got her walking and got her taking kind of taking care of herself and so she was able to back home bidirectional format: and they worked with her and got her walking and got her taking kind of taking care of herself and so she was able to mask back home unidirectional format: and they worked with her and got her walking and got her taking kind of taking care of herself and so she was able to ## neural language model performance we evaluated the performance of various neural network language models on the prosper dataset. confounding random arrive come drive go walk arrive come drive go walk arrive come drive go walk 0.00 0.25 0.50 0.75 1.00 by-item means verb figure 4: human item means by verb type and corpus (@xmath0, @xmath1 ## conclusion in this paper, we present prosper, a dataset for probing how humans and language models infer spatial perspective from text. we use prosper's come/go prediction task to gather novel psycholinguistic data and evaluate transformer-based language models. we find that the best bidirectional model, roberta, performs similarly to humans, providing tentative evidence that grounded information is not required for spatial language acquisition. despite near-equal accuracy, we find only a medium correlation between roberta and human scores, suggesting different perspective inference strategies. we explored two strategies proposed in prior psycholinguistic work, but ultimately argued in support of a novel genre frequency bias effect: humans perform best in conversation-like contexts, while roberta performs best in third-person narrative contexts. although many of the observed trends support the genre frequency bias, it did not explain the full pattern of results. we hope that prosper's annotated subset will aid future work exploring perspective inference strategies. the observed genre frequency bias is important for both neural network and human language understanding. for cognitive scientists, it fuels an existing debate on cognitive biases in perspective inference. for applications where human-like use of perspectival language is important, like navigation, story generation, and human-robot interaction, it suggests that conversational training data may improve model robustness.