Spatial and Temporal Language Understanding: Representation, Reasoning, and Grounding (2024)

Abstract: This tutorial provides an overview of the cutting edge research on spatial and temporal language understanding. We also cover some essential background material from various subdisciplines to this topic, which we believe will enrich the CL community’s appreciation of the complexity of spatiotemporal reasoning.

## description this tutorial provides an overview of the cutting edge research on spatial and temporal language understanding. we also cover some essential background material from various subdisciplines to this topic, which we believe will enrich the cl community's appreciation of the complexity of spatiotemporal reasoning. one of the essential functions of natural language is to express spatial and temporal relationships between objects and events. linguistic constructs can encode highly complex, relational structures of objects, events, and spatiotemporal relations between them. spatiotemporal language understanding is useful in many research areas and real-world applications. extending two past tutorials on spatial language in emnlp-2020 and coling-2022, we propose this new tutorial that jointly discusses both spatial and temporal semantics for the first time; we also want to take this opportunity to showcase the challenges we still face today in spatiotemporal reasoning, even with state-of-the-art large language models. this topic recently has attracted the attention of various sub-communities in the intersection of natural language, computer vision, and robotics. the complexity of spatiotemporal language understanding and its importance in downstream tasks that involve grounding the language in the physical world has become evident to the nlp research community. the recent evaluation results on large generative language models such as chatgpt show these models struggle with spatial and temporal reasoning while comparatively spatial reasoning appeared harder than temporal reasoning for these models @xcite . while these two aspects of semantics are highly related, there are rare efforts with a focus on a combination of these two semantic aspects. we hope such a tutorial makes the connections more explicit and inspires new ideas for future research in the intersection of spatial and temporal semantic understanding in language and when language is combined with vision and action. similar to various aspects of symbolic semantic representations of language, standardizing tasks related to spatiotemporal language is challenging. it has been rather hard to obtain a set of concepts and relationships together with a formal meaning representation that applies to all real-world situations @xcite @xcite @xcite @xcite @xcite . this has resulted in research on spatiotemporal language learning and reasoning becoming diverse, task-specific, and, to some extent, not comparable. while formal meaning representation is a general issue for language understanding, formalizing spatiotemporal concepts and building formal reasoning and machine learning models based on these concepts have a wealth of prior foundational work that can be exploited and linked to language understanding. in this tutorial, we overview five main themes: 1) spatiotemporal semantic representation; 2) spatiotemporal information extraction and; 3) spatiotemporal qualitative representation and reasoning; 4) reasoning over spatial and temporal information with pre-trained and large generative language models; 5) downstream applications that require spatiotemporal reasoning including language grounding, robotics, navigation, dialogue systems and other tasks that require combining vision and language. these are detailed in three categories in the detailed outline provided in a later section. we cover the research on using spatial concepts for language grounding using spatial commonsense about object affordances @xcite , composi-tional referring expressions, and robotic navigation @xcite . the semantic representation section covers the research that attempted to arrive at a common set of basic concepts and relationships @xcite @xcite as well as making existing corpora interoperable @xcite @xcite @xcite . we discuss the existing qualitative and quantitative representation and reasoning models that can be used for the investigation of interoperability of machine learning and reasoning over spatial and temporal semantics @xcite . spatiotemporal language meaning representation includes research on cognitive and linguistically motivated semantic representations, knowledge representation and ontologies, qualitative and quantitative representation models used for formal meaning representation, and various annotation schemas and efforts for creating specialized corpora. we discuss various datasets that either focus on spatiotemporal annotations or downstream tasks that need spatial and temporal language learning and reasoning. particularly, natural language visual reasoning data @xcite and question-answering data @xcite . moreover, we highlight the lack of research on learning representations that are spatiotemporally rich and point to a few sparse works in this area. we refer to meaning representations and foundation models currently being developed when processing video data which might be inspiring @xcite @xcite . we overview the existing models for extraction of spatial and temporal information from language, both the abstract semantic extraction @xcite @xcite @xcite and extractions driven by various target tasks and applications. we will discuss the recent datasets and results that are probing language models' ability in spatial language understanding using spatial question answering, visual questions answering @xcite @xcite @xcite @xcite and also in the recent diffusion models @xcite . finally, we overview the usage of spatiotem-poral semantics by various downstream tasks and killer applications including language grounding @xcite , navigation @xcite , self-driving cars @xcite robotics @xcite @xcite , dialogue systems @xcite and human-machine interaction, and geographical information systems and knowledge graphs @xcite . spatiotemopral semantics is very closely connected and relevant to the visualization of natural language and grounding language into perception, central to dealing with configurations in the physical world and motivating a combination of vision and language for a richer understanding of time and space. the related tasks include text-to-scene, text-to-video, conversion; image captioning; spatial and visual (image/video) question answering; and spatial understanding in multimodal settings @xcite for robotics and navigation tasks and language grounding @xcite . the current research using end-to-end monolithic deep models fails to solve complex tasks that need deep language understanding and reasoning capabilities (hudson and manning, 2019). throughout this tutorial, we will highlight the importance of combining learning and reasoning for spatiotemporal language understanding and its influence on the semantic representation and type of the learning models as well as the performance on various applications. regarding the question of reasoning, we (a) point out the role of qualitative and quantitative formal representations in helping spatiotemporal reasoning based on natural language and the possibility of learning such representations from data to support compositionality and inference @xcite ; and (b) examine how continuous representations contribute to supporting reasoning and alternative hypothesis formation in learning @xcite . we point to the cutting-edge research that shows the influence of explicit representation of concepts @xcite . the main goal of this tutorial is to combine these current related efforts from different communities and application domains into one unified treatment, to identify the challenges, problems and future directions for spatiotemporal language understanding. ## outline the tutorial will cover the following syllabus. ## prerequisites and reading list familiarity with machine learning and natural language processing will be helpful for tutorial attendees. our selected reading list is as follows. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pp. 4606-4613. 2016. parisa @xcite . in: pustejovsky j., ide n. (eds.), handbook of linguistic annotation springer verlag. in proceedings of emnlp-2020, pages 1158-1172. prior to that, qiang was an applied scientist at alexa (2020-2022) and a research scientist at the allen institute for ai (2019-2020). qiang received his ph.d. from the university of illinois at urbana-champaign in 2019 in electrical and computer engineering. qiang's research interests span in information extraction, question answering, and the application of weak supervision methods in these nlp problems in both theoretical and practical aspects. email: qiangning.01@gmail.com. webpage: https: // www.qiangning.info/ • james pustejovsky is the tjx feldberg chair in computer science at brandeis university, where he is also chair of the linguistics program, chair of the computational linguistics ma program, and director of the lab for linguistics and computation. he received his b.s. from mit and his ph.d. from umass at amherst. he has worked on computational and lexical semantics for 25 years and is the chief developer of generative lexicon theory. since 2002, he has been working on the development of a platform for temporal reasoning in language, called tarsqi ( www.tarsqi.org ). pustejovsky is the chief architect of timeml and iso-timeml, a recently adopted iso standard for temporal information in language, as well as the recently adopted standard, iso-space, a specification for spatial information in language. he has developed a modeling framework for representing linguistic expressions and interactions as multimodal simulations. this platform, voxml, enables realtime communication between humans and computers or robots for joint tasks, utilizing speech, gesture, gaze, and action. he is currently working with robotics researchers in hri to allow the voxml platform to act as both a dialogue management system as well as a simulation environment that reveals realtime epistemic state and perceptual input to a computational agent. his areas of interest include computational semantics, temporal and spatial reasoning, language annotation for machines. email: jamesp@brandeis.edu. webpage: http://www.pusto.com .