Learning Language through Grounding (2025)

Abstract: Grounding has been a long-standing concept in natural language processing (NLP) and computational linguistics (CL). This tutorial provides a historical overview and introduces recent advances in learning language through grounding, with a particular emphasis on the latter. We will begin by tracing the history of grounding and presenting a unified perspective on the term. In Parts II to IV, we will delve into recent progress in learning lexical semantics, syntax, and complex meanings through various forms of grounding. We will conclude by discussing future directions and open challenges, particularly those related to the growing trend of large language models and scaling.

## introduction humans can learn and use language efficiently and effectively. importantly, human language acquisition, learning, and use are grounded to contexts in the world, drawing on information from various sensory modalities such as visual and auditory signals, as well as interactions with other people and the surrounding physical environment. to better understand human language learning and build intelligent language agents, modeling language acquisition and learning has been a long-term interest in the fields of computational linguistics (cl) and natural language processing (nlp), drawing crossdisciplinary interest from fields such as cognitive science and robotics @xcite . recently, we have all witnessed that advances in machine learning, especially those driven by large-scale multimodal data, have significantly advanced the field of grounded language learning @xcite @xcite . although language grounding has emerged as a promising direction in language understanding research and is drawing growing attention from the cl and nlp community @xcite , the usage of the term grounding has diverged @xcite . two major lines of work on grounding started almost contemporaneously in different areas: [1; semantic grounding] in artificial intelligence (ai), which refers to connecting language units to the world that gives language meanings @xcite , and [2; communicative grounding] in psychology, which means finding common ground between agents in a communication setting @xcite . given the interdisciplinary nature and rapid advancement of cl and nlp, especially driven by the emerging trend of language models and scaling, the convergence of various fields has made it imperative to re-examine the concept of grounding in the current context. this tutorial will start by reviewing the evolution of the concept of grounding in the literature and introduce a unified view of grounding @xcite . after that, we will introduce recent work on computational approaches that model the acquisition and learning of natural language by grounding natural language to the world, including learning lexical semantics, syntax, complex forms of semantics, and pragmatics in various grounded settings. the tutorial will conclude with a discussion of future directions and open problems in the field, particularly regarding the emerging trend of language models and scaling. tutorial type. this tutorial should be classified as cutting-edge in cl/nlp, as most of the content was introduced within the last ten years; however, it will also include introductory content that familiarizes the audience with the necessary backgrounds. related tutorials. there have been two *acl tutorials focusing on communicative grounding, i.e., building common ground through dialogue communications @xcite ; in contrast, this tutorial will (1) place greater emphasis on semantic grounding when introducing exemplar work and (2) delve deeper into the pragmatic aspects of language use in communicative grounding, which has not been explicitly mentioned in previous tutorials. additionally, while a related naacl tutorial by a member of the instructor team introduces spatial and temporal language understanding @xcite , this tutorial will offer a broader overview of the grounded language learning problems in general. target audience. this tutorial aims to attract a broad audience. we expect that the tutorial will be of interest to researchers and students in cl/nlp, particularly those who are interested in multimodal machine learning and grounded language learning. we expect the audience to understand basic concepts in calculus (derivatives and calculus), linear algebra (matrix, linear transformation), probability (probability distribution, expectation, and variance), and statistics (sampling from a distribution). familiarity with neural networks, common practices in deep learning, and basic linguistic concepts is desired but not required. ## outline of tutorial content we anticipate that the tutorial will be about 3 hours long. after an introduction to grounding, we break down the tutorial with respect to particular learning objectives. we will review the history of grounding @xcite @xcite , and introduce the unified definition of grounding @xcite . in particular, grounding, in this tutorial, refers to processing the primary data with supervision from another source, where the two sources of data have positive mutual information. we will exemplify the definition through connection to existing work such as visual grounding (e.g., @xcite , acoustic grounding (e.g., @xcite , factual grounding (e.g., @xcite , and cross-lingual grounding (e.g., @xcite . we refer to alikhani and stone (2020) for building common ground through communication. in the first 10 minutes, we will introduce the background and focus on recent advances in the remaining time. work on learning lexical semantics through interaction @xcite or learning lexicon to compose sentence-level meanings @xcite will be deferred to part iv. a brief introduction of related syntactic knowledge, such as constituency @xcite , dependency @xcite , and combinatory categorial grammars (ccgs; @xcite , will be presented in the first 10 minutes of this part to help the audience better understand the content. we will focus on recent approaches to learning syntax through visual grounding and crosslingual grounding in the rest of the time. efforts on joint learning of syntax and semantics (e.g., @xcite will be delivered in part iv. this part will cover three topics for 20 minutes each: learning semantics through grounding, learning pragmatics through grounded interaction, and learning cross-lingual text representations through cross-lingual grounding. there will be a 5-minute q&a session at the end of each part to address questions from the audience. a 30-minute coffee break will be held after part iii. we anticipate that 30% of the tutorial content will be work by at least one of the instructors, with the rest coming from other research groups.