Embodied Language Learning: Opportunities, Challenges, and Future Directions (2024)

Abstract: While large language and vision-language models showcase impressive capabilities, they face a notable limitation: the inability to connect language with the physical world. To bridge this gap, research has focused on embodied language learning, where the language learner is situated in the world, perceives it, and interacts with it. This article explores the current standing of research in embodied language learning, highlighting opportunities and discussing common challenges. Lastly, it identifies existing gaps from the perspective of language understanding research within the embodied world and suggests potential future directions.

## introduction besides observing their surroundings, humans actively contribute to their understanding of the world by interacting with it and communicating with others @xcite . this interactive experience is integral to language acquisition and understanding @xcite . a corresponding notion of world scopes @xcite , namely corpus, internet, perception, embodiment, and social, has been proposed to measure progress in language understanding research. while today's large language models (llms) have exhibited powerful capabilities @xcite , their textual training data constrain them to the internet world scope. in turn, large visionlanguage models, trained on image-text corpora, fall within the world of perception. significant research efforts (see §2) have been devoted to transitioning into the embodied realm. however, lying at the intersection of language and robotics research, embodied language learning has been primarily explored from a robotics perspective, with a focus on the general use of language in robotics @xcite , embodied visionlanguage tasks @xcite @xcite , or foundation models for decision-making @xcite or as agents @xcite . the main contribution of this article is providing an overview of embodied language learning from the perspective of language understanding research, summarizing relevant background ( §2), opportunities ( §3), challenges ( §4), and research gaps ( §5). throughout the article, embodied language learning is taken to be the process of language acquisition by a language learner, referred to as an agent or a robot, while it is situated in a physical or a virtual world which it perceives and interacts with through action taking; hence, grounding language in its percepts and actions. ## embodied language learning current dominant approaches to language modeling, as represented by llms, involve training on a vast quantity of textual data. however, as symbol tokens cannot be grounded in other symbol tokens @xcite and meaning is perceived to be residing in the connection between language and extrinsic non-symbolic representations @xcite @xcite ; lake and murphy, 2020), merely training language models on as many textual corpora as possible remains insufficient for capturing meaning @xcite @xcite . it is essential to ground textual corpora in extra-linguistic data @xcite . meaning can then be captured to the extent reflected in such data @xcite . one source of grounding data is perception (harnad, 1990), including visual, tactile, and auditory inputs @xcite . efforts have been directed towards vision-language models that attempt to ground language in the visual input, learning alignments between the modalities. nevertheless, studies on language acquisition among infants @xcite suggested that mere perception is inadequate, and that language cannot be learned from a television @xcite . meaning and language understanding have been investigated with regard to embodiment theories in cognitive science @xcite , which highlight the significance of situated actions @xcite . evidence has supported that meaning representations are grounded in embodied experiences and sensorimotor interactions with the world @xcite @xcite . this has motivated research in embodied language learning, where language is grounded in both perception and action @xcite . much of the research in embodied language learning has been focused on robot learning, following two approaches: embodied exploration and embodied instruction following (see appendix a). in the first approach, a robot interacts with objects while receiving natural language descriptions of its actions and/or object attributes @xcite @xcite . in the second approach, a robot is given a natural language instruction and learns to execute a corresponding short-horizon skill @xcite @xcite @xcite or plan and carry out sequences of actions to reach a corresponding long-horizon goal @xcite @xcite @xcite . recently, @xcite experimented with a robot assigned a nonlanguage task in an environment with language annotations that are useful but not required for task completion. in all approaches, the robot learns to ground language in its sensorimotor experience. ## opportunities & prospects opportunities of grounding language manifest in the various dimensions of understanding it unlocks @xcite , which are discussed below. ## conclusion despite the associated challenges of data scarcity, generalizability, and simulator realism, learning language through embodiment is crucial for establishing the connection between language and the world. with the opportunities it holds for an enhanced understanding of attributes, world dy-namics and affordances, as well as metaphors and abstract concepts, embodied language learning enables a robust language understanding that is essential for language-supported robots. by identifying current research gaps from a language understanding perspective, this article aims to motivate future efforts towards fully exploiting the embodiment experience and more rigorously evaluating embodied language models for their language modeling capabilities.