L anguage (Re)modelling: T owards Embodied Language Understanding (2020)

Abstract: While natural language understanding (NLU) is advancing rapidly, today’s technology differs from human-like language understanding in fundamental ways, notably in its inferior efficiency, interpretability, and generalization. This work proposes an approach to representation and learning based on the tenets of embodied cognitive linguistics (ECL). According to ECL, natural language is inherently executable (like programming languages), driven by mental simulation and metaphoric mappings over hierarchical compositions of structures and schemata learned through embodied interaction. This position paper argues that the use of grounding by metaphoric reasoning and simulation will greatly benefit NLU systems, and proposes a system architecture along with a roadmap towards realizing this vision.

## introduction question-answering (qa) is currently one of the predominant methods of training deep-learning models for general, open-domain language understanding @xcite . while qa is a ver-satile, broadly-applicable framework, recent studies have shown it to be fraught with pitfalls @xcite . a recent workshop on qa for reading comprehension suggested that "there is growing realization that the traditional supervised learning paradigm is broken [...] -we're fitting artifacts" @xcite . in many respects, the problems of nlu mirror those of artificial intelligence (ai) research in general. @xcite seminal work identified a significant common factor at the root of problems in general ai. the current deep-learning paradigm is a statistical pattern-recognition approach predominantly applied to relatively narrow task-specific prediction. in contrast, human cognition supports a wide range of inferences (planning, action, explaining, etc.), hinting at a view of intelligence focused on model-building, specifically, mental models: rich, structured, manipulable, and explainable representations useful for performing in dynamic, uncertain environments. this distinction motivates the quest for a new cognitively-inspired modelbuilding learning paradigm for general ai, which has inspired fruitful subsequent research and discussion (e.g., @xcite ). the observation that nlu and general ai share a common central problem (task-specific predictionbased learning), and the growing realization that deeper text understanding requires building mental models @xcite , motivate the search for an nlu analog of the cognitively-inspired model building paradigm. amid recent position papers highlighting significant differences between human language understanding and current nlu systems @xcite , here we take a more focused look at mental models; challenges arising due to their embodied nature, their importance in general nlu, and how we might begin integrating them into current approaches. mainstream nlu work, be it entirely distributional, such as bert @xcite , or also involving symbolic knowledge representation @xcite , seldom addresses mental models directly. crucially, such approaches lack the interactive worlds within which mental models foot_0 are learned jointly through language and embodied action. the most closely related lines of work to the present proposal are grounded approaches, which feature worlds in the form of interactive environments, and address mapping text to programs (executable semantic parses) (e.g., @xcite @xcite . however, while well-aligned with a model-building paradigm, typically such approaches have been limited to short or synthetic literal language and narrow domains assuming predefined environments. embodied approaches to general nlu, as advocated here, are few and far between. mostly, examples fall under the construction grammar framework @xcite . however, despite their intellectual merit, they were not operationalized to scale readily for mainstream applications (see §3). this position paper argues that executable semantic parsing and grounded approaches to nlu constitute a first step in a much larger program, whose outline is set forth, for general language understanding through embodied cognitive linguistics (ecl). following much cognitive science research (see §3, §4), this paper posits that (1) execution or simulation is a central part of semantics, essential for addressing some of the persistent difficulties in text understanding, and (2) metaphoric inference capabilities are central to knowledge representation, and facilitate grounded understanding of general language. importantly, capacities for both simulation and metaphor are emergent, borne of embodied interaction within an external world. our contributions are: we analyze inherent limitations of sota statistical language models applied to nlu and propose a framework to address these limitations. the novelty of this approach stems from bringing together ideas from the cognitive science literature, the wider ai community, and nlu. this framework constitutes a path to generalize current execution-based methods towards more general language understanding. the world contains 2 crates. each crate contains 4 boxes. oranges and apples are objects. each box may contain up to 5 objects. objects can be moved from one box to another. objects can be removed from boxes or crates. there are two apples in the first box in the first crate. there is one orange and one apple in the second box of the second crate. first, the apples were transfered from the first box of the first crate to the first box of the second crate. next, all apples were removed from the second crate. ## embodied cognitive linguistics: a model building paradigm turning to cognitive science for inspiration, we focus on embodied cognitive linguistics (ecl), an important paradigm directly addressing both desiderata. this section presents a brief overview and key tenets of ecl, specifically the theoretical foundations @xcite and @xcite developed. most contemporary cognitive accounts of language incorporate concepts from ecl to some degree. a full review is out of scope of this work; see @xcite and §4, §5 for discussion in the nlu context. early cognitive theories assumed a disembodied, symbolic representation of knowledge @xcite , separate from the brain's modal systems (vision, motor control, etc.). in contrast, the embodied cognition (ec) view, based on widespread empirical findings, focuses on the role of the body in cognition. in this view, knowledge is stored using multimodal representations (mental imagery, memories, etc.) that arise from embodied experience and action in the world @xcite . ecl postulates that linguistic representations and other, higher-level cognitive functions are deeply grounded in neural modal systems @xcite . this view is compelling, as it addresses the grounding problem @xcite by linking between high-level symbolic constituents of mental representations and experience or action in the physical world @xcite . note that embodiment is far from an end-all for language comprehension: for example, social and cultural aspects too are crucial @xcite . still, ecl laid important conceptual foundations also underlying subsequent accounts: hypothesis 2 (metaphoric representation): human concepts are expressible through hierarchical, compositional, metaphoric mappings over a limited vocabulary of embodied schema. abstract concepts are expressed using more literal concepts. ## linking ecl to nlu and embodied ai research we propose a unifying view of ecl, bringing it closer to contemporary cognitive science and deep learning approaches. this section presents notations and motivating intuitions, further developing the computational framework in §5, §6. the proposal centers around the view of natural language as a kind of neural programming language @xcite , or higher-level cognitive control system for systematically querying and induc- concept symbolic ecl embodied ai primitives basic data structures, operators, variables... schemata: move, container, part-whole... deep neural world & action representations (learned through interaction) knowledge organization a) composition, inheritence b) libraries a) hierarchical, compositional metaphoric mappings b) compiled knowledge executable unit instruction semantic parse ã execution trace intermediate program states mental models t (s, ã) simulation executor emulator † t semantic parsing / grounding parser to executable symbolic program parser to executable neural program o -1 , π preliminaries. at the core of embodied approaches is the partially observable markov decision process (pomdp; @xcite . it governs the relations between states (s), actions (a), observations (o), and rewards (r). of particular interest are the recognition o -1 : @xmath0, policy π : @xmath1, and transition t : s × @xmath2. focusing on mental simulation rather than actual external action, we assume a degree of equivalence between external and internal representations @xcite . we consider internal mental states and actions (s, ã), effecting change to mental models via a learned neural emulator t @xcite . finally, language is considered a form of action @xcite via external and internal utterances (i.e., semantic parses). connecting symbolic & embodied language understanding. executable semantic parsing. involves parsing a novel linguistic input o into a symbolic program a, whose execution foot_2 yields a desired goal state: t o -1 (o) , @xmath3. executable semantic parsing focuses on action in an external, symbolic environment t , and typically doesn't address t , e.g., mapping a natural language question o directly to an executable query a on an sql engine t . ecl semantic parsing. shares the same structure as executable semantic parsing, with the important distinction that simulation is enacted via internal neural representations: t o -1 (o) , ã = s * . the fully neural formulation enables grounded understanding of non-literal language, demonstrated here for the fig. 2 example. metaphoric inference (hyp. 2) facilitates parsing a novel linguistic input o into internal, structured, neural state representations s, ã. accordingly, the utterance @xmath4, the head of the french army" might be parsed to an internal state s composed of a part-whole schema as shown in the figure. the phrase "attacked the russian fort" could be grounded to a parse ã driving simulation over motion and force schemata. the requirement that s and ã should afford mental simulation (hyp. 1) by the neural world emulator t marks an important difference from current neural word embeddings, one that contributes to deeper language understanding; in the resulting mental model t (s, ã), napoleon and the french army likely moved together due to the part-whole relation between them. this inference is non-trivial since it requires implicit knowledge (heads and bodies often move together). indeed, a sota nli model foot_3 considers it "very likely" that the fig. 2 sentence contradicts the entailment that "the french army moved towards the fort but did not enter it." to summarize: ## proposal for an embodied language understanding model formalizing the view characterized above suggests a novel computational model of language understanding. while current statistical models focus on the linguistic signal, research shows that most of the relevant information required for understanding a linguistic message is not present in the words @xcite . accordingly, the ecl view suggests shifting the focus to the mental models that communicators use, and the neural mechanisms used to construct them, e.g., mental simulation. what follows here adapts a relevant cognitiveinspired framework from general ai to the present nlu setting ( §5.1), and discusses computational challenges ( §5.2). note that similar insights have been applied to multi-agent communication problems @xcite , but their application to general nlu has been limited. ## architecture sketch this section presents a schematic ecl-inspired architecture towards the implementation of the comprehension model (c), which addresses the challenges presented in §5.2. fig. 4 shows the proposed architecture. for simplicity, the focus is on a static reading comprehension setting, but the architecture supports richer environments as well. ## implementation roadmap the architecture outlined in §6 is very ambitious; its implementation requires much further research. this section proposes a roadmap to this goal, identifying three sub-goals (fig. 4 ), presented in order of increasing difficulty. broadly speaking, the level of difficulty is determined by which components are assumed as given in the input (here this also means they are hard-coded in a symbolic programming language), and which must be learned. ## conclusions this position paper has proposed an approach to representation and learning based on the tenets of ecl. the proposed architecture, drawing on contemporary cognitive science, aims to address key limitations of current nlu systems through mental simulation and grounded metaphoric inference. we outlined major challenges and suggested a roadmap towards realizing the proposed vision. growing empirical evidence shows that language is intricately intertwined with a vast range of other neural processes. accordingly, this work suggests a symbiotic view of cognitive science, embodied ai, and computational linguistics. by sharing common foundational problems, these fields may better share and co-evolve common solutions. finally, we believe that attaining deeper language understanding must be a large scale effort, beyond the scope of any one research group. we hope that the paradigm presented here will help provide coherence to such efforts. one of our main goals was to stimulate a discussion; moving forward, we welcome comments, feedback, and suggestions.