Neural Event Semantics for Grounded Language Understanding (2021)

Abstract: We present a new conjunctivist framework, neural event semantics (NES), for compositional grounded language understanding. Our approach treats all words as classifiers that compose to form a sentence meaning by multiplying output scores. These classifiers apply to spatial regions (events) and NES derives its semantic structure from language by routing events to different classifier argument inputs via soft attention. NES is trainable end-to-end by gradient descent with minimal supervision. We evaluate our method on compositional grounded language tasks in controlled synthetic and real-world settings. NES offers stronger generalization capability than standard function-based compositional frameworks, while improving accuracy over state-of-the-art neural methods on real-world language tasks.

## introduction capturing the compositional semantics of grounded language is a long-standing goal in natural language processing. composition yields systematicity, and is thus essential to developing systems that can generalize broadly in real-world settings. recent progress with neural module networks @xcite and related models @xcite @xcite have moved neural network methods closer to this goal. these works are largely based on the idea, functionism @xcite , that semantic composition is function composition. in figure 1(a) , function predicates compose by nesting: predicates like ''red'' and ''circle'' operate on sets of elements, progressively filtering them at each project website: https://neural-event-semantics.github.io/ . while functionism is the dominant paradigm in linguistic semantics, there is an intriguing alternative: event semantics @xcite . @xcite is a particularly powerful version of event semantics, wherein the only composition operator is conjunction-structure arises by routing event variables to the function predicates. we illustrate this key difference between paradigms in figure 1 : in a conjunctivist setting (figure 1(b) ), even the relational above has events e 1 , e 2 directly routed as input, rather than taking inputs that are output results of a sequence of filter operations. overall meaning is still preserved, since e 1 concurrently routes to (red, circle) and e 2 to (green, square). all module outputs directly contribute to the final truth value without intermediate steps. altogether, this shift from deriving compositional structure by functional module layout to conjunctive events routing offers a path to improved learnability; we explore the implications of this line of thinking in the context of compositional neural models. we propose neural event semantics (nes), a new conjunctivist framework for compositional grounded language understanding. our work addresses the drawbacks of modern neural module approaches by re-examining the underlying figure 1 : (a) prior neural network methods for compositional semantics, such as neural module networks, derive compositional structure through nested application of function modules. this paradigm, rooted in functionism, is powerful but retains drawbacks to learnability due to its complexity. (b) we propose neural event semantics (nes), a new framework based on conjunctivism, where all words are classifiers and output scores compose by simple multiplication. we call the input spatial regions to these classifiers events: nes derives semantic structure from language by learning how to route event inputs of classifiers for different words in a context-sensitive manner. by relaxing this routing operation with soft attention, nes enables end-to-end differentiable learning without lowlevel supervision for compositional grounded language understanding. semantics framework, shifting from functionism to conjunctivism. the focus of nes revolves around event variables, abstractions of entities in the world (e.g., in images, we can think of events as spatial regions). we treat all words as event classifiers: for each word, a single score indicates the presence of a concept on a specific input (e.g., red, above in figure 1(b) ). we compose output scores from classifiers by multiplication, generalizing logical conjunction. the structural heart of nes is the intermediate soft (attentional) event routing stage, which ensures that these otherwise independent wordlevel modules receive contextually consistent event inputs. in this way, the simple product of all classifier scores accurately represents the intended compositional structure of the full sentence. our nes framework is end-to-end differentiable, able to learn from high-level supervision by gradient descent while providing interpretability at the level of individual words. we evaluate our nes framework on a series of grounded language tasks aimed at assessing its generalizability. we verify the merits of our conjunctivist design in a controlled comparison with functionist methods on the synthetic shapeworld benchmark @xcite . we show nes exhibits stronger systematic generalization over prior techniques, without requiring any low-level supervision. further, we verify the flexibility of the framework in realworld language settings, offering significant gains (+4 to 6 points) in the state-of-the-art accuracy for language and zero-shot generalization tasks on the cic reference game benchmark @xcite . the reliance of these techniques on prespecified module libraries, ground truth supervision on functional module layouts, and/or sample-inefficient reinforcement learning methods @xcite has motivated subsequent work to eschew explicit semantics for recurrent attentional computation techniques @xcite @xcite . this class of more implicit semantics methods offers the benefits of end-to-end differentiability of traditional non-compositional neural networks @xcite , making them better suited for real-world settings. as a trade-off, however, these methods exhibit less systematic generalization than their more explicit counterparts @xcite @xcite . recent work has also suggested that the modular network approach leads to limitations of systematic generalization: functional module layout can lead to entangled concept understanding @xcite . while these works go on to propose mitigating measures, such as module-level pretraining, we consider an orthogonal approach: re-visiting the underlying semantics foundation. this enables us to address the challenges jointly: our nes framework retains the end-to-end learnability of implicit methods, while improving upon the systematic generalizability of explicit ones. ## conclusion in this work, we introduced neural event semantics (nes) for compositional grounded language understanding. our framework's conjunctivist design offers a compelling alternative to designs rooted primarily in function-based semantics: by deriving structure from events and their (soft) routings, nes operates with a simpler composition ruleset (conjunction) and effectively learns semantic concepts without any low-level ground truth supervision. controlled synthetic experiments (shapeworld) show the generalization benefits of our framework, and we demonstrate broader applicability of nes on real-world language data (cic) by significantly improving language and zero-shot generalization over prior state-of-the-art. ultimately, our work shows that deep consideration of the mechanisms for compositional neural methods may yield techniques better suited for differentiable neural modeling, maintaining core expressivity for grounded language understanding tasks. now all terms where |s i | < s * tend to 0, leaving us just the maximum terms in the numerator and denominator where |s i | = s * . thus, equation (9) reduces to 1 1/s * = s * , as desired. further, |f max (s)| has the essential property of always being upper-bounded by s * . we show this by hölder's inequality. let x @xmath0, y @xmath1, and let @xmath2, @xmath3, ∞)). then, thus, with non-negative scores s @xmath4, we have lim β→∞ f max (s) = max(s) and f max (s) ≤ max(s).