PIGL e T : Language Grounding Through Neuro-Symbolic Interaction in a 3 D World (2021)

Abstract: We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language. We factorize PIGLeT into a physical dynamics model, and a separate language model. Our dynamics model learns not just what objects are but also what they do: glass cups break when thrown, plastic ones don’t. We then use it as the interface to our language model, giving us a unified model of linguistic form and grounded meaning. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that result through a literal symbolic representation, or natural language. Experimental results show that our model effectively learns world dynamics, along with how to communicate them. It is able to correctly forecast what happens next given an English sentence over 80% of the time, outperforming a 100x larger, text-to-text approach by over 10%. Likewise, its natural language summaries of physical interactions are also judged by humans as more accurate than LM alternatives. We present comprehensive analysis showing room for future work.

## introduction as humans, our use of language is linked to the physical world. to process a sentence like "the robot turns on the stove, with a pan on it" (figure 1 ) we might imagine a physical pan object. this meaning representation in our heads can be seen as a part of our commonsense world knowledge, about what a pan is and does. we might reasonably predict that the pan will become hot -and if there's an egg on it, it would become cooked . as humans, we learn such a commonsense world model through interaction. young children learn to reason physically about basic objects by manipulating them: observing the properties they have, the robot turns on the stove, with a pan on it. ## pigpen: a resource for neuro-symbolic language grounding we introduce pigpen as a setting for learning and evaluating physically grounded language understanding. an overview is shown in figure 2 . the idea is that an agent gets access to an interactive 3d environment, where it can learn about the world through interaction -for example, that objects such as a vase can become broken if thrown. the goal for a model is to learn natural language meaning grounded in these interactions. ## modeling piglet in this section, we describe our piglet model. first, we learn a neural physical dynamics model m w e p a g b z b w 1 4 u j l 4 h q b e j q j q s s h f f a g q t i k e j u s k 0 v t d x j a t u / k e 3 a f i t p w n n w h s n p w s n w q 1 y 5 4 l q 5 k j z l j g s + f z p 2 / o y v z 8 b g 8 f e d 4 n l u 5 s t x 9 6 6 f 1 2 / c v h v 7 s h / n x k h a u x x t x z u + y 8 e g z x l h l l m o z 5 v g e d n h 0 3 z x o r w f v k d t m j l v 7 l l c q y b s s o j r s j 7 k b g d r o g f r u g f 2 n h d u 3 j s z s 3 p f z 2 y p v h i q s 5 o h t t y y x q n 4 j d e 2 s d o w j j 0 c z / v b b j p t t b y o l e v g z c s j k z t 1 o c 2 j h j s w r q 1 w q b d q 4 s r d c q l n 1 k 3 q a a i d z 8 y i q m l / p i 1 w 7 n 8 v h a j t 5 u c 9 2 7 z n p q 0 l / 2 w b 1 l z 4 n n v m v r v f s d e b i p p h v k v t c 6 i z 0 0 g t x 3 o a v d o f a 0 t n o i f a 3 8 j e f f f z y 7 t 6 0 k v e u e c 0 z 5 q a q j m j 5 3 1 w i z f s y 7 8 n f 3 y p l f w j k m w f z u x / x m u + 8 z n s u b 0 i v 2 p h i t c x b n 6 j f l o a x / v o v a l q g 1 x 6 k u t b l w l o g 9 f p / 7 k x u x b z o g x d v z f j 5 p z s g 5 p d u f j 4 f l 9 9 m j x 6 3 m 3 q h r l h 7 p o h j c f p y r f 5 r y 7 j m f d y i x w m x 8 j x 4 f v w i / g z / f q 7 b j v d m 7 u k j 8 h v p 1 c n f v @xmath0, and second, integrate with a pretrained model of language form. ## experiments we test our model's ability to encode language into a grounded form (pigpen-nlu), and decode that grounded form into language (pigpen-nlg). ## conclusion in this paper, we presented an approach piglet for jointly modeling language form and meaning. we presented a testbed pigpen for evaluating our model, which performs well at grounding language to the (simulated) world.