Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games (2020)

Abstract: In visual guessing games, a Guesser has to identify a target object in a scene by asking questions to an Oracle. An effective strategy for the players is to learn conceptual representations of objects that are both discriminative and expressive enough to ask questions and guess correctly. However, as shown by Suglia et al. (2020), existing models fail to learn truly multi-modal representations, relying instead on gold category labels for objects in the scene both at training and inference time. This provides an unnatural performance advantage when categories at inference time match those at training time, and it causes models to fail in more realistic “zero-shot” scenarios where out-of-domain object categories are involved. To overcome this issue, we introduce a novel “imagination” module based on Regularized Auto-Encoders, that learns context-aware and category-aware latent embeddings without relying on category labels at inference time. Our imagination module outperforms state-of-the-art competitors by 8.26% gameplay accuracy in the CompGuessWhat?! zero-shot scenario (Suglia et al., 2020), and it improves the Oracle and Guesser accuracy by 2.08% and 12.86% in the GuessWhat?! benchmark, when no gold categories are available at inference time. The imagination module also boosts reasoning about object properties and attributes.

## introduction humans do not learn conceptual representations from language alone, but from a wide range of situational information @xcite as highlighted also by property-listing experiments @xcite . when humans experience the concept of "boat", they simulate a new representation by reactivating and aggregating multi-modal representations that reside in their memory and are associated with the concept of "boat" (e.g., what a boat looks like, the action of sailing, etc) @xcite . this simulation process is called perceptual simulation. therefore, it is no wonder that recent trends in learning conceptual representations adopt multi-modal and holistic approaches @xcite wherein abstract distributional lexical representations @xcite learned from text corpora are augmented or refined with perceptual information for concrete and context-aware representations built from visual @xcite , olfactory @xcite , or auditory @xcite modalities. language games between ai agents, inspired by wittgenstein's language games among humans @xcite , are an excellent test bed for such approaches since concepts are expected to emerge when agents are required to communicate to solve specific tasks in specific environments. guesswhat?! @xcite is a prototypical language game of this kind: a guesser has to identify a target object in a scene represented as an image by asking questions to an oracle. learning to ground pixels of the scene into object representations that are relevant for the object category they belong to (category-aware), but are also particularized for the specific scene (context-aware), is fundamental for the guesser to effectively converse with the oracle and vice-versa. 2018 ) rely on gold category labels at test time, thereby failing to ground novel objects from categories not seen during training (e.g., a "pasticciotto", top right) or to properly encode known categories but with unseen visual features (like a "frosted donut", bottom right) since they employ category embeddings c from a predefined set that are fixed for each object. instead, embeddings z learned by our imagination module can be flexibly category-aware allowing them to generalize to unseen categories. we consider a model truly multi-modal if it always uses all the modalities to make decisions. however, existing approaches @xcite rely instead on gold category labels that are assumed to be available also at inference time, thus making these models depend on this modality and discarding the others. this not only poses an unnatural performance advantage for players in controlled benchmark scenarios like the guesswhat?! game when categories at inference time match those at training time, but causes them to fail in more realistic zero-shot scenarios @xcite where players are required to generalize to out-of-domain object categories. for example, consider an agent that during training has only seen glazed donuts, associated with the fixed "donut" category embedding (cf. figure 1 ). at inference time, the model cannot ground visual representations for objects belonging to the "pasticciotto" (an italian pastry) category, since such a category was not in its repertoire. similarly, it will likely represent frosted donuts with a generic "donut" embedding, despite the perceptual differences among different types of donut. in this paper, we tackle the above limitations by introducing a novel imagination module based on regularized auto-encoders @xcite , which are able to derive imagination embeddings directly from perceptual information in the form of the object crop. our formulation of the reconstruction loss allows the model to learn context-aware and category-aware imagination embeddings. thus, removing the need for gold category labels at inference time and greatly improving zero-shot generalization. section 4.2 integrates our imagination component into the oracle model of de @xcite and the guesser model of @xcite . we show that the new imagination models are state-of-the-art in the recently introduced compguesswhat?! benchmark @xcite outperforming current models by 8.26%. it also improves the oracle's and guesser's accuracy (by 2.08% and 12.86%, respectively) in the standard guesswhat?! when no gold category labels are available. lastly, we show that imagining latent object representations greatly helps to reason about object visual properties (i.e., color, shape, etc.), qualifying our module as a generic perceptual simulation component alà @xcite . ## background: guessing games and concept representations guesswhat?! is an instance of a multi-word guessing game @xcite . every game involves two players: an oracle and a guesser conversing about a scene s (a natural image). a scene s can be abstracted into a collection of objects o, each of which is associated with a category c @xmath0, @xmath1, . . . , k}. the aim of the guesser is to identify a target object o * ∈ o by asking questions about s to the oracle. the gameplay of guesswhat?! thus comprises three tasks: i) question generation where the guesser inquires about an object in the scene s given the dialogue generated so far; ii) answer prediction, where the oracle answers @xmath2, no, n/a} given the scene s, question and the target object o * ; and iii) target prediction where the guesser selects a candidate object with the highest relevance score r(o i ). several architectural variants have been proposed to tackle guesswhat?! (cf. section 5 for some related works). in this work we adopt the recent gdse model @xcite , which learns a visually grounded dialogue state used to learn both question generation and target object prediction. as shown below, gdse does not deliver the desired multi-modality needed, therefore we extend it with our imagination component to obtain more effective multi-modal object representations. for successful gameplay, both the guesser and oracle must build representations of the scene that contain specific perceptual information of objects (object-aware), are relevant for the object category they belong to (category-aware), and are specialized to the scene in which the game is played (contextaware). as the scene s is an image, it is natural to associate each object o @xmath3, i.e., a vector v @xmath4.g. resnet-152 @xcite ) based on their bounding box. 1 however, these representations are not sufficient as they are neither context-aware nor category-aware, i.e., they ignore other objects in the scene and do not leverage their category information. gdse and other recent approaches @xcite @xcite coped with the second issue by introducing category embeddings as d c -dimensional continuous representations c @xmath5, . . . , k. once learned, a category embedding c is then concatenated to an 8-dimensional feature vector s i derived from the object bounding box (cf. de vries et al. ( 2017 )). while these embeddings partially solve category-awareness, they are not object-aware. for instance, the embedding for the object category "apple" will be the same regardless of a particular object to be a red or green apple, i.e., most likely a centroid representation of the objects seen only during training. moreover, if during training we only see red apples, at inference time, we will likely fail to detect green apples as belonging to the same category (figure 2(a) ). these issues have gone unnoticed since category embeddings usually boost performances on the original guesswhat?! task, given that gold category labels are also available at inference time. however, this boost is illusory: models relying on this symbolic information to be always available are not learning to exploit all modalities. in fact, a 20% drop in the guesser accuracy if gold category labels are not provided has been reported in @xcite for guesswhat?! and analogous poor results in more realistic benchmarks measuring zero-shot generalization such as compguesswhat?! @xcite . ## imagination module: learning context-and category-aware object representations to overcome the limitations of gdse and competitors and realize a form of perceptual simulation in a learning system, we introduce a generic component-named the imagination module-which learns latent concept representations that are both context-and category-aware, without relying on category labels at inference time. our imagination model can be understood in the context of representation learning via deep generative models @xcite which has been popularized by variational autoencoders (vaes) @xcite , and gans @xcite . specifically, we substantially extend the recently introduced regularized autoencoders (raes) framework @xcite . raes are simplified vaes where stochasticity in the encoder and decoder is dropped in favor of more stable training and more informative embedding learning. in fact, raes do not suffer from several issues known to affect vaes, such as poor convergence and the possibility of learning embeddings that are independent of the input images (cf. @xcite for a detailed discussion). more crucially for our purposes, raes do not have to compromise the informativeness of the learned embeddings with a fixed a-priori structure in the latent space that enables simple sampling (e.g., an isotropic gaussian prior). vaes which need to have such a fixed prior, instead, are deemed to learn embeddings that are less informative w.r.t. objects, categories, and context information. module architecture. figure 2(a) summarizes our imagination module. its aim is to distill a context and category-aware embedding z @xmath6. to this end, we adopt an encoder e φ parameterized by φ that maps a perceptual embedding v i of object o i to its imagined counterpart z i , i.e., e φ (v i ) = z i . a decoder d θ realizes the inverse mapping ṽi = d θ (z i ), with ṽi ∈ r d o being also called the reconstruction of the input v i . as in raes, our per-object loss l img comprises a reconstruction loss (l rec ), weighting how good the reconstructions of d θ are w.r.t. the encoded representations by e φ , and a regularization term (l reg ) enhancing generalization by smoothing the decoder d θ . this leads to the following composite loss: differently from raes, we devise a specific reconstruction loss tailored to learn contextual and category-aware representations. in conventional raes, in fact, the reconstruction loss is defined as the mean squared error (mse) representing the distance between v i and its reconstruction ṽi , so that l rae rec := mse(v i , ṽi ). this loss is purely unsupervised and as such agnostic to object categories or to the scene context. to our aims, we define a custom imagination reconstruction loss l img rec as an instance of a max-margin triplet-loss @xcite , as follows. let c i be the category of object o i with perceptual embedding v i in scene s and let o ¬c @xmath7. our per-object l img rec term is defined as: imagining at inference time. differently from the category embeddings c employed by all previous work, our imagination embeddings z do not depend on gold category labels at inference time, while still being context-aware and category-aware. in fact, once parameters φ have been learned, the encoder e φ contains all the information needed to distill embeddings z independently of l img , which is necessary only at training time. we consider imagination the ability of the model of generating latent representations on-the-fly. therefore, for both guesser and oracle models we consider an object representation for object o i that replaces c i with z i and concatenates it with its spatial information s i (see figures 2(b ) and 2(c) and appendix a.1 for details). by doing so, we consider every gameplay situated in a reference scene as an experience where our imagination module is able to derive a latent conceptual representation simply by "looking" at objects, realizing a perceptual simulator @xcite . we plan to investigate how to combine label-dependent category embeddings c with our imagination embeddings z, similarly to how some vae variants tackle semi-supervised classification scenarios @xcite . ## experimental investigation to assess the impact of using the imagination embeddings against the category embeddings, we use two evaluation benchmarks: guesswhat?! and compguesswhat?!. more information about the training procedure can be found in appendix a.2. ## related work concerning unsupervised learning of concept representations, @xcite first learn modalityspecific representations and then fuse them into a unified representation for each concept. however, they rely on hand-crafted bags of visual features, making the approach laborious to extend to new domains and games. @xcite cope with this issue by relying on cnn models to extract latent features from images for instances of specific objects. @xcite use a margin loss but in the context of maximizing the similarity between the visual representation of a noun phrase and its corresponding text representation. similarly, @xcite learn a mapping between the resnet features and the word embeddings of a concept. as discussed in section 2, unlike our imagination embeddings, these purelyperceptual representations are neither category-aware nor context-aware. @xcite present a multi-modal model that uses a denoising auto-encoder framework. unlike us, they do not use perceptual information as input but rely on an attribute-based representation derived from an additional attribute predictor. however, they do use a reconstruction loss (cross-entropy loss for attribute prediction) and an auxiliary category loss during training. their training scheme is more complex as they first separately train the ae for each modality and then fuse them, which we avoid by adopting a single end-to-end architecture. @xcite used vaes to learn grounded representations for lexical concepts. however, as discussed in section 3, vaes are not as well suited as raes to representation learning for our imagination module. in the context of guessing games, all the previous approaches rely on categories embeddings @xcite @xcite @xcite (see section 2). our imagination component can be flexibly integrated in any of them by replacing the category embeddings with imagination embeddings. ## conclusions we argued that existing models for learning grounded conceptual representations fail to learn compositional and generalizable multi-modal representations, relying instead on the use of category labels for every object in the scene both at training and inference time @xcite . to address this, we introduced a novel "imagination" module based on regularized auto-encoders, that learns a context-aware and category-aware latent embedding for every object directly from its image crop, without using category labels. we showed state-of-the-art performance in the compguesswhat?! zeroshot scenario @xcite , outperforming current models by 8.26% in gameplay accuracy while performing comparably on the other tasks to models which use category labels at training time. the imagination-based model also shows improvements of 2.08% and 12.86% in oracle and guesser accuracy. finally, we conducted an extensive error analysis and showed that imagination embeddings help to reason about object visual properties and attributes. for future work, we plan to 1) integrate category labels at training time in a more principled way following advances in semi-supervised learning @xcite ; 2) improve the multi-task learning procedure presented in @xcite to optimize at the same time multiple tasks of different complexities.