Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text (2021)

Abstract: Communicating with humans is challenging for AIs because it requires a shared understanding of the world, complex semantics (e.g., metaphors or analogies), and at times multi-modal gestures (e.g., pointing with a finger, or an arrow in a diagram). We investigate these challenges in the context of Iconary, a collaborative game of drawing and guessing based on Pictionary, that poses a novel challenge for the research community. In Iconary, a Guesser tries to identify a phrase that a Drawer is drawing by composing icons, and the Drawer iteratively revises the drawing to help the Guesser in response. This back-and-forth often uses canonical scenes, visual metaphor, or icon compositions to express challenging words, making it an ideal test for mixing language and visual/symbolic communication in AI. We propose models to play Iconary and train them on over 55,000 games between human players. Our models are skillful players and are able to employ world knowledge in language models to play with words unseen during training.

## introduction communicating with humans is a long-standing goal in ai, and has been studied in the context of natural language for decades. many of the key challenges in this task, such as using a shared understanding of the world, commonsense reasoning, and metaphor are, however, not language-specific, but are instead general-purpose tools that humans use when communicating through other modalities as well. for example, understanding what means in a text conversation requires grasping metaphor (it is unlikely to be literally suggesting one should put on a party hat), or understanding a sign with a truck swerving requires common-sense reasoning (the intent is to show slippery conditions, not to suggest drivers ought to begin swerving themselves). humans can easily adapt to these different modalities, as well as use visual/symbolic tools (e.g., pointing with a finger, or an arrow in a diagram) that cannot be used in a text-only context. to build and test ais for this skill, we introduce the first task and large-scale dataset for multimodal communication by creating iconary, a game of drawing and guessing based on pictionary, along with a dataset of games with human players, proposing automatic and online game playing metrics, and constructing proficient iconary ais. in iconary, one player (the drawer) draws an image for a phrase by arranging icons (including the ability to rotate or change the sizes of icons) on a canvas, and a second player (the guesser) guesses what phrase the drawing represents. we use icons so we can focus on the high-level semantics of the drawings, and to make the game easier to play online. the guesser then makes a series of attempts to guess the phrase using only the drawing. if the guesser is unsuccessful, the drawer can revise the drawing, and the cycle repeats until time runs out or the guesser is successful. figure 1 shows an example of an iconary game, played between a human player and our ai player. iconary combines several key comprehension challenges. first, non-literal imagery, since most words in our dataset do not have directly corresponding icons so players will often use visual metaphor (e.g., a school bus and book for 'textbook') or reference canonical examples (e.g., lit and unlit light for 'turning off') to convey words. second, visual similarity, since icons can also be composed to draw objects, such as using concentric circles to draw a dartboard. third, annotations, because drawers often use arrows, circles, or crosses to indicate motion or to guide the interpretation of the image. fourth, state tracking, because apprentice cook weighing food child cook weighing food adult cook weighing food chief cook weighing food male cook weighing food young cook weighing food chef cook weighing food man in kitchen chopping person cooking with knives figure 1 : examples of gameplay between human players and our models. snapshots show the progression (left to right) of two games, with the human player guessing in the top row and drawing in the bottom. guesses in each round are shown beneath the drawing for that round, and are color-coded (@xmath0, @xmath1. the first game shows tdrawer drawing 'origami' with a sushi icon (presumably to indicate japan), a turning icon and finally a paper icon once the human has guessed 'folds'. the second game shows tguesser correctly guessing 'apprentice' by interpreting the icons for baby, adult and knife. the words 'origami' and 'apprentice' do not appear in the training data for either model. see the appendix for more qualitative results. we present a large dataset for iconary by having human players play with each other -a collection of 56k games in train, in-domain (ind) dev and test sets with 5k games, and out-of-domain (ood) dev and test sets with 1k and 3k games respectively that contain words not seen during training. our proposed models, tdrawer and tguesser, leverage world knowledge in the t5 @xcite pre-trained language model and have been carefully adapted to draw and guess words not observed during training. we measure performance using automated metrics, but our main results are shown by having our ais play games with human players. tdrawer and tguesser perform remarkably well on the ind sets (68.3% and 96.0% win rates), but are also able to play impressively with human players on the ood sets (41.7% and 62.9% win rates), demonstrating their ability to extract and integrate world knowledge for unseen game-play words from language models. figure 1 shows some interesting games played by our models with human partners on the ood set. while our models are capable players, skilled human players outperform them on the ood sets (a smaller margin of 4.6% at guessing but a sizeable margin of 21.0% at drawing). an error analysis shows that most errors occur for unseen words, particularly verbs, compound words, and examples with complex drawings, such as those requiring fine-grained positional information. our quantitative and qualitative analysis suggests ample room for future research in this new, rich and complex domain. ## models we propose tguesser and tdrawer to play iconary. both models condition on the current game state, meaning the previous drawings, guesses and, for tdrawer, the game phrase, and then generate either text to guess the phrase (for tguesser), or a sequence of special tokens that encode a drawing (for tdrawer). although this involves a visual modality, we propose to use language models for this task because (1) the icon names can be used to understand the drawing and (2) iconary often requires using word knowledge (e.g., mapping person and thumb icons to 'hitchhiking' or milk and ice cream icons to 'milkshake') that is known to be captured by these models @xcite . to do this, we encode the game state as text and apply the t5 (raffel drawing phrase meteor destroying an observatory previous guesses telescope watching an asteroid comet destroying an estate game state drawing: huge comet, 2 smoke, telescope, huge dome. phrase: _ destroying an _. meteor destroying an observatory. phrase encoding fill-in-the-blank encoding drawing: huge comet, 2 smoke, telescope, huge dome. phrase: <@xmath2. ## experimental setup in this section, we specify our metrics and baselines. we use t5-3b for tguesser, but t5-large for tdrawer since it generates longer sequences and therefore uses more memory. other hyperparameters and training details are in the appendix. ## related work there is a long history of using games as a testbed for ai. traditionally these have been adversarial strategy games like chess @xcite , go @xcite , and many others @xcite @xcite a few cooperative games have been studied, like codenames @xcite or hanabi @xcite , that are similar to iconary in that they require players to communicate in order to achieve a shared goal. however, those games severely limit means of communication, whereas iconary allows a rich variety of communication strategies through the use of drawings, and contains language beyond single words. relating text to visual imagery has also been studied in many forms @xcite . generating text that describes visual input, as done in iconary, has been studied in visual dialog @xcite , image captioning @xcite , and describing videos @xcite . training models to produce images from text has been studied for captions @xcite , image specifications @xcite , and dialogue @xcite . unlike in these works, the drawings in iconary are not photographic and constructed to communicate a phrase. as a result, they can be non-literal and deictic, which makes understanding them a significantly different challenge. using a pre-trained language model to understand mixed language and visual input has been considered by @xcite , who use features produced by object detectors or other visual understanding systems as input to gpt-2 @xcite to generate natural language rationales. scialom et al. (2020) also show bert @xcite can be trained for visual question generation @xcite . we also find combining high-level visual features with a pre-trained language model is an effective way to generate visually relevant text, although again our focus is on drawings rather than photographs. figurative text is well studied @xcite @xcite , but non-literal imagery has mostly only been explored in the context of parsing charts or diagrams. this includes food webs @xcite , science diagrams @xcite , charts @xcite or for geometry problems @xcite . while this can involve related skills like understanding arrows or using icons to represent concepts, diagrams are usually used to convey technical information and therefore are unlikely to use things like visual metaphor, scenes, or icon compositions to signal words. the back-and-forth of iconary follows a dialogue structure where the guesser is seeking information from the drawer. a similar format can be found in dialogue qa datasets @xcite @xcite , and task-oriented dialogue in general similarly requires understanding the intent of a human communicator @xcite . iconary, however, makes this a multimodal process. ## conclusion we have presented the game iconary, a large dataset of human/human games, and our proposed iconary models. this represents the first test for complex multimodal communication between humans and ais, and is left as an open challenge to the community.