Abstract Visual Reasoning with Tangram Shapes (2022)

Abstract: We introduce KiloGram, a resource for studying abstract visual reasoning in humans and machines. Drawing on the history of tangram puzzles as stimuli in cognitive science, we build a richly annotated dataset that, with >1k distinct stimuli, is orders of magnitude larger and more diverse than prior resources. It is both visually and linguistically richer, moving beyond whole shape descriptions to include segmentation maps and part labels. We use this resource to evaluate the abstract visual reasoning capacities of recent multi-modal models. We observe that pre-trained weights demonstrate limited abstract reasoning, which dramatically improves with fine-tuning. We also observe that explicitly describing parts aids abstract reasoning for both humans and models, especially when jointly encoding the linguistic and visual inputs.

## introduction reference is a core function of natural language that relies on shared conventions and visual concepts. for example, in english, a speaker may use the term dog to refer to a particular animal of the species canis familiaris, or, through abstraction, to an object with a less strongly conventionalized name, such as the shape at the top of figure 1 . a speaker might refer to such a shape as looking like a dog, and even point to its parts, like its head and tail, despite having few visual features in common with the ordinary referent. comprehension and generation of references are critical for systems to engage in natural language interaction, and have been studied extensively with focus on ordinary references (e.g., @xcite @xcite @xcite , in contrast to the visual abstraction illustrated in figure 1 . each annotation includes a wholeshape description (bold), segmentation to parts (in color), and naming of parts (linked to each part). the top example shows low variability with near-perfect agreement, while the bottom shows high variability with divergence of language and segmentation. we address this gap by adopting an influential paradigm for probing human coordination in the cognitive science literature: reference games with abstract tangram shapes (e.g. @xcite @xcite . unlike photographs of natural objects, where there is often a single canonical label, tangrams are fundamentally ambiguous. while some shapes fall under strong existing conventions and elicit consensus about appropriate names (e.g., figure 1 , top), others are characterized by weaker conventions (e.g., figure 1 , bottom) and every speaker may arrive at a distinct but valid description @xcite . while such diversity is a key consideration motivating their use as stimuli, existing behavioral studies have typically been limited to a relatively small set of 10-20 shapes, highly restricting the overall diversity of the stimulus class. it also limits their applicability for training and analyzing vision and language models, where significantly more data is necessary. in this paper, we significantly expand this resource. we introduce kilogram, 1 a large collec-tion of tangrams with rich language annotations. kilogram dramatically improves on existing resources along two dimensions. first, we curate and digitize 1,016 shapes, creating a set that is two orders of magnitude larger than collections used in existing work. this set dramatically increases coverage over the full range of naming variability, providing a more comprehensive view of human naming behavior. second, rather than treating each tangram as a single whole shape, our images are vector graphics constructed from the original component puzzle pieces. this decomposition enables reasoning about both whole shapes and their parts. we use this new collection of digitized tangram shapes to collect a large dataset of textual descriptions, reflecting a high diversity of naming behaviors. while existing work has focused on naming the complete shape, we also ask participants to segment and name semantically meaningful parts. we use crowdsourcing to scale our annotation process, collecting multiple annotations for each shape, thereby representing the distribution of annotations it elicits, rather than a single sample. in total, we collect 13,404 annotations, each describing a complete object and its segmented parts. the potential of kilogram is broad. for example, it enables the data-driven scaling of studies of human interactions and models of whole-part reasoning in language and vision models. in this paper, we use kilogram to evaluate the visual reasoning capacities of recent pre-trained multi-modal models, focusing on generalizing concepts to abstract shapes. we observe limited generalization of this type in pre-trained models, but significant improvements following fine-tuning with our data. we also see how explicitly referring to and visualizing parts can help reference resolution. data and code, as well as a data viewer are available at: https://lil.nlp.cornell.edu/kilogram/ . ## background and related work abstract or ambiguous visual stimuli have been widely used to investigate how human partners coordinate when talking about things in the absence of strong naming conventions going back to @xcite . tangrams as stimuli were introduced by @xcite . these shapes are all built from the same seven primitives, but elicit a wide range of figurative descriptions that conceptualize shapes in different ways @xcite @xcite @xcite @xcite @xcite . it has been observed that some shapes are easier or harder to describe @xcite @xcite , a property known as nameability or codability, which has also been studied with non-tangram shapes (e.g., line drawings; snodgrass and @xcite @xcite . even though diversity is a key consideration in working with tangrams, existing stimuli sets are relatively small, limiting their usefulness as nlp benchmarks, where scale is critical. even the largest studies of variability in naming (e.g., @xcite have used a relatively small set of 60 tangrams. @xcite present a resource that is related and complementary to ours, including 332 png-formatted tangrams with whole-shape naming annotations in french. contemporary pre-trained vision and language approaches can be categorized along an axis characterizing how they encode the data, from jointly encoding the two inputs @xcite @xcite to encoding them separately @xcite . joint encoding aims to capture tighter interaction between the input modalities compared to separate encoding, but is generally more computationally expensive, and can only operate on multi-modal input. we study recent models on both ends: vilt @xcite for joint encoding and clip @xcite for separate encoding. these models are typically evaluated on image captioning (e.g., @xcite or visual question answering (e.g., @xcite benchmarks. several benchmarks, such as nlvr @xcite and winoground @xcite , aim for more focused evaluations with a focus on compositionality. we build on these efforts, but target generalization through abstraction using visually ambiguous stimuli. this is inspired by the role of abstraction in human cognition. abstraction is a key step in human perception @xcite that is critical for generalization @xcite @xcite , and forms the shared foundation on which human language communication is layered @xcite @xcite . our focus on part dehttp://annotate.edu/tangram/parts this part looks like composition is aligned with how part identification plays an important role in human abstraction (tversky and hemenway, 1984). ## data collection we scan a large set of tangram puzzles to vector graphics, and crowdsource annotations of natural language descriptions and part segmentations. ## data analysis the language and concepts annotators use reflect varying degrees of consensus around conventions for describing the appearance of shapes and their parts. for analysis, we preprocess the annotations by lowercasing, tokenizing, lemmatizing, and removing stop words using nltk @xcite . we use the larger full set for our analyses (section 3.3), unless otherwise noted. for a broad overview of the types of concepts evoked, we manually tag 250 randomly sampled annotations: 30.8% use human-like concepts (e.g., dancer), 31.2% animate but non-human concepts (e.g., dog), and 38.0% non-animate concepts (e.g., house). we examine how part words differ across whole-shape concepts by extracting head words from whole-shape and part descriptions. figure 3 shows the distribution of part head words for each of 272 whole-shape head words @xmath0, ranked in order of frequency. figure a.2 in the appendix illustrates how the most common part word head is used in different tangrams. a central problem of visual abstraction is the degree of ambiguity or subjectivity that a shape evokes across different people @xcite : some descriptions have higher consensus than others. we define three measures of variability along different dimensions: shape naming divergence (snd), part naming divergence (pnd), and part segmentation agreement (psa). ## visual reasoning with tangrams we use kilogram to evaluate the reasoning of clip @xcite and vilt @xcite through a reference game task, where the model is given a textual description and selects the corresponding image from a set of images. formally, given a textual description x and a set of k images @xmath1, . . . , i k }, the task is to select the image i @xmath2. we cast the task as computing a similarity score f (x, i i ) between the description x and an image i i . we select the corresponding image as i * = arg max i @xmath3, i i ). ## discussion kilogram provides a new window into the visual abstraction capacity of grounded language models and their ability to generalize concepts beyond their photographic appearance, an integral component of human concept representations @xcite . our experiments show that there is significant room to improve pre-trained models, which should be able to perform zero-shot reference game tasks without fine-tuning as well as humans do @xcite . the improved performance after fine-tuning indicates the multi-modal architecture itself has the potential for higher performance, which current pre-training regimes likely do not support. in particular, vilt's improved performance as a function of additional part information suggests that more structured concept alignment may play a role in this effort (e.g., between parts expressed as lexical items and the corresponding elements of the image). while we focused on the task of reference resolution, kilogram is also well-suited for production tasks (e.g., generating human-like distributions of descriptions or coloring named parts on a blank tangram) as well as instruction-following tasks (e.g., placing pieces in the described configuration to reconstruct a tangram). more broadly, our data emphasizes the need for maintaining well-calibrated distributions over the many different possible ways that people may conceptualize or talk about things, rather than collapsing to a "best" prediction. ## limitations although randomly constructed reference games provide an interpretable evaluation metric, they also pose several limitations. performance is limited by the fact that descriptions were elicited for isolated images. these descriptions do not reflect the kind of pragmatic reasoning commonly deployed by human speakers in reference games to resolve ambiguities @xcite . in other words, annotators were not able to anticipate the necessary level of detail to disambiguate the object from a specific context of distractors, hence the descriptions may be underinformative. randomly generated reference games may include ambiguities that make them impossible to solve (e.g., two objects that could both plausibly be described as a bird). the possible performance ceiling on these games is likely below 100%. extending the data through interactive reference games is an important direction for future work. likewise, our studies of baseline human performance on this task are preliminary. we found that participants clustered into higher-and lower-performing groups, likely reflecting attentional and motivational factors (e.g., some participants may not have fully attended to the provided part information). a better understanding of human behavior is critical before making any clear conclusions comparing humans and model performance. ultimately, models only outperformed mean human performance significantly only after fine-tuning on approximately 6,600 example reference games. our resource contribution and analysis are focused on english. while the data collection design does not make language-specific assumptions, it depends on the availability of proficient speakers, which is limited in contemporary crowdsourcing services for certain languages. our large collection of visual stimuli is well suited to extend our data collection to other languages and cultures, which may display different abstractions. this is an important direction for future work. extending our analysis to other languages depends on the availability of pre-trained models in these languages, which may be limited by the availability of aligned language vision data and the computational resources required for pre-training. prior work. we sample the 62 tangrams from full to represent the diversity of tangrams using the first set of annotations we collect. we plot the annotated tangrams by average log perplexity of whole-shape descriptions with 1 100 smoothing and psa and apply a 5 × 5 grid to the plot @xcite . using perplexity and psa allows us to sample a set of tangrams with diverse degrees of annotation and segmentation agreement. with a relatively high smoothing factor, we are able to spread out the data points, because the majority of the data set has high divergence in descriptions. we randomly pick 12 periphery points to collect more annotations for outliers, uniformly sample 25 from all the 1004 tangrams, and randomly sample 25, one from each grid, to represent the entire distribution. we calculate average log perplexity of wholeshape annotations for each tangram. let x(1) , . . . , x(n) be annotations for a tangram, where each annotation is a sequence of tokens x(j) = ⟨x 1 , . . . , x m (j) ⟩ of length m (j) . we create a language model p (j) for every annotation x(j) using all other n -1 annotations for the tangram: