In-Context Analogical Reasoning with Pre-Trained Language Models (2023)

Abstract: Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences. While it is thought to be essential for robust reasoning in AI systems, conventional approaches require significant training and/or hard-coding of domain knowledge to be applied to benchmark tasks. Inspired by cognitive science research that has found connections between human language and analogy-making, we explore the use of intuitive language-based abstractions to support analogy in AI systems. Specifically, we apply large pre-trained language models (PLMs) to visual Raven’s Progressive Matrices (RPM), a common relational reasoning test. By simply encoding the perceptual features of the problem into language form, we find that PLMs exhibit a striking capacity for zero-shot relational reasoning, exceeding human performance and nearing supervised vision-based methods. We explore different encodings that vary the level of abstraction over task features, finding that higher-level abstractions further strengthen PLMs’ analogical reasoning. Our detailed analysis reveals insights on the role of model complexity, in-context learning, and prior knowledge in solving RPM tasks.

## introduction humans are constantly presented with novel problems and circumstances. rather than understand them in isolation, we try to connect them with past experiences. with any luck, we might find an analogy: a mapping between relevant aspects of this new situation and a past situation, which helps form abstractions that allow us to reason more effectively in the future @xcite . analogy is thought to underpin humans' robust reasoning and problem solving capabilities (hofstadter and @xcite ) are an analogy-making task where one must infer the missing matrix item based on abstract rules instantiated in the first two rows. to demonstrate the potential analogical reasoning skills in pre-trained language models, we develop languagebased abstractions over their key perceptual features, then prompt them to select the completion of the matrix. sander, 2013), and thus it is believed to be prerequisite in order to enable the same in ai systems. however, conventional approaches struggle with analogy-making, and are trained on thousands of examples to achieve any success on benchmark tasks. this is unsatisfying, as humans are capable of analogy-making without explicit training, and such analogy-making should enable zero-shot generalization to new situations @xcite . interestingly, a body of work in cognitive science suggests that analogy-making and relational reasoning are connected to humans' symbol system and language capabilities @xcite . for example, @xcite finds that members of an amazonian tribe that count only with words for "one," "two," and "many" struggle to make analo-gies with higher numbers. further, @xcite find that deaf children whose sign language does not involve spatial relations are outperformed by hearing children on a spatial relational reasoning task, while @xcite find that assigning even nonsensical names to relations enhances children's relational reasoning. all of this demonstrates that language serves as a powerful way for humans to abstract and better reason about the overwhelming and complex percepts we encounter in the world. in this work, we explore whether language may serve a similar purpose in ai systems. specifically, we apply contemporary autoregressive pre-trained language models (plms) to raven's progressive matrices (rpm), an example of which is shown in figure 1 . rpm is a widely used psychometric test for relational reasoning that requires inducing an abstract rule from just two examples of short sequences of groups of shapes, and then applying the rule to complete a new partial sequence @xcite . this task makes minimal assumptions about the test taker's prior knowledge, and is thus thought to provide a good estimate for general intelligence @xcite . on the raven dataset @xcite , we find that given the ability to perceive key features of rpms, large plms exhibit a surprising capacity for zero-shot relational reasoning, approaching that of supervised vision-based deep learning approaches and even humans. we propose three levels of abstraction over the language features of the task using name assignment and task decomposition, and find that each abstraction further strengthens plms' relational reasoning. our results and detailed analysis offer insights on plm performance, including the role of models' complexity, in-context learning, and prior knowledge in emergent relational reasoning, and suggest that they could play an important role in future cognitive architectures for analogy-making. 2 2 related work past work has studied analogy in ai across various domains. @xcite provides a comprehensive overview of these efforts, especially those applied in idealized symbolic domains. here, symbolic and probabilistic methods have traditionally been applied @xcite @xcite . however, these approaches typically require hard-coding domainspecific concepts, and require substantial search through domain knowledge to operate on their target problems, thus making them unscalable. the creation of large-scale image datasets for analogy tasks here @xcite @xcite have enabled further research with deep learning and neuro-symbolic methods @xcite @xcite , which bring the advantage of requiring less ad-hoc encoding of domain knowledge, but require thousands of training examples to learn the tasks, still limiting their generalization capability. other work has explored ai systems' analogymaking in real-world domains, including in natural images @xcite and language @xcite @xcite , especially lexical analogies @xcite @xcite @xcite . however, these domains make it difficult to control the prior knowledge required to solve tasks @xcite , and in the context of recent generative foundation models that are extensively pre-trained on natural data, it becomes difficult to separate analogy learning from distributional patterns that can be overfit. unlike prior work, we apply such foundation models for language to analogical reasoning in a zero-shot setting, bypassing the requirement of hard-coding domain knowledge or training models on task-specific data. furthermore, while contemporaneous work has applied plms to a variety of simpler relational reasoning tasks in language @xcite , we systematically explore the advantage of using language to abstract over complex visual features of the task, opening questions about how the powerful symbol systems learned in plms may support robust, perception-driven reasoning in future ai systems. ## raven's progressive matrices raven's progressive matrices (rpm) are abstract relational reasoning tasks used in cognitive psychology to test humans' analogy-making @xcite . each instance of rpm is a matrix consisting of 9 items arranged in a square, the last of which must be selected from a set of choices. each item consists of several perceptual attributes, such as shape, color, or more abstract features. within each row of the matrix, a relation is applied over these attributes, such as progression of numerical values associated with these attributes. given the first two rows of the matrix, the challenge of the task is to identify the relations being applied to items, and apply them analogously in the third row to infer the missing ninth item. successfully solving an rpm requires tackling two sub-problems: perception of each item's attributes, and reasoning over multiple items' attributes to infer and apply relations. ## methods in order to apply plms to raven, we abstract the visual features of the task into language. our abstractions are intentionally applied on a per-item basis to tackle the perception problem of the task without giving the plm explicit hints toward the reasoning problem (which requires capturing patterns over multiple items). this allows us to focus on evaluating the reasoning capabilities of plms. 3 first, we introduce our multi-level abstractions for the raven dataset. 4 then we formally define the interface between plms and the rpm task. ## experimental results now, we can examine the impact each of these language-based abstractions has on the performance of transformer-based, autoregressive plms in relational reasoning on raven. to further understand their impact with respect to model complexity, we evaluate a range of model sizes: foot_6 opt 125m, 1.3b, and 13b @xcite , along with gpt-3 @xcite . 9 models are evaluated on a random subset of 500 testing examples from each sub-task of raven. after introducing some comparison approaches, we present the experimental results from our applied abstractions on plms' entity-level, layoutlevel, and component-level relational reasoning. afterward, we dive deeper with an analysis on how both our abstractions and in-context learning contribute to model performance. ## conclusion in this work, we explored the ability of large plms to perform zero-shot analogical reasoning in visual raven's progressive matrices (rpm). upon the simplest mapping to language, they can achieve striking results, while applying higher-level naming and decomposition abstractions over the task features further raises performance to the level of humans and supervised approaches in some cases. we find that while ordinal naming abstractions are a powerful way to enable analogical reasoning in larger plms, decomposition abstractions that break the task down into atomic parts conserve their working memory such that even smaller plms under 1b parameters can achieve competitive performance on this challenging problem. our detailed analysis revealed insights about which features of the task plms best capture, their robustness to distracting features, and the role of in-context learning and prior knowledge in picking up this complex task. surprisingly, we find that even without two complete rows of prior context from the matrix, gpt-3 175b and smaller models can achieve above-chance performance on the task, raising questions about the objectivity and true role of prior knowledge in rpm tasks, which are assumed to require minimal prior knowledge. these results also raise some questions about the role plms may play in future ai systems capable of analogy. while previously thought to be a difficult problem for ai systems, plms can solve the reasoning step of analogy easily given strong abstractions over visual perception. many of these abstractions are intuitive and commonly researched in computer vision, including the detection of object types, sizes, colors, counts, and global arrangements. as such, future work may dive deeper into the challenging problem of generalized perception across domains, where we must robustly tease apart the key features of tasks and experiences that may facilitate analogy-making, e.g., in recognizing the commonalities between a physical bridge and the bridge of a song @xcite . recent efforts toward understanding how humans describe abstract visual features in language by mapping them to natural concepts foot_10 are a promising direction toward this goal @xcite .