Probing Cross-Modal Representations in Multi-Step Relational Reasoning (2021)

Abstract: We investigate the representations learned by vision and language models in tasks that require relational reasoning. Focusing on the problem of assessing the relative size of objects in abstract visual contexts, we analyse both one-step and two-step reasoning. For the latter, we construct a new dataset of three-image scenes and define a task that requires reasoning at the level of the individual images and across images in a scene. We probe the learned model representations using diagnostic classifiers. Our experiments show that pretrained multimodal transformer-based architectures can perform higher-level relational reasoning, and are able to learn representations for novel tasks and data that are very different from what was seen in pretraining.

## introduction intelligence is classically described as "the ability to see the similarities among dissimilar things and the dissimilarities among similar things" (thomas acquinas, 1225-1274, reported by @xcite . developing systems that can reason over objects and their relations is indeed a long-standing goal of artificial intelligence research, as argued by @xcite . in recent years, huge progress toward this goal has been made in the language and vision community. starting from @xcite and @xcite , a wealth of studies have focused on language-driven visual reasoning, namely the problem of reasoning about an image given some linguistic input. generally speaking, there are two main types of problems in visual reasoning datasets (see @xcite : non-relational, requiring models to focus only on a given object (e.g., answering the question "what material is the cube made of?"), and relational, requiring models to pay attention to several or even all the objects in the image (e.g., indi-cating whether the statement "there are four cubes that are red" is true or false). relational problems call for higher-level abilities, such as counting or directly comparing objects, both of which involve recognising the (dis)similarities among things. in this paper, we focus on an important but understudied, relational reasoning task: assessing the relative size of objects in visual contexts, that is, determining whether an object counts as 'big' or 'small' in an image. we define a multi-step relational reasoning problem formulated as a sentence verification task. we construct a dataset of three-image scenes where a given target object, e.g., a blue triangle, is present in each image: two images have target objects with the same contextually-defined size and one image stands out in this regard. the task requires verifying whether a simple natural language statement standing for a first-order logical form describes a scene, e.g., "there is exactly one blue triangle that is small in its image in this scene" (figure 1 ). such multi-step relational reasoning is at play in many real-life situations: e.g., the same exact pan may count as 'big' in all contexts except a restaurant kitchen. we experiment with two types of models to solve this task: a modular neural network @xcite and lxmert, a pre-trained multimodal transformer @xcite . we probe the learned representations of lxmert to assess whether, and to what extent, it has learned the underlying structure of the data. by means of two experiments with probing classifiers @xcite @xcite , we first verify that it is able to perform the task at the image level (i.e., to compute the relative size of the target object at the image level); then, we test its ability to reason at the multi-image level and detect the image that stands out. the experiments show that lxmert is able to solve the multi-step relational reasoning task there is exactly one blue triangle that is small in its image in this scene there are exactly two blue triangles that are small in their images in this scene there are exactly two blue triangles that are big in their images in this scene with an accuracy of 88.8%, and that the majority of errors occur when the relative size of the target object is difficult to determine. our analyses show that (i) in most cases, different attention heads in lxmert specialise to localising the smallest and biggest objects in the images, (ii) that the cross-modal representations learned appear encode a threshold function that controls whether an object is 'big' or 'small' in an image, and (iii) that a simple diagnostic classifier successfully identifies the instance that stands out in a three-image scene. taken together, these findings lend further support to the advanced reasoning abilities of pretrained transformer-based architectures, showing that they can perform higher-level relational reasoning and are able to deal with novel tasks and novel data, including synthetic data not available during pre-training. foot_0 ## problem formulation we investigate multi-step relational reasoning by formulating the problem as a visually grounded sentence verification task (see figure 1 ). given a pair scene, statement consisting of a visual scene and a statement about such scene, the task consists in classifying the statement as either true or false. in our setup, a scene consists of 3 images: img 1 , img 2 , img 3 , each including an instance of the target object (e.g., a blue triangle) together with other geometrical shapes of the same type (e.g., triangles of other colours). a statement paired with a scene is of the following form: "there is exactly one blue triangle that is small in its image in this scene" or "there are exactly two blue triangles that are big in their im-ages in this scene". as we will explain in detail in sec. 4.1, the dataset is created such that the target object counts as either 'big' or 'small' in only one of the three images in a scene. arguably, solving the task requires the following two steps of relational reasoning: (1) identifying whether the target object counts as either 'big' or 'small' in each image, and (2) counting how many images include a big/small target. however, in our setup there is no direct supervision for any of these steps. in other words, the training data does not indicate which images contain an object that counts as big/small nor explicitly how many images contain a big/small target. ## results in @xcite , models were shown to make more errors when the area of the queried object is closer to the threshold (see eq. 1). 8 this is identical to the approach followed by @xcite to finetune lxmert for nlvr2 classification. we check if this is the case also for lxmert on our 3pos1 task. to do so, we consider the cases where the model gives a wrong prediction. among the 3 images in a scene, we take the one with the lowest distance from the threshold. we then check whether the model makes more errors when such distance is lower, i.e., when there is at least one image in the scene with a borderline size. as reported in ## analysis at the individual image level our results show that lxmert achieves a high level of accuracy on our visually-grounded sentence verification task on the three-image 3pos1 ## analysis at the multi-image level in the previous section, we analysed the model representations at the level of the independent images. here, we probe the representations with respect to the entire three-image scene. first, we investigate whether the representations encode information on the overall configuration of the scene (sec. 7.1). second, we probe their effectiveness in identifying the odd-one-out image in the scene (sec. 7.2). in both analyses, we use diagnostic classifiers, foot_9 that take as input the concatenation of the three imagestatement cross-modal representations (eqn. 3). ## conclusion we performed an in-depth analysis of the representations learned by the pretrained multimodal transformer lxmert when performing relational reasoning. we proposed a multimodal reasoning task that requires multi-step relational reasoning and showed that lxmert can perform the task with high accuracy. our analysis reveals that the lts, single gpu tesla v100-sxm2, and nvidia driver 455.38, cuda 10.1, and 24gb ram.