What kinds of errors do reference resolution models make and what can we learn from them? (2022)

Abstract: Referring resolution is the task of identifying the referent of a natural language expression, for example “the woman behind the other woman getting a massage”. In this paper we investigate which are the kinds of referring expressions on which current transformer based models fail. Motivated by this analysis we identify the weakening of the spatial natural constraints as one of its causes and propose a model that aims to restore it. We evaluate our proposed model on different datasets for the task showing improved performance on the most challenging kinds of referring expressions. Finally we present a thorough analysis of the kinds errors that are improved by the new model and those that are not and remain future challenges for the task.

## introduction in the context of vision and language modelling, reference resolution can be understood as the task of identifying a region in an image referred by a natural language expression. this task is also known as referring expression comprehension (rec). it is closely related to the visual grounding problem in the sense that in both cases the goal is to identify the parts of the image that support or "ground" a given linguistic expression into the real world. the difference lies in that the referred region in rec is expected to be unique, while there may be multiple support regions in the case of visual grounding, e.g. the expression "a person" might refer to multiple instances of the person class, all of which can be seen as grounding candidates for the natural language phrase. also note that, in the case of rec, the referred region may correspond to an actual physical object (or groups of objects) or to an abstract visual element that can be perceptually grouped into a meaningful entity, e.g. "the patch of grass at the bottom". rec aims at identifying a specific object or region unambiguously @xcite @xcite . although the use of natural language queries to guide localization @xcite has been explored in the computer vision literature in the past, it has mainly focused on the use of rather simple expressions involving class names and intrinsic visual attributes (e.g. colors). humans use more complex expressions that include spatial and order relations, relative attributes, meronimy, etc. compared to other recognition problems, rec goes beyond simple visuallinguistic matching and requires some (primitive, implicit) form of visual-linguistic reasoning, e.g. an expression like "the person to the left of the tree" requires that in order to locate the target object ("the person") one has to look at its relative position ("to the left of") with respect to a different element in the scene ("the tree"). rec and visual grounding are particularly relevant to other visuallinguistic problems like visual question answering @xcite and visual dialog @xcite , where being able to link different linguistic elements (words, phrases and syntactic relations) in a sentence to actual regions in an image helps establishing a "common ground" between the agents that take part in the communication process @xcite . our research question is what kinds of errors do reference resolution models make and what can we learn from them?. this paper makes the following contributions 1 . the paper is structured as follows. in section 2 we discuss related work from both psycholinguistics and machine learning, section 3 digs into the particularities of the rec and visual grounding problems and analyzes the types of errors made by current models for the rec task. in section 4 we propose a model that takes the observed errors into account and section 5 evaluates the new model on different datasets. section 6 presents a meticulous analysis of the errors that are improved by the new model and those that are not and remain future challenges. ## related work the automatic processing of referring expressions (res) has been studied for a long time @xcite . back then their semantic representation was a logical form and res were classified into intrinsic (e.g. "the big red car") and relational (e.g. "the car by the pedestrian") corresponding to unary and binary logical predicates, respectively @xcite . in the psycholinguistics community, reference were studied as a collaborative process and focused on the construction of shared knowledge @xcite through the use of language. @xcite proposed a theory in which an speaker 1) introduces a referent, 2) collaborates with the hearer to locate the referred object in some shared set of objects and 3) refers to the totality of objects that satisfy the re. viewed as a collaborative process, reference becomes a fundamental phenomena underlying all kinds of grounded dialogs @xcite . res are a linguistically rich constructs. phenomena like overspecification, that is when an re has more attributes than actually needed, has been shown to help identification when the redundant attribute is easy to recognize @xcite . vagueness and the use of gradable attributes emphasize the context dependance when using res @xcite ); something called "big" in one scene may be seen as "small" in another. res with syntactic ambiguities were also shown to be used in human dialogs @xcite . @xcite showed that even with fairly simple scenes human speakers frequently use relational descriptions to identify objects. in this paper we adapt the classification of referring expressions (res) proposed in @xcite to the domain of 2d photos of the world. krahmer and van deemter typification includes the types: instrinsic (that they call unary predicates), relational (that they call binary predicates), set (that refer to a group of objects), and gradable (that we described above). intrinsic and relational res differ in the number of objects that are involved in the description. intrinsic res only involve the referent while relational res use one or more additional objects to identify the target. referring expressions that identify sets use properties that are shared by the elements in the set. for example the referents of "the white cats" are all cats and white. in general references to set use plural definite descriptions to identify them. the datasets used in this paper are supposed to refer to a single referent and not to a set of referents so we restrict our classification in this paper to singular res. finally, krahmer and van deemter last type corresponds to gradable res. res referring to objects in 2d photos of the world frequently use spatial properties that are gradable. for example, the attribute to the right of is gradable in that "the empty sky to the right of the statue" might refer to the sky touching the statue on the right or the sky further to the right. in this paper we restrict our analysis of gradable properties to spatial properties. we agree with @xcite that careful analysis of datasets and proposed models is crucial to make progress in rec. they performed an analysis of rec models by modifying or completely removing the res and showed that the models could exploit biases on particular datasets to achieve competitive performance. in our work we do a disaggregated performance analysis and meticulous error analysis and not only rely on automatic performance metrics. the introduction of the transformer architecture by @xcite enabled interesting and novel ways of fusing visual and linguistic information @xcite @xcite . besides differences on the way both modalities are merged (single-vs two-branch models, crossmodal attention layer design, etc.), these models also differ on the tasks they are (pre-)trained for, ranging from masked token prediction and text-image matching @xcite to more elaborated strategies such as label-region alignments @xcite or scene-graph prediction @xcite . most of these tasks can be seen as local (word-to-region) or global (text-to-image) matching tasks and, although effective for pre-training, they do not contemplate other phenomena such as composition and indirection that can be observed in grounding problems. although large-scale pre-training has been shown to be effective for grounding @xcite , such methods require the availability of large collections of image-text pairs with explicit alignments between image regions and phrases in text. because of the scale of such data and the cost of extensive annotations, these approaches have not gone beyond image-text matching phenomena. in this work, we take inspiration from a recently proposed family of transformer-based architectures that tackle the rec problem as a regression problem, i.e. given an image and a query expression, directly predict the bounding box coordinates of the object or image region referred by it @xcite . ## problem definition and motivation given an image and a natural language expression, the goal of rec is to predict the location of the referred object or region, e.g. by predicting the coordinates of the bounding box that encloses it more tightly. the expression may include diverse linguistic constructs such as ellipsis, prepositional phrases, conjunctions, etc. in what follows, we first introduce a simplified yet performant version of the model proposed by @xcite . this model will serve as a strong baseline in our experiments because it allow us to evaluate the differences in performance between the baseline and the proposed extensions. next, we propose a linguistically motivated classification scheme which allows for a disaggregated analysis that will guide the rest of the paper. finally, we discuss some motivating results in light of our baseline and different types of expressions. ## restoring referential spatiality the types of expressions that have the worst accuracy, relational and ordinal, can be associated to limitations of the model in capturing richer spatial relations that go beyond simple matching (names to image and absolute locations to image). in what follows, we propose two generic strategies to inject stronger visual priors into the model after the cross-modal fusion block. in fig. 3 we illustrate how we reformulate the prediction head of the original model in fig. 1 . we describe the modified architecture below. first, due to the one-to-one alignment of the embedding sequence at the input and output of the transformer, we can identify the first block of hw elements as a re-encoding of the original visual embeddings, modulated by the input expression e. we can rearrange this block as a tensor of size h ⇥ w ⇥ d so as to restore the spatial structure lost after the flattening operation. this operation is represented as the unflatten block in fig. 3 . we next add a small convolutional network with m layers (convolution layer with a 3⇥3 kernel and stride of 1, batch normalization and a relu nonlinearity), on top of which we attach two different heads: a target regression head as before and a box segmentation head, as shown in fig. 3 . the box regression head is the same as the base model but applied to the average pooled features after the convolutional block, while the additional segmentation head consists of a simple convolutional layer whose output is normalized to the interval [0, 1] by a sigmoid activation. the goal of this head is to provide a spatial consistency constraint while avoiding the need of requiring additional data. supervisory signal for this branch is obtained trivially from the existing annotations, i.e. by creating a binary mask from the bounding box coordinates of the target. learning is formulated by adding an auxiliary pixel-wise binary classification term to the main loss. the intuition behind adding m convolutional blocks after the cross-modal transformer as well as the mask prediction head is to ensure that the spatial consistency of the representation that has been broken is restored. the locality and hierarchical nature of the (stacked) convolutions as well as the complementarity of the segmentation and regression heads proves to be an effective approach to improve the accuracy of some kinds of spatial expressions as we will see in the next two sections. ## error analysis in this section we analyze the predicted output of the extended and baseline models for the imageexpression pairs of the referitgame validation set. the figures in this section show the extended model prediction in green and the baseline model prediction in orange. the comparative error analysis goal is to shed light over the kinds of linguistic and visual phenomena that the models are able to handle and those they are not. the analysis was divided into two parts and was carried out on a total of 351 examples. first, we explored referring expressions where the green model correctly predicted the ground truth and improved over the baseline depicted in orange. two such examples are in fig. 4 . second, we analysed cases where both models failed, exemplified in fig. 5 . more examples are shown in in appendix b. we identified different abilities a model should have in order correctly solve a broader set of expressions. they are listed below. fig. 4 and fig. 5 illustrate each of the skills. the first three types of errors require the re to be relational. meronimic errors require a relation between an object and a part of it. viewpoint errors occur in res that are not only relational, but also spatial in general. they use a landmark, related to the referent, to change the point of view of the interpreter. directional errors can occur when the direction of the relation in the re is misinterpreted. frequently this relation is spatial but not necessarily, it could correspond to an order established by other property, such as size (e.g. "from the smallest to the biggest). the last four types of errors may happen for all kinds of res. although implicit errors are more frequent in long res such as relational res. ## discussion and conclusions in this work we studied the kinds of errors that reference resolution models make. in particular, relational expressions caused a lot of errors in rec and that motivated the proposal of a model that improves over the previous sota for the task by restoring the referential spaciality. we used a new training objective which is segmentation prediction and added convolutional layers to a transformer. looking at accuracy only can obscure and hide common errors these model might have. we performed an error analysis and identified which skills the model would need in order to perform correctly. we found that our proposed model improves in dealing with fuzzy objects and meronimy, but still finds it difficult with other skills. by performing this error analysis we learnt there is yet a lot of work to do in making models consider viewpoint or being able to deal with implicit information (conveyed by the common visual context). our findings can help have a finer grained look at the predictions of models. this can be relevant for different areas of nlp like grounding, situated dialog systems and human-computer interaction as referring is a crucial skill in communication.