Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions (2020)

Abstract: Visual referring expression recognition is a challenging task that requires natural language understanding in the context of an image. We critically examine RefCOCOg, a standard benchmark for this task, using a human study and show that 83.7% of test instances do not require reasoning on linguistic structure, i.e., words are enough to identify the target object, the word order doesn’t matter. To measure the true progress of existing models, we split the test set into two sets, one which requires reasoning on linguistic structure and the other which doesn’t. Additionally, we create an out-of-distribution dataset Ref-Adv by asking crowdworkers to perturb in-domain examples such that the target object changes. Using these datasets, we empirically show that existing methods fail to exploit linguistic structure and are 12% to 23% lower in performance than the established progress for this task. We also propose two methods, one based on contrastive learning and the other based on multi-task learning, to increase the robustness of ViLBERT, the current state-of-the-art model for this task. Our datasets are publicly available at https://github.com/aws/aws-refcocog-adv .

## introduction visual referring expression recognition is the task of identifying the object in an image referred by a natural language expression @xcite @xcite . figure 1 shows an example. this task has drawn much attention due to its ability to test a model's understanding of natural language in the context of visual grounding and its application in downstream tasks such as image retrieval @xcite ) and question answering @xcite . to track * work done in part while aa was intern at amazon ai. progress on this task, various datasets have been proposed, in which real world images are annotated by crowdsourced workers @xcite . recently, neural models have achieved tremendous progress on these datasets @xcite . however, multiple studies have suggested that these models could be exploiting strong biases in these datasets @xcite . for example, models could be just selecting a salient object in an image or a referring expression without recourse to linguistic structure (see figure 1 ). this defeats the true purpose of the task casting doubts on the actual progress. in this work, we examine refcocog dataset @xcite , a popular testbed for evaluating referring expression models, using crowdsourced workers. we show that a large percentage of samples in the refcocog test set indeed do not rely on linguistic structure (word order) of the expressions. accordingly, we split refcocog test set into two splits, ref-easy and ref-hard, where linguistic structure is key for recognition in the latter but not the former ( §2). in addition, we create a new out-of-distribution 1 dataset called ref-adv using ref-hard by rewriting a referring expression such that the target object is different from the original annotation ( §3). we evaluate existing models on these splits and show that the true progress is at least 12-23% behind the established progress, indicating there is ample room for improvement ( §4). we propose two new models, one which make use of contrastive learning using negative examples, and the other based on multi-task learning, and show that these are slightly more robust than the current state-of-the-art models ( §5). ## importance of linguistic structure refcocog is the largest visual referring expression benchmark available for real world images @xcite . unlike other referring expression datasets such as refcoco and ref-coco+ @xcite , a special care has been taken such that expressions are longer and diverse. we therefore choose to examine the importance of linguistic structure in refcocog. @xcite observed that when the words in a referring expression are shuffled in random order, the performance of existing models on re-fcocog drops only a little. this suggests that models are relying heavily on the biases in the data than on linguistic structure, i.e., the actual sequence of words. ideally, we want to test models on samples where there is correlation between linguistic structure and spatial relations of objects, and any obscurity in the structure should lead to ambiguity. to filter out such set, we use humans. we randomly shuffle words in a referring expression to distort its linguistic structure, and ask humans to identify the target object of interest via predefined bounding boxes. each image in ref-cocog test set is annotated by five amazon mechanical turk (amt) workers and when at least three annotators select a bounding box that has high overlap with the ground truth, we treat it as a correct prediction. following @xcite , we set 0.5 iou (intersection over union) as the threshold for high overlap. given that there are at least two objects in each image, the optimal performance of a random choice is less than 50%. 2 however, we observe that human accuracy on distorted examples is 83.7%, indicating that a large portion of refcocog test set is insensitive to linguistic structure. based on this observation, we divide the test set into two splits for fine-grained evaluation of models: ref-easy contains samples insensitive to linguistic structure and ref-hard contains sensitive samples (statistics of the splits are shown in ## an out-of-distribution dataset due to unintended annotation artifacts in ref-cocog, it is still possible that models could perform well on ref-hard without having to rely on linguistic structure, e.g., by selecting frequent objects seen during training time. essentially, ref-hard is an in-distribution split. to avoid this, we create ref-adv, an adversarial test set with samples that may be fall out of training distribution. we take each sample in ref-hard and collect additional referring expressions such that the target object is different from the original object. we chose the target objects which humans are most confused with when the referring expression is shuffled (as described in the previous section). for each target object, we ask three amt workers to write a referring expression while retaining most content words in the original referring expression. in contrast to the original expression, the modified expression mainly differs in terms of the structure while sharing several words. for example, in figure 1 , the adversarial sample is created by swapping pastry and blue fork and making plate as the head of pastry. we perform an extra validation step to filter out bad referring expressions. in this step, three additional amt workers select a bounding box to identify the target object, and we only select the samples where at least two workers achieve @xmath0.5 with the target object. since the samples in ref-adv mainly differ in linguistic structure with respect to ref-hard, we hope that a model which does not make use of linguistic structure (and correspondingly spatial relations between objects) performs worse on ref-adv even when it performs well on ref-hard due to exploiting biases in the training data. figure 2 shows several examples from the ref-easy, ref-hard, and ref-adv splits. we note that easy: a dinning table with cake and drinks hard: a chair with a purse hanging from it adv: the purse which is hanging from a chair easy: bus hard: bus in the middle of the crowd adv: the crowd that the bus is in the middle of easy: the larger of two giraffes hard: a giraffe eating leaves off the tree adv: the giraffe that is not eating leaves off the tree easy: a blue snowboard hard: a woman wearing a blue jacket and orange glasses next to a woman with a white hood adv: a woman with a white hood, next to a woman wearing orange glasses and a blue jacket. easy: water in a tall, clear glass hard: the glass of water next to the saucer with the cup on it adv: the cup on the saucer, next to the glass of water easy: the short blue bike on the right hard: the blue bike behind the red car adv: the red car behind the blue bike easy: the man with the glasses on hard: a man holding a cake that is not wearing a tie adv: the man holding a cake that is wearing a tie easy: a green cushion couch with a pillow hard: a green couch across from a white couch adv: a white couch across from a green couch ref-adv expressions are longer on average than ref-easy and ref-hard (figure 6 in appendix) and consists of rich and diverse spatial relationships (figure 7 in appendix). concurrent to our work, gardner et al. ( 2020 ) also propose perturbed test splits for several tasks by modifying in-domain examples. in their setup, the original authors of each task create perturbed examples, whereas we use crowdworkers. closest to our work is from @xcite who also use crowdworkers. while we use perturbed examples to evaluate robustness, they also use them to improve robustness (we propose complementary methods to improve robustness §5). moreover, we are primarily concerned with the robustness of models for visual expression recognition task, while @xcite focus on different tasks (e.g., sentiment, natural language inference). ## diagnosing referring expression recognition models we evaluate the following models, most of which are designed to exploit linguistic structure. cmn (compositional modular networks; @xcite ) grounds expressions using neural modules by decomposing an expression @xmath1, relation, @xmath2. the subject and object are localized to the objects in the image using a localization module while the relation between them is modeled using a relationship module. the full network learns to jointly decompose the input expression into a triple while also recognizing the target object. groundnet @xcite ) is similar to cmn, however it makes use of rich linguistic structure (and correspondingly rich modules) as defined by an external syntactic parser. @xcite generalizes cmn to flexibly adapt to expressions that cannot be captured by the fixed template of cmn. it introduces new modules and also uses an attention mechanism to weigh modules. vilbert @xcite , the state-of-the-art model for referring expression recognition, uses a ref-adv expressions are longer on average than ref-easy and ref-hard (figure 6 in appendix) and consists of rich and diverse spatial relationships (figure 7 in appendix). concurrent to our work, gardner et al. ( ## diagnosing referring expression recognition models we evaluate the following models, most of which are designed to exploit linguistic structure. cmn (compositional modular networks; @xcite ) grounds expressions using neural modules by decomposing an expression @xmath3, relation, @xmath4. the subject and object are localized to the objects in the image using a localization module while the relation between them is modeled using a relationship module. the full network learns to jointly decompose the input expression into a triple while also recognizing the target object. groundnet @xcite ) is similar to cmn, however it makes use of rich linguistic structure (and correspondingly rich modules) as defined by an external syntactic parser. mattnet @xcite generalizes cmn to flexibly adapt to expressions that cannot be captured by the fixed template of cmn. it introduces new modules and also uses an attention mechanism to weigh modules. vilbert @xcite , the state-of-the-art model for referring expression recognition, uses a pretrain-then-transfer learning approach to jointly learn visiolinguistic representations from largescale data and utilizes them to ground expressions. this is the only model that does not explicitly model compositional structure of language, but bert-like models are shown to capture syntactic structure latently @xcite . ## conclusion our work shows that current datasets and models for visual referring expressions fail to make effective use of linguistic structure. although our proposed models are slightly more robust than existing models, there is still significant scope for improvement. we hope that ref-hard and ref-adv will foster more research in this area.