Resilience through Scene Context in Visual Referring Expression Generation (2024)

Abstract: Scene context is well known to facilitate humans’ perception of visible objects. In this paper, we investigate the role of context in Referring Expression Generation (REG) for objects in images, where existing research has often focused on distractor contexts that exert pressure on the generator. We take a new perspective on scene context in REG and hypothesize that contextual information can be conceived of as a resource that makes REG models more resilient and facilitates the generation of object descriptions, and object types in particular. We train and test Transformer-based REG models with target representations that have been artificially obscured with noise to varying degrees. We evaluate how properties of the models’ visual context affect their processing and performance. Our results show that even simple scene contexts make models surprisingly resilient to perturbations, to the extent that they can identify referent types even when visual information about the target is completely missing.

## introduction objects do not appear randomly in the world that surrounds us, but they occur in predictable spatial, semantic, or functional configurations and relations to their environment. research on human perception shows that we "see the world in scenes" @xcite , and that prior experience and knowledge of the world helps us to efficiently process visual stimuli. even with an extremely short glimpse at an image, humans remember essential semantic aspects of the scene and object arrangement @xcite . this rapid scene understanding allows us to handle the complexity of the visual world and to recognize objects in context, e.g., when they are not fully visible @xcite . today's systems for vision and language (v&l) commonly process visual inputs that represent 1 code, models and data for this project are available at: https://github.com/clause-bielefeld/reg-scene-context "real-world" scenes (e.g. @xcite @xcite which, to some extent, exhibit the regularities that human perception is known to be exploiting. yet, it is not clear how current v&l systems process context and whether they rely on strategies of scene understanding similar to humans. in this paper, we aim to investigate this question for referring expression generation (reg, @xcite , a controlled set-up that is well established in nlg research, by testing how scene context supports reference generation for objects that are difficult to recognize. whereas classical reg algorithms mostly build on pre-defined symbolic representations @xcite , neural generation models in visual reg have to extract object properties from low-level visual representations (i.e., photographs) of the target and its context @xcite . this even applies to properties as fundamental as the type of an object, i.e. how it is named in the expression. under ideal conditions, determining a referent's type and properties can be regarded as a relatively simple task, but it becomes non-trivial in the presence of imperfect visual information, occlusion or noise. here, in light of previous findings on human scene understanding (cf. section 2), scene context can be expected to be of great support. however, to date, little is known as to how processes of scene understanding and object type identification interact in reg. in this work, we hypothesize that visual scene context makes reg models more resilient, i.e., it allows them to recalibrate predictions that were based on imperfect target representations. to test this, we use a novel and highly controllable experimental setup for reg: we train and test different transformer-based model architectures with target representations that have been artificially obscured with varying degrees of noise (cf. figure 1 ), simulating scenarios that are common in the real world but insufficiently represented in current reg datasets. we provide the models with different context representations and compare their performance on common quality metrics and a focused human evaluation of their ability to determine referent types. our results show that context makes models surprisingly resilient to perturbations in target representations, to the extent that they can identify referent types even when information about the objects themselves is completely missing. we believe that these results open up new perspectives on how information about the structure and content of surrounding scenes facilitate the description of objects in reg and related tasks. ## background human scene understanding research on human vision and perception emphasizes the fact that scenes are not mere collections of objects @xcite . when humans view a scene, they do not simply recognize the objects in it, but understand it as a coherent whole. @xcite observe that humans perceive the so-called gist of a scene rapidly and even when local information is missing (e.g. blurred). other experiments indicate that contextual information can facilitate the recognition of visible objects across different tasks @xcite @xcite , and that incongruent context can also be misleading @xcite demonstrating that the human vision exploits learned knowledge about regularities of the visual word for visual processing @xcite @xcite @xcite . scenes, objects, and image captioning much research on v&l is concerned with modeling the generation and understanding of image descriptions, e.g. in image captioning @xcite @xcite , among many others). yet, many captioning tasks focus on rather object-centric descriptions that mention objects and their spatial relationships @xcite . a common representation of scene context in image captioning is scene graphs @xcite , which are usually modeled via spatial relations between bounding boxes of objects. @xcite propose a new task and dataset that foregrounds scene-level instead of object-centric descriptions. another perspective on scene knowledge in captioning models is coming from work that focuses on probing them with perturbed or systematically varied images: @xcite find that captioning with extremely reduced inputs of labeled object layouts performs surprisingly well. related to this, @xcite show that image captioning models often rely on regularities in object occurrences, to the extent that they fail to generalize to new combinations of objects. their solution is to generate unseen combinations and challenge models on these. our goal in this work is complementary: we aim to understand how exactly generation models may be able to leverage regular scene knowledge and patterns of object co-occurrence, and how this may facilitate the handling of imperfect visual information. reg and scene context reg is concerned with the generation of descriptions that distinguish a particular object in a given visual context, cf. krahmer and van deemter 2012. in past years, reg research has largely transitioned from symbolic settings to visual reg, focusing on referring expressions for objects in photographs @xcite . recent models usually build on image captioning models but are adapted to generate more pragmatically informative expressions, using e.g. training objectives @xcite , comprehension modules @xcite , reinforcement agents @xcite or decoding strategies @xcite . visual reg models usually process different forms of context information. whereas some models encode differences in appearance between targets and surrounding objects @xcite @xcite @xcite , others use representations of the global image @xcite @xcite @xcite , both commonly supplemented with the relative position and size of the target in the image. on a conceptual level, however, recent work in visual reg generally follows the traditional paradigm by dale and reiter 1995, i.e. context is mainly considered in terms of so-called distractor or competitor objects, that are similar to the target and must therefore be excluded by naming differences @xcite , but see ilinykh and dobnik 2023 for context influences in object naming). in this view, context "exerts pressure", as the speaker needs to reason about which attributes and words make the expression unambiguous @xcite . in this paper, we investigate how contextual information can be conceived as a resource that makes the generation of descriptions easier rather than harder. ## discussion and conclusion our findings show that contextual information about the surroundings of referents makes reg models more resilient against perturbations in visual target representations. even if no target information is present at all, context allows reg models to maintain good results in automatic quality metrics and to identify referent types with high accuracy, as shown in the human evaluation results. this holds for different kinds of context: while especially the trf vis model is able to leverage scene information from resnet encodings of image contents outside the target bounding box, the same applies to symbolic scene representations, as included in trf sym and cc sym . this adds another perspective to basic assumptions of the reg paradigm, where context information is considered important mainly to ensure that references can be resolved without ambiguity. here, we show, that it is also a valuable source for further communicative goals, i.e. the truthfulness of generated expressions. interestingly, while related studies on human perception emphasize the importance of e.g. learned co-occurrence patterns between objects, our subsequent analysis rather points to implicitly learned copying strategies that appear to be highly effective for the relatively regular refcoco data. while this can also be seen as exploiting scene patterns, it is fundamentally different from the ways in which scene information is interpreted by humans (cf. section 2). therefore, we see an urgent need for data more representative of real-world scenarios to further investigate the impact of scene context on multimodal language generation. overall, our results indicate that the influence of visual context in reg is more multifaceted than reflected in previous studies. importantly, this study only provides an initial spotlight, as research in related fields suggests that there are other and more complex ways in which visual scene context may facilitate reference production. with this in mind, we strongly advocate further research into scene context at the interface of perceptual psychology and v&l generation.