C o NAN : A Complementary Neighboring-based Attention Network for Referring Expression Generation (2020)

Abstract: Daily scenes are complex in the real world due to occlusion, undesired lighting conditions, etc. Although humans handle those complicated environments well, they evoke challenges for machine learning systems to identify and describe the target without ambiguity. Most previous research focuses on mining discriminating features within the same category for the target object. One the other hand, as the scene becomes more complicated, human frequently uses the neighbor objects as complementary information to describe the target one. Motivated by that, we propose a novel Complementary Neighboring-based Attention Network (CoNAN) that explicitly utilizes the visual differences between the target object and its highly-related neighbors. These highly-related neighbors are determined by an attentional ranking module, as complementary features, highlighting the discriminating aspects for the target object. The speaker module then takes the visual difference features as an additional input to generate the expression. Our qualitative and quantitative results on the dataset RefCOCO, RefCOCO+, and RefCOCOg demonstrate that our generated expressions outperform other state-of-the-art models by a clear margin.

## introduction generating referring expressions @xcite @xcite @xcite , which identify target objects with simple words and phrases in everyday discourse, has attracted attention from both computer vision (cv) and natural language processing (nlp) communities. with the rapid development of rnns @xcite @xcite and the emergence of transformers @xcite , machine learning systems can generate linguistically correct expressions in most cases. however, the remaining issue in the referring expression generation (reg) field is to avoid ambiguities (i.e. the generated expression should refer to a unique target object). this issue becomes increasingly important when referring to an object in complex daily scenes where occlusion, undesired lighting conditions, complex formation of objects, occur regularly. this complex nature inhibits the system from mining the unique features automatically for the target object among its visually similar ones. previous works have mainly investigated model architectures to generate less ambiguous expressions. speaker-listener models @xcite are widely adopted to encourage a speaker to generate expressions that can be comprehended by listener model. further, @xcite employs a reinforcer module to reward the system if the generated expression is bonded to the target object. in order to find discriminative features for the target object and generate less ambiguous expression, plenty of research utilizes the visual differences between the target objects and the objects that belong to the same category determined by an object detector. however, when multiple visually similar objects appear in the scene, mining the discriminative features for the target object becomes challenging. instead, a human would use the surrounding objects to help clarify the target one. motivated by that, we propose a complementary neighboring-based attention network (conan) that explicitly utilizes and highlights the visual difference between the target object and its neighbors, instead of mining discriminative feature within a class. conan first finds and computes visual differences of the k spatial neighbors for each target object, and then uses the attentional ranking module to rank the potential contribution of each the neighbor object. finally, the speaker (expression generator) in conan additionally takes the top-m ranking visual differences together with the target object and the global representation as inputs to generate referring expressions. note that the conan is compatible with most current learning-based expression generation systems. in particular, we adopt slr @xcite as our baseline system. experimental evaluation shows a significant improvement for the generated expression compared to the state-of-the-art on the three refcoco datasets. ## model in this section, we present a complementary neighboring-based attention network (conan) for generating unambiguous referring expressions. in particular, we first extract k neighbor objects for each target object, detailed in section 3.1. to mine discriminating features for the target object, visual differences between it and its neighbors are utilized as complementary inputs. to better encode the local context, we also employ an attentional ranking strategy that weighs the neighbors to select meaningful ones in section 3.2. finally, we present an expression generating module in section 3.3 that takes the attentional visual difference, target object feature, and the global features as inputs to generate high quality expressions. ## conclusion in this work, we present an approach to explicitly mining complementary aspects for the target object in the local scene. in particular, the visual differences between the target and its neighbors are adopted. instead of using all of the neighbors, we employ an attentional ranking module to filter out irrelevant neighbor objects. finally, the speaker module is built upon the global features, target object features, and our complementary neighbor features to generate the expression. our quantitative results show that conan effectively enhances the performance for referring expression generation outperforming other state-of-the-art methods by a clear margin. besides, our qualitative results state that conan has the potential to give out the more descriptive expression for each target object sometimes even far superior to the ground truth.