K nowledge Supports Visual Language Grounding: A Case Study on Colour Terms (2020)

Abstract: In human cognition, world knowledge supports the perception of object colours: knowing that trees are typically green helps to perceive their colour in certain contexts. We go beyond previous studies on colour terms using isolated colour swatches and study visual grounding of colour terms in realistic objects. Our models integrate processing of visual information and object-specific knowledge via hard-coded (late) or learned (early) fusion. We find that both models consistently outperform a bottom-up baseline that predicts colour terms solely from visual inputs, but show interesting differences when predicting atypical colours of so-called colour diagnostic objects. Our models also achieve promising results when tested on new object categories not seen during training.

## introduction research on human perception has shown that world knowledge supports the processing of sensory information @xcite . for instance, humans have been found to use their knowledge about typical colours of an object when perceiving an instance of that object, in order to compensate for, e.g., perceptually challenging illumination conditions and achieve colour constancy @xcite . thus, the visual perception of object colours can be thought of as leveraging top-down knowledge for bottom-up processing of sensory input, in accordance with traditional approaches in psychology (e.g. @xcite . the integration of visual information and world knowledge in perception, however, is far from obvious, with views ranging from processing through bidirectionally connected bottom-up and top-down components to the assumption that visual and conceptual representations themselves are inseparably intertwined @xcite . a lot of recent work in language & vision (l&v) has looked at grounding language in realistic sensory information, e.g. images of complex, real-world scenes and objects @xcite . in l&v, however, the use of top-down knowledge has mostly been discussed in the context of zero-shot or few-shot learning scenarios where few or no visual instances of a particular object category are available @xcite . 1 we present a simple experiment on language grounding that highlights the great potential of top-down processing even for very common words with a lot of visual instances: we learn to ground colour terms in visual representations of real-world objects and show that model predictions improve strongly when incorporating prior knowledge and assumptions about the object itself. we investigate visual grounding of colour terms by combining bottom-up and top-down modeling components based on early and late fusion strategies, reflecting different interpretations about the integration of visual and conceptual information in human perception. we find that these strategies lead to differ-ent predictions, especially for atypical colours of objects that do have a strong tendency towards a certain colour. foot_2 ## related work even recent work on colour terms has mostly been using artificial datasets with descriptions of isolated colour swatches that show a single hue, primarily examining effects of context and conversational adequacy in colour naming @xcite @xcite @xcite . however, object colours bear a range of additional challenges for perception and grounding: (i) chromatic variation due to lighting and shading @xcite , (ii) effects of conventionalization as in e.g. red hair @xcite and (iii) the inherent complexity of real-world objects @xcite , e.g. a tree with green leaves and a brown trunk is typically called green (see figure 1 ). in human cognition, several recalibration strategies support the constant perception of object colours given these challenges. in addition to bottom-up driven strategies like the chromatic adaption to situational sources of light, this also includes mechanisms such as the memory colour effect: the automatic perception of canonical colours that accompanies the recognition of objects with characteristic hues @xcite . our aim in this work is to transfer knowledge-based recalibration mechanisms to the automatic classification of object colours. @xcite and @xcite propose pixelwise approaches for modeling colour naming in natural images, accounting for factors such as illumination and non-uniform object colours. @xcite assign colour terms as labels to colour values of individual pixels and then average over these labels to obtain a colour term for an image region. we use their model as one of our baselines in section 4. however, they do not take into account object-specific colour tendencies. @xcite classify colour histograms for objects in real-world images. they train object-specific classifiers that recalibrate a bottom-up classifier, but only obtain a small improvement from recalibration. we implement a general top-down component that can be 3 models we focus on the effect of knowledge in language grounding and adopt a slightly idealized setting for modeling: we assume that the object type is available during training and testing. following e.g. @xcite ; @xcite ; @xcite , we distinguish early and late fusion as a way of integrating modeling components with different sources of information. figure 2 illustrates our models, which we describe below. bottom-up this component relies solely on sensory input and is implemented as a feed-forward network trained to predict colour terms from 3dimensional rgb histograms (representing the polychromatic distribution of colour values in complex objects). the output layer has a softmax over the 11 basic colour terms @xcite . for comparability, we adopt the architecture in @xcite (input layer: 512 nodes, hidden layers with 240 and 24 nodes and relu activation, output layer: 11 nodes, drop-out: 0.2). we did not obtain improvements when testing other colour spaces. we also tried visual features extracted with a neural object recognizer @xcite which only give a small improvement over colour histograms. thus, in section 4, we report results only for rgb histograms, as they are more transparent as representations and do not include any conceptual information on objects. top-down this component relies only on conceptual information about the object which consists of assignments of objects to object types and colour distributions for object types reflected in the data. thus, this classifier predicts colour terms given only the object type, which is supposed to mimic the memory colour effect discussed in section 2. we use (pre-trained) word embeddings for object types that are not fine-tuned during training. hence, top-down and the combined models can be tested on unseen object types. we use 100-dimensional pre-trained glove embeddings @xcite . the embedding layer is followed by a hidden layer (24 nodes, relu activation, drop-out set to 0.2). ## discussion and conclusion as in human perception, knowledge about typical object properties seems to be a valuable source of information for visual language grounding. our fusion models clearly outperform a bottom-up baseline that relies solely on visual input. we also showed that the fusion architecture matters: the early integration of visual and conceptual information and their shared processing appears to be beneficial when colour diagnostic objects have atypical colours. however, even early fusion does not yet achieve a perfect balance between top-down and bottom-up processing. future work should look at more complex fusion strategies, possibly coupled with bottom-up recalibration mechanisms @xcite to further enhance colour classification under difficult illumination conditions. our experiment on objects unseen during training looks promising but can be extended towards a more general approach that interfaces colour prediction with object recognition.