Visually Grounded Concept Composition (2021)

Abstract: We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images. We propose Concept and Relation Graph (CRG), which builds on top of constituency analysis and consists of recursively combined concepts with predicate functions. Meanwhile, we propose a concept composition neural network called Composer to leverage the CRG for visually grounded concept learning. Specifically, we learn the grounding of both primitive and all composed concepts by aligning them to images and show that learning to compose leads to more robust grounding results, measured in text-to-image matching accuracy. Notably, our model can model grounded concepts forming at both the finer-grained sentence level and the coarser-grained intermediate level (or word-level). Composer leads to pronounced improvement in matching accuracy when the evaluation data has significant compound divergence from the training data.

## introduction visually grounded text expressions denote the images they describe. these expressions of visual concepts are naturally organized hierarchically in sub-expressions. the organization reveals structural relations that do not manifest when the subexpressions are studied in isolation. for example, the phrase "a soccer ball in a gift-box" is a compound of two shorter phrases, i.e., "a soccer ball" and "a gift-box", but carries the meaning of the spatial relationship "something in something" that goes beyond the two shorter phrases separately. the compositional structure of the grounded expression requires a concept learner to understand what primitive concepts are visually appearing and how the compound relating multiple primitives modifies their appearance. existing approaches @xcite @xcite @xcite tackle visual grounding via end-to-end learning, which typically learns to align image and text information using neural networks without explicitly modeling their compositional structures. while neural networks have shown strong generalization capabilities in test examples that are i.i.d to the training distribution @xcite , they often struggle in dealing with out-of-domain examples of novel compositional structures, in many tasks such as visual reasoning @xcite @xcite , semantic parsing @xcite , and (grounded) command following @xcite @xcite . in this work, we investigate how complex concepts, composed of simpler ones, are grounded in images at sentences, phrases and tokens levels. in particular, we investigate whether the structures of how these concepts are composed can be exploited as a modeling prior to improve visual grounding. to this end, we design concept & relation graph (crg), which is derived from constituency parse trees. the resulting crg is a graph-structured database where concept nodes encode language expressions of concepts and their visual denotations (e.g., a set of images corresponding to the concept), and predicate nodes define how a concept is semantically composed from its child concepts. our graph is related to the denotation graph @xcite but differs in two key aspects. first, our graph extracts the concepts without specially crafted heuristic rules 1 . secondly, crg's predicate can encode richer information explicitly than the subsumption relationships implicitly expressed in the denotation graphs. an illustrative figure of the graph is shown in figure 1 . in addition to crg, we propose concept composition transformer (composer) that leverages the structure of text expressions to recursively encode the grounded concept embeddings, from coarse-level such as the noun words that refer to objects, to finer-grained ones with multiple levels of compositions. transformer @xcite is used as a building block in our model, to encode the predicates, and perform grounded concept composition. we learn composer using the task of visual-semantic alignment. unlike traditional approaches, we perform hierarchical learning of visual-semantic alignment, which aligns the image to words, phrases, and sentences, and preserves the order of matching confidences. we conduct experiments on multi-modal matching and show that composer achieves strong grounding capability in both sentence-to-image and phrase-to-image retrieval on the popular benchmarks. we validate the generalization capability of composer by designing an evaluation procedure for a more challenging compositional generalization task that uses test examples with maximum compound divergence (mcd) to the training data @xcite . experiments show that composer is more robust to the compositional generalization than other approaches. our contributions are summarized as below: we introduce multi-modal concept and relation graph (crg), a graph composed of concept and predicate nodes, which compose visually grounded descriptive phrases and sentences. figure 1 provides an illustrative example. the concepts include sentences and intermediate phrases, shown as blue nodes. the primitives are the leaf nodes (typically noun words) that refer to visual objects, shown as green nodes. the predicates (red nodes) are n-ary functions that define the meaning of the concept composition. their "signatures" consist of lexicalized templates, the number of arguments, and the syntactic type of the arguments. they combine primitives or simpler concepts into more complex ones. ## composer: recursive modeling of the compositional structure the main idea of composer is to recursively compose primitive concepts into sentences of complex structure, using composition rules defined by the predicates. figure 2 presents a conceptual diagram of the high-level idea. concretely, it first takes the primitive word embedding as the inputs and performs cross-modal attention to obtain their visually grounded word embeddings. next, the composer calls the composition procedure to modify or combine primitive or intermediate concepts, according to the description of its predicates. at the end of this recursive procedure, we obtain the desired sentence concept embedding. in the rest of this section, we first discuss the notation and backgrounds, then introduce how primitives and predicates are encoded ( § 3.1), and present the recursive composition procedures in detail ( § 3.2). finally, we discuss the learning objectives ( § 3.3). notation. we denote a paired image and sentence as (x, y) and the corresponding concepts and predicate for a tree (x, u , e), where u , e corresponds to the set of primitives and the set of predicates, respectfully. we also denote all concepts from a sentence y to be c, where u ⊂ c and @xmath0. ## related work generalization in grounded language understanding. many evaluation methods are proposed to assess the model's generalization capabilities in grounded language understanding. johnson et al. (2017 ) proposes a synthetic dataset, i.e. clevr, to evaluate the generalization of visual question answering models to novel objects and attributes. misra et al. (2017) proposes to evaluate compositional generalization capability of visual models w.r.t. short phrases consist of attributes and objects. chaplot et al. (2018) and hermann et al. @xcite evaluate rl agents' capability to generalize to a novel composition of shape, size, and color in 3d simulators, which shows that rl agents generalize poorly. gscan @xcite perform a systematic benchmark to assess command following in a grounded environment. in this work, we focus on assessing model composition generalization under the visual context. compositional networks. state-of-the-art visually grounded language learning typically use deep transformer models @xcite such as vilbert @xcite , lxmert @xcite and uniter @xcite . though being effective for data over i.i.d distribution, these models do not explicitly exploit the structure of the language and are thus prone to fail on compositional generalization. in contrast, another thread of works @xcite @xcite @xcite parse the language into an executable program composed as a graph of atomic neural modules, where each module is designed to perform atomic tasks and are learned end-to-end. such models show almost perfect performances on synthetic benchmarks @xcite but perform subpar on the real-world data @xcite that are noisy and highly variable. unlike them, we propose using a compositional neural network based on the transformer architecture, which extends state-of-the-art neural networks to explicitly exploits language structure. ## experiment in this section, we perform experiments to validate the proposed composer model on the tasks of sentence-to-image retrieval and phrase-to-image retrieval. we begin with introducing the setup in § 5.1 and then present the main results in § 5.2, comparing models for their in-domain, cross-dataset evaluation, and compositional generalization performance. finally, we perform an analysis and ablation study of our model design in § 5.3. ## conclusion in this paper, we propose the concept and relation graph (crg) to explore the compositional structure in visually grounded text data. we further develop a novel concept composition neural network (composer) on top of the crg, which leverages the explicit structure to compose concepts from word-level to sentence-level. we conduct extensive experiments to validate our model on image-text matching benchmarks. comparing with prior methods, composer achieves significant improvements, particularly in zero-shot cross-dataset transfer and compositional generalization. despite these highlights, there are also many challenges that composer does not address in the scope of this paper. first, it requires high-quality parsing results to achieve strong performances, which may not be readily available in languages beyond english. moreover, similar to other recursive neural networks, composer is also computationally resource demanding, which sets a limit to its scalability to large-scale data. as mentioned in the main paper, we parse the sentence and convert it into a tree of concepts and primitives. particularly, we first perform constituency parsing using the self-attention parser @xcite . the concept criterion defined for the flickr30k and coco dataset contains several principles: (1) if the constituent is a word, it is a primitive concept if its part-of-speech (pos) tag is one of the following: { }. after all the concepts are extracted, we take the remaining words in the current input text expression as the predicate that combines those concepts and use the tag to represent syntactic blank. concrete examples can be found in the