Cross-Modality Relevance for Reasoning on Language and Vision (2020)

Abstract: This work deals with the challenge of learning and reasoning over language and vision data for the related downstream tasks such as visual question answering (VQA) and natural language for visual reasoning (NLVR). We design a novel cross-modality relevance module that is used in an end-to-end framework to learn the relevance representation between components of various input modalities under the supervision of a target task, which is more generalizable to unobserved data compared to merely reshaping the original representation space. In addition to modeling the relevance between the textual entities and visual entities, we model the higher-order relevance between entity relations in the text and object relations in the image. Our proposed approach shows competitive performance on two different language and vision tasks using public benchmarks and improves the state-of-the-art published results. The learned alignments of input spaces and their relevance representations by NLVR task boost the training efficiency of VQA task.

## introduction real-world problems often involve data from multiple modalities and resources. solving a problem at hand usually requires the ability to reason about the components across all the involved modalities. examples of such tasks are visual question answering (vqa) @xcite and natural language visual reasoning (nlvr) @xcite . one key to intelligence here is to identify the relations between the modalities, combine and reason over them for decision making. deep learning is a prominent technique to learn representations of the data for decision making for various target tasks. it has achieved supreme performance based on large scale corpora @xcite . however, it is a challenge to learn joint representations for crossmodality data because deep learning is data-hungry. there are many recent efforts to build such multimodality datasets @xcite @xcite @xcite @xcite . researchers develop models by joining features, aligning representation spaces, and using transformers @xcite . however, generalizability is still an issue when operating on unobserved data. it is hard for deep learning models to capture high-order patterns of reasoning, which is essential for their generalizability. there are several challenging research directions for addressing learning representations for crossmodality data and enabling reasoning for target tasks. first is the alignment of the representation spaces for multiple modalities; second is designing architectures with the ability to capture high-order relations for generalizability of reasoning; third is using pre-trained modules to make the most use of minimal data. an orthogonal direction to the above-mentioned aspects of learning is finding relevance between the components and the structure of various modalities when working with multi-modal data. most of the previous language and visual reasoning models try to capture the relevance by learning representations based on an attention mechanism. finding relevance, known as matching, is a fundamental task in information retrieval (ir) @xcite . benefiting from matching, transformer models gain the excellent ability to index, retrieve, and combine features of underlying instances by a matching score @xcite , which leads to the state-of-the-art performance in various tasks @xcite . however, the matching in the attention mechanism is used to learn a set of weights to highlight the importance of various components. in our proposed model, we learn representations directly based on the relevance score inspired by the ideas from ir models. in contrast to the attention mechanism and transformer models, we claim that the relevance patterns are as important. with proper alignment of the representation spaces of different input modalities, matching can be applied to those spaces. the idea of learning relevance patterns is similar to siamese networks @xcite which learn transferable patterns of similarity of two image representations for one-shot image recognition. similarity metric between two modalities is shown to be helpful for aligning multiple spaces of modalities @xcite . the contributions of this work are as follows: 1) we propose a cross-modality relevance (cmr) framework that considers entity relevance and highorder relational relevance between the two modalities with an alignment of representation spaces. the model can be trained end-to-end with customizable target tasks. 2) we evaluate the methods and analyze the results on both vqa and nlvr tasks using vqa v2.0 and nlvr 2 datasets respectively. we improve state-of-the-art on both tasks' published results. our analysis shows the significance of the patterns of relevance for the reasoning, and the cmr model trained on nlvr 2 boosts the training efficiency of the vqa task. ## related work language and vision tasks. learning and decision making based on natural language and visual information has attracted the attention of many researchers due to exposing many interesting research challenges to the ai community. among many other efforts @xcite @xcite , antol et al. proposed the vqa challenge that contains open-ended questions about images that require an understanding of and reasoning about language and visual components. suhr et al. proposed the nlvr task that asks models to determine whether a sentence is true based on the image. attention based representation. transformers are stacked self-attention models for general purpose sequence representation @xcite . they have been shown to achieve extraordinary success in natural language processing not only for better results but also for efficiency due to their parallel computations. self-attention is a mechanism to reshape representations of components based on relevance scores. they have been shown to be effective in generating contextualized representations for text entities. more importantly, there are several efforts to pre-train huge transformers based on large scale corpora @xcite @xcite on multiple popular tasks to enable exploiting them and performing other tasks with small corpora. researchers also extended transformers with both textual and visual modalities @xcite @xcite @xcite . sophisticated pre-training strategies were introduced to boost the performance @xcite . however, as mentioned above, modeling relations between components is still a challenge for the approaches that try reshaping the entity representation space while the relevance score can be more expressive for these relations. in our cmr framework, we model high-order relations in relevance representation space rather than the entity representation space. matching models. matching is a fundamental task in information retrieval (ir). there are ir models that focus on comparing the global representation matching @xcite , the local components (a.k.a terms) matching @xcite , and hybrid methods @xcite . our relevance framework is partially inspired by the local components matching which we apply here to model the relevance of the components of the model's inputs. however, our work differs in several significant ways. first, we work under the cross-modality setting. second, we extend the relevance to a highorder, i.e. model the relevance of entity relations. third, our framework can work with different target tasks, and we show that the parameters trained on one task can boost the training of another. ## cross-modality relevance cross-modality relevance (cmr) aims to establish a framework for general purpose relevance in various tasks. as an end-to-end model, it encodes the relevance between the components of input modalities under task-specific supervision. we further add a high-order relevance between relations that occur in each modality. figure 1 shows the proposed architecture. we first encode data from different modalities with single modality transformers and align the encoding spaces by a cross-modality transformer. we consistently refer to the words in text and objects in images (i.e. bounding boxes in images) as "entities" and their representations as "entity representations". we use the relevance between the components of the two modalities to model the relation between them. the relevance includes the relevance between their entities, as shown in the "entity relevance", and high-order relevance between their relations, as shown in the "relational relevance". we learn the representations of the affinity matrix of relevance score by convolutional layers and fully-connected layers. finally, we predict the output by a non-linear mapping based on all the relevance representations. this architecture can help to solve tasks that need reasoning on two modalities based on their relevance. we argue that the parameters trained on one task can boost the training of the other tasks that deal with multi-modality reasoning. in this section, we first formulate the problem. then we describe our cross-modality relevance (cmr) model for solving the problem. the architecture, loss function, and training procedure of cmr are explained in detail. we will use the vqa and nlvr tasks as showcases. ## conclusion in this paper, we propose a novel cross-modality relevance (cmr) for language and vision reasoning. particularly, we argue for the significance of relevance between the components of the two modalities for reasoning, which includes entity relevance and relational relevance. we propose an end-to-end cross-modality relevance framework that is tailored for language and vision reasoning. we evaluate the proposed cmr on nlvr and vqa tasks. our approach exceeds the state-of-the-art on nlvr 2 and vqa v2.0 datasets. moreover, the model trained on nlvr 2 boosts the training of vqa v2.0 dataset. the experiments and the empirical analysis demonstrate cmr's capability of modeling relational relevance for reasoning and consequently its better generalizability to unobserved data. this result indicates the significance of relevance patterns. our proposed architectural component for capturing relevance patterns can be used independently from the full cmr architecture and is potentially applicable for other multi-modal tasks.