Visual-Textual Alignment for Graph Inference in Visual Dialog (2020)

Abstract: As a conversational intelligence task, visual dialog entails answering a series of questions grounded in an image, using the dialog history as context. To generate correct answers, the comprehension of the semantic dependencies among implicit visual and textual contents is critical. Prior works usually ignored the underlying relation and failed to infer it reasonably. In this paper, we propose a Visual-Textual Alignment for Graph Inference (VTAGI) network. Compared with other approaches, it makes up the lack of structural inference in visual dialog. The whole system consists of two modules, Visual and Textual Alignment (VTA) and Visual Graph Attended by Text (VGAT). Specially, the VTA module aims at representing an image with a set of integrated visual regions and corresponding textual concepts, reflecting certain semantics. The VGAT module views the visual features with semantic information as observed nodes and each node learns the relationship with others in visual graph. We also qualitatively and quantitatively evaluate the model on VisDial v1.0 dataset, showing our VTAGI outperforms previous state-of-the-art models.

## introduction cross-modal semantic understanding has become an attractive challenge in natural language processing and computer vision, inspiring many tasks such as image captioning @xcite and visual question answering (vqa) @xcite @xcite . however, in these missions, the co-reference between vision and language is usually performed in a single round and they do not have many interactions with human over a period of time. in 2017, das et al. introduced a continuous conversational task, visual dialog @xcite . this task needs an ai agent to answer a sequence of questions based on visually-grounded information and contextual information from a dialog history. recently, a manual investigation @xcite on the visual dialog dataset (visdial) tried to figure out how many questions can be answered with images and how many of them need conversation history to be answered. the investigation shows that around 80% of the questions can be answered with images and about 20% of the questions need the knowledge from dialog history. therefore, one of the key challenges in visual dialog is how to effectively utilize these underlying contents in the textual and visual information, i.e., input questions, dialog history and input image. in previous works, such as rva @xcite and dan @xcite , both tended to explicitly reason over past dialog interactions by referring back to previous references, but they ignored the underlying relational structure which contributes to dialog inference. nowadays, researchers have attempted to consider the fixed graph attention or embedding to resolve the problem with structural representations @xcite . they focused on the textual modality but neglected the rich underlying information in the image. in this task, despite its significance to artificial intelligence and human-computer interaction, the agent requires understanding a series of multi-modal entities, and reasons the rich information in both vision figure 1 : an overview of vtagi. we present two modules, vta and vgat. vta takes the attended visual features and attended textual features including question and history as inputs, resulting in integrated image representations reflecting semantics of certain objects in the image. vgat aims to construct a visual graph combined with textual context. the relations among the nodes are top-5 important. for example, the most thickest link between the node v 1 and v k indicates the most important dependencies of them. ⊗ and denote matrix multiplication and element-wise product, respectively. and language. an ideal inference algorithm should be able to find out the underlying relational structure and give a reasonable answer based on this structure. to address aforementioned problem, we pay more attention on visual-textual relation and propose the vtagi in figure 1 to explore potential information for structural inference. the agent will first obtain the question features, history features and visual features by employing different attention mechanisms. however, the semantics of the visual features and the textual concepts are usually inconsistent, and the representations of the image lack of global structural information. thus, the vta module is to align the visual features and global textual contents with their relevant counterparts in each image domain. as a result, the visual features contain more specific semantic information. for example, in figure 1 , the visual feature v 1 contains the semantic information of "giraffe, only, beside, trees", because each visual feature considers all contextual information. in order to infer more reasonably and connect all of the individual visual features, we design the vgat module to construct a visual graph that shows the different relationships among various visual features. for example, considering the feature v 1 in figure 1 , the thickest link between the v 1 and v k indicates the most important relationship between the two features. this module learns how to select other nodes related to the current node. in the last step in this process, each visual feature node in this structural module is connected to its related nodes. through the two modules, the final visual features possess more related information and they are intra-connected in the graph, which are beneficial to inference. ## related work visual dialog. most studies on the task of visual dialog introduced by das et al. @xcite can be categorized into four groups. fusion-based models: late fusion (lf) @xcite and hierarchical recurrent network (hre) @xcite directly encoded the multi-modal inputs (image, question, dialog history) and decoded the answer. attention-based models: memory network (mn) @xcite , history-conditioned image attention (hciae) @xcite , sequential co-attention (coatt) @xcite and synergistic co-attention network (sync) @xcite computed attended representations of inputs. visual co-reference resolution (vcor)-based models: attention memory (amem) @xcite , neural module networks (corefnmn) @xcite , recursive visual attention mechanism (rva) @xcite and dual attention network (dan) @xcite , these solutions clarified ambiguous expressions (e.g., he, she, they) in the text and focused on explicit visual co-reference resolution. graph-based models attempt to construct some structures to obtain more underlying information. zheng et al. @xcite designed a structural inference model based on an em-style (expectation-maximization) gnns (graph neural networks) to conduct the textual co-reference. schwartz et al. @xcite proposed a factor graph mechanism and constructed the graph over all the multi-modal features. guo et al. @xcite utilized the word-level attention of question to construct a context-aware graph. the aforementioned graph-related models did not highlight the visual features and their relationships. while, in our work, which also belongs to the forth group, building a relational graph based on visual objects can contain more information. visual-semantic alignment. in image captioning, @xcite ) introduced the notion of visual-semantic alignment, which was based on a novel combination of convolution neural network over image regions, bidirectional recurrent neural network over sentences and a structural objective that aligned two modalities through a multi-modal embedding. in the field of vqa, some recent efforts @xcite @xcite have also been dedicated to studying similar alignment between image and question. to acquire integrated image representations, they normally aligned the visual features and textual concepts, which were beneficial to explore the latent relation. in this paper, we align heterogeneous modalities (image, question and history) based on distinct attention mechanisms, to make each region in the image possessing more specific and detailed contents. especially, we make the visual features contain two levels of semantic information. by adding history features, the visual features can acquire the global textual information. and through integrating question features, visual features also get the logical contents. graph neural network. the concept of graph neural network (gnn) was first proposed by @xcite , who extended existing neural networks for processing the data represented in graph domain. gnns have been applied in various tasks @xcite @xcite @xcite . the core was to combine the graphical structural representation with neural networks. the gnn follows a strategy that controls how the representation vector of a node calculated by its neighboring nodes to capture specific patterns of a graph. the neighborhood connectivity information in gnns is unrestricted and potentially irregular, giving them greater applicability than convolutional neural networks (cnns), which impose a fixed regular neighborhood structure. in this paper, we apply gnn to learn the relation among visual features with multi-modal contexts for inferring answers. through this gnn, each visual feature can connect with other associated features. ## proposed approach in this section, we firstly define the visual dialog task as in das et al. @xcite . formally, a visual dialog agent takes image i, question q t and dialog history h t as input. among them, the q t is asked in the current round t. the h t is consist of q&a pairs till round t-1, while in the first round it only contains the caption c about the image i. the agent is required to return an answer a @xmath0, a t 2 , • • • , a t 100 } to the q t , by ranking a list of 100 candidate answers in a discriminative manner. we will present the language features and the image features in section 3.1, followed by section 3.2 describing vta module. finally, the detailed information of vgat module is provided in section 3.3. ## conclusion in this paper, we introduce visual-textual alignment for graph inference (vtagi) network based on graph method for the visual dialog task. rather than relying on the visual attention maps in prior works,