S eq D ial N : Sequential Visual Dialog Network in Joint Visual-Linguistic Representation Space (2021)

Abstract: The key challenge of the visual dialog task is how to fuse features from multimodal sources and extract relevant information from dialog history to answer the current query. In this work, we formulate a visual dialog as an information flow in which each piece of information is encoded with the joint visual-linguistic representation of a single dialog round. Based on this formulation, we consider the visual dialog task as a sequence problem consisting of ordered visual-linguistic vectors. For featurization, we use a Dense SymmetricCo-Attention network (Nguyen and Okatani,2018) as a lightweight vison-language joint representation generator to fuse multimodal features (i.e., image and text), yielding better computation and data efficiencies. For inference, we propose two Sequential Dialog Networks (SeqDialN): the first uses LSTM(Hochreiter and Schmidhuber,1997) for information propagation (IP) and the second uses a modified Transformer (Vaswani et al.,2017) for multi-step reasoning (MR). Our architecture separates the complexity of multimodal feature fusion from that of inference, which allows simpler design of the inference engine. On VisDial v1.0 test-std dataset, our best single generative SeqDialN achieves 62.54% NDCG and 48.63% MRR; our ensemble generative SeqDialN achieves 63.78% NDCG and 49.98% MRR, which set a new state-of-the-art generative visual dialog model. We fine-tune discriminative SeqDialN with dense annotations and boost the performance up to 72.41% NDCG and 55.11% MRR. In this work, we discuss the extensive experiments we have conducted to demonstrate the effectiveness of our model components. We also provide visualization for the reasoning process from the relevant conversation rounds and discuss our fine-tuning methods. The code is available at https://github.com/xiaoxiaoheimei/SeqDialN .

## introduction visual dialog has attracted increasing research interest as an emerging field, bringing together aspects of computer vision, natural language processing, and dialog systems. in this task, an ai agent is required to hold a meaningful dialog with humans in natural, conversational language about visual content. specifically, given an image, a dialog history, and a query about the image, the agent has to ground the query in image, infer context from history, and answer the query accurately @xcite . our work is inspired by the use of visuallinguistic joint representation to erase the modality gap, where we embed the visual signals into the text snippets for each dialog round. in this way, we convert a visual dialog into an ordered vector sequence, where each vector is the joint visual-linguistic representation of a specific dialog round. rather than using vilbert @xcite , we chose dense symmetric co-attention @xcite as a lightweight joint visual-linguistic representation generator. in contrast to visdial-bert @xcite , which concatenates all rounds of the dialog history into a single textual input for vilbert @xcite , we keep each dialog round separate. keeping this inherent sequential structure from the visual dialog allows us to reason across the dialog history to find the most query-relevant dialog rounds. by viewing visual dialog task as a vector sequence, we propose two sequential networks to tackle the problem. fig. 1 illustrates a conceptual overview of the proposed method. the visual features and language embeddings are learned from two independent domains. they are fed into the dense symmetric co-attention network @xcite to produce a visual-linguistic vector sequence in the joint visual-linguistic feature space. our baseline model, the information propagation network (seqipn), which uses a lstm @xcite to summarize the visual-linguistic sequence, outperforms other well-known baselines @xcite , on ndcg metric by a large @xmath0.5. multi-step reasoning network (se-qmrn) is based on transformer @xcite . we expect the multi-head attention mechanism of transformer better captures the relationship within the visual linguistic sequence. we achieve multi-step reasoning by stacking several transformers to refine attentions in high level semantic space. seqmrn outperforms visdial-bert @xcite @xmath1.5% on ndcg when trained with comparable amount of data, while using 30% less parameters. the pipeline in fig. 1 facilitates the combination of different word embeddings and seqdialn models. in this work, we compare two kinds of pre-trained word representations: glove @xcite and distilbert @xcite . the ablation test shows that seqmrn with distilbert embedding yields the best performance. further experiment reveals seqdialn sets a new state-of-the-art generative visual dialog model. vldialog and ndcgfinetune @xcite tune with dense annotations 3 . training on the dense annotation 3 makes these models perform very well on the ndcg metric but poorly on the others because the dense annotation 3 dataset doesn't correlate well with the original ground-truth answer to the question @xcite . in this work, we propose a reweighting method to mitigate the damage to non-ndcg metrics in fine-tuning process, which make our best model outperform @xcite on mrr by a large margin at the cost of a little lower ndcg than them. the main contributions of this paper is three fold. (1) we formulate the visual dialog task as reasoning from a sequence in the joint visuallinguistic representation space. (2) we propose two sequential networks to tackle the visual dia-log task in the joint visual-linguistic representation space. (3) we set a new state-of-the-art generative visual dialog model. recently, dense co-attention network (dcn) @xcite proposes a symmetric co-attention layer to address vqa tasks. dcn is "dense symmetric" because it makes each visual region aware of the existence of each question word and vice versa. this fine-granularity co-attention enables dcn to discriminate subtle differences or similarities between vision and language features. in this work, we use dcn as the generator of joint visual-linguistic representation. ## approach the visual dialog task @xcite is formulated as follows: at time t, given a query q t grounded in image i, and dialog history (including the image caption c) h @xmath2, (q 1 , a 1 ), • • • , (q t-1 , a t-1 )} as additional context. for discriminative task, the goal is to rank 100 candidate answers a @xmath3, a 2 t , • • • , a 100 t }. for generative task, the goal is to generate an answer in natural language. the task requires the agent to predict the ground truth answer and rank other feasible answers as high as possible. as illustrated in fig. 1 , we rely on faster- @xcite to extract features corresponding to salient image regions @xcite . the vision feature of image i is represented as f @xmath4, where n @xmath5. q t and each item in h is padded or truncated to the same length d l . thus, each sentence s is represented as f @xmath6, where d e being the dimension of the word embedding. to facilitate further discussion, we denote d h as the dimension of the hidden state throughout this section. ## conclusion we presented sequential visual dialog network (seqdialn) based on a novel idea that treats dialog rounds as a visual-linguistic vector sequence. we explore both discriminative and generative models and set up a new state-of-the-art generative visual dialog model. even though our model is trained only on visdial v1.0 dataset, it achieves competitive performance against other models trained on much larger vision-language datasets, which facilitates its deployment in industrial environment.