Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer (2021)

Abstract: Visual dialog is a task of answering a sequence of questions grounded in an image using the previous dialog history as context. In this paper, we study how to address two fundamental challenges for this task: (1) reasoning over underlying semantic structures among dialog rounds and (2) identifying several appropriate answers to the given question. To address these challenges, we propose a Sparse Graph Learning (SGL) method to formulate visual dialog as a graph structure learning task. SGL infers inherently sparse dialog structures by incorporating binary and score edges and leveraging a new structural loss function. Next, we introduce a Knowledge Transfer (KT) method that extracts the answer predictions from the teacher model and uses them as pseudo labels. We propose KT to remedy the shortcomings of single ground-truth labels, which severely limit the ability of a model to obtain multiple reasonable answers. As a result, our proposed model significantly improves reasoning capability compared to baseline methods and outperforms the state-of-the-art approaches on the VisDial v1.0 dataset. The source code is available at https://github.com/gicheonkang/SGLKT-VisDial .

## introduction recently, visually-grounded dialogue @xcite @xcite has attracted increasing research interest due to its potential impact on many real-world applications (e.g., aiding visually impaired user). notably, visual dialog (visdial) @xcite , which extends visual question answering (vqa) @xcite @xcite † corresponding authors. there are two fundamental challenges in visdial: we argue that existing approaches in visdial show limited reasoning capability due to the way they approach the task: soft-attention and one-hot encoded labels. first, soft-attention restricts the ability to represent various types of semantic relationships in the dialog. as we illustrate in figure 1 , some questions in the dialog (q1-q4) are semantically dependent on previous utterances, while others (q6) are independent, due to an abrupt change in topic. furthermore, previous topics could be readdressed later in the dialog (q5). however, soft-attention, which is based on the softmax function, always assigns a non-zero weight to all previous utterances, which results in dense (i.e., fully-connected) relationships. moreover, the sum of attention weights should be one due to the sum-to-1 constraint of the softmax function. herein lies the problem: even for questions that are partly dependent (q5 in figure 1 ) or independent (q6 in figure 1 ) from the dialog history, all previous utterances are still considered and integrated into the contextual representations. as a consequence, the dialog agent could overly rely on the dialog history, even when the dialog history is irrelevant to the given question. second, the model that utilizes the one-hot encoded labels learns to predict the single ground-truth answer only. however, similar to vqa, the given question is associated with one or several answers from a set of candidate answers. therefore, the one-hot labels could suppress several plausible answers, assigning unreasonably low prediction probabilities to them. in this paper, we propose two methods to remedy the conceptual shortcomings of the current approaches discussed above. first, we introduce a sparse graph learning (sgl) method that predicts sparse structures of the visually-grounded dialog. in the graph structure, each node corresponds to a round of the dialog, and edges represent the semantic relationships between the rounds. sgl constructs the representations of each node by embedding the given image and each round of dialog in a joint fashion. sgl then infers two types of edge weights: binary (i.e., 0 or 1) and score edges. it ultimately discovers the sparse and weighted structures (e.g., (c) in figure 1 ) by incorporating the two edge weights. furthermore, we design a new structural loss function to encourage sgl to infer explicit and reliable dialog structures by leveraging a structural supervision. next, to identify multiple possible answers, we treat visdial as a regression task that predicts the correctness of each candidate answer individually, instead of a traditional setting that estimates the sum-to-1 scores over the candidate answers. to this end, we propose a knowledge transfer (kt) method that extracts the soft scores of each candidate answer from the teacher model @xcite . the soft scores are used to optimize for multiple possible answers. we expect this work to shed light on the above challenges that have not been explicitly addressed in visual dialog. the main contributions of our paper are as follows. first, we propose a sparse graph learning (sgl) approach that builds sparse structures of the visually-grounded dialog. by leveraging a new structural loss function, sgl learns the semantic relationships among dialog rounds in an explicit way. second, we introduce a knowledge transfer (kt) method to encourage the model to find multiple possible answers to the given question. third, the model that utilizes sgl and kt achieves the new state-of-the-art results on the visdial v1.0 dataset. we perform comprehensive analysis to validate the effectiveness of sgl and kt. finally, we conduct a qualitative analysis of each proposed method. graph neural networks @xcite have sparked a tremendous interest at the intersection of deep neural networks and structural learning approaches. recently, graph learning networks (glns) were proposed by @xcite , with the goal of reasoning over underlying structures of input data. glns consider unstructured data and dynamic domains (e.g., timevarying domain). our method belongs to the group of glns. cb-glns (on et al., 2020) attempt to discover the compositional structure of long video data with a graph-cut algorithm @xcite . however, sgl is different from previous studies in that sgl learns to build sparse structures adaptively, not relying on a predefined algorithm, and the dataset we use is highly multimodal. knowledge transfer technique has been mainly explored to compress a large model into a small model @xcite without a significant drop in accuracy. the idea of knowledge transfer was later popularized under the name of knowledge distillation (kd) @xcite . in kd, the knowledge of the large model (i.e., teacher model) is transferred to the small model (i.e., student model) as a form of supervision signal. then, the student model learns to mimic the behavior of the teacher model by using the supervision signal and a pre-defined distillation loss function. our knowledge transfer (kt) approach shares this same spirit. however, we repurpose kt to cast visdial as a regression of scores for candidate answers. accordingly, the soft targets from the teacher model are utilized as supervision for the correctness of each candidate answer which was originally unlabeled. ## sparse graph learning the visual dialog task @xcite is defined as follows: given an image i, a caption c describing the image, a dialog history in our approach, we consider the task as a graph g @xmath0, e t ) with t + 1 nodes (i.e., vertices), where (v 0 , v 1 , ..., v t-1 ) and (v t ) correspond to the node for the previous dialog history and the current question, respectively. each node v @xmath1. the semantic dependencies among the nodes are represented as weighted edges the goal of our approach is to discover a sparse and weighted adjacency matrix a @xmath2. to implement the pipeline above, we propose a sparse graph learning (sgl) method that consists of two modules (see figure 2 ): (1) a node embedding module that embeds the visual-linguistic representations for each round of the dialog and ## knowledge transfer the conventional assumption in visdial is that there is one correct answer for each question from a set of candidate answers. accordingly, the onehot encoded single ground-truth label is used as standard supervision. however, the given question can indeed be associated with one or several answers. for this reason, a few works @xcite have applied an additional fine-tuning strategy on dense labels foot_2 for the validation split to improve the model's ability to predict multiple correct answers. instead of using the finetuning approach, we propose a knowledge transfer (kt) method to optimize several correct answers simultaneously in a single training procedure. kt extracts the soft scores of each candidate answer from the fine-tuned teacher model, p1+p2 @xcite , and uses these scores as pseudo labels. we choose the p1+p2 for their strong performance on retrieving several appropriate answers for the given question. specifically, we combine the dense score vector y dense @xmath3: ## conclusions we propose sgl and kt to remedy the shortcomings of previous work: soft-attention and one-hot labels. experimental results illustrate the effectiveness of our approach. sgl with kt achieves the new state-of-the-art performance on the visdial v1.0 dataset. we believe that the idea of selectively paying attention to desired information is widely applicable to various research fields, and kt can be generally adopted to improve answer prediction.