Neuro-Symbolic Visual Dialog (2022)

Abstract: We propose Neuro-Symbolic Visual Dialog (NSVD) —the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connectionist methods on two key challenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demonstrate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog—a relative improvement of more than 10% over the state of the art—while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against incomplete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes.

## introduction modelled after human-human communication, visual dialog involves reasoning about a visual scene through multiple question-answering rounds in natural language @xcite . its multi-round nature gives rise to one of its unresolved key challenges: co-reference resolution @xcite . that is, as dialogs unfold over time, questions tend to include more and more pronouns, such as "it", "that", and "those" that have to be resolved to the appropriate previouslymentioned entities in the scene. co-reference resolution is profoundly challenging @xcite , even for models specifically designed for this task @xcite . existing models follow a purely connectionist approach and suffer from several limitations: first, they require large amounts of training data, which is prohibitive for most settings. second, these models are not explainable, making it difficult to troubleshoot their logic when co-references are incorrectly resolved. finally, current models lack generalisability, in particular for real-world dialogs that include incomplete or inaccurate dialog histories, longer dialogs than those seen during training, or unseen question types. while neuro-symbolic hybrid models have proven effective as a more robust, explainable, and data-efficient alternative, e.g. for vqa @xcite , video qa @xcite , or commonsense reasoning @xcite , they have not yet been explored for visual dialog. we fill this gap by proposing neuro-symbolic visual dialog (nsvd) -the first neuro-symbolic method geared towards visual dialog. our method combines three novel contributions to disentangle vision and language understanding from reasoning: first, it introduces two different program generators: a caption and a question program generator, the former of which induces a program from the caption to initialise the knowledge base of the executor at the beginning of each dialog. second, a question program generator that predicts a program in each round using not only the current question but also the dialog history. we describe two variants of this generator: one that uses a question encoder to concatenate the caption and questionanswer pairs of previous rounds to encode the dialog history, as well as one that stacks them. third, a symbolic executor with a dynamic knowledge base keeps track of all entities mentioned in the dialog. nsvd also addresses another limitation of existing models that was "hidden" by the dominant evaluation scheme used so far: vanishing questionanswering performance over the course of the dia-log. adopted from vqa @xcite , the dominant scheme assumes that the model has full access to the dialog history, in particular all ground truth answers. we argue that this assumption is overly optimistic and overestimates real-world performance on the visual dialog task. we instead propose a more realistic and stricter evaluation scheme in which prediction in the current round is conditioned on previous predicted answers. this scheme better represents real-world dialogs in which communication partners rarely know whether their previous answers were correct or not. through extensive experiments on clevr-dialog @xcite , we show that our models are significantly better at resolving coreferences and at maintaining performance over many rounds. using our stricter evaluation scheme, we still achieve an accuracy of 99.72% while requiring only a fraction of the training data. our results further suggest that nsvd has a higher mean first failure round, is more robust to incomplete dialog histories, and generalises better to dialogs that are up to three times longer than those seen during training as well as to unseen question types and scenes. the contributions of our work are threefold: (1) we introduce the first neuro-symbolic visual dialog model that is more robust again incomplete histories, is significantly better at resolving co-references and at maintaining performance over more rounds on clevr-dialog. visual dialog. das et al. introduced early visual dialog models that used an encoder-decoder approach to rank a set of possible answers. others explored explicit reasoning based on the dialog structure @xcite @xcite . however, these models focused mainly on the real-world visdial dataset @xcite . although popular, this dataset is not wellsuited to study a key challenge of visual dialog, i.e. co-reference resolution, because it lacks complete annotation of all images and dialogs. several works have focused on co-reference resolution in videos @xcite and 3d data @xcite . kottur et al. introduced clevr-dialog -a fully-annotated diagnostic dataset for multi-round visual reasoning with a grammar grounded in the clevr scene graphs. more recently, shah et al. introduced models that build on the memory, attention, and composition (mac) network @xcite . although their models achieved promising results on clevr-dialog, they are computationally and memory inefficient @xcite . while the neuro-symbolic approach has significant potential to address these shortcomings, it has not yet been explored for visual dialog. ## method our method consists of four components (see figure 1 ): a scene understanding method, a program generator with caption and question encoders and a decoder, and a symbolic program executor with a dynamic knowledge base. scene understanding. we used a pre-trained mask r-cnn @xcite to predict segmentation masks and attributes (colour, shape, material, size) for each entity in the visual scene. we then learned the 3d coordinates of each segment paired with the original image using a resnet-34 (he bi-lstm sa ar fc output attended features caption encoder caption answer caption q 1 a 1 q i-1 a i-1 ... q i history question encoder encoder mask r-cnn cnn i d 0 shape cylinder cylinder size big big material color x y z small 1 2 sphere rubber metal metal green cyan red -0.21 -0.42 -0.19 -1.24 -1.15 -1.41 0.17 0.29 0.16 original scene stacked objects scene understanding decoder decoder program generator caption program question program program executor executor dynamic knowledge base update fetch figure 1 : overview of our method: first, a structured scene representation is created. then, a caption program is induced and run by our executor to initialise its knowledge base. at each subsequent round, the question and the history are used to induce a program that answers the question and updates the dynamic knowledge base. ## experiments we modified the publicly available code for clevr-dialog @xcite to generate datasets with ground truth caption and question programs required to train our program generators. similar to @xcite , we used the 70, 000 training and 15, 000 validation clevr images as our visual groundings when generating the dialogs. we left out the clevr test images because they lack ground truth scene annotations. for each image, we generated five dialogs each consisting of @xmath0. we used 1, 000 training images and their corresponding dialogs to create a validation set and tested our models and the baselines on the dialogs generated using the clevr validation images. performance evaluation. alongside the answer accuracy, the first failure round (ffr), i.e. the number of dialog rounds necessary for a model to make its first mistake, is commonly used to evaluate visual dialog models. although popular, this metric has one major limitation as it only allows us to compare the performance of models across datasets with the same dialog length but not across datasets with different ones. thus, we propose the normalised first failure round (nffr) ∈ [0, 1] as an improvement and use it alongside the answer accuracy to assess the performance of all models. see appendix a.2 for more details. ## results visual dialog performance. after validating our implemented logic (see appendix a.4), we compared the performance of our models with the visual dialog mac networks introduced by @xcite . we limited our comparison to their top three performing models given that these outperformed the previous state of the art @xcite by 30% in accuracy. furthermore, we compared our models to the hybrid code networks (hcn) @xcite that also operate on symbolic dialog state representation but follow a different approach to parse programs than our generative one. they represent programs as templates in an action space and select the one with the highest probability during inference. this action space might become intractable if the dsl has many functions and arguments. ## conclusion and future work we proposed neuro-symbolic visual dialog (nsvd) -the first hybrid method to combine deep learning and symbolic program execution for multiround visual reasoning -and a new, stricter and more realistic evaluation scheme for visual dialog. our method outperforms state-of-the-art purely-connectionist baselines on clevr-dialog and sets a new near-perfect test accuracy of 99.72%. furthermore, nsvd has a higher mean first failure round, is more robust to incomplete dialog histories, and generalises better to longer dialogs and to unseen question types and scenes. these performance improvements are not to be seen in isolation of the stricter supervision our models have as they require a supervised fine-tuning of a mask-rcnn in addition to a supervised training of the program parsers. finally, additional evaluations show that our models generalise to other scene domains (appendix a.7) and we expect them to even generalise to naturalistic datasets as well if they provide the necessary supervision requirements for our models similar to the vqa scenario @xcite . to the best of our knowledge, such datasets for visual dialog do not yet exist.