upplementing Missing Visions via Dialog for Scene Graph Generatio

Most current AI systems rely on the premise that the input visual data are sufficient to achieve competitive performance in various computer vision tasks. However, the current task setup rarely considers the challenging, yet common practical situations where the complete visual data may be inaccessible due to various reasons (*e.g.*, restricted view range and occlusions). To this end, we investigate a computer vision task setting with incomplete visual input data. Specifically, we exploit the Scene Graph Generation (SGG) task with various levels of visual data missingness as input. While insufficient visual input naturally leads to performance drop, we propose to supplement the missing visions via the natural language dialog interactions to better accomplish the task objectives. We design a model-agnostic Supplementary Interactive Dialog (*SI-Dial*) framework that can be jointly learned with most existing models, endowing the current AI systems with the ability of question-answer interactions in natural language. We demonstrate the feasibility of such task setting with missing visual input and the effectiveness of our proposed dialog module as the supplementary information source through extensive experiments and analysis, by achieving promising performance improvement over multiple baselines.

Introduction
============

AI systems related to computer vision research have witnessed great progress in recent years. The majority of the current AI systems rely on full access to sufficient visual data (*e.g.*, clear images or multiple frames from the video clip) to achieve best performance in various vision-related tasks such as the scene graph generation (SGG) @imp [@motifs; @graphrcnn; @tang2020unbiased], action recognition @ac1 [@ac2; @ac3; @ac4], and visual question answering (VQA) @antol2015vqa [@vqa2; @vqa3; @vqa4; @vqa5; @vqa6; @vqa7]. The standard experimental settings for the above tasks rarely consider the situations where the input visual data may be insufficient to achieve the task objectives. However, in practical scenarios, the missingness in the visual data is a more than common issue that can be caused by various reasons. For example, certain objects within a single image may be occluded by other objects during photography, which makes it difficult to identify their category and to infer the precise scene graph in the SGG task. Similarly, a video clip may fail to capture the entire moving or action process of a person due to the fixed camera position and limited view range, and thus becomes insufficient to clearly recognize the actions. To this end, we consider the computer vision task setting with insufficient visual input in this work.

As the primary information source to various computer vision tasks, the visual input data play a significant role in most existing works to achieve competitive and promising performance. It is reasonable to expect the performance drop under the task setting with incomplete visual input. To tackle the problem, we propose to supplement the missing visual data from another information source: the natural language dialog. Our motivations to utilize the natural language dialog as the supplementary information source are mainly two-fold: On the one hand, from the realistic perspective, humans rely on the multi-sensory systems from various modalities (*e.g.*, vision, audio, and language) to understand the surrounding world, and it is intuitive for them to ask questions about the insufficient information given a specific task to fulfill. On the other hand, from the view of research works and AI systems, previous studies in the field of vision and language @das2017learning [@de2017guesswhat; @lu2017knowing; @anderson2018bottom] have demonstrated the advantage of combining the two modalities for better performance. Specifically, @zhu2020describing [@zhu2021saying] have shown preliminary potential of using the natural language dialog in case of limited visual input. However, different from the previously proposed video description task, our work focuses on a more general computer vision task setting with insufficient visual input, and exploits the feasibility of such challenge, yet practical setting with more refined and controlled level of data missingness.

To implement the proposed idea of supplementing the insufficient visual input via the natural language dialog, we introduce a model-agnostic interactive dialog framework, which can be jointly learned with most existing models and endows the models with the capability to communicate in the form of natural language question-answer interactions. The proposed Supplementary Interactive Dialog (*SI-Dial*) framework stimulate the realistic scenarios with two agents, where one agent plays the role of the actual AI system given insufficient visual input, and the other plays the role of human user or evaluator that can answer the raised questions with needed information. In fact, there are two ways to formulate the interactions in terms of the answerer. One possible formulation is to attach the *SI-Dial* framework to the existing AI systems and stimulate the answerer by directly providing the answers. Another formulation is similar to @zhu2020describing [@zhu2021saying], in which the answerer is also modeled by a separate agent that can be jointly learned and optimized during the pipeline training. The former method has the potential to achieve better performance by providing the corresponding ground truth answers to the raised questions, while the later one is more flexible and can be possibly applied in a wider context such as robotics.

Author names
============

Each author name must be followed by:

-   A newline <span>\\\\</span> command for the last author.

-   An <span>\\And</span> command for the second to last author.

-   An <span>\\and</span> command for the other authors.

Affiliations
============

After all authors, start the affiliations section by using the <span>\\affiliations</span> command. Each affiliation must be terminated by a newline <span>\\\\</span> command. Make sure that you include the newline on the last affiliation too.

Mapping authors to affiliations
===============================

If some scenarios, the affiliation of each author is clear without any further indication (*e.g.*, all authors share the same affiliation, all authors have a single and different affiliation). In these situations you don't need to do anything special.

In more complex scenarios you will have to clearly indicate the affiliation(s) for each author. This is done by using numeric math superscripts <span>\${\^$i,j, \ldots$}\$</span>. You must use numbers, not symbols, because those are reserved for footnotes in this section (should you need them). Check the authors definition in this example for reference.

Emails
======

This section is optional, and can be omitted entirely if you prefer. If you want to include e-mails, you should either include all authors' e-mails or just the contact author(s)' ones.

Start the e-mails section with the <span>\\emails</span> command. After that, write all emails you want to include separated by a comma and a space, following the same order used for the authors (*i.e.*, the first e-mail should correspond to the first author, the second e-mail to the second author and so on).

You may "contract" consecutive e-mails on the same domain as shown in this example (write the users' part within curly brackets, followed by the domain name). Only e-mails of the exact same domain may be contracted. For instance, you cannot contract "jamesmeyer@example.org" and "jamesmeyer@example.org" because the domains are different.

[^1]: Contact Author

{'source': 'arXiv_out/2204'}