Category-Based Strategy-Driven Question Generator for Visual Dialogue (2021)

Abstract: GuessWhat?! is a task-oriented visual dialogue task which has two players a guesser and anoracle. Guesser aims to locate the object supposed by oracle by asking several Yes/No questions which are answered by oracle. How to ask proper questions is crucial to achieve the final goal of the whole task. Previous methods generally use an word-level generator which is hard to grasp the dialogue-level questioning strategy. They often generate repeated or useless questions. This paper proposes a sentence-level category-based strategy-driven question generator(CSQG) to explicitly provide a category based questioning strategy for the generator. First we encode the image and the dialogue history to decide the category of the next question to be generated. Thenthe question is generated with the helps of category-based dialogue strategy as well as encoding of both the image and dialogue history. The evaluation on large-scale visual dialogue dataset GuessWhat?! shows that our method can help guesser achieve 51.71% success rate which is the state-of-the-art on the supervised training methods.

## introduction goal-oriented visual dialogue is one of the multi-modal task which has gained increasing attentions. it can usually be explained as a task which using a couple of visual related turns of communication to reach a specific goal, such as guesswhich @xcite , visdial @xcite , multimodal dialogs (mmd) @xcite , guesswhat?! @xcite and so-on. guesswhat?! @xcite is a two-player game which can be named guesser and oracle. given an image, the oracle choose an object in the image without telling the guesser. guesser reads the image and asks several yes/no questions, at the meanwhile oracle uses 'yes @xcite a' to answer them. after a fixed turns of questioning, guesser should guess the chosen object in image and one game finishes. as is mentioned above, there are two sub tasks: one is ask question(qgen), and the other is guess. figure 1 gives a sample in guesswhat?! dataset. to reach the goal of the entire game, the agent for task qgen should be able to make use of information from both the dialogue history and the image. de vries et al. @xcite use an encoderdecoder model to generate questions, which is the first qgen model. after that, some researchers manage to improve the generation quality by attention mechanism. zhuang et al. @xcite proposed a parallel attention (plan) network. deng et al. @xcite proposed accumulated attention mechanism (a-att). xu et al. @xcite focused on boosting the agent to generate highly efficient questions and obtains reliable visual attentions and proposed @xcite . on the other hand, @xcite introduced reinforcement learning (rl). @xcite proposed visual dialogue state tracking (vdst) to represent visual dialogue state and update with the change of the distribution on objects. when humans play the game, they often have some strategies which aims to minimize the scope of the object and reaches the goal as quickly as possible. for example, in visual dialogue dataset guesswhat?!, people might first ask the type of the object, once they determine the right type of the object, they then is it a donut ? is it on the left ? is it on the right ? is it the whole donut ? does it have pink icing ? is it touching the donut with the sprinkles ? does it have chocolate icing ? is it the third donut? yes no yes yes no yes no yes ## related work task-oriented visual dialogue is a new and critical research direction for dialogue systems. it needs not only the ability of traditional text-based dialogue @xcite @xcite , but also image representation and multi-modal fusion capabilities @xcite . as one of the widely used dataset, guesswhat?! (de vries et al., 2017) has three sub-tasks, c c l 2 0 2 1 qgen, oracle and guesser. qgen is extensively studied due to its key role in the entire game. @xcite @xcite . there are mainly two lines of researches on qgen task. one line is the improvement on model structure. @xcite proposed the first model for the task as a baseline. it uses the fc8-layer of vgg-16 model @xcite to provide image embedding and a hierarchical rnn-based model to encode the dialogue history. the generator is performed by rnn-based model which concatenates image and context representation and generates responses. in order to have more clear image representation, resnet @xcite is introduced by the work from shekhar et al. @xcite which extracts the image representation by the second to last layer of resnet152. due to the lack representation capability of static embedding, object detection models are involved. xu et al. @xcite and @xcite introduced faster-rcnn @xcite and receive a better performance. many of other researches focus on the improvement of linguistic information encoding. other than the baseline method, some attention-based models are proposed to get dynamic visual and linguistic embedding. plan network @xcite is proposed to provide dynamic visual information at each round by computing the attention on different regions. accumulated attention @xcite consists of three kinds of attention which are query, image and objects attention. answer-driven visual state estimator @xcite ) uses an answer-based attention transfer method to exploit different answers in different ways to update the visual attention at each step. the other line is on model learning. @xcite first used reinforce learning on this task. temperature policy gradient @xcite is proposed to make balance of exploration and exploitation while selecting words. visual dialogue state tracking (vdst) @xcite is proposed to update the change of the distribution on objects. our work is basically on the first line. inspired by human cognitive psychology @xcite . and current controlled text generation tasks (smith et al., 2020), we add control factors to influence the direction of text generation. to prepare questions controlled information, we first classify them like krishna et al. @xcite at the earliest for vqa task. different from them, we merge the unreasonable classifications and used a more accurate model-based method instead of keywords to label questions' categories. in qgen task, our csqg predicts the probability distribution of category and determines the certain category in this turn based on the explicit prior-probability strategy. as we can see, there is currently no work to add the question category as an element to the generation process. our method is based on human cognition and has better interpretability and logic. the specific effect will also be confirmed in subsequent experiments. ## proposed model in this section, we will introduce our question generation model csqg in detail. denote the dialogue history as h , h t-1 = [q 0 , a 0 , q 1 , a 1 ...q t-1 , a t-1 ] as history before the current turn, andi as the image representation. our model can be described as q @xmath0, i) where q t is the generated question for user in this turn. the overall structure of the model is shown in figure 2 . there are three module: (1) a multi-modality encoder with a hierarchical rnn-based neural network and an image-text attention structure, (2) a category predictor and (3) a generation decoder. ## experiments to access the performance, we evaluate our models on the large visual dialogue dataset guesswhat?!, which contains 155,281 dialogues and 821,955 question-answer pairs. these dialogues are based on 66537 images with 134,074 different objects. to make a fair contrast between ours and the previous, we divided guesswhat?! dataset according to the same way from others. all dialogues are split into train, dev and test dataset, 70%, 15%, 15%. ## conclusions we introduced csqg, a category-based strategy-driven question generator which is guided with category information. at first, we predict the initial probability of each class by classifier and joint learning training method. then we introduce an update strategy based on the prior probability of each category in each turn. the updated category guides the decoder to generate the next question. experiments shows that our proposed model achieves 51.7% on new games in large visual dialogue dataset guesswhat?!, which is the state of the art under supervise learning. the ablation experiments further proved that both category guided and prior probability have positive effect on the success rate of the game. our csqg model is training only on supervised learning. however, due to the coherence, interactivity and dynamics of visual dialogue, there is a lot of room for improvement in reinforcement learning. in the future, we will explore how to produce category prediction which is more consistent with the overall goal through reinforcement learning. and this will balance the goals of interpretability and game success rate.