Original: L ARGE -S CALE A NSWERER IN Q UESTIONER ’ S M IND FOR V ISUAL D IALOG Q UESTION G ENERATION
Revised: L ARGE -S CALE A NSWERER IN Q UESTIONER ’ S M IND FOR V ISUAL D IALOG Q UESTION G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Answerer in Questioner’s Mind (AQM) is an information-theoretic framework that has been recently proposed for task-oriented dialog systems.
Revised: Answerer in Questioner’s Mind (AQM) is an information-theoretic framework that has been recently proposed for task-oriented dialog systems.

Original: AQM beneﬁts from asking a question that would maximize the information gain when it is asked.
Revised: AQM beneﬁts from asking a question that would maximize the information gain when it is asked.

Original: However, due to its intrinsic nature of explicitly calculating the information gain,
Revised: However, due to its intrinsic nature of explicitly calculating the information gain,

Original: AQM has a limitation when the solution space is very large.
Revised: AQM has a limitation when the solution space is very large.

Original: To address this, we propose AQM+ that can deal with a large-scale problem and ask a question that is more coherent to the current context of the dialog.
Revised: To address this, we propose AQM+ that can deal with a large-scale problem and ask a question that is more coherent to the current context of the dialog.

Original: We evaluate our method on
Revised: We evaluate our method on

Original: GuessWhich, a challenging task-oriented visual dialog problem, where the number of candidate classes is near 10K.
Revised: GuessWhich, a challenging task-oriented visual dialog problem, where the number of candidate classes is near 10K.

Original: Our experimental results and ablation studies show that AQM+ outperforms the state-of-the-art models by a remarkable margin with a reasonable approximation.
Revised: Our experimental results and ablation studies show that AQM+ outperforms the state-of-the-art models by a remarkable margin with a reasonable approximation.

Original: In particular, the proposed AQM+ reduces more than 60% of error as the dialog proceeds, while the comparative algorithms diminish the error by less than 6%.
Revised: In particular, the proposed AQM+ reduces more than 60% of error as the dialog proceeds, while the comparative algorithms diminish the error by less than 6%.

Original: Based on our results, we argue that AQM+ is a general task-oriented dialog algorithm that can be applied for non-yes-or-no responses.
Revised: Based on our results, we argue that AQM+ is a general task-oriented dialog algorithm that can be applied for non-yes-or-no responses.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent advances in deep learning have led an end-to-end neural approach to task-oriented dialog problems that can reduce a laborious labeling task on states and intents (Bordes & Weston, 2017).
Revised: Recent advances in deep learning have led an end-to-end neural approach to task-oriented dialog problems that can reduce a laborious labeling task on states and intents (Bordes & Weston, 2017).

Original: Many researchers have applied sequence-to-sequence models (Vinyals & Le, 2015) that are trained in a supervised learning (SL) and a reinforcement learning (RL) fashion to generate an appropriate sentence for the task.
Revised: Many researchers have applied sequence-to-sequence models (Vinyals & Le, 2015) that are trained in a supervised learning (SL) and a reinforcement learning (RL) fashion to generate an appropriate sentence for the task.

Original: In SL approaches, given the dialog histories so far, the model predicts the distribution of the responses from the task-oriented system (Eric & Manning, 2017; de Vries et al., 2017; Zhao et al., 2018).
Revised: In SL approaches, given the dialog histories so far, the model predicts the distribution of the responses from the task-oriented system (Eric & Manning, 2017; de Vries et al., 2017; Zhao et al., 2018).

Original: However, the SL approach typically requires a lot of training data to deal with unseen scenarios and cover all trajectories of the vast action space of dialog systems (Wen et al., 2016).
Revised: However, the SL approach typically requires a lot of training data to deal with unseen scenarios and cover all trajectories of the vast action space of dialog systems (Wen et al., 2016).

Original: Furthermore, because the SL-based model does not consider the sequential characteristic of the dialog, the error may propagate over time that causes an inconsistent dialog (Li et al., 2017; Zhao & Eskenazi, 2016).
Revised: Furthermore, because the SL-based model does not consider the sequential characteristic of the dialog, the error may propagate over time that causes an inconsistent dialog (Li et al., 2017; Zhao & Eskenazi, 2016).

Original: To address this issue, RL has been applied to the problem (Strub et al., 2017; Das et al., 2017b).
Revised: To address this issue, RL has been applied to the problem (Strub et al., 2017; Das et al., 2017b).

Original: By learning the intrinsic planning policy and the reward function, RL approach enables the models to generate a consistent dialog and generalize better on unseen scenarios.
Revised: By learning the intrinsic planning policy and the reward function, RL approach enables the models to generate a consistent dialog and generalize better on unseen scenarios.

Original: However, these methods struggle to ﬁnd a competent RNN model that uses back-propagation, owing to the complexity of learning a series of sentences (Lee et al., 2018).
Revised: However, these methods struggle to ﬁnd a competent RNN model that uses backpropagation, owing to the complexity of learning a series of sentences (Lee et al., 2018).

Original: As an alternative, Lee et al.
Revised: As an alternative, Lee et al.

Original: (2018) have recently proposed “Answerer in Questioner’s Mind” (AQM) algorithm that does not depend on a limited capacity of RNN models to cover an entire dialog.
Revised: (2018) have recently proposed Answerer in Questioners Mind (AQM) algorithm that does not depend on a limited capacity of RNN models to cover an entire dialog.

Original: AQM treats the problems as twenty question games and selects the question that gives a maximum information gain.
Revised: AQM treats the problems as twenty question games and selects the question that gives a maximum information gain.

Original: Unlike the other approaches, AQM beneﬁts from explicitly calculating the posterior distribution and ﬁnding a solution analytically.
Revised: Unlike the other approaches, AQM beneﬁts from explicitly calculating the posterior distribution and ﬁnding a solution analytically.

Original: The authors showed promising results in the taskoriented dialog problem, such as GuessWhat (de Vries et al., 2017), where a questioner tries to ﬁnd an object that is in answerer’s mind via a series of Yes/No questions.
Revised: The authors showed promising results in the task-oriented dialog problem, such as GuessWhat (de Vries et al., 2017), where a questioner tries to ﬁnd an object that is in answerer’s mind via a series of Yes/No questions.

Original: The candidates are conﬁned to the objects that are presented in the given image (less than ten on average).
Revised: The candidates are conﬁned to the objects that are presented in the given image (less than ten on average).

Original: However, this simpliﬁed task may not be general enough to practical problems where the number of objects, questions and answers are typically unrestricted.
Revised: However, this simpliﬁed task may not be general enough to practical problems where the number of objects, questions and answers are typically unrestricted.

Original: For example, GuessWhich is a generalized version of GuessWhat that has a greater number of class candidates (9,628 images) and a dialog that consists of sentences beyond yes or no (Das et al., 2017b).
Revised: For example, GuessWhich is a generalized version of GuessWhat that has a greater number of class candidates (9,628 images) and a dialog that consists of sentences beyond yes or no (Das et al., 2017b).

Original: Because the computational complexity vastly increased to explicitly calculate the information gain over the size of the entire search space, the original AQM algorithm is not scalable to a large scale problem.
Revised: Because the computational complexity vastly increases to explicitly calculate the information gain over the size of the entire search space, the original AQM algorithm is not scalable to a large scale problem.

Original: More speciﬁcally, the number of the unit calculation for information gain in GuessWhat is 10 (number of objects) × 2 (Yes/No), while that of GuessWhich is 10 , 000 (number of images) × ∞ (answer is a sentence) which makes the computation intractable.
Revised: More speciﬁcally, the number of the unit calculation for information gain in GuessWhat is 10 (number of objects) × 2 (Yes/No), while that of GuessWhich is 10 , 000 (number of images) × ∞ (answer is a sentence) which makes the computation intractable.

Original: One of the interesting ideas Lee et al.
Revised: One of the interesting ideas Lee et al.

Original: (2018) suggested is to retrieve an appropriate question from the training set.
Revised: (2018) suggested is to retrieve an appropriate question from the training set.

Original: Retrieval-based models, which are basically discriminative models that select a response from a predeﬁned candidate set of system responses, are often used in task-oriented dialog taskswhere it is critical not to generate sentences that are ill-structured or irrelevant to the task (Bordes & Weston, 2017; Seo et al., 2017a;
Revised: Retrieval-based models, which are basically discriminative models that select a response from a predeﬁned candidate set of system responses, are often used in task-oriented dialog tasks (Bordes & Weston, 2017; Seo et al., 2017a;

Original: Liu & Perez, 2017).
Revised: Liu & Perez, 2017).

Revised: It is critical not to generate sentences that are ill-structured or irrelevant to the task.

Original: However, such a discriminative approach does not ﬁt well with complicated task-oriented visual dialog tasks, because asking an appropriate question considering the visual context is crucial to successfully tackle the problem.
Revised: However, such a discriminative approach does not ﬁt well with complicated task-oriented visual dialog tasks, because asking an appropriate question considering the visual context is crucial to successfully tackle the problem.

Original: It is noticeable that AQM achieved high performance even with a retrieval-based approach in GuessWhat by making the candidate set of questions form the training set.
Revised: It is noticeable that AQM achieved high performance even with a retrieval-based approach in GuessWhat by making the candidate set of questions form the training set.

Original: However, Han et al.
Revised: However, Han et al.

Original: (2017) pointed out that there exist dominant questions in GuessWhat which can be generally applied to all images (contexts), such as “is it left?’’
Revised: (2017) pointed out that there exist dominant questions in GuessWhat which can be generally applied to all images (contexts), such as “is it left?

Original: or “is it human?’’.
Revised: or “is it human?.

Original: Since GuessWhich is a more complicated task where questions dominant for the game are less likely to exist, it is another reason why the original AQM is difﬁcult to be applied.
Revised: Since GuessWhich is a more complicated task where questions dominant for the game are less likely to exist, it is another reason why the original AQM is difﬁcult to be applied.

Original: To address this, we propose a more generalized version of AQM, dubbed AQM+.
Revised: To address this, we propose a more generalized version of AQM, dubbed AQM+.

Original: Compared to the original AQM, the proposed AQM+ can easily handle the increased number of questions, answers, and candidate classes by employing an approximation based on subset sampling.
Revised: Compared to the original AQM, the proposed AQM+ can easily handle the increased number of questions, answers, and candidate classes by employing an approximation based on subset sampling.

Original: Particularly, unlike AQM, AQM+ generates candidate questions and answers at every turn, and then select one of them to ask a question.
Revised: Particularly, unlike AQM, AQM+ generates candidate questions and answers at every turn, and then selects one of them to ask a question.

Original: Because our algorithm considers the previous history of the dialog, AQM+ can generate a more contextual question.
Revised: Because our algorithm considers the previous history of the dialog, AQM+ can generate a more contextual question.

Original: To understand the practicality and demonstrate the superior performance of our method, we conduct extensive experiments and quantitative analysis on
Revised: To understand the practicality and demonstrate the superior performance of our method, we conduct extensive experiments and quantitative analysis on

Original: GuessWhich.
Revised: GuessWhich.

Original: Experimental results show that our model could successfully deal with the answers in sentence and signiﬁcantly decrease 61.5% of the error while the SL and RL methods decrease less than 6% of the error.
Revised: Experimental results show that our model could successfully deal with the answers in sentence and signiﬁcantly decrease 61.5% of the error while the SL and RL methods decrease less than 6% of the error.

Original: The ablation study shows that our information gain approximation is reasonable.
Revised: The ablation study shows that our information gain approximation is reasonable.

Original: Increasing the number of sampling by eight times brought only a marginal improvement of percentile mean rank (PMR) from 94.63% to 94.79%, which indicates that our model can effectively approximate the distribution over the large search space with a small number of sampling.
Revised: Increasing the number of sampling by eight times brought only a marginal improvement of percentile mean rank (PMR) from 94.63% to 94.79%, which indicates that our model can effectively approximate the distribution over the large search space with a small number of sampling.

Original: Overall, our experimental results provide meaningful insights on how AQM framework can further provide an additional improvement on top of the SL and RL approaches
Revised: Overall, our experimental results provide meaningful insights on how AQM framework can further provide an additional improvement on top of the SL and RL approaches

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We propose AQM+ that extends the AQM framework toward the more general and complicated tasks.
Revised: • We propose AQM+ that extends the AQM framework toward the more general and complicated tasks.

Original: AQM+ can handle a more complicated problem where the number of candidate classes is extremely large.
Revised: AQM+ can handle a more complicated problem where the number of candidate classes is extremely large.

Original: • At every turn, AQM+ generates a question considering the context of the previous dialog, which is desirable in practice.
Revised: • At every turn, AQM+ generates a question considering the context of the previous dialog, which is desirable in practice.

Original: In particular, AQM+ generates candidate questions and answers at every turn to ask an appropriate question in the context.
Revised: In particular, AQM+ generates candidate questions and answers at every turn to ask an appropriate question in the context.

Original: • AQM+ outperforms comparative deep learning models by a large margin in Guesswhich, a challenging task-oriented visual dialog task.
Revised: • AQM+ outperforms comparative deep learning models by a large margin in Guesswhich, a challenging task-oriented visual dialog task.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: A task-oriented visual dialog problem has recently been paid attention in the ﬁeld of computer vision and natural language processing (Kim et al., 2017).
Revised: A task-oriented visual dialog problem has recently been paid attention in the ﬁeld of computer vision and natural language processing (Kim et al., 2017).

Original: GuessWhat is one of the famous task-oriented dialog tasks, where the goal is to ﬁgure out a target object in the image through a dialog that the answerer has in mind (de Vries et al., 2017).
Revised: GuessWhat is one of the famous taskoriented dialog tasks, where the goal is to ﬁgure out a target object in the image through a dialog that the answerer has in mind (de Vries et al., 2017).

Original: However, GuessWhat is relatively an easy task because it only allows the answer form of yes or no.
Revised: However, GuessWhat is relatively an easy task because it only allows the answer form of yes or no.

Original: The baseline visual question answering (VQA) model achieves 78.5%.
Revised: The baseline visual question answering (VQA) model achieves 78.5%.

Original: In the object guessing task (i.e., GuessWhat task itself), the state-of-the-art averaged accuracy of SL, RL (Zhang et al., 2018b), and AQM (Lee et al., 2018) reached 44.6% and 60.8%, and 72.9% at the 5th round, respectively.
Revised: In the object guessing task (i.e., GuessWhat task itself), the state-of-the-art averaged accuracy of SL, RL (Zhang et al., 2018b), and AQM (Lee et al., 2018) reached 44.6% and 60.8%, and 72.9% at the 5th round, respectively.

Original: Random guessing baseline has an accuracy of 16.0% (Han et al., 2017), thus RL algorithms achieve 53.3% error decrease, whereas AQM achieves 67.7%.
Revised: Random guessing baseline has an accuracy of 16.0% (Han et al., 2017), thus RL algorithms achieve 53.3% error decrease, whereas AQM achieves 67.7%.

Original: GuessWhich is a cooperative two-player game that one player tries to ﬁgure out an image out of 9,628 that another has in mind (Das et al., 2017b).
Revised: GuessWhich is a cooperative two-player game that one player tries to ﬁgure out an image out of 9,628 that another has in mind (Das et al., 2017b).

Original: GuessWhich uses Visual Dialog dataset (Das et al., 2017a) which includes human dialogs on MSCOCO images (Lin et al., 2014) as well as the captions that are generated.
Revised: GuessWhich uses Visual Dialog dataset (Das et al., 2017a) which includes human dialogs on MSCOCO images (Lin et al., 2014) as well as the captions that are generated.

Original: Although GuessWhich is similar to GuessWhat, it is more challenging in every task including asking a question, giving an answer, and guessing the target class.
Revised: Although GuessWhich is similar to GuessWhat, it is more challenging in every task including asking a question, giving an answer, and guessing the target class.

Original: For example, unlike GuessWhat that can be answered in yes or no, the answer can be an arbitrary sentence in GuessWhich.
Revised: For example, unlike GuessWhat that can be answered in yes or no, the answer can be an arbitrary sentence in GuessWhich.

Original: Therefore, the VQA task in the Visual Dialog dataset is much studied than the GuessWhat dataset (Lu et al., 2017; Seo et al., 2017b).
Revised: Therefore, the VQA task in the Visual Dialog dataset is much studied than the GuessWhat dataset (Lu et al., 2017; Seo et al., 2017b).

Original: Similar to GuessWhat, SL and RL approaches have been applied to solve the GuessWhich task and they showed a moderate increase in performance (Das et al., 2017b; Jain et al., 2018;
Revised: Similar to GuessWhat, SL and RL approaches have been applied to solve the GuessWhich task and they showed a moderate increase in performance (Das et al., 2017b; Jain et al., 2018;

Original: Zhang et al., 2018a).
Revised: Zhang et al., 2018a).

Original: However, based on the authors’ recent Github implementation 1 of the papers in ICCV (Das et al., 2017b), SL and RL methods have shown that only 6% of error is diminished through the dialog compared to the zeroth turn baselines which only use generated caption.
Revised: However, based on the authors’ recent Github implementation 1 of the papers in ICCV (Das et al., 2017b), SL and RL methods have shown that only 6% of error is diminished through the dialog compared to the zeroth turn baselines which only use generated caption.

Original:  A LGORITHM : AQM+ 3. P ROBLEM S ETTING
Revised:  A LGORITHM : AQM+ 3. P ROBLEM S ETTING

Original: In our experiments, a questioner bot (Qbot) and an answerer bot (Abot) cooperatively communicate to achieve the goal via natural language.
Revised: In our experiments, a questioner bot (Qbot) and an answerer bot (Abot) cooperatively communicate to achieve the goal via natural language.

Original: Under the AQM framework, at each turn t , Qbot generates an appropriate question q t and guesses the target class c given a previous history of the dialog h t − 1 = ( q 1: t − 1 , a 1: t − 1 , h 0 ) .
Revised: Under the AQM framework, at each turn t , Qbot generates an appropriate question q t and guesses the target class c given a previous history of the dialog h t − 1 = ( q 1: t − 1 , a 1: t − 1 , h 0 ) .

Original: Here, a t is the t -th answer and h 0 is an initial context that can be obtained before the start of the dialog.
Revised: Here, a t is the t -th answer and h 0 is an initial context that can be obtained before the start of the dialog.

Original: We refer to the random variables of target class and the tth answer as C and A t , respectively.
Revised: We refer to the random variables of target class and the t -th answer as C and A t , respectively.

Original: Note that the t -th question is not a random variable in our information gain calculation.
Revised: Note that the t -th question is not a random variable in our information gain calculation.

Original: To distinguish from the random variables, we use a bold face for a set notation of target class, question, and answers; i.e. C , Q , and A .
Revised: To distinguish from the random variables, we use a bold face for a set notation of target class, question, and answers; i.e. C , Q , and A .

Original: Figure 1 explains the AQM+ algorithm applied to GuessWhich game.
Revised: Figure 1 explains the AQM+ algorithm applied to GuessWhich game.

Original: In Figure 1, c is the image with three elephants, q 1 is “Are there many people?”, a 1 is “Yes it is.”, a 2 is “How many elephants?”, and h 0 is “There are elephants walking in the zoo.”
Revised: In Figure 1, c is the image with three elephants, q 1 is “Are there many people?”, a 1 is “Yes it is.”, a 2 is “How many elephants?”, and h 0 is “There are elephants walking in the zoo.”

Original: In GuessWhich game, C is the set of test images whose size is 9,628.
Revised: In GuessWhich game, C is the set of test images whose size is 9,628.

Original: The size of Q and A is theoretically inﬁnity as questions and answers can be more than one word.
Revised: The size of Q and A is theoretically inﬁnity as questions and answers can be more than one word.

Original: 3. P RELIMINARY : SL, RL, AND AQM A PPROACHES
Revised: 3. P RELIMINARY : SL, RL, AND AQM A PPROACHES

Original: In SL and RL approaches (Das et al., 2017b; Jain et al., 2018; Zhang et al., 2018a), Qbot consists of two RNN modules.
Revised: In SL and RL approaches (Das et al., 2017b; Jain et al., 2018; Zhang et al., 2018a), Qbot consists of two RNN modules.

Original: One is “Qgen”, a question generator ﬁnding the solution that maximizes its distribution p † ; i.e. q ∗ t = argmax p † ( q t | h t − 1 ) .
Revised: One is “Qgen”, a question generator ﬁnding the solution that maximizes its distribution p † ; i.e. q ∗ t = argmax p † ( q t | h t − 1 ) .

Original: The other is a “Qscore”, a class guesser using score function for each class f ‡
Revised: The other is a “Qscore”, a class guesser using score function for each class f ‡

Original: ( c | h t ) .
Revised: ( c | h t ) .

Original: Two RNN modules can either be fully separated two RNNs (Strub et al., 2017), or share some recurrent layers but have a different output layer for each (Das et al., 2017b).
Revised: Two RNN modules can either be fully separated two RNNs (Strub et al., 2017), or share some recurrent layers but have a different output layer for each (Das et al., 2017b).

Original: On the other hand, in the previous AQM approach (Lee et al., 2018), these two RNN-based models are substituted to the calculation that explicitly ﬁnds an analytic solution.
Revised: On the other hand, in the previous AQM approach (Lee et al., 2018), these two RNN-based models are substituted to the calculation that explicitly ﬁnds an analytic solution.

Original: It ﬁnds a question that maximizes information gain or mutual information ˜
Revised: It ﬁnds a question that maximizes information gain or mutual information ˜

Original: I , i.e. q ∗ t = argmax q t [Equation]
Revised: I , i.e. q ∗ t = argmax q t ∈ Q fix ˜

Original: Q
Revised: I

Original: [Equation] ˜ I [ C, A t ; q 
Revised: [ C, A t ; q t , h t − 1 ]

Original: [Table]
Revised: [Table]

Original: an information gain calculation function by Equation

Original: [Table]
Revised: [Table]

Original: [Table]

Original: indA Like SL, aprxAgen is trained from training data depA
Revised: Like SL, aprxAgen is trained from training data depA

Original: Like RL, aprxAgen is trained from the dialog with Abot trueA aprxAgen shares the parameter with Abot where ˜ I
Revised: Like RL, aprxAgen is trained from the dialog with Abot trueA aprxAgen shares the parameter with Abot where ˜ I

Original: [ C, A t ; q t , h t − 1 ] = (cid:88) 
Revised: [ C, A t ; q t , h t − 1 ] = (cid:88) c

Original: [Equation]
Revised: [Equation]

Original:  Here, a posterior function ˆ p can be calculated with a following equation in a sequential way, where ˆ p (cid:48) is a prior function given h 0 .
Revised: C (cid:88) [Equation] [Equation] Here, a posterior function ˆ p can be calculated with a following equation in a sequential way, where ˆ p (cid:48) is a prior function given h 0 .

Original: [Equation]
Revised: [Equation]

Original: In AQM, Equation 1 and Equation 2 can be explicitly calculated from the model.
Revised: In AQM, Equation 1 and Equation 2 can be explicitly calculated from the model.

Original: For ease of reference, let us name every component one by one.
Revised: For ease of reference, let us name every component one by one.

Original: A module that calculates an information gain ˜ I is referred to as “Qinfo” and a module that ﬁnds an approximated answer distribution ˜ p ( a t | c, q t , h t − 1 ) is referred to as “aprxAgen”.
Revised: A module that calculates an information gain ˜ I is referred to as “Qinfo” and a module that ﬁnds an approximated answer distribution ˜ p ( a t | c, q t , h t − 1 ) is referred to as “aprxAgen”.

Original: In AQM, aprxAgen is a model distribution that Qbot has in mind where the target is the true distribution of an answer generator ¯ p ( a t | c, q t , h t − 1 ) , which is referred to as “Agen”.
Revised: In AQM, aprxAgen is a model distribution that Qbot has in mind where the target is the true distribution of an answer generator ¯ p ( a t | c, q t , h t − 1 ) , which is referred to as “Agen”.

Original: Finally, “Qpost” denotes a posterior ˆ p calculation module for guessing a target class.
Revised: Finally, “Qpost” denotes a posterior ˆ p calculation module for guessing a target class.

Original: As AQM use full set of C and A , the complexity depends on the size of C and A .
Revised: As AQM uses full set of C and A , the complexity depends on the size of C and A .

Original: For the question selection, AQM uses a predeﬁned set of candidate questions ( Q fix ), which is not changed for a different turn.
Revised: For the question selection, AQM uses a predeﬁned set of candidate questions ( Q fix ), which is not changed for a different turn.

Original: 3. AQM+ A LGORITHM
Revised: 3. AQM+ A LGORITHM

Original: In this paper, we propose AQM+ algorithm, which uses sampling-based approximation, for tackling the large-scale task-oriented dialog problem.
Revised: In this paper, we propose AQM+ algorithm, which uses sampling-based approximation, for tackling the large-scale task-oriented dialog problem.

Original: The core differences of AQM+ from the previous AQM are summarized as follows:
Revised: The core differences of AQM+ from the previous AQM are summarized as follows:

Original: • The candidate question set Q t,gen is sampled from p † ( q t | h t − 1 ) using a beam search at every turn.
Revised: • The candidate question set Q t,gen is sampled from p † ( q t | h t − 1 ) using a beam search at every turn.

Original: Previously, Lee et al.
Revised: Previously, Lee et al.

Original: (2018) used a predeﬁned set of candidate questions Q fix .
Revised: (2018) used a predeﬁned set of candidate questions Q fix .

Original: For example, one way to obtain Q fix is to select questions from the training dataset randomly, called “randQ”.
Revised: For example, one way to obtain Q fix is to select questions from the training dataset randomly, called “randQ”.

Original: • The answerer model (aprxAgen, ˜ p ) that Qbot has in mind is not a binary classiﬁer (yes/no) but an RNN generator.
Revised: • The answerer model (aprxAgen, ˜ p ) that Qbot has in mind is not a binary classiﬁer (yes/no) but an RNN generator.

Original: In addition, aprxAgen does not assume ˜ p ( a t | c, q t ) =
Revised: In addition, aprxAgen does not assume ˜ p ( a t | c, q t ) =

Original: ˜ p ( a t | c, q t , h t − 1 ) , which is not even an appropriate assumption when the previous and current questions are sequentially related.
Revised: ˜ p ( a t | c, q t , h t − 1 ) , which is not even an appropriate assumption when the previous and current questions are sequentially related.

Original: For example, p ( a 2 = “yes” | c, q 2 = “is left?” )
Revised: For example, p ( a 2 = “yes” | c, q 2 = “is left?” )

Original: (cid:54) = p ( a 2 = “yes” | c, q 2 = “is left?” , a 1 = “yes” , q 1 = “is right?” ) .
Revised: (cid:54) = p ( a 2 = “yes” | c, q 2 = “is left?” , a 1 = “yes” , q 1 = “is right?” ) .

Original: Regardless of the left term, the probability of the right term is almost zero.
Revised: Regardless of the left term, the probability of the right term is almost zero.

Original: • To approximate the information gain of each question, the subsets of A and C are also sampled at every turn.
Revised: • To approximate the information gain of each question, the subsets of A and C are also sampled at every turn.

Original: The previous algorithm used full set of A and C .
Revised: The previous algorithm used full set of A and C .

Original: We describe an additional explanation on our information gain approximation, infogain topk as below.
Revised: We describe an additional explanation on our information gain approximation, infogain topk as below.

Original: Infogain topk The equation for Infogain topk is as follows:
Revised: Infogain topk The equation for Infogain topk is as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˆ p reg and ˜ p reg is a normalized version of ˆ p over C t,topk and ˜ p over A t,topk ( q t ) , respectively.
Revised: where ˆ p reg and ˜ p reg is a normalized version of ˆ p over C t,topk and ˜ p over A t,topk ( q t ) , respectively.

Original: Here, ˜ p (cid:48) reg is obtained by using both ˆ p reg and ˜ p reg as follows:
Revised: Here, ˜ p (cid:48) reg is obtained by using both ˆ p reg and ˜ p reg as follows:

Original: [Equation]
Revised: [Equation]

Original: Each set is constructed by the following procedures
Revised: Each set is constructed by the following procedures

Original: • C t,topk ← top-K posterior test images (from Qpost ˆ p ( c | h t − 1 ) )
Revised: • C t,topk ← top-K posterior test images (from Qpost ˆ p ( c | h t − 1 ) )

Original: • Q t,gen ← top-K likelihood questions using the beam search (from Qgen p † ( q t | h t − 1 ) )
Revised: • Q t,gen ← top-K likelihood questions using the beam search (from Qgen p † ( q t | h t − 1 ) )

Original: • A t,topk ( q t ) ← top-1 generated answers from aprxAgen for each question q t and each class in C t,topk (from aprxAgen ˜ p ( a t | c, q t , h t − 1 ) )
Revised: • A t,topk ( q t ) ← top-1 generated answers from aprxAgen for each question q t and each class in C t,topk (from aprxAgen ˜ p ( a t | c, q t , h t − 1 ) )

Original: Top-K samples may lead our approximation to be biased toward plausible (high-probability) candidate classes and plausible candidate answers.
Revised: Top-K samples may lead our approximation to be biased toward plausible (high-probability) candidate classes and plausible candidate answers.

Original: However, we chose to use top-K samples because our main goal is to reduce the entropy over plausible candidate classes and answers, not over the whole candidate classes and answers.
Revised: However, we chose to use top-K samples because our main goal is to reduce the entropy over plausible candidate classes and answers, not over the whole candidate classes and answers.

Original: In general, the AQM+ algorithm can deal with various problems where | C t,topk | , | Q t,gen | , and | A t,topk ( q t ) | are all different.
Revised: In general, the AQM+ algorithm can deal with various problems where | C t,topk | , | Q t,gen | , and | A t,topk ( q t ) | are all different.

Original: Here, | · | denotes the cardinality of a set.
Revised: Here, | · | denotes the cardinality of a set.

Original: We can vary the size of each set and control the complexity of the AQM+ algorithm.
Revised: We can vary the size of each set and control the complexity of the AQM+ algorithm.

Original: In our experiments, however, we mainly considered the problem when | C t,topk | = | Q t,gen | = |
Revised: In our experiments, however, we mainly considered the problem when | C t,topk | = | Q t,gen | = |

Original: A t,topk ( q t ) | .
Revised: A t,topk ( q t ) | .

Original: More speciﬁcally, | C t,topk | is equal to | A t,topk ( q t ) | because our model ﬁnds a single best answer a t given a pair ( q t , c ) that maximizes ˜ p ( a t | c, q t , h t − 1 ) .
Revised: More speciﬁcally, | C t,topk | is equal to | A t,topk ( q t ) | because our model ﬁnds a single best answer a t given a pair ( q t , c ) that maximizes ˜ p ( a t | c, q t , h t − 1 ) .

Original: Therefore, | A t,topk | = | Q t,gen | · | C t,topk | per information gain calculation where A t,topk =
Revised: Therefore, | A t,topk | = | Q t,gen | · | C t,topk | per information gain calculation where A t,topk =

Original: { A t,topk ( q t ) | q t ∈ Q t,gen } .
Revised: { A t,topk ( q t ) | q t ∈ Q t,gen } .

Original: For the detailed explanation, see Algorithm 1 in Appendix A.
Revised: For the detailed explanation, see Algorithm 1 in Appendix A.

Original: We also explain the extended sampling method on candidate answers for cases where A (cid:54) =
Revised: We also explain the extended sampling method on candidate answers for cases where A (cid:54) =

Original: C is required.
Revised: C is required.

Original: In the extended method, aprxAgen ﬁrst generates top-m answers for each candidate question and each candidate class, where m is the smallest integer satisfying | A | ≤ | C | · m .
Revised: In the extended method, aprxAgen ﬁrst generates top-m answers for each candidate question and each candidate class, where m is the smallest integer satisfying | A | ≤ | C | · m .

Original: After that, the candidate answers are randomly removed, leaving only | A | answers.
Revised: After that, the candidate answers are randomly removed, leaving only | A | answers.

Original: 3. L EARNING
Revised: 3. L EARNING

Original: In all SL, RL, and AQM frameworks, Qbot needs to be trained to approximate the answer-generating probability of Abot.
Revised: In all SL, RL, and AQM frameworks, Qbot needs to be trained to approximate the answer-generating probability of Abot.

Original: In AQM approach, aprxAgen does not share the parameters with Agen, and therefore also needs to be trained to approximate Agen.
Revised: In AQM approach, aprxAgen does not share the parameters with Agen, and therefore also needs to be trained to approximate Agen.

Original: AQM can train aprxAgen by the learning strategy of the SL or RL approach.
Revised: AQM can train aprxAgen by the learning strategy of the SL or RL approach.

Original: We explain two learning strategies of AQM framework below: indA and depA. In SL approach, Qgen and Qscore are trained from the training data, which have the same or similar distribution to that of the training data used in training Abot.
Revised: We explain two learning strategies of AQM framework below: indA and depA. In SL approach, Qgen and Qscore are trained from the training data, which have the same or similar distribution to that of the training data used in training Abot.

Original: Likewise, in indA setting of AQM approach, aprxAgen is trained from the training data.
Revised: Likewise, in indA setting of AQM approach, aprxAgen is trained from the training data.

Original: In RL approach, Qbot uses dialogs made by the conversation of Qbot and Abot and the result of the game as the objective function (i.e. reward).
Revised: In RL approach, Qbot uses dialogs made by the conversation of Qbot and Abot and the result of the game as the objective function (i.e. reward).

Original: Likewise, in depA setting of AQM approach, aprxAgen is trained from the questions in the training data and following answers obtained in the conversation between Qbot and
Revised: Likewise, in depA setting of AQM approach, aprxAgen is trained from the questions in the training data and following answers obtained in the conversation between Qbot and

Original: Abot.
Revised: Abot.

Original: We also use the term trueA, referring to the setting where aprxAgen is the same as Agen, i.e. they share the same parameters.
Revised: We also use the term trueA, referring to the setting where aprxAgen is the same as Agen, i.e. they share the same parameters.

Original: Both the previous AQM algorithm and the proposed AQM+ algorithm use these learning strategies.
Revised: Both the previous AQM algorithm and the proposed AQM+ algorithm use these learning strategies.

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETTING
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETTING

Original: GuessWhich Task GuessWhich is a two player game played by Qbot and Abot.
Revised: GuessWhich Task GuessWhich is a two player game played by Qbot and Abot.

Original: The goal of GuessWhich is to ﬁgure out a correct answer out of 9,628 test images by asking a sequence of questions.
Revised: The goal of GuessWhich is to ﬁgure out a correct answer out of 9,628 test images by asking a sequence of questions.

Original: Abot can see the randomly assigned target image, which is unknown to Qbot.
Revised: Abot can see the randomly assigned target image, which is unknown to Qbot.

Original: Qbot only observes a caption of the image generated from Neuraltalk2 (Vinyals & Le, 2015).
Revised: Qbot only observes a caption of the image generated from Neuraltalk2 (Vinyals & Le, 2015).

Original: To achieve the goal, Qbot asks a series of questions, to which Abot responds with a sentence.
Revised: To achieve the goal, Qbot asks a series of questions, to which Abot responds with a sentence.

Original: Comparative Models We compare AQM+ with three comparative models, SL-Q, RL-Q, and RLQA (Das et al., 2017b).
Revised: Comparative Models We compare AQM+ with three comparative models, SL-Q, RL-Q, and RLQA (Das et al., 2017b).

Original: In SL-Q, Qbot and Abot are trained separately from the training data.
Revised: In SL-Q, Qbot and Abot are trained separately from the training data.

Original: In
Revised: In

Original: RL-Q, Qbot is initialized by the Qbot trained by SL-Q and then is ﬁne-tuned by RL.
Revised: RL-Q, Qbot is initialized by the Qbot trained by SL-Q and then is ﬁne-tuned by RL.

Original: Abot is the same as the Abot trained by SL-Q, and is not ﬁne-tuned further.
Revised: Abot is the same as the Abot trained by SL-Q, and is not ﬁne-tuned further.

Original: In the original paper (Das et al., 2017b), it was referred to as Frozen-A.
Revised: In the original paper (Das et al., 2017b), it was referred to as Frozen-A.

Original: By the way, in an RL-QA setting, not only Qbot but also
Revised: By the way, in an RL-QA setting, not only Qbot but also

Original: Abot is concurrently trained with Qbot.
Revised: Abot is concurrently trained with Qbot.

Original: In the original paper, it was referred to as RL-full-QAf.
Revised: In the original paper, it was referred to as RL-full-QAf.

Original: We also compare our AQM+ with “Guesser” algorithm.
Revised: We also compare our AQM+ with “Guesser” algorithm.

Original: Guesser asks a question generated from SL-Q algorithm and calculates posterior by Qpost of AQM+.
Revised: Guesser asks a question generated from SL-Q algorithm and calculates posterior by Qpost of AQM+.

Original: Non-delta vs. Delta
Revised: Non-delta vs. Delta

Original: Hyperparameter Setting The important issue in our GuessWhich experiment is delta setting.
Revised: Hyperparameter Setting The important issue in our GuessWhich experiment is delta setting.

Original: In the paper of Das et al.
Revised: In the paper of Das et al.

Original: (2017b), SL-Q, RL-Q, and RL-QA algorithms achieve moderate increases of the performance.
Revised: (2017b), SL-Q, RL-Q, and RL-QA algorithms achieve moderate increases of the performance.

Original: In SL-Q, 88.5% of percentile mean rank (PMR) is improved to 90.9%.
Revised: In SL-Q, 88.5% of percentile mean rank (PMR) is improved to 90.9%.

Original: In RL-QA, 90.6% of PMR is improved to 93.3%.
Revised: In RL-QA, 90.6% of PMR is improved to 93.3%.

Original: Here, 93.3% of PMR at the zeroth turn means that the model can predict the correct image to be more likely than the other 8,983 images out of 9,628 candidates after exploiting the caption information solely.
Revised: Here, 93.3% of PMR at the zeroth turn means that the model can predict the correct image to be more likely than the other 8,983 images out of 9,628 candidates after exploiting the caption information solely.

Original: However, Das et al.
Revised: However, Das et al.

Original: (2017b) found that another hyperparameter setting, delta, makes much progress on their algorithm.
Revised: (2017b) found that another hyperparameter setting, delta, makes much progress on their algorithm.

Original: Delta setting refers to different weights on loss and learning decay rate.
Revised: Delta setting refers to different weights on loss and learning decay rate.

Original: Based on the authors’ recent report on Github, SL-Q and RL-QA methods have shown that less than 6% of error is diminished through the dialog compared to the zeroth turn baseline which only uses generated caption.
Revised: Based on the authors’ recent report on Github, SL-Q and RL-QA methods have shown that less than 6% of error is diminished through the dialog compared to the zeroth turn baseline which only uses generated caption.

Original: The
Revised: The

Original: PMR of the target (class) image which only uses the caption is around 95.5, but the dialog does not improve the PMR to more than 95.8.
Revised: PMR of the target (class) image which only uses the caption is around 95.5, but the dialog does not improve the PMR to more than 95.8.

Original: We use both non-delta setting (the setting in the original paper) and delta setting (the setting in Github) to test the performance of AQM+.
Revised: We use both non-delta setting (the setting in the original paper) and delta setting (the setting in Github) to test the performance of AQM+.

Original: Other Experimental Setting As shown in Figure 2, our model uses ﬁve modules, Qgen, Qscore, aprxAgen, Qinfo, and Qpost.
Revised: Other Experimental Setting As shown in Figure 2, our model uses ﬁve modules, Qgen, Qscore, aprxAgen, Qinfo, and Qpost.

Original: We use the same Qgen and Qscore modules as the comparative SL-Q model.
Revised: We use the same Qgen and Qscore modules as the comparative SLQ model.

Original: In Visual Dialog, Qgen and Qscore share one RNN structure and have different outputlayers [Table] w
Revised: In Visual Dialog, Qgen and Qscore share one RNN structure and have different output

Original: [Table]
Revised: [Table]

Original: depA

Original: AQM
Revised: AQM

Original: [Table]
Revised: [Table]

Original: w

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original:  for each.
Revised: layers for each.

Original: The prior function is obtained from ˆ p (cid:48)
Revised: The prior function is obtained from ˆ p (cid:48)

Original: ( c | h 0 )
Revised: ( c | h 0 )

Original: ∝ exp ( λ · f ‡ ( c | h 0 )) using Qscore, where λ is a balancing hyperparameter between prior and likelihood.
Revised: ∝ exp ( λ · f ‡ ( c | h 0 )) using Qscore, where λ is a balancing hyperparameter between prior and likelihood.

Original: We set | C t,topk | = | Q t,gen | =
Revised: We set | C t,topk | = | Q t,gen | =

Original: | A t,topk ( q t ) | = 20.
Revised: | A t,topk ( q t ) | = 20.

Original: The epoch for SL-Q is 60.
Revised: The epoch for SL-Q is 60.

Original: The epoch for RL-Q and RL-QA is 20 for non-delta, and 15 for delta, respectively.
Revised: The epoch for RL-Q and RL-QA is 20 for non-delta, and 15 for delta, respectively.

Original: 4. C OMPARATIVE R ESULTS
Revised: 4. C OMPARATIVE R ESULTS

Original: Figure 3 shows the PMR of the target image for our AQM+ and comparative models across the rounds.
Revised: Figure 3 shows the PMR of the target image for our AQM+ and comparative models across the rounds.

Original: Figure 3a corresponds to the non-delta setting in the original paper (Das et al., 2017b) and
Revised: Figure 3a corresponds to the non-delta setting in the original paper (Das et al., 2017b) and

Original: Figure 3b corresponds to the delta setting proposed in the Github code.
Revised: Figure 3b corresponds to the delta setting proposed in the Github code.

Original: We see that SL-Q and RL-QA do not signiﬁcantly improve the performance after a few rounds, especially for the delta setting.
Revised: We see that SL-Q and RL-QA do not signiﬁcantly improve the performance after a few rounds, especially for the delta setting.

Original: In delta setting, SL-Q increases their performance from 95.45% to 95.72% at 10th round, and RL-QA increases their performance from 95.44% to 95.69%.
Revised: In delta setting, SL-Q increases their performance from 95.45% to 95.72% at 10th round, and RL-QA increases their performance from 95.44% to 95.69%.

Original: It means that error drop of SL-Q and RL-QA algorithms is 5.74% and 5.33%, respectively.
Revised: It means that error drop of SL-Q and RL-QA algorithms is 5.74% and 5.33%, respectively.

Original: On the other hand,
Revised: On the other hand,

Original: AQM-indA increases its PMR from 95.45% to 96.53% at the ﬁfth round and reach 97.17% at the 10th round.
Revised: AQM-indA increases its PMR from 95.45% to 96.53% at the ﬁfth round and reaches 97.17% at the 10th round.

Original: Likewise, AQM-depA increases its PMR from 95.45% to 97.48% at the ﬁfth round and reach 98.25% at the 10th round, decreasing 61.5% of error.
Revised: Likewise, AQM-depA increases its PMR from 95.45% to 97.48% at the ﬁfth round and reach 98.25% at the 10th round, decreasing 61.5% of error.

Original: Note that Guesser w/
Revised: Note that Guesser w/

Original: indA achieves 96.37% at the 10th round, outperforming SL-Q by a signiﬁcant margin.
Revised: indA achieves 96.37% at the 10th round, outperforming SL-Q by a signiﬁcant margin.

Original: It shows that not only the question generation but also the guessing mechanism affects the performance degeneration of SL and RL algorithms.
Revised: It shows that not only the question generation but also the guessing mechanism affects the performance degeneration of SL and RL algorithms.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: No Caption Experiment We test our AQM+ algorithm where no caption information exists.
Revised: No Caption Experiment We test our AQM+ algorithm where no caption information exists.

Original: For the zeroth prediction, we simply replace the prior function from Qscore with a uniform function.
Revised: For the zeroth prediction, we simply replace the prior function from Qscore with a uniform function.

Original: Since
Revised: Since

Original: Qgen in either SL-Q or RL-QA is trained also assuming the existence of the caption, we tried two alternative settings to approximate experiments without a caption.
Revised: Qgen in either SL-Q or RL-QA is trained also assuming the existence of the caption, we tried two alternative settings to approximate experiments without a caption.

Original: The ﬁrst trial is the zero-caption experiment, where the caption vector is ﬁlled with zeros.
Revised: The ﬁrst trial is the zero-caption experiment, where the caption vector is ﬁlled with zeros.

Original: The second trial is the random-caption experiment, where the caption vector is replaced with a random caption vector, which is not related to the target image.
Revised: The second trial is the random-caption experiment, where the caption vector is replaced with a random caption vector, which is not related to the target image.

Original: Figure 4a shows that AQM+ performs well for both zero-caption and randomcaption setting.
Revised: Figure 4a shows that AQM+ performs well for both zero-caption and randomcaption setting.

Original: By contrast, SL-Q and RL-QA do not work at all.
Revised: By contrast, SL-Q and RL-QA do not work at all.

Original: It seems SL-Q and RL-QA are not trained on the situation where zero-caption vector or even totally wrong caption vector comes.
Revised: It seems SL-Q and RL-QA are not trained on the situation where zero-caption vector or even totally wrong caption vector comes.

Original: Though training SL-Q and RL-QA for these situations can increase their performance, it is evident that SL and RL algorithms are not robust to unexpected environments.
Revised: Though training SL-Q and RL-QA for these situations can increase their performance, it is evident that SL and RL algorithms are not robust to unexpected environments.

Original: Likewise, we also run no caption experiments for depA setting.
Revised: Likewise, we also run no caption experiments for depA setting.

Original: For more ablation studies, see Figure 7 in Appendix B.
Revised: For more ablation studies, see Figure 7 in Appendix B.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Random Candidate Answers Experiment One of our main arguments is that generating candidate questions from Qscore and candidate answers from aprxAgen at every turn makes AQM+ effectively deal with general and complicated task-oriented dialogs.
Revised: Random Candidate Answers Experiment One of our main arguments is that generating candidate questions from Qscore and candidate answers from aprxAgen at every turn makes AQM+ effectively deal with general and complicated task-oriented dialogs.

Original: Supporting the argument, we conducted the experiments under the setting where the answer set is randomly selected from the training data and then ﬁxed.
Revised: Supporting the argument, we conducted the experiments under the setting where the answer set is randomly selected from the training data and then ﬁxed.

Original: Random selection of candidate answers decreases the performance from 94.64% to 92.78% at indA, non-delta, and the 10th round.
Revised: Random selection of candidate answers decreases the performance from 94.64% to 92.78% at indA, non-delta, and the 10th round.

Original: Appendix B also includes a discussion on the setting with a predeﬁned candidate question set Q fix .
Revised: Appendix B also includes a discussion on the setting with a predeﬁned candidate question set Q fix .

Original: Number of
Revised: Number of

Original: QAC
Revised: QAC

Original: Experiment
Revised: Experiment

Original: We also changed the size of subset
Revised: We also changed the size of subset

Original: K = | Q t,gen | = |
Revised: K = | Q t,gen | = |

Original: A t,topk ( q t ) | = | C t,topk | to check our efﬁciency of information gain approximation, using non-delta setting.
Revised: A t,topk ( q t ) | = | C t,topk | to check our efﬁciency of information gain approximation, using non-delta setting.

Original: Figure 5a shows the experimental results.
Revised: Figure 5a shows the experimental results.

Original: Note that AQM+ with the setting of K = 1 corresponds to Guesser.
Revised: Note that AQM+ with the setting of K = 1 corresponds to Guesser.

Original: In the setting of non-delta and indA, 94.64% of PMR is achieved when K is 20, whereas 94.79% is achieved when K is 40.
Revised: In the setting of non-delta and indA, 94.64% of PMR is achieved when K is 20, whereas 94.79% is achieved when K is 40.

Original: Note that 8 times (2 x 2 x 2) complexity increase just improves 0.15% of PMR, showing the efﬁciency of the setting of K =20 in our experiments.
Revised: Note that 8 times (2 x 2 x 2) complexity increase just improves 0.15% of PMR, showing the efﬁciency of the setting of K =20 in our experiments.

Original: On the other hand, this result also implies that increasing K would make further improvement on the performance.
Revised: On the other hand, this result also implies that increasing K would make further improvement on the performance.

Original: Likewise, in depA setting, changing K from 20 to 40 increases the PMR from 97.44% to 97.77%.
Revised: Likewise, in depA setting, changing K from 20 to 40 increases the PMR from 97.44% to 97.77%.

Original: For more ablation studies, see Figure 8 in Appendix B.
Revised: For more ablation studies, see Figure 8 in Appendix B.

Original: We also changed the size of each subset, | Q t,gen | , | A t,topk ( q t ) | , and | C t,topk | .
Revised: We also changed the size of each subset, | Q t,gen | , | A t,topk ( q t ) | , and | C t,topk | .

Original: Figure 5b-d shows the results.
Revised: Figure 5b-d shows the results.

Original: | Q t,gen | has the most effect, whereas | A t,topk ( q t ) | has the least effect.
Revised: | Q t,gen | has the most effect, whereas | A t,topk ( q t ) | has the least effect.

Original: Generated Questions and Selected Images Figure 6 shows the top-k images selected by AQM+’s posterior.
Revised: Generated Questions and Selected Images Figure 6 shows the top-k images selected by AQM+’s posterior.

Original: Non-delta and indA setting is used.
Revised: Non-delta and indA setting is used.

Original: The ﬁgure shows that relevant images to the caption remained after few dialog turns.
Revised: The ﬁgure shows that relevant images to the caption remained after few dialog turns.

Original: The bottom number in the image denotes posterior of the image
Revised: The bottom number in the image denotes posterior of the image

Original: AQM+ thinks of.
Revised: AQM+ thinks of.

Original: We also compare selected examples of generated dialog of SL-Q, RL-QA, and
Revised: We also compare selected examples of generated dialog of SL-Q, RL-QA, and

Original: AQM+ w/ indA for delta setting.
Revised: AQM+ w/ indA for delta setting.

Original: See Figure 10 in Appendix C for the results.
Revised: See Figure 10 in Appendix C for the results.

Original:  D ISCUSSION 5. D IFFICULTY OF G UESS W HICH
Revised:  D ISCUSSION 5. D IFFICULTY OF G UESS W HICH

Original: According to our results, we infer that PMR degradation of comparative SL and RL models during the dialog is not caused by forgetting dialog context to ask an appropriate question.
Revised: According to our results, we infer that PMR degradation of comparative SL and RL models during the dialog is not caused by forgetting dialog context to ask an appropriate question.

Original: Comparative results between AQM+ and Guesser show that the improvement from AQM+’s
Revised: Comparative results between AQM+ and Guesser show that the improvement from AQM+’s

Original: Qpost is signiﬁcant, which implies that the major constraint of SL and RL is the limited capacity of RNN and its softmax score function.
Revised: Qpost is signiﬁcant, which implies that the major constraint of SL and RL is the limited capacity of RNN and its softmax score function.

Original: Another reason for the poor performance lies in the current status of VQA models.
Revised: Another reason for the poor performance lies in the current status of VQA models.

Original: According to
Revised: According to

Original: Das et al. (2017a), the model structure used in  our experiments can already reach 41.2% for answer retrieval accuracy from 100 candidate answers, solely using the question without exploiting image and history information.
Revised: Das et al. (2017a), they discovered a variety of models, one of which is used in both the study of Das et al. (2017b) and our experiments, and can already reach 41.2% for answer retrieval accuracy from 100 candidate answers, solely using the question without exploiting image and history information.

Original: Fully exploiting these factors, however, increases the performance only slightly to 45.5%.
Revised: Fully exploiting these factors, however, increases the performance only slightly to 45.5%.

Original: As discrimination on different images relies on image and history information, Qbot suffers to gain meaningful information through the dialog.
Revised: As discrimination on different images relies on image and history information, Qbot suffers to gain meaningful information through the dialog.

Original: Therefore, applying AQM+ to the GuessWhich problem means that we not only solve a very complicated problem, but also ﬁnd that the AQM framework is applicable to the situation where the answer has high uncertainty.
Revised: Therefore, applying AQM+ to the GuessWhich problem means that we not only solve a very complicated problem, but also ﬁnd that the AQM framework is applicable to the situation where the answer has high uncertainty.

Original: 5. N OTES ON C OMPARATIVE A NALYSIS
Revised: 5. N OTES ON C OMPARATIVE A NALYSIS

Original: Fine-tuning both Qbot and Abot through RL Though RL-QA is the main setting in the work of Das et al.
Revised: Fine-tuning both Qbot and Abot through RL Though RL-QA is the main setting in the work of Das et al.

Original: (2017b), there are some reports indicating that ﬁne-tuning both Qbot and Abot is unfair (de Vries et al., 2017; Han et al., 2017).
Revised: (2017b), there are some reports indicating that ﬁne-tuning both Qbot and Abot is unfair (de Vries et al., 2017; Han et al., 2017).

Original: If the distribution of Abot is not ﬁxed during RL,
Revised: If the distribution of Abot is not ﬁxed during RL,

Original: Qbot and Abot can make their own language which is not compatible to natural language (Kottur et al., 2017).
Revised: Qbot and Abot can make their own language which is not compatible to natural language (Kottur et al., 2017).

Original: To prevent this problem, many studies added the objective function of language model during RL (Zhu et al., 2017; Das et al., 2017b).
Revised: To prevent this problem, many studies added the objective function of language model during RL (Zhu et al., 2017; Das et al., 2017b).

Original: However, even though the generated dialog is tuned to be like human dialog, the performance of RL-QA on the conversation with human would decrease compared to SL-Q, because the distribution of Abot become far from human’s
Revised: However, even though the generated dialog is tuned to be like human dialog, the performance of RL-QA on the conversation with human would decrease compared to SL-Q, because the distribution of Abot become far from human’s

Original: (Chattopadhyay et al., 2017; Lee et al., 2018).
Revised: (Chattopadhyay et al., 2017; Lee et al., 2018).

Original: Moreover, achieving a good performance by ﬁne-tuning both Qbot and Abot is much easier than ﬁne-tuning only Qbot (Zhu et al., 2017; Han et al., 2017).
Revised: Moreover, achieving a good performance by ﬁne-tuning both Qbot and Abot is much easier than ﬁne-tuning only Qbot (Zhu et al., 2017; Han et al., 2017).

Original: Thus, it is reasonable to compare AQM+ w/ indA and AQM+ w/ depA with SL-Q and RL-Q, respectively.
Revised: Thus, it is reasonable to compare AQM+ w/ indA and AQM+ w/ depA with SL-Q and RL-Q, respectively.

Original: Compuational Cost AQM+ at K =20 uses 20 × 20 × 20 calculations for information gain.
Revised: Compuational Cost AQM+ at K =20 uses 20 × 20 × 20 calculations for information gain.

Original: On the other hand, the previous AQM requires 20 ×∞× 9628 calculations for information gain, which makes the computationof the previous AQM intractable.
Revised: On the other hand, the previous AQM requires 20 ×∞× 9628 calculations for information gain, which makes the computation intractable.

Original: Even if we use only 100 candidate answers, which is in the Visual Dialog dataset (Das et al., 2017a), the previous AQM requirestimes as many calculations (20M) as AQM+.
Revised: Even if we use only 100 candidate answers, which is in the Visual Dialog dataset (Das et al., 2017a), the previous AQM requires 2500 times as many calculations (20M) as AQM+.

Original: On the other hand, AQM+ requires more calculations and thus requires more inference time than SL or RL AQM+ generates one question within around 3s at K =20, whereas SL generates one question within 0.1s.
Revised: On the other hand, AQM+ requires more calculations and thus requires more inference time than SL or RL. AQM+ generates one question within around 3s when K =20, whereas SL generates one question within 0.1s.

Original: We used Tesla P40 for our experiments.
Revised: We used Tesla P40 for our experiments.

Original: Though the complexity of our information gain is O ( K 3 ) , K does not increase the time required for the whole inference in proportion to the cube of K, when K =20.
Revised: Though the complexity of our information gain is O ( K 3 ) , K does not increase the time required for the whole inference in proportion to the cube of K, when K =20.

Original: It is because calculating the information gain is not the sole resource-intensive part in the whole inference process and we do parallel processing for the calculation of aprxAgen using GPU.
Revised: It is because calculating the information gain is not the sole resource-intensive part in the whole inference process and we do parallel processing for the calculation of aprxAgen using GPU.

Original: The inference time would further decrease if more parallel processing techniques are applied.
Revised: The inference time would further decrease if more parallel processing techniques are applied.

Original: 5. T OWARD P RACTICAL A PPLICATIONS
Revised: 5. T OWARD P RACTICAL A PPLICATIONS

Original: There are plenty of potential future works to improve the performance of AQM+ in real task-oriented dialog applications.
Revised: There are plenty of potential future works to improve the performance of AQM+ in real task-oriented dialog applications.

Original: For example, robust task-oriented dialog systems are required for appropriately replying to user’s questions (Li et al., 2017) and responding for chit-chat style conversation (Zhao et al., 2017).
Revised: For example, robust task-oriented dialog systems are required for appropriately replying to user’s questions (Li et al., 2017) and responding for chit-chat style conversation (Zhao et al., 2017).

Original: The question quality can also be improved by diverse beam search approaches (Vijayakumar et al., 2016;
Revised: The question quality can also be improved by diverse beam search approaches (Vijayakumar et al., 2016;

Original: Li et al., 2016), which prevent sampling similar questions for the candidate set.
Revised: Li et al., 2016), which prevent sampling similar questions for the candidate set.

Original: We highlight two issues described below; online learning and fast inference.
Revised: We highlight two issues described below; online learning and fast inference.

Original: Online Learning For a novel answerer, ﬁne-tuning on the dialog model is required (Krause et al., 2018).
Revised: Online Learning For a novel answerer, ﬁne-tuning on the dialog model is required (Krause et al., 2018).

Original: If the experiences of many users are available, model-agnostic meta learning (MAML) (Finn et al., 2017) can be applied for few-shot learning.
Revised: If the experiences of many users are available, model-agnostic meta learning (MAML) (Finn et al., 2017) can be applied for few-shot learning.

Original: Updating the hyperparameter λ in an online manner, which balances the effect of the prior and the likelihood, can also be effective in practice.
Revised: Updating the hyperparameter λ in an online manner, which balances the effect of the prior and the likelihood, can also be effective in practice.

Original: If the answer distribution of user is different from our aprxAgen, we can increase λ to decrease the effect of the likelihood.
Revised: If the answer distribution of user is different from our aprxAgen, we can increase λ to decrease the effect of the likelihood.

Original: Fast Inference AQM+’s time complexity can be decreased further by changing the structure of aprxAgen.
Revised: Fast Inference AQM+’s time complexity can be decreased further by changing the structure of aprxAgen.

Original: In speciﬁc, we can apply diverse methods such as skipping the update of hidden states in some steps (Seo et al., 2018), using convolution networks or self-attention networks (Yu et al., 2018; Vaswani et al., 2017), substituting matrix multiplication operation for hidden state update to weighted addition (Yu & Liu, 2018), and direct information gain inference from the neural networks (Belghazi et al., 2018).
Revised: In speciﬁc, we can apply diverse methods such as skipping the update of hidden states in some steps (Seo et al., 2018), using convolution networks or self-attention networks (Yu et al., 2018; Vaswani et al., 2017), substituting matrix multiplication operation for hidden state update to weighted addition (Yu & Liu, 2018), and direct information gain inference from the neural networks (Belghazi et al., 2018).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Asking appropriate questions in practical applications has recently been paid attention (Rao &
Revised: Asking appropriate questions in practical applications has recently been paid attention (Rao &

Original: Daum´e III, 2018; Buck et al., 2018).
Revised: Daum´e III, 2018; Buck et al., 2018).

Original: We proposed AQM+ algorithm that is a large-scale extension of AQM framework.
Revised: We proposed AQM+ algorithm that is a large-scale extension of AQM framework.

Original: AQM+ can ask an appropriate question considering the context of the dialog, handle the responses in a sentence form, and efﬁciently estimate information gain of the target class with a given question.
Revised: AQM+ can ask an appropriate question considering the context of the dialog, handle the responses in a sentence form, and efﬁciently estimate information gain of the target class with a given question.

Original: This improvement makes our AQM framework to step forward toward practical task-oriented applications.
Revised: This improvement makes our AQM framework to step forward toward practical task-oriented applications.

Original: AQM+ not only outperforms the comparative SL and
Revised: AQM+ not only outperforms the comparative SL and

Original: RL algorithms, but also enlarges the gap between AQM+ and the comparative algorithms comparing to the performance gaps reported in GuessWhat.
Revised: RL algorithms, but also enlarges the gap between AQM+ and the comparative algorithms comparing to the performance gaps reported in GuessWhat.

Original: AQM+ acheives more than 60% error decreases through the dialog, whereas the comparative algorithms only achieve 6% error decreases.
Revised: AQM+ acheives more than 60% error decreases through the dialog, whereas the comparative algorithms only achieve 6% error decreases.

Original: Moreover, the performance of AQM+ can be boosted further by employing the models recently proposed in the visual dialog ﬁeld such as other question generator models (Jain et al., 2018) and question answering models (Kottur et al., 2018)
Revised: Moreover, the performance of AQM+ can be boosted further by employing the models recently proposed in the visual dialog ﬁeld such as other question generator models (Jain et al., 2018) and question answering models (Kottur et al., 2018)

Original: A PPENDIX A. AQM+ A LGORITHM
Revised: A PPENDIX A. AQM+ A LGORITHM

Original: The question generating process of AQM+ used in our GuessWhich experiments are as follows.
Revised: The question generating process of AQM+ used in our GuessWhich experiments are as follows.

Original: Algorithm
Revised: Algorithm

Original: ˆ p (cid:48)
Revised: ˆ p (cid:48)

Original: ( c | h 0 )
Revised: ( c | h 0 )

Original: ∝ exp( λ · f ‡ ( c | h 0 )) for t = 1: T do
Revised: ∝ exp( λ · f ‡ ( c | h 0 )) for t = 1: T do

Original: C t,topk ← top-K posterior test image (from Qpost ˆ p ( c | h t − 1 ) )
Revised: C t,topk ← top-K posterior test image (from Qpost ˆ p ( c | h t − 1 ) )

Original: Q t,gen ← top-K likelihood questions using beam search (from Qgen p † ( q t | h t − 1 ) )
Revised: Q t,gen ← top-K likelihood questions using beam search (from Qgen p † ( q t | h t − 1 ) )

Original: A t,topk ( q t ) ← generated answers from aprxAgen for question q t and each class in C t,topk (from aprxAgen ˜ p ( a t | c, q t , h t − 1 ) )
Revised: A t,topk ( q t ) ← generated answers from aprxAgen for question q t and each class in C t,topk (from aprxAgen ˜ p ( a t | c, q t , h t − 1 ) )

Original: q t ← argmax q (cid:48) t ∈ Q t,gen ˜ I
Revised: q t ← argmax q (cid:48) t ∈ Q t,gen ˜ I

Original: [ C, A t ; q (cid:48) t , a 1: t − 1 , q 1: t − 1 ] with A t,topk ( q t ) and C t,topk in Eq.
Revised: [ C, A t ; q (cid:48) t , a 1: t − 1 , q 1: t − 1 ] with A t,topk ( q t ) and C t,topk in Eq.

Original: Get a t from Agen ¯ p ( a t | c, q t , h t − 1 )
Revised: Get a t from Agen ¯ p ( a t | c, q t , h t − 1 )

Original: Update Qpost ˆ p ( c | h t )
Revised: Update Qpost ˆ p ( c | h t )

Original: ∝ ˜ p ( a t | c, q t , h t − 1 ) · ˆ p ( c | h t − 1 ) in Eq.end for
Revised: ∝ ˜ p ( a t | c, q t , h t − 1 ) · ˆ p ( c | h t − 1 ) in Eq.end for

Original: A PPENDIX B. A BLATION S TUDY
Revised: A PPENDIX B. A BLATION S TUDY

Original: Figure 7 shows the results of the number of QAC ablation experiment on depA and trueA, in the non-delta setting.
Revised: Figure 7 shows the results of the number of QAC ablation experiment on depA and trueA, in the non-delta setting.

Original: The effect of K decreases in trueA compared to indA, which indicates that the similarity between the distribution of aprxAgen and Agen is related to the effectiveness of large K.
Revised: The effect of K decreases in trueA compared to indA, which indicates that the similarity between the distribution of aprxAgen and Agen is related to the effectiveness of large K.

Original: Figure 8 shows the results of the no caption experiment on depA and trueA, in the non-delta setting.
Revised: Figure 8 shows the results of the no caption experiment on depA and trueA, in the non-delta setting.

Original: Figure 9 shows the experimental results on the model where AQM+’s
Revised: Figure 9 shows the experimental results on the model where AQM+’s

Original: Qinfo is used as the questiongenerator and SL’s Qscore is used as the guesser.
Revised: Qinfo is used as the questiongenerator and SL’s Qscore is used as the guesser.

Original: AQM+’s Qinfo does not improve the performance of SL’s guesser (Qscore).
Revised: AQM+s Qinfo does not improve the performance of SLs guesser (Qscore).

Original: Our analysis of the results is as follows.
Revised: Our analysis of the results is as follows.

Original: For delta setting, the SL guesser is not able to obtain the information from the answers.
Revised: For delta setting, the SL guesser is not able to obtain the information from the answers.

Original: For the non-delta case, not dialog history but caption information gives dominant information to SL’s guesser.
Revised: For the non-delta case, not dialog history but caption information gives dominant information to SLs guesser.

Original: The questions which often appear with caption thus gave a more clear signal for the target class for SL’s guesser.
Revised: The questions which often appear with caption thus gave a more clear signal for the target class for SLs guesser.

Original: Figure 9a shows that
Revised: Figure 9a shows that

Original: SL-Q performs better than RL-Q in the early phase, but SL-Q’s performance decreases faster than that of RL-Q in the later phase.
Revised: SL-Q performs better than RL-Q in the early phase, but SL-Qs performance decreases faster than that of RL-Q in the later phase.

Original: It is because SL-Q generates the question to be more likely to have co-appeared with the caption than RL-Q. Likewise, AQM+’s question does not help SL’s guesser because AQM+ generates questions that are more independent of the caption.
Revised: It is because SL-Q generates the question to be more likely to have co-appeared with the caption than RL-Q. Likewise, AQM+s question does not help SLs guesser because AQM+ generates questions that are more independent of the caption.

Original: We conducted the experiments under the setting where a predeﬁned candidate question set Q fix is used.
Revised: We conducted the experiments under the setting where a predeﬁned candidate question set Q fix is used.

Original: The discussion section in the work of Lee et al.
Revised: The discussion section in the work of Lee et al.

Original: (2018) includes an experimental setting in which the candidate questions are generated from an end-to-end SL model only at the ﬁrst turn.
Revised: (2018) includes an experimental setting in which the candidate questions are generated from an end-to-end SL model only at the ﬁrst turn.

Original: We refer to this setting as gen1Q, as in the previous AQM paper.
Revised: We refer to this setting as gen1Q, as in the previous AQM paper.

Original: Figure 10 shows the results of gen1Q ablation study.
Revised: Figure 10 shows the results of gen1Q ablation study.

Original: Note that this setting of | Q | =100 requires ﬁve times as many computations to calculate the information gain as the original AQM+, despite gen1Q performs even worse than
Revised: Note that this setting of | Q | =100 requires ﬁve times as many computations to calculate the information gain as the original AQM+, despite gen1Q performs even worse than

Original: Guesser baseline.
Revised: Guesser baseline.

Original: Another noticeable phenomenon is that there is no signiﬁcant performance loss in trueA setting.
Revised: Another noticeable phenomenon is that there is no signiﬁcant performance loss in trueA setting.

Original: Since aprxAgen in trueA knows the exact probability of Abot’s answer, by exploiting such an aprxAgen, Qbot in trueA can clearly distinguish between different classes by capturing even the subtle differences in answer distributions given similar questions.
Revised: Since aprxAgen in trueA knows the exact probability of Abot’s answer, by exploiting such an aprxAgen, Qbot in trueA can clearly distinguish between different classes by capturing even the subtle differences in answer distributions given similar questions.

Original: We also performed the experiments under the setting where Q fix comes from training data.
Revised: We also performed the experiments under the setting where Q fix comes from training data.

Original: Figure 11 shows the results of randQ ablation study.
Revised: Figure 11 shows the results of randQ ablation study.

Original: The baseline method with this Q fix showed accuracy degradation.
Revised: The baseline method with this Q fix showed accuracy degradation.

Original: Regardless of the PMR, we point out that randQ retrieves questions relevant to neither the caption nor the target image.
Revised: Regardless of the PMR, we point out that randQ retrieves questions relevant to neither the caption nor the target image.

Original: It is why we generate candidate questions from a seq-to-seq model.
Revised: It is why we generate candidate questions from a seq-to-seq model.

Original: Figure 12 shows the results of the no history experiment.
Revised: Figure 12 shows the results of the no history experiment.

Original: Dialog history helps to guess the target image but is not critical.
Revised: Dialog history helps to guess the target image but is not critical.

Original: Ablating history makes the performance decrease by 0.22% and 0.56% for indA and depA in non-delta, respectively, and 0.46% and 0.21% for indA and depA in delta, respectively.
Revised: Ablating history makes the performance decrease by 0.22% and 0.56% for indA and depA in non-delta, respectively, and 0.46% and 0.21% for indA and depA in delta, respectively.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX C. G ENERATING S ENTENCES
Revised: A PPENDIX C. G ENERATING S ENTENCES

Original: Figure 13 shows selected examples of generated questions in delta setting.
Revised: Figure 13 shows selected examples of generated questions in delta setting.

Original: Though delta setting boosts to increase PMR of the zeroth turn much, it degenerates the question quality, especially for
Revised: Though delta setting boosts to increase PMR of the zeroth turn much, it degenerates the question quality, especially for

Original: RL-QA.
Revised: RL-QA.

Original: Moreover, RL-QA tends to concentrate on the ﬁrst turn, leaving questions and answers of the remaining turns meaningless.
Revised: Moreover, RL-QA tends to concentrate on the ﬁrst turn, leaving questions and answers of the remaining turns meaningless.