Original: T OWARDS I NTERPRETABLE C HIT - CHAT : O PEN D OMAIN D IALOGUE G ENERATION WITH D IALOGUE A CTS
Revised: T OWARDS I NTERPRETABLE C HIT - CHAT : O PEN D OMAIN D IALOGUE G ENERATION WITH D IALOGUE A CTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Conventional methods model open domain dialogue generation as a black box through end-to-end learning from large scale conversation data.
Revised: Conventional methods model open domain dialogue generation as a black box through end-to-end learning from large scale conversation data.

Original: In this work, we make the ﬁrst step to open the black box by introducing dialogue acts into open domain dialogue generation.
Revised: In this work, we make the ﬁrst step to open the black box by introducing dialogue acts into open domain dialogue generation.

Original: The dialogue acts are generally designed and reveal how people engage in social chat.
Revised: The dialogue acts are generally designed and reveal how people engage in social chat.

Original: Inspired by analysis on real data, we propose jointly modeling dialogue act selection and response generation, and perform learning with human-human conversations tagged with a dialogue act classiﬁer and a reinforcement approach to further optimizing the model for long-term conversation.
Revised: Inspired by analysis on real data, we propose jointly modeling dialogue act selection and response generation, and perform learning with human-human conversations tagged with a dialogue act classiﬁer and a reinforcement approach to further optimizing the model for long-term conversation.

Original: With the dialogue acts, we not only achieve signiﬁcant improvement over state-of-the-art methods on response quality for given contexts and long-term conversation in both machine-machine simulation and human-machine conversation, but also are capable of explaining why such achievements can be made.
Revised: With the dialogue acts, we not only achieve signiﬁcant improvement over state-of-the-art methods on response quality for given contexts and long-term conversation in both machine-machine simulation and human-machine conversation, but also are capable of explaining why such achievements can be made.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Conversational agents are becoming ubiquitous recently.
Revised: Conversational agents are becoming ubiquitous recently.

Original: Through human-machine conversation, such agents either help users complete speciﬁc tasks (Young et al., 2013) or engage them in social chat (Vinyals & Le, 2015).
Revised: Through human-machine conversation, such agents either help users complete speciﬁc tasks (Young et al., 2013) or engage them in social chat (Vinyals & Le, 2015).

Original: Depending on application scenarios, various conversational agents have been designed including chatbots, personal assistants, and automated customer service, etc.
Revised: Depending on application scenarios, various conversational agents have been designed including chatbots, personal assistants, and automated customer service, etc.

Original: Traditional research on conversational agents focuses on task-oriented dialogue systems (Young et al., 2013) where task speciﬁc dialogue acts are handcrafted in a form of slot-value pairs.
Revised: Traditional research on conversational agents focuses on task-oriented dialogue systems (Young et al., 2013) where task speciﬁc dialogue acts are handcrafted in a form of slot-value pairs.

Original: On the one hand, through slot-ﬁlling, the dialogue acts make conversations in such systems interpretable and controllable; on the other hand, they also hinder scaling such systems to new domains.
Revised: On the one hand, through slot-ﬁlling, the dialogue acts make conversations in such systems interpretable and controllable; on the other hand, they also hinder scaling such systems to new domains.

Original: To escape from the limitation, recent interest of research moves to end-to-end dialogue learning without any assumptions on dialogue acts.
Revised: To escape from the limitation, recent interest of research moves to end-to-end dialogue learning without any assumptions on dialogue acts.

Original: Most of the effort is paid to non-task-oriented chit-chat
Revised: Most of the effort is paid to non-task-oriented chit-chat

Original: (Vinyals &
Revised: (Vinyals &

Original: Le, 2015), and there are also a few studies on task-oriented dialogues (Bordes & Weston, 2017; Eric & Manning, 2017).
Revised: Le, 2015), and there are also a few studies on task-oriented dialogues (Bordes & Weston, 2017; Eric & Manning, 2017).

Original: Without dialogue acts, these work directly constructs a response by learning from large scale data with neural networks, and thus is easy to scale to new domains.
Revised: Without dialogue acts, these work directly constructs a response by learning from large scale data with neural networks, and thus is easy to scale to new domains.

Original: On the other hand, due to the absence of dialogue acts, it is hard to interpret the emergence of a response to a dialogue context and predict where the conversation will ﬂow to.
Revised: On the other hand, due to the absence of dialogue acts, it is hard to interpret the emergence of a response to a dialogue context and predict where the conversation will ﬂow to.

Original: In this work, we aim to achieve interpretability and controllability in non-task-oriented dialogues.
Revised: In this work, we aim to achieve interpretability and controllability in non-task-oriented dialogues.

Original: To this end, we introduce dialogue acts into open domain dialogue generation.
Revised: To this end, we introduce dialogue acts into open domain dialogue generation.

Original: Open domain dialogue generation has been widely applied to chatbots which aim at engaging users by keeping conversation going, but is seldom in the harness of dialogue acts.
Revised: Open domain dialogue generation has been widely applied to chatbots which aim at engaging users by keeping conversation going.

Original: Existing work concentrates on generating relevant and diverse responses for a context.
Revised: Existing work concentrates on generating relevant and diverse responses for a context.

Original: However, it is not clear if relevance and diversity are sufﬁcient to engagement in interactions.
Revised: However, it is not clear if relevance and diversity are sufﬁcient to engagement in interactions.

Original: Therefore, we investigate the following problems: (1) if we can properly design dialogue acts that can enable us to understand engagement in human-human open domain conversation; (2) how to learn a dialogue generation model with the dialogue acts; and (3) how the model performs in practice and if the performance can be explained by the dialogue acts.
Revised: Therefore, we investigate the following problems: (1) if we can properly design dialogue acts that can enable us to understand engagement in human-human open domain conversation; (2) how to learn a dialogue generation model with the dialogue acts; and (3) how the model performs in practice and if the performance can be explained by the dialogue acts.

Original: To examine how people engage in social chat, we establish a general dialogue act taxonomy for open domain conversation by extending the existing work with high-level dialogue acts regarding to conversational context.
Revised: To examine how people engage in social chat, we establish a general dialogue act taxonomy for open domain conversation by extending the existing work with high-level dialogue acts regarding to conversational context.

Original: The taxonomy, when applied to real data, gives rise to an interesting ﬁnding that in addition to replying with relevance and diversity , people are used to driving their social chat by constantly switching to new contexts and properly asking questions.
Revised: The taxonomy, when applied to real data, gives rise to an interesting ﬁnding that in addition to replying with relevance and diversity , people are used to driving their social chat by constantly switching to new contexts and properly asking questions.

Original: Such behaviors are less explored before, and thus are difﬁcult for the existing end-to-end learning methods to imitate.
Revised: Such behaviors are less explored before, and thus are difﬁcult for the existing end-to-end learning methods to imitate.

Original: To mimic human behaviors, we propose jointly modeling dialogue act selection and response generation in open domain dialogue generation.
Revised: To mimic human behaviors, we propose jointly modeling dialogue act selection and response generation in open domain dialogue generation.

Original: The dialogue model is speciﬁed with neural networks.
Revised: The dialogue model is speciﬁed with neural networks.

Original: We propose learning from human-human interactions by ﬁtting the model to large scale real world dialogues tagged with a dialogue act classiﬁer and further optimizing the policy of act selection for long-term conversation through a reinforcement learning approach.
Revised: We propose learning from human-human interactions by ﬁtting the model to large scale real world dialogues tagged with a dialogue act classiﬁer and further optimizing the policy of act selection for long-term conversation through a reinforcement learning approach.

Original: Our model enjoys several advantages over the existing models: (1) the dialogue acts provide interpretation to response generation from a discourse perspective; (2) the dialogue acts enhance diversity of responses by expanding the search space from language to act × language;and (3) the dialogue  act selection is compatible with post-engineering work (e.g., combination with rules), and thus allows engineers to ﬂexibly control their systems through picking responses from their desired dialogue acts.
Revised: Our model enjoys several advantages over the existing models: (1) the dialogue acts provide interpretation to response generation from a discourse perspective; (2) the dialogue acts enhance diversity of responses by expanding the search space from language to act × language; (3) the dialogue acts improve engagement in human-machine conversations; and (4) the dialogue act selection is compatible with post-engineering work (e.g., combination with rules), and thus allows engineers to ﬂexibly control their systems through picking responses from their desired dialogue acts.

Original: Evaluation results on large scale test data indicate that our model can signiﬁcantly outperform state-of-the-art methods in terms of quality of generated responses regarding to given contexts and lead to long-term conversation in both machine-machine simulation and human-machine conversation in a way similar to how human behave in their interactions.
Revised: Evaluation results on large scale test data indicate that our model can signiﬁcantly outperform state-of-the-art methods in terms of quality of generated responses regarding to given contexts and lead to long-term conversation in both machine-machine simulation and human-machine conversation in a way similar to how human behave in their interactions.

Original: Our contributions in this work include: (1) design of dialogue acts that represent human behavior regarding to conversational context and insights from analysis of human-human interactions with the design; (2) joint modeling of dialogue act selection and response generation in open domain dialogue generation; (3) proposal of a supervised learning approach and a reinforcement learning approach for model optimization; (4) empirical veriﬁcation of the effectiveness of the model through automatic metrics, human annotations, machine-machine simulation, and human-machine conversation.
Revised: Our contributions in this work include: (1) design of dialogue acts that represent human behavior regarding to conversational context and insights from analysis of human-human interactions with the design; (2) joint modeling of dialogue act selection and response generation in open domain dialogue generation; (3) proposal of a supervised learning approach and a reinforcement learning approach for model optimization; (4) empirical veriﬁcation of the effectiveness of the model through automatic metrics, human annotations, machine-machine simulation, and human-machine conversation.

Original: [Table]
Revised: [Table]

Original:  D IALOGUE A CTS FOR O PEN D OMAIN C ONVERSATION
Revised:  D IALOGUE A CTS FOR O PEN D OMAIN C ONVERSATION

Original: We ﬁrst deﬁne dialogue acts, and then describe the data for learning and the insights we obtain from the data.
Revised: We ﬁrst deﬁne dialogue acts, and then describe the data for learning and the insights we obtain from the data.

Original: Finally, we elaborate how we build the classiﬁer with neural networks.
Revised: Finally, we elaborate how we build the classiﬁer with neural networks.

Original: 2. D EFINITION OF D IALOGUE A CTS
Revised: 2. D EFINITION OF D IALOGUE A CTS

Original: Our dialogue acts are inherited from the existing work on 1-on-1 live chats and twitter (Kim et al., 2010; Ivanovic, 2005).
Revised: Our dialogue acts are inherited from the existing work on 1-on-1 live chats and twitter (Kim et al., 2010; Ivanovic, 2005).

Original: Similar to (Oraby et al., 2017), we organize the 12 acts in (Ivanovic, 2005) which originate from the42 tags (Jurafsky et al., 1997; Stolcke et al., 2006) based on the DAMSL annotation scheme (Core & Allen, 1997) into high-level dialogue acts: “statement” and “expressive” are merged as “statement”; “yes-no question” and “open question” are combined as “question”; “yes-answer”, “no-answer”, and “response-ack” are collapsed as “answer”; and other tags are treated as “others”.
Revised: Similar to (Oraby et al., 2017), we organize the 12 acts in (Ivanovic, 2005) which originate from the annotation scheme (Core & Allen, 1997) into high-level dialogue acts: “statement” and “expressive” are merged as “statement”; “yes-no question” and “open question” are combined as “question”; “yes-answer”, “no-answer”, and “response-ack” are collapsed as “answer”; and other tags are treated as “others”.

Original: On top of these acts, we further deﬁne two high-level dialogue acts that describe how people behave regarding to conversational context in their interactions.
Revised: On top of these acts, we further deﬁne two high-level dialogue acts that describe how people behave regarding to conversational context in their interactions.

Original: As will be seen later, the extension may bring us further insights on engagement in social chat.
Revised: As will be seen later, the extension may bring us further insights on engagement in social chat.

Original: Details of the dialogue acts are described in Table 1.
Revised: Details of the dialogue acts are described in Table 1.

Original: The high-level dialogue acts in Table 1 are generally applicable to open domain dialogues from various sources in different languages such as Twitter, Reddit, Facebook, Weibo ( www.weibo.com ), and Baidu Tieba ( https://tieba.baidu.com/ ), etc.
Revised: The high-level dialogue acts in Table 1 are generally applicable to open domain dialogues from various sources in different languages such as Twitter, Reddit, Facebook, Weibo ( www.weibo.com ), and Baidu Tieba ( https://tieba.baidu.com/ ), etc.

Original: One can extend the taxonomy by deﬁning ﬁner-grained dialogue acts and learn their generation models with the approaches described later.
Revised: One can extend the taxonomy by deﬁning ﬁner-grained dialogue acts and learn their generation models with the approaches described later.

Original: Existing annotated data sets (e.g., the Switchboard Corpus 1 ) do not have dialogue acts regarding to conversational context.
Revised: Existing annotated data sets (e.g., the Switchboard Corpus 1 ) do not have dialogue acts regarding to conversational context.

Original: Therefore, it is not clear how such dialogue acts depict human behavior in interactions, and there are no large scale data available for learning dialgoue generation with the dialogue acts either.
Revised: Therefore, it is not clear how such dialogue acts depict human behavior in interactions, and there are no large scale data available for learning dialgoue generation with the dialogue acts either.

Original: To resolve these problems, we build a data set.
Revised: To resolve these problems, we build a data set.

Original: 2. D ATA S ET
Revised: 2. D ATA S ET

Original: We crawled 30 million dyadic dialogues (conversations between two people) from Baidu Tieba.
Revised: We crawled 30 million dyadic dialogues (conversations between two people) from Baidu Tieba.

Original: Baidu Tieba is the largest Reddit-like forum in China which allows users to post and comment on others’ post.
Revised: Baidu Tieba is the largest Reddit-like forum in China which allows users to post and comment on others’ post.

Original: Two people can communicate with each other through one posting a comment and the other one replying to the comment.
Revised: Two people can communicate with each other through one posting a comment and the other one replying to the comment.

Original: Data in Baidu Tieba covers a large variety of topics, and thus can be viewed as a simulation of open domain conversation in a chatbot.
Revised: Data in Baidu Tieba covers a large variety of topics, and thus can be viewed as a simulation of open domain conversation in a chatbot.

Original: We randomly samplemillion dialogues as a training set, 90 thousand dialogues as a validation set, and 1000 dialogues as a test set.
Revised: We randomly samplemillion dialogues as a training set, 90 thousand dialogues as a validation set, and 1000 dialogues as a test set.

Original: These data are used to learn a dialogue generation model later.
Revised: These data are used to learn a dialogue generation model later.

Original: We employ the Standford
Revised: We employ the Standford

Original: Chinese word segmenter 2 to tokenize utterances in the data.
Revised: Chinese word segmenter 2 to tokenize utterances in the data.

Original: Table 2 reports statistics of the data.
Revised: Table 2 reports statistics of the data.

Original: [Table]
Revised: [Table]

Original: For dialogue act learning
Revised: For dialogue act learning

Original: [Table]
Revised: [Table]

Original: we randomly sample 500 dialogues from the training set and recruitnative speakers to label dialogue acts 3 for each utterance according to the deﬁnitions in Table 1.
Revised: we randomly sample 500 dialogues from the training set and recruitnative speakers to label dialogue acts 3 for each utterance according to the deﬁnitions in Table 1.

Original: Table 3 shows a labeling example from one annotator.
Revised: Table 3 shows a labeling example from one annotator.

Original: Each utterance receives 3 labels, and the
Revised: Each utterance receives 3 labels, and the

Original: Fleiss’ kappa of the labeling work is 0 . 45 , indicating moderate agreement among the labelers
Revised: Fleiss’ kappa of the labeling work is 0 . 45 , indicating moderate agreement among the labelers

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A: 万 里 长 城 很 漂 亮 ！
Revised: A: 万 里 长 城 很 漂 亮 ！

Original: The Great Wall of China is beautiful!
Revised: The Great Wall of China is beautiful!

Original: [Table]
Revised: [Table]

Original: B: 你 在 长 城 看 日 落 了 吗 ？
Revised: B: 你 在 长 城 看 日 落 了 吗 ？

Original: Did you see the sunset on the Great Wall?
Revised: Did you see the sunset on the Great Wall?

Original: [Table]
Revised: [Table]

Original: A: 是 的 ， 那 是 最 漂 亮 的 景 色
Revised: A: 是 的 ， 那 是 最 漂 亮 的 景 色

Original: Yes, it’s the most beautiful scenery.
Revised: Yes, it’s the most beautiful scenery.

Original: [Table]
Revised: [Table]

Original: B: 上 次 我 去 的 时 候 人 很 多
Revised: B: 上 次 我 去 的 时 候 人 很 多

Original: It was very crowded when I visited there last time
Revised: It was very crowded when I visited there last time

Original: [Table]
Revised: [Table]

Original: A: 我 只 待 了一 小 会 儿 ， 人 太多 了 ！ I only stayed there for a while.
Revised: A: 我 只 待 了一 小 会 儿 ， 人 太多 了 ！ I only stayed there for a while.

Original: Too many vistors!
Revised: Too many vistors!

Original: [Table]
Revised: [Table]

Original: 2. I NSIGHTS FROM THE LABELED DATA
Revised: 2. I NSIGHTS FROM THE LABELED DATA

Original: The frequencies of the dialogue acts in terms of percentages of the total number of utterances in the labeled data are CM.S 55 .
Revised: The frequencies of the dialogue acts in terms of percentages of the total number of utterances in the labeled data are CM.S 55 .

Original: 8 %, CM.Q 11 .
Revised: 8 %, CM.Q 11 .

Original: 7 %, CM.A 12 .
Revised: 7 %, CM.A 12 .

Original: 2 %, CS.S 12 .
Revised: 2 %, CS.S 12 .

Original: 4 %, CS.Q 4 .
Revised: 4 %, CS.Q 4 .

Original: 8 %, CS.A 2 %, and
Revised: 8 %, CS.A 2 %, and

Original: O 1 . 1 %.
Revised: O 1 . 1 %.

Original: In addition to the numbers, we also get further insights from the data that are instructive to our dialogue generation learning:
Revised: In addition to the numbers, we also get further insights from the data that are instructive to our dialogue generation learning:

Original: Context switch is a common skill to keep conversation going.
Revised: Context switch is a common skill to keep conversation going.

Original: In fact, we ﬁnd that 78 .
Revised: In fact, we ﬁnd that 78 .

Original: 2 % dialogues contain at least one CS.
Revised: 2 % dialogues contain at least one CS.

Original: * act.
Revised: * act.

Original: The average number of turns of dialogues that contain at least one
Revised: The average number of turns of dialogues that contain at least one

Original: CS.
Revised: CS.

Original: * is 8 . 4 , while the average number of turns of dialogues that do not contain a CS.
Revised: * is 8 . 4 , while the average number of turns of dialogues that do not contain a CS.

Original: * is 7 .
Revised: * is 7 .

Original: When dialogues are shorter than 5 turns, only 47 % of them contain a CS.
Revised: When dialogues are shorter than 5 turns, only 47 % of them contain a CS.

Original: *, but when dialogues exceedturns, more than 85 % of them contain a CS.
Revised: *, but when dialogues exceedturns, more than 85 % of them contain a CS.

Original: Because there are no speciﬁc goals in their conversations, people seldom stay long in one context.
Revised: Because there are no speciﬁc goals in their conversations, people seldom stay long in one context.

Original: The average number of turns before context switch is 3 . 39 .
Revised: The average number of turns before context switch is 3 . 39 .

Original: We also observed consecutive context switch in many dialogues ( 43 . 7 %).
Revised: We also observed consecutive context switch in many dialogues ( 43 . 7 %).

Original: The numbers suggest dialogue generation with smooth context switch and moderate context maintenance.
Revised: The numbers suggest dialogue generation with smooth context switch and moderate context maintenance.

Original: Question is an important building block in open domain conversation.
Revised: Question is an important building block in open domain conversation.

Original: In fact, 13 . 9 % CM.
Revised: In fact, 13 . 9 % CM.

Original: * are
Revised: * are

Original: CM.Q and the percentage is even higher in CS.
Revised: CM.Q and the percentage is even higher in CS.

Original: * which is 20 .
Revised: * which is 20 .

Original: People need to ask questions in order to maintain contexts.
Revised: People need to ask questions in order to maintain contexts.

Original: The average number of turns of contexts with questions (i.e., consecutive
Revised: The average number of turns of contexts with questions (i.e., consecutive

Original: CM.
Revised: CM.

Original: * with at least one CM.Q) is 3 .
Revised: * with at least one CM.Q) is 3 .

Original: 92 , while the average number of turns of contexts without questions is only 2 .
Revised: 92 , while the average number of turns of contexts without questions is only 2 .

Original: The observation indicates that a good dialogue model should be capable of asking questions properly, as suggested by Li et al.
Revised: The observation indicates that a good dialogue model should be capable of asking questions properly, as suggested by Li et al.

Original: (2017a).
Revised: (2017a).

Original: A further step to study human’s questioning behavior is to look into types and functions of questions.
Revised: A further step to study human’s questioning behavior is to look into types and functions of questions.

Original: We leave it as future work.
Revised: We leave it as future work.

Original: The observations raise new challenges that are difﬁcult for the existing end-to-end methods to tackle (e.g., smoothly interleaving context blocks with switch actions), and thus encourage us to create a new model.
Revised: The observations raise new challenges that are difﬁcult for the existing end-to-end methods to tackle (e.g., smoothly interleaving context blocks with switch actions), and thus encourage us to create a new model.

Original: Note that these observations may relate to dialogue scenarios (e.g., chatting online instead of face-to-face) and cultures, but we ignore these factors and just study how to learn the conversational patterns from the data with a principled approach.
Revised: Note that these observations may relate to dialogue scenarios (e.g., chatting online instead of face-to-face) and cultures, but we ignore these factors and just study how to learn the conversational patterns from the data with a principled approach.

Original: The learning approach is generally applicable to other data.
Revised: The learning approach is generally applicable to other data.

Original: To perform learning, we need to build a classiﬁer that can automatically tag large scale dialogues with dialogue acts.
Revised: To perform learning, we need to build a classiﬁer that can automatically tag large scale dialogues with dialogue acts.

Original: 2. D IALOGUE A CT C LASSIFICATION
Revised: 2. D IALOGUE A CT C LASSIFICATION

Original: We aim to learn a classiﬁer c from D A
Revised: We aim to learn a classiﬁer c from D A

Original: = { d i } Ni =1 where d i = { ( u i, 1 , a i, 1 ) , . . .
Revised: = { d i } Ni =1 where d i = { ( u i, 1 , a i, 1 ) , . . .

Original: , ( u i,n i , a i,n i ) } represents a dialogue with u i,k the k -th utterance and a i,k the labeled dialogue act.
Revised: , ( u i,n i , a i,n i ) } represents a dialogue with u i,k the k -th utterance and a i,k the labeled dialogue act.

Original: Given a new dialogue d = { u 1 , . . .
Revised: Given a new dialogue d = { u 1 , . . .

Original: u n } , c can sequentially tag the utterances in d with dialouge acts by taking u i , u i − 1 , and the predicted a i − 1 as inputs and outputting a vector c ( u i , u i − 1 , a i − 1 ) where the j -th element representing the probability of u i being tagged as the j -th dialogue act.
Revised: u n } , c can sequentially tag the utterances in d with dialouge acts by taking u i , u i − 1 , and the predicted a i − 1 as inputs and outputting a vector c ( u i , u i − 1 , a i − 1 ) where the j -th element representing the probability of u i being tagged as the j -th dialogue act.

Original: We parameterize c ( · , · , · ) using neural networks.
Revised: We parameterize c ( · , · , · ) using neural networks.

Original: Speciﬁcally, u i and u i − 1 are ﬁrst processed by bidirectional recurrent neural networks with gated recurrent units (biGRUs) (Chung et al., 2014) respectively.
Revised: Speciﬁcally, u i and u i − 1 are ﬁrst processed by bidirectional recurrent neural networks with gated recurrent units (biGRUs) (Chung et al., 2014) respectively.

Original: Then the last hidden states of the two biGRUs are concatenated with an embedding of a i − 1 and fed to a multi-layer perceptron (MLP) to calculate a dialogue act distribution.
Revised: Then the last hidden states of the two biGRUs are concatenated with an embedding of a i − 1 and fed to a multi-layer perceptron (MLP) to calculate a dialogue act distribution.

Original: Formally, suppose that u i =
Revised: Formally, suppose that u i =

Original: ( w i, 1 , . . .
Revised: ( w i, 1 , . . .

Original: , w i,n ) where w i,j is the embedding of the j -th word, then the j -th hidden state of the biGRU is given by h i,j = [ −→ h i,j ; ←− h i,j ] where −→ h i,j is the j -th state of a forward
Revised: , w i,n ) where w i,j is the embedding of the j -th word, then the j -th hidden state of the biGRU is given by h i,j = [ −→ h i,j ; ←− h i,j ] where −→ h i,j is the j -th state of a forward

Original: GRU, ←− h i,j is the j -th state of a backward GRU, and [ · ; · ] is a concatenation operator.
Revised: GRU, ←− h i,j is the j -th state of a backward GRU, and [ · ; · ] is a concatenation operator.

Original: −→ h
Revised: −→ h

Original: i,j and ←− h i,j are calculated by
Revised: i,j and ←− h i,j are calculated by

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have h i − 1 ,j as the j -th hidden state of u i − 1 .
Revised: Similarly, we have h i − 1 ,j as the j -th hidden state of u i − 1 .

Original: Let e ( a i − 1 ) be the embedding of a i − 1 , then c ( u i , u i − 1 , a i − 1 ) is deﬁned by a two-layer MLP:
Revised: Let e ( a i − 1 ) be the embedding of a i − 1 , then c ( u i , u i − 1 , a i − 1 ) is deﬁned by a two-layer MLP:

Original: [Equation]
Revised: [Equation]

Original: where we pad zeros for u 0 and a 0 in c ( u 1 , u 0 , a 0 ) .
Revised: where we pad zeros for u 0 and a 0 in c ( u 1 , u 0 , a 0 ) .

Original: We learn c ( · , · , · ) by minimizing cross entropy with D A .
Revised: We learn c ( · , · , · ) by minimizing cross entropy with D A .

Original: Let p j ( a i ) be the probability of a i being the j -th dialogue act and c ( u i , u
Revised: Let p j ( a i ) be the probability of a i being the j -th dialogue act and c ( u i , u

Original: i − 1 ,
Revised: i − 1 ,

Original: a i − 1 )
Revised: a i − 1 )

Original: [ j ] be the j -th element of c ( u i , u
Revised: [ j ] be the j -th element of c ( u i , u

Original: i − 1 , a i − 1 ) , then the objective function of learning is formulated as
Revised: i − 1 , a i − 1 ) , then the objective function of learning is formulated as

Original: [Equation]
Revised: [Equation]

Original: Encoder 1 v 2 v 3 v 4 v 5 v
Revised: Encoder 1 v 2 v 3 v 4 v 5 v

Original: Decoder
Revised: Decoder

Original: Attention 2 ' v 3 ' v 4 ' v 6 v 1 ' v 5 ' v ...
Revised: Attention 2 ' v 3 ' v 4 ' v 6 v 1 ' v 5 ' v ...

Original: Softmax i a 1,1 i w  1,2 i w  1,3 i w  1,4 i w  1,5 i w  ,1 i w ,2 i w ,3 i w ,4 i w ,5 i w (a) generation network 𝑤 1 , 1 𝑤 1 ,𝑤 1 , 3 𝑤 1 ,𝑤 2 , 1 𝑤 2 ,𝑤 2 ,𝑤 3 , 1 𝑤 3 ,𝑤 3 , 3 𝑤 3 ,𝑤 4 , 1 𝑤 4 ,𝑤 4 , 3 𝑤 4 ,𝑎 1 𝑎 2 𝑎 3 𝑎
Revised: Softmax i a 1,1 i w  1,2 i w  1,3 i w  1,4 i w  1,5 i w  ,1 i w ,2 i w ,3 i w ,4 i w ,5 i w (a) generation network 𝑤 1 , 1 𝑤 1 ,𝑤 1 , 3 𝑤 1 ,𝑤 2 , 1 𝑤 2 ,𝑤 2 ,𝑤 3 , 1 𝑤 3 ,𝑤 3 , 3 𝑤 3 ,𝑤 4 , 1 𝑤 4 ,𝑤 4 , 3 𝑤 4 ,𝑎 1 𝑎 2 𝑎 3 𝑎

Original: CM.S
Revised: CM.S

Original: CM.S CM.Q
Revised: CM.S CM.Q

Original: CM.A CS.S
Revised: CM.A CS.S

Original: CS.Q CS.A 0.50 0.
Revised: CS.Q CS.A 0.50 0.

Original: 𝑤 2 ,ℎ 1𝑢 ℎ 2𝑢 ℎ 3𝑢 ℎ 4𝑢 ℎ 1 𝑎 ℎ 2𝑎 ℎ 3𝑎 ℎ 4𝑎
Revised: 𝑤 2 ,ℎ 1𝑢 ℎ 2𝑢 ℎ 3𝑢 ℎ 4𝑢 ℎ 1 𝑎 ℎ 2𝑎 ℎ 3𝑎 ℎ 4𝑎

Original: Other 0.
Revised: Other 0.

Original: MLP (b) policy network
Revised: MLP (b) policy network

Original: We randomly split the labeled dialogues as 400 / 30 / 70 dialogues with 3280 / 210 / 586 utterances for training/validation/test.
Revised: We randomly split the labeled dialogues as 400 / 30 / 70 dialogues with 3280 / 210 / 586 utterances for training/validation/test.

Original: Details of learning are given in Appendix 7.2.
Revised: Details of learning are given in Appendix 7.2.

Original: The learned classiﬁer achieves an accuracy of 70 .
Revised: The learned classiﬁer achieves an accuracy of 70 .

Original: 1 % on the test data.
Revised: 1 % on the test data.

Original: We employ it to tag the training, validation, and test sets in Table 2.
Revised: We employ it to tag the training, validation, and test sets in Table 2.

Original:  D IALOGUE G ENERATION M ODEL
Revised:  D IALOGUE G ENERATION M ODEL

Original: We present dialogue generation learning using large scale dialogues tagged with dialogue acts.
Revised: We present dialogue generation learning using large scale dialogues tagged with dialogue acts.

Original: Then, we describe model optimization with reinforcement learning for long-term conversation.
Revised: Then, we describe model optimization with reinforcement learning for long-term conversation.

Original: 3. S UPERVISED L EARNING
Revised: 3. S UPERVISED L EARNING

Original: We aim to learn a dialogue generation model g from D =
Revised: We aim to learn a dialogue generation model g from D =

Original: { d i } Ni =1 where d i = { ( u i, 1 , a i, 1 ) , . . .
Revised: { d i } Ni =1 where d i = { ( u i, 1 , a i, 1 ) , . . .

Original: , ( u i,n i , a i,n i ) } refers to a human-human dialogue with u i,k the k -th utterance and a i,k the dialogue act tagged by the classiﬁer in Section 2.4.
Revised: , ( u i,n i , a i,n i ) } refers to a human-human dialogue with u i,k the k -th utterance and a i,k the dialogue act tagged by the classiﬁer in Section 2.4.

Original: Given s i
Revised: Given s i

Original: = { ( u 1 , a 1 ) , . .
Revised: = { ( u 1 , a 1 ) , . .

Original: , ( u i − 1 , a i − 1 ) } as a new dialogue session, g ( s i ) can generate a response as the next turn of the dialogue.
Revised: , ( u i − 1 , a i − 1 ) } as a new dialogue session, g ( s i ) can generate a response as the next turn of the dialogue.

Original: Our dialogue model consists of a policy network and a generation network.
Revised: Our dialogue model consists of a policy network and a generation network.

Original: A dialogue act is ﬁrst selected from the policy network according to the conversation history, and then a response is generated from the generation network based on the conversation history and the dialogue act.
Revised: A dialogue act is ﬁrst selected from the policy network according to the conversation history, and then a response is generated from the generation network based on the conversation history and the dialogue act.

Original: Formally, the dialogue model can be formulated as
Revised: Formally, the dialogue model can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where a (cid:63)i =
Revised: where a (cid:63)i =

Original: O
Revised: O

Original: ( p a ( a i | s i )) is the selected dialogue act for the i -th turn, and r i is the response of the i -th turn.
Revised: ( p a ( a i | s i )) is the selected dialogue act for the i -th turn, and r i is the response of the i -th turn.

Original: p a is the policy network and p r is the generation network.
Revised: p a is the policy network and p r is the generation network.

Original: O ( · ) refers to a dialogue act select operation according to the value of the policy network.
Revised: O ( · ) refers to a dialogue act select operation according to the value of the policy network.

Original: A simple deﬁntion of O ( p a ( a i | s i )) is
Revised: A simple deﬁntion of O ( p a ( a i | s i )) is

Original: [Equation]
Revised: [Equation]

Original: where A is the space of dialogue acts.
Revised: where A is the space of dialogue acts.

Original: One can also customize O ( · ) with more complicated rules to achieve controllability or further optimization (e.g., improving response diveristy by selecting multiple acts) of their systems.
Revised: One can also customize O ( · ) with more complicated rules to achieve controllability or further optimization (e.g., improving response diveristy by selecting multiple acts) of their systems.

Original: Figure 1(b) shows the architecture of the policy network.
Revised: Figure 1(b) shows the architecture of the policy network.

Original: The utterance sequence and the act sequence are encoded with a hierarchical encoder and a GRU encoder respectively.
Revised: The utterance sequence and the act sequence are encoded with a hierarchical encoder and a GRU encoder respectively.

Original: Then, the last hidden states of the two encoders are concatenated and fed to an MLP to calculate a probability distribution of dialogue acts for the next turn.
Revised: Then, the last hidden states of the two encoders are concatenated and fed to an MLP to calculate a probability distribution of dialogue acts for the next turn.

Original: Formally, ∀ u j ∈ s i , u j is ﬁrst transformed to hidden vectors { h uj,k } n (cid:48)
Revised: Formally, ∀ u j ∈ s i , u j is ﬁrst transformed to hidden vectors { h uj,k } n (cid:48)

Original: j k =1 through a biGRU parameterized as Equation (1).
Revised: j k =1 through a biGRU parameterized as Equation (1).

Original: Then, { h uj,n (cid:48) j } i − 1 j =1 is processed by a GRU parameterized as t k = f u GRU ( t k − 1 , h uk,n (cid:48) k ) .
Revised: Then, { h uj,n (cid:48) j } i − 1 j =1 is processed by a GRU parameterized as t k = f u GRU ( t k − 1 , h uk,n (cid:48) k ) .

Original: In parallel, { a 1 , . . .
Revised: In parallel, { a 1 , . . .

Original: , a i − 1 } is transformed to { h ak } i − 1 k =1 by h ak = f a GRU ( h ak − 1 , e ( a k )) .
Revised: , a i − 1 } is transformed to { h ak } i − 1 k =1 by h ak = f a GRU ( h ak − 1 , e ( a k )) .

Original: p a ( a i | s i ) is then deﬁned by p
Revised: p a ( a i | s i ) is then deﬁned by p

Original: [Equation]
Revised: [Equation]

Original: We build the generation network in a sequence-to-sequence framework.
Revised: We build the generation network in a sequence-to-sequence framework.

Original: Here, we simplify p r ( r i | s i , a i ) as p r ( r i | a i , u i − 1 , u i − 2 ) since decoding natural language responses from long conversation history is challenging.
Revised: Here, we simplify p r ( r i | s i , a i ) as p r ( r i | a i , u i − 1 , u i − 2 ) since decoding natural language responses from long conversation history is challenging.

Original: Figure 1(a) illustrates the architecture of the generation network.
Revised: Figure 1(a) illustrates the architecture of the generation network.

Original: The only difference from the standard encoder-decoder architecture with an attention mechanism is that in encoding, we concatenate u i − 1 and u i − 2 , and attach a i to the top of the long sentence as a special word.
Revised: The only difference from the standard encoder-decoder architecture with an attention mechanism is that in encoding, we concatenate u i − 1 and u i − 2 , and attach a i to the top of the long sentence as a special word.

Original: The technique here is similar to that in zero-shot machine translation (Johnson et al., 2016).
Revised: The technique here is similar to that in zero-shot machine translation (Johnson et al., 2016).

Original: More formulation details can be found in Appendix 7.1.
Revised: More formulation details can be found in Appendix 7.1.

Original: The dialogue model is then learned by minimizing the negative log likelihood of D :
Revised: The dialogue model is then learned by minimizing the negative log likelihood of D :

Original: [Equation]
Revised: [Equation]

Original: where d i,<k = { ( u i, 1 , a i, 1 ) , . . .
Revised: where d i,<k = { ( u i, 1 , a i, 1 ) , . . .

Original: , ( u i,k − 1 , a i,k − 1 ) } .
Revised: , ( u i,k − 1 , a i,k − 1 ) } .

Original: Through supervised learning, we ﬁt the dialogue model to human-human interactions in order to learn their conversational patterns.
Revised: Through supervised learning, we ﬁt the dialogue model to human-human interactions in order to learn their conversational patterns.

Original: However, supervised learning does not explicitly encourage long-term conversation (e.g., 45 .
Revised: However, supervised learning does not explicitly encourage long-term conversation (e.g., 45 .

Original: 35 % dialogues in our training set are no more than 5 turns).
Revised: 35 % dialogues in our training set are no more than 5 turns).

Original: This motivates us to further optimize the model through a reinforcement learning approach.
Revised: This motivates us to further optimize the model through a reinforcement learning approach.

Original: 3. R EINFORCEMENT L EARNING
Revised: 3. R EINFORCEMENT L EARNING

Original: We optimize the dialogue model through self-play (Li et al., 2016b; Lewis et al., 2017) where we let two models learned with the supervised approach talk to each other in order to improve their performance.
Revised: We optimize the dialogue model through self-play (Li et al., 2016b; Lewis et al., 2017) where we let two models learned with the supervised approach talk to each other in order to improve their performance.

Original: In the simulation, a dialogue is initialized with a message sampled from the training set.
Revised: In the simulation, a dialogue is initialized with a message sampled from the training set.

Original: Then, the two models continue the dialogue by alternately taking the conversation history as an input and generating a response (top one in beam search) until T turns ( T = 20 in our experiments).
Revised: Then, the two models continue the dialogue by alternately taking the conversation history as an input and generating a response (top one in beam search) until T turns ( T = 20 in our experiments).

Original: To speed up training and avoid generated responses diverging from human language, we ﬁx the generation network and only optimize the policy network by reinforcement learning.
Revised: To speed up training and avoid generated responses diverging from human language, we ﬁx the generation network and only optimize the policy network by reinforcement learning.

Original: Thus, the policy in learning is naturally deﬁned by the policy network p a ( a i | s i ) with s i =
Revised: Thus, the policy in learning is naturally deﬁned by the policy network p a ( a i | s i ) with s i =

Original: { ( u 1 , a 1 ) , . . .
Revised: { ( u 1 , a 1 ) , . . .

Original: , ( u i − 1 , a i − 1 ) } a state and a i an action.
Revised: , ( u i − 1 , a i − 1 ) } a state and a i an action.

Original: We deﬁne a reward function r ( a i , s i ) as
Revised: We deﬁne a reward function r ( a i , s i ) as

Original: [Equation]
Revised: [Equation]

Original: where E [ len ( a i , s i )] is the expected conversation length after taking action a i under state s i , E [ rel ( a i , s i )] is the expected response relevance within the conversation, α = 0 .
Revised: where E [ len ( a i , s i )] is the expected conversation length after taking action a i under state s i , E [ rel ( a i , s i )] is the expected response relevance within the conversation, α = 0 .

Original: 67 , and β = 0 .
Revised: 67 , and β = 0 .

Original: Through Equation (8), we try to encourage actions that can lead to long (measured by E [ len ( a i , s i )] ) and reasonable (measured by E [ rel ( a i , s i )] ) conversations.
Revised: Through Equation (8), we try to encourage actions that can lead to long (measured by E [ len ( a i , s i )] ) and reasonable (measured by E [ rel ( a i , s i )] ) conversations.

Original: To estimate E [ len ( a i , s i )] and E [ rel ( a i , s i )] , we ﬁx ( s i , a i ) and construct a dialogue set { d (cid:48)
Revised: To estimate E [ len ( a i , s i )] and E [ rel ( a i , s i )] , we ﬁx ( s i , a i ) and construct a dialogue set { d (cid:48)

Original: i,j } Nj =1 ( N = 10 in our experiments) by sampling after ( s i , a i ) with self-play.
Revised: i,j } Nj =1 ( N = 10 in our experiments) by sampling after ( s i , a i ) with self-play.

Original: ∀ j , d (cid:48)
Revised: ∀ j , d (cid:48)

Original: i,j = ( s i , u j,i +1 , . . .
Revised: i,j = ( s i , u j,i +1 , . . .

Original: , u j,n i,j ) where ∀ k , u j,i + k is randomly sampled from the top 5 beam search results of p r conditioned on the most probable dialogue act given by p a for that turn.
Revised: , u j,n i,j ) where ∀ k , u j,i + k is randomly sampled from the top 5 beam search results of p r conditioned on the most probable dialogue act given by p a for that turn.

Original: Inspired by (Li et al., 2016b), we terminate a simulated dialogue if (1) cosine ( e ( u i − 1 ) , e ( u i )) > 0 . 9 && cosine ( e ( u i )) , e ( u i +1 )) > 0 .
Revised: Inspired by (Li et al., 2016b), we terminate a simulated dialogue if (1) cosine ( e ( u i − 1 ) , e ( u i )) > 0 . 9 && cosine ( e ( u i )) , e ( u i +1 )) > 0 .

Original: 9 , or (2) cosine ( e ( u i − 1 ) , e ( u i +1 )) > 0 . 9 , or (3) the length of the dialogue reaches T , where e ( · ) denotes the representation of an utterance given by the encoder of p r .
Revised: 9 , or (2) cosine ( e ( u i − 1 ) , e ( u i +1 )) > 0 . 9 , or (3) the length of the dialogue reaches T , where e ( · ) denotes the representation of an utterance given by the encoder of p r .

Original: Condition (1) means three consecutive turns are (semantically) repetitive, and Condition (2) means one agent gives repetitive responses in two consecutive turns.
Revised: Condition (1) means three consecutive turns are (semantically) repetitive, and Condition (2) means one agent gives repetitive responses in two consecutive turns.

Original: Both conditions indicate a high probability that the conversation falls into a bad inﬁnite loop.
Revised: Both conditions indicate a high probability that the conversation falls into a bad inﬁnite loop.

Original: E [ len ( a i , s i )] and E [ rel ( a i , s i )] are then estimated by
Revised: E [ len ( a i , s i )] and E [ rel ( a i , s i )] are then estimated by

Original: [Equation]
Revised: [Equation]

Original: where d i,j<k
Revised: where d i,j<k

Original: = ( u 1 , . .
Revised: = ( u 1 , . .

Original: , u j,k − 1 ) , and m ( · , · ) is the dual LSTM model proposed in (Lowe et al., 2015) which measures the relevance between a response and a context.
Revised: , u j,k − 1 ) , and m ( · , · ) is the dual LSTM model proposed in (Lowe et al., 2015) which measures the relevance between a response and a context.

Original: We train m ( · , · ) with the 30 million crawled data through negative sampling.
Revised: We train m ( · , · ) with the 30 million crawled data through negative sampling.

Original: The objective of learning is to maximize the expected future reward:
Revised: The objective of learning is to maximize the expected future reward:

Original: [Equation]
Revised: [Equation]

Original: The gradient of the objective is calculated by Reinforce algorithm (Williams, 1992):
Revised: The gradient of the objective is calculated by Reinforce algorithm (Williams, 1992):

Original: [Equation]
Revised: [Equation]

Original: where the baseline b t is empirically set as 1 | A | (cid:80) a t ∈
Revised: where the baseline b t is empirically set as 1 | A | (cid:80) a t ∈

Original: A r ( a t , s t ) .
Revised: A r ( a t , s t ) .

Original:  E XPERIMENT 4. E XPERIMENT S ETUP
Revised:  E XPERIMENT 4. E XPERIMENT S ETUP

Original: Our experiments are conducted with the data in Table 2.
Revised: Our experiments are conducted with the data in Table 2.

Original: The following methods are employed as baselines: (1) S2SA : sequence-to-sequence with attention (Bahdanau et al., 2015) in which utterances in contexts are concatenated as a long sentence.
Revised: The following methods are employed as baselines: (1) S2SA : sequence-to-sequence with attention (Bahdanau et al., 2015) in which utterances in contexts are concatenated as a long sentence.

Original: We use the implementation with Blocks ( https://github.com/mila-udem/blocks ); (2) HERD : the hierarchical encoder-decoder model in (Serban et al., 2016) implemented with the source code available at ( https://github.com/julianser/hed-dlg-truncated ); (3) VHERD : the hierarchical latent variable encoder-decoder model in (Serban et al., 2017) implemented with the source code available at ( https://github.com/julianser/hed-dlg-truncated ); and (4) RL-S2S : dialogue generation with reinforcement learning (Li et al., 2016b).
Revised: We use the implementation with Blocks ( https://github.com/mila-udem/blocks ); (2) HERD : the hierarchical encoder-decoder model in (Serban et al., 2016) implemented with the source code available at ( https://github.com/julianser/hed-dlg-truncated ); (3) VHERD : the hierarchical latent variable encoder-decoder model in (Serban et al., 2017) implemented with the source code available at ( https://github.com/julianser/hed-dlg-truncated ); and (4) RL-S2S : dialogue generation with reinforcement learning (Li et al., 2016b).

Original: We implement the algorihtm by ﬁnishing the code at ( https://github.com/liuyuemaicha/
Revised: We implement the algorihtm by ﬁnishing the code at ( https://github.com/liuyuemaicha/

Original: Deep-Reinforcement-Learning-for-Dialogue-Generation-in-tensorflow ).
Revised: Deep-Reinforcement-Learning-for-Dialogue-Generation-in-tensorflow ).

Original: Dull responses are deﬁned as in (Li et al., 2016b) and listed in Appendix 7.3.
Revised: Dull responses are deﬁned as in (Li et al., 2016b) and listed in Appendix 7.3.

Original: All baseline models are implemented with the recommended conﬁgurations in the existing literatures.
Revised: All baseline models are implemented with the recommended conﬁgurations in the existing literatures.

Original: We denote our Dialogue Act aware Generation Model with only Supervised Learning as
Revised: We denote our Dialogue Act aware Generation Model with only Supervised Learning as

Original: SL-DAGM, and the full model (supervised learning + reinforcement learning) as RL-DAGM.
Revised: SL-DAGM, and the full model (supervised learning + reinforcement learning) as RL-DAGM.

Original: Implementation details are given in Appendix 7.3.
Revised: Implementation details are given in Appendix 7.3.

Original: 4. R ESPONSE G ENERATION FOR G IVEN C ONTEXTS
Revised: 4. R ESPONSE G ENERATION FOR G IVEN C ONTEXTS

Original: The ﬁrst experiment is to check if the proposed models can generate high-quality responses regarding to given contexts.
Revised: The ﬁrst experiment is to check if the proposed models can generate high-quality responses regarding to given contexts.

Original: To this end, we take the last turn of each test dialogue as ground truth, and feed the previous turns as a context to different models for response generation.
Revised: To this end, we take the last turn of each test dialogue as ground truth, and feed the previous turns as a context to different models for response generation.

Original: Top one responses from beam search (beam size = 20 ) of different models are collected, randomly shufﬂed, and presented to 3 native speakers to judge their quality.
Revised: Top one responses from beam search (beam size = 20 ) of different models are collected, randomly shufﬂed, and presented to 3 native speakers to judge their quality.

Original: Each response is rated by the three annotators under the following criteria: 2 : the response is not only relevant and natural, but also informative and interesting
Revised: Each response is rated by the three annotators under the following criteria: 2 : the response is not only relevant and natural, but also informative and interesting

Original: ; 1 : the response can be used as a reply, but might not be informative enough (e.g.,“Yes, I see” etc.); 0 : the response makes no sense, is irrelevant, or is grammatically broken.
Revised: ; 1 : the response can be used as a reply, but might not be informative enough (e.g.,“Yes, I see” etc.); 0 : the response makes no sense, is irrelevant, or is grammatically broken.

Original: [Table]
Revised: [Table]

Original: Table 4(a) summarizes the annotation results.
Revised: Table 4(a) summarizes the annotation results.

Original: Improvements from our models over the baseline methods are statistically signiﬁcant (t-test, p-value < 0 . 01 ).
Revised: Improvements from our models over the baseline methods are statistically signiﬁcant (t-test, p-value < 0 . 01 ).

Original: In addition to human annotations, we also compare different models using automatic metrics with the the ground truth.
Revised: In addition to human annotations, we also compare different models using automatic metrics with the the ground truth.

Original: These metrics include (1) BLEU (Papineni et al., 2002) which measures term overlap of two responses; (2) embedding based metrics (Liu et al., 2016) such as Embedding Average (Average), Embedding Extrema (Extrema), and Embedding Greedy (Greedy) which measure similarity of two responses in a semantic space; and (3) ratios of distinct unigrams (distinct-1) and bigrams (distinct-2) in the generated responses which are employed in (Li et al., 2015) to measure response diversity.
Revised: These metrics include (1) BLEU (Papineni et al., 2002) which measures term overlap of two responses; (2) embedding based metrics (Liu et al., 2016) such as Embedding Average (Average), Embedding Extrema (Extrema), and Embedding Greedy (Greedy) which measure similarity of two responses in a semantic space; and (3) ratios of distinct unigrams (distinct-1) and bigrams (distinct-2) in the generated responses which are employed in (Li et al., 2015) to measure response diversity.

Original: Table 5 reports the automatic evaluation results.
Revised: Table 5 reports the automatic evaluation results.

Original: We can see that one beneﬁt brought by the dialogue acts is that diversity of responses is signiﬁcantly improved.
Revised: We can see that one beneﬁt brought by the dialogue acts is that diversity of responses is signiﬁcantly improved.

Original: This is supported by the much more 2 responses from the two models in Table 4(a) and the signiﬁcant improvement on distinct n-grams in Table 5.
Revised: This is supported by the much more 2 responses from the two models in Table 4(a) and the signiﬁcant improvement on distinct n-grams in Table 5.

Original: The reason is easy to understand: we search a response not only from a language space, but also from an act space.
Revised: The reason is easy to understand: we search a response not only from a language space, but also from an act space.

Original: The dimension of dialogue acts provides further variations to the generated responses.
Revised: The dimension of dialogue acts provides further variations to the generated responses.

Original: On the other hand, due to the diversity, responses from our models may diverge from the ground truth sometimes.
Revised: On the other hand, due to the diversity, responses from our models may diverge from the ground truth sometimes.

Original: This is why
Revised: This is why

Original: [Table] [Table]
Revised: [Table] [Table]

Original: improvements on other automatic metrics are not signiﬁcant.
Revised: improvements on other automatic metrics are not signiﬁcant.

Original: To further explain the advantages of our models, we show an example in Table 6.
Revised: To further explain the advantages of our models, we show an example in Table 6.

Original: In addition to responses from SL-DAGM and RLDAGM which are selected from the dialogue acts obtained by Equation (5), we also show responses from other reasonable but not selected acts.
Revised: In addition to responses from SL-DAGM and RLDAGM which are selected from the dialogue acts obtained by Equation (5), we also show responses from other reasonable but not selected acts.

Original: With dialogue acts, responses from our models become really rich, from conﬁrmation (CM.Q) to an open question (CS.Q) and then to a long informative statement (CS.S).
Revised: With dialogue acts, responses from our models become really rich, from conﬁrmation (CM.Q) to an open question (CS.Q) and then to a long informative statement (CS.S).

Original: More importantly, the dialogue acts let us know why we have such responses: both SL-DAGM and RL-DAGM try to switch to new topics (e.g., Xiamen, noodle, and plan etc.)
Revised: More importantly, the dialogue acts let us know why we have such responses: both SL-DAGM and RL-DAGM try to switch to new topics (e.g., Xiamen, noodle, and plan etc.)

Original: in order to continue the conversation.
Revised: in order to continue the conversation.

Original: One can also change the ﬂow of the conversation by picking responses from other dialogue acts.
Revised: One can also change the ﬂow of the conversation by picking responses from other dialogue acts.

Original: The example demonstrates that in addition to good performance, our models enjoy good interpretability and controllability as well.
Revised: The example demonstrates that in addition to good performance, our models enjoy good interpretability and controllability as well.

Original: We show more such examples in
Revised: We show more such examples in

Original: Appendix 7.4.
Revised: Appendix 7.4.

Original: 4. E NGAGEMENT T EST
Revised: 4. E NGAGEMENT T EST

Original: Secondly, we study conversation engagement with the proposed models.
Revised: Secondly, we study conversation engagement with the proposed models.

Original: Experiments are conducted through machine-machine simulation and human-machine conversation.
Revised: Experiments are conducted through machine-machine simulation and human-machine conversation.

Original: In both experiments, we compare SL-DAGM and RL-DAGM with RL-S2S, as RL-S2S is the only baseline optimized for future success.
Revised: In both experiments, we compare SL-DAGM and RL-DAGM with RL-S2S, as RL-S2S is the only baseline optimized for future success.

Original: Responses from all models are randomly sampled from the top 5 beam search results.
Revised: Responses from all models are randomly sampled from the top 5 beam search results.

Original: Average length of dialogues is employed as an evaluation metric.
Revised: Average length of dialogues is employed as an evaluation metric.

Original: Machine-machine simulation is conducted in a way similar to (Li et al., 2016b) in which we let two bots equipped with the same model talk with each other in 1000 simulated dialogues.
Revised: Machine-machine simulation is conducted in a way similar to (Li et al., 2016b) in which we let two bots equipped with the same model talk with each other in 1000 simulated dialogues.

Original: Each dialogue is initialized with the ﬁrst utterance of a test example, and terminated according to the termination conditions for reward estimation in Section 3.2.
Revised: Each dialogue is initialized with the ﬁrst utterance of a test example, and terminated according to the termination conditions for reward estimation in Section 3.2.

Original: In human-machine conversation, we recruit 5 native speakers as testers and ask them to talk with the bots equipped with the three models.
Revised: In human-machine conversation, we recruit 5 native speakers as testers and ask them to talk with the bots equipped with the three models.

Original: Every time, a bot is randomly picked for a tester, and the tester does not know which model is behind.
Revised: Every time, a bot is randomly picked for a tester, and the tester does not know which model is behind.

Original: Every tester ﬁnishes 100 dialogues with each bot.
Revised: Every tester ﬁnishes 100 dialogues with each bot.

Original: To make a fair comparison, we let the bots start dialgoues.
Revised: To make a fair comparison, we let the bots start dialgoues.

Original: A starting message in a dialogue is randomly sampled from the test data and copiedtimes for all the 3 bots (a tester can skip the message if he/she cannot understand it).
Revised: A starting message in a dialogue is randomly sampled from the test data and copiedtimes for all the 3 bots (a tester can skip the message if he/she cannot understand it).

Original: A dialogue is terminated if (1) the tester thinks the conversation cannot be continued (e.g., due to bad relevance or repetitive content etc.); or (2) the bot gives repetitive responses in two consecutive turns (measured by cosine ( e ( u i − 1 ) , e ( u i +1 )) > 0 . 9 ).
Revised: A dialogue is terminated if (1) the tester thinks the conversation cannot be continued (e.g., due to bad relevance or repetitive content etc.); or (2) the bot gives repetitive responses in two consecutive turns (measured by cosine ( e ( u i − 1 ) , e ( u i +1 )) > 0 . 9 ).

Original: Dialogue acts in human turns are tagged by the classiﬁer in
Revised: Dialogue acts in human turns are tagged by the classiﬁer in

Original: Section 2.4.
Revised: Section 2.4.

Original: The evaluation metric is calculated with the total 500 dialogues for each model.
Revised: The evaluation metric is calculated with the total 500 dialogues for each model.

Original: Table 4(b) reports the evaluation results.
Revised: Table 4(b) reports the evaluation results.

Original: In both experiments, SL-DAGM and RL-DAGM can lead to longer conversations, and the improvements from both models over the baseline are statistically signiﬁcant (t-test, p-value < 0 . 01 ).
Revised: In both experiments, SL-DAGM and RL-DAGM can lead to longer conversations, and the improvements from both models over the baseline are statistically signiﬁcant (t-test, p-value < 0 . 01 ).

Original: Improvements in human-machine conversation are smaller than those in machine-machine simulation, indicating the gap between the simulation environment and the real conversation environment and encouraging us to consider online optimization in humanmachine conversations in the future.
Revised: Improvements in human-machine conversation are smaller than those in machine-machine simulation, indicating the gap between the simulation environment and the real conversation environment and encouraging us to consider online optimization in humanmachine conversations in the future.

Original: RL-DAGM is better than SL-DAGM in both experiments, indicating the efﬁcacy of reinforcement learning.
Revised: RL-DAGM is better than SL-DAGM in both experiments, indicating the efﬁcacy of reinforcement learning.

Original: The reason that our models are better is that they captured conversational patterns in human-human interactions and obtained further optimization through reinforcement learning.
Revised: The reason that our models are better is that they captured conversational patterns in human-human interactions and obtained further optimization through reinforcement learning.

Original: First, the models can pro-actively switch contexts in a smooth way.
Revised: First, the models can pro-actively switch contexts in a smooth way.

Original: In machine-machine simulation, 65 . 4 % (SL) and 94 .
Revised: In machine-machine simulation, 65 . 4 % (SL) and 94 .

Original: 4 % (RL) dialogues contain at least one CS.
Revised: 4 % (RL) dialogues contain at least one CS.

Original: *; and in human-machine conversation, the two percentages are 38 .
Revised: *; and in human-machine conversation, the two percentages are 38 .

Original: 1 % (SL) and 48 .
Revised: 1 % (SL) and 48 .

Original: 1 % (RL) respectively.
Revised: 1 % (RL) respectively.

Original: More interestingly, in machine-machine simulation, average lengths of dialogues without CS.
Revised: More interestingly, in machine-machine simulation, average lengths of dialogues without CS.

Original: * are only 4 .
Revised: * are only 4 .

Original: 78 (SL) and 2 .
Revised: 78 (SL) and 2 .

Original: 67 (RL) respectively which are comparable with or even worse than RL-S2S, while average lengths of dialogues with CS.
Revised: 67 (RL) respectively which are comparable with or even worse than RL-S2S, while average lengths of dialogues with CS.

Original: * are 8 .
Revised: * are 8 .

Original: 66 (SL) and 8 .
Revised: 66 (SL) and 8 .

Original: (RL) respectively.
Revised: (RL) respectively.

Original: The results demonstrate the importance of context switch for engagement in open domain conversation and one signﬁcant effect of RL is promoting context switch in interactions for future engagment even with sacriﬁce on relevance of the current turn (e.g., more 0 responses than SL-DAGM in Table 4(a)).
Revised: The results demonstrate the importance of context switch for engagement in open domain conversation and one signﬁcant effect of RL is promoting context switch in interactions for future engagment even with sacriﬁce on relevance of the current turn (e.g., more 0 responses than SL-DAGM in Table 4(a)).

Original: Second, the models can drive conversations by asking questions.
Revised: Second, the models can drive conversations by asking questions.

Original: In machine-machine simulation, 36 .
Revised: In machine-machine simulation, 36 .

Original: 5 % (SL) and 32 .
Revised: 5 % (SL) and 32 .

Original: 4 % (RL) dialogues contain at least one question.
Revised: 4 % (RL) dialogues contain at least one question.

Original: The percentages in human-machine conversation are 17 .
Revised: The percentages in human-machine conversation are 17 .

Original: 7 % (SL) and 22 .
Revised: 7 % (SL) and 22 .

Original: 3 % (RL) respectively.
Revised: 3 % (RL) respectively.

Original: We give more analysis in Appendix 7.5.
Revised: We give more analysis in Appendix 7.5.

Original: 4. D ISCUSSION
Revised: 4. D ISCUSSION

Original: Finally, we study how the generated responses are affected by the dialogue acts.
Revised: Finally, we study how the generated responses are affected by the dialogue acts.

Original: We collect generated responses from a speciﬁc dialogue act for the contexts of the test dialogues, and characterize the responses with the following metrics: (1) distinct-1 and distinct-2; (2) words out of context (OOC): ratio of words that are in the generated responses but not contained by the contexts; and (3) average length of the generated responses (Ave Len).
Revised: We collect generated responses from a speciﬁc dialogue act for the contexts of the test dialogues, and characterize the responses with the following metrics: (1) distinct-1 and distinct-2; (2) words out of context (OOC): ratio of words that are in the generated responses but not contained by the contexts; and (3) average length of the generated responses (Ave Len).

Original: [Table]
Revised: [Table]

Original: Table 7 reports the results 4 .
Revised: Table 7 reports the results 4 .

Original: In general, responses generated from CS.
Revised: In general, responses generated from CS.

Original: * are longer, more informative, and contain more new words than responses generated from CM.
Revised: * are longer, more informative, and contain more new words than responses generated from CM.

Original: *, which has been illustrated by the example in Table 6.
Revised: *, which has been illustrated by the example in Table 6.

Original: Another interesting ﬁnding is that statements and answers are generally more informative than questions in both CS.
Revised: Another interesting ﬁnding is that statements and answers are generally more informative than questions in both CS.

Original: * and CM.
Revised: * and CM.

Original: In addition to these metrics, we also calculate BLEU scores and embedding based metrics, but do not observe signiﬁcant difference among responses from different dialogue acts.
Revised: In addition to these metrics, we also calculate BLEU scores and embedding based metrics, but do not observe signiﬁcant difference among responses from different dialogue acts.

Original: The reason might be that these metrics are based on comparsion of the generated responses and human responses, but human responses in the test set are inherently mixture of responses from different dialogue acts.
Revised: The reason might be that these metrics are based on comparsion of the generated responses and human responses, but human responses in the test set are inherently mixture of responses from different dialogue acts.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Existing dialogue models are either built for open domain conversation or for speciﬁc task completion.
Revised: Existing dialogue models are either built for open domain conversation or for speciﬁc task completion.

Original: Regarding to the former, a common practice is to learn a generation model in an end-to-end fashion.
Revised: Regarding to the former, a common practice is to learn a generation model in an end-to-end fashion.

Original: On top of the basic sequence-to-sequence with attention architecture (Vinyals & Le, 2015;
Revised: On top of the basic sequence-to-sequence with attention architecture (Vinyals & Le, 2015;

Original: Shang et al., 2015), various extensions have been proposed to tackle the “safe response” problem 
Revised: Shang et al., 2015), various extensions have been proposed to tackle the “safe response” problem (Li et al., 2015; Mou et al., 2016;

Original: (Li et al., 2015; Mou et al., 2016; Xing et al., 2017); to model complicated structures of conversation contexts (Serban et al., 2016; Sordoni et al., 2015); to bias responses to some speciﬁc persona or emotions (Li et al., 2016a;
Revised: Xing et al., 2017); to model complicated structures of conversation contexts (Serban et al., 2016; Sordoni et al., 2015); to bias responses to some speciﬁc persona or emotions (Li et al., 2016a;

Original: Zhou et al., 2017); and to pursue better optimization strategies (Li et al., 2017b; 2016b).
Revised: Zhou et al., 2017); and to pursue better optimization strategies (Li et al., 2017b; 2016b).

Original: On the other line of research, POMDP (Young et al., 2013) breaks down the development of task-oriented dialogue systems into natural language understanding (Yao et al., 2014; Henderson et al., 2014), dialogue management (Mrkˇsi´c et al., 2016), and response generation (Wen et al., 2015).
Revised: On the other line of research, POMDP (Young et al., 2013) breaks down the development of task-oriented dialogue systems into natural language understanding (Yao et al., 2014; Henderson et al., 2014), dialogue management (Mrkˇsi´c et al., 2016), and response generation (Wen et al., 2015).

Original: Recently, researchers also consider learning task-oriented dialogue models in an end-to-end way (Wen et al., 2016; 2017; Bordes & Weston, 2017).
Revised: Recently, researchers also consider learning task-oriented dialogue models in an endto-end way (Wen et al., 2016; 2017; Bordes & Weston, 2017).

Original: In this work, we introduce dialogue acts into open domain dialogue generation.
Revised: In this work, we introduce dialogue acts into open domain dialogue generation.

Original: Although Zhao et al.
Revised: Although Zhao et al.

Original: (2017) have leveraged dialogue acts in the Switchboard Corpus as prior knowledge in dialogue generation, the dialogue acts in this work are generally designed for explaining engagement in social chat.
Revised: (2017) have leveraged dialogue acts in the Switchboard Corpus as prior knowledge in dialogue generation, the dialogue acts in this work are generally designed for explaining engagement in social chat and modelled as policies.

Original: To the best of our knowledge, we are the ﬁrst who design dialogue acts to explain social interactions and control open domain dialogue generation.
Revised: To the best of our knowledge, we are the ﬁrst who design dialogue acts to explain social interactions, control open domain response generation, and guide human-machine conversations.

Original: Before us, some researchers have proposed analyzing open domain dialogues with dialogue acts (Kim et al., 2010; 2012;
Revised: Before us, some researchers have proposed analyzing open domain dialogues with dialogue acts (Kim et al., 2010; 2012;

Original: Oraby et al., 2017; Ivanovic, 2005; Wallace et al., 2013; Wu et al., 2005;
Revised: Oraby et al., 2017; Ivanovic, 2005; Wallace et al., 2013; Wu et al., 2005;

Original: Ritter et al., 2010).
Revised: Ritter et al., 2010).

Original: These work, however, stops at performing utterance classiﬁcation or clustering.
Revised: These work, however, stops at performing utterance classiﬁcation or clustering.

Original: Our dialogue act design is inspired by these work, but we not only exploit the dialogue acts to interpret open domain dialogues, but also conduct dialogue generation with the dialogue acts.
Revised: Our dialogue act design is inspired by these work, but we not only exploit the dialogue acts to interpret open domain dialogues, but also conduct dialogue generation with the dialogue acts.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We study open domain dialogue generation with generally designed dialogue acts that can describe human behavior in social interactions.
Revised: We study open domain dialogue generation with generally designed dialogue acts that can describe human behavior in social interactions.

Original: To mimic such behavior, we propose jointly modeling dialogue act selection and response generation, and perform both supervised learning with a learned dialogue act classiﬁer and reinforcement learning for long-term conversation. Empirical studies on response generation for given contexts, machine-machine simulation, and human-machine conversation show that the proposed models can signiﬁcantly outperform state-of-the-art methods.
Revised: To mimic such behavior, we propose jointly modeling dialogue act selection and response generation, and perform both supervised learning with a learned dialogue act classiﬁer and reinforcement learning for long term response generation for given contexts machine-machine simulation, and human-machine conversation show that the proposed models can signiﬁcantly outperform state-of-the-art methods

Original:  A PPENDIX 7. G ENERATION N ETWORK
Revised:  A PPENDIX 7. G ENERATION N ETWORK

Original: Suppose that x i
Revised: Suppose that x i

Original: =
Revised: =

Original: [ a i ; u i − 1 ; u i − 2 ] =
Revised: [ a i ; u i − 1 ; u i − 2 ] =

Original: ( w i, 1 , . . .
Revised: ( w i, 1 , . . .

Original: , w i,n (cid:48) i ) where w i,k is the embedding of the k -th word, then the k -th hidden state of the encoder is given by v i,k =
Revised: , w i,n (cid:48) i ) where w i,k is the embedding of the k -th word, then the k -th hidden state of the encoder is given by v i,k =

Original: [ −→ v i,k ; ←− v i,k ] where
Revised: [ −→ v i,k ; ←− v i,k ] where

Original: [Equation]
Revised: [Equation]

Original: Positions of u − 1 and u 0 in x 1 and x 2 are padded with zeros.
Revised: Positions of u − 1 and u 0 in x 1 and x 2 are padded with zeros.

Original: Let r i =
Revised: Let r i =

Original: ( w (cid:48)
Revised: ( w (cid:48)

Original: i, 1 , . .
Revised: i, 1 , . .

Original: . , w (cid:48)
Revised: . , w (cid:48)

Original: i,T ) , then in decoding the j -th word w (cid:48)
Revised: i,T ) , then in decoding the j -th word w (cid:48)

Original: i,j , { v i, 1 , . .
Revised: i,j , { v i, 1 , . .

Original: . , v i,n (cid:48)
Revised: . , v i,n (cid:48)

Original: i } is summarized as a context vector c i,j through an attention mechanism:
Revised: i } is summarized as a context vector c i,j through an attention mechanism:

Original: [Equation]
Revised: [Equation]

Original: where v and W α are parameters, and v (cid:48)
Revised: where v and W α are parameters, and v (cid:48)

Original: i,j − 1 is the ( j − 1) -th hidden state of the decoder GRU in which v (cid:48)
Revised: i,j − 1 is the ( j − 1) -th hidden state of the decoder GRU in which v (cid:48)

Original: i,j is calculated by
Revised: i,j is calculated by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The generation probability of w i,j is then deﬁned as
Revised: The generation probability of w i,j is then deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where I ( w (cid:48)
Revised: where I ( w (cid:48)

Original: i,j ) is a vector with only one element 1 indicating the index of w (cid:48)
Revised: i,j ) is a vector with only one element 1 indicating the index of w (cid:48)

Original: i,j in the vocabulary.
Revised: i,j in the vocabulary.

Original: p r
Revised: p r

Original: ( r i | a i , u i − 1 , u i − 2 ) is ﬁnally deﬁned as
Revised: ( r i | a i , u i − 1 , u i − 2 ) is ﬁnally deﬁned as

Original: [Equation]
Revised: [Equation]

Original: 7. D ETAILS OF L EARNING THE D IALOGUE A CT C LASSIFIER
Revised: 7. D ETAILS OF L EARNING THE D IALOGUE A CT C LASSIFIER

Original: We randomly split the 500 labeled dialogues as 400 , 30 , and 70 dialogues for training, validation, and test respectively.
Revised: We randomly split the 500 labeled dialogues as 400 , 30 , and 70 dialogues for training, validation, and test respectively.

Original: Utterances in the three sets are 3280 , 210 , and 586 respectively.
Revised: Utterances in the three sets are 3280 , 210 , and 586 respectively.

Original: In training, we represent dialogue acts as probability distributions by averaging the labels given by the three annotators.
Revised: In training, we represent dialogue acts as probability distributions by averaging the labels given by the three annotators.

Original: For example, if an utterance is labeled as “CM.S”, “CM.S”, and “CS.S”, then the probability distribution is (0 .
Revised: For example, if an utterance is labeled as “CM.S”, “CM.S”, and “CS.S”, then the probability distribution is (0 .

Original: In test, we predict the dialogue act of an utterance u i by arg max j g ( u i , u
Revised: In test, we predict the dialogue act of an utterance u i by arg max j g ( u i , u

Original: i − 1 , a i − 1 )
Revised: i − 1 , a i − 1 )

Original: [ j ] .
Revised: [ j ] .

Original: To avoid overﬁtting, we pre-train word embeddings using word2vecwith an embedding size of 200 on the 30 million data and ﬁx them in training.
Revised: To avoid overﬁtting, we pre-train word embeddings using word2vecwith an embedding size of 200 on the 30 million data and ﬁx them in training.

Original: We set the embedding size of the dialogue acts and the hidden state size of the biGRUs as 100 , and the dimensions of the ﬁrst layer and the second layer of the MLP as 200 and 7 respectively.
Revised: We set the embedding size of the dialogue acts and the hidden state size of the biGRUs as 100 , and the dimensions of the ﬁrst layer and the second layer of the MLP as 200 and 7 respectively.

Original: We optimize the objective function (3) using back-propagation and the parameters are updated by stochastic gradient descent with AdaDelta algorithm
Revised: We optimize the objective function (3) using back-propagation and the parameters are updated by stochastic gradient descent with AdaDelta algorithm

Original: (Zeiler, 2012).
Revised: (Zeiler, 2012).

Original: The best performing model on the validation data is picked up for test.
Revised: The best performing model on the validation data is picked up for test.

Original: 7. D ETAILS OF IMPLEMENTATION OF THE DIALOGUE GENERATION MODEL
Revised: 7. D ETAILS OF IMPLEMENTATION OF THE DIALOGUE GENERATION MODEL

Original: In learning of the generation network, we set the size of word embedding as 620 and the size of hidden vectors as 1024 in both the encoder and the decoder.
Revised: In learning of the generation network, we set the size of word embedding as 620 and the size of hidden vectors as 1024 in both the encoder and the decoder.

Original: Both the encoder vocabulary and the decoder vocabulary contain 30 , 000 words.
Revised: Both the encoder vocabulary and the decoder vocabulary contain 30 , 000 words.

Original: Words out of the vocabularies are replaced by a special token “UNK”.
Revised: Words out of the vocabularies are replaced by a special token “UNK”.

Original: We employ AdaDelta algorithm
Revised: We employ AdaDelta algorithm

Original: (Zeiler, 2012) to train the generation network with a batch size 128 .
Revised: (Zeiler, 2012) to train the generation network with a batch size 128 .

Original: We set the initial learning rate as 1 . 0 and reduce it by half if perplexity on validation begins to increase.
Revised: We set the initial learning rate as 1 . 0 and reduce it by half if perplexity on validation begins to increase.

Original: We stop training if the perplexity on validation keeps increasing in two successive epochs.
Revised: We stop training if the perplexity on validation keeps increasing in two successive epochs.

Original: In learning of the policy network, we set the size of word embedding, the size of dialogue act, and the size of hidden states of the biGRU as 100 .
Revised: In learning of the policy network, we set the size of word embedding, the size of dialogue act, and the size of hidden states of the biGRU as 100 .

Original: There are 50 neurons in the ﬁrst layer of the MLP and 7 neurons in the second layer of the MLP.
Revised: There are 50 neurons in the ﬁrst layer of the MLP and 7 neurons in the second layer of the MLP.

Original: Vectors in the policy network have smaller sizes than those in the generation network because the complexity of dialogue act prediction is much lower than language generation.
Revised: Vectors in the policy network have smaller sizes than those in the generation network because the complexity of dialogue act prediction is much lower than language generation.

Original: In reinforcement learning, the size of mini-batch is 60 and learning rate is ﬁxed as 0 .
Revised: In reinforcement learning, the size of mini-batch is 60 and learning rate is ﬁxed as 0 .

Original: To estimate the reward, we train a dual LSTM (Lowe et al., 2015) with the size of word embedding and the size of hidden states as 100 .
Revised: To estimate the reward, we train a dual LSTM (Lowe et al., 2015) with the size of word embedding and the size of hidden states as 100 .

Original: Responses from the simulated dialogues are generated with a beam size 20 .
Revised: Responses from the simulated dialogues are generated with a beam size 20 .

Original: In RL-S2S, we deﬁne 8 responses as dull responses according to the frequency of responses in the training set.
Revised: In RL-S2S, we deﬁne 8 responses as dull responses according to the frequency of responses in the training set.

Original: Table 8 gives the responses.
Revised: Table 8 gives the responses.

Original: [Table]
Revised: [Table]

Original: M ORE E XAMPLES OF R ESPONSE G ENERATION
Revised: M ORE E XAMPLES OF R ESPONSE G ENERATION

Original: We compare SL-DAGM and RL-DAGM with baseline models in terms of response quality for given contexts with more examples in Table 9.
Revised: We compare SL-DAGM and RL-DAGM with baseline models in terms of response quality for given contexts with more examples in Table 9.

Original: 7. F URTHER ANALYSIS ON ENGAGMENT TEST
Revised: 7. F URTHER ANALYSIS ON ENGAGMENT TEST

Original: Table 10 gives some examples on machine-machine simulation.
Revised: Table 10 gives some examples on machine-machine simulation.

Original: Unlike the dialogues from RL-S2S which quickly converge to loops, dialogues from our models smoothly move forward under the management of the dialogue acts.
Revised: Unlike the dialogues from RL-S2S which quickly converge to loops, dialogues from our models smoothly move forward under the management of the dialogue acts.

Original: The dialogue acts let us know why such responses are generated and make the simulated dialogues closer to human dialogues with moderate context continuation and jumping out of the contexts at proper timing.
Revised: The dialogue acts let us know why such responses are generated and make the simulated dialogues closer to human dialogues with moderate context continuation and jumping out of the contexts at proper timing.

Original: Table 11 and Table 12 show some examples from the test of human-machine conversation.
Revised: Table 11 and Table 12 show some examples from the test of human-machine conversation.

Original: We denote a machine turn as “M” and a human turn as “H”.
Revised: We denote a machine turn as “M” and a human turn as “H”.

Original: After each example, we give the reason of termination in which “EOD-H” means the dialogue is terminated by the tester and “EOD-R” means the dialogue is terminated by the repetition check with the next generated turn attached.
Revised: After each example, we give the reason of termination in which “EOD-H” means the dialogue is terminated by the tester and “EOD-R” means the dialogue is terminated by the repetition check with the next generated turn attached.

Original: Compared to dialogues with the baseline, dialogues with our models can go deeper with much richer content, although a side-effect is that sometimes responses from CS.
Revised: Compared to dialogues with the baseline, dialogues with our models can go deeper with much richer content, although a side-effect is that sometimes responses from CS.

Original: * might be nonsense (e.g., the ﬁrst example of SL-DAGM).
Revised: * might be nonsense (e.g., the ﬁrst example of SL-DAGM).

Original: This sheds light on our future direction to further improve the generation network with knowledge.
Revised: This sheds light on our future direction to further improve the generation network with knowledge.

Original: In addition to the qualitative results, we also show quantitative results of human-machine conversation test in terms of different testers in Figure 2.
Revised: In addition to the qualitative results, we also show quantitative results of human-machine conversation test in terms of different testers in Figure 2.

Original: Although there exists variance among the testers, the overall trend is consistent with the numbers in Table 4(b).
Revised: Although there exists variance among the testers, the overall trend is consistent with the numbers in Table 4(b).

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 玩 的 吗 ？ B: 新 加 坡 有 什么 好 玩 的 ？ 我 想 去 看看
Revised: 玩 的 吗 ？ B: 新 加 坡 有 什么 好 玩 的 ？ 我 想 去 看看

Original: 。 A: I can learn English in Singa- pore and Malaysia.
Revised: 。 A: I can learn English in Singa- pore and Malaysia.

Original: B: Are there any interesting places in Singapore?
Revised: B: Are there any interesting places in Singapore?

Original: A: Are there any interesting places?
Revised: A: Are there any interesting places?

Original: B: Are there any interesting places in Singapore?
Revised: B: Are there any interesting places in Singapore?

Original: I want to have a tour in Singapore.
Revised: I want to have a tour in Singapore.

Original: A: 去 新 加 坡 马 来 西 亚 可 以 学 英 语 。 (CM.S) B: 你 英 文 名 是 什么 ？ (CS.Q)
Revised: A: 去 新 加 坡 马 来 西 亚 可 以 学 英 语 。 (CM.S) B: 你 英 文 名 是 什么 ？ (CS.Q)

Original: A: 我 应 该 可 以 写出 来 。 (CM.S) B: 那 你 姓 什么 。 (CM.S)
Revised: A: 我 应 该 可 以 写出 来 。 (CM.S) B: 那 你 姓 什么 。 (CM.S)

Original: A: 我 什么 时 候 写出 来 ？ (CS.Q)
Revised: A: 我 什么 时 候 写出 来 ？ (CS.Q)

Original: A: I can learn English in Singa- pore and Malaysia.
Revised: A: I can learn English in Singa- pore and Malaysia.

Original: (CM.S) B: What is your English name?
Revised: (CM.S) B: What is your English name?

Original: (CS.Q)
Revised: (CS.Q)

Original: A: I think I can write it out.
Revised: A: I think I can write it out.

Original: (CM.S) B: What is your family name?
Revised: (CM.S) B: What is your family name?

Original: (CM.S)
Revised: (CM.S)

Original: A:
Revised: A:

Original: When shall I write my English name?
Revised: When shall I write my English name?

Original: (CS.Q)
Revised: (CS.Q)

Original: A: 去 新 加 坡 马 来 西 亚 可 以 学 英 语 。 (CM.S) B: 你 也 想 去 新 加
Revised: A: 去 新 加 坡 马 来 西 亚 可 以 学 英 语 。 (CM.S) B: 你 也 想 去 新 加

Original: 坡 学 英 语 ？ (CM.Q)
Revised: 坡 学 英 语 ？ (CM.Q)

Original: A: 一 样 一 样 。 (CM.A) B: 你 是 什么专业 的 ？ (CS.Q) A: 我 们 学 校 有 什 么 专 业 ？ (CM.Q) A: I can learn English in Singa- pore and Malaysia.
Revised: A: 一 样 一 样 。 (CM.A) B: 你 是 什么专业 的 ？ (CS.Q) A: 我 们 学 校 有 什 么 专 业 ？ (CM.Q) A: I can learn English in Singa- pore and Malaysia.

Original: (CM.S) B: You also want to go there to learn English?
Revised: (CM.S) B: You also want to go there to learn English?

Original: (CM.Q)
Revised: (CM.Q)

Original: A: Yep.
Revised: A: Yep.

Original: (CM.A) B: What are you major in?
Revised: (CM.A) B: What are you major in?

Original: (CS.Q)
Revised: (CS.Q)

Original: A: What majors do we have?
Revised: A: What majors do we have?

Original: (CM.Q)
Revised: (CM.Q)

Original: [Table]
Revised: [Table]

Original: M: 我 去 试试 ！ (CM.S) H: 快 去 吧 ，
Revised: M: 我 去 试试 ！ (CM.S) H: 快 去 吧 ，

Original: [Equation]
Revised: [Equation]

Original: 酵 素 要 坚
Revised: 酵 素 要 坚

Original: 持 。 (CM.S) M:
Revised: 持 。 (CM.S) M:

Original: [Equation]
Revised: [Equation]

Original: 有
Revised: 有

Original: [Equation]
Revised: [Equation]

Original: 的
Revised: 的

Original: [Equation]
Revised: [Equation]

Original: 多
Revised: 多

Original: [Equation]
Revised: [Equation]

Original: (CS.Q) H: 有 ， 不 贵 。 (CM.A) EOD-R ( 请 问 药 店
Revised: (CS.Q) H: 有 ， 不 贵 。 (CM.A) EOD-R ( 请 问 药 店

Original: [Equation]
Revised: [Equation]

Original: 卖 的 吗 ？ 多 少 钱 ？ )
Revised: 卖 的 吗 ？ 多 少 钱 ？ )

Original: M: I am full with only soup.
Revised: M: I am full with only soup.

Original: (CM.S) H:
Revised: (CM.S) H:

Original: Then you can lose weight.
Revised: Then you can lose weight.

Original: (CS.S)
Revised: (CS.S)

Original: M:Do you eat albumen powder?
Revised: M:Do you eat albumen powder?

Original: (CS.Q) H:Nope.
Revised: (CS.Q) H:Nope.

Original: (CM.A)
Revised: (CM.A)

Original: M: What is enzyme powder?
Revised: M: What is enzyme powder?

Original: (CS.Q)
Revised: (CS.Q)

Original: H:Something can make you slim.
Revised: H:Something can make you slim.

Original: (CM.A) M: I want to try it!
Revised: (CM.A) M: I want to try it!

Original: (CM.S) H: Go ahead, and do not give up.
Revised: (CM.S) H: Go ahead, and do not give up.

Original: (CM.S) M: Can I get enzyme powder in pharmacy?
Revised: (CM.S) M: Can I get enzyme powder in pharmacy?

Original: How much is it?
Revised: How much is it?

Original: (CS.Q)
Revised: (CS.Q)

Original: H:
Revised: H:

Original: Yes you can.
Revised: Yes you can.

Original: It’s not expensive.
Revised: It’s not expensive.

Original: (CM.A) EOD-R (Can I get it in pharmacy?
Revised: (CM.A) EOD-R (Can I get it in pharmacy?

Original: How much?)
Revised: How much?)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: for
Revised: for

Original: [Table]
Revised: [Table]

Original: you
Revised: you

Original: [Table]
Revised: [Table]

Original: it?
Revised: it?

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) H: 嗯 我 听 你 的 。 (CM.A) M: 画 浓 妆 会 不 会 气 色 很 差 ？ (CS.Q) H: 不 会 的 。 (CM.A) M: 那 有 什么 副 作 用 ？ (CS.Q) H: 皮 肤 可 能 不 好 。 (CM.S) EOD-R
Revised: ) H: 嗯 我 听 你 的 。 (CM.A) M: 画 浓 妆 会 不 会 气 色 很 差 ？ (CS.Q) H: 不 会 的 。 (CM.A) M: 那 有 什么 副 作 用 ？ (CS.Q) H: 皮 肤 可 能 不 好 。 (CM.S) EOD-R

Original: ( 有 什么 副 作 用 ？ )
Revised: ( 有 什么 副 作 用 ？ )

Original: M: Light makeup looks good for you.
Revised: M: Light makeup looks good for you.

Original: (CM.S) H:
Revised: (CM.S) H:

Original: Well, you are right.
Revised: Well, you are right.

Original: (CM.A) M: Does heavy makeup make your complexion look bad?
Revised: (CM.A) M: Does heavy makeup make your complexion look bad?

Original: [Table] [Table]
Revised: [Table] [Table]

Original: )H:
Revised: )H:

Original: [Table] [Table]
Revised: [Table] [Table]

Original: CM.
Revised: CM.

Original: [Table]
Revised: [Table]

Original: ) M: Then is there any side-effect?
Revised: ) M: Then is there any side-effect?

Original: (CS.Q) H:
Revised: (CS.Q) H:

Original: My skin will become worse.
Revised: My skin will become worse.

Original: (CM.S) EOD-R (is there any side-effect?)
Revised: (CM.S) EOD-R (is there any side-effect?)