Original: GPT-C RITIC :A N O FFLINE R EINFORCEMENT L EARNING FOR E ND - TO -E ND T ASK -O RIENTED D IALOGUE S YSTEMS
Revised: GPT-C RITIC : O FFLINE R EINFORCEMENT L EARNING FOR E ND - TO -E ND T ASK -O RIENTED D IALOGUE S YSTEMS

Original: A BSTRACT
Revised: A BSTRACT

Original: Training a task-oriented dialogue agent can be naturally formulated as ofﬂine reinforcement learning (RL) problem, where the agent aims to learn a conversational strategy to achieve user goals, only from a dialogue corpus.
Revised: Training a task-oriented dialogue agent can be naturally formulated as ofﬂine reinforcement learning (RL) problem, where the agent aims to learn a conversational strategy to achieve user goals, only from a dialogue corpus.

Original: It is very challenging in terms of RL since the natural language action space is astronomical, while feasible (syntactically and semantically correct) actions are very sparse.
Revised: It is very challenging in terms of RL since the natural language action space is astronomical, while feasible (syntactically and semantically correct) actions are very sparse.

Original: Thus, standard
Revised: Thus, standard

Original: RL methods easily fail and generate responses diverging from human language, even when ﬁne-tuning a powerful pre-trained language model.
Revised: RL methods easily fail and generate responses diverging from human language, even when ﬁne-tuning a powerful pre-trained language model.

Original: In this paper, we introduce GPT-Critic, an ofﬂine RL method for task-oriented dialogue.
Revised: In this paper, we introduce GPT-Critic, an ofﬂine RL method for task-oriented dialogue.

Original: GPT-Critic is built upon GPT-2, ﬁne-tuning the language model through behavior cloning of the critic-guided self-generated sentences.
Revised: GPT-Critic is built upon GPT-2, ﬁne-tuning the language model through behavior cloning of the critic-guided self-generated sentences.

Original: GPT-Critic is essentially free from the issue of diverging from human language since it learns from the sentences sampled from the pre-trained language model.
Revised: GPT-Critic is essentially free from the issue of diverging from human language since it learns from the sentences sampled from the pre-trained language model.

Original: In the experiments, we demonstrate that our algorithm outperforms the state-of-the-art in the task-oriented dialogue benchmarks including MultiWOZ 2.0 and ConvLab.
Revised: In the experiments, we demonstrate that our algorithm outperforms the state-of-the-art in the task-oriented dialogue benchmarks including MultiWOZ 2.0 and ConvLab.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Building an end-to-end task-oriented dialogue agent is one of the promising applications of natural language processing (NLP) tasks, yet challenging due to large language action spaces and limited availability of human-annotated data.
Revised: Building an end-to-end task-oriented dialogue agent is one of the promising applications of natural language processing (NLP) tasks, yet challenging due to large language action spaces and limited availability of human-annotated data.

Original: Recently, large-scale pre-trained language models (LM) have achieved remarkable successes in various NLP tasks with prohibitively large vocabulary (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020; Raffel et al., 2019).
Revised: Recently, large-scale pre-trained language models (LM) have achieved remarkable successes in various NLP tasks with prohibitively large vocabulary (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020; Raffel et al., 2019).

Original: The current best performing end-to-end conversational agents for a task-oriented dialogue system utilize a pre-training on largescale corpus and ﬁne-tuning on downstream tasks (Ham et al., 2020; Yang et al., 2021; Lin et al., 2020; Peng et al., 2021).
Revised: The current best performing end-to-end conversational agents for a task-oriented dialogue system utilize a pre-training on largescale corpus and ﬁne-tuning on downstream tasks (Ham et al., 2020; Yang et al., 2021; Lin et al., 2020; Peng et al., 2021).

Original: This combination of pre-training and ﬁne-tuning signiﬁcantly improves overall performance in the task-oriented dialogues.
Revised: This combination of pre-training and ﬁne-tuning signiﬁcantly improves overall performance in the task-oriented dialogues.

Original: However, supervised ﬁne-tuning (i.e. imitation learning of the dialogue corpus) alone may not be sufﬁcient to learn an optimal dialogue strategy since the corpus often contains suboptimal dialogues collected from human participants of diverse expertise levels.
Revised: However, supervised ﬁne-tuning (i.e. imitation learning of the dialogue corpus) alone may not be sufﬁcient to learn an optimal dialogue strategy since the corpus often contains suboptimal dialogues collected from human participants of diverse expertise levels.

Original: Thus, in order to optimize the task performance of the conversational agent, goaloriented training (i.e. reinforcement learning) is an essential and promising direction to pursue.
Revised: Thus, in order to optimize the task performance of the conversational agent, goaloriented training (i.e. reinforcement learning) is an essential and promising direction to pursue.

Original: Training a task-oriented conversational agent from a dialogue corpus can be naturally formulated as ofﬂine reinforcement learning (RL) problem (Levine et al., 2020; Fujimoto et al., 2019; Jaques et al., 2020), which offers the prospect to optimize the policy solely from the ﬁxed dataset without online environment interaction.
Revised: Training a task-oriented conversational agent from a dialogue corpus can be naturally formulated as ofﬂine reinforcement learning (RL) problem (Levine et al., 2020; Fujimoto et al., 2019; Jaques et al., 2020), which offers the prospect to optimize the policy solely from the ﬁxed dataset without online environment interaction.

Original: Most of the existing ofﬂine RL methods are built on the off-policy ActorCritic framework, which performs iterative optimization of the policy (i.e. actor) and the actionvalue function (i.e. critic) (Fujimoto et al., 2019; Janner et al., 2019; Kumar et al., 2020).
Revised: Most of the existing ofﬂine RL methods are built on the off-policy ActorCritic framework, which performs iterative optimization of the policy (i.e. actor) and the actionvalue function (i.e. critic) (Fujimoto et al., 2019; Janner et al., 2019; Kumar et al., 2020).

Original: Yet, a naive application of these ofﬂine RL methods generally results in poor dialogue strategies which generate responses in no way similar to human language (Lewis et al., 2017;
Revised: Yet, a naive application of these ofﬂine RL methods generally results in poor dialogue strategies which generate responses in no way similar to human language (Lewis et al., 2017;

Original: Zhao et al., 2019; Jang et al., 2020).
Revised: Zhao et al., 2019; Jang et al., 2020).

Original: Weighted behavior cloning (BC) (Wang et al., 2020) is one of the representative ofﬂine RL algorithms, which is free from the issue of diverging from human language.
Revised: Weighted behavior cloning (BC) (Wang et al., 2020) is one of the representative ofﬂine RL algorithms, which is free from the issue of diverging from human language.

Original: Weighted BC amounts
Revised: Weighted BC amounts

Original: [Figure]
Revised: [Figure]

Original: Critic-guided Self-generation
Revised: Critic-guided Self-generation

Original: Policy
Revised: Policy

Original: Critic 𝑎𝑎𝑎 𝑁 𝑎 𝑡∗ ⋯ to ﬁltering out bad actions and imitating good actions.
Revised: Critic 𝑎𝑎𝑎 𝑁 𝑎 𝑡∗ ⋯ to ﬁltering out bad actions and imitating good actions.

Original: In the context of task-oriented dialogues, that would be equivalent to simply dropping the unsuccessful dialogues from the corpus.
Revised: In the context of task-oriented dialogues, that would be equivalent to simply dropping the unsuccessful dialogues from the corpus.

Original: However, dropping a whole dialogue from training would be wasteful, since they may still contain some taskspeciﬁc information that is useful to properly respond to user requests in the intermediate steps.
Revised: However, dropping a whole dialogue from training would be wasteful, since they may still contain some taskspeciﬁc information that is useful to properly respond to user requests in the intermediate steps.

Original: In this paper, we present an ofﬂine RL algorithm for task-oriented dialogue, which can be adopted for any generative pre-trained language model.
Revised: In this paper, we present an ofﬂine RL algorithm for task-oriented dialogue, which can be adopted for any generative pre-trained language model.

Original: Our algorithm, GPT-Critic, aims to revise unsuccessful dialogues into successful ones, rather than removing them as done in weighted BC.
Revised: Our algorithm, GPT-Critic, aims to revise unsuccessful dialogues into successful ones, rather than removing them as done in weighted BC.

Original: It starts with ﬁne-tuning the GPT-2 model and learning the action-value function (critic) using the dialogue corpus.
Revised: It starts with ﬁne-tuning the GPT-2 model and learning the action-value function (critic) using the dialogue corpus.

Original: Then, GPT-Critic generates a strategically promising action that is selected based on the value estimated by the critic.
Revised: Then, GPT-Critic generates a strategically promising action that is selected based on the value estimated by the critic.

Original: GPT-Critic updates the policy through behavior cloning of the critic-guided self-generated responses.
Revised: GPT-Critic updates the policy through behavior cloning of the critic-guided self-generated responses.

Original: This is in contrast to the previous methods that perform weighted behavior cloning on the dialogue corpus, where the action choice is restricted to the support in the dataset (Wang et al., 2020).
Revised: This is in contrast to the previous methods that perform weighted behavior cloning on the dialogue corpus, where the action choice is restricted to the support in the dataset (Wang et al., 2020).

Original: Compared to traditional actor-critic methods, since GPT-Critic does not rely on policy gradient and updates the policy within the support of generated actions from the GPT-2, it thus inherits GPT-2’s ability to generate human-like responses.
Revised: Compared to traditional actor-critic methods, since GPT-Critic does not rely on policy gradient and updates the policy within the support of generated actions from the GPT-2, it thus inherits GPT-2’s ability to generate human-like responses.

Original: In the experiments, we demonstrate that GPT-Critic outperforms the state-of-the-art end-to-end dialogue agent in the task-oriented dialogue benchmarks including MultiWOZ 2.0 (Budzianowski et al., 2018) and ConvLab (Zhu et al., 2020).
Revised: In the experiments, we demonstrate that GPT-Critic outperforms the state-of-the-art end-to-end dialogue agent in the task-oriented dialogue benchmarks including MultiWOZ 2.0 (Budzianowski et al., 2018) and ConvLab (Zhu et al., 2020).

Original:  B ACKGROUND 2. O FFLINE R EINFORCEMENT L EARNING FOR T ASK -O RIENTED D IALOGUES
Revised:  B ACKGROUND 2. O FFLINE R EINFORCEMENT L EARNING FOR T ASK -O RIENTED D IALOGUES

Original: We consider the task-oriented dialogue system that can be modeled as a partially observable Markov decision process (POMDP) (Williams & Young, 2007) deﬁned by tuple (cid:104) S, A, O, T, Z, R, γ (cid:105) where
Revised: We consider the task-oriented dialogue system that can be modeled as a partially observable Markov decision process (POMDP) (Williams & Young, 2007) deﬁned by tuple (cid:104) S, A, O, T, Z, R, γ (cid:105) where

Original: S is the set of environment states s = (cid:104) g, h (cid:105) (underlying state that consists of the user goal g and dialogue history h ), A is the set of actions a (a sequence of tokens which represents dialogue act and system response ), O is the set of observations o (user utterance), T ( s (cid:48) | s, a ) =
Revised: S is the set of environment states s = (cid:104) g, h (cid:105) (underlying state that consists of the user goal g and dialogue history h ), A is the set of actions a (a sequence of tokens which represents dialogue act and system response ), O is the set of observations o (user utterance), T ( s (cid:48) | s, a ) =

Original: Pr( s t +1 =
Revised: Pr( s t +1 =

Original: s (cid:48) | s t = s, a t = a ) is the transition function, Z ( o | s (cid:48) , a )
Revised: s (cid:48) | s t = s, a t = a ) is the transition function, Z ( o | s (cid:48) , a )

Original: =
Revised: =

Original: Pr( o t +

Revised: Pr( o t +

Original: =
Revised: =

Original: o | s t +

Revised: o | s t +

Original: = s (cid:48) , a t = a ) is the observation probability, R ( g, h, a ) is the reward function indicating the utility of executing action a in history h and the user goal g , and γ ∈ (0 , 1) is a discount factor.
Revised: = s (cid:48) , a t = a ) is the observation probability, R ( g, h, a ) is the reward function indicating the utility of executing action a in history h and the user goal g , and γ ∈ (0 , 1) is a discount factor.

Original: The history at time step t , h t = { o 0 , a 0 , . . .
Revised: The history at time step t , h t = { o 0 , a 0 , . . .

Original: o t − 1 , a t − 1 , o t } , is a sequence of all previous observations and actions.
Revised: o t − 1 , a t − 1 , o t } , is a sequence of all previous observations and actions.

Original: Since the underlying state s (e.g. user goal) is not directly observable, the agent makes decisions based on the entire observation-action history.
Revised: Since the underlying state s (e.g. user goal) is not directly observable, the agent makes decisions based on the entire observation-action history.

Original: The policy π ( a t | h t ) is mapping from history h t to a probability distribution over A .
Revised: The policy π ( a t | h t ) is mapping from history h t to a probability distribution over A .

Original: The goal is to ﬁnd an optimal policy π ∗ that maximizes the expected cumulative rewards, i.e. π ∗ = arg max π E π [ (cid:80) ∞ t =

Revised: The goal is to ﬁnd an optimal policy π ∗ that maximizes the expected cumulative rewards, i.e. π ∗ = arg max π E π [ (cid:80) ∞ t =

Original: γ t R ( g, h t , a t )] .
Revised: γ t R ( g, h t , a t )] .

Original: The action-value function of policy π is deﬁned as Q π ( h, a ) :
Revised: The action-value function of policy π is deﬁned as Q π ( h, a ) :

Original: =
Revised: =

Original: E π [ (cid:80) ∞ t =

Revised: E π [ (cid:80) ∞ t =

Original: γ t R ( g, h t , a t ) |
Revised: γ t R ( g, h t , a t ) |

Original: h 0 = h, a 0 = a ] , where Q π is a unique solution of the Bellman equation: Q π ( h, a ) =
Revised: h 0 = h, a 0 = a ] , where Q π is a unique solution of the Bellman equation: Q π ( h, a ) =

Original: E g
Revised: E g

Original: [ R ( g, h, a )] + γ E π [ Q π ( h (cid:48) , a (cid:48) )] .
Revised: [ R ( g, h, a )] + γ E π [ Q π ( h (cid:48) , a (cid:48) )] .

Original: User
Revised: User

Original: DB
Revised: DB

Original: BS
Revised: BS

Original: Act
Revised: Act

Original: Resp ⋯
Revised: Resp ⋯

Original: Value
Revised: Value

Original: Linear Linear
Revised: Linear Linear

Original: Linear
Revised: Linear

Original: GPTAction ( 𝒂 𝒕 )
Revised: GPTAction ( 𝒂 𝒕 )

Original: Dialogue history ( 𝒉 𝒕 )
Revised: Dialogue history ( 𝒉 𝒕 )

Original: Linear
Revised: Linear

Original: User
Revised: User

Original: DB
Revised: DB

Original: BS
Revised: BS

Original: Act
Revised: Act

Original: Resp
Revised: Resp

Original: Value
Revised: Value

Original: Linear Linear
Revised: Linear Linear

Original: Linear Linear
Revised: Linear Linear

Original: Transformer 𝑸(𝒉 𝒕 ,𝒂 𝒕 )
Revised: Transformer 𝑸(𝒉 𝒕 ,𝒂 𝒕 )

Original: User
Revised: User

Original: BS
Revised: BS

Original: Linear
Revised: Linear

Original: Critic
Revised: Critic

Original: Using ofﬂine RL for dialogue policy optimization, the agent optimizes the policy from the precollected dataset D
Revised: Using ofﬂine RL for dialogue policy optimization, the agent optimizes the policy from the precollected dataset D

Original: = {{ ( g j , h jt , a jt , r jt , h jt +1 )
Revised: = {{ ( g j , h jt , a jt , r jt , h jt +1 )

Original: Tt =0 } Nj =1 } without online environment interaction during the intermediate stages of training.
Revised: Tt =0 } Nj =1 } without online environment interaction during the intermediate stages of training.

Original: Prior ofﬂine RL algorithms (Fujimoto et al., 2019; Janner et al., 2019; Kumar et al., 2020) rely on off-policy actor-critic method, where the critic network is trained by minimizing the temporal differnce error with respect to the target policy π :
Revised: Prior ofﬂine RL algorithms (Fujimoto et al., 2019; Janner et al., 2019; Kumar et al., 2020) rely on off-policy actor-critic method, where the critic network is trained by minimizing the temporal differnce error with respect to the target policy π :

Original: [Equation]
Revised: [Equation]

Original: where ¯ φ is the parameters of the target network.
Revised: where ¯ φ is the parameters of the target network.

Original: As discussed in the prior work (Fujimoto et al., 2019; Kumar et al., 2020), optimizing this loss can be challenging in the ofﬂine RL setting due to the overestimation issue in the bootstrapping process by taking out-of-distribution (OOD) actions to evaluate the value of the next state.
Revised: As discussed in the prior work (Fujimoto et al., 2019; Kumar et al., 2020), optimizing this loss can be challenging in the ofﬂine RL setting due to the overestimation issue in the bootstrapping process by taking out-of-distribution (OOD) actions to evaluate the value of the next state.

Original: 2. E ND - TO -E ND T ASK -O RIENTED D IALOGUE S YSTEM
Revised: 2. E ND - TO -E ND T ASK -O RIENTED D IALOGUE S YSTEM

Original: We focus on the MultiWOZ 2.0 dataset (Budzianowski et al., 2018), which is a representative benchmark for task-oriented dialogue.
Revised: We focus on the MultiWOZ 2.0 dataset (Budzianowski et al., 2018), which is a representative benchmark for task-oriented dialogue.

Original: The MultiWOZ dataset is a fully-annotated corpus of human-human task-oriented conversations, which is collected via the Wizard-of-Oz setting (Kelley, 1984).
Revised: The MultiWOZ dataset is a fully-annotated corpus of human-human task-oriented conversations, which is collected via the Wizard-of-Oz setting (Kelley, 1984).

Original: The traditional approach to building a task-oriented dialogue system adopts a modular pipeline, which consists of the following four modules: 1) A natural language understanding (NLU) module (Kim et al., 2017; Zhu et al., 2020) identiﬁes the user’s intent and extracts the information of slots and their values, 2)
Revised: The traditional approach to building a task-oriented dialogue system adopts a modular pipeline, which consists of the following four modules: 1) A natural language understanding (NLU) module (Kim et al., 2017; Zhu et al., 2020) identiﬁes the user’s intent and extracts the information of slots and their values, 2)

Original: A Dialogue state tracking (DST) module (Williams et al., 2013) infers the belief state, 3)
Revised: A Dialogue state tracking (DST) module (Williams et al., 2013) infers the belief state, 3)

Original: A dialogue policy (POL) module decides the system action, 4)
Revised: A dialogue policy (POL) module decides the system action, 4)

Original: A natural language generation (NLG) module (Wen et al., 2015) generates the system response corresponding to the system action.
Revised: A natural language generation (NLG) module (Wen et al., 2015) generates the system response corresponding to the system action.

Original: Recently, end-to-end task-oriented dialogue methods leveraging the pre-trained language model have been proposed (Yang et al., 2021;
Revised: Recently, end-to-end task-oriented dialogue methods leveraging the pre-trained language model have been proposed (Yang et al., 2021;

Original: Ham et al., 2020; Lin et al., 2020; Peng et al., 2021; Hosseini-Asl et al., 2020), and signiﬁcantly improves overall performance in the task-oriented dialogues.
Revised: Ham et al., 2020; Lin et al., 2020; Peng et al., 2021; Hosseini-Asl et al., 2020), and signiﬁcantly improves overall performance in the task-oriented dialogues.

Original: In this paper, our algorithm is built upon UBAR (Yang et al., 2021), which is based on GPT-2 (Radford et al., 2019) and currently the state-of-the-art end-to-end dialogue agent for the MultiWOZ domain.
Revised: In this paper, our algorithm is built upon UBAR (Yang et al., 2021), which is based on GPT-2 (Radford et al., 2019) and currently the state-of-the-art end-to-end dialogue agent for the MultiWOZ domain.

Original:  O FFLINE R EINFORCEMENT L EARNING FOR E ND - TO -E ND T ASK -O RIENTED D IALOGUE S YSTEMS
Revised:  O FFLINE R EINFORCEMENT L EARNING FOR E ND - TO -E ND T ASK -O RIENTED D IALOGUE S YSTEMS

Original: The corpus collected from human-human conversations inevitably contains unsuccessful dialogues in terms of task completion.
Revised: The corpus collected from human-human conversations inevitably contains unsuccessful dialogues in terms of task completion.

Original: For example, approximately 20% dialogues of the MultiWOZ dataset fail to meet the user goal.
Revised: For example, approximately 20% dialogues of the MultiWOZ dataset fail to meet the user goal.

Original: Therefore, a naive behavior cloning of the whole dataset would limit the performance of the conversational agent since the dataset includes a lot of unsuccessful dialogues: an agent that imitates failure would be inevitably suboptimal.
Revised: Therefore, a naive behavior cloning of the whole dataset would limit the performance of the conversational agent since the dataset includes a lot of unsuccessful dialogues: an agent that imitates failure would be inevitably suboptimal.

Original: Yet, dropping the unsuccessful dialogues from the corpus as done in weighted BC is also undesirable, since they may contain some task-speciﬁc information that is useful to properly respond to user requests.
Revised: Yet, dropping the unsuccessful dialogues from the corpus as done in weighted BC is also undesirable, since they may contain some task-speciﬁc information that is useful to properly respond to user requests.

Original: We thus aim to revise unsuccessful dialogues into successful ones in order to prevent repeating the past failure while improving the task performance.
Revised: We thus aim to revise unsuccessful dialogues into successful ones in order to prevent repeating the past failure while improving the task performance.

Original: In this section, we present GPT-Critic, an ofﬂine RL algorithm for task-oriented dialogue.
Revised: In this section, we present GPT-Critic, an ofﬂine RL algorithm for task-oriented dialogue.

Original: Our GPTCritic is analogous to Actor-Critic method: GPT (Actor) decides which action to take while the
Revised: Our GPTCritic is analogous to Actor-Critic method: GPT (Actor) decides which action to take while the

Original: Critic informs how good the action was and provides a signal for policy improvement.
Revised: Critic informs how good the action was and provides a signal for policy improvement.

Original: Still, GPTCritic is distinct from the Actor-Critic methods in that it does not rely on the policy gradients, which are generally known to cause the issue of diverging from human language (Lewis et al., 2017; Zhao et al., 2019).
Revised: Still, GPTCritic is distinct from the Actor-Critic methods in that it does not rely on the policy gradients, which are generally known to cause the issue of diverging from human language (Lewis et al., 2017; Zhao et al., 2019).

Original: Instead, we sample a set of action candidates using GPT-2 and pick the best one using the critic, which constitutes a revised dialogue corpus.
Revised: Instead, we sample a set of action candidates using GPT-2 and pick the best one using the critic, which constitutes a revised dialogue corpus.

Original: Then, we perform supervised ﬁne-tuning of the GPT-2 on the revised dialogue corpus.
Revised: Then, we perform supervised ﬁne-tuning of the GPT-2 on the revised dialogue corpus.

Original: This learning procedure of our GPT-Critic does not hurt the agent’s capability to generate human-like sentences, given that the generated action candidates were all natural-looking sentences due to the power of large pre-trained LM.
Revised: This learning procedure of our GPT-Critic does not hurt the agent’s capability to generate human-like sentences, given that the generated action candidates were all natural-looking sentences due to the power of large pre-trained LM.

Original: Our algorithm is built upon the GPT-2 but it can be adopted for any generative pre-trained language model.
Revised: Our algorithm is built upon the GPT-2 but it can be adopted for any generative pre-trained language model.

Original: 3. P OLICY E VALUATION
Revised: 3. P OLICY E VALUATION

Original: Our GPT-Critic starts by training the action-value function (i.e. critic), which can evaluate the candidates for the response.
Revised: Our GPT-Critic starts by training the action-value function (i.e. critic), which can evaluate the candidates for the response.

Original: The architecture of the critic network basically follows GPT-2 with employing different last layers to compute the Q-value.
Revised: The architecture of the critic network basically follows GPT-2 with employing different last layers to compute the Q-value.

Original: The parameterization of the critic network Q φ is designed to share the parameters of the Transformer (Vaswani et al., 2017) layers of GPT-2, where the parameters of the Transformer layers are only updated during the policy improvement step.
Revised: The parameterization of the critic network Q φ is designed to share the parameters of the Transformer (Vaswani et al., 2017) layers of GPT-2, where the parameters of the Transformer layers are only updated during the policy improvement step.

Original: The critic network is trained by minimizing the temporal difference error with respect to the dataset D :
Revised: The critic network is trained by minimizing the temporal difference error with respect to the dataset D :

Original: [Equation]
Revised: [Equation]

Original: where ¯ φ is the parameters of the target network.
Revised: where ¯ φ is the parameters of the target network.

Original: Note that Eq.
Revised: Note that Eq.

Original: (2) is an on-policy evaluation on the dataset D , which can be optimized very stably since every a t +1 is always an in-distribution sample of D .
Revised: (2) is an on-policy evaluation on the dataset D , which can be optimized very stably since every a t +1 is always an in-distribution sample of D .

Original: This is in contrast to Eq.
Revised: This is in contrast to Eq.

Original: (1), which requires evaluation of out-of-distribution actions sampled from the target policy π .
Revised: (1), which requires evaluation of out-of-distribution actions sampled from the target policy π .

Original: The OOD action-value estimation can be very unreliable if the target policy deviates much from the dataset.
Revised: The OOD action-value estimation can be very unreliable if the target policy deviates much from the dataset.

Original: This kind of on-policy evaluation has been explored in the ofﬂine RL context for stable policy optimization (Brandfonbrener et al., 2021; Goo & Niekum, 2021), but they are limited to only one-step policy improvement: once the policy π is improved by the initial on-policy Q -function (i.e. π ( s ) =
Revised: This kind of on-policy evaluation has been explored in the ofﬂine RL context for stable policy optimization (Brandfonbrener et al., 2021; Goo & Niekum, 2021), but they are limited to only one-step policy improvement: once the policy π is improved by the initial on-policy Q -function (i.e. π ( s ) =

Original: arg max a Q ( s, a ) ), the new policy deviates from the dataset policy, thus it requires off-policy evaluation for further policy iteration.
Revised: arg max a Q ( s, a ) ), the new policy deviates from the dataset policy, thus it requires off-policy evaluation for further policy iteration.

Original: In contrast, our GPT-Critic performs policy improvement by generating an improved dataset based on the learned critic, where we can perform on-policy evaluation on the new dataset again.
Revised: In contrast, our GPT-Critic performs policy improvement by generating an improved dataset based on the learned critic, where we can perform on-policy evaluation on the new dataset again.

Original: As a consequence, GPT-Critic can enjoy the stable multi-step policy iteration through alternation between on-policy evaluation and policy improvement via revising dataset, which will be discussed in the following section.
Revised: As a consequence, GPT-Critic can enjoy the stable multi-step policy iteration through alternation between on-policy evaluation and policy improvement via revising dataset, which will be discussed in the following section.

Original: 3. P OLICY I MPROVEMENT VIA D ATASET R EVISION
Revised: 3. P OLICY I MPROVEMENT VIA D ATASET R EVISION

Original: In the task-oriented dialogues, the reward is given by the external program provided as a part of the dataset, which checks whether the user goal is satisﬁed by examining the dialogue history.
Revised: In the task-oriented dialogues, the reward is given by the external program provided as a part of the dataset, which checks whether the user goal is satisﬁed by examining the dialogue history.

Original: To generate the improved dataset, we adopt the common automatic evaluation of dialogue systems, where the agent generates dialogue act and system response on every system turn with ﬁxed user utterances.
Revised: To generate the improved dataset, we adopt the common automatic evaluation of dialogue systems, where the agent generates dialogue act and system response on every system turn with ﬁxed user utterances.

Original: More formally, the GPT-Critic generates a new dataset containing revised responses by:
Revised: More formally, the GPT-Critic generates a new dataset containing revised responses by:

Original: [Equation]
Revised: [Equation]

Original: where { a k } N is a set of N response candidates generated from the policy π (i.e. ﬁne-tuned GPT-2), and D i is the dataset at i -th iteration.
Revised: where { a k } N is a set of N response candidates generated from the policy π (i.e. ﬁne-tuned GPT-2), and D i is the dataset at i -th iteration.

Original: In the task-oriented dialogues, a reward function R ( g, h, a ) is provided that can compute a reward given a user goal, dialogue history, and system action.
Revised: In the task-oriented dialogues, a reward function R ( g, h, a ) is provided that can compute a reward given a user goal, dialogue history, and system action.

Original: The revised reward r ∗ t =
Revised: The revised reward r ∗ t =

Original: R ( g, h t , a ∗ t ) is computed by given user goal, dialogue history, and revised system action a ∗ t .
Revised: R ( g, h t , a ∗ t ) is computed by given user goal, dialogue history, and revised system action a ∗ t .

Original: The dialogue history is a sequence of all previous observations and actions, thus the revised history h ∗ t +

Revised: The dialogue history is a sequence of all previous observations and actions, thus the revised history h ∗ t +

Original: =
Revised: =

Original: { o 0 , a 0 , . . .
Revised: { o 0 , a 0 , . . .

Original: , o t , a ∗ t , o t +1 } is deﬁned by replacing the original action a t of h t +1 with the revised action a ∗ t .
Revised: , o t , a ∗ t , o t +1 } is deﬁned by replacing the original action a t of h t +1 with the revised action a ∗ t .

Original: The examples of revised responses can be found in Appendix B.
Revised: The examples of revised responses can be found in Appendix B.

Original: In order to address the prohibitively large language action spaces, we explicitly consider the set of response candidates that are generated from the ﬁne-tuned GPT-2.
Revised: In order to address the prohibitively large language action spaces, we explicitly consider the set of response candidates that are generated from the ﬁne-tuned GPT-2.

Original: The GPT-Critic selects the
Revised: The GPT-Critic selects the

Original: Input: Training dataset D 

Revised: Input: Training dataset D

Original: = {{ ( g j , h jt , a jt , r jt , h jt +1 )
Revised: = {{ ( g j , h jt , a jt , r jt , h jt +1 )

Original: Tt =0 } Nj =1 } , policy network (GPT)
Revised: Tt =0 } Nj =1 } , policy network (GPT)

Original: π θ , critic network Q φ
Revised: π θ , critic network Q φ

Original: Fine-tune the initial policy represented by GPT-2 model (e.g. UBAR) for each iteration i do
Revised: Fine-tune the initial policy represented by GPT-2 model (e.g. UBAR) for each iteration i do

Original: Update critic by minimizing the temporal difference error until convergence: arg min φ
Revised: Update critic by minimizing the temporal difference error until convergence: arg min φ

Original: E (
Revised: E (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ,a t ,
Revised: ,a t ,

Original: [Table]
Revised: [Table]

Original: t ,
Revised: t ,

Original: [Table]
Revised: [Table]

Original: ,a t +1 )
Revised: ,a t +1 )

Original: ∼D i (cid:20)(cid:16)
Revised: ∼D i (cid:20)(cid:16)

Original: [Table]
Revised: [Table]

Original: + γQ
Revised: + γQ

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) (cid:17)
Revised: ) (cid:17)

Original: [Table]
Revised: [Table]

Original: (cid:21)
Revised: (cid:21)

Original: Update dataset by critic-guided self-generation:
Revised: Update dataset by critic-guided self-generation:

Original: D i +
Revised: D i +

Original: [Table]
Revised: [Table]

Original: = { ( g, h t , a ∗ t ,
Revised: = { ( g, h t , a ∗ t ,

Original: [Table]
Revised: [Table]

Original: ∗ t , h ∗ t +1 ) | a ∗ t = arg max a ∈{ a i } N { a i } N ∼ π iθ ( h t )
Revised: ∗ t , h ∗ t +1 ) | a ∗ t = arg max a ∈{ a i } N { a i } N ∼ π iθ ( h t )

Original: Q φ ( h t , a ) where h t ∈ D i }
Revised: Q φ ( h t , a ) where h t ∈ D i }

Original: Update policy by behavior cloning of critic-guided self-generated dataset:
Revised: Update policy by behavior cloning of critic-guided self-generated dataset:

Original: Early stop according on validation set) arg
Revised: Early stop according on validation set) arg

Original: [Table]
Revised: [Table]

Original: E (
Revised: E (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: t )
Revised: t )

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ( a
Revised: ( a

Original: [Table]
Revised: [Table]

Original: t )] end for most promising response by calculating the Q-values over the response candidates.
Revised: t )] end for most promising response by calculating the Q-values over the response candidates.

Original: GPT-Critic then performs behavior cloning of critic-guided self-generated dialogues:
Revised: GPT-Critic then performs behavior cloning of critic-guided self-generated dialogues:

Original: [Equation]
Revised: [Equation]

Original: (4) where θ is the parameters of GPT-2.
Revised: (4) where θ is the parameters of GPT-2.

Original: The policy improvement of GPT-Critic is performed by behavior cloning of generated dialogues from the GPT-2, thus GPT-Critic inherits GPT-2’s ability to generate human-like responses.
Revised: The policy improvement of GPT-Critic is performed by behavior cloning of generated dialogues from the GPT-2, thus GPT-Critic inherits GPT-2’s ability to generate human-like responses.

Original: We can theoretically show that the updated policy by the above policy improvement step has a higher value than the old policy.
Revised: We can theoretically show that the updated policy by the above policy improvement step has a higher value than the old policy.

Original: Furthermore, we can also theoretically show that updated policy by the higher number of candidate actions has a higher value than the policy updated by the lower number of candidate actions.
Revised: Furthermore, we can also theoretically show that updated policy by the higher number of candidate actions has a higher value than the policy updated by the lower number of candidate actions.

Original: We formalize this result in Theorem 1.
Revised: We formalize this result in Theorem 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Policy Improvement)
Revised: (Policy Improvement)

Original: Given a policy π and the number of sampling actions N ≥ 1 , If we update the new policy π new N by
Revised: Given a policy π and the number of sampling actions N ≥ 1 , If we update the new policy π new N by

Original: [Equation]
Revised: [Equation]

Original: then Q π new N ( s, a ) ≥ Q π ( s, a ) ∀ s, a always holds.
Revised: then Q π new N ( s, a ) ≥ Q π ( s, a ) ∀ s, a always holds.

Original: Furthermore, for any N, M such that N ≥ M ≥ 1 , Q π new N ( s, a ) ≥ Q π new M ( s, a ) ∀ s, a always holds.
Revised: Furthermore, for any N, M such that N ≥ M ≥ 1 , Q π new N ( s, a ) ≥ Q π new M ( s, a ) ∀ s, a always holds.

Original: (Proof in Appendix A.)
Revised: (Proof in Appendix A.)

Original: We describe our algorithm, GPT-Critic, in Algorithm 1, that alternates between policy evaluation and policy improvement via revising the dataset until the policy performance converges.
Revised: We describe our algorithm, GPT-Critic, in Algorithm 1, that alternates between policy evaluation and policy improvement via revising the dataset until the policy performance converges.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: End-to-End Task-Oriented Dialogue Systems.
Revised: End-to-End Task-Oriented Dialogue Systems.

Original: The traditional approach to building a task-oriented dialogue system adopts a modular pipeline, which consists of natural language understanding, dialogue state tracking, dialogue policy, and natural language generation.
Revised: The traditional approach to building a task-oriented dialogue system adopts a modular pipeline, which consists of natural language understanding, dialogue state tracking, dialogue policy, and natural language generation.

Original: Recently, pre-trained LMbased end-to-end task-oriented dialogue agents that all sub-tasks recast as a single sequence prediction problem have been proposed (Ham et al., 2020; Hosseini-Asl et al., 2020), and signiﬁcantly improved overall performance in the task-oriented dialogues.
Revised: Recently, pre-trained LMbased end-to-end task-oriented dialogue agents that all sub-tasks recast as a single sequence prediction problem have been proposed (Ham et al., 2020; Hosseini-Asl et al., 2020), and signiﬁcantly improved overall performance in the task-oriented dialogues.

Original: There are a number of variants of
Revised: There are a number of variants of

Original: GPT-2-based end-to-end task-oriented dialogue agents.
Revised: GPT-2-based end-to-end task-oriented dialogue agents.

Original: Yang et al.
Revised: Yang et al.

Original: (2021) leverage the entire dialogue session of every dialogue turn.
Revised: (2021) leverage the entire dialogue session of every dialogue turn.

Original: Peng et al.
Revised: Peng et al.

Original: (2021) adopt transfer learning and machine teaching for training a GPT-2-based dialogue agent.
Revised: (2021) adopt transfer learning and machine teaching for training a GPT-2-based dialogue agent.

Original: Lin et al.
Revised: Lin et al.

Original: (2020) present efﬁcient dialogue state tracking with a minimal generation length, then leverage pre-trained language models for task-oriented dialogues.
Revised: (2020) present efﬁcient dialogue state tracking with a minimal generation length, then leverage pre-trained language models for task-oriented dialogues.

Original: Reinforcement Learning for Task-Oriented Dialogue Systems.
Revised: Reinforcement Learning for Task-Oriented Dialogue Systems.

Original: Applying the standard RL methods straightforwardly to optimize a task-oriented dialogue agent causes the issue of diverging from human language.
Revised: Applying the standard RL methods straightforwardly to optimize a task-oriented dialogue agent causes the issue of diverging from human language.

Original: To address this problem, interleaving reinforcement learning with supervised learning has been proposed but it is still not free from the issue of diverging from human language (Lewis et al., 2017).
Revised: To address this problem, interleaving reinforcement learning with supervised learning has been proposed but it is still not free from the issue of diverging from human language (Lewis et al., 2017).

Original: Recently, the latent representation models for language actions have been introduced to address the aforementioned problem (Zhao et al., 2019; Yarats & Lewis, 2018).
Revised: Recently, the latent representation models for language actions have been introduced to address the aforementioned problem (Zhao et al., 2019; Yarats & Lewis, 2018).

Original: They disentangle the semantics of the utterance and the natural language generation, and then perform goal-based training in the space of the latent variables instead of directly optimizing utterances.
Revised: They disentangle the semantics of the utterance and the natural language generation, and then perform goal-based training in the space of the latent variables instead of directly optimizing utterances.

Original: However, they cannot be directly applied to large-scale pre-trained language models that are not designed in a way that works inherently with discrete latent variables.
Revised: However, they cannot be directly applied to large-scale pre-trained language models that are not designed in a way that works inherently with discrete latent variables.

Original: Jaques et al.
Revised: Jaques et al.

Original: (2020) use KLcontrol to restrict the policy to stay close to its prior policy, but it still suffers from divergence from human language even with carefully chosen hyper-parameters.
Revised: (2020) use KLcontrol to restrict the policy to stay close to its prior policy, but it still suffers from divergence from human language even with carefully chosen hyper-parameters.

Original: Furthermore, Jang et al.
Revised: Furthermore, Jang et al.

Original: (2020) adopt
Revised: (2020) adopt

Original: Bayes-adaptive Monte-Carlo planning to negotiation dialogue then use it as a policy improvement operator.
Revised: Bayes-adaptive Monte-Carlo planning to negotiation dialogue then use it as a policy improvement operator.

Original: This approach can prevent the issue of diverging from human language through the policy improvement based on behavior cloning of self-generated dialogues.
Revised: This approach can prevent the issue of diverging from human language through the policy improvement based on behavior cloning of self-generated dialogues.

Original: However, they assume a user model that is difﬁcult enough to be considered another problem.
Revised: However, they assume a user model that is difﬁcult enough to be considered another problem.

Original: Ofﬂine Reinforcement Learning.
Revised: Ofﬂine Reinforcement Learning.

Original: There have been extensive studies on ofﬂine RL (Fujimoto et al., 2019; Levine et al., 2020; Kumar et al., 2020; Wang et al., 2020).
Revised: There have been extensive studies on ofﬂine RL (Fujimoto et al., 2019; Levine et al., 2020; Kumar et al., 2020; Wang et al., 2020).

Original: Most of prior works are built on the off-policy actor-critic framework, and they focus on the overestimation issue by taking the
Revised: Most of prior works are built on the off-policy actor-critic framework, and they focus on the overestimation issue by taking the

Original: OOD actions (Kumar et al., 2019; Lee et al., 2020; Fujimoto et al., 2019;
Revised: OOD actions (Kumar et al., 2019; Lee et al., 2020; Fujimoto et al., 2019;

Original: Jaques et al., 2020; Kumar et al., 2020).
Revised: Jaques et al., 2020; Kumar et al., 2020).

Original: However, a naive application of these ofﬂine RL methods suffer from the issue of diverging from human language in the task-oriented dialogues (Lewis et al., 2017; Zhao et al., 2019;
Revised: However, a naive application of these ofﬂine RL methods suffer from the issue of diverging from human language in the task-oriented dialogues (Lewis et al., 2017; Zhao et al., 2019;

Original: Jang et al., 2020).
Revised: Jang et al., 2020).

Original: On the other hand, there are a number of recent works on weighted behavior cloning, where a policy is trained by a variant of supervised learning loss (Wang et al., 2020; Peng et al., 2019; Siegel et al., 2020).
Revised: On the other hand, there are a number of recent works on weighted behavior cloning, where a policy is trained by a variant of supervised learning loss (Wang et al., 2020; Peng et al., 2019; Siegel et al., 2020).

Original: The weighted behavior cloning approaches ﬁlter out bad actions, then perform behavior cloning on high-quality data.
Revised: The weighted behavior cloning approaches ﬁlter out bad actions, then perform behavior cloning on high-quality data.

Original: However, in the task-oriented dialogues, simply dropping the unsuccessful dialogues from the corpus is undesirable, since they may contain some task-speciﬁc information that is useful to properly respond to user requests.
Revised: However, in the task-oriented dialogues, simply dropping the unsuccessful dialogues from the corpus is undesirable, since they may contain some task-speciﬁc information that is useful to properly respond to user requests.

Original: Our GPT-Critic aims to revise unsuccessful dialogues into successful ones, which is in contrast to the weighted behavior cloning on the ﬁxed training dataset, where the action choice is restricted to the support in the dataset (Wang et al., 2020; Peng et al., 2019; Siegel et al., 2020).
Revised: Our GPT-Critic aims to revise unsuccessful dialogues into successful ones, which is in contrast to the weighted behavior cloning on the ﬁxed training dataset, where the action choice is restricted to the support in the dataset (Wang et al., 2020; Peng et al., 2019; Siegel et al., 2020).

Original: More recently, Chen et al.
Revised: More recently, Chen et al.

Original: (2021) introduce
Revised: (2021) introduce

Original: Decision Transformer, a Transformer-based architecture that casts the problem of RL as conditional sequence modeling.
Revised: Decision Transformer, a Transformer-based architecture that casts the problem of RL as conditional sequence modeling.

Original: These ofﬂine RL methods based on behavior cloning are directly applied to the task-oriented dialogues without aforementioned issue, but their results are similar to that of behavior cloning in the task-oriented dialogues.
Revised: These ofﬂine RL methods based on behavior cloning are directly applied to the task-oriented dialogues without aforementioned issue, but their results are similar to that of behavior cloning in the task-oriented dialogues.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we show the experimental results of GPT-critic on both automatic evaluation and human evaluation.
Revised: In this section, we show the experimental results of GPT-critic on both automatic evaluation and human evaluation.

Original: First, we evaluate the performances of GPT-Critic on the MultiWOZ 2.
Revised: First, we evaluate the performances of GPT-Critic on the MultiWOZ 2.

Original: (Budzianowski et al., 2018) as dataset-based automatic evaluation, compared with baseline methods including ofﬂine RL algorithms.
Revised: (Budzianowski et al., 2018) as dataset-based automatic evaluation, compared with baseline methods including ofﬂine RL algorithms.

Original: Second, for more realistic evaluation, we conduct a simulator-based evaluation on the ConvLab framework (Zhu et al., 2020).
Revised: Second, for more realistic evaluation, we conduct a simulator-based evaluation on the ConvLab framework (Zhu et al., 2020).

Original: Third, we also conduct the human evaluation to evaluate the quality of generated responses.
Revised: Third, we also conduct the human evaluation to evaluate the quality of generated responses.

Original: Finally, we give a qualitative analysis of our method using generated dialogue examples on the training dataset of MultiWOZ 2.0, which shows how GPT-Critic improves the performance through the behavior cloning of self-generated dialogues.
Revised: Finally, we give a qualitative analysis of our method using generated dialogue examples on the training dataset of MultiWOZ 2.0, which shows how GPT-Critic improves the performance through the behavior cloning of self-generated dialogues.

Original: The qualitative analysis with generated dialogue examples can be found in Appendix B.
Revised: The qualitative analysis with generated dialogue examples can be found in Appendix B.

Original: 5. E XPERIMENTAL S ETUP
Revised: 5. E XPERIMENTAL S ETUP

Original: We implement GPT-Critic based on the HuggingFace
Revised: We implement GPT-Critic based on the HuggingFace

Original: Transformers library (Wolf et al., 2019) and codebase of UBAR (Yang et al., 2021), which is a GPT-2-based current state-of-the-art end-to-end task-oriented dialogue agent for the MultiWOZ 2.0 dataset.
Revised: Transformers library (Wolf et al., 2019) and codebase of UBAR (Yang et al., 2021), which is a GPT-2-based current state-of-the-art end-to-end task-oriented dialogue agent for the MultiWOZ 2.0 dataset.

Original: For the generative pre-trained language model, we use DistilGPT2 (Sanh et al., 2019), a distilled version of GPT-2.
Revised: For the generative pre-trained language model, we use DistilGPT2 (Sanh et al., 2019), a distilled version of GPT-2.

Original: Figure 2 shows the architecture of our policy and critic network based on GPT-2.
Revised: Figure 2 shows the architecture of our policy and critic network based on GPT-2.

Original: We design the parameterization of the critic network to share the parameters of the Transformer layers of GPT-2, where the parameters of the Transformer layers are only updated during the policy improvement step.
Revised: We design the parameterization of the critic network to share the parameters of the Transformer layers of GPT-2, where the parameters of the Transformer layers are only updated during the policy improvement step.

Original: For the hyperparame-
Revised: For the hyperparame-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ters of ﬁne-tuning the GPT-2 model, we follow the setting in the public code of UBAR (Yang et al., 2021).
Revised: ters of ﬁne-tuning the GPT-2 model, we follow the setting in the public code of UBAR (Yang et al., 2021).

Original: We use N = 5 for the number of candidate actions { a k } N , and the set of candidate actions are constructed by vanilla softmax sampling from the policy, rather than beam search, to collect diverse actions.
Revised: We use N = 5 for the number of candidate actions { a k } N , and the set of candidate actions are constructed by vanilla softmax sampling from the policy, rather than beam search, to collect diverse actions.

Original: For each behavior cloning iteration, all models are ﬁne-tuned with a training dataset from the pre-trained GPT-2 and early stop according to the loss on the validation set.
Revised: For each behavior cloning iteration, all models are ﬁne-tuned with a training dataset from the pre-trained GPT-2 and early stop according to the loss on the validation set.

Original: 5. E VALUATION ON THE M ULTI W O Z D ATASET
Revised: 5. E VALUATION ON THE M ULTI W O Z D ATASET

Original: We evaluate our algorithm on the MultiWOZ 2.0 dataset, which is one of the representative taskoriented dialogue benchmarks.
Revised: We evaluate our algorithm on the MultiWOZ 2.0 dataset, which is one of the representative taskoriented dialogue benchmarks.

Original: The MultiWOZ 2.0 is a large-scale multi-domain Wizard-of-Oz dataset, where a tourist (i.e. user) converses with a clerk (i.e. system) at the information center in a touristic city.
Revised: The MultiWOZ 2.0 is a large-scale multi-domain Wizard-of-Oz dataset, where a tourist (i.e. user) converses with a clerk (i.e. system) at the information center in a touristic city.

Original: It consists of 8438/1000/1000 dialogues for training/validation/testing.
Revised: It consists of 8438/1000/1000 dialogues for training/validation/testing.

Original: For end-to-end evaluation on the MultiWOZ 2.0 dataset, we use the following automatic evaluation metrics: 1) Inform : evaluates whether the system provides an appropriate entity, 2) Success : evaluates whether the system answers all the requested information, 3) BLEU : measures the ﬂuency of the generated response (Papineni et al., 2002).
Revised: For end-to-end evaluation on the MultiWOZ 2.0 dataset, we use the following automatic evaluation metrics: 1) Inform : evaluates whether the system provides an appropriate entity, 2) Success : evaluates whether the system answers all the requested information, 3) BLEU : measures the ﬂuency of the generated response (Papineni et al., 2002).

Original: We also report the Combined Score as an overall quality measure (Combined = ( Inform + Success ) × 0 .
Revised: We also report the Combined Score as an overall quality measure (Combined = ( Inform + Success ) × 0 .

Original: 5 + BLEU).
Revised: 5 + BLEU).

Original: We compare the performance of GPT-Critic with the following algorithms: 1) SFN+RL (Mehri et al., 2019), a seq2seq network that incorporates several pre-trained dialogue modules into a neural dialogue model, 2) DAMD
Revised: We compare the performance of GPT-Critic with the following algorithms: 1) SFN+RL (Mehri et al., 2019), a seq2seq network that incorporates several pre-trained dialogue modules into a neural dialogue model, 2) DAMD

Original: (Zhang et al., 2020), the domain-aware multi-decoder network with multiaction data augmentation method, 3)
Revised: (Zhang et al., 2020), the domain-aware multi-decoder network with multiaction data augmentation method, 3)

Original: SimpleTOD (Hosseini-Asl et al., 2020), a GPT-2-based endto-end dialogue agent that all sub-tasks recast as a single sequence prediction problem, 4)
Revised: SimpleTOD (Hosseini-Asl et al., 2020), a GPT-2-based endto-end dialogue agent that all sub-tasks recast as a single sequence prediction problem, 4)

Original: SOLOIST
Revised: SOLOIST

Original: (Peng et al., 2021), a GPT-2-based end-to-end dialogue agent with transfer learning and machine teaching, 5) MinTL (Lin et al., 2020), an efﬁcient dialogue state tracking method with a minimal generation length by predicting the difference between old and new states, 6) UBAR (Yang et al., 2021), a GPT-2-based end-to-end dialogue agent that leverages the entire dialogue session of every dialogue turn.
Revised: (Peng et al., 2021), a GPT-2-based end-to-end dialogue agent with transfer learning and machine teaching, 5) MinTL (Lin et al., 2020), an efﬁcient dialogue state tracking method with a minimal generation length by predicting the difference between old and new states, 6) UBAR (Yang et al., 2021), a GPT-2-based end-to-end dialogue agent that leverages the entire dialogue session of every dialogue turn.

Original: We implement our algorithm into the codebase of UBAR (Yang et al., 2021), and the result of UBAR is reproduced by adapting its code to the same evaluation settings as other papers 1 .
Revised: We implement our algorithm into the codebase of UBAR (Yang et al., 2021), and the result of UBAR is reproduced by adapting its code to the same evaluation settings as other papers 1 .

Original: Moreover, we also compare the data augmentation method, D ATA
Revised: Moreover, we also compare the data augmentation method, D ATA

Original: A UGMENTATION , which is naively ﬁne-tuning the GPT-2 model with additionally generated data by vanilla softmax sampling from the trained policy.
Revised: A UGMENTATION , which is naively ﬁne-tuning the GPT-2 model with additionally generated data by vanilla softmax sampling from the trained policy.

Original: In addition, we also compare with recent ofﬂine RL algorithms that are free from the issue of diverging from human language: 1) CRR (Wang et al., 2020), a value-ﬁltered regression method that performs weighted behavior cloning of ofﬂine dataset, 2) Decision Transformer (Chen et al., 2021), a Transformer-based architecture that casts the problem of RL as conditional sequence modeling.
Revised: In addition, we also compare with recent ofﬂine RL algorithms that are free from the issue of diverging from human language: 1) CRR (Wang et al., 2020), a value-ﬁltered regression method that performs weighted behavior cloning of ofﬂine dataset, 2) Decision Transformer (Chen et al., 2021), a Transformer-based architecture that casts the problem of RL as conditional sequence modeling.

Original: For a fair comparison, we use the same pre-trained GPT-2 model as a policy network to train the
Revised: For a fair comparison, we use the same pre-trained GPT-2 model as a policy network to train the

Original: CRR and the Decision Transformer.
Revised: CRR and the Decision Transformer.

Original: Moreover, to show that the policy-gradient-based standard RL algorithms suffer from diverging from human language, we also provide examples of responses generated by policy-gradient-based standard RL algorithm in Appendix D.
Revised: Moreover, to show that the policy-gradient-based standard RL algorithms suffer from diverging from human language, we also provide examples of responses generated by policy-gradient-based standard RL algorithm in Appendix D.

Original: [Table]
Revised: [Table]

Original: Table 1 and Table 2 show the results of policy iteration.
Revised: Table 1 and Table 2 show the results of policy iteration.

Original: Table 1 shows the performance of training dataset and critic-guided self-generated dialogues being used for each policy improvement step.
Revised: Table 1 shows the performance of training dataset and critic-guided self-generated dialogues being used for each policy improvement step.

Original: Table 2 reports the intermediate performance of behavior cloning of training dataset and criticguided self-generated dialogues in each of the policy iteration.
Revised: Table 2 reports the intermediate performance of behavior cloning of training dataset and criticguided self-generated dialogues in each of the policy iteration.

Original: As shown in Table 1 and Table 2, the performance of critic-guided self-generated dialogues is improved gradually; the performance of GPT-Critic is also consistently improved through the behavior cloning of improved dataset.
Revised: As shown in Table 1 and Table 2, the performance of critic-guided self-generated dialogues is improved gradually; the performance of GPT-Critic is also consistently improved through the behavior cloning of improved dataset.

Original: Table 3 summarizes the overall performance of GPT-Critic and baseline algorithms in end-to-end response generation setting, where the generated dialogue state and generated dialogue act are used for the DB search and response generation.
Revised: Table 3 summarizes the overall performance of GPT-Critic and baseline algorithms in end-to-end response generation setting, where the generated dialogue state and generated dialogue act are used for the DB search and response generation.

Original: The results show that GPT-Critic achieved the best performance in terms of inform rate, success rate, and combined score.
Revised: The results show that GPT-Critic achieved the best performance in terms of inform rate, success rate, and combined score.

Original: Moreover, the performance of GPT-Critic on the BLEU score matches those of other pre-trained LM-based methods, since
Revised: Moreover, the performance of GPT-Critic on the BLEU score matches those of other pre-trained LM-based methods, since

Original: GPT-Critic inherits GPT-2’s
Revised: GPT-Critic inherits GPT-2’s

Original: ability to generate human-like responses through the behavior cloning of responses generated by GPT-2.
Revised: ability to generate human-like responses through the behavior cloning of responses generated by GPT-2.

Original: The results show that GPT-Critic improves the task performance of the agent without the issue of diverging from human language.
Revised: The results show that GPT-Critic improves the task performance of the agent without the issue of diverging from human language.

Original: In addition, as can be shown in
Revised: In addition, as can be shown in

Original: Table 3, the naive data augmentation is not effective since it will not change the GPT’s sampling distribution in principle.
Revised: Table 3, the naive data augmentation is not effective since it will not change the GPT’s sampling distribution in principle.

Original: For the results of ofﬂine RL baselines, CRR and Decision Transformer show the results that do not diverge from human-language, since their policy is also trained by behavior cloning.
Revised: For the results of ofﬂine RL baselines, CRR and Decision Transformer show the results that do not diverge from human-language, since their policy is also trained by behavior cloning.

Original: However, both algorithms show limited performance because they perform behavior cloning on a ﬁxed dataset.
Revised: However, both algorithms show limited performance because they perform behavior cloning on a ﬁxed dataset.

Original: CRR has achieved remarkable success in continuous control tasks by performing weighted behavior cloning of training dataset ﬁltered by critic, but it does not effectively perform in the task-oriented dialogues because of data scarcity.
Revised: CRR has achieved remarkable success in continuous control tasks by performing weighted behavior cloning of training dataset ﬁltered by critic, but it does not effectively perform in the task-oriented dialogues because of data scarcity.

Original: Furthermore, to evaluate the Decision Transformer, we adopt a delayed return where the agent receives the cumulative reward at the end of dialogue, since the agent cannot observe user goal.
Revised: Furthermore, to evaluate the Decision Transformer, we adopt a delayed return where the agent receives the cumulative reward at the end of dialogue, since the agent cannot observe user goal.

Original: Therefore, without observing the user goal at test time, Decision
Revised: Therefore, without observing the user goal at test time, Decision

Original: Transformer reduces to the behavior cloning of successful dialogues.
Revised: Transformer reduces to the behavior cloning of successful dialogues.

Original: 5. E VALUATION ON C ONV L AB E VALUATOR
Revised: 5. E VALUATION ON C ONV L AB E VALUATOR

Original: In order to evaluate the performance of dialogue agents in an end-to-end fashion, we conduct simulator-based evaluation on ConvLab (Zhu et al., 2020).
Revised: In order to evaluate the performance of dialogue agents in an end-to-end fashion, we conduct simulator-based evaluation on ConvLab (Zhu et al., 2020).

Original: ConvLab is an open-source toolkit that enables to build task-oriented dialogue systems and perform an end-to-end evaluation.
Revised: ConvLab is an open-source toolkit that enables to build task-oriented dialogue systems and perform an end-to-end evaluation.

Original: The simulator-based evaluation is more reliable than dataset-based automatic evaluation because it evaluates the performance while interacting with the user simulator.
Revised: The simulator-based evaluation is more reliable than dataset-based automatic evaluation because it evaluates the performance while interacting with the user simulator.

Original: To interact with dialogue systems,
Revised: To interact with dialogue systems,

Original: ConvLab provides an agenda-based user simulator (Schatzmann et al., 2007) that consists of a BERT (Devlin et al., 2019) for NLU, a rule-based policy, and a template-based NLG.
Revised: ConvLab provides an agenda-based user simulator (Schatzmann et al., 2007) that consists of a BERT (Devlin et al., 2019) for NLU, a rule-based policy, and a template-based NLG.

Original: We compare the performance of GPT-Critic with baseline algorithms interacting with the same user simulator and user goals.
Revised: We compare the performance of GPT-Critic with baseline algorithms interacting with the same user simulator and user goals.

Original: We report the results with the following metrics:
Revised: We report the results with the following metrics:

Original: 1) Complete : evaluates whether the system completes the goal, 2) Success : evaluates whether all the user requests have been informed and the booked entities satisfy the constraints, 3) Book : evaluates how many booked entities satisfy the user constraints, 4)
Revised: 1) Complete : evaluates whether the system completes the goal, 2) Success : evaluates whether all the user requests have been informed and the booked entities satisfy the constraints, 3) Book : evaluates how many booked entities satisfy the user constraints, 4)

Original: Inform (Precision / Recall / F1) : evaluates how many user requests have been informed, 5) Turn (success / all) : evaluates the average turn number for successful/all dialogues.
Revised: Inform (Precision / Recall / F1) : evaluates how many user requests have been informed, 5) Turn (success / all) : evaluates the average turn number for successful/all dialogues.

Original: We describe the performance of GPT-Critic and baselines in Table 7.
Revised: We describe the performance of GPT-Critic and baselines in Table 7.

Original: Each algorithm is tested for 1000 runs with randomly sampled user goal.
Revised: Each algorithm is tested for 1000 runs with randomly sampled user goal.

Original: The results show that GPT-Critic achieves the best
Revised: The results show that GPT-Critic achieves the best

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: performance in all metrics related to task accomplishment.
Revised: performance in all metrics related to task accomplishment.

Original: However, they also show that GPT-Critic takes longer dialogue turn for the task accomplishment because GPT-Critic is trained by maximizing the success rate without considering the dialogue turn.
Revised: However, they also show that GPT-Critic takes longer dialogue turn for the task accomplishment because GPT-Critic is trained by maximizing the success rate without considering the dialogue turn.

Original: 5. H UMAN E VALUATION
Revised: 5. H UMAN E VALUATION

Original: We also conduct human evaluation on Amazon Mechanical Turk (AMT) to assess the quality of generated responses of GPT-Critic and baseline algorithms, using the evaluation protocol as in (Yang et al., 2021; Lin et al., 2020; Zhang et al., 2020).
Revised: We also conduct human evaluation on Amazon Mechanical Turk (AMT) to assess the quality of generated responses of GPT-Critic and baseline algorithms, using the evaluation protocol as in (Yang et al., 2021; Lin et al., 2020; Zhang et al., 2020).

Original: Speciﬁcally, human workers on AMT were asked to read the context and generated response by interactive simulation via ConvLab, then score the following two evaluation metrics on a Likert scale (1-5): 1) Appropriateness : evaluates whether the generated responses are appropriate for the given context, 2) Fluency : evaluates whether the generated responses are comprehensible and human-like.
Revised: Speciﬁcally, human workers on AMT were asked to read the context and generated response by interactive simulation via ConvLab, then score the following two evaluation metrics on a Likert scale (1-5): 1) Appropriateness : evaluates whether the generated responses are appropriate for the given context, 2) Fluency : evaluates whether the generated responses are comprehensible and human-like.

Original: We compare the performance of GPT-Critic with same baselines on ConvLab evaluation.
Revised: We compare the performance of GPT-Critic with same baselines on ConvLab evaluation.

Original: Figure 3 summarizes the overall results of human evaluation, where 60 workers evaluate the quality of 30 randomly selected dialogues for each algorithm.
Revised: Figure 3 summarizes the overall results of human evaluation, where 60 workers evaluate the quality of 30 randomly selected dialogues for each algorithm.

Original: The results show that GPT-Critic signiﬁcantly outperforms baseline algorithms in appropriateness which is related to task accomplishment.
Revised: The results show that GPT-Critic signiﬁcantly outperforms baseline algorithms in appropriateness which is related to task accomplishment.

Original: Moreover, the result of ﬂuency shows that GPT-Critic does not hurt the agent’s capability to generate human-like sentences.
Revised: Moreover, the result of ﬂuency shows that GPT-Critic does not hurt the agent’s capability to generate human-like sentences.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented GPT-Critic, an ofﬂine RL algorithm for task-oriented dialogue system, which can be adopted for any generative pre-trained language model.
Revised: We presented GPT-Critic, an ofﬂine RL algorithm for task-oriented dialogue system, which can be adopted for any generative pre-trained language model.

Original: GPT-Critic aims to learn an end-to-end task-oriented dialogue agent without the issue of diverging from human language.
Revised: GPT-Critic aims to learn an end-to-end task-oriented dialogue agent without the issue of diverging from human language.

Original: GPT-Critic starts with ﬁne-tuning the GPT-2 model and learning the critic using the dialogue corpus.
Revised: GPT-Critic starts with ﬁne-tuning the GPT-2 model and learning the critic using the dialogue corpus.

Original: Then, GPTCritic updates the policy through the behavior cloning of the critic-guided self-generated responses, thus it is essentially free from the issue of diverging from human language.
Revised: Then, GPTCritic updates the policy through the behavior cloning of the critic-guided self-generated responses, thus it is essentially free from the issue of diverging from human language.

Original: In the experiments, we demonstrated that GPT-Critic outperforms the state-of-the-art algorithms in the task-oriented dialogue benchmarks including MultiWOZ 2.0 and ConvLab.
Revised: In the experiments, we demonstrated that GPT-Critic outperforms the state-of-the-art algorithms in the task-oriented dialogue benchmarks including MultiWOZ 2.0 and ConvLab.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was supported by the National Research Foundation (NRF) of Korea (NRF2019R1A2C1087634, NRF-2021M3I1A1097938) and Institute of Information & communications
Revised: This work was supported by the National Research Foundation (NRF) of Korea (NRF2019R1A2C1087634, NRF-2021M3I1A1097938) and Institute of Information & communications

Original: Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
Revised: Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: (No.20190-00075, No.2020-0-00940, No.2021-0-02068)
Revised: (No.20190-00075, No.2020-0-00940, No.2021-0-02068)

Original: A P OLICY I MPROVEMENT T HEOREM
Revised: A P OLICY I MPROVEMENT T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: (Policy Improvement)
Revised: (Policy Improvement)

Original: Given a policy π and the number of sampling actions N ≥ 1 , If we update the new policy π new N by
Revised: Given a policy π and the number of sampling actions N ≥ 1 , If we update the new policy π new N by

Original: [Equation]
Revised: [Equation]

Original: then Q π new N ( s, a ) ≥ Q π ( s, a ) ∀ s, a always holds.
Revised: then Q π new N ( s, a ) ≥ Q π ( s, a ) ∀ s, a always holds.

Original: Furthermore, for any N, M such that N ≥ M ≥ 1 , Q π new N ( s, a ) ≥ Q π new M ( s, a ) ∀ s, a always holds.
Revised: Furthermore, for any N, M such that N ≥ M ≥ 1 , Q π new N ( s, a ) ≥ Q π new M ( s, a ) ∀ s, a always holds.

Original: Proof.
Revised: Proof.

Original: For any s, a, N ≥ M ,
Revised: For any s, a, N ≥ M ,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] 
Revised: [Table] [Equation] [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: For the case of N = 1 , note that π new1 is simply reduced to the π
Revised: For the case of N = 1 , note that π new1 is simply reduced to the π

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof:
Revised: which concludes the proof:

Original: [Equation]
Revised: [Equation]

Original: , N ≥ M ≥ 1 .
Revised: , N ≥ M ≥ 1 .

Original: B Q UALITATIVE A NALYSIS OF S ELF - GENERATED D IALOGUES
Revised: B Q UALITATIVE A NALYSIS OF S ELF - GENERATED D IALOGUES

Original: WOZ
Revised: WOZ

Original: Original Dialogue / Self-generated Dialogue
Revised: Original Dialogue / Self-generated Dialogue

Original: User Utterance i would like a moderately prices restaurant in the east part of town.
Revised: User Utterance i would like a moderately prices restaurant in the east part of town.

Original: i would like the phone number and postcode, please.
Revised: i would like the phone number and postcode, please.

Original: Dialogue State [restaurant] pricerange moderate area east
Revised: Dialogue State [restaurant] pricerange moderate area east

Original: Dialogue Act [restaurant] [select] food [inform] choice area
Revised: Dialogue Act [restaurant] [select] food [inform] choice area

Original: System Response we have [value choice] matching your request in the [value area] part of town.
Revised: System Response we have [value choice] matching your request in the [value area] part of town.

Original: do you prefer [value food] or [value food]?
Revised: do you prefer [value food] or [value food]?

Original: Dialogue Act * [restaurant] [inform] food price area phone name postcode
Revised: Dialogue Act * [restaurant] [inform] food price area phone name postcode

Original: System Response * [value name] is a [value price] [value food] restaurant in the [value area] part of town.
Revised: System Response * [value name] is a [value price] [value food] restaurant in the [value area] part of town.

Original: their phone number is [value phone] and their postcode is [value postcode].
Revised: their phone number is [value phone] and their postcode is [value postcode].

Original: SNG
Revised: SNG

Original: Original Dialogue / Self-generated Dialogue
Revised: Original Dialogue / Self-generated Dialogue

Original: User Utterance i am looking for the nearest police station
Revised: User Utterance i am looking for the nearest police station

Original: Dialogue StateDialogue Act [police] [inform] phone
Revised: Dialogue StateDialogue Act [police] [inform] phone

Original: System Response the nearest police station is in parkside cambridge, the telephone number is [value phone].
Revised: System Response the nearest police station is in parkside cambridge, the telephone number is [value phone].

Original: Dialogue Act * [police] [inform] name phone address postcode
Revised: Dialogue Act * [police] [inform] name phone address postcode

Original: System Response * the nearest police station is [value name], located in [value address] [value postcode].
Revised: System Response * the nearest police station is [value name], located in [value address] [value postcode].

Original: their phone number is [value phone].
Revised: their phone number is [value phone].

Original: is there anything else i can help you with?
Revised: is there anything else i can help you with?

Original: PMUL
Revised: PMUL

Original: Original Dialogue / Self-generated Dialogue
Revised: Original Dialogue / Self-generated Dialogue

Original: User Utterance i heard cityroomz is a good hotel.
Revised: User Utterance i heard cityroomz is a good hotel.

Original: can you give me some information on this hotel?
Revised: can you give me some information on this hotel?

Original: Dialogue State [hotel] name cityroomz
Revised: Dialogue State [hotel] name cityroomz

Original: Dialogue Act [hotel] [inform] area price
Revised: Dialogue Act [hotel] [inform] area price

Original: System Response it is located in the [value area] and is priced in the [value price] range.
Revised: System Response it is located in the [value area] and is priced in the [value price] range.

Original: Dialogue Act * [hotel] [inform] area name internet price parking type stars [offerbook]
Revised: Dialogue Act * [hotel] [inform] area name internet price parking type stars [offerbook]

Original: System Response * [value name] is a [value type] in the [value area].
Revised: System Response * [value name] is a [value type] in the [value area].

Original: it is [value price], has [value stars] stars, and offers free wiﬁ and parking.
Revised: it is [value price], has [value stars] stars, and offers free wiﬁ and parking.

Original: would you like me to book it for you?
Revised: would you like me to book it for you?

Original: MUL
Revised: MUL

Original: Original Dialogue / Self-generated
Revised: Original Dialogue / Self-generated

Original: [Table]
Revised: [Table]

Original: In this section, we provide a qualitative analysis on the critic-guided self-generated responses in
Revised: In this section, we provide a qualitative analysis on the critic-guided self-generated responses in

Original: GPT-Critic.
Revised: GPT-Critic.

Original: We show the critic-guided self-generated dialogue examples in Table 5 that illustrates how GPT-Critic improves the performance through the behavior cloning of self-generated dialogues, compared with unsuccessful dialogues on the training dataset of MultiWOZ.
Revised: We show the critic-guided self-generated dialogue examples in Table 5 that illustrates how GPT-Critic improves the performance through the behavior cloning of self-generated dialogues, compared with unsuccessful dialogues on the training dataset of MultiWOZ.

Original: Each example demonstrates the critic-guided self-generated dialogue act and delexicalized system response, compared with the original dialogue.
Revised: Each example demonstrates the critic-guided self-generated dialogue act and delexicalized system response, compared with the original dialogue.

Original: The generated responses contain all the requests of user with abundant information, whereas the original responses of unsuccessful dialogues do not contain all the requested information.
Revised: The generated responses contain all the requests of user with abundant information, whereas the original responses of unsuccessful dialogues do not contain all the requested information.

Original: GPT-Critic improves the performance through the behavior cloning of these revised responses.
Revised: GPT-Critic improves the performance through the behavior cloning of these revised responses.

Original: Moreover, Table 5 shows that the generated dialogues do not diverge from human language.
Revised: Moreover, Table 5 shows that the generated dialogues do not diverge from human language.

Original: Since GPT-Critic updates the policy through behavior cloning of the self-generated human-like responses, GPT-Critic is essentially free from the issue of diverging from human language.
Revised: Since GPT-Critic updates the policy through behavior cloning of the self-generated human-like responses, GPT-Critic is essentially free from the issue of diverging from human language.

Original: C Q UALITATIVE E XAMPLES OF S TANDARD R EINFORCEMENT L EARNING A LGORITHM
Revised: C Q UALITATIVE E XAMPLES OF S TANDARD R EINFORCEMENT L EARNING A LGORITHM

Original: In this section, we provide examples of responses generated by standard RL algorithm (REINFORCE) to show that the policy-gradient-based standard RL algorithms suffer from diverging from human language.
Revised: In this section, we provide examples of responses generated by standard RL algorithm (REINFORCE) to show that the policy-gradient-based standard RL algorithms suffer from diverging from human language.

Original: As can be shown in Table 6, policy-gradient-based RL algorithm generates responses which is diverging from human language.
Revised: As can be shown in Table 6, policy-gradient-based RL algorithm generates responses which is diverging from human language.

Original: WOZ
Revised: WOZ

Original: Original Dialogue / Generated Dialogue
Revised: Original Dialogue / Generated Dialogue

Original: User Utterance i need a restaurant.
Revised: User Utterance i need a restaurant.

Original: Dialogue StateDialogue Act [restaurant] [request] area
Revised: Dialogue StateDialogue Act [restaurant] [request] area

Original: System Response in what part of town?
Revised: System Response in what part of town?

Original: Dialogue Act * [restaurant] [request] name
Revised: Dialogue Act * [restaurant] [request] name

Original: System Response * do you know what you are looking for?
Revised: System Response * do you know what you are looking for?

Original: i can direct you to something you will need.
Revised: i can direct you to something you will need.

Original: if you think of it, please let me know.
Revised: if you think of it, please let me know.

Original: goodbye.
Revised: goodbye.

Original: i hope everything works.
Revised: i hope everything works.

Original: thank you for calling.
Revised: thank you for calling.

Original: goodbye.
Revised: goodbye.

Original: i hope everything works.
Revised: i hope everything works.

Original: PMUL
Revised: PMUL

Original: Original Dialogue / Generated Dialogue
Revised: Original Dialogue / Generated Dialogue

Original: User Utterance awesome.
Revised: User Utterance awesome.

Original: i also need a place to eat in the moderate range.
Revised: i also need a place to eat in the moderate range.

Original: i am thinking maybe in the city centre.
Revised: i am thinking maybe in the city centre.

Original: Dialogue State [restaurant] pricerange moderate area centre
Revised: Dialogue State [restaurant] pricerange moderate area centre

Original: [Table]
Revised: [Table]

Original: day wednesday arrive 17: destination
Revised: day wednesday arrive 17: destination

Original: [Table]
Revised: [Table]

Original: departure
Revised: departure

Original: [Table]
Revised: [Table]

Original: liverpool street people
Revised: liverpool street people

Original: Dialogue Act [restaurant] [select] [inform] area food choice price
Revised: Dialogue Act [restaurant] [select] [inform] area food choice price

Original: System Response there are [value choice] [value price] restaurant s in the [value area], 4 serve [value food], and 1 serves [value food] food.
Revised: System Response there are [value choice] [value price] restaurant s in the [value area], 4 serve [value food], and 1 serves [value food] food.

Original: which would you prefer?
Revised: which would you prefer?

Original: Dialogue Act * -
Revised: Dialogue Act * -

Original: System Response * alright, have a wonderful day!
Revised: System Response * alright, have a wonderful day!

Original: you have a wonderful day as well.
Revised: you have a wonderful day as well.

Original: have a great day!!!!!
Revised: have a great day!!!!!

Original: you have a nice day as well!!!!!!!!!!!!!!!
Revised: you have a nice day as well!!!!!!!!!!!!!!!

Original: PMUL
Revised: PMUL

Original: Original Dialogue / Generated Dialogue
Revised: Original Dialogue / Generated Dialogue

Original: User Utterance i need a hotel and i
Revised: User Utterance i need a hotel and i

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: [Table]
Revised: [Table]

Original: please
Revised: please

Original: [Table]
Revised: [Table]

Original: speciﬁc
Revised: speciﬁc

Original: [Table]
Revised: [Table]

Original: type
Revised: type

Original: [Table]
Revised: [Table]

Original: stars
Revised: stars

Original: [Table]
Revised: [Table]

Original: you
Revised: you

Original: [Table]
Revised: [Table]

Original: like
Revised: like

Original: [Table]
Revised: [Table]

Original: D E VALUATION FOR THE QUALITY OF GENERATED DIALOGUE STATES AND DIALOGUE ACTS
Revised: D E VALUATION FOR THE QUALITY OF GENERATED DIALOGUE STATES AND DIALOGUE ACTS

Original: [Table]
Revised: [Table]

Original: We additionally conducted the experiments with UBAR and GPT-Critic to explicitly evaluate the generated dialogue states and dialogue acts (rather than evaluating the ﬁnal system response).
Revised: We additionally conducted the experiments with UBAR and GPT-Critic to explicitly evaluate the generated dialogue states and dialogue acts (rather than evaluating the ﬁnal system response).

Original: The table below shows the performance for predicted dialogue state(Joint accuracy / Slot accuracy) and predicted dialogue act (Dialogue Act f1), where the mean performance and the standard error are reported.
Revised: The table below shows the performance for predicted dialogue state(Joint accuracy / Slot accuracy) and predicted dialogue act (Dialogue Act f1), where the mean performance and the standard error are reported.

Original: As the table presents, GPT-Critic outperforms UBAR on Dialogue Act f1, which measures the performance of dialogue policy prediction.
Revised: As the table presents, GPT-Critic outperforms UBAR on Dialogue Act f1, which measures the performance of dialogue policy prediction.

Original: However, in the case of dialogue state tracking, there is no signiﬁcant performance gap between GPT-Critic and UBAR since our GPT-Critic revises only the dialogue act and system response (which are considered as an action in GPT-Critic) but not the dialogue state in the dataset.
Revised: However, in the case of dialogue state tracking, there is no signiﬁcant performance gap between GPT-Critic and UBAR since our GPT-Critic revises only the dialogue act and system response (which are considered as an action in GPT-Critic) but not the dialogue state in the dataset.