Original: Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System
Revised: Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System

Original: Anonymous ACL submission
Revised: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Task-Oriented Dialogue (TOD) systems are designed to carry out specific tasks by trackingdialogue states and generating appropriate responses to help users achieve defined goals.
Revised: Task-Oriented Dialogue (TOD) systems are designed to carry out specific tasks by trackingdialogue states and generating appropriate responses to help users achieve defined goals.

Original: Recently, end-to-end dialogue models pre-trainedbased on large datasets have shown promising performance in the conversational system.
Revised: Recently, end-to-end dialogue models pre-trainedbased on large datasets have shown promising performance in the conversational system.

Original: However, they share the same parameters totrain tasks of the dialogue system (NLU, DST,
Revised: However, they share the same parameters totrain tasks of the dialogue system (NLU, DST,

Original: NLG), so debugging each task is challenging.
Revised: NLG), so debugging each task is challenging.

Original: Also, they require a lot of effort to fine-tunelarge parameters to create a task-oriented chatbot, making it difficult for non-experts to handle.
Revised: Also, they require a lot of effort to fine-tunelarge parameters to create a task-oriented chatbot, making it difficult for non-experts to handle.

Original: Therefore, we intend to train relativelylightweight and fast models compared to PLM.
Revised: Therefore, we intend to train relativelylightweight and fast models compared to PLM.

Original: In this paper, we propose an End-to-end TODsystem with Task-Optimized Adapters whichlearn independently per task, adding only smallnumber of parameters after fixed layers of pretrained network.
Revised: In this paper, we propose an End-to-end TODsystem with Task-Optimized Adapters whichlearn independently per task, adding only smallnumber of parameters after fixed layers of pretrained network.

Original: We also enhance the performance of the DST and NLG modules throughreinforcement learning, overcoming the learning curve that has lacked at the adapter learning and enabling the natural and consistent response generation that is appropriate for thegoal.
Revised: We also enhance the performance of the DST and NLG modules throughreinforcement learning, overcoming the learning curve that has lacked at the adapter learning and enabling the natural and consistent response generation that is appropriate for thegoal.

Original: Our method is a model-agnostic approachand does not require prompt-tuning as only input data without a prompt.
Revised: Our method is a model-agnostic approachand does not require prompt-tuning as only input data without a prompt.

Original: As results of theexperiment, our method shows competitive performance on the MultiWOZ benchmark compared to the existing end-to-end models.
Revised: As results of theexperiment, our method shows competitive performance on the MultiWOZ benchmark compared to the existing end-to-end models.

Original: In particular, we attain state-of-the-art performanceon the DST task of 2.2 dataset 1 .
Revised: In particular, we attain state-of-the-art performanceon the DST task of 2.2 dataset 1 .

Original:  Introduction
Revised:  Introduction

Original: Task-oriented dialogue systems are trained toachieve specific goal to enhance efficiency and convenience in various fields such as customer service centers and healthcare information retrieval.
Revised: Task-oriented dialogue systems are trained toachieve specific goal to enhance efficiency and convenience in various fields such as customer service centers and healthcare information retrieval.

Original: Task-oriented dialogue systems are divided intokey components: understanding the user’s intent
Revised: Task-oriented dialogue systems are divided intokey components: understanding the user’s intent

Original: (NLU), tracking the current dialogue states (DST), and generating responses based on previous sessions (NLG).
Revised: (NLU), tracking the current dialogue states (DST), and generating responses based on previous sessions (NLG).

Original: Pipeline-based systems separately train each component, so they have the advantage of optimizing each module and raising the performance of a given task.
Revised: Pipeline-based systems separately train each component, so they have the advantage of optimizing each module and raising the performance of a given task.

Original: User feedback is, however, difficult to propagate to each module, and inputs to the component is dependent on the result of the previous module (Chen et al., 2017).
Revised: User feedback is, however, difficult to propagate to each module, and inputs to the component is dependent on the result of the previous module (Chen et al., 2017).

Original: Recently, dialogue systems have been trained in an end-toend manner with transfer learning or pre-training networks with large dialogue corpora.
Revised: Recently, dialogue systems have been trained in an end-toend manner with transfer learning or pre-training networks with large dialogue corpora.

Original: However, building efficient end-to-end TOD systems requires a large amount of data and has some limitations due to parameter sharing.
Revised: However, building efficient end-to-end TOD systems requires a large amount of data and has some limitations due to parameter sharing.

Original: End-to-end models backpropagate to transfer the gradients of the output and end back to the entire neural network.
Revised: End-to-end models backpropagate to transfer the gradients of the output and end back to the entire neural network.

Original: They pose an issue of parameter efficiency as updating all parameters for every downstream scenario.
Revised: They pose an issue of parameter efficiency as updating all parameters for every downstream scenario.

Original: Also, it is challenging to debug each task and take task-flow characteristics into account.
Revised: Also, it is challenging to debug each task and take task-flow characteristics into account.

Original: Therefore, we propose a simple structure of adding adapters to the core modules (NLU, DST,
Revised: Therefore, we propose a simple structure of adding adapters to the core modules (NLU, DST,

Original: NLG) of the TOD system as shown in Figure 1.
Revised: NLG) of the TOD system as shown in Figure 1.

Original: By using the adapters, it is possible to optimize each task with only a small amount of training parameters, remaining the pre-trained model’s parameters fixed.
Revised: By using the adapters, it is possible to optimize each task with only a small amount of training parameters, remaining the pre-trained model’s parameters fixed.

Original: Additionally, it is safe from the catastrophic forgetting problem (French, 1999), which causes pre-trained models to lose important skills acquired during the pre-training process.
Revised: Additionally, it is safe from the catastrophic forgetting problem (French, 1999), which causes pre-trained models to lose important skills acquired during the pre-training process.

Original: The key is to apply a transfer learning strategy that yields compact and extensible downstream models in the dialogue system (Houlsby et al., 2019).
Revised: The key is to apply a transfer learning strategy that yields compact and extensible downstream models in the dialogue system (Houlsby et al., 2019).

Original: This makes it easy for people to train large-scale end-to-end TOD models.
Revised: This makes it easy for people to train large-scale end-to-end TOD models.

Original: Also, by applying REINFORCE (Sutton et al., 1999), we attempt to reduce the expected score gap caused by the small parameters of adapter compared to the full fine-tuning.
Revised: Also, by applying REINFORCE (Sutton et al., 1999), we attempt to reduce the expected score gap caused by the small parameters of adapter compared to the full fine-tuning.

Original: Specifically, we use
Revised: Specifically, we use

Original: Joint Goal Accuracy, and weighted sum of BLEU and Success rate as rewards for training DST and
Revised: Joint Goal Accuracy, and weighted sum of BLEU and Success rate as rewards for training DST and

Original: [Figure]
Revised: [Figure]

Original: a place to dine in the center that s expensive.
Revised: a place to dine in the center that s expensive.

Original: [system] I have several options for you; do you prefer
Revised: [system] I have several options for you; do you prefer

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: or British
Revised: or British

Original: [Figure]
Revised: [Figure]

Original: [user]
Revised: [user]

Original: [Figure]
Revised: [Figure]

Original: of
Revised: of

Original: [Figure]
Revised: [Figure]

Original: would be fine, as long as it is a bit expensive.
Revised: would be fine, as long as it is a bit expensive.

Original: [Figure]
Revised: [Figure]

Original: I get the
Revised: I get the

Original: [Figure]
Revised: [Figure]

Original: number for your recommendation?
Revised: number for your recommendation?

Original: [Figure]
Revised: [Figure]

Original: DST
Revised: DST

Original: [Figure]
Revised: [Figure]

Original: NLG
Revised: NLG

Original: [Figure]
Revised: [Figure]

Original: [user] I need a place to dine in the center that s expensive.
Revised: [user] I need a place to dine in the center that s expensive.

Original: [system] I have several options for you; do you prefer
Revised: [system] I have several options for you; do you prefer

Original: African, Asian, or British food?
Revised: African, Asian, or British food?

Original: [user] Any sort of food would be fine, as long as it is a bit expensive.
Revised: [user] Any sort of food would be fine, as long as it is a bit expensive.

Original: Could I get the phone number for your recommendation?
Revised: Could I get the phone number for your recommendation?

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: NLG adapter.
Revised: NLG adapter.

Original: To best of our knowledge, this is firstwork that use Joint Goal Accuracy as a reward for
Revised: To best of our knowledge, this is firstwork that use Joint Goal Accuracy as a reward for

Original: E2E TOD system.
Revised: E2E TOD system.

Original: To address the aforementioned problems, wepropose a T ask-
Revised: To address the aforementioned problems, wepropose a T ask-

Original: O ptimized A dapter for an end-toend T ask-
Revised: O ptimized A dapter for an end-toend T ask-

Original: O riented D ialogue system ( TOATOD )applying reinforcement learning to DST and NLGtasks.
Revised: O riented D ialogue system ( TOATOD )applying reinforcement learning to DST and NLGtasks.

Original: In summary, our key contributions are asfollows:
Revised: In summary, our key contributions are asfollows:

Original: • We present a new architecture that can debug
Revised: • We present a new architecture that can debug

Original: per task of the end-to-end model using the
Revised: per task of the end-to-end model using the

Original: separated adapters.
Revised: separated adapters.

Original: • Without updating the original parameters of
Revised: • Without updating the original parameters of

Original: PLM, we train end-to-end TOD system effi-
Revised: PLM, we train end-to-end TOD system effi-

Original: ciently with a few trainable parameters.
Revised: ciently with a few trainable parameters.

Original: • It is a novel approach to design a reward func-
Revised: • It is a novel approach to design a reward func-

Original: tion not just for NLG, but also for DST task
Revised: tion not just for NLG, but also for DST task

Original: with metric-aware reinforcement learning.
Revised: with metric-aware reinforcement learning.

Original: • The performance of the proposed approach
Revised: • The performance of the proposed approach

Original: outperforms on the DST task of MultiWOZ
Revised: outperforms on the DST task of MultiWOZ

Original: 2.2 and shows comparable results to full
Revised: 2.2 and shows comparable results to full

Original: finetuning on the NLU and NLG tasks.
Revised: finetuning on the NLU and NLG tasks.

Original:  Background
Revised:  Background

Original: Pipeline-based Task-Oriented Dialogue System
Revised: Pipeline-based Task-Oriented Dialogue System

Original: Conventional task-oriented dialogue systems usually based on the pipeline method, consisted of language understanding (NLU), dialogue state tracking (DST), policy learning (POL), and languagegeneration (NLG).
Revised: Conventional task-oriented dialogue systems usually based on the pipeline method, consisted of language understanding (NLU), dialogue state tracking (DST), policy learning (POL), and languagegeneration (NLG).

Original: This kind of modularization allows for each component to be optimized independently, making it easier to update and understandhow the model is working.
Revised: This kind of modularization allows for each component to be optimized independently, making it easier to update and understandhow the model is working.

Original: Pipeline-based systems,however, have several limitations.
Revised: Pipeline-based systems,however, have several limitations.

Original: Each of the modules train sequentially, so it is hard to align modulesto the common optimization targets (Liu and Lane, 2018).
Revised: Each of the modules train sequentially, so it is hard to align modulesto the common optimization targets (Liu and Lane, 2018).

Original: This makes the system more complex and harder to backpropagate cumulated errors.
Revised: This makes the system more complex and harder to backpropagate cumulated errors.

Original: The performance of the previous components affects the next modules, so if upper modules perform poorly, errors that occurred earlier may propagate and be amplified in downstream components (Liu and Lane, 2018).
Revised: The performance of the previous components affects the next modules, so if upper modules perform poorly, errors that occurred earlier may propagate and be amplified in downstream components (Liu and Lane, 2018).

Original: End-to-end Task-Oriented Dialogue System Endto-end task-oriented dialogue systems, on the other hand, are easier to optimize and train to directly map the input to output in a single model.
Revised: End-to-end Task-Oriented Dialogue System Endto-end task-oriented dialogue systems, on the other hand, are easier to optimize and train to directly map the input to output in a single model.

Original: They can leverage large amounts of data for robust learning and the entire system can be optimized under end-to-end settings, which leads to better performance.
Revised: They can leverage large amounts of data for robust learning and the entire system can be optimized under end-to-end settings, which leads to better performance.

Original: A general approach for building end-to-end systems is to fine-tune pre-trained language models (Budzianowski and Vuli´c, 2019).
Revised: A general approach for building end-to-end systems is to fine-tune pre-trained language models (Budzianowski and Vuli´c, 2019).

Original: This approach utilizes the strength of pre-trained networks, which help the models to leverage the pre-trained knowledge while also adapting to task-specific data.
Revised: This approach utilizes the strength of pre-trained networks, which help the models to leverage the pre-trained knowledge while also adapting to task-specific data.

Original: For example, SimpleTOD (Hosseini-Asl et al., 2020) solved task-oriented dialogue as causal language modeling task using several versions of GPT (Radford and Narasimhan, 2018;
Revised: For example, SimpleTOD (Hosseini-Asl et al., 2020) solved task-oriented dialogue as causal language modeling task using several versions of GPT (Radford and Narasimhan, 2018;

Original: Radford et al.).
Revised: Radford et al.).

Original: Pre-training of Dialogue Language Model Recently, methods with pre-training dialogue language model (Wu et al., 2020; Zhang et al., 2020b;
Revised: Pre-training of Dialogue Language Model Recently, methods with pre-training dialogue language model (Wu et al., 2020; Zhang et al., 2020b;

Original: Peng et al., 2021;
Revised: Peng et al., 2021;

Original: Su et al., 2022;
Revised: Su et al., 2022;

Original: He et al., 2022a), instead of fine-tuning pre-trained networks have outperformed the previous baselines on the benchmark.
Revised: He et al., 2022a), instead of fine-tuning pre-trained networks have outperformed the previous baselines on the benchmark.

Original: For instance, SPACE-3 (He et al., 2022a) captures the contextualized knowledge from largescale dialogue corpora by pre-training the unified language model.
Revised: For instance, SPACE-3 (He et al., 2022a) captures the contextualized knowledge from largescale dialogue corpora by pre-training the unified language model.

Original: However, there are still some issues with these methods.
Revised: However, there are still some issues with these methods.

Original: A large amount of parameters is required for training backbone models like
Revised: A large amount of parameters is required for training backbone models like

Original: BERT, T5 (Raffel et al., 2020), GPT, and UniLM(Dong et al., 2019).
Revised: BERT, T5 (Raffel et al., 2020), GPT, and UniLM(Dong et al., 2019).

Original: As shown in Table 1, T5 baseand T5 small require the trainable parameters over220M and 60M respectively.
Revised: As shown in Table 1, T5 baseand T5 small require the trainable parameters over220M and 60M respectively.

Original: And they disregardthe task-flow features of task-oriented dialogue systems.
Revised: And they disregardthe task-flow features of task-oriented dialogue systems.

Original: Also, it’s still hard to debug per module because the model parameters are shared and jointlyoptimized.
Revised: Also, it’s still hard to debug per module because the model parameters are shared and jointlyoptimized.

Original: PPTOD (Su et al., 2022) integratedmodules into a unified model with task-specificprompts and alleviated the error accumulation inplug-and-play way.
Revised: PPTOD (Su et al., 2022) integratedmodules into a unified model with task-specificprompts and alleviated the error accumulation inplug-and-play way.

Original: Still, this method is not completely free from the interference among tasks dueto fully shared parameters.
Revised: Still, this method is not completely free from the interference among tasks dueto fully shared parameters.

Original: Adapter tuning for NLP Since the pre-trainedmodels for NLP tasks have become mainstream,they are mainly used for transfer learning downstream tasks.
Revised: Adapter tuning for NLP Since the pre-trainedmodels for NLP tasks have become mainstream,they are mainly used for transfer learning downstream tasks.

Original: However, a parameter efficiency issuehas been raised because updating all PLM parameters is expensive for every downstream scenario.
Revised: However, a parameter efficiency issuehas been raised because updating all PLM parameters is expensive for every downstream scenario.

Original: To address the issue, the adapter module is proposed to transfer PLM like BERT with parameterefficient tuning (Houlsby et al., 2019) and showscomparable performance to full fine-tuning.
Revised: To address the issue, the adapter module is proposed to transfer PLM like BERT with parameterefficient tuning (Houlsby et al., 2019) and showscomparable performance to full fine-tuning.

Original: Themain idea behind the adapter is to train the network to the downstream task with task-specific parameters while maintaining the original pre-trainedparameters.
Revised: Themain idea behind the adapter is to train the network to the downstream task with task-specific parameters while maintaining the original pre-trainedparameters.

Original: The module is composed of two feedforward layers and a non-linear layer, which can beinserted into the transformer blocks of an end-toend model.
Revised: The module is composed of two feedforward layers and a non-linear layer, which can beinserted into the transformer blocks of an end-toend model.

Original: It projects d-dimensional input featuresinto a smaller dimension m , and then projects backinto the original dimension, so the total of parameters add per layer with biases is 2 md + d + m .
Revised: It projects d-dimensional input featuresinto a smaller dimension m , and then projects backinto the original dimension, so the total of parameters add per layer with biases is 2 md + d + m .

Original: The number of additional parameters per task can be restricted by setting m < d (Li et al., 2021).
Revised: The number of additional parameters per task can be restricted by setting m < d (Li et al., 2021).

Original: In most of the previous research, the adapter was used for efficient learning.
Revised: In most of the previous research, the adapter was used for efficient learning.

Original: But in this approach, we adapt the adapter modules not just for efficient learning (Stickland and Murray, 2019), but also task-optimized learning as recent studies that have mainly focused on separating parameters (Lin et al., 2021;
Revised: But in this approach, we adapt the adapter modules not just for efficient learning (Stickland and Murray, 2019), but also task-optimized learning as recent studies that have mainly focused on separating parameters (Lin et al., 2021;

Original: Feng et al., 2022; Bapna and Firat, 2019).
Revised: Feng et al., 2022; Bapna and Firat, 2019).

Original: Reinforcement learning
Revised: Reinforcement learning

Original: for Text Generation
Revised: for Text Generation

Original: Although token-based supervised learning is a widely adopted training method in text generation tasks, as highlighted by Ranzato et al., 2016, there are two major problems associated with this approach.
Revised: Although token-based supervised learning is a widely adopted training method in text generation tasks, as highlighted by Ranzato et al., 2016, there are two major problems associated with this approach.

Original: The first problem is the exposure bias problem, where during training, the model is exposed to the ground-truth outputs, thus allowing it to learn to generate text that is similar to the training data.
Revised: The first problem is the exposure bias problem, where during training, the model is exposed to the ground-truth outputs, thus allowing it to learn to generate text that is similar to the training data.

Original: However, during evaluation, the model is not exposed to the ground-truth outputs and instead, generates texts based on the previous words generated by the model itself.
Revised: However, during evaluation, the model is not exposed to the ground-truth outputs and instead, generates texts based on the previous words generated by the model itself.

Original: This can lead to errors and a significant deviation in the generated text from the training data.
Revised: This can lead to errors and a significant deviation in the generated text from the training data.

Original: The second problem is the tokenlevel loss problem, where the training objective function is based solely on individual words’ prediction, neglecting the predicted token sequence’s overall coherence and fluency.
Revised: The second problem is the tokenlevel loss problem, where the training objective function is based solely on individual words’ prediction, neglecting the predicted token sequence’s overall coherence and fluency.

Original: To address these problems, researchers have applied reinforcement learning to text generation tasks (Ranzato et al., 2016;
Revised: To address these problems, researchers have applied reinforcement learning to text generation tasks (Ranzato et al., 2016;

Original: Li et al., 2016; Paulus et al., 2018; Chen et al., 2020; Wang et al., 2021; Ye et al., 2022) using metrics such as BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) as rewards for sequence-level training.
Revised: Li et al., 2016; Paulus et al., 2018; Chen et al., 2020; Wang et al., 2021; Ye et al., 2022) using metrics such as BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) as rewards for sequence-level training.

Original: In our study, we also use the REINFORCE method and task-oriented dialogue metrics to continually train our model after supervised learning, not only to address these challenges but also to mitigate performance degradation caused by the use of an Adapter.
Revised: In our study, we also use the REINFORCE method and task-oriented dialogue metrics to continually train our model after supervised learning, not only to address these challenges but also to mitigate performance degradation caused by the use of an Adapter.

Original:  Methodology
Revised:  Methodology

Original: 3. Adapter for each task (NLU, DST, NLG)
Revised: 3. Adapter for each task (NLU, DST, NLG)

Original: The Adapter module is designed to adapt the pretrained network for each of the tasks (NLU, DST,
Revised: The Adapter module is designed to adapt the pretrained network for each of the tasks (NLU, DST,

Original: NLG) in a task-oriented dialogue system.
Revised: NLG) in a task-oriented dialogue system.

Original: As illustrated in Figure 2, the adapters are inserted after the feed-forward layer following the multi-head
Revised: As illustrated in Figure 2, the adapters are inserted after the feed-forward layer following the multi-head

Original: [Table]
Revised: [Table]

Original: attention layer of the transformer blocks of theend-to-end model.
Revised: attention layer of the transformer blocks of theend-to-end model.

Original: It enables the model to learntask-specific representations while preserving theshared parameters learned during pre-training.
Revised: It enables the model to learntask-specific representations while preserving theshared parameters learned during pre-training.

Original: Asdescribed in Table 1, our model consists of largesize frozen shared parameters per task and smallsize of trainable parameters that account for about14% of the entire network.
Revised: Asdescribed in Table 1, our model consists of largesize frozen shared parameters per task and smallsize of trainable parameters that account for about14% of the entire network.

Original: While original network’s parameters are frozen, the j th adapter oftask i ∈ {NLU, DST, NLG}, A ij computes as below:
Revised: While original network’s parameters are frozen, the j th adapter oftask i ∈ {NLU, DST, NLG}, A ij computes as below:

Original: [Equation]
Revised: [Equation]

Original: The output of the j th feed-forward layer withresidual connection in the transformer block is represented as H j ∈ R n × d , where n is the input dimension, and d is the hidden dimension.
Revised: The output of the j th feed-forward layer withresidual connection in the transformer block is represented as H j ∈ R n × d , where n is the input dimension, and d is the hidden dimension.

Original: As shownin Figure 2, the overall architecture of the adaptermodule includes multiple feed-forward projections,referred to as down-projection and up-projection,followed by layer normalization (LN).
Revised: As shownin Figure 2, the overall architecture of the adaptermodule includes multiple feed-forward projections,referred to as down-projection and up-projection,followed by layer normalization (LN).

Original: The downprojection with W down ∈ R
Revised: The downprojection with W down ∈ R

Original: d
Revised: d

Original: × h projects the input
Revised: × h projects the input

Original: H j , which is passed by the ReLU activation function, and then the up-projection with W up ∈ R h × dprojects the output back to the original dimension.
Revised: H j , which is passed by the ReLU activation function, and then the up-projection with W up ∈ R h × dprojects the output back to the original dimension.

Original: The bottleneck dimension h is a hyperparameter toproject the original input to a smaller dimension.
Revised: The bottleneck dimension h is a hyperparameter toproject the original input to a smaller dimension.

Original: And each adapter has a residual connection to avoidvanishing gradient (Rebuffi et al., 2017;
Revised: And each adapter has a residual connection to avoidvanishing gradient (Rebuffi et al., 2017;

Original: He et al.,2016).
Revised: He et al.,2016).

Original: 3. Metric-Aware Reinforcement Learning
Revised: 3. Metric-Aware Reinforcement Learning

Original: for DST & NLG module
Revised: for DST & NLG module

Original: The overall loss function for the metric-aware reinforcement learning is given by the equation (2):
Revised: The overall loss function for the metric-aware reinforcement learning is given by the equation (2):

Original: [Equation]
Revised: [Equation]

Original: The equation describes how the network updatesthe parameters in order to maximize both the likelihood of the generated response (token loss) and the quality of the response (policy loss).
Revised: The equation describes how the network updatesthe parameters in order to maximize both the likelihood of the generated response (token loss) and the quality of the response (policy loss).

Original: The token loss can be described as a categorical cross-entropy loss.
Revised: The token loss can be described as a categorical cross-entropy loss.

Original: y ∗ denotes the predicted probability of ground truth and y is the target probability.
Revised: y ∗ denotes the predicted probability of ground truth and y is the target probability.

Original: The token loss
Revised: The token loss

Original: CE ( y, y ∗ ) , which measures how well a set of predicted token probabilities match the target tokens for a given context of dialogue when reinforcement learning.
Revised: CE ( y, y ∗ ) , which measures how well a set of predicted token probabilities match the target tokens for a given context of dialogue when reinforcement learning.

Original: By applying REINFORCE method, the network can update weights towards the direction that allows the model for getting more rewards even when the reward function is non-differentiable.
Revised: By applying REINFORCE method, the network can update weights towards the direction that allows the model for getting more rewards even when the reward function is non-differentiable.

Original: [Equation]
Revised: [Equation]

Original: The policy loss J policy ( θ ) is introduced to measure of how well the model can generate a token sequence with high probabilities that result in high rewards.
Revised: The policy loss J policy ( θ ) is introduced to measure of how well the model can generate a token sequence with high probabilities that result in high rewards.

Original: ˆ y denotes the predicted token sequence by model.
Revised: ˆ y denotes the predicted token sequence by model.

Original: The policy loss described in the equation (3) is calculated as the negative log probability of the token sequence that has the highest probability multiplied by the reward.
Revised: The policy loss described in the equation (3) is calculated as the negative log probability of the token sequence that has the highest probability multiplied by the reward.

Original: The parameter α in the overall loss (2) is a hyperparameter, that is a scalar value between 0 and 1 to weigh the importance of these losses.
Revised: The parameter α in the overall loss (2) is a hyperparameter, that is a scalar value between 0 and 1 to weigh the importance of these losses.

Original: In this way, the model is trained to predict the correct labels (categorical cross-entropy loss) and to make good decisions that result in high rewards (policy loss) at the same time.
Revised: In this way, the model is trained to predict the correct labels (categorical cross-entropy loss) and to make good decisions that result in high rewards (policy loss) at the same time.

Original: We define the reward functions of the DST and NLG modules as follows:
Revised: We define the reward functions of the DST and NLG modules as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y )
Revised: y )

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: The Reward DST is calculated as the sum of the
Revised: The Reward DST is calculated as the sum of the

Original: Joint Goal Accuracy (JGA) and a constant value of 1.
Revised: Joint Goal Accuracy (JGA) and a constant value of 1.

Original: The JGA measures how well the model predicted the values for every slot in the dialogue turnsy using JGA as a reward, the moda el is encouraged to accurately track the state of the dialogue, which is crucial for generating appropriate responses and improving the performance of task-oriented dialogue systems.
Revised: The JGA measures how well the model predicted the values for every slot in the dialogue turnsy using JGA as a reward, the moda el is encouraged to accurately track the state of the dialogue, which is crucial for generating appropriate responses and improving the performance of task-oriented dialogue systems.

Original: Reward
Revised: Reward

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the equation (5), BLEU score and Success rate are used as rewards to guide the learning process of NLG module.
Revised: In the equation (5), BLEU score and Success rate are used as rewards to guide the learning process of NLG module.

Original: y u denotes the ground truth token sequence of each utterance, and ˆ y u denotes  the predicted token sequence.
Revised: y u denotes the ground truth token sequence of each utterance, and ˆ y u denotes  the predicted token sequence.

Original: To calculate the success rate, we apply the batch of a session-level,not an utterance-level.
Revised: To calculate the success rate, we apply the batch of a session-level,not an utterance-level.

Original: y and ˆ y without u meanthe session level ground truth and prediction.
Revised: y and ˆ y without u meanthe session level ground truth and prediction.

Original: Theweighting factor β is adjusted to balance these twometrics.
Revised: Theweighting factor β is adjusted to balance these twometrics.

Original: The hyperparameter may have to be carefully chosen because there is a trade-off relationship between these two metrics in the RL setting(Wu et al., 2021), where increasing one metric maycome at the cost of decreasing the other.
Revised: The hyperparameter may have to be carefully chosen because there is a trade-off relationship between these two metrics in the RL setting(Wu et al., 2021), where increasing one metric maycome at the cost of decreasing the other.

Original: We experiment to choose α and β on the Section 6.1.3 and6.1.4.
Revised: We experiment to choose α and β on the Section 6.1.3 and6.1.4.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: 4. Datasets
Revised: 4. Datasets

Original: We experiment our method for dialogue state tracking (DST) and end-to-end response generation(NLG) tasks on the MultiWOZ 2.1 and 2.2 datasets,and the intent prediction (NLU) task with the Banking77, CLINC150, and HWU64 datasets.
Revised: We experiment our method for dialogue state tracking (DST) and end-to-end response generation(NLG) tasks on the MultiWOZ 2.1 and 2.2 datasets,and the intent prediction (NLU) task with the Banking77, CLINC150, and HWU64 datasets.

Original: MultiWOZ2.1, 2.
Revised: MultiWOZ2.1, 2.

Original: The
Revised: The

Original: MultiWOZ(Budzianowski et al., 2018) dataset has beenwidely used to evaluate the performance of TODsystems.
Revised: MultiWOZ(Budzianowski et al., 2018) dataset has beenwidely used to evaluate the performance of TODsystems.

Original: It consists of 8438, 1000, and 1000 fortraining, dev, test sets with multi-turn dialogues,collected through a Wizard-of-Oz (WOZ) setup.
Revised: It consists of 8438, 1000, and 1000 fortraining, dev, test sets with multi-turn dialogues,collected through a Wizard-of-Oz (WOZ) setup.

Original: The dialogues cover a wide range of domainsand topics.
Revised: The dialogues cover a wide range of domainsand topics.

Original: MultiWOZ 2.2 (Zang et al.
Revised: MultiWOZ 2.2 (Zang et al.

Original: , 2020)is the improved version of MultiWOZ 2.1 (Ericet al., 2020) that has corrected annotation errors,inconsistencies, and ontology issues, also addedspan annotations to standardize.
Revised: , 2020)is the improved version of MultiWOZ 2.1 (Ericet al., 2020) that has corrected annotation errors,inconsistencies, and ontology issues, also addedspan annotations to standardize.

Original: Banking 77 (Casanueva et al., 2020)
Revised: Banking 77 (Casanueva et al., 2020)

Original: This datasetis a collection of 77 real-life customer bankingservice queries.
Revised: This datasetis a collection of 77 real-life customer bankingservice queries.

Original: It consists of 13,083 utterances.
Revised: It consists of 13,083 utterances.

Original: Each query is labeled with a single intent, however,it is hard to differentiate because they correspondto very similar tasks.
Revised: Each query is labeled with a single intent, however,it is hard to differentiate because they correspondto very similar tasks.

Original: CLINC150 (Larson et al., 2019)
Revised: CLINC150 (Larson et al., 2019)

Original: This dataset ismulti-domain dataset which contains 23,700 utterances that cover 150 intent classes over 10 domains.
Revised: This dataset ismulti-domain dataset which contains 23,700 utterances that cover 150 intent classes over 10 domains.

Original: HWU64 (Liu et al., 2019)
Revised: HWU64 (Liu et al., 2019)

Original: This dataset consistsof 25,716 examples.
Revised: This dataset consistsof 25,716 examples.

Original: It maps user utterances tostructured, but mode abstract.
Revised: It maps user utterances tostructured, but mode abstract.

Original: The data providesannotation with the 64 intents from 21 differentdomains.
Revised: The data providesannotation with the 64 intents from 21 differentdomains.

Original: [Table]
Revised: [Table]

Original: 4. Baselines & Settings
Revised: 4. Baselines & Settings

Original: For the DST task, we compare our models with other strong baselines including TRADE (Wu et al., 2019),
Revised: For the DST task, we compare our models with other strong baselines including TRADE (Wu et al., 2019),

Original: DS-DST (Zhang et al., 2020a),
Revised: DS-DST (Zhang et al., 2020a),

Original: DST-Picklist (Zhang et al., 2020a), TripPy (Heck et al., 2020), ConvBERT+DG+Multi (Mehri et al., 2020a), TripPy+SaCLog (Dai et al., 2021), SimpleTOD (Hosseini-Asl et al., 2020), AG-DST (Tian et al., 2021), UniLM, SPACE-3, D3ST (Zhao et al., 2022),
Revised: DST-Picklist (Zhang et al., 2020a), TripPy (Heck et al., 2020), ConvBERT+DG+Multi (Mehri et al., 2020a), TripPy+SaCLog (Dai et al., 2021), SimpleTOD (Hosseini-Asl et al., 2020), AG-DST (Tian et al., 2021), UniLM, SPACE-3, D3ST (Zhao et al., 2022),

Original: SDP-DST (Lee et al., 2021), and PPTOD.
Revised: SDP-DST (Lee et al., 2021), and PPTOD.

Original: On the NLG task, we choose models trained on PLM in an end-to-end setting such as DoTS (Jeon and Lee, 2021), PPTOD, UBAR (Yang et al., 2021), MTTOD (Lee, 2021), RSTOD (Cholakov and Kolev, 2022), GALAXY (He et al., 2022b),
Revised: On the NLG task, we choose models trained on PLM in an end-to-end setting such as DoTS (Jeon and Lee, 2021), PPTOD, UBAR (Yang et al., 2021), MTTOD (Lee, 2021), RSTOD (Cholakov and Kolev, 2022), GALAXY (He et al., 2022b),

Original: MinTL (Lin et al., 2020), SOLOLIST, BORT (Sun et al., 2022a) and Mars (Sun et al., 2022b).
Revised: MinTL (Lin et al., 2020), SOLOLIST, BORT (Sun et al., 2022a) and Mars (Sun et al., 2022b).

Original: For the NLU task, we compare TOATOD with existing baselines of each dataset.
Revised: For the NLU task, we compare TOATOD with existing baselines of each dataset.

Original: We use T5 small and T5 base as backbone models initialized with PPTOD (Su et al., 2022)’s pretrained weights, which are trained on a large dialogue dataset.
Revised: We use T5 small and T5 base as backbone models initialized with PPTOD (Su et al., 2022)’s pretrained weights, which are trained on a large dialogue dataset.

Original: As described in the Appendix A, the bottleneck dimension h of the adapter is 1/2 of
Revised: As described in the Appendix A, the bottleneck dimension h of the adapter is 1/2 of

Original: [Table]
Revised: [Table]

Original: the hidden dimension of the T5 model.
Revised: the hidden dimension of the T5 model.

Original: We use the
Revised: We use the

Original: Adafactor (Shazeer and Stern, 2018) optimizer with15 epochs and set batch size as 16, learning rate of1e-4 during supervised learning of DST and NLGtasks.
Revised: Adafactor (Shazeer and Stern, 2018) optimizer with15 epochs and set batch size as 16, learning rate of1e-4 during supervised learning of DST and NLGtasks.

Original: We sweep a wide range of learning rates:{ 1e-5, 1e-6, 1e-7 } .
Revised: We sweep a wide range of learning rates:{ 1e-5, 1e-6, 1e-7 } .

Original: For reinforcement learning, wetrain models 10 epochs for DST and 3 epochs for
Revised: For reinforcement learning, wetrain models 10 epochs for DST and 3 epochs for

Original: NLG.
Revised: NLG.

Original: We do not train NLG-optimized adapters forthe MultiWOZ 2.2, because there are not significantchanges of response annotations from MultiWOZ2.1 and intend to train robust models with noiseddataset.
Revised: We do not train NLG-optimized adapters forthe MultiWOZ 2.2, because there are not significantchanges of response annotations from MultiWOZ2.1 and intend to train robust models with noiseddataset.

Original: We follow the preprocessing method from
Revised: We follow the preprocessing method from

Original: UBAR to delexicalize slot values for each systemresponses.
Revised: UBAR to delexicalize slot values for each systemresponses.

Original: We evaluate our models using the olderversion of the standardized evaluation script for
Revised: We evaluate our models using the olderversion of the standardized evaluation script for

Original: MultiWOZ 2.1, and the newly opened version for
Revised: MultiWOZ 2.1, and the newly opened version for

Original: MultiWOZ 2.2, released by Nekvinda and Dusek,2021.
Revised: MultiWOZ 2.2, released by Nekvinda and Dusek,2021.

Original: It has been adopted by the official MultiWoZdataset github 2 .
Revised: It has been adopted by the official MultiWoZdataset github 2 .

Original: Other implementation details aredescribed in Appendix C.
Revised: Other implementation details aredescribed in Appendix C.

Original:  Experimental Results
Revised:  Experimental Results

Original: 5. Dialogue State Tracking
Revised: 5. Dialogue State Tracking

Original: We evaluate our models on the DST task with the
Revised: We evaluate our models on the DST task with the

Original: MultiWOZ 2.1 & 2.2 datasets.
Revised: MultiWOZ 2.1 & 2.2 datasets.

Original: We compute Joint
Revised: We compute Joint

Original: Goal Accuracy on the test set, which measures howmany values are filled accurately compared to theground truth states for all slots.
Revised: Goal Accuracy on the test set, which measures howmany values are filled accurately compared to theground truth states for all slots.

Original: Joint Goal Accuracy is considered as more difficult and importantmetric in most research (Zhou and Small, 2019;
Revised: Joint Goal Accuracy is considered as more difficult and importantmetric in most research (Zhou and Small, 2019;

Original: Dey et al., 2022), because once wrong prediction has been made, it cannot get the score at that turn.
Revised: Dey et al., 2022), because once wrong prediction has been made, it cannot get the score at that turn.

Original: 5.1. Evaluation Result
Revised: 5.1. Evaluation Result

Original: We compare our best models, TOATOD small and
Revised: We compare our best models, TOATOD small and

Original: TOATOD base to the models trained with a pretrained network.
Revised: TOATOD base to the models trained with a pretrained network.

Original: Table 2 shows that our models are competitive to the end-to-end models on the current benchmark.
Revised: Table 2 shows that our models are competitive to the end-to-end models on the current benchmark.

Original: In the 2.1 dataset, our models show a relatively good performance, despite the small number of trainable parameters.
Revised: In the 2.1 dataset, our models show a relatively good performance, despite the small number of trainable parameters.

Original: As shown in the Table 5, the Joint Goal Accuracy of TOATOD base only with task-optimized adapter (SL) is 53.33, which is slightly lower than other models using T5 base as backbone.
Revised: As shown in the Table 5, the Joint Goal Accuracy of TOATOD base only with task-optimized adapter (SL) is 53.33, which is slightly lower than other models using T5 base as backbone.

Original: So, we reduce the performance degradation applying metric-aware REINFORCE and report the final results.
Revised: So, we reduce the performance degradation applying metric-aware REINFORCE and report the final results.

Original: Trainable parameters of our models are less than 1/2 of the models with the smallest parameters.
Revised: Trainable parameters of our models are less than 1/2 of the models with the smallest parameters.

Original: The result implies that the adapter module helps the network more adaptable to the DST task by only activating a few parts of the model.
Revised: The result implies that the adapter module helps the network more adaptable to the DST task by only activating a few parts of the model.

Original: Because of relatively small parameters, our model is more robust to overfitting problem with confused labels.
Revised: Because of relatively small parameters, our model is more robust to overfitting problem with confused labels.

Original: As mentioned in Section 4.1,
Revised: As mentioned in Section 4.1,

Original: MultiWOZ 2.2 is the cleaned version of 2.1 dataset, so the performance is better in the 2.2.
Revised: MultiWOZ 2.2 is the cleaned version of 2.1 dataset, so the performance is better in the 2.2.

Original: Among the top results on the 2.2 dataset, our models obtain state-of-the-art performance.
Revised: Among the top results on the 2.2 dataset, our models obtain state-of-the-art performance.

Original: It demonstrates that TOATOD optimizes well on the given task remaining the prior knowledge learned from the pre-trained network.
Revised: It demonstrates that TOATOD optimizes well on the given task remaining the prior knowledge learned from the pre-trained network.

Original: 5. End-to-End Response Generation
Revised: 5. End-to-End Response Generation

Original: We test our methods with end-to-end response generation (NLG) task on the MultiWOZ 2.1 & 2.2 asin DST evaluation.
Revised: We test our methods with end-to-end response generation (NLG) task on the MultiWOZ 2.1 & 2.2 asin DST evaluation.

Original: Four metrics are used to measure the quality of generated responses.
Revised: Four metrics are used to measure the quality of generated responses.

Original: We measure if the system provides the appropriate entity(Inform rate), answers all the requested information (Success rate), and responds fluently (BLEUscore).
Revised: We measure if the system provides the appropriate entity(Inform rate), answers all the requested information (Success rate), and responds fluently (BLEUscore).

Original: And the Combined Score for end-to-endresponse generation is computed as ’BLEU+0.5 ×(Inform+Success)’.
Revised: And the Combined Score for end-to-endresponse generation is computed as ’BLEU+0.5 ×(Inform+Success)’.

Original: Under the end-to-end settings,the models have to predict proper dialogue statesand then generate responses based on the states.
Revised: Under the end-to-end settings,the models have to predict proper dialogue statesand then generate responses based on the states.

Original: 5.2. Evaluation Result
Revised: 5.2. Evaluation Result

Original: From Table 3, our models achieve comparable results (1.x point different from SOTA model) in alldatasets.
Revised: From Table 3, our models achieve comparable results (1.x point different from SOTA model) in alldatasets.

Original: The adapter module helps each part of themodel be fine-tuned independently, therefore wecan optimize the DST and NLG task respectively.
Revised: The adapter module helps each part of themodel be fine-tuned independently, therefore wecan optimize the DST and NLG task respectively.

Original: Our model performs well on NLG task based on thebelief states from DST modules and base knowledge gained from the large-scale dialog datasetduring pre-training.
Revised: Our model performs well on NLG task based on thebelief states from DST modules and base knowledge gained from the large-scale dialog datasetduring pre-training.

Original: TOATOD base attains the bestscore on the Inform and Success rate.
Revised: TOATOD base attains the bestscore on the Inform and Success rate.

Original: It showsthat the reinforcement learning of our approach iseffective for adjusting trade-off problem between
Revised: It showsthat the reinforcement learning of our approach iseffective for adjusting trade-off problem between

Original: BLEU score and others.
Revised: BLEU score and others.

Original: 5. Intent Classification
Revised: 5. Intent Classification

Original: [Table]
Revised: [Table]

Original: We test our models on the NLU task with Banking77, CLINC150, and HWU64.
Revised: We test our models on the NLU task with Banking77, CLINC150, and HWU64.

Original: Intent predictionis the task to identify the intent behind a given in- put.
Revised: Intent predictionis the task to identify the intent behind a given in- put.

Original: The task is normally framed as a classification problem, so we set the metric as turn accuracy.
Revised: The task is normally framed as a classification problem, so we set the metric as turn accuracy.

Original: 5.3. Evaluation Result
Revised: 5.3. Evaluation Result

Original: From Table 4, while our models do not achieve the highest score on NLU task, it is important to note that they perform well in relation to the size when compared to other models.
Revised: From Table 4, while our models do not achieve the highest score on NLU task, it is important to note that they perform well in relation to the size when compared to other models.

Original: This highlights the effectiveness of our task-optimized adapter approach in achieving a balance between model performance and efficiency.
Revised: This highlights the effectiveness of our task-optimized adapter approach in achieving a balance between model performance and efficiency.

Original:  Further Analysis and Discussion
Revised:  Further Analysis and Discussion

Original: 6. Reinforcement Learning
Revised: 6. Reinforcement Learning

Original: w/o Reinforcement Learning
Revised: w/o Reinforcement Learning

Original: [Table]
Revised: [Table]

Original: Described on the Table 5, after applying reinforcement learning, the performance of DST and NLG modules are enhanced.
Revised: Described on the Table 5, after applying reinforcement learning, the performance of DST and NLG modules are enhanced.

Original: Models with reinforcement learning obtain best JGA on the DST task and get the highest Combined Score on the NLG task.
Revised: Models with reinforcement learning obtain best JGA on the DST task and get the highest Combined Score on the NLG task.

Original: The BLEU scores fall slightly but the Combined
Revised: The BLEU scores fall slightly but the Combined

Original: Score rise with Inform and Success, which means that incorporating reinforcement learning into the training process leads our model to complete tasks more efficiently.
Revised: Score rise with Inform and Success, which means that incorporating reinforcement learning into the training process leads our model to complete tasks more efficiently.

Original: We conduct hyperparameter tuning only for the TOATOD base , so there is some gap in the degree of performance improvement between
Revised: We conduct hyperparameter tuning only for the TOATOD base , so there is some gap in the degree of performance improvement between

Original: TOATOD small and TOATOD base .
Revised: TOATOD small and TOATOD base .

Original: 6.1. Hyperparameters of REINFORCE
Revised: 6.1. Hyperparameters of REINFORCE

Original: The hyperparameter α is used for balancing the importance between cross-entropy loss and policy loss.
Revised: The hyperparameter α is used for balancing the importance between cross-entropy loss and policy loss.

Original: We re-train DST and NLG modules by optimizing the mixed objective function to get higher rewards.
Revised: We re-train DST and NLG modules by optimizing the mixed objective function to get higher rewards.

Original: The second hyperparameter β is a scaling factor to control the trade-off between BLEU score
Revised: The second hyperparameter β is a scaling factor to control the trade-off between BLEU score

Original: [Table]
Revised: [Table]

Original: and Success rate for NLG module.
Revised: and Success rate for NLG module.

Original: The BLEUscore is calculated based on the number of matching n-grams between the generated text and thereference text.
Revised: The BLEUscore is calculated based on the number of matching n-grams between the generated text and thereference text.

Original: A higher score indicates that thegenerated text is more similar to the label text.
Revised: A higher score indicates that thegenerated text is more similar to the label text.

Original: Success rate measures the proportion of dialogues,where the model successfully completes the task.
Revised: Success rate measures the proportion of dialogues,where the model successfully completes the task.

Original: A higher success rate means the model is better atachieving the user’s goal of the dialogue.
Revised: A higher success rate means the model is better atachieving the user’s goal of the dialogue.

Original: In somecases, however, the model may generate text that isclose to the reference text (high BLEU score) butnot relevant to the current dialogue (low Successrate).
Revised: In somecases, however, the model may generate text that isclose to the reference text (high BLEU score) butnot relevant to the current dialogue (low Successrate).

Original: So, we aim to reduce the gap between twometrics via hyperparameter tuning.6.1.
Revised: So, we aim to reduce the gap between twometrics via hyperparameter tuning.6.1.

Original: α of DST-optimized adapterα 1.
Revised: α of DST-optimized adapterα 1.

Original: JGA 54.
Revised: JGA 54.

Original: Slot F 92.
Revised: Slot F 92.

Original: In the Table 7, the result of α experiment on DSTtask indicates that 1.0 yielded the best performance,suggesting the significance of the policy loss inthe overall loss function.
Revised: In the Table 7, the result of α experiment on DSTtask indicates that 1.0 yielded the best performance,suggesting the significance of the policy loss inthe overall loss function.

Original: By maximizing policyloss, the model is encouraged to make decisionsthat result in high rewards, which improves theperformance of the DST task.
Revised: By maximizing policyloss, the model is encouraged to make decisionsthat result in high rewards, which improves theperformance of the DST task.

Original: 6.1. α and β of NLG-optimized adapter
Revised: 6.1. α and β of NLG-optimized adapter

Original: We experiment with several combinations of α andβ within the range of { α : 0.3, 0.5, 0.7, 0.9, 1.0 / β :0.4, 0.5, 0.7, 0.9 } .
Revised: We experiment with several combinations of α andβ within the range of { α : 0.3, 0.5, 0.7, 0.9, 1.0 / β :0.4, 0.5, 0.7, 0.9 } .

Original: Regardless of hyperparameters,performances improve after applying the reinforcement learning.
Revised: Regardless of hyperparameters,performances improve after applying the reinforcement learning.

Original: The hyperparameters of α =0.5 andβ =0.7, result the best Combined Score, which is5.44 point higher than the performance of the supervised learning.
Revised: The hyperparameters of α =0.5 andβ =0.7, result the best Combined Score, which is5.44 point higher than the performance of the supervised learning.

Original: As shown in the Figure 3, on the experiment after removing impact of CE loss ( α =1), we found that the higher the β , success rate increases compared to BLEU score as we expected.
Revised: As shown in the Figure 3, on the experiment after removing impact of CE loss ( α =1), we found that the higher the β , success rate increases compared to BLEU score as we expected.

Original: As shown in Table 6, when the β is fixed, the bigger weight (smaller α ) on the CE loss ensures the higher BLEU score.
Revised: As shown in Table 6, when the β is fixed, the bigger weight (smaller α ) on the CE loss ensures the higher BLEU score.

Original: On the other hand, the success rate started to decline from the certain point.
Revised: On the other hand, the success rate started to decline from the certain point.

Original: Trade-off issue appears at this point.
Revised: Trade-off issue appears at this point.

Original: Therefore, α and β need to be properly tuned in NLG task for good performance.
Revised: Therefore, α and β need to be properly tuned in NLG task for good performance.

Original:  Conclusion
Revised:  Conclusion

Original: We propose TOATOD, task-optimized adapters for an end-to-end task dialogue system.
Revised: We propose TOATOD, task-optimized adapters for an end-to-end task dialogue system.

Original: By adapting task-optimized adapters, we utilize the end-to-end models without updating the pre-trained parameters and enabling debugging per task, which is different from previous research.
Revised: By adapting task-optimized adapters, we utilize the end-to-end models without updating the pre-trained parameters and enabling debugging per task, which is different from previous research.

Original: In addition, we apply REINFORCE algorithm with metric-aware reward function directly not only on the NLG task but also DST task to prevent score degradation.
Revised: In addition, we apply REINFORCE algorithm with metric-aware reward function directly not only on the NLG task but also DST task to prevent score degradation.

Original: As a result, we attain comparable performance to the previous SOTA models on every benchmark with very small number of trainable parameters.
Revised: As a result, we attain comparable performance to the previous SOTA models on every benchmark with very small number of trainable parameters.

Original: For the
Revised: For the

Original: DST task of MultiWOZ2.2, our TOATOD model outperforms the current SOTA systems.
Revised: DST task of MultiWOZ2.2, our TOATOD model outperforms the current SOTA systems.

Original: Limitations
Revised: Limitations

Original: We train the task-optimized adapters based on thepre-trained weights of dialogue LM.
Revised: We train the task-optimized adapters based on thepre-trained weights of dialogue LM.

Original: Therefore, ifapplied to other dialogue tasks such as chit-chatand conversational QA system, the performancecould be lower than that shown in our research.
Revised: Therefore, ifapplied to other dialogue tasks such as chit-chatand conversational QA system, the performancecould be lower than that shown in our research.

Original: And we need future works to clarify the reasonwhy the performance was better on the MultiWOZ2.2 dataset, which is expected that our model doesnot overfit to the confused labels.
Revised: And we need future works to clarify the reasonwhy the performance was better on the MultiWOZ2.2 dataset, which is expected that our model doesnot overfit to the confused labels.

Original: Our model inferences in on the end-to-end manner, but trains likemodular system for each task.
Revised: Our model inferences in on the end-to-end manner, but trains likemodular system for each task.

Original: End-to-end learningis currently under study.
Revised: End-to-end learningis currently under study.

Original: We could adapt the multitask end-to-end learning to our method, which maylead to the better performance.
Revised: We could adapt the multitask end-to-end learning to our method, which maylead to the better performance.

Original: Also, we could analyze the inner working of task-optimized adaptersapplying XAI technologies.
Revised: Also, we could analyze the inner working of task-optimized adaptersapplying XAI technologies.

Original: Ethics Statement
Revised: Ethics Statement

Original: We honor the ethical codes set out in the ACL codeof Ethics.
Revised: We honor the ethical codes set out in the ACL codeof Ethics.

Original: All of the datasets used in our studyare from previous studies and do not have privacyissues.
Revised: All of the datasets used in our studyare from previous studies and do not have privacyissues.

Original: Appendices
Revised: Appendices

Original: [Table]
Revised: [Table]

Original: A Units of Adapters
Revised: A Units of Adapters

Original: [Table]
Revised: [Table]

Original: In this experiment, we evaluate the performance with several bottleneck dimensions, h = 256, 128, 64 with TOATOD small , which is 1/2, 1/4, 1/8 size of T5 small ’s embedding dimension of 512.
Revised: In this experiment, we evaluate the performance with several bottleneck dimensions, h = 256, 128, 64 with TOATOD small , which is 1/2, 1/4, 1/8 size of T5 small ’s embedding dimension of 512.

Original: The main focus of the experiment is to investigate the effect of the bottleneck dimension (h) of the adapter module on the performance of the model.
Revised: The main focus of the experiment is to investigate the effect of the bottleneck dimension (h) of the adapter module on the performance of the model.

Original: To evaluate the performance of the model, we use the Joint
Revised: To evaluate the performance of the model, we use the Joint

Original: Goal Accuracy for DST task.
Revised: Goal Accuracy for DST task.

Original: We keep the other hyperparameters constant across all the experiments, including learning rate of 1e-4 and evaluate the performance on the test set of MultiWOZ 2.1.
Revised: We keep the other hyperparameters constant across all the experiments, including learning rate of 1e-4 and evaluate the performance on the test set of MultiWOZ 2.1.

Original: We report the result on the Table 8.
Revised: We report the result on the Table 8.

Original: The result implies that there is a trade-off between the bottleneck dimension and the performance of the model.
Revised: The result implies that there is a trade-off between the bottleneck dimension and the performance of the model.

Original: As the bottleneck dimension increases, the performance of the model also improves.
Revised: As the bottleneck dimension increases, the performance of the model also improves.

Original: The best performance is achieved with a bottleneck dimension of 256, where the JGA is 51.59.
Revised: The best performance is achieved with a bottleneck dimension of 256, where the JGA is 51.59.

Original: It is important to carefully choose the bottleneck dimension when using the adapter module in the task-oriented dialogue system.
Revised: It is important to carefully choose the bottleneck dimension when using the adapter module in the task-oriented dialogue system.

Original: As described in the Table 1, the trainable parameters of our model with bottleneck dimension of 256 is still signifi-  cantly smaller than the PLM’s parameters, so wechoose the size of 1/2.
Revised: As described in the Table 1, the trainable parameters of our model with bottleneck dimension of 256 is still signifi-  cantly smaller than the PLM’s parameters, so wechoose the size of 1/2.

Original: B w/o Reinforcement Learning of
Revised: B w/o Reinforcement Learning of

Original: TOATOD
Revised: TOATOD

Original: [Table]
Revised: [Table]

Original: C Implementation Details
Revised: C Implementation Details

Original: We used 8 A100 (80G) GPUs, but they were fullyused during only reinforcement learning.
Revised: We used 8 A100 (80G) GPUs, but they were fullyused during only reinforcement learning.

Original: Duringsupervised learning, we used 4 GPUs.
Revised: Duringsupervised learning, we used 4 GPUs.

Original: While reinforcement learning of the DST task-optimizedadapters, we set learning rate as 1e-5, and batchsize as 32 (utterance-level).
Revised: While reinforcement learning of the DST task-optimizedadapters, we set learning rate as 1e-5, and batchsize as 32 (utterance-level).

Original: On the contrary, whilereinforcement training of the NLG task-optimizedadapters, we set learning rate as 1e-6 and batch sizeas 4 (session-level).
Revised: On the contrary, whilereinforcement training of the NLG task-optimizedadapters, we set learning rate as 1e-6 and batchsize as 4 (session-level).

Original: During the entire training process, we set random seed as 42.
Revised: During the entire trainingprocess, we set random seed as 42.

Original: And for the NLUtask, we used 1 RTX A5000 (24GB) GPUs andtrained models without reinforcement learning.
Revised: And for the NLU task, we used 1 RTX A5000 (24GB) GPUand trained models without reinforcement learning.

Original: Weset batch size as 64 and across runs with differentrandom seeds.
Revised: We set batch size as 64 and single run with randomseeds.

Original: When training T5 small and T5 basefor the Banking 77 and CLINC150, we used learning rates of 0.001 and 0.15.
Revised: When training T5 small and T5 base for the Banking 77 and CLINC150, we used learning ratesof 0.001 and 0.15.

Original: And we used learningrates of 0.01 and 0.1 for the T5 small and T5 basewith the HWU64.
Revised: And we used learning rates of0.01 and 0.1 for the T5 small and T5 base with the HWU64.