Dialog Action-Aware Transformer for Dialog Policy Learning (2023)

Abstract: Recent works usually address Dialog policy learning DPL by training a reinforcement learning (RL) agent to determine the best dialog action. However, existing works on deep RL require a large volume of agent-user interactions to achieve acceptable performance. In this paper, we propose to make full use of the plain text knowledge from the pre-trained language model to accelerate the RL agent’s learning speed. Specifically, we design a dialog action-aware transformer encoder (DaTrans), which integrates a new fine-tuning procedure named masked last action task to encourage DaTrans to be dialog-aware and distill action-specific features. Then, DaTrans is further optimized in an RL setting with ongoing interactions and evolves through exploration in the dialog action space toward maximizing long-term accumulated rewards. The effectiveness and efficiency of the proposed model are demonstrated with both simulator evaluation and human evaluation.

## introduction a task-oriented dialog system that can serve users on certain tasks has increasingly attracted research efforts. dialog policy learning (dpl) aiming to determine the next abstracted system output plays a key role in pipeline task-oriented dialog systems @xcite . recently, it has shown great potential for using reinforcement learning (rl) based methods to formulate dpl @xcite @xcite . a lot of progress is being made in demonstration-based efficient learning methods @xcite @xcite @xcite . among these methods, dialog state tracking (dst), comprising all information required to determine the * equal contribution response, is an indispensable module. however, dst inevitably accumulates errors from each module of the system. recent pre-trained language models (plms) gathering knowledge from the massive plain text show great potential for formulating dpl without dst. recently, the studies on plms for dialog, including bert-based dialog state tracking @xcite and gpt-2 based dialog generation @xcite are not centred on dpl. to this end, we proposed the dialog action-oriented transformer encoder termed as datrans, for efficient dialog policy training. datrans is achieved by a dialog act-aware fine-tuning task, which encourages the model to distil the dialog policy logic. specifically, rather than commonly used tasks, like predicting randomly masked words in the input (mlm task) and classifying whether the sentences are continuous or not (nsp task) @xcite , datrans is fine-tuned by predicting the masked last acts in the input action sequences (termed as mla task). after that, datrans works as an rl agent which evolves toward maximizing longterm accumulated rewards through interacting with a user simulator. following the traditional rl-based dialog policy learning framework, the main novelty of datrans is that it integrates a proposed dialog action-aware fine-tuning task (mla), which helps to extract action-specific features from historical dialog action sequences to improve dialog policy learning. the empirical results prove the excellent performance of datrans. our main contributions include 1) we propose the datrans that integrates the dialog act-aware fine-tuning task to extract the dialog policy logic from the plain text; 2) we validate the efficiency and effectiveness of the proposed model on a multi-domain benchmark with both simulator and human evaluation. ## approach we cast the dialog policy learning problem as a markov decision process and optimize the policy with deep reinforcement learning approaches. rl usually involves an interactive process (as shown in figure 1 ), during which the dialog agent's behavior should choose actions that tend to increase the long-turn sum of rewards given by the user. it can learn to do this over time, by systematic trials and errors until reaches the optimal. in our setting, the dialog agent is encoded with the proposed datrans, which perceives the state s and determines the next action a a . we consider a transformer decoder-based policy model, which takes text concatenating of tuples containing a domain name, an intent type, and slot names as input and determines the next action. ## experiments and results we first conduct the simulator evaluation to assess the datrans' performance of learning efficiency, the robustness of fine-tuning corpus, and domain adaptation. besides, the case study and human evaluation are conducted and the results are presented in section d & e in appendix. in our experiment, nlu and nlg modules are ignored since the interactions are made with dialog actions. notably, datrans can be equipped with any nlu and nlg models. two datasets, multiwoz @xcite and schema-guided dialog (sgd) @xcite are involved. we leverage a public available agenda-based user simulator @xcite ) setup on multiwoz. the details of the dataset, implementation, and the user simulator are illustrated in the appendix. ## conclusion and future work in this paper, we investigate the pre-trained language model enhancing the reinforcement learning agent for dialog policy learning. we propose datrans, which is equipped with a new fine-tuning task that masks the last dialog action to extract the dialog logic for efficient dialog policy learning. the evaluation results show the effectiveness of the proposed datrans in terms of learning efficiency and domain adaptation ability.