Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning (2022)

Abstract: When a natural language generation (NLG) component is implemented in a real-world task-oriented dialogue system, it is necessary to generate not only natural utterances as learned on training data but also utterances adapted to the dialogue environment (e.g., noise from environmental sounds) and the user (e.g., users with low levels of understanding ability). Inspired by recent advances in reinforcement learning (RL) for language generation tasks, we propose ANTOR, a method for A daptive N atural language generation for T ask- O riented dialogue via R einforcement learning. In ANTOR, a natural language understanding (NLU) module, which corresponds to the user’s understanding of system utterances, is incorporated into the objective function of RL. If the NLG’s intentions are correctly conveyed to the NLU, which understands a system’s utterances, the NLG is given a positive reward. We conducted experiments on the MultiWOZ dataset, and we confirmed that ANTOR could generate adaptive utterances against speech recognition errors and the different vocabulary levels of users.

## introduction in task-oriented dialogue systems, the role of the natural language generation (nlg) component is to convert a system's intentions, called dialogue acts (das), into natural language utterances and to convey das accurately to users @xcite . in recent years, data-driven language generation methods @xcite using neural networks have been introduced to nlg for task-oriented dialogue systems, enabling natural utterance generation. when such nlg is implemented in a realistic environment, however, it is essential to generate not only natural utterances as learned on training data but also utterances adapted to the dialogue environment and the user. for example, when interacting in a noisy environment, such as in a place with loud background noise or through a telephone, the system needs to use sentences and vocabulary that are less likely to be misrecognized. in addition, if the user is a child or a second language learner, it is necessary to generate utterances in plain terms that the user can easily understand. therefore, it is essential for the nlg module to adaptively generate utterances for the dialogue environment and the user in real-world situations. however, it is challenging to implement optimal nlg using only supervised learning because it is not practical to create training data for every environment or user. recently, for many generative tasks, such as machine translation, summary generation, and dialogue generation in open domains, many methods using reinforcement learning (rl) have been proposed. in these studies, non-differentiable objective functions, such as generated text quality and subjective user preferences, are used to optimize the language generation model and achieve high performance. with this background in mind, this study proposes a method for adaptive natural language generation for task-oriented dialogue via reinforcement learning (antor) foot_0 , which adapts to the dialogue environment and the user. in our method, a reward function using a natural language understanding (nlu) model is set up, and a pretrained nlg model is fine-tuned by using rl. that is, the nlg generates a system utterance for a given da, and the nlu provides a positive reward if it can successfully recognize the original da from the utterance. experiments using the multiwoz dataset @xcite are conducted with multiple environments and users simulating real-world conditions, such as speech recognition errors and the different vocabulary levels of users. our contribution is threefold: the results show that antor could generate utterances with words less likely to cause speech recognition errors. the results confirmed that antor was able to generate utterances using vocabulary appropriate for each vocabulary level. ## experiments we wanted to confirm that antor is capable of generating utterances adapted to the dialogue environment and the user. to verify the effectiveness of antor, we conducted experiments using simulations. ## case study to see how antor improved the performance of nlu, we analyzed the behavior of antor. in this analysis, we used bert because we thought that a difference from gpt-2 could be clearly seen since the f1 was improved more by using bert than milu as in in clean environments, but similar behaviors were observed for noisy environments and different user vocabulary levels. first, we listed the intents and slots in the das for which the nlu prediction accuracy was considerably improved by the utterances of antor compared with those of gpt-2 ( note that since humans will not have the problems that the nlu had here because humans have a better understanding, we expect antor to adapt differently when interacting with humans. ## summary and future work this paper investigated whether nlg can generate utterances adapted to the dialogue environment and the user via rl. we proposed a method, an-tor, and conducted experiments using multiwoz to confirm that antor can generate such utterances for multiple nlus with different model architectures. in addition, we also consistently confirmed the effectiveness of antor for noisy environments and a user's vocabulary levels. for future work, we plan to evaluate whether antor optimized for nlu is also effective for humans. we are also interested in extending our method for practical use (e.g., real-time adaptation to users in an online dialogue environment). furthermore, we would like to utilize methods to optimize an entire system with rl, such as @xcite and @xcite , so that all modules of a system can be adapted to users.