Post-processing Networks: Method for Optimizing Pipeline Task-oriented Dialogue Systems using Reinforcement Learning (2022)

Abstract: Many studies have proposed methods for optimizing the dialogue performance of an entire pipeline task-oriented dialogue system by jointly training modules in the system using reinforcement learning. However, these methods are limited in that they can only be applied to modules implemented using trainable neural-based methods. To solve this problem, we propose a method for optimizing a pipeline system composed of modules implemented with arbitrary methods for dialogue performance. With our method, neural-based components called post-processing networks (PPNs) are installed inside such a system to post-process the output of each module. All PPNs are updated to improve the overall dialogue performance of the system by using reinforcement learning, not necessitating each module to be differentiable. Through dialogue simulation and human evaluation on the MultiWOZ dataset, we show that our method can improve the dialogue performance of pipeline systems consisting of various modules.

## introduction task-oriented dialogue systems can be classified into two categories: pipeline systems, in which multiple modules take on a sequential structure, and neural-based end-to-end systems @xcite @xcite . a typical pipeline system consists of four modules @xcite : natural language understanding (nlu), dialogue state tracking (dst), policy, and natural language generation (nlg). each module can be implemented individually using various methods (e.g., rule-based and neural-based) @xcite . in a pipeline system, the inputs and outputs of each module are explicit, making it easy for humans to interpret. 1 our code is publicly available at https://github. com/nu-dialogue/post-processing-networks however, since each module is processed sequentially, errors in the preceding module can easily propagate to the following ones, and the performance of the entire system cannot be optimized @xcite . this results in low dialogue performance of the entire system @xcite . in contrast, neural-based methods can optimize entire neural-based end-to-end systems, which allows for less error propagation than pipeline systems and high dialogue performance @xcite . the drawback of these methods is the large amount of annotation data required to train systems @xcite . compared with pipeline systems, neuralbased end-to-end systems are also less interpretable and more difficult to adjust or add functions. to marry the benefits of both pipeline and endto-end systems, methods @xcite @xcite have been proposed for optimizing an entire pipeline system in an end-to-end fashion by using reinforcement learning (rl) (figure 1(a) ). these methods are powerful because they jointly train and finetune neural-based implementations of the modules, such as nlu, policy, and nlg, by using rl. however, these methods may not always be applicable because there may be situations in which modules can only be implemented with rules or the modules' internals cannot be accessed, such as with a web api. with this background, we propose a method for optimizing an entire pipeline system composed of modules implemented in arbitrary methods. we specifically focus on modules that output fixed sets of classes (i.e., nlu, dst, and policy) and install neural-based components (post-processing networks; ppns) in the system to post-process the outputs of these modules, as shown in figure 1(b) . each ppn modifies the output of each module by adding or removing information as necessary to facilitate connections to subsequent modules, resulting in a better flow of the entire pipeline. to enable the appropriate post-processing for the entire system, each ppn uses the states of all modules in the system when executing post-processing. the post-processing of each ppn is optimized using rl so that the system can improve its dialogue performance, e.g., task success. a major advantage of our method is that each module does not need to be trainable since ppns are trained instead. to evaluate the effectiveness of our method, we applied ppns to pipeline systems consisting of modules implemented with various methods (e.g., rule-based and neural-based) on the basis of the multiwoz dataset @xcite and conducted experiments by using dialogue simulation and human participants. the contributions of this study are as follows. our study is related to optimizing an entire dialogue system with a modular architecture. @xcite proposed a method for implementing all the functions of nlu, dst, policy, and nlg modules by using neural networks, enabling the entire system to be trained. @xcite incorporated both a decoder for generating belief states (i.e., dst module) and a response-generation decoder (i.e., nlg module) into a sequence-tosequence model @xcite . zhang et al. (2020a) also proposed a method for jointly optimizing a system that includes three decoders that respectively execute the functions of dst, policy, and nlg. @xcite extended the method of @xcite by jointly optimizing four decoders that generate user dialogue acts (das), belief states, system das, and system responses. however, these systems are trained in a supervised manner and require large amounts of data @xcite . our study is related to improving the dialogue performance of a pipeline system by using rl. @xcite and @xcite implemented dst and policy in a neural model and used the deep q network @xcite algorithm to optimize the system to achieve robustness against errors that occur in interactions. @xcite proposed a policy-learning optimization method for real users by combining supervised learning, imitation learning, and rl. @xcite proposed a method for training a response-generation model by using rl while using the hidden states of the learned nlu, policy, and nlg. methods have been proposed @xcite for building a pipeline system with individually trained modules and finetuning specific modules by using rl, which significantly improved the performance of the overall system. these methods are powerful because they can fine-tune a system directly through rl. however, they can only be applied to systems consisting of specific differentiable modules implemented using neural-based methods, not to systems consisting of non-differentiable modules. our method is independent of the module-implementation method, trainability of each module in pipeline systems, and combination of modules. ## proposed method we developed our method to improve the dialogue performance of an entire pipeline system by optimizing the output of each module through postprocessing. post-processing means modifying the output by adding or removing information from the actual output of the module. with our method, inadapter output vocabulary ϭ ŝŷĩžƌŵͳśžƚğůͳăƌğă ϯ ŝŷĩžƌŵͳśžƚğůͳƚǉɖğ ϯ ŝŷĩžƌŵͳśžƚğůͳɖƌŝđğ ϰ ƌğƌƶğɛƚͳśžƚğůͳɖƌŝđğ ⁝ ⁝ ďǉğͳőğŷğƌăů ϭ ϭ ϭ ϭ ⁝ ϭ output vector copy original values intent value ŝŷĩžƌŵͳśžƚğůͳăƌğă ğăɛƚ ŝŷĩžƌŵͳśžƚğůͳɖƌŝđğ đśğăɖ mlp vector outadapter output vocabulary ϭ ŝŷĩžƌŵͳśžƚğůͳăƌğă ϯ ŝŷĩžƌŵͳśžƚğůͳƚǉɖğ ϯ ŝŷĩžƌŵͳśžƚğůͳɖƌŝđğ ϰ ƌğƌƶğɛƚͳśžƚğůͳɖƌŝđğ ⁝ ⁝ ďǉğͳőğŷğƌăů output intent value ŝŷĩžƌŵͳśžƚğůͳăƌğă ğăɛƚ ƌğƌƶğɛƚͳśžƚğůͳɖƌŝđğ ϭ ϭ ϭ ϭ ⁝ ϭ figure 3: procedure in which inadapter converts output label o into vector v and outadapter restores vector v ′ into output label o ′ by using output vocabulary (in this case, output labels are das of nlu). value information, which cannot be encoded in v, is copied directly from o when creating o ′ . ## experiments to confirm the effectiveness of our method, we applied ppns to several different pipeline systems and evaluated dialogue performance using dialogue simulation. we also carried out a human evaluation. ## conclusions and future work we proposed a method for optimizing pipeline dialogue systems with post-processing networks (ppns). through dialogue simulation and human evaluation experiments on the multiwoz dataset, we showed that the proposed method is effective for a pipeline system consisting of modules with various models. for future work, we plan to design more sophisticated rewards in rl such as module-specific rewards. we also plan to extend ppns to handle natural language generation by implementing them using transformer-based models. we are also considering to apply ppns to modules dealing with speech recognition and multi-modal processing.