Graph Neural Network Policies and Imitation Learning for Multi-Domain Task-Oriented Dialogues (2022)

Abstract: Task-oriented dialogue systems are designed to achieve specific goals while conversing with humans. In practice, they may have to handle simultaneously several domains and tasks. The dialogue manager must therefore be able to take into account domain changes and plan over different domains/tasks in order to deal with multi-domain dialogues. However, learning with reinforcement in such context becomes difficult because the state-action dimension is larger while the reward signal remains scarce. Our experimental results suggest that structured policies based on graph neural networks combined with different degrees of imitation learning can effectively handle multi-domain dialogues. The reported experiments underline the benefit of structured policies over standard policies.

## related work fundamental hierarchical reinforcement learning @xcite @xcite has inspired a previous string of works on dialogue management @xcite @xcite . recently, the use of structured hierarchy with gnn @xcite rather than a set of classical feedforward networks (fnn) enables the learning of non-independent sub-policies @xcite 2020a). these works adopted the domain independent parametrisation (dip) that standardizes the slots representation into a common feature space to eliminate the domain dependence. it allows policies to deal with different slots in the same way. it is therefore possible to build policies that handle a variable number of slots and that transfer to different domains on similar tasks @xcite . our contribution differs from chen et al. (2020b) on three points: first we perform our experiments on convlab @xcite which is a dedicated multi-domain framework; second, the dialogue state tracker (dst) output is not discarded when activating the domain; third, we adapt the gnn structure to each domain by keeping the relevant nodes while sharing the edge's weights. the reward sparsity can be bypassed by guiding the learning through the injection of some knowledge via a teacher policy. this approach, called imitation learning (il) @xcite , can be declined from pure behaviour cloning (bc) where the agent only learns to mimic its teacher to pure reinforcement learning (rl) where no hint is provided @xcite @xcite . ## extended gnn policies with imitation we adopt the multi-task setting as presented in convlab, in which a single dialogue can have the following tasks: (i) find, in which the system requests information in order to query a database and make an offer; (ii) book, in which the system requests information in order to book the item. a single dialogue can also contain multiple domains such as hotel, restaurant, attraction, train, etc. our method, illustrated in figure 1 , is designed to adapt: (i) at the domain-level (i.e. be scalable to changes in the number of slots), and (ii) at the multi-domain-level (i.e. be scalable to changes of domain). for each dialogue turn, it works as follow: first, the dst module chooses which domain to activate. then, the multi-domain belief state (and action space) is projected into the active domain (i.e only the dip nodes corresponding to the active domain are kept) as shown in figure 1a . afterwards, we apply the gnn message passing as @xcite but only among the domain specific dip nodes in the decision making module (figure 1b ). ## experiments we performed an ablation study: (i) by progressively extending the baseline to our proposed gnns and (ii) by guiding the exploration with il. all the experiments were restarted 10 times with random initialisations and the results evaluated on 500 dialogues were averaged. each learning trajectory was kept up to 10,000 dialogues with a step of 1,000 dialogues in order to analyse the variability and stability of the methods. ## evaluation we evaluate the dialogue manager and the dialogue system both with simulated users. ## conclusion we studied structured policies like gnn combined with some imitation learning that effectively handle multi-domain dialogues. the results of our largescale experiments on convlab confirm that an actor-critic based policy with a gnn structure can solve multi-domain multi-task dialogue problems. finally, we evaluated our best policy (acgos) in a complete dialogue system with simulated users. it overcomes the baselines and it is comparable to the handcrafted policy. a limitation of current policies in convlab, including ours, is that the robustness to noisy inputs is not specifically addressed as it had been done in pydial @xcite . it could be also interesting to study the impact of incorporating real human feed-backs and demonstrations instead of a handcrafted teacher. the gnn structured policies combined with imitation learning avoid sparsity, while being data efficient, stable and adaptable. they are relevant for covering multi-domain task dialogue problems.