Few-Shot Structured Policy Learning for Multi-Domain and Multi-Task Dialogues (2023)

Abstract: Reinforcement learning has been widely adopted to model dialogue managers in task-oriented dialogues. However, the user simulator provided by state-of-the-art dialogue frameworks are only rough approximations of human behaviour. The ability to learn from a small number of human interactions is hence crucial, especially on multi-domain and multi-task environments where the action space is large. We therefore propose to use structured policies to improve sample efficiency when learning on these kinds of environments. We also evaluate the impact of learning from human vs simulated experts. Among the different levels of structure that we tested, the graph neural networks (GNNs) show a remarkable superiority by reaching a success rate above 80% with only 50 dialogues when learning from simulated experts. They also show superiority when learning from human experts, although a performance drop was observed. We therefore suggest to concentrate future research efforts on bridging the gap between human data, simulators and automatic evaluators in dialogue frameworks.

## introduction multi-domain multi-task dialogue systems are designed to complete specific tasks in distinct domains such as finding and booking a hotel or a restaurant @xcite . a domain is formally defined as a list of slots with their valid values. the most common task, the information-seeking task, is usually modelled as a slot-filling data-query problem in which the system requests constraints to the user and proposes items that fulfil those constraints. the design of a dialogue manager (dms) is costly: hand-crafted policies require a lot of engineering, pure supervised learning (or behaviour cloning) requires a lot of expert demonstrations, and pure reinforcement learning requires a lot of user interactions to converge. the simulators provided with frameworks, such as pydial @xcite or convlab @xcite , are only rough approximations of human behaviour and the ability to learn from a small number of human interactions remains crucial. this is especially true on multi-domain and multi-task environments where the action space is large @xcite . a popular approach to reduce these costs is to wire some knowledge about the problem into the policy model, namely: few shot learning @xcite . in particular, structured policies like graph neural networks (gnns) are known to be well suited to handle a variable number of slots and domains for the information-seeking task @xcite . in this paper, we explore structured policies based on gnn. a graph in a gnn is fully connected and directed. each node represents a sub-policy associated with a slot, while a directed edge between two nodes represents a message passing. for studying sample efficiency, we analyse the dialogue success rate of structured policies once trained in a supervised way from expert demonstrations. we consider two types of demonstrations: human experts extracted from the multiwoz dataset @xcite , and simulated experts generated by letting the convlab's handcrafted policy interact with a simulated user. we perform large scale experiments. we study the impact of different levels of structure (see them in figure 2 ) on policy success rate after a limited number of dialogue demonstrations. for each level of structure, we also compare two sources of demonstrations: simulated and human dialogues. we show a notable result: our structured policies are able to reach a success rate above 80% with only 50 when following a simulated expert in con-vlab. to the best of our knowledge there are not previous works that studied the impact of structure for dialogue policy in a few-shot setting. another important finding is that few-shot learning from human demonstrations is harder, producing a lower success rate. this can be explained first by the large variability of human strategies that is not covered by simulated users which stick to more repetitive -easy to learn -dialogue patterns. another explanation could be an evaluation bias, simulated dialogues are more in line with artificial evaluators. the remainder of this paper is structured as follows. we present the related work in section 2. section 3 presents the proposed gnns from demonstrations. the experiments and evaluation are described in sections 4 and 5 respectively. finally, we conclude in section 6. ## related work few shot learning takes advantage of prior knowledge to avoid overloading the empirical risk minimiser when the number of available examples is small. in particular, prior knowledge can be used to constrain hypothesis space (i.e. model parameters) with parameter sharing or tying in order to reduce reliance on data acquisition and on data annotation @xcite . prior knowledge can be built into dialogue systems by imposing a structure in the neural network architecture. a first approach is to use hierarchical reinforcement learning that divides a main problem into several simpler sub-problems. we refer to @xcite that introduces semi-markov decision process using temporal abstraction and to @xcite that introduces sub-markov decision process using state partition. in the scope of the paper, a hierarchical policy corresponds to a metacontroller that chooses to activate a domain and we have one sub-policy per domain @xcite @xcite . in the same vein, graph neural networks (gnns) have been explored in a wide range of domains because of their empirical success and their theoretical properties which explains its efficiency: the abilities of generalisation, stability and expressiveness @xcite . gnns are suitable for applications where the data have a graph structure i.e where the graph outputs are supposed to be permutation-invariant or equivariant to the input features @xcite . in single-domain dialogue environments, this architecture has been adapted to model the dm in @xcite and @xcite . they have shown that gnns generalise between similar dialogue slots, manage a variable number of slots and transfer to different domains that perform similar tasks. we thus adopt in this work the domain independent parametrisation (dip) @xcite , which standardises the slots representation into a common feature space. in this work, as in @xcite and @xcite , we propose to improve multi-domain covering by learning a generic policy based on gnn. but unlike them, (i) we use a multi-domain multi-task setting, in which several domains and tasks can be evoked in a dialogue; (ii) the dialogue state tracker (dst) output is not discarded when activating the domain; and (iii) we adapt the gnn structure to each domain by keeping the relevant nodes while sharing the edge's weights. ## structured policies with expert demonstrations in order to investigate the impact of structured policies with behaviour cloning in improving sample efficiency in multi-domain multi-task dialogue environments, we introduce the dialogue state and action spaces for structured policies and we present the different policies and the experts' nature. ## experiments in this section we explain the experimental setup, the proposed models and the evaluation metrics. ## evaluation first, we evaluate the dialogue manager performance when talking to a simulated user. second, we evaluate the learned policies within the entire dialogue system both with simulated and with real users. the evaluations have been done within con-vlab. ## conclusion we investigated in this work the impact of policy structure and experts on success rate in few-shot learning for multi-domain multi-task dialogues. promising results were obtained: hierarchical and generic gnn policies are able to achieve remarkable performance with few dialogues and few training iterations when following a simulated expert. this confirms the growing interest for these neural structures. we also present an important finding: the policy performance degrades in few-shot learning when using human demonstrations. this fact questions the alignment between dialogue evaluators and human strategies in state-of-the-art dialogue frameworks.