This paper proposes a Markov Decision Process and reinforcement learning based approach for domain selection in a multidomain Spoken Dialogue System built on a distributed architecture . In the proposed framework , the domain selection problem is treated as sequential planning instead of classification , such that confirmation and clarification interaction mechanisms are supported . In addition , it is shown that by using a model parameter tying trick , the extensibility of the system can be preserved , where dialogue components in new domains can be easily plugged in , without re-training the domain selection policy . The experimental results based on human subjects suggest that the proposed model marginally outperforms a non-trivial baseline . Due to growing demand for natural humanmachine interaction , over the last decade Spoken Dialogue Systems ( SDS ) have been increasingly deployed in various commercial applications ranging from traditional call centre automation ( e.g. AT&T " Lets Go ! " bus information system ( Williams et al . , 2010 ) ) to mobile personal assistants and knowledge navigators ( e.g. Apple 's Siri R , Google Now TM , Microsoft Cortana , etc . ) or voice interaction for smart household appliance control ( e.g. Samsung Evolution Kit for Smart TVs ) . Furthermore , latest progress in openvocabulary Automatic Speech Recognition ( ASR ) is pushing SDS from traditional single-domain information systems towards more complex multidomain speech applications , of which typical examples are those voice assistant mobile applications . Recent advances in SDS have shown that statistical approaches to dialogue management can result in marginal improvement in both the naturalness and the task success rate for domainspecific dialogues ( Lemon and Pietquin , 2012 ; Young et al . , 2013 ) . State-of-the-art statistical SDS treat the dialogue problem as a sequential decision making process , and employ established planning models , such as Markov Decision Processes ( MDPs ) ( Singh et al . , 2002 ) or Partially Observable Markov Decision Processes ( POMDPs ) ( Thomson and Young , 2010 ; Young et al . , 2010 ; Williams and Young , 2007 ) , in conjunction with reinforcement learning techniques ( Jurčíček et al . , 2011 ; Jurčíček et al . , 2012 ; Gašić et al . , 2013a ) to seek optimal dialogue policies that maximise long-term expected ( discounted ) rewards and are robust to ASR errors . However , to the best of our knowledge , most of the existing multi-domain SDS in public use are rule-based ( e.g. ( Gruber et al . , 2012 ; Mirkovic and Cavedon , 2006 ) ) . The application of statistical models in multi-domain dialogue systems is still preliminary . Komatani et al . ( 2006 ) and Nakano et al . ( 2011 ) utilised a distributed architecture ( Lin et al . , 1999 ) to integrate expert dialogue systems in different domains into a unified framework , where a central controller trained as a data-driven classifier selects a domain expert at each turn to address user 's query . Alternatively , Hakkani-Tür et al . ( 2012 ) adopted the well-known Information State mechanism ( Traum and Larsson , 2003 ) to construct a multi-domain SDS and proposed a discriminative classification model for more accurate state updates . More recently , Gašić et al . ( 2013b ) proposed that by a simple expansion of the kernel function in Gaussian Process ( GP ) reinforcement learning ( Engel et al . , 2005 ; Gašić et al . , 2013a ) , one can adapt pre-trained dialogue policies to handle unseen slots for SDS in extended domains . In this paper , we use a voice assistant applica- Figure 1 : The distributed architecture of the voice assistant system ( a simplified illustration ) . tion ( similar to Apple 's Siri but in Chinese language ) as an example to demonstrate a novel MDP-based approach for central interaction management in a complex multi-domain dialogue system . The voice assistant employs a distributed architecture similar to ( Lin et al . , 1999 ; Komatani et al . , 2006 ; Nakano et al . , 2011 ) , and handles mixed interactions of multi-turn dialogues across different domains and single-turn queries powered by a collection of information access services ( such as web search , Question Answering ( QA ) , etc . ) . In our system , the dialogues in each domain are managed by an individual domain expert SDS , and the single-turn services are used to handle those so-called out-of-domain requests . We use featurised representations to summarise the current dialogue states in each domain ( see Section 3 for more details ) , and let the central controller ( the MDP model ) choose one of the following system actions at each turn : ( 1 ) addressing user 's query based on a domain expert , ( 2 ) treating it as an out-of-domain request , ( 3 ) asking user to confirm whether he / she wants to continue a domain expert 's dialogue or to switch to out-of-domain services , and ( 4 ) clarifying user 's intention between two domains . The Gaussian Process Temporal Difference ( GPTD ) algorithm ( Engel et al . , 2005 ; Gašić et al . , 2013a ) is adopted here for policy optimisation based on human subjects , where a parameter tying trick is applied to preserve the extensibility of the system , such that new domain experts ( dialogue systems ) can be flexibly plugged in without the need of re-training the central controller . Comparing to the previous classification-based methods ( Komatani et al . , 2006 ; Nakano et al . , 2011 ) , the proposed approach not only has the advantage of action selection in consideration of long-term rewards , it can also yield more robust policies that allow clarifications and confirmations to mitigate ASR and Spoken Language Understanding ( SLU ) errors . Our human evaluation results show that the proposed system with a trained MDP policy achieves significantly better naturalness in domain switching tasks than a non-trivial baseline with a hand-crafted policy . The remainder of this paper is organised as follows . Section 2 defines the terminology used throughout the paper . Section 3 briefly overviews the distributed architecture of our system . The MDP model and the policy optimisation algorithm are introduced in Section 4 and Section 5 , respectively . After this , experimental settings and evaluation results are described in Section 6 . Finally , we discuss some possible improvements in Section 7 and conclude ourselves in Section 8 . In this paper , we introduce an MDP framework for learning domain selection policies in a complex multi-domain SDS . Standard problem formulation is modified with tied model parameters , so that the entire system is extensible and new domain experts can be easily integrated without re-training the policy . This expectation is confirmed by empirical experiments with human subjects , where the proposed system marginally beats a non-trivial baseline and demonstrates proper extensibility . Several possible improvements are discussed , which will be the central arc of our future research . Thomas Robert Gruber , Adam John Cheyer , Dag

Challenge: Although existing works show that statistical approaches to dialogue management tasks outperform rule-based counterparts, they are not used in real world applications.
Approach: They propose to use Markov Decision Process and reinforcement learning for domain selection in multi-domain spoken dialogue systems to achieve extensibility in domains.
Outcome: The proposed system outperforms rule-based baseline systems with a hand-crafted policy on human-based evaluation of naturalness in domain switching tasks.