A Sequence-to-sequence Approach for Numerical Slot-filling Dialog Systems (2020)

Abstract: Dialog systems capable of filling slots with numerical values have wide applicability to many task-oriented applications. In this paper, we perform a particular case study on the “number_of_guests” slot-filling in hotel reservation domain, and propose two methods to improve current dialog system model on 1. numerical reasoning performance by training the model to predict arithmetic expressions, and 2. multi-turn question generation by introducing additional context slots. Furthermore, because the proposed methods are all based on an end-to-end trainable sequence-to-sequence (seq2seq) neural model, it is possible to achieve further performance improvement on increasing dialog logs in the future.

## introduction task-oriented dialog systems which assist users to complete tasks like hotel reservation, are drawing great attentions among both research and industry. compared to conventional pipelined system, recently emerging end-to-end trainable dialog systems are showing many favorable characteristics -because of the neural models that directly learn from chatlogs of human-to-human conversation employed, such systems hold the promise of low data preparation cost, flexible response generation and the ability to evolve with new data. in this work, we are going to explore the possibility to bring the end-to-end trainable dialog system model to the hotel reservation chatbot application, where we encounter two new problems: 1. numerical slots-filling and 2. multi-turn dialog management. to the best of our knowledge, both of them can not be fully solved using currently available end-to-end systems. in this paper, we will focus on these two problems and propose possible workarounds which can lead to satisfactory results. ## problem description the hotel reservation application requires a dialog system to fill three slots with integernumber of adults (slot:num adult), number of primary school children (aged 6-12) (slot:num c6 12), and number of preschool children (aged 0-5) (slot:num c0 5). these numerical slots are necessary because the applicable room plan (number of beds, quantity of amenities) and the pricing (food cost etc.) vary on the number of adults and children. this numerical slotfilling problem is also widely applicable to other domains such as restaurant reservation or flight booking, with slightly different slot configurations. the challenges of building such dialog system mainly lie in two aspects. first challenge is the difficulty in the numerical slot value inference. unlike most task-oriented dialog systems or datasets such as @xcite ; @xcite , where the slot filling can be either solved as a named entity extraction problem or a multi-label classification problem, the numerical slot-filling requires additional reasoning and calculation. for examples the simple expression "my wife and me" means 2 adults, and "4 including 1 baby" implies 3 adults. and moreover, the numerical inference sometimes involves with multi-turn dialog context, which brings to the second challenge. second challenge is the multi-turn dialog management. many previous task-oriented dialog systems are designed in a turn-wise manner e.g. @xcite -the systems ask the question for particular slot in each turn and expect user to give explicit answer within that turn. if no exact slot value can be extracted from the response, the system will simply repeat the same question. this behavior is unfavorable for the numerical slot-filling, because of the likely ambiguity in the user responses. for example, no target slot value can be determined from the user response "4 people including 2 kids", while human agent may ask drill-down questions such as "how old are the children" to address this ambiguity. to achieve this human-level conversation, a dialog system capable of managing multiturn strategy such as asking drill-down questions, is desirable. ## present methods several end-to-end model architectures have been proposed for task-oriented dialog system. @xcite proposed a modularly connected neural networks to enable end-to-end training. later @xcite simplified this architecture to a single sequence-to-sequence model (sequicity), which not only reduced the training cost but also improved the performance. more recently more advanced model like hagan has been applied to end-to-end learning @xcite . @xcite also explored the possibility of applying recent large pre-trained language model such as bert and gpt-2 to the task-oriented dialog system. after survey and review on different models, we consider the squicity framework a particularly good point to start because of its simplicity and extendability. its key idea is to encode the dialog states (slot values) into a text format which can be concatenated to the target utterances, so that any seq2seq models can handle both slot-filling and language generation at the same time. in this way, the model complexity and the training procedure are greatly simplified (refer to original paper for more details). recently published t5 model @xcite also demonstrates the promising performance and the wide applicability of such textto-text format training. therefore we consider this sequicity framework using seq2seq model has great potential and lower maintenance cost for commercial applications, and in this paper we chose this framework as our base model. ## conclusion and future work in this paper, we proposed two methods for improving the original end-to-end dialog system on numerical slot-filling. by training the model to predict arithmetic expressions, the dialog system can perform numeric reasoning more robustly, and with newly included context slots, the dialog system is able to generate multi-turn questions for ambiguous user responses. future work may include extending the current seq2seq network to more recent large-scale pretrained models such as roberta, as suggested in @xcite , for a better performance in reasoning task. and also the proposed multi-turn dialog management approach should be extensively tested on other slots and domains.