D ialog USR : Complex Dialogue Utterance Splitting and Reformulation for Multiple Intent Detection (2022)

Abstract: While interacting with chatbots, users may elicit multiple intents in a single dialogue utterance. Instead of training a dedicated multi-intent detection model, we propose DialogUSR, a dialogue utterance splitting and reformulation task that first splits multi-intent user query into several single-intent sub-queries and then recovers all the coreferred and omitted information in the sub-queries. DialogUSR can serve as a plug-in and domain-agnostic module that empowers the multi-intent detection for the deployed chatbots with minimal efforts. We collect a high-quality naturally occurring dataset that covers 23 domains with a multi-step crowd-souring procedure. To benchmark the proposed dataset, we propose multiple action-based generative models that involve end-to-end and two-stage training, and conduct in-depth analyses on the pros and cons of the proposed baselines.

## introduction thanks to the technological advances of natural language processing (nlp) in the last decade, modern personal virtual assistants like apple siri, amazon alexa have managed to interact with end users in a more natural and human-like way. taking chatbots as human listeners, users may elicit multiple intents within a single query. for example, in figure 1 , a single user query triggers the inquiries on both highspeed train ticket price and the weather of destination. to handle multi-intent user queries, a straightforward solution is to train a dedicated natural language understanding (nlu) system for multi-intent detection. @xcite first adopted hierarchical structures to identify multiple user intents. @xcite explored the joint multi-intent and slot-filling task with a recurrent neural network. @xcite figure 1 : the task illustration for dialogusr. it serves as a plug-in module that empowers multi-intent detection capability for deployed single-intent nlu systems. to provide an alternative way towards understanding multi-intent user queries, we propose complex dialogue utterance splitting and reformulation (dialogusr) task with corresponding benchmark dataset that firstly splits the multi-intent query into several single-intent sub-queries and then recover the coreferred and omitted information in the subqueries, as illustrated in fig 1 . with the proposed task and dataset, the practitioners can train a multiintent query rewriting model that serves as a plug-in module for the existing chatbot system with minimal efforts. the trained transformation models are also domain-agnostic in the sense that the learned query splitting and rewriting skills in dialogusr are generic for multi-intent complex user queries from diverse domains. we employ a multi-step crowdsourcing procedure to annotate the dataset for dialogusr which covers 23 domains with 11.6k instances. the naturally occurring coreferences and omissions account for 62.5% of the total human-written sub-queries, which conforms to the genuine user preferences. specifically we first collect initial queries from 2 chinese task-oriented nlu datasets that cover real-world user-agent interactions, then ask the annotators to write the subsequent queries as they were sending multiple intents to the chatbots, finally we aggregate the human written sub-queries and provide completed sub-queries if coreferences and omissions are involved. we also employ multiple screening and post-checking protocols in the entire data creation process, in order to ensure the high quality of the proposed dataset. for baseline models, we carefully analyze the transformation from the input multi-intent queries to the corresponding single-intent sub-queries and summarize multiple rewriting actions, including deletion, splitting, completion and causal completion which are the local edits in the generation. based on the summarized actions, we proposed three types of generative baselines: end-toend, two-stage and causal two-stage models which are empowered by strong pretrained models, and conduct a series of empirical studies including the exploration on the best action combination, the model performance on different training data scale and existing multi-intent nlu datasets. we summarize our contributions as follows 1 : h%p :; chec [sp] from h%p :; z[ chec [sp] there ^"_ abs abs step1. initial query collection +,-./012345678 check the high-speed train from xiamen to nanjing on friday afternoon task-oriented query datasets fghi:+,a/j12klh -./0345678 hi, i wanna check the high-speed train that departs from xiamen and arrives in nanjing on friday afternoon sampling sentence simplification step2. follow-up query creation '(* +,-./012345678 check the high-speed train from xiamen to nanjing on friday afternoon figure 2 : the overview for the data collection procedure of dialogusr. firstly we sample initial queries from task-oriented nlu datasets (sec. 2.1), then we hire crowdsource workers to write follow-up queries (sec. 2.2). to aggregate the annotated queries, we propose text filler templates (marked in red, sec. 2.3) and post-processing procedure. finally we ask annotators to recover the missing information in the incomplete utterances (marked in blue, sec. 2.4). models are domain-agnostic in the sense that the learned query splitting, coreference/omission recovery skills are generic for distinct domains 3) presumably mid is more difficult than single intent detection (sid) given the same intent/slot ontology. from the perspective of task (re)formulation, dialogusr is the first to convert a mid task to multiple sid tasks (the philosophy of 'divide and conquer') with a relatively low error propagation rate, providing an alternative and effective way to handle the mid task. ## dataset creation we collect a high quality dataset via a 4-step crowdsourcing procedure as illustrated in fig 2 . ## dataset analysis dataset statistics in total, after accumulating annotations for several rounds, we obtain 11,669 instances. we conduct 6 rounds of annotation, increasing the annotation scale with each round (ranging from ∼100 instances/round to ∼4000 instances/round). on average, an aggregated multiintent complex query from the proposed dialo-gusr dataset comprises 36.7 chinese characters by assembling 3.6 single-intent queries (including initial and follow-up queries). after recovering missing information in the query completion phase (sec 2.4), the average lengths of completed initial query, first follow-up query, second follow-up query and third follow-up query are 11.9, 12.3, 12.4, 10.8 respectively. we split the dataset into train, validation and test sets with sizes of 10,169 , 500, 1,000 respectively. ## experiment settings model setting we experiment with a variety of pretrained models via hugging face transformers @xcite , including mt5 @xcite with three different parameter scales, namely t5-base (580m), t5-large (1.2b), t5-xl (3.7b) , and mbart-large @xcite with 340m parameters as the backbones for the end-to-end and two-stage models. they are all multi-lingual pre-trained models that support both chinese and english dialogusr. we use the adam optimizer (kingma and ba, 2015) with the learning rate of 0.00003 and train the models for maximum 9 epochs on 4-8 a100 gpus. evaluation metrics viewing dialohusr as a sequence generation task, i.e. concatenating the segmented single-intent queries with semicolons like q4 in fig 5 , we use bleu-4 @xcite , meteor @xcite , rouge-l @xcite , which are three commonly used automatic evaluation metrics to measure the ngram similarity with the reference in the token level. we also propose two new sentence-level metrics, namely split accuracy (sacc) and exact match (em) to evaluate the model performance for dialogusr. specifically sacc measures the ratio of correct query splitting. we consider a multi-n n n n, upvwmx89y: !"#@xmath0%&'()*+,-. \z]^< 3=>?@ \z]^qr!4%bc,defg check the high-speed train from xiamen to nanjing on friday afternoon [sp] how long does the journey take [sp] then check out the special food there. _k[kmkx8ay: !"#@xmath1%&'()*+,-. \z]' ()*+,-.<3=>?@ \z]^!4%*+, defg check the high-speed train from xiamen to nanjing on friday afternoon [sp] how long does it take to travel from xiamen to nanjing in highspeed train [sp] check out the special cuisine in nanjing `lwcl[ `oav[kmk: zmkv9x8dye!"#@xmath2%&'()*+,-. check the high-speed train from xiamen to nanjing on friday @xmath5;x8hye!"#@xmath3%&'()*+,-. \z]^< 3=>?@ \z]^qr!4%bc,defg fg !4%*+,defg check the high-speed train from xiamen to nanjing on friday afternoon [sp] how long does the journey take [sp] then check out the special food @xmath6;jy: 89 → 8b mtolcmlik xnpoky: 89 → 8; → 8b mtolcmlik x`lcwl[y: 89 → 8; → \8d → 8h → 8if ## analysis and discussions baseline performance figure 7 : the demonstration of generated outputs for different baseline models. the query marked in red is wrong due to the missing destination of the train, while the query marked in blue is a paraphrase of the the reference. models, we observe that for mt5-base and mt5large, the causal-style two-stage model is the clear winner among the three models, which shows that the query-by-query transformation (@xmath7. for mt5-xl, the performance gap between two-stage and end-to-end baselines is largely reduced, indicating powerful trained models may close the gap between different baselines. we also report the model performance on the existing multi-intent detection datasets, namely mixs-nips and mixatis. as mentioned in sec 3, both of them are created by inserting specific conjunctions between two complete single-intent queries from the snips @xcite or atis @xcite datasets, without any coreference or omission phenomenon. in other words, both of them can be effectively solved with an endto-end model using the delete and split actions. the large performance gap of the same model on the mixatis/ mixsnips and the proposed dialo-gusr verifies that the multi-intent query splitting and reformulation task is far from solved. ## related work incomplete utterance restoration to convert multi-turn incomplete dialogue into multiple singleturn complete utterance, two major paradigms are available currently. one straight-forward way is to consider it as a sequence-to-sequence problem, using models including rnn @xcite , trans-pg+bert @xcite and t5 with importance token selection @xcite . and since the source and target utterances are highly overlapped, another approach is to edit rather than generate from scratch, specifying the operation by sequence tagging. pan et al. (2019 multi-intent detection spoken language understanding (slu) which consists of intent detection and slot filling is the core in spoken dialogue systems @xcite . intent detection mainly aims to classify a given utterance with its intents from user inputs. considering this strong correlation between the two tasks, some joint models are proposed based on the multi-task learning framework. @xcite @xcite @xcite . @xcite proposed the gate mechanism to explore incorporating the intent information for slot filling. convolutional-lstm and capsule network have been proposed to solve the problem @xcite . @xcite shows that 52% utterances are multi-intent in the amazon internal dataset which indicate that in real world scenario, however, users often input utterance containing multi-intent. therefore, @xcite first adopted hierarchical structures to identify multiple user intents. @xcite associate multi-intent detection with slots filling via graph attention network. larson and leach (2022) offers a thorough overview on the existing multi-intent detection datasets. except from mixatis and mixsnips datasets, top @xcite contains multiintent queries annotated in a hierarchical manner which dramatically improves the expressive power while dialogusr contains queries and rewriting queries which can bridge the single-intent dection and multi-intent detection and also decoupling the query intent detection section and multi-intent query separation section. nlu++ @xcite has been collected, filtered and carefully annotated by dialogue nlu experts while dialogusr queries are created by human annotators and aggregated by rules and evaluated by model which lead to a lower cost of data annotation than nlu++. ## conclusion we propose dialogusr, a dialog utterance splitting and reformulation task and corresponding dataset, for multi-intent detection in the conversational agents. the model trained on dialogusr can serve as a domain-agnostic and plug-in module for the existing product chatbots with minial efforts. the proposed dataset contains 11.6k high quality instances that cover 23 domains with a multi-step annotation process. we propose multiple action-based generative baselines to benchmark the dataset and analyze their pros and cons through a series of investigations.