A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots (2022)

Abstract: A slot value might be provided segment by segment over multiple-turn interactions in a dialog, especially for some important information such as phone numbers and names. It is a common phenomenon in daily life, but little attention has been paid to it in previous work. To fill the gap, this paper defines a new task named Sub-Slot based Task-Oriented Dialog (SSTOD) and builds a Chinese dialog dataset SSD for boosting research on SSTOD. The dataset includes a total of 40K dialogs and 500K utterances from four different domains: Chinese names, phone numbers, ID numbers and license plate numbers. The data is well annotated with sub-slot values, slot values, dialog states and actions. We find some new linguistic phenomena and interactive manners in SSTOD which raise critical challenges of building dialog agents for the task. We test three state-of-the-art dialog models on SSTOD and find they cannot handle the task well on any of the four domains. We also investigate an improved model by involving slot knowledge in a plug-in manner. More work should be done to meet the new challenges raised from SSTOD which widely exists in real-life applications. The dataset and code are publicly available via https://github.com/shunjiu/SSTOD .

## introduction task-oriented dialogs help users accomplish specific tasks such as booking restaurants or accessing technical support services by acquiring task-related slots through multi-turn dialogs. many advances have been achieved under an assumption that each slot value is informed or updated as a whole in a single turn by default @xcite @xcite ,. but in real-world dialogs, some slot values are often provided in a much more complicated manner. we take phone numbers as an example. users tend to inform an agent a sequence of 0-9 digits segment by segment across several turns as exemplified in figure 1 . accordingly, the agent needs to confirm, update or record the recognized sub-slot values. we regard these scenarios as sstod task. the sstod is very common when people communicate telephone numbers, names and so on. specifically, as shown in figure 1 , the sstod task raises several critical new challenges which have not been tackled in building dialog agents: (1) multi-segment informing: the segments could be informed in many different complex ways. as exampled in figure 1 , the user informed two subslots "136" and "361555" sequentially. the agent should discriminate whether the snippet "36" in "361555" is a partial repeat of "136" or a duplicated component of a whole slot value. (2) sub-slot locating: differing from updating a whole slot value in traditional slot filling, in sstod, the agent demands to precisely locate the part of values that needs to be updated. the situation is exacerbated when there are more than one similar sub-slots. to the best of our knowledge, the existing dialog benchmarks, such as atis @xcite , multiwoz @xcite , cross-woz @xcite , and sgd @xcite do not contain the dialogs illustrated in figure 1 , which makes the dialog agents optimized on them fail dramatically at conversing in sub-slot dialogs. to address the above challenges, we develop the sub-slot dialog (ssd) dataset which contains most popular sub-slot dialog scenarios including phone numbers (a sequence of digits 0-9), id numbers (much longer digit sequence), person names (a sequence of chinese characters), and license plate numbers (a mix of chinese characters, digits and english letters). the dataset is originated from the real-world human-to-human conversations, then richly labeled and reprocessed by crowdsourcing. although the dataset is in chinese, the development methodology depicted in this work is also applicable to other languages. under the setting of sstod, we present an improved model, ubar + , on the basis of ubar @xcite and the large pretrained model gpt2 @xcite . ubar + equips ubar with a knowledge prediction module to correct automatic speech recognition (asr) errors and discriminate the ambiguities, and achieves better performance on ssd. we also provide a rule-based user simulator to evaluate the system. our main contributions are: ## task and dataset we first give a defination of sstod, then introduce how to build the ssd dataset, and give some analyses on the dataset. ## experiments using the ssd dataset as a dialog state tracking benchmark, we conduct a comprehensive analysis of the challenges through an empirical approach and validate the effectiveness of the proposed ubar + method. ## related work we can group the datasets for task-oriented dialog systems by whether the two parts involved in the dialogs are humans or machines: human-to-human (h2h), machine-to-machine (m2m) and human-tomachine (h2m) collecting methods. h2h corpora are derived by asking a human user to talk with a human agent. to mimic the conversations between human and machine, h2h datasets ubiquitously apply the wizard-of-oz approach @xcite @xcite , which a human agent pretends as machine to talk to a human user and the human user believes the other side is a machine. however, it costs tremendous effort to construct such a h2h dataset. m2m datasets which are generated by simulated systems and simulated users take much less work to construct than h2h datasets with the same scale. however, the naturalness and diversity of m2m datasets are questioned @xcite @xcite . h2m @xcite @xcite hires crowd workers to chat with a machine system and the conversations are more diverse and natural than m2m. we integrate the m2m and h2m approaches by boosting the generated m2m datasets through crowdsource rewriting to obtain more diverse and natural dialogs with less effort. the datasets might be also grouped by the single-domain and the multi-domain. the early datasets are mostly single-domain. for example, atis @xcite , by m2m strategy, is a system to help people make air travel plans; a h2m corpus, let's go public @xcite , contains consultation dialogs of bus schedule information; two datasets for buying a movie ticket and reserving a restaurant table are collected by m2m @xcite . single-domain systems generally fill slots within a single turn and thereby slot values are relatively independent. recently, multi-domain datasets grab more attention. multi-woz @xcite , one of the most popular datasets, consists of wizard-of-oz largescale multi-domain conversations. a m2m dataset, sgd @xcite , generates multi-domain dialogs, guided by the predefined schema. cross-woz @xcite states how slots in one domain relate to the following domains by reference. nevertheless, none of the above datasets, with single domain or multiple domains, look into sub-slot cases as ssd does. in sstod, we have to not only locate the related previous sub-slots through complicated expressions, but also tile the pieces of value into a correct sequence without duplication, missing, and errors under the assistance of external knowledge. ## conclusions and future work in this paper, we propose a sub-slot based task sstod which has not brought to the public. to help the exploration of the task, we build a textual dialog dataset ssd which covers four popular domains and contains natural noise brought by asr module. ssd stems from the real human-to-human dialogs and can be utilized as a benchmark for slot a knowledge char knowledge 黄 草头黄 ('黄' has the radical '⺾'.) 林 双木林 ('林' has double '木'.) 王 三横一竖王 (three horizontal bars and one verscal bar 'wang'.) 赢 亡口月贝凡 ('赢' is composed of '亡', '口', '月', '贝' and '凡'.) 赛 下面一个贝的赛 ('赛' has the component '贝') 怡 竖心旁加一个台湾的台的那个怡 ('怡' is a combinason of the radical '忄' and '台' from taiwan.) figure 6: some different types of knowledge on chinese characters. char knowledge explanason 宝 宝宝 baby 宝贵 precious 宝马 bmw 淘宝 taobao 宝石 gemstone 宝藏 treasure 珠宝 jewelry 宝玉 precious jade 宝物 gems 宝箱 treasure chest 支付宝 alipay 宝盖头 chinese radical '⼧' 小宝贝儿 li]le baby 上面一个宝盖头，下面一个玉字 '⼧' above, '玉' below 宝字盖加个玉 '⼧' and '玉' 宝盖下面一个玉的宝 '玉' under '⼧' figure 6 shows some different types of knowledge. the word "黄" is described with its radical. and it is necessary to use whole components to explain "林", "王" and "赢" . in the fourth row, only a part of the word "赛", "贝", is enough to disambiguate homonyms. in the last example, "台" also needs explanation besides "怡". overall, the knowledge description is challenging for systems to get the correct char. some pieces of knowledge about "宝" are shown in figure 7 . it contains phrase-based knowledge, structure-based knowledge and hybrid knowledge. the way to explain one character is various and the number of one character's knowledge is large.