Adapting Text-based Dialogue State Tracker for Spoken Dialogues (2023)

Abstract: Although there have been remarkable advances in dialogue systems through the dialogue systems technology competition (DSTC), it remains one of the key challenges to building a robust task-oriented dialogue system with a speech interface. Most of the progress has been made for text-based dialogue systems since there are abundant datasets with written cor- pora while those with spoken dialogues are very scarce. However, as can be seen from voice assistant systems such as Siri and Alexa, it is of practical importance to transfer the success to spoken dialogues. In this paper, we describe our engineering effort in building a highly successful model that participated in the speech-aware dialogue systems technology challenge track in DSTC11. Our model consists of three major modules: (1) automatic speech recognition error correction to bridge the gap between the spoken and the text utterances, (2) text-based dialogue system (D3ST) for estimating the slots and values using slot descriptions, and (3) post-processing for recovering the error of the estimated slot value. Our experiments show that it is important to use an explicit automatic speech recognition error correction module, post-processing, and data augmentation to adapt a text-based dialogue state tracker for spoken dialogue corpora.

## introduction task-oriented dialogue (tod) systems aim to successfully perform various tasks, such as hotel reservations, train information retrieval, and food ordering, by interacting with users. in particular, with the popularity of voice assistant systems such as siri and alexa, it is important to be able to take spoken languages as inputs to tod systems. these tod systems with the speech interface is referred to as spoken dialogue systems. however, due to the differences in many aspects between text conversation and speech conversation, * these authors contributed equally. the existing tod systems for text conversation can fail to achieve good performance for speech conversation. to this end, several tasks such as atis @xcite and competitions (dstc2 foot_0 , dstc10 knowledge-grounded task-oriented dialogue modeling on spoken conversations track foot_1 ) were proposed. in the knowledge-grounded taskoriented dialogue modeling on spoken conversations track in dstc10, various methods were applied to spoken dialogue systems. @xcite introduced a method to reduce automatic speech recognition (asr) error by making predictions using multiple hypotheses of several asr models together. also, @xcite proposed a method to reduce the discrepancy between written conversation and spoken conversation by augmenting the dataset based on ontology and pattern or generating noisy addition of spoken conversation through a text-to-speech (tts)->asr pipeline. since only asr hypothesis data was provided in knowledge-grounded task-oriented dialogue modeling on spoken conversations track in dstc10, it was difficult to effectively deal with errors generated in asr, and most of the studies were focused on data augmentation to make up for the lack of data. on the other hand, in the speech-aware dialogue system technology track in dstc11, in addition to the asr hypothesis data, audio, transcripts used to generate the audio and augmented data were provided, allowing a more diverse approach. for this motivation, we studied how to solve problems caused by the propagation of asr errors to the model and how to solve errors such as incorrect proper nouns generated by spoken input and participated in the speech-aware dialogue systems technology challenge track in dstc11. in this paper, we propose a model, which allows good performance for spoken utterance input as speaker utterance dialog state user i'd like to find out if there are any 4-star rated guesthouses in cambridge hotel-@xmath0. what area do you prefer? user i'm open to any area as long as there is free wifi. system then i recommend the a and b guest house. would you like me to book you a room? user does that also have free parking available? system no, it doesn't. should i recommend you a place with free parking instead? user no, but i am looking for a particular restaurant. its name is called bangkok city system bangkok city is an expensive thai restaurant in the centre of town. they are located at 24 green street city centre. their postcode is cb23jx. would you like a reservation? user all i needed today was the address, thank you ## speech-aware dialog systems task here we provide background on the speech-aware dialogue system technology track in dstc11, which was introduced to overcome the challenges in implementing a good speech dialogue system with the previous speech dialog system benchmarks. ## methods in this section, we describe our model that participated in the speech-aware dialogue systems technology challenge track in dstc11. figure 1 presents our model architecture. our system consists of three modules: (1) asr error correction, (2) text based dialogue system, and (3) postprocessing. the overview of the process followed by our model is as follows: asr error correction (t5) "[str] please get me a ticket for one that leaves at 12:49 a m a m and send me the reference number [end]" "[str] please get me a ticket for one that leaves at 0:49 am and send me the reference number. [end]" input corrected input "[str] please get me a ticket for one that leaves at 0:49 am and send me the reference number. [end]" transcript figure 2: asr correction example (multiwoz mul0207.json) ## experiments we trained two t5 @xcite ) models for asr error correction and the description dialogue system using huggingface. t5-base(220m) was used for asr error correction and t5-base, large(770m) for the description dialog system. we used pre-trained weights by @xcite and tokenized each sentence into sub-words using t5tokenizer foot_5 . for fine-tuning, we used the adamw optimizer @xcite with β 1 = 0.9, β 2 = 0.999, weight @xmath1.01 and a learning rate of 0.0001. we used a batch size of 8, an input length of 256, the output length of 256 for asr error correction, and batch size of 6 (3 for t5-large), an input length of 1024, and an output length of 512 for the description dialogue system. ## conclusion in this paper, we proposed an asr dialogue pipeline model to solve the speech-aware dialogue system technology track in dstc11. first, we reduce the gap when applying spoken data to textbased dialogue systems by modifying asr results that differ from the original text such as time formats or special characters through the asr error correction module. secondly, the text-based dialogue system, a model including descriptions with several techniques, was able to get good performance even with low input data feature quality of audio data. lastly, by recovering errors such as proper nouns that cannot be caught in asr error correction through post-processing, a dialogue system that produces high performance even on the speech database was created. in this way, we have created a model that can overcome the limitations of speech in several parts, and in the case of t5, although the model is very sensitive to parameter size, we can confirm that it is a model with high potential for development by performing well even in a small size model.