C opy T 5: Copy Mechanism and Post-Trained T5 for Speech-Aware Dialogue State Tracking System (2023)

Abstract: In a real-world environment, Dialogue State Tracking (DST) should use speech recognition results to perform tasks. However, most existing DST research has been conducted in text-based environments. This study aims to build a model that efficiently performs Automatic Speech Recognition-based DST. To operate robustly against speech noise, we used CopyT5, which adopted a copy mechanism, and trained the model using augmented data including speech noise. Furthermore, CopyT5 performed post-training using the masked language modeling method with the MultiWOZ dataset in T5 in order to learn the dialogue context better. The copy mechanism also mitigated name entity errors that may occur during DST generation. Experiments confirmed that data augmentation, post-training, and the copy mechanism effectively improve DST performance.

## introduction task-oriented dialogue systems are used in various fields and are intimately connected to our daily lives. they can help users perform tasks that are frequently encountered in daily life, such as restaurant reservations and train ticket reservations. however, as most of these dialogue systems are implemented using a text-based dialogue corpus, they are very weak when implemented in actual speech interfacebased dialogue systems. in actual human speech, automatic speech recognition (asr) data contains errors generated because of similar words, inaccurate pronunciation, and noise in the environment being considered, making it difficult to use it directly for learning conversation system models. various attempts have been proposed to implement a robust speech interface-based dialogue system, such as @xcite and @xcite . in track 3 of the dstc11 summer track, we aim to secure a dialogue system in consideration of speech recognition environments that include speech noise and paraphrasing. accordingly, we propose a dialogue system based on voice audio and evaluate our model based on a given dataset. we adopt simpletod (hosseini-asl et al., 2020) in an end-to-end manner, which is robust in noisylabeled annotation among task-oriented dialogue systems. to secure a dialogue corpus of various expressions, @xcite and laug @xcite toolkit were applied to augment the text corpus, and post-training @xcite was performed to better understand the secured text data. furthermore, in order to improve the generation error occurring in the normal dst model, we propose a dialogue system that applies an effective copy mechanism @xcite to out-ofvocabulary resolution and applies certain efficient post-processing techniques. ## conclusion in this study, dst was performed using the copyt5 model to which the copy mechanism was applied. the copy mechanism contributed to performance improvement by suitably extracting object names that appeared in the conversation. in addition, post-training helped copyt5 improve performance in the dst fine-tuning step by prelearning conversational domain data. however, in the large-sized copyt5 model, the copy mechanism and post-training were not effective, but the performance improved as a result of applying data augmentation together.