Speech-Aware Multi-Domain Dialogue State Generation with ASR Error Correction Modules (2023)

Abstract: Prior research on dialogue state tracking (DST) is mostly based on written dialogue corpora. For spoken dialogues, the DST model trained on the written text should use the results (or hypothesis) of automatic speech recognition (ASR) as input. But ASR hypothesis often includes errors, which leads to significant performance drop for spoken dialogue state tracking. We address the issue by developing the following ASR error correction modules. First, we train a model to convert ASR hypothesis to ground truth user utterance, which can fix frequent patterns of errors. The model takes ASR hypotheses of two ASR models as input and fine-tuned in two stages. The corrected hypothesis is fed into a large scale pre-trained encoder-decoder model (T5) for DST training and inference. Second, if an output slot value from the encoder-decoder model is a name, we compare it with names in a dictionary crawled from Web sites and, if feasible, replace with the crawled name of the shortest edit distance. Third, we fix errors of temporal expressions in ASR hypothesis by using hand-crafted rules. Experiment results on the DSTC 11 speech-aware dataset, which is built on the popular MultiWOZ task (version 2.1), show that our proposed method can effectively mitigate the performance drop when moving from written text to spoken conversations.

## introduction conversational agents are evolving rapidly in the last decade and providing solutions for almost every industry including e-commerce, travel, finance, food & beverage, hospitality, gaming and healthcare. the agents that can perform certain tasks or provide transactional services such as restaurant booking and hotel booking need to understand user's intent and track the dialogue states throughout the conversation, known as dst task. the performance of the agents very much depends on the dst results for a fluent and accurate task com-pletion. with the advancement in ai and machine learning, especially the breakthroughs in natural language processing such as attention, sequenceto-sequence model and the transformer architecture, the research on dst has attracted much attention in the past few years. dst is typically formulated as the problem of estimating user's goal in the form of a list of slot-value pairs when a dialogue progresses with multiple turns. when multiple domains are involved in the conversation (e.g. multiwoz datasets @xcite ), the task becomes more challenging because the developed dst model needs to deal with multiple domains and their slot values in every dialogue turn. different dst models have been developed. the methodologies can be broadly classified into two types. one is classification based method @xcite @xcite and another one is generation based method @xcite . however, both types of methods are mostly developed for written dialogues. these models do not perform well on spoken dialogues, because the audio form of user utterances in spoken dialogues should be transcribed by an automatic speech recognition (asr) system, and the transcription involves asr errors. consider a multi-domain spoken dialogue with both ground truths and asr hypotheses of user utterances below. the ground truth user utterances are associated with, if any, their dialogue states that are affected by asr errors observed in the corresponding asr hypotheses. for instance, digits in the ground truth (e.g. "7", "2") are converted into english words in the corresponding asr hypothesis (e.g. "seven", "two"). a name (e.g. "rosepine", "naturita") is incorrectly split into two words (e.g. "rose pine") and incorrectly transcribed (e.g. "natur") in some asr hypotheses. the word "pm" is incorrectly transcribed into two words "p m". an example is given below with both ground truth and asr hypothesis from the user side. user (ground-truth) : i am looking to get some information on the resort at summerlin. user (asr hypothesis) : i am looking to get some information on the resort at summerlin system : the resort at summerlin is in the centre of town and in the expensive price range. user (ground-truth) : great can you get me a room for 7 people for 2 nights starting saturday? (hotel-@xmath0, hotel-@xmath1: great can you get me a room for seven people for two nights starting saturday system : yes, i can! you're booked with reference number i34y32y9. is there anything else i can help you with user (ground-truth) : yes, i also need to find a train going to rosepine. (train-@xmath2: yes i also need to find a train going to rose pine system : okay, i have quite a few. can you give me some more information? user (ground-truth) : i'm departing from naturita on saturday. i want to arrive by 12:58 pm. (train-@xmath3, train-@xmath4:58 pm") user (asr hypothesis) : i'm departing from natur on saturday i want to arrive by 12:58 p m system : i would recommend train id tr5049 which leaves naturita at 11:46 am and arrives in rosepine at 12:36 pm. there are earlier options as well if you wish. user (ground-truth) : that would be perfect. for 7 people. (train-@xmath5: that would be perfect for seven people in this work, we address the issue that errors in the asr hypotheses of spoken dialogues affect dialogue states, by learning to fix them before passing the asr hypotheses as input to a dst model and also by fixing them after the dst model generates slot values. prior work have investigate the possibility to correct errors from asr engines based on edit alignment @xcite , non-auto-regressive generation @xcite and auto-regressive sequence-to-sequence models @xcite . here, we investigate the possibility to merge the multi-source asr predictions using a pre-trained sequence-to-sequence model as shown in figure 1 . we experiment with dstc 11 speech-aware dataset. results and ablation studies show that our proposed approach can effectively mitigate the noise introduced by asr hypothesis and performance drop when moving from written text to spoken conversations. ## methodology our model has three parts: pre-processing, encoderdecoder, and post-processing. in the stage of preprocessing, we in particular fix asr errors in asr hypothesis that is given as input to the encoderdecoder model, which generates dialogues states as output. in the stage of post-processing, we further try to fix remaining errors in the generated dialog states. ## conclusions we present our submissions to the speech aware dialog systems technology challenge of dstc 11. our works are based on pre-trained encoderdecoder language models. we enhance their performance by integrating asr error correction, name correction and time correction modules. our asr error correction module utilizes two asr models synergistically. we crawled names from web sites and use them for the name correction module, instead of relying on the given training data. we manually wrote rules to fix errors of temporal expressions in asr hypothesis.