OLISIA : a Cascade System for Spoken Dialogue State Tracking (2023)

Abstract: Though Dialogue State Tracking (DST) is a core component of spoken dialogue systems, recent work on this task mostly deals with chat corpora, disregarding the discrepancies between spoken and written language. In this paper, we propose OLISIA, a cascade system which integrates an Automatic Speech Recognition (ASR) model and a DST model. We introduce several adaptations in the ASR and DST modules to improve integration and robustness to spoken conversations. With these adaptations, our system ranked first in DSTC11 Track 3, a benchmark to evaluate spoken DST. We conduct an in-depth analysis of the results and find that normalizing the ASR outputs and adapting the DST inputs through data augmentation, along with increasing the pre-trained models size all play an important role in reducing the performance discrepancy between written and spoken conversations.

## introduction a majority of recent research on task-oriented dialogue (tod) systems has focused on chat corpora such as multiwoz @xcite . with voice assistants becoming more prominent in our daily lives, there has been a renewed interest in spoken dialogue systems @xcite . however, state-of-the-art systems trained on chats face robustness issues when dealing with spoken inputs @xcite . in a tod system, the role of dst is to predict at each turn and based on the dialogue history the current belief state, i.e. a condensed and updated representation of the user needs. dst plays a central role as the system relies on the belief state to decide which action to take next. the belief state is typically frame-based and represented as a list @xmath0, @xmath1. while both cascade and end-to-end approaches have been well studied for spoken language understanding (slu, @xcite ), there has been little recent work on spoken dst. considering the entire dialogue context, as opposed to only the current turn, requires tricky strategies for end-to-end systems @xcite . in order to leverage state-of-the-art models, a cascade approach with separate asr and dst components is thus preferred. however, these two components do not benefit from joint optimization and often lack integration. to address these shortcomings, we propose olisia, a cascade system composed of an asr model and a dst model. olisia integrates these two components through adaptations in the asr outputs and the dst inputs. with this design, our system ranked first in the speech-aware dialog systems technology challenge (dstc11 track 3), foot_1 a benchmark to evaluate spoken dst models. in this paper, we describe our cascade spoken dst system along with the proposed adaptations. additionally, we conduct an analysis of these adaptations based on the results from various evaluation setups. our contributions can be summarized as follows. in the context of the speech-aware dialog systems technology challenge, we show • the need for post-processing the asr output in a pipeline; ## related work much of the research in task-oriented dialogue (tod) systems initially focused on spoken dialogue, for instance leveraging probabilistic modeling to account for the uncertainty associated with noisy utterances @xcite @xcite . the first editions of the dialogue state tracking challenge foot_2 (dstc1 & dstc2, williams et al. (2013); henderson et al. (2014)) introduced the first standardized benchmark for dst, releasing annotated spoken dialogue corpora. however, the focus of research in tod gradually shifted to chats, assuming that the upstream asr model would be able to provide accurate transcriptions. recently, there has been a renewed interest in spoken dialogue to address the lack of attention on the differences between spoken and textual inputs @xcite . dstc10 track 2 proposed a dst task on spoken conversations which stimulated work on this aspect, though only the n-best asr hypotheses were provided without audio data. when audio-only data is available, end-to-end spoken language understanding (slu) systems @xcite are often preferred because they benefit from joint optimization. although cascade approaches suffer from error propagation because the textual natural language understanding (nlu) model does not consider the uncertainty of the asr transcriptions, they remain competitive. in fact several adaptation techniques can boost the cascade's performance. performing hypothesis rescoring with a language model specifically trained on the targeted domain proves to be effective when data is available in high quality and large quantity @xcite . we rather adopt a post-processing approach such as spelling correction which can also help aligning the transcriptions with the targeted domain @xcite , especially when focusing on domainspecific words. with the recent advances in text-to-speech (tts) technologies, adapting asr models by finetuning them on synthetic speech of the target domain @xcite @xcite ) is now common. however, with only synthetic speech in the training set, such a finetuning might degrade the performances on human speech @xcite . simulating asr hidden representations from text in order to train an end-to-end slu model reaches higher named-entity recognition (ner) performances than training it on the synthesised speech @xcite . therefore, relying on the text nlu model to tolerate and correct some errors of the asr model seems more adapted in our setting. recent approaches have focused on data augmentation techniques to simulate spoken data @xcite @xcite in order to make the language understanding components of tod systems more robust to spoken inputs. other approaches have sought to leverage the multiple hypotheses provided by the upstream asr model in the hope that these different hypotheses complement each other to help language understanding @xcite @xcite . similarly, others used a more compact representation of these hypotheses, such as word confusion networks @xcite . ## speech-aware dialog systems technology challenge the lack of recent work on spoken dialogue can be attributed in part to the lack of available datasets. track 3 of the dialog systems technology challenge 11 foot_3 seeks to promote work on spoken dialogue by releasing a spoken version of multi-woz. this multi-domain (restaurant, hotel, attraction, taxi, train, hospital and police) wizard-of-oz dataset is a large-scale human-human task-oriented conversational corpus commonly used for training and evaluating dialogue state tracking (dst), policy optimization and end-to-end dialogue modeling systems. the goal of this track is to characterize the performance of dst models in the presence of asr errors and speech phenomena such as disfluencies. the organizers released a new version of multiwoz 2.1 (eric et al., 2019) with user utterances voiced by crowdworkers, as illustrated in figure 1. despite being widely used by the research community, multiwoz has been shown to exhibit an entity bias and a large overlap in the distribution of slot-values between the training and the evaluation sets which can lead to memorization in generative models @xcite . to encourage generalization, the organizers introduced modifications in the dev and test sets: the values for the slots hotel-name, restaurant-name, train-departure and train-destination were replaced with unseen entities, and time mentions were offset by a constant amount. user utterances in the dev and test sets are vocalized by crowdworkers. a speech synthesized version of the training data is also provided in the aim of assessing the validity of such data to mitigate the lack of real spoken conversations. two verbatim versions of the dev set are provided to the participants, i.e. user utterances are vocalized as is by a tts system (tts-v) and human crowdworkers (human-v). the test set includes the same setup along with a third version containing paraphrased user utterances vocalized by humans to sound more natural (human-p). 5 system submissions are evaluated using joint goal accuracy (jga) and slot error rate (ser), defined as follows: ## method in this section, we present our cascade approach with an asr component which converts the user spoken inputs into text and a dst component which predicts the current dialogue state from the transcript of the previous turns. the overall architecture of the system is shown in figure 2 on the asr side, given that the turns are perfectly segmented, we can easily transcribe the user's turns with open ai whisper @xcite transformer model with a forced english decoding. on the dst side, we use a generative dst model based on a pre-trained t5 model @xcite , which proved to be more robust to spoken inputs than an extractive model in preliminary experiments. the input to the dst model consists of the entire dialogue history at a given turn, with agent and user utterances separated by delimiter tokens. at each turn, the model outputs the current dialogue state from scratch in the form @xmath2, slot-@xmath3. formally, let u i and a i respectively be user and agent utterances at turn i. the input at turn t is linearized by concatenating the utterances (u 0 , a 1 , ..., a t -1 , u t ) and prepending the delimiters "user:" and "agent:". the output dialogue state is linearized as a semicolon-separated list of strings "slot-@xmath4. our contribution lies in the adaption of the transcriptions outputted from the asr for dst and the adaptation of the dst component to handle speech specificities, which are discussed in the next two ## results we propose an overview of the challenge's results in section 5.1 and further analysis of the impact of each adaptation on our system's performance in the following sections. ## limitations overall, the dataset released in this challenge is a good step towards bridging the gap between written and spoken dialogue systems. however, as the user utterances were read aloud by humans, this spoken data lacks in spontaneity associated with actual speech. it would be interesting to see if our findings hold on spontaneous spoken dialogue. one limitation of our system lies in the use of transformer-based models for both asr and dst. while these models provide attractive performances they come with their own limitation of quadratic memory and limited input size. dialogues in multiwoz are relatively short and this was not a concern in this work, but this could become problematic when dealing with longer, more realistic conversations. one alternative would be to reduce the input context to the most recent turns and include the linearized previous dialogue state in the dst input. another concern is the use of large pre-trained models to achieve competitive results. as pointed out before @xcite , training these large models on abundant data requires a substantial electrical consumption. in light of the human impact on the environment, we should promote efficiency as the main performance factor rather than metric scores. ## conclusion and future work this work introduced olisia, a cascade system for spoken dst that integrates an asr and a dst model through several adaptations. we used asr normalization and dst data augmentation to adjust each component to its counterpart. we have shown the importance of these adaptations in improving robustness to spoken inputs and our system achieved first place in the speech-aware dialog systems technology challenge. 14 post-evaluation experiments while having user turns as speech and agent turns as text is a natural setup for a spoken dialogue system, this mix of modalities makes it more challenging to develop end-to-end systems. recent progress on speech and text multimodal models could prove to be useful in addressing this problem @xcite . another possibility would be to exploit agent turns using an audio only model, either as synthesized speech or as intermediary asr representations @xcite . this work focused on a cascade approach but a thorough comparison of end-to-end and cascade approaches could also be helpful in further research on spoken dialogue systems.