DSTC -11: Speech Aware Task-Oriented Dialog Modeling Track (2023)

Abstract: Most research on task oriented dialog modeling is based on written text input. However, users interact with practical dialog systems often using speech as input. Typically, systems convert speech into text using an Automatic Speech Recognition (ASR) system, introducing errors. Furthermore, these systems do not address the differences in written and spoken language. The research on this topic is stymied by the lack of a public corpus. Motivated by these considerations, our goal in hosting the speech-aware dialog state tracking challenge was to create a public corpus or task which can be used to investigate the performance gap between the written and spoken forms of input, develop models that could alleviate this gap, and establish whether Text-to-Speech-based (TTS) systems is a reasonable surrogate to the more-labor intensive human data collection. We created three spoken versions of the popular written-domain MultiWoz task – (a) TTS-Verbatim: written user inputs were converted into speech waveforms using a TTS system, (b) Human-Verbatim: humans spoke the user inputs verbatim, and (c) Human-paraphrased: humans paraphrased the user inputs. Additionally, we provided different forms of ASR output to encourage wider participation from teams that may not have access to state-of-the-art ASR systems. These included ASR transcripts, word time stamps, and latent representations of the audio (audio encoder outputs). In this paper, we describe the corpus, report results from participating teams, provide preliminary analyses of their results, and summarize the current state-of-the-art in this domain.

## introduction in recent years, automatic speech recognition (asr) and natural language processing (nlp) models have converged to utilize common components like transformers and encoder/decoder modules. they increasingly rely on large amounts of data, large model sizes and large amounts of compute resources. this is a substantial departure from a previous era when asr and nlp utilized different modeling architectures, chosen to inject domain-specific knowledge and constraints. this shift to a common paradigm has stimulated research in fusing audio and text modalities to substantially improve performance in tasks such as audio intent classification and speech-to-speech translation. the fusion of modalities in many cases allows the direct optimization of the endtask, overcoming the hurdles of the older cascaded approaches that often led to accumulation of errors. despite the general trend to develop end-to-end models for various tasks, spoken dialog systems stick out as a sore thumb. most practical systems utilize a cascaded approach where the output of a general asr system is fed into a dialog model trained separately on written domain. this mismatch between written and spoken inputs to the dialog models is not well-studied, largely due to the lack of a public task with spoken user inputs. research into combined audio-text models is limited by the lack of paired data. while the paired data requirement can be relaxed to some extent for training data via un-or self-supervised training techniques, test sets with paired data are crucial for model evaluation. in addition to an evaluation task, a training set with spoken input would also be helpful in quantifying the gains from supervised learning and recent advances in self-supervised learning. the focus of our effort was to bring most benefit to the community with the limited resources available. while a wizard-of-oz style data collection in spoken domain would have been ideal to fully investigate all the phenomena of spoken domain, that would be extremely labor-intensive especially in annotating the dialog states and was beyond the scope of our effort. instead, we chose to create a spoken version of a well-studied writtendomain task, the multiwoz task. one advantage of this approach was that we could directly compare the performance of the spoken version with the continuing advances in the written domain. knowing that the current text-to-speech (tts) systems have come a long way, we also chose to create a tts-version of the multiwoz training corpus as a surrogate for the human spoken version. in the speech-aware dstc11 challenge, participants are asked to infer the dialog states from the sequence of agent (text-input) and user (audio-input) turns. we evaluated the performance on three versions of audio inputs -tts-verbatim, human-verbatim (humans speaking the written user inputs), and human-paraphrased (humans paraphrasing the written user inputs). aside from the audio-inputs, we provided transcripts from a state-of-the-art asr system trained on 33k hours of people's speech corpus to encourage participation from teams that did not have an easy access to asr systems. the transcripts were accompanied with timestamps of the words and the latent representations of the acoustic encoder, which could be used by participants to train joint audio-text encoders. in the course of developing this challenge, we developed a cascaded baseline system with data augmentation and report performances on a few variants of cascaded systems. in the process, we uncovered a bias in the multiwoz evaluation task, the slot values in the evaluation set have a substantial overlap with those of the training set. to address this bias, we created a new version of the multiwoz evaluation set, the dev-dsct11 and eval-dstc11. we observed that the new task surfaces many of the challenges in practical spoken dialog systems associated with mismatch in modalities, inability to recover from asr errors, and more generally difficulty of extracting semantically relevant information from audio signals. ## related work motivated by similar consideration as ours, @xcite organized a dst challenge in 2021 where they created a task using spoken humanhuman dialogs on tourist information in san francisco for three target domains: hotel, restaurant, and attraction. one of the serious limitations of the challenge was that the audio data was not released, only the asr transcripts. the transcripts had an error rate of about 26.25% which is significantly higher than the average performance of most state-of-the-art asr systems. the larger fo-cus of their effort was on evaluating correctness of detecting knowledge-seeking turns, identifying the knowledge snippets and knowledge supplied in the generated responses. prior to this effort, there have been much smaller efforts with fewer domains and dialogs in dstc2 and dstc3 @xcite . here again the organizers only provided asr transcripts with error rates in high 20s and low 30s, which limited the utility of the corpus as the asr systems improved over time. meanwhile considerable progress has been achieved in improving the naturalness of dialog systems, for example, with chat-bots like meena (adiwardana et al., 2020), raising expectations of interacting with dialog systems using spoken language. similarly, the convergence of model architectures for asr and nlp has stimulated research in creating joint audio-text encoders that could potentially compensate for asr errors and other spoken language phenomena @xcite @xcite @xcite @xcite , where they propose different approaches to align the speech input (frames, phones, utterances) to the corresponding text units. however, none of them have been evaluated on dialog models due to the lack of a dialog task and corpus with audio input. we hope the task and corpus released in this work will bridge this gap along with other recent speech understanding tasks such as superb and slurp @xcite 3 data we chose to create a spoken version of multiwoz (2.1 version) so we could directly compare the written and spoken versions and reuse the annotations of dialog state labels, avoiding the labor-intensive process of annotating reference labels. ## challenge task and evaluation the main task in the dstc11 challenge is to infer the dialog states correctly from the audio inputs that were provided corresponding to the redesigned multiwoz test set, redesigned as described in section 3.1. for ablation studies, the participants were required to submit their results for three conditions: tts-verbatim, human-verbatim and humanparaphrased, which were collected as described in section 3.2 and 3.3. the participants were free to use their own asr system or the outputs provided from our baseline asr system, described in section 5.1. the performance in the challenge is measured using the standard joint goal accuracy (jga) as a primary metric and slot error rate (ser) as a secondary metric for fine-grained comparisons. in the literature, results are often difficult to compare since research groups apply their own output normalization before scoring the results. to sidestep the resulting confusion, this challenge is evaluated using standard evaluation tool published on github @xcite . we created a patch to support the results from this challenge and published the golden reference for our redesigned evaluation sets, which can be downloaded from the dstc11 challenge website https://dstc11. dstc.community . the participants were allowed to utilize any publicly available data or checkpoints, but no private data to allow fair comparisons. they were allowed to augment data and no restrictions were imposed on model sizes or computational costs. ## results we received 11 submissions from 6 teams and the results are shown in figures 2 and 3. the full set of numbers are also available in the link under the dstc11 challenge website https://dstc11. dstc.community . we provide a high level overview of the results, but refer to the team's system descriptions for details. while we provided data to build direct audioto-dst and tightly coupled models, all teams chose a cascaded approach with separately trained asr and dst models. many teams employed an explicit asr error correction model and re-trained their dst models on asr hypotheses together with various forms of tts-based data augmentation. within this general approach teams experimented with several variations and as a result the performance across submissions vary substantially. the highest performing submission obtains a jga of 37.9% while the lowest performance is at 18.2%. f-p f-s a-s c-p c-s b-p b-s a-p d-s e-p d-p 0 5 10 15 20 25 30 35 tts-verbatim human-verbatim human-paraphrased figure 3: slot error rate (ser) of team's submissions ## conclusions in this paper, we describe a new corpus for stimulating research in modeling spoken dialogs that builds on popular written dialog corpus, multiwoz. having a spoken version of the same evaluation set allows researchers to study and bridge the performance gap between written and spoken dialog models. in the course of characterizing the task, we observed that there is substantial overlap between the slot values in the training and evaluation sets of the original multiwoz corpus. we redesigned the evaluation sets (dev-dstc11 and eval-dstc11) by sampling new non-overlapping slot values and show that the new sets captures the weakness of the written dialog models better. we released three versions of the task -tts-verbatim, human-verbatim and human-paraphrased, where the last two were collected from crowd-workers. one caveat of this collection is that disfluencies such as speech repair are not as well represented as in natural conversations. we have characterized this gap using a baseline cascaded asr-dst system using asr and dst models. while the performance improves with model size and data augmentation, even the best models (d3st-xxl) show substantial drop in performance when switching from written version to the spoken version (53.8% to 26.1% jga). retraining the d3st-xxl model on the asr hypotheses, improves the performance to 30.9% jga but still leaves substantial ground to be covered. we report the results of 11 submissions from the dstc11 challenge. the dominant paradigm across teams was to rely on large language models for dst. several submissions inserted asr error correction modules of different complexities. the performance of the teams varied from a high of 37.9% to a low of 18.2% jga. one area of research that is not well explored is better utilization of the latent representations of the audio encoder in the asr in the downstream dst models. similarly, we hope the release of the audio and the audio encoders outputs will allow researchers to evaluate the power of joint audio-text encoders on dialog tasks.