PRESTO : A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs (2023)

Abstract: Research interest in task-oriented dialogs has increased as systems such as Google Assistant, Alexa and Siri have become ubiquitous in everyday life. However, the impact of academic research in this area has been limited by the lack of datasets that realistically capture the wide array of user pain points. To enable research on some of the more challenging aspects of parsing realistic conversations, we introduce PRESTO, a public dataset of over 550K contextual multilingual conversations between humans and virtual assistants. PRESTO contains a diverse array of challenges that occur in real-world NLU tasks such as disfluencies, code-switching, and revisions. It is the only large scale human generated conversational parsing dataset that provides structured context such as a user’s contacts and lists for each example. Our mT5 model based baselines demonstrate that the conversational phenomenon present in PRESTO are challenging to model, which is further pronounced in a low-resource setup.

## introduction virtual dialog agents (a.k.a. "assistants") are increasingly becoming a part of our everyday life. from setting up alarms to controlling one's home environment, users now converse with their devices to accomplish many tasks previously done manually. parsing task-oriented dialogs is the problem of understanding the user's intent and any associated arguments so that the assistant can fulfill the requested task. given the prominence of data-driven methods in nlp, public availability of relevant datasets dictates which problems can be effectively studied by the research community at large. datasets ## dataset characteristics in this section, we discuss the characteristics of presto dataset that set it apart from existing datasets: native conversations in multiple languages, code-switched utterances, user revisions, disfluencies, and structured context. native speakers. the presto dataset only includes utterances provided by native speakers of the language, with no translation. prior large multilingual datasets for conversational semantic parsing such as mtop @xcite and @xcite contain non-english conversations obtained by translating english conversations to other languages, foot_0 resulting in unnatural and synthetic utterances which are unlikely to be spoken by native speakers of the non-english language. for example, an arabic example in massive: does not resemble native arabic speech of any dialect. the corresponding english utterance in the massive dataset is: i need to have location services on can you check. code-switched utterances. multilingual users often mix words from two languages in the same utterance, as shown in fig. 2 . recognizing the difficulty of parsing such utterances, we asked bilingual data contributors to provide mixed utterances, and marked these examples with a special tag 'code-switching' to enable isolated research focused on addressing this phenomenon. we have 4 such tags in our dataset which are all grouped under the broader user-revision linguistic phenomenon. the tags are: correct-action, correct-argument, within-turn-correction, and cancel-action. sometimes the revision is necessary due to a mistake made by the virtual assistant, as in fig. 1 (correct-argument). at other times, the user may simply change their mind about what they want the assistant to do which may happen in the same utterance, e.g., "add bread no no wait add wheat bread to my shopping list", or in a later utterance, e.g., "sorry add wheat bread instead." another common revision users make is to cancel their requests, e.g., "sorry don't add anything". fig. 1 shows examples of some of these revisions. disfluencies. due to the spoken nature of most conversations with virtual assistants, user utterances frequently have disfluencies such as repeated phrases and filler words, as shown in fig. 3 . the gold parse provided in the dataset shows the correct interpretation for the last utterance, leveraging the structured context. dataset # languages multi-turn explicit labeled conversational context phenomena presto 6 ✓ ✓ code-switching, disfluencies, user-revisions, coreferences * treedst (cheng et al., 2020) 1 ✓ ✗ coreferences smcalflow (andreas et al., 2020) 1 ✓ ✗ coreferences, user-revisions multiwoz @xcite 1 ✓ ✗ coreferences * dstc10 @xcite 1 ✓ ✗ disfluencies * top @xcite 1 ✗ ✗ -mtop @xcite 6 ✗ ✗ -massive @xcite 51 ✗ ✗ -snips @xcite 1 ✗ ✗ -multiatis++ @xcite 9 ✗ ✗ - other statistics. presto covers 34 intents, including the intent 'other' which is a catch-all intent used for all out-of-scope utterances and constitutes 11.3% of all examples in the dataset. the average number of the previous user turns is around 2.6, which is representative of the relatively short conversations users tend to have with virtual assistants today. related datasets there have been multiple related datasets which are listed in in terms of representations, there are 2 popular paradigms, a direct semantic parse typically consisting of intents and slots vs maintaining an explicit dialog state. with the popularity of pretrained text-to-text models, the natural modeling choice for all of these has converged to similar models. in fact, our context representation can be seen as a constant dialog state for that conversation. while other datasets have focussed on multi-intent and compositional utterances @xcite , presto emphasizes on labelled conversational phenomena and explicit context, which are important for modeling real-world task-oriented dialogues. ## data collection the scope for data collection is based on the app actions built-in intents available for 3rd-party an-droid developers, which cover a wide spectrum of domains including finance, health, fitness, food, transportation, social, communications, shopping among others. 4 our dataset has over 34 unique intents and 370 arguments across 6 languages. contextual examples. data contributors were asked to have conversations with a virtual assistant simulator based on different sets of instructions for each language and for each type of targeted linguistic phenomenon (e.g., code-switching). each data collection request targeted a single intent and was initialized with potentially relevant structured context, e.g., contacts, notes or lists, which can optionally be referenced by data contributors in the utterances they produce. the initial seed contacts, notes and lists are authored by native speakers and are shared for all examples in one request. for example, when initiating a call, data contributors may use one of the names in the contacts (e.g., "call mom"), but they may also use the name of a business not in the contact list (e.g., "call mcdonalds"). each user utterance from the collected conversations is exported as a candidate example for semantic annotation by trained expert linguists who produce two types of annotations. 5 first, linguists decide whether the example is in scope or not. out of scope examples include incoherent or nonsen-sical utterances, unsupported arguments and outof-scope intents. then, they choose the intent and nested arguments as shown in fig. 1 . finally, they added semantic tags which indicate which linguistic phenomena of interest are expressed in this example, such as, correct-argument, cancel-action, code-switching, etc. ## experiments in this section, we demonstrate a few experimental setups which are enabled by presto, provide baseline results for future work, and summarize our findings. ## conclusion we introduce presto, a over 550k-examples, six-language dataset for parsing realistic taskoriented dialogs. presto is enriched with a high representation of contextual examples, code switching, user revisions and disfluencies. our initial results demonstrate that it is possible for our models to perform well on various linguistic phenomena, but only when the model is exposed to a very large number of training examples with the given phenomenon. we observe that feeding a simple representation of the structured context to the model does not yield large improvements on contextual examples. we also notice that multilingual models perform better than monolingual models in lower data regimes. open questions. by introducing a new dataset with unique characteristics, this work deliberately asks more questions than it answers. some of the obvious modeling questions which this dataset can help answer are: how do we train models which perform well on conversational phenomenon like code-switching, user revisions and disfluencies, with a relatively small number of training examples that exhibit these phenomena? how do we identify examples for which context is critical to understanding the user's utterance? how can structured context be represented in our models most optimally? how can we increase cross-lingual supervision in multilingual models to make them more data efficient? we hope that presto is useful in answering these and other questions about conversational semantic parsing.