NLU ++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue (2022)

Abstract: We present NLU++, a novel dataset for natural language understanding (NLU) in task-oriented dialogue (ToD) systems, with the aim to provide a much more challenging evaluation environment for dialogue NLU models, up to date with the current application and industry requirements. NLU++ is divided into two domains (BANKING and HOTELS) and brings several crucial improvements over current commonly used NLU datasets. 1) NLU++ provides fine-grained domain ontologies with a large set of challenging multi-intent sentences combined with finer-grained and thus more challenging slot sets. 2) The ontology is divided into domain-specific and generic (i.e., domain-universal) intents that overlap across domains, promoting cross-domain reusability of annotated examples. 3) The dataset design has been inspired by the problems observed in industrial ToD systems, and 4) it has been collected, filtered and carefully annotated by dialogue NLU experts, yielding high-quality annotated data. Finally, we benchmark a series of current state-of-the-art NLU models on NLU++; the results demonstrate the challenging nature of the dataset, especially in low-data regimes, and call for further research on ToD NLU.

## introduction research on task-oriented dialogue (tod) systems @xcite has become a key aspect in industry: e.g., tod is used to automate telephone customer service tasks ranging from hospitality over healthcare to banking @xcite @xcite . typical tod systems still rely on a modular design: (i) the natural language understanding yes, i need this card to arrive before 3pm on jan 14 intents: affirm, card, arrival, less_lower_before intents: greet, change, spa, booking (nlu) module maps user utterances into a domainspecific set of intent labels and values @xcite @xcite , followed by (ii) the policy module, which makes decisions based on the information extracted by the nlu @xcite @xcite the nlu module is a critical part of any tod system, as it must extract the relevant information from the user's utterances. the information relevance is denoted by the structured dialogue domain ontology, which enables the policy module to make decisions about next system actions. the domain ontology covers the information on 1) intents and 2) slots, see figure 1 . the former is aimed at extracting general conversational ideas (i.e., the user's intents) and corresponds to the standard nlu task of intent detection (id); the latter extracts specific slot values and corresponds to the nlu task of slot labeling (sl) @xcite . 1 in order to make the policy operational and tractable, nlu should extract only the minimal information required by the policy. therefore, the ontologies differ for each domain of tod application and are typically built from scratch for each domain. consequently, this makes domain-relevant nlu data extremely expensive to collect and annotate, and prevents its reusability @xcite . due to this, nlu research in recent years has heavily focused on very data-efficient models that can effectively operate in low-data regimes. current state-of-the-art (sota) nlu models leverage large pretrained language models (plms) @xcite @xcite and fine-tune them with small task-specific datasets @xcite @xcite at the same time, the progress in creation of nlu datasets has not kept up with the impressive pace of nlu methodology development. however, designing domain ontologies and nlu datasets is also critical for steering further progress in nlu, both from methodology and application perspective. put simply, current publicly available nlu datasets do not keep up to date with current industry/application requirements for many reasons. 1) they are usually crowdsourced by untrained annotators (thus typically optimised for quantity rather than quality), yielding examples with low lexical diversity and prone to annotation errors. 2) they typically assume one intent per example, and thus enable only much simpler single-label id experiments; such setups are not realistic in more complex industry settings (see figure 1 again ) and lead to unnecessarily large intent sets. 3) their ontologies are tied to specific domains, making it difficult to reuse already available annotated data in other domains. 4) the complexity of the defined tasks and ontologies is limited; the undesired artefact is that current nlu datasets might overestimate the nlu models' abilities, and are not able to separate models any more performance-wise. 2 2 for instance, for some standard and commonly used nlu datasets such as atis @xcite and snips @xcite , the results of sota models are all in the region of 97-98 f1, with new models getting statistically insignificant gains which might be due to overfitting to the test set or even some remaining annotation errors. in order to address all these gaps, we introduce nlu++, a novel nlu dataset which provides highquality nlu data annotated by dialogue experts. nlu++ provides multi-intent, slot-rich and semantically varied nlu data, and is inspired by a number of nlu challenges which tod systems typically face in production environments. unlike previous id datasets, examples are annotated with multiple labels, named intent modules foot_1 (see finally, we run a series of experiments on nlu++ with current sota id and sl models, demonstrating the challenging nature of nlu++ and ample room for future improvement, especially in low-data setups. our benchmark comparisons also demonstrate strong performance and shed new light on the (ability of) recently emerging qa-based nlu models @xcite @xcite , consisting of spoken queries on flightrelated information. 4 over the next two decades, very few nlu resources were released. 5 the lack of tod nlu resources ended in 2013, with the beginning of the 'dialogue state tracking (dst) era' @xcite @xcite . instead of just classifying each turn of the user, dst deals with keeping track of the user's goal over the entire dialogue history, i.e., all the previous user and system turns. several datasets where released during the dst challenges, all of them comprising simple intent sets (usually tagged as dialogue acts). in order to adapt to the increasing data requirements of deep learning models, increasingly larger dialogue datasets have been released in recent years @xcite @xcite . however, the design of tod datasets comes with some profound differences to datasets for e.g. machine translation or speech recognition, which affect current tod datasets. 1) the domain-specific nature of tod datasets made the data tied to its ontologies, not allowing data reusability across different domains. current nlu trends, inspired by such production requirements, thus deviate from previous dstoriented nlu research in two main aspects. first, the models went back to focusing on single-turn utterances, which 1) simplifies the nlu design and 2) renders the nlu tasks more tractable. 6 the requirement of fast development cycles also instigated more research on nlu (i.e., id and sl tasks) in low-data scenarios. this way, systems can be developed and maintained faster by reducing the data collection and annotation effort. in addition, the nlu focus shifted from ontologies with only a handful of simple intents and slots @xcite to complex ontologies with much larger intent sets @xcite @xcite . inspired by these nlu datasets and empowered by transfer learning with plms and sentence encoders @xcite @xcite , there have been great improvements in single-turn nlu systems recently, especially in low-data scenarios @xcite @xcite @xcite @xcite . current gaps in nlu datasets. however, existing nlu datasets are still not up to the current industry requirements. 1) they use crowdworkers for data collection and annotation, often through simple rephrasings; they thus suffer from low lexical diversity and annotation errors @xcite . ## nlu++ dataset the nlu++ dataset has been designed with the aim of addressing some of the major shortcomings of the current nlu datasets. in what follows, we describe the main improvements and new evaluation opportunities offered by nlu++. ## experiments and results in hope to establish nlu++ as a more challenging production-oriented testbed for dialogue nlu, especially in low-data scenarios, we evaluate a series of current cutting-edge models for both nlu tasks: intent detection ( §4.1) and slot labeling ( §4.2). our aim is to assess and analyse their performance across different setups, and provide solid baseline reference points for future evaluations on nlu++. data setups. unless noted otherwise, for both tasks we adopt the standard k-fold cross-validation ## conclusion we have presented nlu++, a novel dataset for task-oriented dialogue (tod) nlu that overcomes the shortcomings of previous nlu evaluation sets. nlu++ presents a multi-intent and slot-rich ontology, defines generic and domain-specific intents and slots to promote data reusability, and it focuses on the creation of high-quality complex examples and annotations collected by dialogue experts. experimental results show that nlu++ raises the bar with respect to current nlu benchmarks, helping better discriminate and compare the performance of current state-of-the-art nlu models, particularly in low-data setups. we hope that nlu++ will be valuable in guiding future modeling efforts for tod nlu, both in academia and in industry. limitations and future work. this work has shown that a better design of the intent set can improve data reusability. however, the current ontology does not cover generic sets of intents exhaustively, and we acknowledge a (sometimes) fine line between truly generic intents versus intents 'anecdotally' shared by two domains (e.g., refund). further, the boundaries of some generic intents can sometimes be unclear and difficult to annotate, even for expert annotators. 18 future work should try to ground the set of generic intents. further, we believe that span-based annotation might be sub-optimal for canonical values such as times and dates, where small differences in the span would lead to evaluation errors but would not suppose a problem for the value to be parsed. in addition, separating time and date intervals in different slots increases the difficulty of the annotations and models need to learn a more conflicting set of slots. further, nlu++ currently provides fine-grained slots such as date_from, date_to and date to enable more complex scenarios, but such a design might slow down annotation process and make it cumbersome. future work includes rethinking the sl task for these slots. finally, while single-turn nlu is more dataefficient and easier to model, some user utterances only make sense in the presence of context from the previous system utterance. while some previous datasets @xcite deal with this issue with the help of extra annotations indicating if a slot has been requested, in this work we opt for using non-contextualised slots such as number and time and let the policy handle the contextualisation. however, future work should start looking into nlu datasets composed by system + user turns. before data collection: all the data has been collected by workers of polyai limited and all the annotators are also employees of polyai limited. during data collection: we did not include any personal information (e.g. personal names or addresses) and all the examples that included any had been fully anonymised or removed from the dataset. all the names in the dataset are created by randomly concatenating names and surnames from the list of the top 10k names from the us registry. upon collection, the dataset has undergone an additional check by the internal ethics committee of the company. nlu++ is licensed under cc-by-4.0. general medium savings is the intent asking about the savings account? banking low domain specific current is the intent asking about the current account? banking low adjectives business is the intent to ask something about the business account? banking low credit is the intent asking about something related to credit? banking low debit is the intent asking about something related to debit? banking low contactless is the intent to ask about contactless? banking low international is the intent to ask about something related to international issues? banking medium account is the intent asking about some account? banking low domain specific transfer_payment is the intent to ask about something related to a transfer, banking low nouns/entities payment or deposit? appointment is the intent to ask about something about an appointment? banking medium arrival is the intent to ask about the arrival of something? banking medium balance is the intent to ask about balance? banking medium card is the intent to ask about something related to a card or cards? banking low cheque is the intent to ask about cheque? banking low direct_debit is the intent to ask about direct debit? banking low standing_order is the intent asking about a standing order? banking low fees_interests is the intent to ask about fees or interests? banking medium loan is the intent to ask about loans? banking low mortgage is the intent asking about mortgage? banking low overdraft is the intent to ask about ovedraft? banking low withdrawal is the intent to ask about withdrawals? banking low pin is the intent to ask something about the pin number? banking low refund is the intent to ask about some refund? banking, hotels low check_in is the intent to ask about check in? hotels medium check_out is the intent to ask about check out? hotels medium restaurant is the intent to ask something related to restaurant? hotels medium swimming_pool is the intent to ask something related to the swimming pool? hotels low parking is the intent to ask something related to parking? hotels low pets is the intent to ask something related to pets? hotels medium accesibility is the intent to ask something related to accessibility? hotels medium booking is the intent to talk about some booking? hotels medium wifi is the intent to ask something related to wifi or wireless? hotels low gym is the intent to ask something related to gym? hotels low spa is the intent to ask something related to spa or beauty services? hotels high room_ammenities is the intent to ask something related to some room amenities? hotels high housekeeping is the intent to talk about housekeeping issues? hotels medium room_service is the intent to talk about room service? hotels medium