SLURP : A Spoken Language Understanding Resource Package (2020)

Abstract: Spoken Language Understanding infers semantic meaning directly from audio data, and thus promises to reduce error propagation and misunderstandings in end-user applications. However, publicly available SLU resources are limited. In this paper, we release SLURP, a new SLU package containing the following: (1) A new challenging dataset in English spanning 18 domains, which is substantially bigger and linguistically more diverse than existing datasets; (2) Competitive baselines based on state-of-the-art NLU and ASR systems; (3) A new transparent metric for entity labelling which enables a detailed error analysis for identifying potential areas of improvement. SLURP is available at https://github.com/pswietojanski/slurp .

## introduction traditionally, spoken language understanding (slu) uses a pipeline transcribing audio into text using automatic speech recognition (asr), which is then mapped into a semantic structure via natural language understanding (nlu). however, this modular approach is prone to error propagation from noisy asr transcriptions, and asr in turn is not able to disambiguate based on semantic information. end-to-end (e2e) approaches on the other hand, can benefit from joint modelling. one of the main bottlenecks for building e2e-slu systems, however, is the lack of large and diverse datasets of audio inputs paired with corresponding semantic structures. publicly available datasets to date are limited in terms of lexical and semantic richness @xcite , number of vocalizations @xcite , domain coverage @xcite and semantic contexts @xcite . in this paper, we present the spoken language understanding resource package (slurp), a publicly available multi-domain dataset for e2e-slu, which is substantially bigger and more diverse than existing slu datasets. slurp is a collection of ~72k audio recordings of single turn user interactions with a home assistant, annotated with three levels of semantics: scenario, action and entities, as in fig. 1 , including over 18 different scenarios, with 46 defined actions and 55 different entity types as listed on https://github.com/pswietojanski/slurp . 1 in order to further support slu development, we propose slu-f1, a new metric for entity prediction, which is specifically designed to assess error propagation in structured e2e-slu tasks. this metric has 3 main advantages over the commonly used accuracy/f1 metric, aimed at supporting slu developers: first, it computes a distribution rather than a single score. this distribution is (1) inspectable and interpretable by system developers, and (2) can be converted into a confidence score which can be used in the system logic (akin to previously available asr confidence scores). finally, the distribution reflects errors introduced by asr and their impact on nlu and thus (3) gives an indication of the scope of improvement that can be gained by e2e approaches. using this metric, we evaluate 4 baseline systems that represent competitive pipeline approaches, i.e. 2 state-of-the-art nlu systems and 2 asr engines. we conduct a detailed error analysis of cases where e2e could have made a difference, i.e. error propagation and semantic disambiguation. ## related work the first corpora containing both audio and semantic annotation reach as far back as the the air travel information system (atis) corpus @xcite and the switchboard-damsl labeling project @xcite . however, it was not until recently when the first e2e approaches to slu were introduced @xcite . since then, one of the main research questions is how to overcome data sparsity by e.g. using transfer learning @xcite , or pre-training @xcite . here, we present a new corpus, slurp, which is considerably bigger than previously available corpora. in particular, we directly compare our dataset to the two biggest e2e-slu datasets for the english language: the snips benchmark @xcite and the fluent speech command (fsc) corpus @xcite . with respect to these resources, slurp contains ~6 times more sentences than snips, ~2.5 times more audio examples than fsc, while covering 9 times more domains and being on average 10 times lexically richer than both fsc and snips, see section 3.3. slurp represents the first e2e-slu corpus of this size for the english language. the only existing comparable project is represented by the castlu dataset @xcite for chinese mandarin. ## slurp metrics the standard metric for evaluating e2e-slu is accuracy, which is defined as "the accuracy of all slots for an utterance taken together -that is, if the predicted intent differs from the true intent in even one slot, the prediction is deemed incorrect" @xcite . however, this notion of accuracy is problematic when it comes to evaluating entities, as it does not account for the interplay between semantic mislabelling and textual misalignment. nor does it differentiate between entity label and lexical filler, as in fig. 4 , where lexical filler is defined as span over tokens in the original sentence. 5 formally, given a sentence s, let e and ê be the set of gold and predicted entities, respectively. each e @xmath0, f @xmath1, while f @xmath2, . . . , t n ] is the lexical filler, defined as a span of consecutive tokens of s such that 1 ≤ @xmath3. similarly, predicted entities are of the form êk = lk , @xmath4. in spanbased metrics, two entities e 1 and e 2 are identical (e 1 =:= e 2 ) when both labels and lexical fillers are the same (l 1 = l 2 ∧f 1 = f 2 ). a match is thus found only whenever the gold and predicted entities are identical, i.e. e i =:= êk . this evaluation method holds in nlu because entities are tagged over the same textual sequence. when evaluating e2e-slu, where entities are identified out of a wave form, this strict coupling with the token sequence may no longer apply. note that pipeline systems for slu are affected as well since they operate over asr transcribed sentences, which can consistently differ from the original gold transcription. to account for this mismatch, we propose slu-f1, a new metric which does not overly penalise misalignments caused by asr errors. in addition, it is able to capture the quality of transcriptions and entity tagging errors at the same time in a single metric. as such, this metric allows to directly compare e2e and pipeline systems. in particular, slu-f1 combines span-based f1 evaluation with a text-based distance measure dist, e.g. wer. the equality property =:= is relaxed by allowing gold and predicted entities (e i and êk ) to match (e i =:= êk ) when the corresponding labels are identical (l @xmath5, even when the fillers are not identical. in this case we increment the true positives (tps) by 1. to account for lexical distance/ mismatch, we compute the dist between gold and predicted fillers (dist(f i , fk )), and increment the false positives (fps) and false negatives (fns) of this amount, as in algorithm 1. in the case of a predicted entity label matching with more than one gold entity, e.g. when two or more entities with the same label are present, we opt for a non-conservative approach, selecting the gold annotation minimising the dist as a candidate. the assumption is that the pair of entities is most likely referring to the same text span. we use two distance functions to capture different aspects of possible transcription mistakes: wer (word-f1) and the normalised levenshtein distance on character level (char-f1). wer is a strict token-level metric, which outputs errors/null matches whenever a mismatching or misalignment of tokens is observed. the character-based levenshtein distance, on the other hand, offers the opposite perspective. by computing character-based similarities, it is much less susceptible to small variations of input strings, and thus better accounting for local transcription errors which do not affect nlu tagging. for example, word-f1 will penalise small morphological differences e.g. singular vs. plural as in pizza vs. pizzas, which are often seen in transcriptions. this over-penalises nlu outputs, e.g. the tagging of pizzas may be semantically correct. char-f1 on the other hand does not over-penalise nlu, but it also may provide a positive score when two fillers have similar characters, but are semantically and phonetically unrelated. in other words, word-f1 shows the influence of asr on nlu, whereas char-f1 gives an indication of nlu performance despite transcription noise. these dist-f1 metrics (@xmath6. they fundamentally differ for the adopted string matching schema: any dist-f1 considers string ordering to score string similarity, while the fuzzy mechanism is instead order invariant. consider the illustrative entity tagging example in figure 4 . here, aaronson has been wrongly transcribed into aron's son, and morning has been wrongly tagged with date. a dist-f1 will score the predicted entities as follows: both [event name: brunch] and [date: saturday] contribute with a +1 to the tps, since both label and filler correspond to gold information. the wrong label associated with morning increases the fps of 1, although it is correctly transcribed. it follows that the entity timeofday is not predicted, increasing the fns of 1. finally, [person: aron's son] is correctly labelled, but its filler is partially wrong. it thus contributes to the tps by 1, but fps and fns are both incremented by dist(aaronson, aron's son). ## discussion slurp is not only bigger, but also a magnitude more challenging than previous datasets. the purpose of this new data release is not to provide yet another benchmark dataset, but to provide a usecase inspired new challenge, which is currently beyond the capabilities of sota e2e approaches (due to scalability, lack of data efficiency, etc.). developing an appropriate model architecture is left for future work. for this reason, in this work we focus on benchmarking existing approaches. we show that sota modular approaches are able to provide a strong baseline for this challenging data, which has yet to be met by sota e2e systems. we also argue that our modular baseline is closer to how real-world applications build slu systems, nevertheless often overlooked when testing e2e systems. as such, we consider our sota modular baseline a major novel contribution. ## conclusion in this paper, we present slurp, a new resource package for slu. first, we present a novel dataset, which is substantially bigger than other publicly available resources. we show that this dataset is also more challenging by first conducting a linguistic analysis, and then demonstrating the reduced performance of state-of-the-art asr and nlu systems. second, we propose the new slu-f1 metric for evaluating entity prediction in slu tasks. in a detailed error analysis we demonstrate that the distribution of this metric can be inspected by system developers to identify error types and system weaknesses. finally, we analyse the performance of two state-of-the-art nlu systems on asr data. we find that a sequential decoding approach for slu, which starts from the more abstract notion of scenario and action produces better results for entity tagging, than an approach which works bottom up, i.e. starting from the entities. our error analysis suggests that this is due to the former approach being able to better account for noise by priming entity tagging, which is a more challenging task than scenario or action recognition. in future work, we hope that slurp will be a valuable resource for developing e2e-slu systems, as well as more traditional pipeline approaches to slu. the next step is to extend slurp with spontaneous speech, which would again increase its complexity, but also move it one step closer to real-life applications.