Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models (2022)

Abstract: End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation. However, these systems model sequence labeling as a sequence prediction task causing a divergence from its well-established token-level tagging formulation. We build compositional end-to-end SLU systems that explicitly separate the added complexity of recognizing spoken mentions in SLU from the NLU task of sequence labeling. By relying on intermediate decoders trained for ASR, our end-to-end systems transform the input modality from speech to token-level representations that can be used in the traditional sequence labeling framework. This composition of ASR and NLU formulations in our end-to-end SLU system offers direct compatibility with pre-trained ASR and NLU systems, allows performance monitoring of individual components and enables the use of globally normalized losses like CRF, making them attractive in practical scenarios. Our models outperform both cascaded and direct end-to-end models on a labeling task of named entity recognition across SLU benchmarks.

## introduction sequence labeling (sl) is a class of natural language understanding (nlu) tasks. these systems tag each word in a sentence to provide insights into the sentence structure and meaning @xcite ). an sl system that processes unstructured text, first encodes the context and relationships of words in the sentence using an encoder and then labels each token @xcite @xcite . however, when dealing with spoken utterances, sequence labeling introduces an additional complexity of 1 our code and models are publicly available as part of the espnet-slu toolkit: https://github.com/espnet/ espnet . * equal contribution. siddharth is now at google. also recognizing the mentions of the labels @xcite . sl in spoken language understanding (slu) has been approached by two schools of thought, (1) that seek to recognize the spoken words using an automatic speech recognition (asr) engine and then tag the mentions using an nlu engine in a cascaded manner @xcite @xcite , and (2) that seek to recognize and tag the mentions directly from speech in an end-to-end (e2e) framework @xcite . prior work has shown that cascaded systems suffer due to error propagation @xcite from the asr into the nlu engine, which can be overcome in an e2e framework. however, unlike cascaded models, e2e systems cannot utilize the vast abundance of nlu research @xcite as they re-define the sl problem as a complex sequence prediction problem where the sequence contains both the tags and its mentions. inspired by the principles of task compositionality in sl for slu, we seek to bring both schools of thought together. our conjecture is that we can build compositional e2e systems that first convert the spoken utterance to a sequence of token representations @xcite , which can then be used to train token-wise classification systems as per the nlu formulation. by also conditioning our token-wise classification on speech, our compositional e2e system allows recovery from errors made while creating token representations. we instantiate our formulation on a popular sl task of named entity recognition (ner) and (1) present the efficacy of our compositional e2e ner-slu system on benchmark slu datasets @xcite surpassing both the cascaded and direct e2e systems §5.2. (2) our compositional model consists of asr and nlu components compatible with pre-trained asr and ner-nlu models §5.3. (3) our e2e systems ex-hibit transparency towards categorizing errors by enabling the evaluation of individual components of our model in isolation §5.4. the paper first describes the traditional sl formulation ( §2), and discusses shortcomings in current slu formulations ( §3). section §4 presents our compositional e2e model that can overcome these shortcomings. we then evaluate these approaches towards the sl task of ner ( §5). ## sequence labeling (sl) sl systems tag each word, w i , of a text sequence, @xmath0, . . . , n } of length n and vocabulary v, with a label from a label set l, {w @xmath1. this produces a label sequence, @xmath2, . . . , n } of the same length n . using decision theory, sequence labeling models seek to output ŷ from a set of all possible tag sequence l n , conditional random field: @xcite aims to directly compute the posterior of the entire label sequence y given the sentence s: token classification model: since the advent of strong contextual modeling using transformer based models, sequence labeling can also be treated as token classification @xcite , a simplification over memm estimations @xcite , with the assumption that the current tag is conditionally independent to previous tag. these models are still effective as h l is able to model the full context s for every word w l . in cases like ner, where an entity can span multiple words, these problems are modeled using bio tags @xcite , where begin (b), inside (i) tags are added for entities and an outside (o) tag for non-entity words, extending the tag set vocabulary from l to l ′ = {l b ⊕ l i |@xmath3.when modeled using sub-word tokens the tags can be aligned to the first sub-word token of the word and the remaining ones can be marked with a special token ∅ giving l ′′ = l ′ ∪ {∅}. ## sequence labeling in slu sequence labeling in slu introduces an added complexity of recognizing mentions on top of textbased sl tasks ( §2) as they aim to predict the tag and its mentions directly from a spoken sequence. given a sequence of d dimensional speech feature of length t frames, @xmath4, . . . , t }, these systems seek to estimate the label sequence ŷ from cascaded slu @xcite @xcite models p (y |x) from p (y |s) using an nlu framework ( §2) and p (s|x) using an asr model @xcite @xcite , assuming condi- once ŝ is estimated, ŷ can be estimated using eq 1. although this enables realizing ŷ using two well studied frameworks, the independence assumption doesn't allow recovery from errors in estimating ŝ. direct end-to-end slu @xcite @xcite systems avoid cascading errors by directly modeling p (y |x) in a single monolithic model. to achieve this while being able to recognize the spoken mentions, these systems enrich y with transcripts s, y @xmath5, . . . , n ′ }, where n ′ is the length of y e . this can be modeled using an autoregressive decoder as: however this new formulation cannot utilize the well studied sequence labeling framework §2. additionally, this applies an extra burden of labeling along with alignment on the decoder and makes understanding the errors made by these systems particularly difficult. for example, eq 13 gives non-zero likelihood to a corrupt sequence with only labels and no words as y @xmath6. ## compositional end-to-end slu we propose to bring the two paradigms together in a compositional end-to-end system, by extending over the cascaded slu formulation using searchable intermediate framework @xcite : this system can be realized with two sub-networks as shown in figure 1 , where: ## spoken named entity recognition to show the effectiveness of our compositional e2e slu model we build spoken ner systems on two publically available slu datasets, slue @xcite and slurp @xcite (dataset and preparation details in §a.2). we compare our compositional e2e system with cascaded and direct e2e systems. we also compare with another compositional e2e system that predicts the enriched transcript ( §3) using a decoder like @xcite instead of label sequence (i.e. y e instead of y in eq. 15) using a token level classification sub-network. we refer to this baseline model as "compositional e2e slu with direct e2e formulation". slurp is evaluated using slu-f1 @xcite which weighs the entity labels with the word and character error rate of the predicted mentions and slue using f1 @xcite which evaluates getting both the mention and the entity label exactly right. we also compute label-f1 for both datasets which considers only the entity label. we report micro-averaged f1 for all results. ## conclusion we propose to combine text based sequence labeling framework into the speech recognition framework to build a compositional end-to-end model for slu. our compositional e2e models not only show superior performance over cascaded and direct end-to-end slu systems, but also bring the power of both these systems in a single framework. these models can utilise pretrained sub task components and exhibit transparency like cascaded systems, while avoiding error propagation like direct end-to-end systems.