1-step Speech Understanding and Transcription Using CTC Loss (2023)

Abstract: Recent studies have made some progress in refining end-to-end (E2E) speech recognition encoders by applying Connectionist Temporal Classification (CTC) loss to enhance named entity recognition within transcriptions. However, these methods have been constrained by their exclusive use of the ASCII character set, allowing only a limited array of semantic labels. We propose 1SPU, a 1-step Speech Processing Unit which can recognize speech events (e.g: speaker change) or an NL event (Intent, Emotion) while also transcribing vocal content. It extends the E2E automatic speech recognition (ASR) system’s vocabulary by adding a set of unused placeholder symbols, conceptually akin to the <pad> tokens used in sequence modeling. These placeholders are then assigned to represent semantic events (in form of tags) and are integrated into the transcription process as distinct tokens. We demonstrate notable improvements on the SLUE benchmark and yields results that are on par with those for the SLURP dataset. Additionally, we provide a visual analysis of the system’s proficiency in accurately pinpointing meaningful tokens over time, illustrating the enhancement in transcription quality through the utilization of supplementary semantic tags.

## introduction this paper extends the capabilities of end-to-end automatic speech recognition (e2e asr) systems from traditional transcription tasks to more nuanced speech understanding processes @xcite . presently, deep neural networks (dnns) are proficient in applying greedy or beam search decoding to predict vocabulary labels from audio inputs over time @xcite . these networks typically produce a sub-word text token for each segment of a continuous speech stream, focusing on transcription fidelity. however, the transcription process has evolved with recent efforts to embed additional metadata, such as named entity boundaries and utterance classification @xcite . multitask learning frameworks utilizing separate loss functions for transcription and classification have emerged @xcite , and there has been success in identifying sequences of intents using non-autoregressive models within a speech stream @xcite . although recognizing a limited set of named entities using ctc loss by marking them with specific begin and end tokens in transcriptions has been demonstrated @xcite . methods like these are often use special ascii characters for events, constrained by the size ascii character set. morever these special characters are originally defined to be useful for meaningful language transcription. this also happens a as asr encoders are generally considered as intermediaries between speech and natural language (nl) models. in response, our study proposes a single-step, integrated speech processing unit (1spu) using e2e speech encoders. 1spu not only transcribes spoken content but also annotates it with tags indicating various nlp and speech events, including intent and entity recognition as well as speaker changes. by incorporating user_def ined_symbols into the asr vocabulary @xcite , akin to text-based entity tagging tokens, we enhance the encoder's lexicon. these symbols are introduced during the fine-tuning stage and are only used if present in the fine-tuning data, thereby maintaining computational efficiency. we fine-tune a pre-trained conformer asr model to perform transcription while also producing intent labels and entity tags, all under the ctc loss function. the model outputs transcriptions using standard greedy decoding and can function in both offline and streaming modes. our experimentation focuses on mono-channel speech and yields promising results on the slue @xcite and slurp @xcite benchmarks. we posit that incorporating a language model (lm) could significantly elevate these results. furthermore, we present visual evidence that utilizing context from previous utterances can enhance transcription and understanding in dual-channel human-human interactions, particularly in marking speaker changes. this ongoing work has the potential to improve prompting strategies and facilitate the development of multi-modal and multitasking e2e asr applications in streaming contexts. the contributions of this paper are summarized as follows: ## related work a common practice is to transform the normalized token sequences produced by automatic speech recognition (asr) systems into a written form that is more amenable for downstream dialog system components to process @xcite . this reformatted written output then serves as the basis for extracting structured information, such as intents and slot values, which are crucial for the continuation of a dialog @xcite . of late, there has been an emerging trend towards utilizing neural encoders that are optimized directly on speech inputs, a methodology often referred to as end-to-end spoken language understanding (e2e slu) @xcite . text-based sequence tagging: the domain of natural language processing has seen extensive research in employing sequence-to-sequence (seq-2-seq) models to parse and extract structured information from textual data. this encompasses advancements in identifying the positions and classifications of entities within a text @xcite , generating structured summaries of content @xcite , and constructing synctactic parse trees through timely tag predictions during the generation of language (cabot and navigli, 2021). despite their efficacy, these approaches, which are primarily designed for monologic text, do not adequately address the dynamic nature of spoken dialogue. e2e asr: modern automatic speech recognition (asr) systems excel at converting spoken language into a sequence of lexical tokens, effectively transcribing spoken words. this transcription process typically involves refining pre-trained self-supervised neural networks or initiating training from the ground up with non-autoregressive connectionist temporal classification (ctc) loss or a sequential loss @xcite to recognize speech tokens @xcite . transcriptions are obtained either by greedy decoding or beam-search decoding using a language model. ## 1spu: 1-step speech processing unit for 1-step speech processing unit (1spu) we leverage a commercially available end-to-end automatic speech recognition (e2e asr) system, initializing it with a tokenizer that includes d userdefined dummy tokens. these labels are akin to unique tokens, much like a '<@xmath0, and are not utilized during the asr's initial training phase. thus, the incorporation of these inactive tokens does not impact the duration of training or inference as they can be masked during the e2e asr optimization process. subsequently, we re-purpose the pre-trained speech encoder to generate transcriptions that embed semantic tokens indicative of event tags, optimizing with the ctc loss function. this strategy diverges from previous methods that relied on special characters for tagging within transcriptions using ctc loss. our approach, which introduces dummy tags during the asr's pre-training, obviates the need for alterations to the tokenizer or output layer before fine-tuning. we detail a fine-tuning process for a pre-trained speech encoder that empowers it to recognize utterance intent, demarcate the beginning and end of entities, and carry out transcription, using pairs of speech files and tagged transcriptions. an example of a transcribed speech input is: here, calendar_set signifies the intent of the utterance, while the entity tokens person, date, and time terminate with a shared end token. these semantic tokens are mapped to the dummy vocabulary tokens, which are subsequently learned during the fine-tuning stage for the desig-nated task. although the tokens for utterance level semantics can be placed at the beginning or the end, our empirical evidence suggests a slight performance improvement when it is generated at the beginning. in our study, we apply the conformer end-toend asr framework @xcite . we use tokenizer output vocabulary v of size 1024, comprised of 400 un-used tokens d for event labels, 624 transcription tokens l and 1 token representating a blank label. these d tokens are provided to tokenizer as a list of user_defined_symbols ## dataset we evaluate our method on the standardized sluevoxpopuli @xcite and slurp corpus @xcite . ## observations figure 2 illustrates the token output from both a two-step pipeline system and a single-step entity recognition system at each time-step. tokens from the end-to-end automatic speech recognition (e2e asr) are marked in red, whereas tokens pertinent to entities identified by the single-step system are highlighted in blue. in addition to these experiments, we also experiment with propriety human-human dual channel conversations between a caller and an agent. assuming negligible overlap in the audio from both parties, we combine the two channels to create a single-channel audio track. to denote transitions between speakers, we introduce a unique label for "speaker-change." a video demonstration, accessible via the provided link foot_2 , showcases a sample conversation displayed in a format akin to figure 2 . from this, we notice an improvement in both the accuracy of entity prediction and overall transcription, which we attribute to the model's capacity to predict semantic events alongside the speech content. ## conclusions in our study, we demonstrate that simultaneous high-quality event tagging (encompassing intent, entities, and speaker changes) and transcription is feasible with our proposed method. this technique is adaptable across a broad spectrum of events, making it suitable for extensive applications in speech understanding and transcription tasks. we are confident that this methodology can be expanded to accommodate additional languages and various domains due to its multi-tasking nature, facilitated by a single, unified loss function. this makes it an ideal candidate for comprehensive, one-step, automated conversational systems. currently, we are developing a multi-lingual version of this system, which has the capability to perform language translation while concurrently tagging events. additionally, we aim to refine end-to-end automatic speech recognition (e2e asr) systems to respond to event recognition prompts based on user-specified natural language queries.