T oken V erse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR (2024)

Abstract: In traditional conversational intelligence from speech, a cascaded pipeline is used, involving tasks such as voice activity detection, diarization, transcription, and subsequent processing with different NLP models for tasks like semantic endpointing and named entity recognition (NER). Our paper introduces TokenVerse, a single Transducer-based model designed to handle multiple tasks. This is achieved by integrating task-specific tokens into the reference text during ASR model training, streamlining the inference and eliminating the need for separate NLP models. In addition to ASR, we conduct experiments on 3 different tasks: speaker change detection, endpointing, and NER. Our experiments on a public and a private dataset show that the proposed method improves ASR by up to 7.7% in relative WER while outperforming the cascaded pipeline approach in individual task performance. Our code is publicly available: https://github.com/idiap/tokenverse-unifying-speech-nlp

## introduction automated analysis of conversational audios has a wide range of practical applications, including in contact center analytics @xcite . traditionally, conversational audios are transcribed with intermediate voice activity detection (vad) @xcite or endpointing @xcite and diarization @xcite . afterward, separate nlp pipelines are employed on the transcripts to perform tasks such as named entity recognition (ner) @xcite , among others, to comprehend the conversation's structure and content @xcite . using separate models for each subtask (optimized independently) has drawbacks @xcite such as error propagation and a potential mismatch between automatic speech recognition (asr) metrics and the final task. for instance, the best asr hypothesis may not be in this paper, we introduce tokenverse, a neural transducer @xcite model capable of learning asr and multiple additional tasks through the incorporation of task tokens. in contrast to the multi-head based multitasking approaches explored in previous studies @xcite @xcite , tokenverse distinguishes itself by generating tokens directly within the asr hypothesis, as illustrated in fig. 1a . leveraging the transducer architecture @xcite , we can attain text-audio alignment for each output token, including those designated as task tokens. for example, we can perform ner directly in the acoustic domain, presenting potential utility in scenarios such as audio de-identification @xcite . to address challenges in lowresource settings, we use self-supervised (ssl) trained xlsr-53 (conneau et al., 2020) model as an encoder in the transducer setup, leading to the xlsr-transducer (fig. 1b ). previous works aims at modeling several tasks directly from speech using special tokens @xcite , or asr with speaker change detection (scd) @xcite @xcite , vad @xcite , speechto-text translation @xcite , or timestamps @xcite , ner @xcite and multi-speaker asr @xcite . tokenbased multitasking offers multiple benefits, e.g., it has a fix number of parameters while all tasks are predicted with standard decoding without increased latency. however, nlp tasks like ner in conjunction with other tasks from audio domains have not received much attention in the literature. therefore, we consider 3 additional tasks alongside asr: scd, endpointing and ner. these tasks are selected to represent both audio and nlp domains. scd is an audio task @xcite . endpointing can be viewed as an nlp task when conducting semantic endpointing @xcite , or as an audio task @xcite . ner is an nlp task @xcite . they serve as suitable benchmarks for evaluating our proposed method. ## tokenverse through tokenverse, we aim to train a single model for asr (main task), speaker change detection (scd), endpointing, and named entity recognition (ner). this is achieved by augmenting the reference text, with task tokens that denote special events at the acoustic level. ## task-specific baselines, metrics & evaluation protocol in this section, we describe strong independent baselines for each task considered in this work. ## results & discussion automatic speech recognition for the de-finedai (tab. 2) set, including all tasks in tokenverse (exp 2) leads to a 4% relative improvement in wer compared to the baseline asr model ## conclusions in this paper, we show the effectiveness of a tokenbased multitask model on speech and nlp using xlsr-transducer as our asr model, termed to-kenverse. alongside asr, speaker change detection, endpointing and named entity recognition are considered. results on 2 datasets show that our approach improves asr performance while outperforming strong task-specific baselines. ablation experiments suggest that multitask training across different domains can enhance performance on all tasks. our approach offers flexibility for extension to numerous tasks across various domains.