End-to-end ASR to jointly predict transcriptions and linguistic annotations (2021)

Abstract: We propose a Transformer-based sequence-to-sequence model for automatic speech recognition (ASR) capable of simultaneously transcribing and annotating audio with linguistic information such as phonemic transcripts or part-of-speech (POS) tags. Since linguistic information is important in natural language processing (NLP), the proposed ASR is especially useful for speech interface applications, including spoken dialogue systems and speech translation, which combine ASR and NLP. To produce linguistic annotations, we train the ASR system using modified training targets: each grapheme or multi-grapheme unit in the target transcript is followed by an aligned phoneme sequence and/or POS tag. Since our method has access to the underlying audio data, we can estimate linguistic annotations more accurately than pipeline approaches in which NLP-based methods are applied to a hypothesized ASR transcript. Experimental results on Japanese and English datasets show that the proposed ASR system is capable of simultaneously producing high-quality transcriptions and linguistic annotations.

## introduction end-to-end automatic speech recognition (e2e asr), which transcribes speech using a single neural network (nn), has recently gained traction @xcite @xcite @xcite . existing e2e asr models generate audio transcripts by sequentially producing likely graphemes, or multi-graphemic units, from which lexical items of a language can be recovered. however, other linguistic annotations such as phonemic transcripts, part-of-speech (pos) tags, or word boundaries, help understand the underlying audio characteristics @xcite . such linguistic annotations are especially important in natural language processing (nlp) tasks done on audio in o2m models shown in fig. 1 (a), a multitask objective is used in which an extra branch is tasked with estimating the secondary label sequence. for example, in @xcite , the phonemic transcript is produced in addition to the graphemic transcript. the o2m model can estimate each sequence more accurately than separate models responsible for producing phonemic and graphemic transcripts independently. we can implement this approach with less effort by attaching multiple loss functions to the base architecture. however, this o2m model does not explicitly consider dependencies between phonemic and graphemic transcripts. furthermore, aligning phoneme and grapheme sub-sequences requires additional post-processing based on time alignment or alignment across the multiple sequences during inference. performance of downstream nlp tasks built on top of asr outputs will suffer if this post-processing fails to generate alignment. fig. 1(b ) shows an o2o model with a conditional chain mapping. this method for multiple sequence modeling has been applied to dialog modeling @xcite , speaker diarization (fujita et al., 2020a), and multi-speaker asr @xcite . unlike the o2m model, this model can predict a variable number of output sequences while explicitly considering dependencies between the multiple sequences based on the probabilistic chain rule. however, modeling these inter-sequence dependencies requires more complicated neural architectures, and alignment of the sequences still requires post-processing during inference. another option for using o2o models is to output multiple sequences as a single sequence instead of using conditional chain mapping, as shown in fig. 1(c ). for example, in @xcite , the o2o model produces word transcripts by first generating a word's constituent graphemes followed by the word itself. another application, explored in @xcite used the o2o model to produce graphemes followed by speaker role. this approach is the simplest to implement because we can reuse the neural network architecture used to produce the primary sequence to sequence mapping to produce the secondary label sequence (e.g., connectionist temporal classification (ctc) based systems). in contrast to the previous two approaches, the o2o model does not require postprocessing to align the label sequences during inference since the output sequence preserves the alignment between the word and corresponding annotation labels; alignment is only needed for the data preparation stage during training to produce the appropriate target sequences. for this reason, we used the o2o model in this study. this paper proposes to use a state-of-the-art transformer-based e2e asr system @xcite for the o2o model with a single sequence, instead of ctc-based approaches which are frequently supported @xcite . compared with the ctc-based systems, this approach can explicitly model the relationship between the output labels thanks to the autoregressive decoder network, similar to the conditional chain rule model in fig. 1(b) . we also demonstrate improved performance compared to the ctc-based systems. another contribution is that we conducted an extensive empirical evaluation to analyze and demonstrate the utility of our approach. for example, we applied the method to english and japanese asr tasks in which phonemic transcripts and pos tags are simultaneously produced. our approach predicts linguistic annotations correctly even though corresponding graphemes are wrong, while the pipeline approach, in which nlp-based methods are applied to a hypothesized asr transcript, fails. this feature is helpful for the downstream nlp system like slot filling or intent detection. besides, our approach is suitable for on-device applications because the e2e model archives small-footprint prediction @xcite . note that our primary goal is to provide aligned transcripts and linguistic annotations with minimal degradation in asr performance. we are not aiming to improve asr performance. the features of the proposed method are summarized as follows: ## conclusion and future work we proposed a novel e2e asr transformer system for simultaneously estimating transcriptions and linguistic annotations such as phonemic transcripts or pos tags. this paper showed that the proposed asr could estimate these features with sufficient performance and also showed reasonable phoneme and grapheme analyses and attention patterns thanks to the aligned output of both output symbols. in future work, we will extend the proposed approach to predict other linguistic annotations such as named entities.