CUNI Neural ASR with Phoneme-Level Intermediate Step for~ N on- N ative~ SLT at IWSLT 2020 (2020)

Abstract: In this paper, we present our submission to the Non-Native Speech Translation Task for IWSLT 2020. Our main contribution is a proposed speech recognition pipeline that consists of an acoustic model and a phoneme-to-grapheme model. As an intermediate representation, we utilize phonemes. We demonstrate that the proposed pipeline surpasses commercially used automatic speech recognition (ASR) and submit it into the ASR track. We complement this ASR with off-the-shelf MT systems to take part also in the speech translation track.

## introduction this paper describes our submission to non-native speech translation task in iwslt 2020 @xcite . we participate in two sub-tracks: offline speech recognition and offline speech translation from english into czech and german. we focus on the speech recognition, proposing a robust pipeline consisting of two componentsan acoustic model recognizing phonemes, and a phoneme-to-grapheme translation model, see @xcite 1 . we decided to use phonemes as the intermediate representation between the acoustic and the translation model because we believe that conventional grapheme representation is too constrained with complicated rules of mapping speech to a transcript. this issue becomes immense when dealing with dialects and non-native speakers. both models used in our pipeline are end-toend deep neural networks, jasper @xcite for the acoustic model and transformer @xcite for the phoneme-to-grapheme translation model. for punctuating, truecasing, segmenting and translation into czech and german, we use offthe-shelf systems provided by elitr project. the paper is organized as follows: section 2 reviews related work. in sections 3 and 4 we describe models for our speech recognition pipeline and their training. in section 5, we describe the punctuator, truecasor and segmenter, and machine translation into czech and german in section 6. we summarize our submissions in section 7 and conclude in section 8. ## related work this section reviews the related work. ## neural asr with phoneme-level intermediate step our main idea is to couple an end-to-end acoustic model with a specialized "translation" model, which translates phonemes to graphemes and corrects the asr errors. the motivation for the translation step is that the translation model can exploit larger context than a basic convolutional acoustic model. furthermore, we can utilize considerably larger non-speech corpora to train this part of the pipeline. ## phoneme-to-grapheme model we seek a model for translating transcripts written in phonemes into graphemes in the same language. unlike the most studies reviewed in section 2, we propose to use transformer @xcite architecture for phoneme-to-grapheme translation. we believe that transformer is the best option for these tasks. transformer has shown its potential in many nlp tasks. most importantly, we consider its ability to learn the structure of a sentence, see e.g. @xcite . ## punctuation, truecasing and segmentation our asr system produces lowercased, unpunctuated text, but the machine translation works on capitalized, punctuated text, segmented to individual sentences. we use the same birnn punctuator, truecaser and segmenter as @xcite . the punctuator is a bidirectional recurrent neural network by @xcite trained on the english side of czeng @xcite . the truecaser uses tri-grams @xcite . we use a rule-based moses sentence splitter @xcite . more details are in @xcite , section 4.2. ## machine translation our submission to the slt track relies on the mt systems, which are used also by elitr project and are described in their submission to this task @xcite . we do not rely on their validation for this task. as our primary mt systems, we select "wmt18 t2t" for czech and "de t2t" for german, because they were easily accessible name initialization lm rescoring encoder decoder asr (primary) random random no asr lm random random yes asr slt en cs cs en no asr slt lm en cs cs en yes bert en cs bert no we submit also all other machine translation systems for czech and german by elitr with our "asr" source for contrastive evaluation. see @xcite for more details. ## submission summary we participate in two tracks of the non-native speech translation task: speech recognition, and speech translation into both czech and german. in both cases, our submissions are off-line. the acoustic model was initialized from a checkpoint trained on other data than allowed for the task. therefore, our systems are unconstrained. for the speech recognition track, we utilize our speech recognition pipeline in various configurations. we first obtain the phoneme transcripts using the acoustic model. for configurations marked with " lm", we additionally use a phoneme language model during the acoustic model inference. subsequently, we feed these phonetic transcripts to the phoneme-to-grapheme translation model. we have three variants of this model: plain ("asr"), with pre-trained weights from slt ("slt"), and with pre-trained weights from slt for encoder and bert for decoder ("bert"). in this manner, we yield five different configurations for submission (see we do not have our own translation model. to participate in the translation track, we utilize the mt systems of the elitr project, which are mostly transformer neural models. we select as our primary submission the "asr" system.