JHU IWSLT 2022 Dialect Speech Translation System Description (2022)

Abstract: This paper details the Johns Hopkins speech translation (ST) system used in the IWLST2022 dialect speech translation task. Our system uses a cascade of automatic speech recognition (ASR) and machine translation (MT). We use a Conformer model for ASR systems and a Transformer model for machine translation. Surprisingly, we found that while using additional ASR training data resulted in only a negligible change in performance as measured by BLEU or word error rate (WER), aggressive text normalization improved BLEU more significantly. We also describe an approach, similar to back-translation, for improving performance using synthetic dialectal source text produced from source sentences in mismatched dialects.

## introduction in this paper we describe the jhu dialect speech translation submissions and their development. dialects are varieties of a language spoken by a group of people, often in a specific geographic location. in many languages, standard rules of pronunciation, orthography and syntax, but also available data resources are drawn from a single dominant dialect. a challenge for all language technologies, including automatic speech recognition (asr), machine translation (mt), and speech translation (st), is how to deal with non-standard dialects for which no formal orthography, grammar, or even data exist. because many dialects are rarely if ever written, evaluation of asr and mt on dialect speech is not even particularly well defined. however, there are no such problems evaluating speech translation on dialect speech, which here refers to the task of producing target language text from source language audio inputs. a focus of both the dialect speech translation task and our system development, is how to leverage available resources from the standard dialect * equal contribution. to improve performance on non-standard dialects. the dialect translation task focuses specifically on tunisian arabic. arabic and its dialects lie on a dialect continuum unified by a single standardized dialect, modern standard arabic (msa) @xcite . msa is the primary language of formal and written communications (e.g. news broadcasts, parliaments and religion). however, most native arabic speakers use local dialects in daily life, which generally lack a standard written form. certain dialects, such as algerian, tunisian, and moroccan arabic also have strong romance, and berber substrates, and may exhibit a high degree of code-switching, especially with french. traditionally, speech translation systems have been built by cascading asr and mt models to form a speech translation chain @xcite . however, the more recent end-to-end approach @xcite , which directly translates the source speech to target text, is appealing for this task since since both asr and mt are ill-defined for unwritten spoken dialects, and there were relatively large amounts of translated speech (∼160 hrs). we found, somewhat surprisingly during initial experimentation (see rows 1,2 of our systems incorporated three improvements over the provided baseline. 1. we aggressively normalized the tunisian arabic transcripts, which led to improved mt performance. 2. we use additional msa bi-text by pretraining models on these data using a shared bpe model with a large number of bpe units for both the msa and tunisian data. 3. we show that training on synthetic tunisian source sentences instead of the msa source sentences provides small improvements. ## the dialect speech translation task the dialect speech translation task permitted submissions using models trained assuming different resource constraints, called: (a) basic, (b) dialect adaptation, and (c) unconstrained. we refer to these conditions as (a), (b) and (c) in the rest of the paper. ## methods we model the speech translation problem as a two step process. first, input audio is converted to source language text via an asr model. next, an mt model, which may have been trained on entirely different data from the asr model, is used to translate the asr output transcript into target language sentences. this model is known as a cascade model. while cascade models suffer from a few well known problems, such as compounding error and inability to make direct use of the acoustic signal to improve translation quality, their modularity facilitates training on and incorporation of additional resources such as transcribed speech, bi-text, monolingual text, and unlabeled source language audio. we describe how we used these available resources to train the asr and mt models in our st cascade in each data condition. ## experiments to test our approach, we conducted experiments on the asr, mt, and st tasks. in all experiments, unless otherwise stated we performed additional text normalization in order to reduce some of the orthographic variation in the tunisian transcripts. in all experiments and for all languages / dialects, we remove punctuation, using the scripts provided by the organizer. 1 for both tunisian and msa, we convert eastern arabic digits to western arabic digits, and remove diacritics and single character words. we also perform alif/ya/ta-marbuta normalization, which removes distinctions within three sets of characters that are often written inconsistently in dialect arabic and even sometimes in modern standard arabic: alif forms ( @xmath0, , , ), ya forms ( @xmath1, @xmath2, and ta-marbuta forms ( @xmath3, @xmath4. for english, we keep all the text in lowercase, as the evaluation is performed on lowercased english text, and we use moses @xcite ) for text tokenization. it is difficult to assess the normalization affect on the quality of the asr. however, we can measure its effect on the downstream task of translation, described in section 4.2. ## conclusion we have detailed the our submission for the iwslt 2022 dialect speech translation task. we briefly compared end-to-end to cascaded systems and found that cascaded models were slightly outperforming their end-to-end counterparts despite, a relative abundance of training data. we demonstrated that increased text normalization, and back-translation to reduce dialect mismatch improved speech translation performance. finally, we described two ways of using extra mismatched dialect resources and found surprisingly that using additional unlabeled data through the use of the xlsr-53 model resulted in only small improvements. using additional large labeled msa resources resulted in slight improvements to the asr, and modest improvements in mt. future work should expand upon the backtranslation results to determine the optimal method for minimizing the dialect mismatch when augmenting training with additional bi-text.