The AFRL IWSLT 2020 Systems: Work-From-Home Edition (2020)

Abstract: This report summarizes the Air Force Research Laboratory (AFRL) submission to the offline spoken language translation (SLT) task as part of the IWSLT 2020 evaluation campaign. As in previous years, we chose to adopt the cascade approach of using separate systems to perform speech activity detection, automatic speech recognition, sentence segmentation, and machine translation. All systems were neural based, including a fully-connected neural network for speech activity detection, a Kaldi factorized time delay neural network with recurrent neural network (RNN) language model rescoring for speech recognition, a bidirectional RNN with attention mechanism for sentence segmentation, and transformer networks trained with OpenNMT and Marian for machine translation. Our primary submission yielded BLEU scores of 21.28 on tst2019 and 23.33 on tst2020.

## introduction as part of the evaluation campaign for the 2020 international workshop on spoken language translation (iwslt) @xcite , the afrl human language technology team submitted an entry to the offline spoken language translation (slt) task. the goal of this task is to automatically generate cased and punctuated german translations from english audio ted talks using either a cascade of systems or the end-to-end approach. we chose to build upon our previous work @xcite and adopt the cascade approach of using separate systems to perform speech activity detection, automatic speech recognition (asr), sentence segmentation, and machine translation (mt). sections 2 and 3 describe our asr and mt systems, respectively. section 4 presents our results on the development set when the data is manually segmented into sentences, and section 5 descibes our approach to slt on unsegmented data. section 6 provides a post-evaluation analysis of our systems based on sentence length, and section 7 presents our conclusions and future work. ## automatic speech recognition this section describes the english asr system that was developed for the offline speech translation task. first, we sequestered all talks from tedlium-v3 @xcite that were present in tst2014, tst2015, and tst2018. next, language models (lms) were estimated on tedlium-v3 and the same subsets of news crawl and news discussions described in @xcite . the text was formatted as follows: we used the srilm select-vocab tool foot_0 to choose a 100,000 word vocabulary. an interpolated bigram language model (lm) was estimated using the srilm toolkit, and a recurrent neural network (rnn) lm was trained using kaldi @xcite . acoustic models were trained on tedlium-v3 using kaldi. in a preliminary experiment, we found that training on both tedlium-v3 and commonvoice did not lead to a reduction in word error rate (wer), so we decided to only use tedlium-v3. the kaldi system used in these experiments is a factorized time delay neural network (tdnn) with residual network style skip connections. input mel frequency cepstral coefficient 104 (mfcc) features have standard speed perturbation applied (0.9 & 1.1 factor). the initial kaldi finite state transducer (fst) was built with the bigram lm, and the resulting lattices were rescored using the pruned rnnlm lattice rescoring algorithm @xcite . ## machine translation in order to translate the asr output from the previous section, we construct an english-german mt training corpus from allowable sources provided by the organizers foot_1 . we then prepare this corpus in a similar manner as described in gwinnup et al. (2018) and gwinnup et al. (2019) , especially focusing on fasttext @xcite language-id filtering. as a contrast to @xcite we prepare data for additional systems on this same corpus where the source english text has been transformed to resemble output from our asr systems. we then train transformer @xcite based mt systems with the opennmt @xcite and marian (junczys-dowmunt et al., 2018) toolkits. ## manual segmentation in order to evaluate the effect of automatic sentence segmentation on spoken language translation, we manually segmented the tst2014, tst2015, and tst2018 development sets into sentences using the provided reference files. this was done by automatically aligning the reference text using a kaldi asr system and then manually correcting any errors. the kaldi system described in section 2 was then used to generate asr transcripts for each utterance. note that for asr tasks, a development set is typically used to select the lm scale that minimizes the wer; however, in this task our goal is to choose the best translation. we decided to generate 8 different hypotheses for each utterance by varying the asr lm scale over 6, 8, 10, ..., 20, translating each utterance, and then selecting the asr lm scale that yields the best overall bleu score. each asr hypothesis was translated using the mt systems trained on lowercase non-punctuated english text. compared to selecting the asr lm scale to minimize wer, this method yields a very minor improvement with marian (0.06 bleu on tst2014 and tst2018, 0.17 bleu on tst2015), but no improvement with opennmt. in a second set of experiments, an automatic punctuator and text recaser were applied to the english asr text prior to performing translation. compared to the previous approach, one advantage of this method is that we can train a single mt system to translate both asr transcripts and text documents. the punctuator was a bidirectional rnn with attention mechanism that was trained on 4m words of english ted data using the python fork of ottokar tilk's punctuator. 3 the punctuated text was recased using moses and then translated 3 available at: https://pypi.org/project/punctuator opennmt tst2014 23.77 (6.2) tst2015 24.26 (6.5) tst2018 22.90 (8.6) ## automatic segmentation in the previous section, we evaluated our asr and mt systems on audio clips that were manually segmented into sentences. this section considers the task where we are given an audio stream that must be automatically segmented. first, we evaluated a speech activity detector (sad) on each audio file. we used the same neural network based sad as described in our iwslt 2018 paper. automatic segmentation of the test data was performed by evaluating the sad, applying a dynamic programming algorithm to choose the best sequence of states, and defining utterance boundaries at the midpoint of each non-speech segment longer than 0.5 seconds. next, the kaldi asr system was evaluated on each utterance using the same asr lm scales found in the previous section. two different methods were investigated for partitioning the time-aligned words into sentences. in the first method, we simply used the utterance boundaries from the sad to define the sentence boundaries. for the second method, we evaluated the automatic punctuator from section 4 on each utterance, and then defined additional sentence boundaries at words that ended with a period, exclamation, or question mark. our primary submission to the iwslt 2020 offline speech translation task can be summarized as follows. first, a neural network based sad was used to segment each audio file into utterances. next, asr transcripts were generated using kaldi and an automatic punctuator was applied to further split each utterance into sentences. lastly, an opennmt system was used to translate the lowercase non-punctuated english into cased and punctuated german. as a contrasting system, we also submitted the translations obtained using marian. the processing pipeline for the marian system was identical to the opennmt system, except that we did not apply the automatic punctuator (i.e., the sentence boundaries were defined solely on the pause durations from the sad). the opennmt system yielded bleu scores of 21.28 on tst2019 and 23.33 on tst2020; the marian system yielded bleu scores of 21.48 on tst2019 and 23.21 on tst2020. ## post-evaluation analysis in section 5 we found that defining additional sentence boundaries using an automatic punctuator led to a worse performance with marian, but improved performance with opennmt. this led us to wonder if the automatic punctuator was actually helping to identify more correct sentence boundaries, or simply producing shorter sentences that were better translated with opennmt. based on this idea, we decided to analyze how sentence length affects translation performance with each of our systems. first, the english reference text from dev2010, tst2010, tst2013, tst2014, tst2015, and tst2018 was processed using the same steps as described in section 2 to match the expected mt input. marian and opennmt were then used to translate each sentence, and the bleu score was calculated for sentences where the english source included 1-9, 10-19, 20-29, 30-39, 40-49, and 50+ words. ## conclusion and future work with our systems, we found that automatic sentence segmentation led to a decrease of up to -2.51 bleu. the punctuator that we used provides functionality to specify the pause duration after each word when training the punctuator. this could be obtained by automatically aligning the original ted training transcripts; however, due to limited computational resources while working at home, we were not able to investigate this feature. in addition to text features and pause durations, other acoustic features might also prove useful for automatically identifying sentence boundaries. alternatively, it may be interesting to resegment the mt training data to better match the asr segmentation, although this would probably have to be done in an automatic fashion to take advantage of available text-only parallel corpora.