T RANSLATOTRON 2: R OBUST DIRECT SPEECH - TO - SPEECH TRANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We present Translatotron 2 , a neural direct speech-to-speech translation model that can be trained end-to-end.
Translatotron 2 consists of a speech encoder, a phoneme decoder, a mel-spectrogram synthesizer and an attention module that connects all the previous three components.
Experimental results suggest that
Translatotron 2 outperforms the original Translatotron by a large margin in terms of translation quality and predicted speech naturalness, and drastically improves the robustness of the predicted speech by mitigating over-generation, such as babbling or long pause.
We also propose a new method for retaining the source speaker’s voice in the translated speech.
The trained model is restricted to retain the source speaker’s voice, but unlike the original Translatotron, it is not able to generate speech in a different speaker’s voice, making the model more robust for production deployment, by mitigating potential misuse for creating spooﬁng audio artifacts.
When the new method is used together with a simple concatenation-based data augmentation, the trained Translatotron 2 model is able to retain each speaker’s voice for input with speaker turns.
I NTRODUCTION
Speech-to-speech translation (S2ST) is highly beneﬁcial for breaking down communication barriers between people not sharing a common language.
Conventional S2ST systems are often composed of a cascade of three components: automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis (Lavie et al., 1997; Wahlster, 2000; Nakamura et al., 2006).
Very recently, direct speech-to-text translation (ST) is rapidly emerging, and has outperformed the cascade of ASR and MT (Weiss et al., 2017;
Jia et al., 2019a; Di Gangi et al., 2019; McCarthy et al., 2020; Wang et al., 2021b;
Ansari et al., 2020; Anastasopoulos et al., 2021), which makes the cascade of ST and TTS as S2ST feasible (Jia et al., 2019b).
However, works on direct S2ST is very limited.
Compared to cascaded systems, direct S2ST has the potential beneﬁts of: 1) retaining paralinguistic and non-linguistic information during the translation, such as speaker’s voice (Jia et al., 2019b), emotion and prosody; 2) working on languages without written form (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2021a); 3) reduced computational requirements and lower inference latency; 4) avoiding error compounding across sub-systems; 5) easier on handling contents that do not need to be translated, such as names and proper nouns (Jia et al., 2019b).
Translatotron (Jia et al., 2019b) is the ﬁrst model that is able to directly translate speech in one language to speech in another language.
It is also able to retain the source speaker’s voice in the translated speech.
However, its translation quality underperforms a cascade baseline by a large margin, and its synthesized translated speech suffers from robustness issues, such as babbling and long pause, which is an issue shared with the Tacotron 2 TTS model (Shen et al., 2018;
He et al., 2019; Zheng et al., 2019; Battenberg et al., 2020; Shen et al., 2020), since they share the same attention-based approach for synthesizing speech.
In this work we present Translatotron 2 .
The main contributions include:
We propose Translatotron 2 , a novel direct S2ST model that is able to be trained end-to-end; 2.
We conduct experiments suggesting that Translatotron 2 signiﬁcantly outperforms Translatotron, and is comparable to a cascade system, in terms of translation quality, speech naturalness, and speech robustness;
We propose a new method for voice retention in S2ST without relying on any explicit speaker embedding or ID.
The trained model is only able to retain the source speaker’s voice but not able to generate speech in a different speaker’s voice, making it more robust for production deployment by mitigating potential misuse for creating spooﬁng audio artifacts; 4.
We propose a simple concatenation data augmentation, ConcatAug , to enable Translatotronto retain each speaker’s voice when the input includes speaker turns; 5.
We conduct experiment suggesting that Translatotron 2 is efﬁcient on multilingual direct S2ST, in which it obtaines translation quality very close to an ST baseline.
Audio samples from Translatotron 2 are available in the supplementary material.
R ELATED WORKS
S2ST Until very recently, S2ST systems are typically composed of a cascade of ASR, MT, and TTS components (Lavie et al., 1997; Wahlster, 2000; Nakamura et al., 2006; ITU, 2016).
Translatotron (Jia et al., 2019b) is the ﬁrst direct S2ST model, which is a sequence-to-sequence model trained endto-end in a multi-objective task.
It has shown reasonable translation quality and speech naturalness, but still underperformed a baseline of ST + TTS cascade by a large margin.
It also demonstrated the capacity of retaining speaker’s voice during the translation, by leveraging a speaker encoder separately trained in a speaker veriﬁcation task (Wan et al., 2018;
Jia et al., 2018).
A few recent works proposed cascade S2ST systems using learned discrete representation as the intermediate representation instead of text or phoneme.
Tjandra et al.
(2019) introduced such an S2ST system that ﬁrst translated the source speech into a discrete representation of the target speech which was predicted from a separately trained VQ-VAE (Oord et al., 2017), and then used the VQ-VAE decoder to predict the target speech spectrogram from the discrete representation.
Zhang et al.
(2021) additionally trained the VQ-VAE jointly with a supervised phoneme recognition objective in different languages.
Lee et al.
(2021a) used a separately trained vocoder to directly predict waveform from the discrete representation without relying on spectrogram; for the best performance, this vocoder included a duration predictor and an upsampler, akin to a generative TTS model.
All these works require multiple components being trained in multiple steps, but are not able to be trained end-to-end.
Kano et al.
(2021) introduced an end-to-end S2ST model with a cascade of three auto-regressive decoders, and used pre-trained MT and TTS models as teacher models to facilitate the training of the end-to-end model.
It requires pre-trained ASR, MT, and TTS models, and the end-to-end model itself has to be trained in multiple steps.
Unfortunately, despite that these recent works generated translated speech in novel ways without relying on TTS subsystems, most of these works (except for Jia et al. (2019b)) focused only on the translation quality, but did not assess the perceptual quality (Wagner et al., 2019) of the translated speech (e.g. naturalness), which is critical to S2ST.
TTS Translatotron uses a decoder similar to Tacotron 2 (Shen et al., 2018;
Jia et al., 2018), which is an attention-based auto-regressive decoder.
Due to the ﬂexibility of the attention mechanism, they both suffer from robustness issues such as over-generation.
Recent TTS models such as FastSpeech (Ren et al., 2019; 2021), Non-Attentive Tacotron (NAT)
(Shen et al., 2020;
Jia et al., 2021)
and
Parallel Tacotron (Elias et al., 2021b;a), demonstrate that replacing the attention module with a duration-based upsampler yields more robust synthesized speech, as quantitatively evaluated at a large scale in Shen et al.
The synthesizer component in this work resembles these works.
Voice conversion and anti-spooﬁng The performance of voice conversion (VC) has progressed rapidly in the recent years, and is reaching a quality that is hard for automatic speaker veriﬁcation (ASV) systems to detect (Yi et al., 2020).
ASVspoof 2019 (Todisco et al., 2019; Wang et al., 2020) found that it was challenging to detect spoof audios generated from Jia et al.
(2018), which uses the same speaker encoder-based approach as in the original Translatotron.
Such progress poses concerns on related techniques being misused for creating spooﬁng artifacts.
We designed Translatotron 2 with the motivation of avoiding such potential misuse. [Figure]
T RANSLATOTRON
The motivation of the architectural design of Translatotron 2 is to improve on three weaknesses existing in the original Translatotron: 1)
The valuable auxiliary supervision on textual features is not fully utilized during training; 2) Long-sequence-to-long-sequence modelling with attention is difﬁcult because of the ﬂexibility of the attention mechanism; 3) Attention-based speech generation is known to suffer from robustness issues such as over-generation and under-generation.
The proposed Translatotron 2 model solves these problems by using an architecture composed of a source speech encoder, a target phoneme decoder, and a target mel-spectrogram synthesizer.
These three components are connected together by a single attention module.
See Figure 1 for an illustration.
The model is jointly trained with a speech-to-speech translation objective and a speech-to-phoneme translation objective.
As the result, the auxiliary supervision is fully utilized, and the attention is used only for modeling shorter target phoneme sequence but is not directly involved in speech generation.
The following subsections describe the architecture of each components as used in our main experiments.
As shown in Appendix D, using alternative architectures for these components do not signiﬁcantly impact the performance of Translatotron 2.
3. E NCODER
The encoder of Translatotron 2 takes mel-spectrogram of the source speech as the input, and produces a hidden representation.
We use Conformer (Gulati et al., 2020) as the architecture for the encoder.
It ﬁrst subsamples the input mel-spectrogram with a convolutional layer, and then processes it with a stack of Conformer blocks.
Each Conformer block is composed of a feed-forward layer, a selfattention layer, a convolution layer, and a second feed-forward layer.
SpecAugment (Park et al., 2019) is applied at the training time.
3. D ECODER
The autoregressive decoder is assisted with an attention module.
It takes the encoder output as the source values for the attention, and predicts phoneme sequences corresponding to the target translated speech.
We use a stack of LSTM cells as the decoder, along with a multi-head attention (Vaswani et al., 2017).
The combination of the encoder, the decoder, and the attention module is similar to a typical ST model, except that it predicts phonemes instead of subword tokens.
3. S YNTHESIZER
The synthesizer takes the hidden output from the decoder, as well as the context output from the attention module as its input, and synthesizes the target mel-spectrogram.
It is similar to the decoders in typical neural TTS models, such as in Shen et al.
(2018; 2020); Ren et al.
The predicted mel-spectrogram can be converted into waveform using either an estimation algorithm such as Grifﬁn & Lim (1984) or a neural vocoder.
We experimented with a number of different non-attentive architectures for the synthesizer, including autoregressive ones and parallel ones, and found that autoregressive synthesizers worked best.
We followed the architecture and the hyperparameters of the mel-spectrogram decoder from NAT (Shen et al., 2020).
It ﬁrst predicts a duration for each of its input elements using a bidirectional LSTM stack, then upsamples the input sequence with Gaussian weights based on the predicted duration, and ﬁnally uses an auto-regressive LSTM stack to predict the target mel-spectrogram.
The predicted melspectrogram is further reﬁned with a residual convolutional post-net.
This synthesizer is trained jointly with a mel-spectrogram reconstruction loss and a duration prediction loss.
Figure 1(b) illustrates the architecture of this synthesizer.
Unlike in Shen et al.
(2020), we do not use per-phoneme duration labels for training the duration predictor, nor do we apply the FVAE-based alignment.
Instead, only an L 2 loss on the total predicted duration of the entire sequence is used (i.e. the “na ¨ ıve approach” of unsupervised duration modelling in Shen et al.
We anticipate that adopting these approaches could further improve the performance of Translatotron 2, which we leave as future work.
V OICE RETENTION
The original Translatotron (Jia et al., 2019b) demonstrated the capacity of being able to retain the source speaker’s voice in the translated speech, by conditioning its decoder on a speaker embedding generated from a separately trained speaker encoder.
In fact, it is capable of generating the translated speech in a different speaker’s voice, as long as a clip of the target speaker’s recording is used as the reference audio to the speaker encoder, or the embedding of the target speaker is directly available.
While this is impressively powerful, it can potentially be misused for generating spooﬁng audio with arbitrary content, posing a concern for production deployment.
In this work, we use a different approach for retaining the source speaker’s voice in the translated speech.
The trained model is restricted to retaining the source speaker’s voice, but not able to generate speech in a different speaker’s voice.
4. T RAINING TARGETS IN SOURCE SPEAKER ’ S VOICE
To retain speakers’ voices across translation, we train S2ST models on parallel utterances with the same speaker’s voice on both sides.
Such a dataset with human recordings on both sides is extremely difﬁcult to collect, because it requires a large number of ﬂuent bilingual speakers.
Instead, we use a
TTS model that is capable of cross-lingual voice transferring to synthesize such training targets.
We modiﬁed the PnG NAT (Jia et al., 2021) TTS model by incorporating a separately trained speaker encoder (Wan et al., 2018) in the same way as Jia et al.
(2018), and trained it on the LibriTTS corpus (Zen et al., 2019).
The result TTS model is capable of zero-shot voice transferring, but synthesizes in a better quality and more robust than Jia et al.
We used this model to synthesize target speech in the source speaker’s voice in our experiments.
Other TTS models capable of cross-lingual voice modelling, such as Zhang et al.
; Chen et al.
; Xin et al.
(2021), could also be utilized.
4. S PEAKER TURNS
Theoretically, because the target-spectrogram synthesizer in both Translatotron 2 and Translatotron are directly conditioned on the source-spectrogram encoder output, the encoder output may be capable of preserving voice information locally in together with linguistic information, and the decoders and synthesizers may be capable of utilizing such local information for translating linguistic information while preserving local voice information. As a result, such direct S2ST models may be capable of [Table]
[Figure] [Table] [Figure] retaining each source speaker’s voice on input with speaker turns.
However, proper training data is required to enable such models to learn so. [Figure]
ConcatAug To enable direct S2ST models to preserve each speaker’s voice for input with speaker turns, we augmented the training data by randomly sampling pairs of training examples and concatenating the source speech, the target speech, and the target phoneme sequences into new training examples.
The result new examples contain two speakers’ voices in both the source and the target speech, which enables the model to learn on examples with speaker turns.
See Figure 2 for an example of such concatenation and the prediction from Translatotron 2 on it.
Such augmentation does not only enable the model to learn voice retention on speaker turns, but also increases the diversity of the speech content as well as the complexity of the acoustic conditions in the training examples, which can further improve the translation quality of the model, especially on small datasets (as shown in Section 5.1).
Narayanan et al.
(2019) uses a similar augmentation but in a more complicated fashion, for improving ASR performance on multi-speaker inputs.
E XPERIMENTS
We conducted experiments on three datasets, including two Spanish → English datasets and a multilingual → English dataset.
All datatsets use TTS synthesized target speech with 24 kHz sample rate.
The phonemes used only at training time were converted from the transcripts using a proprietary G2P system.
See Table 1 for the details of each dataset.
We evaluated the translation quality, naturalness and robustness of the predicted speech, as well as speaker similarity for voice retention.
Unless described otherwise, in all the experiments, Translatotron 2 models use a 16-layer Conformer encoder with 144 dimension following Gulati et al.
(2020), a 4-layer LSTM decoder, and a RNNbased synthesizer following Shen et al.
A comprehensive table of hyperparameters in available in Appendix A. All models were implemented using the Lingvo framework (Shen et al., 2019).
5. T RANSLATION QUALITY
The same two datasets from Jia et al. (2019b) were used for evaluating the translation quality of
Translatotron 2 when predicts translated speech is in a single female speaker’s voice.
In contrast to
[Table]
Jia et al.
(2019b), we did not augment the source speech to add background noise or reverberation, and we consistently use 80-channel mel-spectrogram as input and 128-channel mel-spectrogram as output for all Translatotron 2 models.
The predicted mel-spectrogram is converted into waveform using the Grifﬁn-Lim algorithm (Grifﬁn & Lim, 1984).
Following Jia et al.
(2019b)
, the translation quality is measured by BLEU on ASR transcribed text (lower case, no punctuation marks).
Because ASR makes errors, such BLEU can be thought a lower bound of the translation quality.
We used an ASR model from Park et al.
(2020), trained on
LibriSpeech (Panayotov et al., 2015) and LibriLight (Kahn et al., 2020) corpora.
We retrained the baseline Translatotron strictly following Jia et al.
(2019b)
and re-evaluated it with this ASR model. As shown in Table 2, the translation quality from Translatotron 2 outperformed the original Translatotron by a large margin and got very close to a strong cascade system.
Applying ConcatAug further improved the translation quality of Translatotron 2 on the smaller Fisher dataset.
The original Translatotron uses the phoneme sequences from both the source and the target sides as auxiliary training tasks.
Translatotron 2 uses only the target phonemes, yet yield better translation quality than Translatotron especially on the smaller Fisher dataset, indicating it is more data efﬁcient. Manual error case analysis revealed high consistency between the target speech prediction and the target phoneme prediction, suggesting more headroom for improving translation quality lies in the encoder and the decoder of Translatotron 2, instead of the synthesizer.
Potential approaches to take include utilizing beam search, self-supervised pre-training (Baevski et al., 2020; Wang et al., 2021b), self-training (Park et al., 2020; Wang et al., 2021b), and weakly supervised data (Jia et al., 2019a).
5. S YNTHESIS NATURALNESS
The naturalness of the predicted speech is evaluated by subjective listening test, reporting 5-scale mean opinion scores (MOS) with 95% conﬁdence interval on 1,000 randomly sampled predictions.
A WaveRNN-based neural vocoder (Kalchbrenner et al., 2018) was used for converting the melspectrograms predicted from S2ST models into waveforms.
The result is reported in Table 2.
The naturalness of the speech predicted from Translatotron 2 is signiﬁcantly better than from the original Translatotron, and is getting close to a cascade system, which uses Tacotron 2, one of the state-of-the-art TTS models, to synthesize the predicted text translation into speech.
Consistent with Jia et al.
(2019b), despite that the training targets in the Conversational dataset is synthesized with a lower quality Grifﬁn-Lim vocoder, the trained S2ST model is able to synthesize with signiﬁcantly better naturalness than the training target when a higher quality neural vocoder is used at inference time. [Table] [Table]
5. R OBUSTNESS
We speciﬁcally evaluated the robustness issue of over-generation in the predicted speech, such as babbling or long pause, measured by unaligned duration ratio (UDR) (Shen et al., 2020) with 1-second threshold.
The ASR transcribed text is used for alignment, using a conﬁdence islands-based forced alignment model (Chiu et al., 2018).
The result is shown in Table 2.
On the Fisher set, the UDR from Translatotron 2 is about 7 times lower than from the original Translatotron [Table] and is about the same as the training target.
On the
Conversational set, the UDR from Translatotron 2 is more than 4 times lower than from the original Translatotron
[Table] and is even about 3 times lower than the training targets.
Note that the training target in the Conversational set is synthesized by the Tacotron 2 TTS model (see Table 1), which by itself suffers from over-generation (He et al., 2019; Zheng et al., 2019; Battenberg et al., 2020; Shen et al., 2020).
The result suggests that Translatotron 2 drastically improved robustness than the original
Translatotron, and is also robust to a small ratio of disﬂuency in the training targets.
5. V OICE RETENTION
To evaluate the ability of retaining speakers’ voices while translating their speech from one language to another, we augmented the Conversational dataset by synthesizing the target speech using a voice-transferring TTS model as described in Section 4.1.
Examples with source speech shorter than 1 second were discarded for the stability of voice transferring.
The result dataset contains parallel utterances with similar voices on both sides.
S2ST models were trained on this dataset without any explicit conditioning on speaker embeddings or IDs (i.e. no speaker encoder for the original
Translatotron).
Translatotron 2 models used a Conformer encoder with a larger dimension (256) than in previous experiments, since its output was expected to carry more acoustic information for voice retention.
Following Jia et al.
(2019b), we also reduced the synthesizer’s pre-net dimension to 16 to encourage it to infer voice information from the encoder output instead of the teacher-forcing label.
5-scale subjective MOS on both naturalenss and speaker similarity was evaluated with 1,000 random samples or pairs of samples from the test set 2 , reported with 95% conﬁdence interval.
As Tableshows, both Translatotron 2 and Translatotron performed well using the new approach for voice retention.
They both obtained about the same speaker similarity MOS as the original Translatotron and signiﬁcantly better translation quality.
Translatotron 2 further outperformed Translatotron in terms of translation quality, which is consistent with the experimental results for translating into a single speaker’s voice.
It is worth to note that with the new approach, the speaker similarity from S2ST models is capped by the same of the training targets, which by itself is low.
This can be partially due to the performance of the voice-transferring TTS model used for synthesizing the training targets, and partially due to the fact that cross-lingual speaker similarity evaluation is more challenging to raters
[Table]
(some rating comments are purely based on language difference), as also observed in Zhang et al.
Obtaining better quality training targets, such as human recordings instead of synthesized speech [Table] may further improve the performance of S2ST models trained with the new approach.
5.4. S PEAKER TURNS
Speaker similarity evaluation with speaker turns on the entire model prediction is challenging because it would require speaker diarization on the predicted speech [Table] and the potential content re-ordering during translation as well as potential model prediction error adds extra difﬁculty.
We approximated by considering the leading/trailing short segments in the predicted speech as corresponding to each of the two speakers in inputs with a single speaker turn.
The evaluation set was constructed by applying the same concatenation augmentation as described in
Section 4.2 on the original test set, so that each utterance contains two speakers’ voices.
Examples with target speech shorter than 2 seconds before concatenation were discarded.
We conducted subjective speaker similarity MOS evaluation between the two entire original utterances before concatenation and the leading/trailing 1.6-second segments 3 of the predicted speech.
The results are shown in Table 4.
It can be seen that the impact of the concatenation augmentation is consistent on Translatotron and Translatotron 2.
When the concatenation augmentation was not used during training, for each source speaker, the similarity compared to the leading/trailing segment in the predicted speech was about the same; and for each segment in the predicted speech, the similarity compared to the ﬁrst/second source speaker was also close.
This suggests that the predicted speech was in a single speaker’s voice, which was optimized for both source speakers at the same time.
When the concatenation augmentation was used, both models obtained signiﬁcantly higher speaker similarity on matched pairs than mismatched pairs, indicating that the models successfully separated two speakers and retained voice for each of them respectively.
It can also be seen that Translatotronobtained signiﬁcantly higher speaker similarity than Translatotron on matched pairs, indicating the effectiveness of Translatotron 2.
Such quantitative evaluation cannot reﬂect how the predicted speech transits from one speaker’s voice into another speaker’s voice.
Listening to the audio samples veriﬁed that the voice change happened instantly without blurry rather than transitionally, suggesting that the encoder outputs preserved the voice information locally in together with the linguistic information, and the synthesizer and the decoders were able to utilize such local information for translating the linguistic while retaining the corresponding voice information locally.
A sample of such speaker turn with the prediction from
Translatotron 2 trained with concatenation augmentation is visualized in Figure 2.
Although Translatotron 2 bears some similarity to cascade systems in terms of the cascade of the decoder and the synthesizer, such voice retention capacity, especially on speaker turns, is very challenging for the latter, as it would require separate speaker diarization and voice encoder.
While ConcatAug is effective on enabling S2ST models to support voice retention on speaker turns, and can further improve the translation quality and the speech naturalness on models with lower performance (e.g. trained on small datasets), it may negatively impact the speech naturalness and similarity on models with strong performance, as shown in Table 2 and Table 3.
This may be
[Table]
explained by the fact that the augmented utterances sound less natural and supporting speaker turns may sacriﬁce model capacity on single-speaker cases.
5. M ULTILINGUAL S2ST
We also conducted experiments to evaluate the performance of multilingual X → En S2ST models.
We trained both a Translatotron 2 model and a Translatotron model on the 4 high-resource languages from the CoVoST 2 corpus (Wang et al., 2021a), using TTS synthesized target speech in a single female speaker’s voice. The original Common Voice (Ardila et al., 2020) data split was followed.
The models were not explicitly conditioned on language IDs.
We modiﬁed the original Translatotron model to use the same Conformer encoder and SpecAugment as in Translatotron 2 for a fair comparison.
Similarly, we only used the target phoneme sequence (in English) for both Translatotron and Translatotron 2.
Translatotron used a decoder with 6-layer LSTM with 1024 dimension; Translatotron 2 used a decoder with 6-layer LSTM with 512 dimension and a synthesizer of 2-layer LSTM withdimension.
Both used the same convoluational post-net as Shen et al.
The total number of parameters in Translatotron 2 was about 10% fewer than in Translatotron.
The translation quality as measured by BLEU on the ASR transcribed text is shown in Table 5.
The BLEU scores from Translatotron 2 signiﬁcantly outperformed the same from Translatotron.
Although the results are not directly comparable between S2ST and ST, 4 the close numbers suggest that Translatotron 2 achieved translation quality comparable to the baseline ST model.
This indicates that Translatotron 2 is also highly effective for multilingual S2ST.
C ONCLUSION
We proposed Translatotron 2 , a neural direct S2ST model that can be trained end-to-end.
The major differences compared to the original Translatotron are: 1) the output from the auxiliary target phoneme decoder is used as an input to the spectrogram synthesizer; 2) the spectrogram synthesizer is duration-based, while still keeping the beneﬁts of the attention mechanism.
Experiments conducted on three different datasets, including multilingual S2ST, suggested that Translatotron 2 outperformed the original Translatotron by a large margin in terms of translation quality and predicted speech naturalness, and drastically improved the robustness of the predicted speech.
We also proposed a new method for retaining the source speaker’s voice in the translated speech.
In contrast to the original Translatotron, S2ST models trained with the new method is restricted to retain the source speaker’s voice, but not able to generate speech in a different speaker’s voice, which makes the model free from potential abuse such as creating spooﬁng audios, thus more robust for production deployment.
When the new method is used together with a simple concatenation data augmentation, the trained Translatotron 2 model is able to retain each speaker’s voice for input with speaker turns.
E THICS STATEMENT
Preserving voices during S2ST is desired as it helps communication.
However, progress on high quality voice cloning poses societal concerns on misuse, such as creating “deepfake” spooﬁng audios.
This work includes a new voice retention method for S2ST that reduces the potential for such misuse, if such models are deployed in production.
R EPRODUCIBILITY S TATEMENT
We conducted experiments on multiple datasets, including two public datasets.
We also provided a comprehensive list of hyperparameters in Appendix A to help reproducing the experimental results
A T ABLE OF HYPER - PARAMETERS
[Table] [Table]
[Table]
B O BJECTIVE SPEAKER SIMILARITY ANALYSIS
Subjective speaker similarity evaluation is costly and has long turnaround.
We explored into alternative objective evaluation using separately trained speaker encoders, such as d-vector (Wan et al., 2018).
We evaluated the voice retention performance using the cosine similarity of the d-vectors.
We ﬁrst checked the scenario that each input contains a single speaker’s recording.
Figure 3 visualizes the afﬁnity matrices of d-vector similarity among different input utterances for a Translatotronmodel.
The outstanding higher similarity values on the diagonals indicate that the model is able to retain the source speaker’s voice in the predicted speech.
We then conducted a detailed evaluation for the voice retention performance for speaker turns.
The experiment setting up was identical to Section 5.4.
[Table]
except that the speaker similarity was measured by d-vector similarity instead of subjective MOS evaluation.
The d-vectors for each source speaker were computed on the entire original utterance before concatenation; the d-vectors for each speaker in the prediction is approximated by computing on the leading/trailing 1.6 seconds of predicted speech.
The results are shown in Table 7.
Consistent with the MOS evaluation results in Table 4, when the concatenation augmentation was not used, the d-vector similarity to each source speaker is about the same regardless if it was compared to the leading or trailing segments, indicating that the predicted speech was in a single speaker’s voice and the model was unable to separate different speakers in the input, but rather optimized for both source speakers at the same time.
When the concatenation augmentation was used, the d-vector similarity was signiﬁcantly higher between matched pairs than between unmatched pairs, indicating that the models were able to separate different speakers in the input and retain their voice in the predicted speech respectively.
However, when these similarities are compared among different models, it seems suggesting that
Translatotron performed better than Translatotron 2, which is in contradictory to the subjective evaluation results in Table 4.
By carefully listening to the audio samples, we found that such discrepancy may be due to that the d-vector model was also sensitive to non-voice related acoustic characteristics, such as reverb and channel noise in the audios.
This is likely a consequence of the fact that in the large-scale training set for the d-vector model used in the evaluation, each speaker is typically associated with a particular recording condition, e.g. recording device and room.
Because the encoder output from the Translatotron model was of signiﬁcantly larger dimension than from the [Table]
[Table] Translatotron 2 model (2048 vs 256), it was capable of carrying more non-voice acoustic information and thus obtained better d-vector similarity [Table] which not necessarily indicating higher speaker similarity
[Table] These results suggest that while such speaker encoder-based objective analysis reveals insightful indications about the performance of the S2ST models, it can be less reliable compared to subjective
MOS evaluation.
Such reliability also highly depends on the training details of the speaker encoder model being used, especially the training corpus.
C N ON - AUTOREGRESSIVE SYNTHESIZER
Following recent non-autoregressive TTS works (Ren et al., 2021;
Guo et al., 2021; Lee et al., 2021b; Elias et al., 2021a) [Table]
we explored using non-autoregressive synthesizer in Translatotron [Table] which may have signiﬁcantly lower latency at inference time.
The experimental results in Table 8 suggested that despite producing comparable BLEU on ASR transcribed text from the Translatotron 2 predictions, using non-autoregressive synthesizer produced signiﬁcantly worse naturalness of the predicted speech.
This is consistent with the observation in TTS in Shen et al. D A BLATION STUDIES
To understand the importance of each components in the Translatotron 2 model, we conducted ablation studies on the CoVoST 2 multilingual X → En dataset as described in Section 5.5.
All models in the ablation used the same input and output features, SpecAugment setting, and learning rate schedule as described in Section 5.5 and
Appendix A.
No auxiliary training target on the source text or phonemes were used.
For models using an RNN-based encoder, we ﬁrst applied the same 4 × time-wise subsampling as used in the Conformer encoder, then used a 8-layer bidirectional LSTM stack with a cell size of 256.
The number of parameters in this RNN-based encoder is close to the same in the Conformer encoder.
For the Translatotron model using a NAT decoder, the same architecture and hyperparameters as the synthesizer of Translatotron 2 was used to replace the original attention-based decoder.
For Translatotron 2 using a non-autoregressive synthesizer, a simple 6-layer
Conformer stack with a dimension of 512 and 8 attention heads was used to replace the autoregressive synthesizer, same as in Appendix C.
This Conformer-based non-autoregressive synthesizer is similar to the Transformer-based decoder in the FastSpeech 2 TTS model (Ren et al., 2021), but performed
[Figure]
better in our experiments.
All the rest hyperparameters follow Appendix A for Translatotron 2, and follow the Conversational model in Jia et al.
(2019b) for Translatotron.
All models were trained for 200K steps with a batch size of 768.
The evaluation was done using the same checkpoints for all the 4 language pairs, picked by the highest average performance on the dev sets.
The results are shown in Table 9.
As can be seen, while the use of Conformer, SpecAugment, and NAT decoder helps the performance of Translatotron 2, replacing them with alternative architectural choices or removing SpecAugment only reduced the performance by a small degree ( < 2 BLEU).
Similarly, directly using these components in the original Translatotron does not bring its performance close to
Translatotron 2.
These results suggest that the performance improvement of Translatotron 2 comes from addressing the weaknesses existing in Translatotron (Section 3), rather than the architectural choices of each individual components.
Consistent with the result in Appendix C, using a non-autoregressive synthesizer in Translatotronobtained comparable translation quality to using an autoregressive synthesizer which is based on the NAT decoder.
However, as Appendix C shows, such non-autoregressive synthesizer may suffer from lower naturalness in the predicted speech.
Directly using the NAT decoder in the original
Translatotron obtained the worst performance because its monotonic nature does not work well for a translation task.
E C ROSS - LINGUAL VOICE TRANSFERRING TTS
The TTS model used for synthesizing training targets in the source speaker’s voice in Section 4. is modiﬁed from the PnG NAT (Jia et al., 2021) TTS model by incorporating a separately trained speaker encoder (Wan et al., 2018) in the same way as Jia et al.
The architecture of this
TTS model is illustrated in Figure 4.
We trained this model on the LibriTTS corpus (Zen et al., 2019), following the hyperparameters in Jia et al.
; Shen et al.
; Wan et al.
The speaker encoder is separately trained in a speaker veriﬁcation task and is frozen during the TTS model training.
At synthesis time, the phonemes and graphemes in the target language and the reference speech in the source language are fed into the model as inputs; and the model produces speech in the target language with the voice from the source speech transferred.