Paper: PolyVoice: Language Models for Speech to Speech Translation (ICLR-2024)

Reviewer comment (Experiments):
It would have been preferable to report performance on a low-resource target language in an unwritten scenario. Such a setup might reveal additional challenges which are not present in a high resource language such as Spanish.

Author response:
As the data of an unwritten language is still very limited in literature, we follow the practice in the previous works, e.g.

Lee A, Gong H, et al. Textless speech-to-speech translation on real data. In Proc. NAACL 2022,

Lee A, Chen P J, et al. Direct speech-to-speech translation with discrete units. In Proc. ACL 2022,

Rongjie Huang, Jinglin Liu, et al. TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation. In Proc. ICLR 2023,

which use a written language with transcriptions discarded to simulate an unwritten scenario.

Reviewer comment (Experiments):
4.1.1: For Chinese->English task, what is the size of the semantic unit inventory for Chinese and English?

Author response:
In our core experiment, we use separated tokenized models for Chinese and English.
In this case, their semantic units are not shared.
The size of the semantic unit set is 500 for Chinese and 1000 for English, respectively. For more details, please refer to Section 4.1.1.

Reviewer comment (Experiments):
I couldn't get a good sense of the training data - specifically how the different prompts from Table 1 are used to synthesize data, and the size of the synthesized dataset: is it the 44M sentences from Table 7 in the appendix, or more because multiple prompts are used? How does the training data compare to the baselines?

Author response:
We thank the reviewer for the constructive and valuable feedback, and we hope our response fully resolves your concern.
 
- **[How the prompts are used]** 

We apologize for an unclear presentation about how the prompts are used to synthesize data. Here is our explanation and we will modify our paper accordingly.

Table 1 shows the prompt templates for different data types. Indeed, there is an instruction set for each data type to form the prompt.
Here we list some examples of our synthetic training data:

*Sample1*: Translate Chinese text " 顾客：打扰一下，是在这里排队吗？ " to English text: Customer: Excuse me, do I queue here?

*Sample2*:  Chinese text " 顾客：打扰一下，是在这里排队吗？ " in English text: Customer: Excuse me, do I queue here?

*Sample3*:  Translate the following sentence, "顾客：打扰一下，是在这里排队吗？"  from Chinese to English: Customer: Excuse me, do I queue here?

*Sample4*: Translate Chinese unit "<zh_21><zh_161><zh_155>...<zh_266><zh_199><zh_16>" to Chinese text: 你带着现金离开，减去3%左右的费用。

*Sample5*: Translate English text "douglas mcgray is going to be our guide you walk through the door, you see the red carpeting, you see someone in a suit. they may be greeting you." to English unit:  <293><63><662>...<6><407><334>

*Sample6*: Translate Chinese unit "<zh_16><zh_37><zh_111>...<zh_266><zh_199><zh_16>" to English unit: <499><334><226>...<544><991><39>

By utilizing diverse construction templates or prompts, the training data can be significantly increased. This approach is also a mainstream method in training Large Language Models (LLMs) to enhance the diversity of instructions and improve model robustness. “44M sentences” refers to the statistical count of the original machine translation (MT) parallel data. They are multiplied by different instruction prompts selected from the instruction set.
 
- **[Training data comparison with the baseline]**

In our paper, we use VALL-E X as our baseline as it is the closest work to us. Regarding the comparison of training data, please refer to the following table, where WenetST represents the end-to-end speech translation training data, expanded upon WenetSpeech by the authors of VALL-E X.

  ——————————————————————————————————————————————————————
  
System     &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &nbsp;&nbsp;       |    **VALL-E X Trans (SpeechUT + VALL-E X )**   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |     **PolyVoice**

ASR data  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;    |  LibriLight (60k hrs), WenetSpeech (10k hrs)   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;  | LibriLight (60k hrs), In-house (60k hrs)

MT data   &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;  &nbsp;&nbsp;&nbsp;&nbsp;     |  73M sentences      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;     &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;          &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;                             | 44M sentences

ST/S2S data  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;    |  WenetST (10k hrs), GigaST (10k hrs)      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;         |  WenetS2S (10k hrs), GigaS2S (10k hrs)

 ——————————————————————————————————————————————————————

We have incorporated an additional in-house ASR dataset, specifically a Chinese ASR dataset, into the training of our U-XLM module. The primary objective behind using this dataset is to enhance the performance of our translation module. As a result of this integration, we have observed a notable improvement of 1-2 points in BLEU scores for the entire system. This increase offsets the performance loss incurred by employing unsupervised discretized units instead of phonemes. Although we do not release this particular ASR dataset, we believe a dataset with similar data volume can reproduce our results.

Reviewer comment (Experiments):
My main concern would be that the ablation studies are not effective for disentangling the many design choices and the many moving parts of the whole architecture. The encoder-decoder vs decoder-only comparison is a good start, but I still don't have a good sense about how well the synthetic data generation works, and how well each of the three models do in isolation.

Author response:
- **[More ablation studies]**
 
Thanks for the reviewer's constructive suggestion. Actually, we have already done some of the ablation studies suggested by the reviewer. In this work, we performed these analyses to evaluate the performance of individual modules. 

For the evaluation of the U-XLM module, we fixed the U-SLM and tested the performance of different translation modules, as shown in Table 4, comparing S2UT with the encoder-decoder structure. 

For the evaluation of the U-SLM module, we fixed the U-XLM and tested the performance of different synthesis modules, e.g. the comparison between U2S and Unit-based Vocoder. The results are posted in Table 2. 

In Table 6, we have analyzed the impact of removing the duration module on the performance of speech synthesis. 

As for the suggestion of replacing U-XLM with an out-of-the-box ASR/MT cascade, since the existing ASR->MT cascade systems do not support speech unit prediction, it is not easy to perform end-to-end evaluation of the cascade system with U-SLM as the back-end synthesis module. We will try our best to implement a unit-based cascade system and report the results.

Reviewer comment (Experiments):
Possible interesting ablations would be removing the duration model, replacing u-xlm with an out-of-the-box asr/mt cascade, removing source speech dependency from u-slm, passing through n-bests between the models, etc. I'm not requesting that all possible ablations should be included, but giving a little bit more color to the paper story would make it much stronger.

Author response:
- **[More ablation studies]**
 
Thanks for the reviewer's constructive suggestion. Actually, we have already done some of the ablation studies suggested by the reviewer. In this work, we performed these analyses to evaluate the performance of individual modules. 

For the evaluation of the U-XLM module, we fixed the U-SLM and tested the performance of different translation modules, as shown in Table 4, comparing S2UT with the encoder-decoder structure. 

For the evaluation of the U-SLM module, we fixed the U-XLM and tested the performance of different synthesis modules, e.g. the comparison between U2S and Unit-based Vocoder. The results are posted in Table 2. 

In Table 6, we have analyzed the impact of removing the duration module on the performance of speech synthesis. 

As for the suggestion of replacing U-XLM with an out-of-the-box ASR/MT cascade, since the existing ASR->MT cascade systems do not support speech unit prediction, it is not easy to perform end-to-end evaluation of the cascade system with U-SLM as the back-end synthesis module. We will try our best to implement a unit-based cascade system and report the results.

Reviewer comment (Reproducibility):
Could you give more details on how the prompts are used?

Author response:
We thank the reviewer for the constructive and valuable feedback, and we hope our response fully resolves your concern.
 
- **[How the prompts are used]** 

We apologize for an unclear presentation about how the prompts are used to synthesize data. Here is our explanation and we will modify our paper accordingly.

Table 1 shows the prompt templates for different data types. Indeed, there is an instruction set for each data type to form the prompt.
Here we list some examples of our synthetic training data:

*Sample1*: Translate Chinese text " 顾客：打扰一下，是在这里排队吗？ " to English text: Customer: Excuse me, do I queue here?

*Sample2*:  Chinese text " 顾客：打扰一下，是在这里排队吗？ " in English text: Customer: Excuse me, do I queue here?

*Sample3*:  Translate the following sentence, "顾客：打扰一下，是在这里排队吗？"  from Chinese to English: Customer: Excuse me, do I queue here?

*Sample4*: Translate Chinese unit "<zh_21><zh_161><zh_155>...<zh_266><zh_199><zh_16>" to Chinese text: 你带着现金离开，减去3%左右的费用。

*Sample5*: Translate English text "douglas mcgray is going to be our guide you walk through the door, you see the red carpeting, you see someone in a suit. they may be greeting you." to English unit:  <293><63><662>...<6><407><334>

*Sample6*: Translate Chinese unit "<zh_16><zh_37><zh_111>...<zh_266><zh_199><zh_16>" to English unit: <499><334><226>...<544><991><39>

By utilizing diverse construction templates or prompts, the training data can be significantly increased. This approach is also a mainstream method in training Large Language Models (LLMs) to enhance the diversity of instructions and improve model robustness. “44M sentences” refers to the statistical count of the original machine translation (MT) parallel data. They are multiplied by different instruction prompts selected from the instruction set.

Reviewer comment (Experiments):
Could you give more details on the synthetic training data?

Author response:
We thank the reviewer for the constructive and valuable feedback, and we hope our response fully resolves your concern.
 
- **[How the prompts are used]** 

We apologize for an unclear presentation about how the prompts are used to synthesize data. Here is our explanation and we will modify our paper accordingly.

Table 1 shows the prompt templates for different data types. Indeed, there is an instruction set for each data type to form the prompt.
Here we list some examples of our synthetic training data:

*Sample1*: Translate Chinese text " 顾客：打扰一下，是在这里排队吗？ " to English text: Customer: Excuse me, do I queue here?

*Sample2*:  Chinese text " 顾客：打扰一下，是在这里排队吗？ " in English text: Customer: Excuse me, do I queue here?

*Sample3*:  Translate the following sentence, "顾客：打扰一下，是在这里排队吗？"  from Chinese to English: Customer: Excuse me, do I queue here?

*Sample4*: Translate Chinese unit "<zh_21><zh_161><zh_155>...<zh_266><zh_199><zh_16>" to Chinese text: 你带着现金离开，减去3%左右的费用。

*Sample5*: Translate English text "douglas mcgray is going to be our guide you walk through the door, you see the red carpeting, you see someone in a suit. they may be greeting you." to English unit:  <293><63><662>...<6><407><334>

*Sample6*: Translate Chinese unit "<zh_16><zh_37><zh_111>...<zh_266><zh_199><zh_16>" to English unit: <499><334><226>...<544><991><39>

By utilizing diverse construction templates or prompts, the training data can be significantly increased. This approach is also a mainstream method in training Large Language Models (LLMs) to enhance the diversity of instructions and improve model robustness. “44M sentences” refers to the statistical count of the original machine translation (MT) parallel data. They are multiplied by different instruction prompts selected from the instruction set.
 
- **[Training data comparison with the baseline]**

In our paper, we use VALL-E X as our baseline as it is the closest work to us. Regarding the comparison of training data, please refer to the following table, where WenetST represents the end-to-end speech translation training data, expanded upon WenetSpeech by the authors of VALL-E X.

  ——————————————————————————————————————————————————————
  
System     &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &nbsp;&nbsp;       |    **VALL-E X Trans (SpeechUT + VALL-E X )**   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |     **PolyVoice**

ASR data  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;    |  LibriLight (60k hrs), WenetSpeech (10k hrs)   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;  | LibriLight (60k hrs), In-house (60k hrs)

MT data   &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;  &nbsp;&nbsp;&nbsp;&nbsp;     |  73M sentences      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;     &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;          &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;                             | 44M sentences

ST/S2S data  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;    |  WenetST (10k hrs), GigaST (10k hrs)      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;         |  WenetS2S (10k hrs), GigaS2S (10k hrs)

 ——————————————————————————————————————————————————————

We have incorporated an additional in-house ASR dataset, specifically a Chinese ASR dataset, into the training of our U-XLM module. The primary objective behind using this dataset is to enhance the performance of our translation module. As a result of this integration, we have observed a notable improvement of 1-2 points in BLEU scores for the entire system. This increase offsets the performance loss incurred by employing unsupervised discretized units instead of phonemes. Although we do not release this particular ASR dataset, we believe a dataset with similar data volume can reproduce our results.

Reviewer comment (Novelty):
The innovations of this work need to be more clearly explained. This work bears strong similarity to VALL-E X. It is important to clarify the difference between the proposed approach and VALL-E X, but the paper did not clearly point out the difference between PolyVoice and VALL-E X to highlight the innovations of the proposed PolyVoice. Both works concatenate source and target semantic units and the source acoustic units to create the prompt for the LM. For PolyVoice, this prompt is created for the duration LM and the speech synthesis LM, respectively. The soundstream codec is reimplemented, but the impact of the reimplementation is not clear.

Author response:
We appreciate the reviewer's comments highlighting similarities between aspects of our work and VALL-E X. The core of our pipeline consists of a speech-to-unit translation (S2UT) component followed by a unit-to-speech (U2S) synthesis component. The unit-to-speech language model (U-SLM) in our U2T module shares conceptual similarities with VALL-E X, while other aspects of our overall model design differ. In response, we clarify key differences in motivation and approach:

1. Our work is primarily motivated by exploring fully language model-based architectures for speech-to-speech translation, as a way to extend recent progress in large language models to this task. To our knowledge, we are the first to propose and evaluate such a fully decoder-only pipeline, demonstrating feasibility and encouraging results.

2.  A key difference is that VALL-E X operates on phonemes, while we aim for a universal framework using discrete units that can be learned from unlabeled speech. This provides greater flexibility for low-resource and unwritten languages. While our current unit extraction quality limits gains over VALL-E X, our approach enables potential optimization with improved units and direct application to any language.

3. There are also significant differences in the modeling approach between VALL-E X and PolyVoice for speech-to-speech translation tasks. In VALL-E X, a traditional pipeline system is used, which relies on external ASR (Automatic Speech Recognition) and ST (Speech-to-Text) modules to recognize phonemes in the source language and target language. On the other hand, PolyVoice employs the U-XLM module, which enables end-to-end speech-to-speech translation. This means that the translation task is performed directly without relying on separate ASR or ST modules. The U-XLM module can directly convert speech units in the source language into synthesized speech units in the target language, providing a seamless translation experience.

Overall, we believe our work makes meaningful contributions in exploring language model-based S2ST, despite some similarities to VALL-E X in the speech synthesis component.
 
- **[Re-implementation of soundstream codec]**

The main reason we have built our own codec model is that Google has not open-sourced their soundstream model. The only publicly available audio codec model was Encodec released by Meta, but its performance is worse than we anticipated. 
We trained our own version of soundstream in order to obtain a better audio codec. And the performance of codecs was compared using a subjective metric very similar to  MUSHRA.

Reviewer comment (Writing):
Some of the key technical presentations lack clarity — specifically, in Section 3.1, when describing Unit-based cross-lingual language model (U-XLM), the paper shows the prompt for encoder-decoder architecture, but should also clarify the prompt for decoder-only LM.

Author response:
*a. Prompt for model training.*

To clarify, the prompts shown in Table 1 are used to format the training data for our decoder-only language model, not an encoder-decoder architecture. Specifically, we construct varied training sequences using different prompts to cover diverse data types like ASR, MT, and S2ST. This allows us to train our language model in a decoder-only manner by modeling the training sequences in an auto-regressive way. 

We do not actually utilize any encoder-decoder architectures in our proposed framework. We compare with encoder-decoder models in Section 5.1, where for those baselines we construct the training data in the same way as previous work
(https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/direct_s2st_discrete_units.md).

Reviewer comment (Reproducibility):
Some of the key technical presentations lack clarity — specifically, in Section 3.1, when describing training, Table 1 shows how ASR, MT, and TTS supervised data are used for training. The paper also mentions that “This approach also enables the direct utilization of unlabeled text and speech data,” yet how to use unlabeled text and speech data for training is not explained here. Instead, based on Section 4.1.1, it seems that one approach is to apply in-house MT and TTS systems on ASR data to create pseudo S2S data. This part needs to be clarified.

Author response:
*b. Data utilization.*

Like text-based language models, our framework allows unlabeled speech and text data to be directly leveraged for training via language modeling objectives. In our current implementation, we construct training data by formatting labeled ASR, MT, and pseudo-S2ST data with specific prompts, as detailed in Table 1. Simultaneously, parallel training data is also trained separately in monolingual form, and these type of training samples do not require construction through prompts.

To clarify, our proposed framework inherently supports training on unlabeled speech and text, similar to standard language model pretraining. We believe that scaling up the training data by incorporating large scales of unlabeled speech and text would enable more powerful models.

Reviewer comment (Evaluation):
In Table 2, the ASV metric evaluates the capability of preserving the voice of the source speaker. Without ground truth target information, PolyVoice achieves 0.38 and 0.38 for hyp vs. src and hyp vs. tgt, while VALL-E X achieves 0.37 and 0.37 respectively. These are very small gains, yet the paper did not discuss this point.

Author response:
*b. Explanation of the ASV metric.*

We acknowledge the reviewer's observation that our framework achieves only slight gains over VALL-E X in voice cloning quality. This is expected given the similar implementations for the speech synthesis component. However, by modeling on semantic units rather than phonemes, our model retains more acoustic information that can lead to higher naturalness and better synthesis effects. While not yet reflected in voice cloning metrics, our experiments provide a path to improve acoustic modeling by using unit representations.

Reviewer comment (Experiments):
In Section 4.3.2, the evaluation of PolyVoice for unwritten language is quite brief — it is only evaluated for the target language treated as an unwritten language. It would be useful to also extend the evaluation to the source or both languages as unwritten languages.

Author response:
*d. Evaluation of unwritten language.*

Thanks for the valuable advice to extend the evaluation to encompass unwritten languages, either as the source language or as both languages. We will incorporate the updated experimental results in the upcoming version. To clarify, the primary objective of experimenting on unwritten language scenarios in this paper is to demonstrate the feasibility of extending PolyVoice to unwritten languages.

Reviewer comment (Evaluation):
In Section 5.1, the discussions are again very brief. More analyses and insights are expected to explain the better ASR-BLEU from the decoder-only setup compared to the encoder-decoder.

Author response:
*e. Better ASR-BLEU from decoder-only over encoder-decoder.*

The comparison between the decoder-only and encoder-decoder models was conducted under the same training data and similar parameters. We hypothesize that the higher ASR-BLEU score achieved by the decoder-only model over the encoder-decoder model can be attributed to two factors. Firstly, the decode-only framework demonstrates superior capability in fitting large-scale training data. Secondly, the output generated by the decoder-only framework exhibits higher fluency, which positively impacts the evaluation of ASR-BLEU.

Reviewer comment (Experiments):
In Section 5.2, for the other tasks (ASR, ST, MT, TTS) in Table 5, since no baseline results are provided, it is not clear how these performances compare to baseline results on this dataset from prior works.

Author response:
*f. Performance of other tasks in Table 5.*

Given the involvement of multi-task learning (MTL), the primary objective of Table 5 is to demonstrate the overall S2ST gain achieved through MTL. Beating other systems in individual tasks is not our objective. We apologize for any misleading and we will improve our paper presentation. Still, as suggested by the reviewer, we would like to complement Table5 with the following table:

——————————————————————————

Task       &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;           | **ASR**↓ |  **ST**↑  &nbsp; |   **MT** ↑ &nbsp; |  **TTS**↓

PolyVoice (w/ MTL)   | 4.46 &nbsp;   | 30.8 |  33.81  |  6.99

Whisper[1]   &nbsp;&nbsp;&nbsp;  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;        | 2.28 &nbsp;  |  18.2 |    -   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |    -

YourTTS[2]        &nbsp;&nbsp;&nbsp;  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;          |  -   &nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;      |    - &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;   |      -  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  &nbsp;&nbsp;  |  3.03

NLLB[3]         &nbsp;&nbsp;&nbsp;  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp; &nbsp;               |  -    &nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;    |    - &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;   |    25.2  &nbsp;&nbsp; |    -

——————————————————————————

The evaluation mentioned above was performed using state-of-the-art open-sourced models, all evaluated on the same test set. As expected, Whisper, trained on substantially larger datasets, outperforms our model in terms of ASR performance. However, our model demonstrates superior results in both ST (Speech-to-Text) and MT (Machine Translation), showcasing the translation quality it offers.
It is worth noting that our translation model's TTS (Text-to-Speech) results are not as satisfactory as YourTTS. This disparity can be attributed to the utilization of a weaker unit vocoder in this abliation studies.

Reviewer comment (Evaluation):
In Section 5.3, when using a mHuBERT model trained with more parameters and more data, the WER decreased (which is explained), but ASV and naturalness got degraded. Insights are expected to explain these results.

Author response:
*g.  Explanation of mHuBERT's performance.*

Our explanation is that a  larger HuBERT trained with more data is stronger in extracting semantic  information. The units extracted from this model contain more semantic information and as a result less prosody and speaker information. Thus the ASV and naturalness degraded.

Reviewer comment (Experiments):
The proposed system uses many in-house data and in-house systems. The amount of data and model sizes need to be clearly compared as well when comparing to baselines.

Author response:
*h. Comparison of the amount of data and model sizes.*

Regarding the model size, the parameters of U-SLM are consistent with Valle-X. And the training data for U-SLM and Valle-X are essentially consistent. Therefore, the comparison of the synthesis module is absolutely fair. 

In the speech-to-speech translation task, Valle-X utilizes SpeechUT[4] as the ASR and ST model, which predicts source phoneme sequences for recognition, and target phoneme sequences for translation, respectively. SpeechUT follows an encoder-decoder architecture, whereas our U-XLM is a decoder-only structure. A direct comparison of parameters of these definitely different frameworks may not provide an intuitive understanding. 

Regarding the amount of training data, please refer to the following table, where WenetST represents the end-to-end speech translation training data, expanded upon WenetSpeech by the authors of VALL-E X.

 ——————————————————————————————————————————————————————
  
System     &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &nbsp;&nbsp;       |    **VALL-E X Trans (SpeechUT + VALL-E X )**   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |     **PolyVoice**

ASR data  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;    |  LibriLight (60k hrs), WenetSpeech (10k hrs)   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;  | LibriLight (60k hrs), In-house (60k hrs)

MT data   &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;  &nbsp;&nbsp;&nbsp;&nbsp;     |  73M sentences      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;     &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;          &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;                             | 44M sentences

ST/S2S data  &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;    |  WenetST (10k hrs), GigaST (10k hrs)      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;         |  WenetS2S (10k hrs), GigaS2S (10k hrs)

 ——————————————————————————————————————————————————————

We have incorporated an additional in-house ASR dataset, specifically a Chinese ASR dataset, into the training of our U-XLM module. The primary objective behind using this dataset is to enhance the performance of our translation module. As a result of this integration, we have observed a notable improvement of 1-2 points in BLEU scores for the entire system. This increase offsets the performance loss incurred by employing unsupervised discretized units instead of phonemes. Although we do not release this particular ASR dataset, we believe a dataset with similar data volume can reproduce our results.

Reviewer comment (Experiments):
The duration and speech synthesis models depend on the translation model. Hence the training of two models depend on one upstream model which can make experimentation slow. At least, the duration model can be attempted to be folded in the translation model as shown in the paper Text-Free Prosody-Aware Generative Spoken Language Modeling (Kharitonov et. al.).

Author response:
The translation front-end model and the speech synthesis back-end model can be trained separately. During the training, the speech of the target language will be converted into discrete units, which on the one hand serve as the training target of the translation module and on the other hand serve as the input of the speech synthesis module. Thus, these two models can be trained in parallel and do not result in slow training time.

Thank you for pointing this out. We may make an attempt on this in the future.

Reviewer comment (Experiments):
Since the authors use CVSS it would be desirable to show the performance on other language pairs from the dataset to make the evaluation more robust.

Author response:
We didn’t run experiments on other language pairs as our tokenizer didn’t support other languages. We will consider extending our work into other language pairs in the future.

Reviewer comment (Experiments):
HuBERT is trained on English only corpus. Did you just apply the model to discretize Chinese speech, or had to adapt it?

Author response:
In this paper, we employed three variations of HuBERT. In the main and analysis experiments (Table 2-5), separate training was conducted for the Chinese and English versions of HuBERT, as described in detail in Section 4.1.1. In the analysis experiment (Table 6), we trained a bilingual HuBERT model that combines both Chinese and English data, as explained in Section 5.3.

Reviewer comment (Experiments):
Why was the specific language pairs that have been evaluated chosen?

Author response:
Chinese->English pair is chosen as they are both high-resource languages. And it is easier to compare our results with other systems (VALL-E X) with this language pair.

English->Spanish pair is chosen as we want to use Spanish to simulate an unwritten language scenario where the amount of available data is relatively limited. Additionally, we directly utilized an open-source multilingual HuBERT model (
https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/textless_s2st_real_data.md , en-es-fr), which also supports the Spanish language.

Reviewer comment (Writing):
Spelling error: "marked" instead of "maked" in section 4.1.1

Author response:
Thank you very much for pointing these out. We will cite the mentioned paper accordingly and fix the spelling error in the revised version.