From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper aims to generate more deterministic representation for direct speech-to-speech translation (S2ST) system, where the system often suffers from various acoustic conditions (e.g., rhythm, pitch, energy) that make generating consistent semantic units challenging. The authors propose bilateral perturbation to address the challenge, where information enhancement and style normalization are leveraged such that CTC decoding can generate units that preserve linguistic information and be invariant across speech sample variants in dierent acoustic conditions. The author also implemented S2ST with non-autoregressive unit decoding mechanism. Efficacy of proposed approaches are showed empirically with several well-known benchmarks in translation quality and unit decoding latency. 

Strength of the paper:
1. Proposed approaches are simple, intuitive and novel.
2. Authors proposed a bag of tricks and show the efficacy of those tricks empirically in performance (BLEU scores) and translation latency. Conducted experiments and analyses are comprehensive and allows readers to understand the effectiveness of each trick.
3. Beyond S2St, proposed approaches can benefit other applications that required units stably encoding semantic information, such as ASR. 

Weakness of the paper:
1. Some technical details and discussion are missing (e.g., the connection between bilateral perturbation and NAR, why indeterministic units are harmful for performance, how much BLEU is sacriced from moving an auto-regressive system to a NAR system, why the authors choose to experiment with main stream languages such as English, even though S2ST is most beneficial for languages that can't be written). The authors did a great job in discussion phase, and provided many additional experiment results and explanation. Most of the concerns are mitigated, if not  cleared. However, the connection between bilateral perturbation and NAR approaches are not as strong as the reviewers/AC are hoping to, which makes the work be viewed more as a bag of (effective) tricks, instead of one clean framework. Lack of one clean framework could limit the impact of this paper.
2. Reviewers and AC found "multimodality" used in the paper (refer to variations) is a bit different from how it's generally interpreted in community (mulitiple sensory or channels of information, such as text, audio, image)

justification_for_why_not_higher_score: - See the bullet 1 of weakness. The connection among proposed approaches (i.e., bilateral perturbation and NAR) are not as strong as the reviewers/AC are hoping to. The lack of strong connection makes the work be viewed more as a bag of tricks (yet effective), instead of one clean/simple framework, which could limit the impact of this paper. 
- Also the proposed approaches can benefit S2St and several speech applications that require semantic units, such as ASR. However, how can the proposed approaches and experiment results be generalized to other community is an open questions.
 
justification_for_why_not_lower_score: - See the strength of the paper for more details. The paper proposed several very helpful approaches for S2ST and potentially beneficial for ASR and other speech applications. The experiment and analysis are comprehensive. The reviewers and AC have confidence that accepting the paper in ICLR would benefit the community and drive research further.

note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper2746/Reviewer_H8Cj
title: Thanks for fully responding to my questions
comment: I appreciate the detailed responses from the authors. All of my questions are addressed. 

Particularly, the WER results on the generated speech across testsets between original units and norm units are convincing to demonstrate that the style-normalized speech preserves linguistic content. Thanks for the clarification on the choice of the chain functions and the order. It's good to know that changing order to peq(fs(pr(S))) doesn't cause significant changes, which aligns with hypothesis. Thanks for explaining the energy perturbation and for including the discussion in the draft. Finally, thanks for sharing the observations on the effectiveness of style normalization on different conditions. This is reasonable.  The plan of future work on this topic is interesting. I'm looking forward to your future progress.

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Response to Reviewer H8Cj
comment: 
We are grateful for your positive review and valuable feedback, and we hope our response fully resolves your concern.

**[About style normalization]**

Thanks for the reviewer's feedback that requests more explanation about rhythm normalization. Unlike pitch and energy attributes, we do not include rhythm normalization in an unsupervised manner (without using text) as it introduces further challenges. For utterance-level rhythm, it could be improper to decide the mean value for normalization since the utterance duration mainly corresponds to linguistic meaning (long content always has long pronounced speech), which could lead to content degradation in generated pseudo-targets. For phoneme or word-level rhythm, the text-audio pairs are required [1] for precise duration prediction, which are unavailable in direct speech-to-speech translation.

**[About HuBERT representation]**

In the style normalization stage, we perturb the acoustic condition by shifting attributes to the mean distribution, and create the acoustic-agnostic "pseudo text". Still natural as shown in the demo page, a normalized speech represents the averaged pattern across the dataset, which is not considered as an out-of-domain sample. 

For easy understanding, we report the word error rate (WER) of generated speech across the test set. As shown in the following table, a minor increase of WER by 0.6\% could be witnessed in samples generated by norm HuBERT units, which is negligible and does not degrade the clearness of a speech. In summary, the encoded representations of the style-normalized speech achieve high performance and inherently preserve linguistic content information.

| Vocoder Input | WER|
|-|:-:|
|GT|/|
| Original Unit | 9.7\%| 
|Norm Unit|10.3\%| 

**[About the choice of the chain function in order]**

We apply the chain function $F = fs(pr(peq(S)))$ to randomly shift the pitch value of original speech $S$, including the following aspects: 1) formant shifting $fs$ to perturb the frequency formant, 2) pitch randomization $pr$ to perturb the f0 contour, 3) random frequency shaping $peq$ to adjust the level of a frequency. 

Since $fs$ and $pr$ are integrated as a joint function in the parselmouth framework [2], we inherit the common practice and do not explore their order. For $peq$, we observe that it is orthogonal to the aforementioned $fs$ and $pr$ techniques, which contribute to the information enhancement side-by-side. For easy comparison, we change its order to $F' = peq(fs(pr(S)))$ and obtain perturbed audios. As presented at demo page, no obvious changes could be witnessed after altering the processing order. 

**[About the energy perturbation]**

Thanks for the reviewer's reminder. We used the average from a log-mel spectrogram in historical development, which is taken place by the waveform-level perturbation in the final version of energy perturbation. Specifically, we used to consider the average from a log-mel spectrogram as energy following [3], after which we altered the values and reconstructed the waveform using a separately-trained neural vocoder. However, we found it more flexible to perturb the audio in the waveform domain directly [4], which is adopted in the information enhancement stage to create speech samples variant in energy conditions. We will include this discussion and revise the manuscript accordingly.

**[About how to improve the effectiveness of style normalization]**

Thanks for the reviewer's feedback that requests more discussion about the style normalization (SN). This stage is expected to eliminate the acoustic variations in the CTC target, and thus its effectiveness becomes different according to the acoustic conditions across datasets: Among expressive speech samples (speaker, emotion and style, etc.), the normalization stage brings a noticeable improvement in reducing acoustic variations, while a relatively smaller promotion could be observed in dull samples. 

To further improve the effectiveness of style normalization, future work includes designing a neural conversion network to normalize the acoustic conditions in speech samples, which could be more powerful than signal processing algorithms. Thanks for the reviewer's suggestion, we will attach ID=4 results in the revised paper.

**[About the typos and grammatical errors in manuscript]**

Thanks for the reviewer’s suggestion. We will fix the mentioned instance in the revised version of the paper.

**[References]**

[1] McAuliffe M, Socolof M, Mihuc S, et al. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. In Proc. InterSpeech 2017.

[2] https://github.com/YannickJadoul/Parselmouth

[3] Ren Y, Hu C, Tan X, et al. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. In Proc. ICLR 2021.

[4] Wang C, Hsu W N, Adi Y, et al. fairseq S\^2: A Scalable and Integrable Speech Synthesis Toolkit. In Proc. EMNLP 2021.

Again, we appreciate your constructive reviews and hope our response can fully resolve your concern.

From: ICLR.cc/2023/Conference/Paper2746/Reviewer_H8Cj
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper aimed at alleviating the acoustic multimodality problem for speech-to-unit approaches for direct speech-to-speech translation (S2ST) system. To address this problem, the authors developed bilateral perturbations (BiP) and leveraged the CTC fine-tuning technology. The bilateral perturbations include information enhancement on CTC input and style normalization on CTC target. Information enhancement applies information bottleneck on acoustic features to create speech sample variants in different acoustic conditions (considering rhythm, pitch, energy) while preserving linguistic content. In contrast, style normalization aims to eliminate acoustic-style information and creates acoustic-agnostic “pseudo text”.  After the bilateral perturbation, ASR system with CTC decoding would do better to generate deterministic representations, hence alleviating the acoustic multimodality problem.  The whole idea of bilateral perturbation and leveraging CTC fine-tuning is simple and intuitive. Analysis of unit error rates showed effectiveness of proposed Bilateral Perturbation in reducing acoustic multimodality. With bilateral perturbation, the authors developed NAR S2ST (TranSpeech), while applying the standard KD technology in NAR to reduce linguistic multimodality and prior work of mask-predict NAR unit decoder. Experiments showed effectiveness of BiP over AR baseline system (basic conformer) by 1.2-4.4 BLEU points and NAR baseline system (Transpeech-Distill) by 0.6-1.8 BLEU points. 
strength_and_weaknesses: Strengths:

(1) The proposed bilateral perturbation approach for CTC fine-tuning to address the  acoustic multimodality problem is simple, intuitive and novel.

(2) With the reduced acoustic multimodality, the paper explored NAR for direct S2ST. As shown in the results, NAR TranSpeech with KD achieved significant speedup over AR direct S2ST without a big degradation in translation accuracy.

(3) The authors investigated a bag of tricks proposed in prior works, including conformer, KD for NAR, mask-predict NAR unit decoder, and other decoding choices including target length beam and NPD. Experiments evaluated efficacy of these tricks and analyzed performance-speed tradeoff, which are useful for future NAR efforts on direct S2ST. 

(4) Experiments and analyses are comprehensive. Experimental results on three language pairs of CVSS-C showed good gains in BLEU from the proposed BiP approach for both AR and NAR direct S2ST.  Analysis of unit error rates showed effectiveness of proposed Bilateral Perturbation in reducing acoustic multimodality.  The authors also compared to cascaded systems, which shows that there is still a lot of room for improvement for direct S2ST systems.

Weaknesses:

(1) The paper is overall clearly written. However, some technical details and discussions are missing and need to be included. Please check out the questions under Clarity.

clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: 

The paper is generally clearly written, but some technical details and discussions are missing and need to be included:

(1)	The style normalization stage did not consider normalizations in rhythm. This needs to be explained.

(2)	The HuBERT representations are pre-trained on natural speech. It would be useful to explain the quality of encoded representations by applying pre-trained HuBERT encoder on the style-normalized speech (whether encoding may suffer from reduced naturalness in the style-normalized speech).

(3)	For pitch information in Information Enhancement, please elaborate the motivation for the choice of the chain function in this order, instead of other orders or other designs. Appendix C only provides details on each function, still misses answers to this question.

(4)	For energy information in Information Enhancement,  please clarify how the average from a log-mel spectrogram is used.

(5)	In the ablation analysis, it would help to add ID=4 results in Table 2. Based these results, it seems that adding SN on top of Basic Conformer + IE only brought quite a small gain on BLEU, as 0.41, 0.05, 0.03 respectively. This discussion needs to be added and more insights on how to improve effect of SN are expected.

(6)	Note that there are some typos, for example, Section 5.1, “BELU” should be “BLEU”.

Novelty: 

The proposed BiP approach is novel, simple, and intuitive. Experiments showed effectiveness of BiP in reducing acoustic multimodality and improving translation accuracy. The NAR direct S2ST achieved significant speedup over AR direct S2ST without a big gap in translation accuracy.

Reproducibility:  

Although no source code is released, Section 5.1 and Appendix B and Appendix C provide sufficient details for reproducibility.


Presentation: 

There are some grammatical errors. For example,
(1)Introduction, “Among the conventional method”, method -> methods
(2)	Introduction, “recently proposed direct S2ST literature demonstrate…”,  demonstrate -> demonstrates 
(3)	There are quite a few subject-verb agreement errors.
(4)	Section 2, “demonstrates its outperformed translation results…”, outperformed -> outperforming

summary_of_the_review: This paper proposed a simple and intuitive bilateral perturbation (BiP) approach leveraging CTC fine-tuning for tackling acoustic multimodality in unit-based direct S2ST. The proposed BiP approach is shown to reduce acoustic multimodality. With this approach, the paper built non-autoregressive direct S2ST, TranSpeech, and achieved significant speedup over AR direct S2ST without a big degradation in translation accuracy. The authors also investigated other prior approaches such as conformer, KD, mask-predict unit decoder and other decoding choices for TranSpeech. The experiments are comprehensive. The proposed BiP approach achieved good BLEU gains over both AR and NAR baselines. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Continuous effort in addressing the reviewer's question
comment: **[Reference]**

[1] Fang Q, Feng Y. Neural Machine Translation with Phrase-Level Universal Visual Representations. In Proc. ACL 2022.

[2] Li B, Lv C, Zhou Z, et al. On Vision Features in Multimodal Machine Translation. In Proc. ACL 2022.

[3] Chen P, Itkina M, Senanayake R, et al. Evidential softmax for sparse multimodal distributions in deep generative models. In Proc. NeurIPS 2021.

[4] Amirian J, Hayet J B, Pettré J. Social ways: Learning multi-modal distributions of pedestrian trajectories with gans. In Proc. CVPR 2019.

[5] Gu J, Bradbury J, Xiong C, et al. Non-autoregressive neural machine translation. In Proc. ICLR 2018.

[6] Ren Y, Tan X, Qin T, et al. Revisiting Over-Smoothness in Text to Speech. In Proc. ACL 2022.

[7] Wang Y, Tian F, He D, et al. Non-autoregressive machine translation with auxiliary regularization. In Proc. AAAI 2019.

[8] Xiao Z, Kreis K, Vahdat A. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In Proc. ICLR 2022.

Again, we appreciate the reviewer's valuable reviews. We are happy to have your question about the unwritten language setting settled, and believe that your valuable comments have improved the work. For your continuous questions, we will try to address them carefully, and we would be grateful if we could hear your feedback regarding our answers to the reviews.

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Continuous effort in addressing the reviewer's question
comment: Thanks again for your constructive comments. We will address your question accordingly, and we hope our continuous response fully resolves your concern.

**[About multimodality]**

Thanks for the reviewer's feedback that requests more clarification about the "multimodality" term to resolve the misunderstanding. 

We agree with the reviewer that "multimodality" could be utilized by researchers to highlight multiple senses of information (e.g., text, speech, image), where researchers aim to build models that can jointly exploit information from multiple sources. For instance, Fang et al.[1] and Li et al.[2] introduce multimodal machine translation (MMT), incorporating visual data as auxiliary information to enhance translation performance. 

As a word with multiple meanings, we would like to clarify that "multimodality" has also been adopted to denote the multimodal (in contrast to unimodal) probability distribution in different scenarios, which is the main indication in this work. From a high-level perspective, Chen et al.[3] introduce a sparse normalization function to preserve the multimodality of probability distributions. Amirian et al.[4] propose to learn the multimodal distributions of pedestrian trajectories with generative NNs. 

Our work inherits the usage and definition of multimodality from previous literature: 1) Linguistic multimodality is proposed in neural machine translation [5], representing that multiple possible target translations correspond to a single sequence in the source language. 2) Acoustic multimodality is adopted from speech synthesis [6], representing that multiple possible speech in different tones correspond to one linguistic transcription due to the acoustic variations.

**[About the proposed bilateral perturbation and NAR]**

We agree with the reviewer that the bilateral perturbation and NAR approach respectively contribute to our goals (high quality and low latency) side-by-side, which are significant in direct speech-to-speech translation (S2ST).

Achieving these two goals is inherently challenging in various machine-learning research, where a line of work [7,8] has proposed novel techniques for different tasks. Sharing a common motivation for high-quality and low-latency translation, the bilateral perturbation serves as a pioneer in achieving these goals. We agree with the reviewer that NAR is an approach to sacrifice accuracy for speed. With the eased multimodal challenge, it could be more practicable to capture the distribution of target translation, where extra gain can be exchanged for speed. We will include this discussion in the revised version of the paper.

**[About the speech perturbation techniques]**

We thank the reviewer for placing a high value on the NAR part of TranSpeech, which is the first implementation and speed up the S2ST inference procedure. We agree with the reviewer that data augmentation/normalization approaches have been long-established techniques in speech feature extraction, while we would like to clarify the main focus and novelty of the proposed bilateral perturbation (BiP). 

Rather than introducing new data re-programming techniques, we adopt traditional signal processing to perturb speech samples. Therefore, instead of attributing the success of BiP to speech re-programming algorithms, we mainly focus on designing a novel pipeline including style normalization and information enhancement stage to perturb information flow in bilateral, and then leverage the CTC fine-tuning for acoustic-agnostic representation.

This novel pipeline learns the "average" information referring to linguistic content and generates deterministic representations, significantly reducing multimodality and promoting translation accuracy. Notably, the proposed bilateral perturbation pipeline could be applied to a broader range of challenges, which we hope may serve as a basis for future attribute-agnostic representation learning research.

From: ICLR.cc/2023/Conference/Paper2746/Reviewer_PYA8
title: Re: Response to Reviewer PYA8 (1/2) 
comment: I don't agree with the author's usage of "multimodality".
In the context of the paper, "speech multimodality" normally means "speech variation".
I don't think this is a common usage for the speech community.
In recent publications, "multimodality" refers to multiple sensory of information e.g. text, speech, image.

Regarding the connection between bilateral perturbation and NAR approach, I don't think there is a strong connection.
NAR is always an approach to sacrifice accuracy for speed.
According to author's response, the relation is that as bilateral perturbation leads to accuracy gain, extra gain can be exchanged for speed.
Thus, it is a bag of two tricks.

Among the two tricks (perturbation and NAR), I don't think the perturbation sounds as there are already a lot of  data augmentation/normalization approaches. (E.g. pitch and energy normalization have been a long-established techniques in speech feature extraction)
The novelty I highlight in this paper is the NAR part, as it is the first implementation of NAR on S2ST.

I agree with the author's response about the unwritten language translation setting.
The original paper "Direct Speech-to-Speech Translation With Discrete Units", which the author uses as a baseline, also uses written language pairs to simulate unwritten language scenario.

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Looking forward to further feedback
comment: Dear Reviewer PYA8,

Thanks again for your constructive comments. We would like to kindly remind you that we tried our best to respond to your concerns with extensive experiments, detailed explanations, and more precise presentations. As the end of the rebuttal period is approaching, we would be grateful if we could hear your feedback regarding our answers to the reviews. We would be happy to answer and discuss if you have further comments.

Best regards, Authors

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Looking forward to further feedback
comment: Dear Reviewer yedS,

Thanks again for your constructive comments. We would like to kindly remind you that we tried our best to respond to your concerns with extensive experiments, detailed explanations, and more precise presentations. As the end of the rebuttal period is approaching, we would be grateful if we could hear your feedback regarding our answers to the reviews. We would be happy to answer and discuss if you have further comments.

Best regards, Authors

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Summary of the response
comment: 
To all reviewers, ACs, and PCs:

We thank all reviewers for the valuable suggestions with their effort and time. Your comments have improved our work. Here we summarize our effort in addressing reviewers' concerns, and please refer to the responses to each reviewer for more details.

**[Extensive experimental comparison to demonstrate the effectiveness of proposed modules.]**

- In response to the reviewer's question, we have extensively included a wild dataset as an additional benchmark to demonstrate the effectiveness of the proposed bilateral perturbation (BiP) technique, and demonstrate that it consistently promotes S2ST by a large margin in challenging scenarios.

- To evaluate the impact of indeterministic training targets, we randomly alter discrete units and witness that the neural vocoder is sensitive to unit variations, showing that the indeterminacy prevents models from precisely reconstructing the meaning of a speech and results in a degradation of S2ST translation accuracy.

- Per the reviewers' suggestions, we include SNR and MOS as evaluation matrices to measure speech quality, verifying the effectiveness of TranSpeech in high-quality speech generation as baseline models.

**[More detailed explanations on model design.]**

- As shown in our responses, we explain the connection between bilateral perturbation and the NAR approach, and show that it is challenging to 1) achieve high translation accuracy due to the acoustic multimodality; 2) build a parallel model upon multimodal S2ST systems as it introduces further indeterminacy.

- In response to the reviewer's question, we have included the motivation and reported the impact of indeterministic training targets. Specifically, 1) the noisy target entangles both linguistic and acoustic information and hurts speech-to-unit translation performance, and 2) the variant units significantly decrease unit-to-speech generation quality.

- For seq2seq prediction, we leverage the success of the ASR model with CTC to learn a frame-wise latent alignment between the input speech and the output tokens, which is reasonable for our monotonic setup and results in faster convergence.


**[More precise definitions and presentation.]**

- As shown in our responses, we present detailed explanations and definitions of "multimodality" and introduce linguistic and acoustic multimodality.

- Per the reviewers' suggestions, we include a discussion on perturbation-based speech reprogramming for a more precise presentation.

- Per the reviewers' suggestions, we provide additional mathematical notations and add a new section for reproducibility statement.

In the meanwhile, we revise the manuscript according to the comments and suggestions of reviewers. In this rebuttal phase, we have tried to address them carefully, and we sincerely hope you can consider increasing your score if you find our reply solves your concern. We are always happy to have a further discussion and answer more questions raised by you.

Best regards, Authors

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Looking forward to further feedback
comment: Dear Reviewers,

Thank you again for the great efforts and the valuable comments.

We have carefully addressed the main concerns in detail. We hope you might find the response satisfactory. As the end of the rebuttal phase is approaching, we would be grateful if we could hear your feedback regarding our answers to the reviews. We will be very happy to clarify any remaining points (if any).

Thanks in advance,

Paper 2746 authors

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Paper Revision
comment: We thank all reviewers for the constructive feedback. Here we summarize the revision of the manuscript according to the comments and suggestions of reviewers:

- In abstract and section 1, we revise the instance according to the reviewer's suggestion.
- In section 1, we provide more insights and explanations on the connection between bilateral perturbation and the NAR approach.
- In section 3.1, we include detailed explanations on how indeterminacy units influence S2ST and put this in appendix C.
- In section 3.3.1, we include more mathematical notations for a clearer presentation.
- In section 5.2, we provide an additional evaluation of speech quality and put this in appendix D.
- In section 5.2, we adopt the mined dataset as an additional benchmark and put this in appendix E.
- In appendix A.2, we add a new subsection to include perturbation-based speech reprogramming as our related works.
- In appendix H, we add a new section to include a reproducibility statement.

To address and clarify these concerns, we carefully run new experimental results with different requirements, modify our manuscripts, and provide responses to your questions and comments. We are always happy to have a further discussion and answer more questions raised by you. 

Thanks in advance,

Paper 2746 authors

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Response to Reviewer 2n3x
comment: We are grateful for your positive review and valuable feedback, and we hope our response fully resolves your concern.

**[About the connections with perturbation-based related works]**

Thanks for the reviewer's suggestion. We include a discussion on perturbation-based speech reprogramming as follows: 

Various approaches that perturb information flow in acoustic models have demonstrated the efficiency in promoting downstream performance. Contentvec designs a random transformation and introduces a teacher-student training mechanism to achieve speaker disentanglement. SpeechSplit, AutoPST, and NANSY perturb the speech variations during the analysis stage to encourage the synthesis stage to use the supplied more stable representations. Voice2Series introduces a novel end-to-end approach that reprograms pre-trained acoustic models by input transformation learning and output label mapping. Wavprompt utilizes the pre-trained audio encoder as part of an ASR to convert the speech in the demonstrations into embeddings digestible to the language model.

In this work, we propose the bilateral perturbation technique with style normalization and information enhancement to reprogram the acoustic conditions in speech. During the fine-tuning stage, it guarantees the speech encoder learns only the linguistic information from acoustic-variant samples, significantly reducing the acoustic multimodality in unit-based S2ST. We envisage that our work serves as an inspiration for future perturbation-based speech reprogramming research.

**[About the impact of different acoustic condition]**

In response to the reviewer's question, we provide a brief primer on each component (e.g., rhythm, pitch, and energy) in Section 3.2. To visualize the acoustic variations, we apply the information bottleneck to perturb different acoustic information and illustrate the samples in Appendix G. 

To further measure how each term introduces indeterminacy in derived units, we calculate the unit error rate (UER) between units derived from original and perturbed speeches as quantitative evaluation in Table 1 in the manuscript: 1) Notably, the rhythm variation of speech samples among different acoustic components leads to a distinct alteration of self-supervised units. 2) Perturbing energy in speech results in a respectively minor multimodal issue, which could be significantly alleviated by the proposed bilateral perturbation technique.


**[About the setup of ASR model with CTC decoding]**

The self-supervised acoustic models for deriving discrete representations assume that the speech feature and output tokens are monotonic. As such, we leverage the success of the ASR model with CTC to learn a frame-wise latent alignment between the input speech and the output tokens. CTC fine-tuning enables the seq2seq prediction models to be monotonic, which is reasonable for our fine-tuning setup and results in faster convergence.

**[About the speech quality assessment]**

Thanks for the reviewer's suggestion. Following the publicly-available implementation fairseq [1], we include the SNR as an evaluation matrix to measure the speech quality across the test set. We approximate the noise by subtracting the output of the enhancement model from the input-noisy speech and then compute the SNR between the two. Further, we conduct crowd-sourced human evaluations with MOS, rated from 1 to 5 and reported with 95\% confidence intervals (CI). For easy comparison, the results are compiled and presented in the following table:

| Method   |    SNR  |      Fr-En BELU       |
|----------|:-------------:|:-------------:|
|Translation GT |/ | 4.22$\pm$0.06
|DirectS2ST	 | 46.45 | 4.01$\pm$0.07
|TextlessS2ST | 47.22  | 4.05$\pm$0.06
| TranSpeech  |  46.56 | 4.03$\pm$0.06 |  


As illustrated in Table, TranSpeech has achieved the SNR and MOS with scores of 46.56 and 4.03 competitive with the baseline systems. Since we apply the publicly-available pre-trained unit vocoder and leave it unchanged for unit-to-speech, we expect our model to exhibit high-quality speech generation as baseline models while achieving a significant improvement in translation accuracy. 


**[About clarity and reproducibility]**

Thanks for the reviewer's suggestion. We have included the motivation and reported the impact of indeterministic training target in Appendix C in the revised version of the paper. Besides, we have attached the perturbation-based methods as related works in Appendix A.2 and the reproducibility statement in Appendix H. 

**[References]**

[1] Wang C, Hsu W N, Adi Y, et al. fairseq S\^2: A Scalable and Integrable Speech Synthesis Toolkit. In Proc. EMNLP 2021.


Again, we thank the reviewer for the insightful reviews and for recommending accepting our paper. We sincerely hope you that can consider raising your score if you find our reply solves your concern.

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Response to Reviewer PYA8 (2/2)
comment: **[About trading BELU score for NAR generation]**

In response to the reviewer's question, Table 2 in the manuscript summarizes the translation accuracy and inference latency among all systems, and we have the following observations: 1) Moving an auto-regressive system to the parallel one (4 vs. 6) results in trading 3.8 BLEU points of translation accuracy on average for 11.04x speedup. 2) As introducing advanced NAR generation choices with length beam and noisy parallel decoding, the gap of translation quality (4 vs. 10) is reduced to 2.7 BLEU points while maintaining a 253\% speedup compared to autoregressive baseline.


**[About the unwritten language translation setting]**

Thanks for the reviewer's feedback that requests more clarification about the experimental setup to resolve the misunderstanding. The unit-based S2ST baseline systems [6,7] conduct studies on shared language pairs (Fr-En, and Es-En, etc) with transcriptions discarded, which has been defined as unwritten language scenarios. Following this practice, we prepare the CVSS-C dataset with transcriptions discarded and train the direct S2ST system without texts. As such, our work inherits the motivation of unit-based S2ST for unwritten language, and the experimental study is considered a fair and appropriate comparison with previous literature. 

To further address the reviewer's concern, we include an additional wild dataset introduced in [7] to demonstrate the effectiveness of the proposed bilateral perturbation technique, which consists of 338k audio-book samples automatically mined from the web. We evaluate across the test set and present the Fr-En results in the following tables:


| Method   |      Fr-En BELU       |
|----------|:-------------:|
| Autoregressive models |
| Basic Conformer | 22.13 | 
| Basic Conformer + BiP | 24.51  |  
| Non-autoregressive models with naive decoding |
|TranSpeech  | 18.98 | 
| TranSpeech + BiP  |  21.15 |  


Regarding the wild dataset without transcription in the unwritten scenario, bilateral perturbation consistently promotes S2ST by a large margin compared with the baselines. To conclude, the proposed technique demonstrates its efficiency in deriving deterministic targets for speech-to-unit translation, and thus we expect our model to exhibit better performance even in challenging wild scenarios.
     

**[About clarity and reproducibility]**

Thanks for the reviewer's suggestion. We have refined the mentioned instance and included a reproducibility statement in Appendix H in the revised version of the paper. 


**[References]**

[1] Pang B, Han T, Nijkamp E, et al. Learning latent space energy-based prior model. In Proc. NeurIPS 2018.

[2] Gu J, Bradbury J, Xiong C, et al. Non-autoregressive neural machine translation. In Proc. ICLR 2018.

[3] Ren Y, Tan X, Qin T, et al. Revisiting Over-Smoothness in Text to Speech. In Proc. ACL 2022.

[4] Qian K, Zhang Y, Gao H, et al. ContentVec: An improved self-supervised speech representation by disentangling speakers. In Proc. ICML 2022.

[5] Polyak A, Adi Y, Copet J, et al. Speech resynthesis from discrete disentangled self-supervised representations. In Proc. Interspeech 2021.

[6] Lee A, Gong H, Duquenne P A, et al. Textless speech-to-speech translation on real data. In Proc. NAACL 2022.

[7] Lee A, Chen P J, Wang C, et al. Direct speech-to-speech translation with discrete units. In Proc. ACL 2022.

Again, we appreciate the reviewer's valuable reviews and believe some misunderstandings are due to our clarity. In this rebuttal phase we will try to address them carefully, and we sincerely hope that you can consider increasing your score if you find our reply solves your concern.

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Response to Reviewer PYA8 (1/2)
comment: We thank the reviewer for the constructive feedback and for considering our work as "significant improvement is achieved" and "significant speed up". We understand that your concerns are mainly related to the paper's clarity, and hope our response resolves your concerns fully.

**[About the usage of multimodality]**

Thanks for the reviewer's feedback that requests more explanations on the term.
"multimodality" describes the data distribution: the target data is not from a simple unimodal distribution [1]. Specifically: 1) in the area of neural machine translation, Gu et al. [2] use "multimodality'' where multiple possible target translations correspond to a single sequence in the source language, which is resorted to linguistic multimodality; 2) in the area of speech synthesis, Yi et al. [3] use "multimodality'' where multiple possible speeches in different tones correspond to one linguistic transcription due to the acoustic variations, which is summarized as acoustic multimodality.


**[About the connection between bilateral perturbation and NAR approach]**

Thanks for the reviewer's feedback that requests more explanations on the connection between bilateral perturbation and the NAR approach. In modern textless speech-to-speech translation (S2ST), our goal is mainly two-fold: 1) high quality: direct S2ST is challenging especially without using the transcription. 2) low latency: high inference speed is essential when considering real-time translation.


Self-supervised discrete units learned from unlabeled speech have emerged to prove their capability in building textless S2ST systems. However, the current development of the unit-based S2ST system is hampered by two major challenges: 
- It is challenging to achieve high translation accuracy due to the acoustic multimodality. Specifically, the self-supervised units derived from speech with the same content could be different due to various acoustic conditions. Thus, the indeterministic training target for speech-to-unit translation fails to yield good results. 
- It is challenging to build a parallel model upon multimodal S2ST systems as it introduces further indeterminacy. A non-autoregressive (NAR) S2ST system generates all tokens in parallel without any limitation of sequential dependency, making it a poor approximation to the actual target distribution. With the acoustic multimodality unsettled, the parallel decoding approaches increasingly burden S2ST capturing the distribution of target translation.

In this work, 1) we propose bilateral perturbation to generate representations agnostic to acoustic variation. This novel technique alleviates the acoustic multimodal issue and leads to a more deterministic target distribution. 2) With eased multimodal challenge, it could be more practicable to establish a NAR S2ST technique and speed up the inference procedure.


**[About the impact of indeterministic training target]**

Thanks for the reviewer's feedback that requests detailed explanations on how indeterminacy units influence S2ST, which is mainly two-fold: 

- For speech-to-unit translation (S2UT), the most desirable speech representations [4] should be the ones that disentangle content information in speech from other interfering acoustic variations. In contrast, the noisy training target entangles both linguistic and acoustic information, bringing difficulties to the convergence of the training procedure (even more critical in NAR without limitation of sequential dependency) and reducing translation robustness to various signal corruptions. Considering the error propagation, the drop in S2UT accuracy distinctly hurts S2ST. 

- In unit-to-speech generation, previous work [5] suggests that representations are better disentangled than lumping acoustic variation together, where the indeterministic and variant units significantly decrease synthesis quality. For easy understanding, we randomly alter the values of discrete units and report the word error rate (WER) of generated samples across the test set. For easy comparison, the results are presented in the following table. It could be witnessed that the neural vocoder is sensitive to the input unit variations, and thus the indeterminacy prevents models from precisely reconstructing the meaning of a speech and results in a degradation of S2ST translation accuracy.


| Vocoder Input | WER      |
|----------|:-------------:|
| GT | /  |
| Original Unit | 9.7\% | 
| 5\% Altered Unit | 15.1\%  |  
| 10\% Altered Unit  | 21.3\% |
| 30\% Altered Unit  | 56.1\% |

From: ICLR.cc/2023/Conference/Paper2746/Authors
title: Response to Reviewer yed5
comment: We are grateful for your positive review and valuable feedback, and we hope our response fully resolves your concern.

**[About the connection between bilateral perturbation and NAR approach]**

Thanks for the reviewer's feedback that requests more explanations on the connection between bilateral perturbation and the NAR approach. In modern textless speech-to-speech translation (S2ST), our goal is mainly two-fold: 1) high quality: direct S2ST is challenging especially without using transcription. 2) low latency: high inference speed is essential when considering real-time translation.

Self-supervised discrete units learned from unlabeled speech have emerged to prove their capability in building textless S2ST systems. However, the current development of the unit-based S2ST system is hampered by two major challenges: 
- It is challenging to achieve high translation accuracy due to the acoustic multimodality. Specifically, the self-supervised units derived from speech with the same content could be different due to various acoustic conditions. Thus, the indeterministic training target for speech-to-unit translation fails to yield good results. 
- It is challenging to build a parallel model upon multimodal S2ST systems as it introduces further indeterminacy. A non-autoregressive (NAR) S2ST system generates all tokens in parallel without any limitation of sequential dependency, making it a poor approximation to the actual target distribution. With the acoustic multimodality unsettled, the parallel decoding approaches increasingly burden S2ST capturing the distribution of target translation.

In this work, 1) we propose bilateral perturbation to generate representations agnostic to acoustic variation. This novel technique alleviates the acoustic multimodal issue and leads to a more deterministic target distribution. 2) With eased multimodal challenge, it could be more practicable to establish a NAR S2ST technique and speed up the inference procedure.

    
**[About the variable benchmarks]**

Thanks for the reviewer's suggestion. We further include a wild dataset introduced in [1] to demonstrate the effectiveness of the proposed bilateral perturbation (BiP) technique. It consists of 338k audio-book samples automatically mined from Librivox with various acoustic conditions (e.g., rhythm, pitch, and energy). We evaluate across the test set and present the Fr-En results in the following tables:

| Method   |      Fr-En BELU       |
|----------|:-------------:|
| Autoregressive models |
| Basic Conformer | 22.13 | 
| Basic Conformer + BiP | 24.51  |  
| Non-autoregressive models with naive decoding |
|TranSpeech  | 18.98 | 
| TranSpeech + BiP  |  21.15 |  


In the mined dataset with a wide variety of acoustic conditions, bilateral perturbation consistently promotes S2ST by a large margin compared with the baselines. To conclude, the proposed technique demonstrates its efficiency in deriving deterministic targets for speech-to-unit translation, and thus we expect S2ST to exhibit better performance in variable benchmarks.


        
**[About clarity]**

Thanks for the reviewer's suggestion. We have refined the mentioned instance in the revised version of the paper. 


**[References]**

[1] Lee A, Gong H, Duquenne P A, et al. Textless speech-to-speech translation on real data. In Proc. ACL 2022.


Again, we appreciate your positive reviews and sincerely hope you can consider raising your score if you find our reply solves your concern.

From: ICLR.cc/2023/Conference/Paper2746/Reviewer_yedS
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposes two approach for an end-to-end speech-to-speech (S2S) translation: bilateral perturbation to focus the learning step on the linguistics information from the speech (i.e., disregarding acoustic properties such as rhythm and pitch that potential affects the consistency of the translation), and a non-auto-regressive unit decoding mechanism (mask-predict). Experimental results show the two approach to improve translation quality and unit decoding latency.   
strength_and_weaknesses: Strengthes
- The proposed style normalization step (that removes acoustic style information), and the enhancement step (that creates variable speech samples without affecting the linguistic information), are the main strength of the proposed S2S model. In addition to generate deterministic representation of the speech signal, the approach can be applied in several use cases as it will allow to better exploit speech representation learned from a self-supervised pre-training step incorporating both linguistic and acoustic signals. 
- Experimental evidences show the proposed approach (I and II), improves translation quality and a significant decoding latency. These improvements can enable the proposed approach to have multiple application use cases (including simultaneous S2S translation).  


Weaknesses
- Although, the proposed approaches demonstrate improvements for the respective metrics (quality and latency), its is hard to connect the two approaches (or at least it is not well motivated in the paper). I understand, the perturbation approach delivers by making NAR decoding of units in S2S translation is relatively easier, given the simplification on the acoustic features. However, its still unclear which is the main theme of the paper (I will assume the NAR approach is a by-product). 
- Experiments from a single benchmark - given the focus on the acoustic attributes, dataset from variable benchmarks (potentially with high variability in the acoustics information) could provide more strength for the experimental section.        
clarity,_quality,_novelty_and_reproducibility: Clarity
- The paper is well written, and understandable with enough background. The scope covering basic definitions of linguistics and acoustic information, illustrating and discussing the proposed approach (Figure 2), through the experiment and discussion part keeps the flow. 

Quality
- There are instance, that require a proof reading, to tune some of the parts of the work (for instance "The
parallel decoding algorithm requires as few as 2 iterations to generate outperformed samples," --> ... to generate samples that outperform ... ?). 

Novelty 
- The proposed approach follows previous work (see Figure 1a), and different from related work, the proposed approach explores more variants of acoustic features (rhythm, pitch, and energy) of the speech representation.
- NAR decoding

Reproducibility
- Model parameters are provided
- 
summary_of_the_review: This work shows interesting results for a S2S translation task, for quality and decoding latency. The proposed approach addresses the modeling of more acoustic attributes that are not explored in previous work. This work allows for a better use of pre-trained self-supervised models, by generating a deterministic representation. Despite the improvements, the experiment setup is narrow and the motivation to combine quality improvement with latency is poorly stated.  
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2746/Reviewer_PYA8
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper address the issues in the recent proposed S2ST framework and propose to solve the problems with two different aspect.
First of all, the paper proposes to alleviate the acoustic multimodal problem by using bilateral perturbation.
Obvious improvement is achieved with this method.
Next, the paper proposes to solve the high latency problem by introducing a non-autoregressive approach.
This leads to a significant speed up compared with the baseline.
strength_and_weaknesses: Strength:
This paper propose two methods to improve the translation accuracy and the inference speed of S2ST systems. Significant improvement is achieved.
It is the first implementation of non-autoregressive S2ST

Weaknesses:
Here are my questions to the paper: 

The author should check the definition of the word "multimodality" and the common usage of this word in the community.

"The proposed bilateral perturbation eases acoustic multimodality and makes it possible for non-autoregressive (NAR) generation" is stated in the introduction. Why the method of bilateral perturbation and the NAR issue is related? 

"the indeterministic training target for speech-to-unit translation fails to yield good results" is stated in section 3.1. What is the reason behind this judgement? The units, which generated from the speech with an unsupervised way, contain noise and it was known since HuBERT. The vocoder behind the S2UT should be able to cover this and generated the target waveform. In other way round, this might be a good thing as the acoustic variation is preserved.

How much BLEU is sacrificed from moving an auto-regressive system to a NAR system?

The major aim of unit-based S2ST is to deal with unwritten languages. 
Without this motivation and without the use of discrete units, I would consider the unit-based S2ST approach as a cascade of speech2text translation and TTS. 
While this paper is claiming their work on the foundation of unit-based S2ST, they are not evaluating  their models with unwritten language translation task.
I doubt that the comparison in Table 2 is not fair as the strength of the baselines should be on unwritten language tasks.



 


 

clarity,_quality,_novelty_and_reproducibility: Clarity: The writing is ambiguous and hard to read. E.g. the first sentence in the abstract.

Quality: Standard.

Novelty: Significant.

Reproducibility: The codes are not provided in the paper.


summary_of_the_review: I would recommend the paper to be accepted if my questions are settled.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2746/Reviewer_2n3x
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: ### Overview

The paper aims to generate much deterministic representation for end-to-end speech translation by characterizing style normalization and information enhancement. Overall, I like the paper on incorporating those ideas for S2ST tasks. 

In general, the author's design of both style normalization and acoustic information enhancement methods could be considered as one attribute-based enhancement sharing with some similar findings on latent space prompting or speech input reprogramming via perturbation to drive the model under different models and conditions. Both works are related to characterizing the perturbation for fine-tuning tasks. 

The authors could add some discussion on those fine-tuning pre-training AM works via perturbations to improve the overall impact of this work. 

I strongly recommend accepting this work. 
- Original 6; to 8 after rebuttal. 


***

**References**

A. Voice2series: Reprogramming acoustic models for time series classification, ICML 21

B. WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Models, Interspeech 2022




strength_and_weaknesses: ### Pros.

1. Startforward but effective approach via the perturbation with fine-tuning CTC 

### Cons.

1. Some related perturbation-based references are missing. 

2. Since Rhythm, Pitch, and Energy have all been used as information enhancement. The impact of each terms are not very clear. 


### Questions. 

1. It is not clear to me that the "ASR model with CTC decoding" has been used as one major setup in this work. Is that any difficulty in using the seq2seq prediction model under the proposed scheme?

2. I heard some high-pitch background noise for Fr-En (https://transpeech.github.io/) examples. Could the authors provide some SNR or speech quality assessment since additional information has been introduced in the proposed method?

***

### Post-Rebuttal

I read the authors' updated version and reply. 

- In general, my Q1 (seq2seq modeling) and Q2 (SNR) have been addressed. The authors have also stated a code release in their future version. 

- After re-reading, I think the major contribution (normalization and enhancement via perturbations) in section 3 is still a little blur in terms of writing since the speech module and notations are defined in their section 4. The authors could consider refine the proposed method in section with mathematical notation (e.g., $X$ and $p$) to improve their future clarification for general readers. 

- I updated my score to accept. This is a ~8 score paper to me. 
clarity,_quality,_novelty_and_reproducibility: Overall the paper with good clarity. Some motivation for the perturbation based fine-tuning could be included. 

In general, there are yet theoretical discussions on the fine-tuning process. But some connections to the previous perturbation-based theoretical study [A] could be included. 

The authors do not mention their code and license. I give a question mark on the reproducibility. [updated: authors state code is to be released. The issue seems to be resolved.]
summary_of_the_review: In general, it is one good work on tackling characterized perturbation and S2ST. Some references are missing but in general, the area of fine-tuning SSL with a customized approach is still new and worthy of more studies. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
details_of_ethics_concerns: No. 
recommendation: 8: accept, good paper