Paper: Correct and speak: accent reduction with minimum supervision (ICLR-2024)

Reviewer comment (Experiments):
Subjective listening tests are at a relatively small scale.

Author response:
Yes, you are right. But that is indeed costly for us, the students. In our test, the raters are college students majoring in foreign languages, who have also been informed about the characteristics of the Indian accent. For example, "Indian-accented speakers may place stress on different syllables, or their intonation may be more flat or elevated, which can make their speech sound more monotone or sing-song to native English listeners." We believe they have the ability to properly conduct accent tests. We also showcase our demo page.

Reviewer comment (Evaluation):
Did the authors consider other ways of measuring the effect of the influencing factors in Table 1? For example, the edit distance between the Hubert sequences of two files.

Author response:
The edit distance could also be a good choice. But we think the Longest Common Subsequence Ratio (LCSR) is more intuitive. In Table 1, first, we want to explore the percentage of pure semantic tokens without speaker identity leak, and the percentage of tokens affected by the accent. When the influencing factor is speaker identity, a 75% LCSR indicates that 25% of tokens have speaker identity leakage. A lower probability of speaker leak tokens explains why the generative model works in a zero-shot evaluation. And when the accent influence is added, the LCSR degrades from 0.75 to 0.54, indicating that approximately 21% of the tokens are affected by the accent.

Reviewer comment (Evaluation):
The subjective analysis in Table 2 has been conducted on a relatively small set of utterances and a small number of evaluators. Could you elaborate more on the reliability of the results in Table 2?

Author response:
We've expanded to 100 test pairs and 20 raters.

Reviewer comment (Presentation):
References Jiang 2023a and Jiang 2023b are the same.

Author response:
Sorry about this. We have corrected this in our updated paper.

Reviewer comment (Presentation):
VALL-E has better speaker similarity in Table 3 but not in Table 7. In Table 7 caption, please mention about the zero-shot condition (as opposed to the SPK comparison in Table 3).

Author response:
Our framework has better speaker similarity. In Table 3, the primary reason for lower Speaker Perplexity (SPK) but comparable values to VALLE-AC in the proposed model is that the speech generated by VALLE-AC (without conversion module) retains more accent information. This suggests that more similar speech acoustic information results in more similar speech representations. This is not fair. Therefore, we present Table 7 to compare our proposed generative-only model with VALLE-AC. To avoid misunderstanding, we directly compare our proposed generative-only module with the baselines in Table 3. The results are shown in Table 3.

Reviewer comment (Writing):
Minor typing issues: Section 4.3.2, first sentence: hubert -> Hubert.

Author response:
Sorry for such typing issues.  We have corrected these.

Reviewer comment (Writing):
Minor typing issues: Section 5.1. Baseline models: Hubert tokens for encoding Employing Hubert tokens for encoding -> please remove repetition.

Author response:
Sorry for such typing issues.  We have corrected these.

Reviewer comment (Writing):
Appendix is not well-formatted and has some typos (e.g. Accemt -> Accent).

Author response:
Sorry about the mistake. We have already made these corrections in our updated paper.

Reviewer comment (Experiments):
Section 3: Please consider comparing hidden units from HuBERT to PPG or other features used in literature. Otherwise the use of hidden units sounds adhoc.

Author response:
In our framework, we require a speech tokenizer that can extract highly semantic, highly compressed tokens. It has been demonstrated that HuBert model satisfies these requirements [1][2][3]. On the other hand, PPG is a frame-level feature that involves more speaker information leakage, making it more suitable for mapping tasks. The baseline model in AC-PPG illustrates that mapping from L2 PPGs(non-native feature) to L1 PPGs(native feature) may not be as effective.

Reviewer comment (Reproducibility):
Section 3.1: Please provide more details on the HuBERT setup. How is the model trained? How many clusters are used for hidden units identifications? These have a significant impact on the analysis results.

Author response:
Thank you for pointing this. We provide additional details in Section 3.1: "we utilize the HuBERT-Base (Hsu et al., 2021) model and a k-means algorithm with 500 clusters to extract HuBERT tokens". The pretrained model can be accessed at https://huggingface.co/facebook/hubert-base-ls960. Additionally, in Section 3.1, we've expanded to 1000 test pairs from 10 pairs. We arrived at similar conclusions, and our experimental results and analyses have been publicly commented and updated in the paper.

Reviewer comment (Experiments):
Section 3.1: In addition, 10 pairs don’t seem to be enough to have the conclusions on the impact of accent and speaker identity from hidden units. Please verify this on a larger dataset.

Author response:
Thank you for pointing this. We provide additional details in Section 3.1: "we utilize the HuBERT-Base (Hsu et al., 2021) model and a k-means algorithm with 500 clusters to extract HuBERT tokens". The pretrained model can be accessed at https://huggingface.co/facebook/hubert-base-ls960. Additionally, in Section 3.1, we've expanded to 1000 test pairs from 10 pairs. We arrived at similar conclusions, and our experimental results and analyses have been publicly commented and updated in the paper.

Reviewer comment (Experiments):
Section 3.2: Similarly, the results are less convincing with limited human ratings. Please consider increasing the number of pairs and the number of raters.

Author response:
In Sec 3.2, we've expanded to 100 test pairs and 20 raters.

Reviewer comment (Reproducibility):
Sample page: Please list an utterance that is used for acoustic prompting as well here.

Author response:
The utterance used for acoustic prompting is also derived from the source non-native speech. And we also add these in sample page. In our framework, we randomly select 3-second segments from the source speech to extract acoustic prompts, aiming to maintain the same speaker identity post-accent conversion. If the original speech is less than 3 seconds, we take the whole sequence of the original speech.

Reviewer comment (Experiments):
Section 5.2: In VCTK experiments, please use more source speakers instead of just one.

Author response:
We follow our baseline model AC-PPG‘s evaluation setting, which utilizes 20 test cases from a speaker in the VCTK dataset which is not parallel dataset. To explore why our model performs effectively through the LCSR metric, we require the parallel datasets like Arctic and L2-Arctic datasets.

Reviewer comment (Experiments):
Section 5.2: Accent: “During the AB test on accent score, the ground truth (GT) speech samples were selected from the native speaker ”bdl” in the L2-ARCTIC dataset.” Is it still the case when the non-native speaker is a female? If so, please consider using a female voice as the reference.

Author response:
As A4, the utterance used for acoustic prompting come from the source non-native speech to maintain the same speaker identity. The native speaker, 'bdl', serves as the ground truth speaker in our accent metric.

Reviewer comment (Experiments):
Section 5.4: Just curious but not required - Does the proposed correction work under one-shot or zero-shot setup? 15-mins of parallel data is already a lot in production scenarios.

Author response:
Our framework operates under zero-shot setup. The six test speakers are unseen by our generative module, and the conversion module has only been exposed to an Indian speaker. We do not require audios with the same content but from the same speakers with different accents. The speaker identities can be different. Furthermore, we are able to acquire 4 hours of such parallel data from the Arctic and L2-Arctic datasets.

Reviewer comment (Novelty):
From the perspective of method, introducing TF-Codec as a contribution seems independent and unrelated to the theme of paper (accent conversion or accent reduction). If the speaking module is replaced with a multi-stage generative model such as Encodec, will it affect the accentedness and speaker similarity of accent conversion?

Author response:
TF-Codec is the core design of our generative model which enables a single-stage causal speech generation. Compared with multi-stage speech generation based on EnCodec, it achieves lower complexity and has higher potential for low-latency applications. Besides complexity, TF-Codec generates speech with higher speaker similarity (0.502 vs 0.429 in terms of SPK）and naturalness (4.08 vs 3.84 in terms of MOS) in ***Table 3***.

Reviewer comment (Writing):
Does it remove all the dupilicated semantic tokens in HuBERT in both correction module and speaking module as stated in Section 3.1?

Author response:
In the training and inference phase, we don't remove duplicated HuBERT semantic tokens. Somehow, it helps to retain the original speech duration.  In ***Section 3.1***,  we remove duplicated semantic tokens to eliminate the effect of the duration of each word in the accent analysis on HuBERT tokens.

Reviewer comment (Writing):
The formula definition in “Pretraining” in Section 4.2. What is the purpose of defining $C^{t-1}$? It seems to be not used. Does $X^{t-1}$ already include $x_t$ or not? This part may be roughly understood, but it is not clear enough.

Author response:
Sorry for the mistake. We correct the formula in our paper.  Please refer to ***Formula (1)*** in ***Section 4.2***.

Reviewer comment (Experiments):
In section 5, what is the difference between “proposed” and “proposed-VC”? Are they the same one? Is “the traditional AC model (Liu et al., 2020)” equal to “AC-PPG”? It should be better to present the name and settings of each model more clearly.

Author response:
***Proposed-VC*** in Section 5 in previous paper is ***Proposed*** without conversion module. Sorry for the unclear expression. We correct these in ***Table 3*** in the updated paper, in which ***Proposed-VC*** refers to ***Generative model(TF-Codec)*** and ***Liu. et al*** refers to existing best machine-learning based AC method available in the public, previously named  ***AC-PPG***.

Reviewer comment (Writing):
Sec 2: "there has not been a parallel corpus that contains pairs of audios having the same contents yet coming from the same speakers in different accents" -- better to restrict such claims as "public corpus", as you don't know proprietary ones.

Author response:
**Q3**：Sec 2: "there has not been a parallel corpus that contains pairs of audios having the same contents yet coming from the same speakers in different accents" -- better to restrict such claims as "public corpus", as you don't know proprietary ones.  
**A3**：We update this "*For accent conversion task, there has not been a public parallel corpus that contains pairs of audios having the same contents yet coming from the same speakers in different accents.*"

Reviewer comment (Writing):
Sec 3.1: `LCSR = LCS / utterance_length` -- clarification is needed because the two utterance can have different lengths.

Author response:
**Q4**：Sec 3.1: LCSR = LCS / utterance_length -- clarification is needed because the two utterance can have different lengths.  
**A4**：We update this "*The length of the parallel data in each pair is similar and the utterance length ranges from 3 seconds to 5 seconds. The smaller utterance length is used to calculate the LCSR of each pair.*"

Reviewer comment (Evaluation):
Sec 3.1: Table 1 is misleading because "accent without phoneme corruption" and "accent with phoneme corruption" implies different speaker. So the trends in the table is completely expected. It cannot draw conclusion as the paper stated, that " the speaker identity causes little impact on the Hubert tokens and the content is not disturbed", "accent feature ... brings larger influence on the Hubert tokens".

Author response:
**Q5**：Sec 3.1: Table 1 is misleading because "accent without phoneme corruption" and "accent with phoneme corruption" implies different speaker. So the trends in the table is completely expected. It cannot draw conclusion as the paper stated, that " the speaker identity causes little impact on the Hubert tokens and the content is not disturbed", "accent feature ... brings larger influence on the Hubert tokens".  
**A5**： We update the description in my paper “As the results shown, with the accent source, HuBert tokens have changed a lot, degrade from 0.75 to 0.57 in terms of LCSR. For those cases with specific phoneme changes, more tokens have changed
from their native references(LCSR: 0.54).”

Reviewer comment (Experiments):
Sec 3.2: which model is used for synthesis? this section looks like experimental results, rather than analysis.

Author response:
**Q6**：Sec 3.2: which model is used for synthesis? this section looks like experimental results, rather than analysis.   
**A6**：The generative model in our proposed framework is used for this analysis experiment. In this section, we evaluate if the accent feature will be extended through the in-context learning. Based on this analysis, we find the accent style can hardly be transferred through the in-context learning in the generative model, which guides us to use the original accent source as the prompt to maintain the speaker identity.

Reviewer comment (Experiments):
Sec 5.1: any explanation on why the same speaker "ASI" is used for both training and evaluating?

Author response:
**Q7**：Sec 5.1: any explanation on why the same speaker "ASI" is used for both training and evaluating?  
**A7**: We utilize the speaker "ASI" for training and use 6 Indian speakers from VCTK, ARCTIC, and L2-ARCTIC for evaluation. We have now corrected this in Sec 5.1“*In the process of fine-tuning the correction model, we utilize data from speaker "bdl" as the native English speaker source ...... and data from speaker "ASI" ....... In the evaluation, we employ data from speaker p248 originating from the VCTK dataset, as well as data from five speakers, specifically ASI, KSP, RRBI, and SVBI, derived from the L2-ARCTIC dataset to assess the performance of models.*”

Reviewer comment (Experiments):
Sec 5.1: "In the AB test for accenteness, participants first listened to native and non-native reference audio. Subsequently, they heard paired speech samples (A and B) with identical content and were asked to choose the one that sounded more native-like." -- will rater be able to infer from the voice identity instead of accent?

Author response:
**Q8**：Sec 5.1: "In the AB test for accenteness, participants first listened to native and non-native reference audio. Subsequently, they heard paired speech samples (A and B) with identical content and were asked to choose the one that sounded more native-like." -- will rater be able to infer from the voice identity instead of accent?  
**A8**:  The native and non-native reference audio initially listened to have different speaker identities compared to the test pairs. Or if you're asking whether the speaker's identity is related to the accent in the test cases? Before the test, the raters have been taught to discern differences in Indian accents. And we direct the speakers to focus solely on accent differences.  These 5 speakers are all indian- accent speakers in l1-l2 arctic dataset.

Reviewer comment (Writing):
Sec 5.4: This seems completely unreadable to me. I don't understand what the sentences mean, and have no idea about what "3.5, 3, 2.7" numbers refer to as they are not presented in the corresponding Table.

Author response:
**Q9**：Sec 5.4: This seems completely unreadable to me. I don't understand what the sentences mean, and have no idea about what "3.5, 3, 2.7" numbers refer to as they are not presented in the corresponding Table.  
**A9**:  Sorry for this mistake. We have already deleted this.