Paper: A Discrete and Variational Approach to Speech Representation Learning (ICLR-2024)

Reviewer comment (Writing):
The formulation presented in the paper is not well-described; Figure 1b does not contribute to the understanding of the proposed approach and should be rebuilt entirely, and the learning process should also be explained in more detail.

Author response:
The
original figure 1(b) is closer to the graphical rather than the
feed-forward process that is more familiar in deep learning. We update
a first draft of figure 1(b) to show the loss and the feed-forward process.

> The formulation presented in the paper is not well-described; Figure 1b does not contribute to the understanding of the 
proposed approach and should be rebuilt entirely, and the learning process should also be explained in more detail.

We have a draft version of [an updated figure 1(b)](https://i.imgur.com/XBfgSO4.png).

Reviewer comment (Experiments):
The use of simplified versions of benchmarking models limits the evidence of performance improvements presented in the paper.

Author response:
Our focus is on the proposed loss function, and the benchmarking is
exactly controlled under the same model architecture. The goal is more
to show the connection among approaches, and less about getting the
best numbers.

Reviewer comment (Theory):
How the proposed model guarantees the identifiability of distributions q(z_i|x_i) and p(z_i|x_\m) is unclear. Is the model train in fully unsupervised or (self-supervised) learning strategy or do the authors uses a force alligment to get the one-hot encoding vectors for all the experiments?

Author response:
We are not sure about the identifiability mentioned here. We certainly
make no guarantee that the auxiliary distribution q will be close to the
posterior. We also want to caution about degeneracy. When the two sets
A and B in equation (1) have overlaps, p and q can collaborate and lead
to degeneracy. We will revise the paragraph before 2.1 to make this clear.

> Is the model train in fully unsupervised or (self-supervised) learning strategy or do the authors uses a force alligment to get the one-hot encoding vectors for all the experiments?

Our variational approach is unsupervised and self-supervised, and does
not rely on forced alignments.

Reviewer comment (Reproducibility):
The authors did not clarify their process to update the codebook; the whole learning process should be explained better.

Author response:
In terms of the training process, the codebook V in equation in (5) and
(6) are jointly optimized with gradient descent when minimizing equation
(3).

Reviewer comment (Experiments):
The authors did not perform any experiment to evaluate the effect of the codebook size, which was arbitrarily set to 100. According to previous results using VQ strategies for speech representation, that is a too-small value.

Author response:
We follow the setting in HuBERT. See Section IV. B. in Hsu et al. (2021) [1].

Reviewer comment (Evaluation):
Is there any reason that explains why the future prediction model outperforms the masked prediction training on speaker verification?

Author response:
Our submission is the few to study speaker verification with causal
self-attention, it is unclear at this point what the potential cause is.

The key hyperparameters that decide what's learned in the representations
are the time shift for future prediction and the masking probability
and masking span in masked prediction. More research is needed to draw
the connection between these hyperparameters and the learned representations.

Reviewer comment (Writing):
The PER acronym is used before definition.

Author response:
We thank the reviewer for catching the errors and will revise accordingly.

Reviewer comment (Theory):
There is an error in equiation (11) second row, las term should be p(z|x_A).

Author response:
We thank the reviewer for catching the errors and will revise accordingly.

Reviewer comment (Experiments):
The theoretical argument is interesting, but the experiments are quite weak. HuBERT and wav2vec 2.0 are crippled, and then the new representation is shown to outperform them. Crippling the baselines might be forgivable if the crippling was irrelevant to the theoretical claims, but it is not. HuBERT is crippled by not retraining the K-means codebook every few epochs, and wav2vec by removing the codebook entropy loss; these are directly relevant to the theoretical claims, and cause the experimental tests to be insufficient proof of the theoretical claims.

Author response:
We thank the reviewer for the valuable and detailed feedback.

There is a concern about our experimental methodology (quoted below)
about the submission intentionally crippling the baselines. However, the
reviewer seems to have misunderstood HuBERT and wav2vec. We will pinpoint
where the confusion comes from and propose revision to our submission.

> The theoretical argument is interesting, but the experiments are quite weak. HuBERT and wav2vec 2.0 are crippled, and then the new representation is shown to outperform them. Crippling the baselines might be forgivable if the crippling was irrelevant to the theoretical claims, but it is not.


1. Based on the quote below, the reviewer claims that HuBERT updates
the k-means codebook during training.

> HuBERT is crippled by not retraining the K-means codebook every few epochs ...

> Against recommendations in the original HuBERT paper, this paper does not re-train HuBERT's codebook between epochs of transformer training. Figure 2 then shows that the proposed method achieves superior performance because it adapts the quantizer representation in a series of modes, which HuBERT cannot do because the authors chose not to allow it. Indeed, re-training the K-means codebook in the manner recommended in the original HuBERT article would probably lead to a similar learning curve to VLB.

The above claim is technically incorrect, in the sense that the codebook
in HuBERT is updated every gradient step, not every few epochs (See Hsu
et al. (2021) for the term e_c in equation (3) [1] .). The targets (i.e.,
that cluster assignment) to frames, however, are updated only after every
iteration (See Hsu et al. Section IV. B.). The targets are updated with a
new round of k-means, so a new codebook is produced every iteration, not
every few epochs. K-means centroids across iterations are not compatible
with each other. For example, each centroid in the first iteration has 39
dimensions (for MFCCs), while each centroid in the second iteration has
768 dimensions (for HuBERT hidden vectors).

Besides, iteration two and onwards in HuBERT are technically not
unsupervised or self-supervised, because the selection of a HuBERT
layer to produce targets relies on phone and cluster purity.

We have stated explicitly in 5.1 that we do not do multiple iterations
in our submission, because our approach can instantiate the HuBERT loss
function for a particular iteration. In principle, it is possible to
instantiate loss functions for multiple iterations, but that is not the
goal of the paper. The goal of the paper is to show that it is possible
to instantiate the HuBERT loss function with our proposed approach.


2. In terms of wav2vec 2.0, there is one concern about the diversity
loss and about our experimental methodology.

> "As opposed to previous work in advocating codebook usage" -- The wording of this paragraph suggests that wav2vec increases a quantity while you decrease the same quantity, which is not true. Your formulation measures D(q||p); diversity loss measures H(q). Indeed, this is where the choice to remove H(q) from your wav2vec implementation is particularly troubling. Wav2vec minimizes -H(q)-Eq[logp(z)], which is exactly D(q||p). 

We admit the wording issue for sending the wrong message. We agree
that our entropy term serves the same purpose as the diversity loss in
wav2vec 2.0.

The point is that the diversity of codebook usage is encouraged and
is part of the natural consequence of our variational bound, while the
diversity loss in wav2vec 2.0 is given without any justification. (See
Section 3.2 in Baevski et al. (2020) [2].) Another difference is that
the diversity loss in wav2vec 2.0 is the entropy of an average within a mini-batch of utterances,
while ours is computed per frame in Eq (3).

> In other words, if you add back the entropy loss, wav2vec is already minimizing exactly the quantity proposed in this paper, and there should be no difference in performance between wav2vec and VLB.

When instantiating wav2vec 2.0 in our approach, the entropy term has to
be 0 because of q being one-hot. There is also no reconstruction term
in wav2vec 2.0, which our approach has. We will revise section 4.3 to
reflect the nuances.

For the experimental methodology, the reviewer solely focuses on the
cross entropy loss.
> The original wav2vec 1.0 paper included all of the components of the proposed approach, but in that paper, the codebook entropy was presented as sort of an ad-hoc method of avoiding mode collapse.
> Crippling the baselines might be forgivable if the crippling was irrelevant to the theoretical claims, but it is not. [...] and wav2vec by removing the codebook entropy loss ...

The above claim ignores the fact that our loss function also includes
a reconstruction term that is missing in wav2vec 2.0. Again, the entropy term has to be 0 because of the one-hot q.

To avoid the confusion, we will revise section 4.3 and detail the
difference between our approach and wav2vec 2.0.

Reviewer comment (Experiments):
Against recommendations in the original HuBERT paper, this paper does not re-train HuBERT's codebook between epochs of transformer training. Figure 2 then shows that the proposed method achieves superior performance because it adapts the quantizer representation in a series of modes, which HuBERT cannot do because the authors chose not to allow it. Indeed, re-training the K-means codebook in the manner recommended in the original HuBERT article would probably lead to a similar learning curve to VLB.

Author response:
1. Based on the quote below, the reviewer claims that HuBERT updates
the k-means codebook during training.

> HuBERT is crippled by not retraining the K-means codebook every few epochs ...

> Against recommendations in the original HuBERT paper, this paper does not re-train HuBERT's codebook between epochs of transformer training. Figure 2 then shows that the proposed method achieves superior performance because it adapts the quantizer representation in a series of modes, which HuBERT cannot do because the authors chose not to allow it. Indeed, re-training the K-means codebook in the manner recommended in the original HuBERT article would probably lead to a similar learning curve to VLB.

The above claim is technically incorrect, in the sense that the codebook
in HuBERT is updated every gradient step, not every few epochs (See Hsu
et al. (2021) for the term e_c in equation (3) [1] .). The targets (i.e.,
that cluster assignment) to frames, however, are updated only after every
iteration (See Hsu et al. Section IV. B.). The targets are updated with a
new round of k-means, so a new codebook is produced every iteration, not
every few epochs. K-means centroids across iterations are not compatible
with each other. For example, each centroid in the first iteration has 39
dimensions (for MFCCs), while each centroid in the second iteration has
768 dimensions (for HuBERT hidden vectors).

Besides, iteration two and onwards in HuBERT are technically not
unsupervised or self-supervised, because the selection of a HuBERT
layer to produce targets relies on phone and cluster purity.

We have stated explicitly in 5.1 that we do not do multiple iterations
in our submission, because our approach can instantiate the HuBERT loss
function for a particular iteration. In principle, it is possible to
instantiate loss functions for multiple iterations, but that is not the
goal of the paper. The goal of the paper is to show that it is possible
to instantiate the HuBERT loss function with our proposed approach.

Reviewer comment (Theory):
The wording "As opposed to previous work in advocating codebook usage" suggests that wav2vec increases a quantity while you decrease the same quantity, which is not true. Your formulation measures D(q||p); diversity loss measures H(q). The choice to remove H(q) from your wav2vec implementation is particularly troubling. Wav2vec minimizes -H(q) - Eq[log p(z)], which is exactly D(q||p). In other words, if you add back the entropy loss, wav2vec is already minimizing exactly the quantity proposed in this paper, and there should be no difference in performance between wav2vec and VLB.

Author response:
2. In terms of wav2vec 2.0, there is one concern about the diversity
loss and about our experimental methodology.

> "As opposed to previous work in advocating codebook usage" -- The wording of this paragraph suggests that wav2vec increases a quantity while you decrease the same quantity, which is not true. Your formulation measures D(q||p); diversity loss measures H(q). Indeed, this is where the choice to remove H(q) from your wav2vec implementation is particularly troubling. Wav2vec minimizes -H(q)-Eq[logp(z)], which is exactly D(q||p). 

We admit the wording issue for sending the wrong message. We agree
that our entropy term serves the same purpose as the diversity loss in
wav2vec 2.0.

The point is that the diversity of codebook usage is encouraged and
is part of the natural consequence of our variational bound, while the
diversity loss in wav2vec 2.0 is given without any justification. (See
Section 3.2 in Baevski et al. (2020) [2].) Another difference is that
the diversity loss in wav2vec 2.0 is the entropy of an average within a mini-batch of utterances,
while ours is computed per frame in Eq (3).

> In other words, if you add back the entropy loss, wav2vec is already minimizing exactly the quantity proposed in this paper, and there should be no difference in performance between wav2vec and VLB.

When instantiating wav2vec 2.0 in our approach, the entropy term has to
be 0 because of q being one-hot. There is also no reconstruction term
in wav2vec 2.0, which our approach has. We will revise section 4.3 to
reflect the nuances.

Reviewer comment (Theory):
One of the differences between future prediction and masked prediction is that, using future prediction, it's possible for each frame to serve two roles: to be predicted by its predecessor frames, while it is also a predictor of future frames. Eq. (2) trivializes this by saying that the sum of all prediction log probabilities is less than or equal to the log probability of predicting the rest of the sequence from the first k frames.

Author response:
3. There is a minor concern about how future prediction is formulated.

> Eq. (2) trivializes this by saying that the sum of all prediction log probabilities is less than or equal to the log probability of predicting the rest of the sequence from the first k frames.

We will rewrite the log probability of equation (2) as below to make clear the factorization of the autoregressive process:
$L_{\text{AR}} \geq -\sum_{t=1}^{T-k}\log p(x_{t+k}|x_{\leq t})$

Reviewer comment (Writing):
There are a large number of grammar mistakes that slow understanding of the paper. Notable among these include the strange wording in the second line of the abstract, and the notational error in the second line of the equation in Appendix A.

Author response:
4. We thank the reviewer for the detailed feedback. We will revise the
submission accordingly.

> There are a large number of grammar mistakes that. Some of them, slowed my understanding of the paper somewhat: notable among these include the strange wording in the second line of the abstract, and the notational error in the second line of the equation in Appendix A.

Reviewer comment (Experiments):
The authors demonstrated the connection between their approach and VQ-APC and many more methods, but they only compared tow wav2vec2 and HuBERT. Also, the authors mostly only test one WSJ. To make the claim stronger, does it make sense to compare to more methods you have mentioned and tested on more downstream datasets?

Author response:
It is indeed interesting to compare to VQ-APC. Given the short timeline,  
we cannot guarantee to include VQ-APC during the rebuttal period, but  
will aim for the camera-ready version.

Reviewer comment (Experiments):
Compared with wav2vec-2 and HuBERT, does the proposed framework have advantages or disadvantages in terms of GPU hours? This analysis could be interesting as the authors are proposing a general framework.

Author response:
We do not observe a significant difference in GPU hours between HuBERT  
and the proposed method. We do find wav2vec-2 more computationally  
expensive due to the negative sampling step.

Reviewer comment (Experiments):
In Table one and two, VLB-base archives even more significant improvement. Does this sound reasonable? My understanding is that, Table one and two are strong evidences on the optimization benefits of the variational framework; However, the baseline can be stronger with more tuning, better initialization, optimizer scheduler, and even more data; That is, the gap between the Hubert/wav2vec-2 and VLB in could much smaller than what is shown in this draft.

Author response:
The gap might be smaller if we fine-tune on the downstream tasks instead  
of freezing the pre-trained models. However, all comparisons share almost the  
same hyperparameters, and we expect any improvement to the baseline to  
transfer to the proposed approach.

Reviewer comment (Experiments):
In Table three, are the rate and distortion calculated on dev93, eval92 or training data? Similar question to figure 2, is the PER curve on dev93 or eval92, or train?

Author response:
The rate and distortion are calculated on the training data (LibriSpeech  
960 hr). The PER curve is evaluated on dev93. We will add more details  
to Table 3 and Figure 2.

Reviewer comment (Theory):
There are some analogy and connection to other model make no sense. For example, "The loss function becomes cross entropy if D contains all possible codes in V , and each code is uniquely sample", this is simply the difference of softmax and contrastive learning, I don't know what this rephrase means. I cannot see the proposed loss generalize anything to contrastive based approach.

Author response:
The contrastive loss is related to the cross-entropy term in the
variational bound. Maximizing InfoNCE is analogous to maximizing the
standard cross-entropy loss if negative samples always contain all
possible values of codes (See Eq. 2 in Kong et al. [1]).

Reviewer comment (Novelty):
Based on (1), the proposed method is more like a unified version of w2v-bert [1] and best-rq [2], both of them using a mlm loss and learn the code on-the-fly without k-means.

Author response:
Indeed, w2v-BERT has both an MLM branch and a contrastive branch.
(See Fig 2. in Chung et al. [2].) Our approach can instantiate MLM and
a contrastive loss, but not both. The goal of our submission is to show
that the two are special cases, not to have both at the same time.

BEST-RQ does not update the codebook during training (See the abstract
in Chiu et al. [3]), and can be seen as a simplified HuBERT. We will
revise 4.2 to include a short discussion on this.

Reviewer comment (Experiments):
Experimental results are weak. No causal baseline been compared.

Author response:
We do have results for a HuBERT trained with causal self-attention,
and will update Table 2 accordingly. Note that our approach is already
better than the causal baseline Baevski et al. (2020) [4] in Table 5, so the results are by no
means weak.

Reviewer comment (Novelty):
The paper is unify causal (predictive) and non-causal (mask based), but none of such unification work been mentioned in the paper. Can the author survey and add it?

Author response:
The closest attempt is Kong et al. [1], and we will revise and point
readers to related work therein.

Reviewer comment (Novelty):
Can the author explain more on the difference of proposed approach versus VQ-CPC? Am I right conceptually it's replacing contrastive predictive coding with mlm loss?

Author response:
There is no reconstruction term in VQ-CPC, while there is one in our approach.