Paper: It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition (ICLR-2024)

Reviewer comment (Experiments):
In evaluating performance, the paper omits a critical baseline: shallow fusion of ASR and an LLM. The method should be compared against (1) end-to-end ASR, (2) shallow fusion of ASR and LLM, and (3) GER-only. The authors compare against (1) and (3) but not (2), which would be important to determine the added value of GER over a simpler, on-the-fly approach.

Author response:
We apologize for the misunderstanding caused. The “static fusion" baseline in our experiment can be viewed as a kind of “shallow 
    fusion” approach. In static fusion, we conduct the grid search on a small unseen validation (200 utterances) to find the suitable 
    weight to balance the weights of LLM and ASR systems. This process is akin to searching for the optimal hyper-parameters of LM’s 
    weight in shallow fusion.

    Additionally, we have some other thoughts on your point about the comparison with shallow fusion. As you said “simple shallow 
    fusion can be done on-the-fly”, the foundation is that we have a well-trained ASR model and a language model (usually trained with 
    in-domain data). However, since UADF does not need any joint training process, it also can be done on-the-fly based on a well- 
    trained ASR model and GER model. In other words, it just replaces an in-domain LM with an in-domain LLM to independently 
    perform GER. Furthermore, because the GER utilizes the LoRA-tuning, the experiment of a single dataset can be completed in less 
    than one day with a single GPU (Nvidia A40 or 3090). Although this still requires more time than training an in-domain LM, GER is 
    more capable of independently predicting transcription, also resulting in a significant improvement to the ASR performance.

Reviewer comment (Evaluation):
There is a risk that the test data may have been seen by LLAMA. For example, the CHiME-4 dataset is based on WSJ text, which is public domain and could have been part of LLAMA’s training data. This raises concerns about the validity of large improvements on CHiME-4. It would be more informative to test on a closed dataset such as CHiME-5 that is less likely to have been memorized.

Author response:
Thanks for your question. This concern has been discussed in Hyporadise paper, we report their observations here: 1) They 
    randomly write some customized sentences with errors that are unseen for LLM, while these errors can be corrected. 2) They 
    employ the T5 model (trained with publicly available C4 dataset) as the error correction backbone model, and it still works on 
    different datasets. 
    To reconfirm this issue, we reviewed the pre-training dataset introduced in LLaMA paper [1], and also checked each subset of 
    training data. No traces of WSJ’s transcription were found in these datasets. Furthermore, considering that WSJ is a paid dataset, its 
    transcription is typically not used for pre-training of LLMs.    

- **Reference**

    [1] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

Reviewer comment (Writing):
Some sections of the paper use excessive, flowery language that should be made more concise. For instance, in the last paragraph of Section 1, phrases like “a pioneering UADF technique,” “adroitly allocates,” and “conspicuously outperforming” could be simplified. Authors should focus on reporting rather than embellishing.

Author response:
Thanks for your comment. We have modified our introduction according to your suggestion.

Reviewer comment (Writing):
The terms “early”, “mid”, and “late” fusion used in the paper may confuse readers familiar with ASR terminology such as shallow, cold, and deep fusion. The authors should either change the term “fusion” or explicitly clarify the differences in the paper.

Author response:
We fully understand your concern. According to your suggestion, we add a paragraph in Section 3.1 to illustrate the relationship to 
    previous works, which includes the shallow fusion, cold fusion, and deep fusion you metioned.

Reviewer comment (Experiments):
Some of the proposed methods depend on specific model choices. For example, “early fusion” assumes the use of Wav2Vec 2.0 as the encoder, and “late fusion” only works with encoder-decoder style models. The dominant ASR strategy in industry is conformer-transducers, which lack such speech tokens and have a 3-dimensional logit space including a blank token. The authors should address how their method would apply to such models.

Author response:
Thank you for bringing up this topic. We think that you provide a potential solution for a practical scenario: the conformer-transducer 
    model provides prompt response for users, and the LLMs perform error correction for high-quality results. To integrate them, we 
    think an asynchronous decoding strategy should be proposed, where LLM and transducer need to launch two separate decoding 
    iterations. When the transducer decoder predicts a “blank” token, it should skip the fusion process. Otherwise, the logits are passed 
    into LLM’s decoding iteration for dynamic fusion. However, under this setting, a primary challenge in asynchronous decoding we can 
    identify is the mismatch in length. A further alignment operation is required to ensure the temporal consistency between the logits 
    from two decoders.

Reviewer comment (Experiments):
It would be better to add some comparisons of the results with LLM scoring.

Author response:
We agree that UADF uses more acoustic information than GER. However, for LLM, this kind of acoustic information is 
    unrecognizable, or even harmful due to the large modality gap, according to the results of early fusion. Our work aims to leverage 
    this acoustic information to improve the ASR results. Therefore, we compared the proposed UADF with early fusion and mid fusion 
    that also utilize the acoustic information. 

    For the rescoring baseline, as GER has surpassed the upper bound of the rescoring-based method (as oracle $o_{nb}$ shown in 
    Hyporadise paper), so we directly compare with the result of GER. We add a further LM rescoring baseline and oracle information in 
    Appendix A.1 Table 6.

Reviewer comment (Theory):
In equation 5, do $f^{llm}$ and $f^{asr}$ need to be of the same length? If they do, how to ensure this?

Author response:
We wonder the “length” you asked is the length of distribution or decoding sequence, but both of them need to have the same 
    length. For distribution, they are the vocabulary size 1D embedding, since our ASR model has been aligned with LLM. For sequence 
    length, as it is auto-regressive decoding, so they predict the current token together including the <eos> (end of sentence), so the 
    sequence length remains consistent throughout.

Reviewer comment (Presentation):
In figure 2, it’s clear the probability of “all” from ASR is high. But it didn’t show where is the probability of “all” from LLM? And what’s the meaning to show the blue cross in the left bottom part and the green cross in the right bottom part?

Author response:
We are sorry for the unclear presentation. In Figure 2, we only show the top-2 candidates from ASR and LLM, where the probability of 
    “all” is too low (close to 0) to be presented. For UADF, we present the top-3 candidates after fusion. We have revised the caption to 
    illustrate it.

Reviewer comment (Experiments):
In the experiments part, for all late fusion experiments, the ASR model needs to be retrained with the training data. But it’s not clear whether all the training data are used to get a unified model or for each task (WSJ, ATIS, Chime-4), the model is updated only with the training data from this task.

Author response:
We retrain the ASR model as the tokenizer mismatch between ASR model and LLM. In other words, if Whisper has the same 
    decoding space with LLaMA, this step is not necessary as Whisper can perform as a general ASR engine. For the dataset you  
    mentioned, we employ a common Whisper encoder and train separated decoder (very small) for each dataset. These decoders 
    share the same decoding space with LLaMA.

Reviewer comment (Reproducibility):
The results in table 2 showed that without calibration, the proposed UADF did not perform better than the static method (WER 1.39 vs. 1.36). This means calibration is crucial for UADF, so the tuning of temperature t1 and t2 becomes a key to make it work. But the paper didn’t give enough information on these two parameters, such as what’s the optimal value, should we tune them for different tasks and so on.

Author response:
Thank you for bringing up this topic. Firstly, as we mentioned in this paper, we find t1 and t2 on each dataset with binary search 
    algorithms on a small validation set including only 200 unseen utterances (they can provide thousands of token examples). This 
    search strategy is also used in classification tasks [1]. However, compared with classification tasks, we have indeed found out that 
    the impact of t1 and t2 are intricate to the WER performance. Now we provide some empirical conclusion to illustrate the impact of 
    t1 and t2 according our experiment:

    (1) t1 acts on the LLM to alleviate its overconfidence problem. When WER is low, LLM tends to always output a confidence very 
    close to 1, which is difficult to be corrected if it’s a wrong prediction. Accordingly, t1 would reduce the peak value of the distribution.  
    (2) t2 acts on the ASR model that also exhibits an overconfidence phenomenon. Meanwhile, the calibration of ASR results in a 
    smoother probability distribution, which encourages more potential candidates to LLMs according to E.q.(10).  
    (3) In general, the benefits of t1 and t2 vary from different dataset due to their different WER levels. So we think the best solution is 
    employing a small validation set to find them for each dataset. Moreover, static fusion baseline also requires a validation set to find a 
    suitable weight (grid search).

Reviewer comment (Evaluation):
Do we have the WER of the top 1 hypothesis in HyPoradise dataset for different tasks? These values may not be useful to validate the proposed method. But it will make the readers understand better why we need to use LLM to refine the N Hypotheses.

Author response:
We have included it in the Appendix A.1 Table 6. Furthermore, we provide a LM rerank baseline and oracle performance for 
    comparison.

Reviewer comment (Experiments):
As noted above, tuning of temperature t1 and t2 becomes a key point to make the proposed method work. Did the author observe the same conclusion for other tasks (WSJ, Chime-4, LRS3)?

Author response:
Thank you for bringing up this topic. Firstly, as we mentioned in this paper, we find t1 and t2 on each dataset with binary search 
    algorithms on a small validation set including only 200 unseen utterances (they can provide thousands of token examples). This 
    search strategy is also used in classification tasks [1]. However, compared with classification tasks, we have indeed found out that 
    the impact of t1 and t2 are intricate to the WER performance. Now we provide some empirical conclusion to illustrate the impact of 
    t1 and t2 according our experiment:

    (1) t1 acts on the LLM to alleviate its overconfidence problem. When WER is low, LLM tends to always output a confidence very 
    close to 1, which is difficult to be corrected if it’s a wrong prediction. Accordingly, t1 would reduce the peak value of the distribution.  
    (2) t2 acts on the ASR model that also exhibits an overconfidence phenomenon. Meanwhile, the calibration of ASR results in a 
    smoother probability distribution, which encourages more potential candidates to LLMs according to E.q.(10).  
    (3) In general, the benefits of t1 and t2 vary from different dataset due to their different WER levels. So we think the best solution is 
    employing a small validation set to find them for each dataset. Moreover, static fusion baseline also requires a validation set to find a 
    suitable weight (grid search).

Reviewer comment (Experiments):
As noted above, tuning of temperature t1 and t2 becomes a key point to make the proposed method work. Do we need to tune these values for different tasks, or we could get them with one task’s data and apply them to other tasks?

Author response:
Thank you for bringing up this topic. Firstly, as we mentioned in this paper, we find t1 and t2 on each dataset with binary search 
    algorithms on a small validation set including only 200 unseen utterances (they can provide thousands of token examples). This 
    search strategy is also used in classification tasks [1]. However, compared with classification tasks, we have indeed found out that 
    the impact of t1 and t2 are intricate to the WER performance. Now we provide some empirical conclusion to illustrate the impact of 
    t1 and t2 according our experiment:

    (1) t1 acts on the LLM to alleviate its overconfidence problem. When WER is low, LLM tends to always output a confidence very 
    close to 1, which is difficult to be corrected if it’s a wrong prediction. Accordingly, t1 would reduce the peak value of the distribution.  
    (2) t2 acts on the ASR model that also exhibits an overconfidence phenomenon. Meanwhile, the calibration of ASR results in a 
    smoother probability distribution, which encourages more potential candidates to LLMs according to E.q.(10).  
    (3) In general, the benefits of t1 and t2 vary from different dataset due to their different WER levels. So we think the best solution is 
    employing a small validation set to find them for each dataset. Moreover, static fusion baseline also requires a validation set to find a 
    suitable weight (grid search).

Reviewer comment (Experiments):
As noted above, tuning of temperature t1 and t2 becomes a key point to make the proposed method work. How much minimum data should we use to tune these values?

Author response:
Thank you for bringing up this topic. Firstly, as we mentioned in this paper, we find t1 and t2 on each dataset with binary search 
    algorithms on a small validation set including only 200 unseen utterances (they can provide thousands of token examples). This 
    search strategy is also used in classification tasks [1]. However, compared with classification tasks, we have indeed found out that 
    the impact of t1 and t2 are intricate to the WER performance. Now we provide some empirical conclusion to illustrate the impact of 
    t1 and t2 according our experiment:

    (1) t1 acts on the LLM to alleviate its overconfidence problem. When WER is low, LLM tends to always output a confidence very 
    close to 1, which is difficult to be corrected if it’s a wrong prediction. Accordingly, t1 would reduce the peak value of the distribution.  
    (2) t2 acts on the ASR model that also exhibits an overconfidence phenomenon. Meanwhile, the calibration of ASR results in a 
    smoother probability distribution, which encourages more potential candidates to LLMs according to E.q.(10).  
    (3) In general, the benefits of t1 and t2 vary from different dataset due to their different WER levels. So we think the best solution is 
    employing a small validation set to find them for each dataset. Moreover, static fusion baseline also requires a validation set to find a 
    suitable weight (grid search).

Reviewer comment (Experiments):
As noted above, tuning of temperature t1 and t2 becomes a key point to make the proposed method work. What’s the optimal value for these two parameters for the tasks in this paper?

Author response:
Thank you for bringing up this topic. Firstly, as we mentioned in this paper, we find t1 and t2 on each dataset with binary search 
    algorithms on a small validation set including only 200 unseen utterances (they can provide thousands of token examples). This 
    search strategy is also used in classification tasks [1]. However, compared with classification tasks, we have indeed found out that 
    the impact of t1 and t2 are intricate to the WER performance. Now we provide some empirical conclusion to illustrate the impact of 
    t1 and t2 according our experiment:

    (1) t1 acts on the LLM to alleviate its overconfidence problem. When WER is low, LLM tends to always output a confidence very 
    close to 1, which is difficult to be corrected if it’s a wrong prediction. Accordingly, t1 would reduce the peak value of the distribution.  
    (2) t2 acts on the ASR model that also exhibits an overconfidence phenomenon. Meanwhile, the calibration of ASR results in a 
    smoother probability distribution, which encourages more potential candidates to LLMs according to E.q.(10).  
    (3) In general, the benefits of t1 and t2 vary from different dataset due to their different WER levels. So we think the best solution is 
    employing a small validation set to find them for each dataset. Moreover, static fusion baseline also requires a validation set to find a 
    suitable weight (grid search).

Reviewer comment (Experiments):
It’s noted that “beta” in eq. (10) is set to 0.5 for the experiments. It didn’t show what’s the performance is if we change it to other values. Will the results be sensitive to this value?

Author response:
We set beta as a fixed value of 0.5 in the ASR experiments that avoids introducing too many hyper-parameters. In practice, the 
    results are not sensitive to beta when it varies from 0.3 to 0.5 on ATIS, WSJ, and ChiME, as the GER performs better than ASR. 
    However, on other datasets, when two systems achieve comparable WER results on validation set (e.g. low SNR condition in AVSR 
    task), we can reduce the value of beta to improve the importance of the second model in E.q.(10).