Paper: TransLLaMa: LLM-based Simultaneous Translation System (ICLR-2024)

Reviewer comment (Evaluation):
Therefore, I urge the authors to provide a real-life inference cost/wall-time comparison to have a better picture of the cost trade-off here and makes the paper complete. I would appreciate and change scores if such report is produced.

Author response:
We have compared our method to the other state-of the-art model on inference wall time and summarize the results in Table 4 in Appendix E. In general, inference times are much longer for our model than for the smaller models we used as baselines (in Fukuda et al. 2023 and Papi et al, 2023). While our model is admittedly slower, several things could be done in future work to improve inference speed:

(1)	Use a shorter system message in the prompt or remove the prompt altogether. The system message we used in the paper is much longer than the average length of the sentence. In fact, the system message is only necessary if the base model is used zero-shot (that is without prior supervised fine-tuning). This can dramatically (by about ~40%) shorten the inference time.

(2)	Use more efficient quantization schemes. We noticed that loading the model in 4 bit increases inference time by about 43% compared to the native 16-bit quantization. We speculate that with more efficient quantization techniques and low bit width arithmetic hardware support (this is a currently very active area of research), inference speed is likely to become less of an issue from a practical point of view.

(3)	Take an end-to-end approach (as in [1, 2]), in which input audio is directly embedded (e. g. with Wav2Vec or HuBERT), reducing delays due to a separate ASR subsystem. In fact, some recent works [3] have already showed that raw audio chunks can be embedded and (after projecting with an adapter) be injected into the prompt as if they were text embeddings.

Encouragingly, GPT-4 (despite being presumably at least an order of magnitude bigger) performs much faster than Llama on the same task. This suggests that even very large language models can, if served efficiently and with proper optimizations, become a viable alternative to current compact SiMT systems. Finally, since Lllama2 (as many other LLMs) are trained predominantly on English text, the tokenizer encodes English more efficiently (that is with fewer tokens on average) than other, less represented, languages. For example, we made three texts from sentences sourced from TED talks, each 11500 characters long, in English, German and Russian and encoded them with the Llama-2 tokenizer. The resulting numbers of tokens were 2890, 3397 and 4129, respectively. This suggests that translating into English will be faster than from English. We now mention these considerations in the final paragraph of the last section (lines 252-265).

[1] Fukuda et al. (2023) NAIST simultaneous speech-to-speech translation system for IWSLT 2023. doi: 10.18653/v1/2023.iwslt-1.31.

[2] Papi et al. (2023) Attention as a guide for simultaneous speech translation. In Proceedings of the 61st Annual Meeting of the ACL.

[3] Fathullah et al. (2023). Prompting Large Language Models with Speech Recognition Abilities. ArXiv.

Reviewer comment (Presentation):
Fig. 2 looks broken.. I guess the key point here is that "away" is aligned to "befreite", but the alignment link is not shown in the original en-de alignment.

Author response:
Thank you for the opportunity to clarify this. The links are established by SimAlign (as we mention on line 123 in the revised manuscript). Indeed, the source word “away” would be aligned with “befreite”, but SimAlign does not treat phrasal verbs (such as “took away” in this case) as a whole, but rather as separate words. In this example, the constituent words of that verb (“took” and “away”) could not be directly linked to any German worlds in the target. This explains the missing link. As we suggest in the paper (lines 116-120 in the revised manuscript), the alignment does not need to be perfect for our method to work.

Reviewer comment (Experiments):
Have you tried small (<=1B) LLMs that would be more practical in a real-life SiMT scenatio?

Author response:
No, we have not. While a small LLM would run faster (and therefore be more practical in the real world), our focus was on exploring the utility and potential of a decoder-only transformer for the SiMT task. These decoder-only transformers tend to come in sizes much larger than 1B parameters. For example, the smallest version of Llama2, the latest open-source LLM at the time of writing the paper, has 7B parameters. There are of course older decoder-only transformers (e. g. GPT-2), but given that already at 13B parameters Llama2 performed significantly worse than at 70B parameters, we speculate that much smaller models would fail the SiMT task unless fully trained on a very large dataset of source-target pairs.

Reviewer comment (Evaluation):
The experiment only presents BLEU scores; it lacks concrete examples of output, which would be beneficial for understanding the translation quality.

Author response:
Thank you for this suggestion. We have added some examples to Appendix F of the revised manuscript.

Reviewer comment (Experiments):
I wonder if there are any experimental results for a setting when the target language is English such as DE-EN (instead of EN-DE). Since LLMs are typically trained mainly in English, I wonder if it makes a difference in the performance between En-X and X-En.

Author response:
We report results only for translation from English because we utilized MuST-c, a common and easily available dataset used to train and benchmark SiMT systems, which includes talks in English and their translations into another language. However, because the base LLM is pre-trained mostly on English text, we can speculate that translation into English should be better that into a foreign language. Moreover, translation into English should be faster, because fewer tokens on average are needed to encode an English text than a text of the same size (in terms of the number of characters) in another, less represented language (e. g. three texts made from sentences sourced from TED talks, each 11500 characters long, in English, German and Russian are encoded into 2890, 3397 and 4129 tokens, respectively, by the Llama-2 tokenizer. We now mention this in the final paragraph of the last section (lines 252-265).

Reviewer comment (Experiments):
Line 161. "We did not use beam search during generation." Is there any reason not to use beam search?

Author response:
Thank you for this question. Although we found beam search to improve translation quality (in locally run Llama2 models), it is not clear how using beam search would affect GPT-3.5 and GPT-4 translation performance. This is because OpenAI API does not expose any beam search-related settings (although it might use them internally). Also, beam search slows down the generation process. For these two reasons, we chose to focus on greedy search.

Reviewer comment (Novelty):
One of the main concerns regarding this paper is the reliance on a reference-based approach for the causal alignment introduced. While the paper claims to propose a novel method, similar ideas have been studied in previous simultaneous translation literature (e.g. [1]). However, the paper lacks a comparative analysis with these existing approaches in the experiment section, making it difficult to assess the novelty and superiority of the proposed method.

Author response:
Without claiming superiority of our approach (in its current implementation) in either computational cost or quality, we aimed to demonstrate, for the first time, that a general-purpose (that is, not pre-trained specifically for translation) decoder-only LLM can perform the SiMT task. The potential of such LLMs for SiMT can be unlocked, as we demonstrate, with minimal fine-tuning. The fact that causal alignment does not have to be perfect for the method to work suggests that further improvements can be achieved with a refined  alignment procedure and/or human simultaneous translations used as targets. We do make an effort to compare our model to a selection of published state-of-the-art SiMT systems. Additionally, we compare these systems and ours in terms of wall time it takes to process one second of audio (Appendix E in the revised manuscript).

Reviewer comment (Experiments):
A significant limitation of the reference-based approach is the potential mismatch between full sentence translation and simultaneous translation. The references used to generate the causal alignment are derived from complete sentence translations, which may not be suitable for the dynamic nature of simultaneous translation. Simultaneous translation requires the model to begin translation based on partial context, and the reference-based approach may not adequately capture the challenges and nuances specific to this task.

Author response:
Thank you for raising this important point. Despite being standard practice in the SiMT literature, using full sentence translations as references for causal alignment has its limitations and may not always be optimal. Indeed, offline translations quite often violate causality (later source words might appear earlier in the target). Causality violation happens less often in simultaneous translation, because interpreters are under pressure to produce translation as early as possible to avoid working memory overload (although causality can often be violated when interpreters translate what has not yet been said (e.g. [1]). On the whole, we agree that offline and simultaneous translations are quite different things, with the latter potentially more suitable for the purpose of causal alignment and LLM fine-tuning. However, due to difficulty of obtaining high-quality datasets of simultaneous translations, we chose to follow the standard practice in the SiMT literature of using TED talks, many of which are translated offline into languages other than English. Sticking to TED talks makes comparisons with previous approaches easier (e. g. SiMT models submitted to IWSLT in the past few years have been tested on the MuST-c dataset primarily sourced from TED talks). Importantly, as we emphasize in our paper, our approach works with encouraging results despite the imperfect alignment procedure and offline targets. 

[1] Amos et al. (2022), Prediction during simultaneous interpreting: Evidence from the visual-world paradigm. Cognition.

Reviewer comment (Evaluation):
Is the comparison in Figure 6 fair, considering that the authors only evaluate their method and two state-of-the-art models based on translation quality without considering latency?

Author response:
Thank you for this question. The comparison is made as fair as possible by keeping average lagging (AL) below about 2000 ms (which we briefly mention on lines 201-202). We also do provide comparative latency performance in Appendix C (Table 3, p. 15). To help the reader, we now refer the reader to Table 3 it in the caption of Figure 6.

Reviewer comment (Evaluation):
Did the authors observe a high variance in the latency of the resulting causal alignment, potentially due to the fact that the gold reference used is designed for full sentence translation rather than simultaneous translation?

Author response:
Thank you for the question. The average translation latency (measured by SimulEval) might depend on the average source-target “gap” (measured in words) resulting from causal alignment, which shifts target words into the future. Since for German, this “gap” is generally larger than for Russian, it is possible for latency metrics for English-German to be larger than those for English-Russian translation. However, based on Table 2 and 3, we did not observe a striking latency difference between these language pairs. Nor did we observe large differences in the variance of the latency estimates between the language pairs tested. A different picture might emerge if causal alignment were performed on simultaneous (rather than offline) translation targets. We leave this to future work.