Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Pair (2024)

Abstract: In Simultaneous Machine Translation (SiMT), training with a simultaneous interpretation (SI) corpus is an effective method for achieving high-quality yet low-latency. However, constructing such a corpus is challenging due to high costs, and limitations in annotator capabilities, and as a result, existing SI corpora are limited. Therefore, we propose a method to convert existing speech translation (ST) corpora into interpretation-style corpora, maintaining the original word order and preserving the entire source content using Large Language Models (LLM-SI-Corpus). We demonstrate that fine-tuning SiMT models using the LLM-SI-Corpus reduces latency while achieving better quality compared to models fine-tuned with other corpora in both speech-to-text and text-to-text settings. The LLM-SI-Corpus is available at https://github.com/yusuke1997/LLM-SI-Corpus.

## introduction simultaneous machine translation (simt) 1 @xcite @xcite translates input in real-time by incrementally processing partial segments rather than waiting the whole sentence completion. while offline machine translation (mt) works without time restrictions, simt begins translating at certain points due to time limitations; therefore, balancing its latency and quality is crucial. this challenge is especially difficult in language pairs with drastically different word orders, such as english and japanese (svo vs. sov) @xcite @xcite . to manage word order differences in simultaneous settings, one strategy is to maintain the source language word order as much as possible to keep transcription translation (offline) si corpus ted talks naist-sic-aligned-st llm-si-corpus (ours) naist-cwmt (test only) si corpus si corpus figure 1 : the corpora used in this study, each created from the same ted talks data. ted talks are accompanied by english-japanese offline mt data. naist-sic-aligned-st @xcite is an si dataset created by transcribing audio data of these talks by human interpreters. naist english-to-japanese chunkwise monotonic translation evaluation dataset 2024 (naist-cwmt) @xcite is manually created based on offline mt data from ted talks, following the cwmt guideline @xcite , and used only for testing purposes. our llm-si-corpus was created by llms based on the cwmt guideline and comprises training, development, and test sets. to address these challenges, @xcite manually created test data (chunk-wise) following chunk-wise monotonic translation (cwmt) guideline @xcite , with fluency and adequacy verified by professional interpreters. a key feature of chunk-wise is its monotonic alignment with the source, maintaining the entire source content, making it well-suited for the goals of simt. cwmt is designed for englishto-japanese si to reduce latency by segmenting sentences into grammatical chunks and translate sequentially. however, despite its potential, the reliance on human labor for dataset creation remains a significant barrier for scaling. therefore, we propose a method to convert existing speech translation (st) corpora into sistyle data (llm-si-corpus), closely maintaining the original word order and preserving the entire source content based on the cwmt guideline using large language odels (llms) as shown in figure 1 . we demonstrated that fine-tuning simt models with the llm-si-corpus, in both text-to-text and speech-to-text settings, achieves better translation quality with minimal latency compared to models fine-tuned with other corpora and the pretrained model. to summarize, our contributions are as follows: ## si-corpus construction with llms to address the limitations of the current si corpus, we leverage llms, which are known for their high translation performance and ability to perform purpose-specific translations based on instructions @xcite . for our purpose, we follow the cwmt guidelines to automatically convert st into si corpora using llms to be more monotonic while maintaining fluency, making it suitable for simt training. ## quality analysis of llm-si corpus quality monotonicity we analyzed the word alignment and evaluated the extent to which monotonicity improved between the source and different reference for gpt-3.5, gpt-4, sic, offline, and chunkwise. we used awesome-align (dou and @xcite to compare the source and reference, and evaluated the alignment consistency using spearman's correlation coefficient. on the other hand, the monotonicity of sic is comparable to that of offline, suggesting that the transcription of a simultaneous interpreter does not necessarily ensure monotonicity with the source. this indicates that such data may not be ideal for training simt models aimed at achieving both minimal latency and high quality. ## experimental setup to evaluate the effectiveness of the llm si-corpus, we conducted experiments in speech-to-text settings. we also conducted text-to-text experiments, as presented in appendix c, which showed a similar trend to the speech-to-text results. we implemented the baseline using fairseq @xcite and simuleval @xcite . speech-to-text settings following the settings of @xcite ; @xcite , we employ pretrained language models for both encoder and decoder using fairseq @xcite , and integrating into the transformer architecture @xcite . we used hubert-large @xcite as the encoder, and we used mbart50 @xcite as the decoder. we trained the model with must-c v2.0 @xcite as continuous pertaining, and then fine-tuned the models for 3k steps, evaluating their performance every 200 steps, and terminated the fine-tuning if there was no improvement in the loss score for eight consecutive evaluations. for decoding policy, we applied test-time wait-k @xcite foot_3 to determine whether the tradeoff between latency and quality is solely a result of differences in the dataset. the value of wait-k ranges from 1 to 17 at two intervals. one unit for k was set to 160 frames and when @xmath0, after reading 3 × 160 frames, the model would write and read alternately. the detailed settings are described in appendix c. training datasets we used must-c v2.0 for en-ja @xcite for pre-training and it is as the baseline @xcite . we then fine-tuned the pre-trained model using different types of data: offline st translation data (offline), naist-sic-aligned-st (sic), which consists of human interpretation transcriptions, and two versions of the llm-si-corpus (gpt-4 and gpt-3.5). all finetuning datasets come from the same audio sources, allowing for a comparison of the impact of different translation styles from each dataset. ## experimental results on speech-to-text evaluation 1: tst-common figure 3 shows the results of speech-to-text experiments. when we focused on bleu-al in figure 3 for @xmath1, @xmath2, and @xmath3, the llm-si-corpus (gpt-3.5 and gpt-4) achieved higher bleu scores than offline, indicating improvements in both latency and quality. however, as the value of k increases, the bleu score in pretrain starts to surpass that of llm-si-corpus and offline when exceeds around @xmath4. this pattern persists across laal and atd as well. this is attributed to the alignment of training and evaluation data, leading to enhanced bleu scores. next, in {bleurt, comet}-{al, laal}, both quality and latency in llm-si-corpus (gpt-3.5 and gpt-4) surpasses offline and pretrain. also, in comet-qe, the llm-si-corpus demonstrates superior quality and latency performance at all latencies in al, laal, and atd, indicating that the model trained on the llm-si-corpus can perform high-quality translations with low latency. despite the trends observed in text-to-text settings, the quality gap remains evident in speech-to-text settings even as k increases. evaluation 2: sic-test figure 4 shows the result of sic-test. focus on bleu-al, the result indicates that the llm-si-corpus exhibits higher quality than offline up to around @xmath5. however, offline and sic perform better as k increases because these align with the training and evaluation data, thereby improving the bleu score. the same trends are observed in laal and atd. next, in {bleurt, comet}-{al, laal, atd}, both quality and latency in llm-si-corpus (gpt-3.5 and gpt-4) surpasses offline and pretrain. the same as in comet-qe, the llm-si-corpus outperforms offline and pretrain at all latencies in al, laal, and atd, indicating that the model trained on the llm-si-corpus can perform highquality translations with low latency. evaluation 3: chunk-wise figure 5 shows that the llm-si-corpus consistently exhibits superior quality and latency performance across all quality evaluation metrics. the quality gap among models is noticeable, particularly when wait-k is small, and remains significant even as wait-k values increase. gpt-4 achieves a better balance between quality and latency than gpt-3.5, likely due to its higher model capabilities. offline achieved comparable results on both tst-common and sic-test, however, in this test set, the results were weaker, indicating that offline has difficulty achieving more monotonic translation. ## discussions we picked several important discussion themes, with further discussions provided in appendix e. ## conclusion and future directions in this study, we proposed a method for converting st corpora to si corpora using llms to improve the monotonicity yet maintain the quality. this corpus creation method follows the cwmt guidelines, focusing on the english-to-japanese direction. to evaluate the effectiveness of our llm-si-corpus, we conducted experiments in three scenarios: a general offline st corpus (tstcommon), an si corpus (sic-test), and a cwmt test corpus (chunk-wise), in both speech-to-text and textto-text settings. in all cases, the simt models fine-tuning with the llm-si-corpus outperformed others, achieving lower latency and higher quality. moreover, while manually constructing si corpora is costly, the llm-si-corpus can be produced for only 20 dollars. therefore, it can be easily applied to other st corpora or adapted to other languages since it utilizes llms. for future work, we plan to explore the application of other si techniques, such as summarization, extend these methods to larger-scale st corpora, and expand their use to speech-to-speech settings.