Simul- M u ST - C : Simultaneous Multilingual Speech Translation Corpus Using Large Language Model (2024)

Abstract: Simultaneous Speech Translation (SiST) begins translating before the entire source input is received, making it crucial to balance quality and latency. In real interpreting situations, interpreters manage this simultaneity by breaking sentences into smaller segments and translating them while maintaining the source order as much as possible. SiST could benefit from this approach to balance quality and latency. However, current corpora used for simultaneous tasks often involve significant word reordering in translation, which is not ideal given that interpreters faithfully follow source syntax as much as possible. Inspired by conference interpreting by humans utilizing the salami technique, we introduce the Simul-MuST-C, a dataset created by leveraging the Large Language Model (LLM), specifically GPT-4o, which aligns the target text as closely as possible to the source text by using minimal chunks that contain enough information to be interpreted. Experiments on three language pairs show that the effectiveness of segmented-base monotonicity in training data varies with the grammatical distance between the source and the target, with grammatically distant language pairs benefiting the most in achieving quality while minimizing latency.

## introduction simultaneous speech translation (sist) begins translating before the source inputs are fully received @xcite @xcite @xcite . as waiting time increases, translation quality improves with more available inputs, some individual services even bring it down by 90 percent. 中には 90%も 削減できる サービス もあります。 いくつかの 個別のサービスは それをさらに 90パーセントまで 引き下げる。 some individual services even bring it down by 90 percent. must-c simul-must-c figure 1 : an example of an english-japanese parallel sentence. in translations from must-c, the word order changes frequently, resulting in a reversed order compared to the source, as indicated by the arrows. on the other hand, translations from simul-must-c, where the salami technique is applied to maintain monotonicity, preserve the source's word order as much as possible, as shown by the arrows. but latency impacts negatively. starting translation immediately reduces latency but limits available inputs and damages quality. to address this trade-off between quality and latency, one might consider using a method by simultaneous interpreters, as they also process inputs in real-time. this technique, i.e., "salami technique" @xcite @xcite , divides a sentence into units that are as short as possible while ensuring each unit contains enough information to be interpreted clearly. interpreters translate each segment into the target language, keeping that the output mirrors the source input syntax, which helps to speed up the translation process. this syntax manipulation on the target side is effective because the syntax is more flexible than word order across different languages @xcite . sist could benefit from this technique by using simultaneous interpretation corpora made by professional interpreters, allowing a model to learn the segmented-base monotonicity through training with such real simultaneous interpretation data @xcite . despite the availability of several simultaneous interpretation corpora @xcite @xcite , their sizes remain limited for effective model training. collecting new data is challenging and costly because it requires simultaneous human interpreters. moreover, interpreters employ tactics, e.g., summarization, and they make mistakes due to the intense time pressure and high cognitive load during interpretation @xcite . relying on real simultaneous interpretation data is challenging due to frequent summarizations and omissions, which are unsuitable for model training. however, the data's monotonicity is necessary to balance latency and quality. therefore, we introduce a segment-base monotonic dataset of simul-must-c (simultaneous multilingual speech translation corpus) by rewriting existing multilingual speech translation corpora, @xcite in figure 1 . based on @xcite , we utilize the salami technique, used in conference interpreting, when prompting large language models (llms) with gpt-4o. this technique involves dividing original sentences into shorter segments that contain enough information to be interpreted, reducing the word order changes in the target language. we investigate the effectiveness of salami technique in a computational approach for simultaneous tasks for multiple language pairs. training models with simul-must-c in speech-to-text settings improves latency minimization and translation quality for language pairs that are grammatically distant, whereas the improvement is less evident for pairs that are grammatically similar. our contributions are as follows: ## word order monotonicity analysis we compared word alignments between source and target sentences in both must-c and simul-must-c translations to investigate word order differences. we used awesome-align @xcite for this comparison and evaluated word order monotonicity using spearman's correlation coefficient. as shown in en-ja source (1) now, / (2) we have some pilot things / (3) that do this / (4) at the 60 to 80 percent level. must-c (4) 60%から80%のレベルで (at the 60 to 80 percent level) / ( 3 ) この処理を行う (do this) / (2) 試験運用を (pilot things) / (3) 行っています (do)。 simul-must-c (1) 今 (now) 、/ (2) いくつかの試験的なものがあり (we have some pilot things)、/ (3) これ を (this) / (4) 60から80パーセントのレベルで (at the 60 to 80 percent level) / (3) 行 います (do)。 en-zh source (1) i / (2) grew up / (3) on a steady diet of / (4) science fiction. must-c (1) 我是 (i) / (4) 在科幻小说 (science fiction) / (3) 的陪伴下 (accompanied by) / (2) 长大 的(grew up)。 simul-must-c (1) 我 (i) / (2) 长 大 在 (grew up) / (3) 稳 定 的 饮 食 (a steady diet) / (4) 科幻小说(science fiction)。 en-de source (1) these are / (2) what people / (3) often / (4) refer to as / (5) the renewable sources. must-c (1) es sind (there are) / (5) die erneuerbaren energien (renewable energies), / (3) wie sie oft (as they often) / (4) genannt warden (be called). simul-must-c (1) dies sind (these are), / (2) was die leute (what people) / (3) oft als die (often) / (5) erneuerbaren quellen (renewable energies) / ( 4 ) bezeichnen (describe). en-zh similarly, ## experimental setup to evaluate the contribution of simul-must-c to improving the quality-latency trade-off, we compare two models: one trained with must-c and the other with simul-must-c. for clarity in our analysis, we present the results of the wait-k @xcite policy. we also evaluated based on the local agreement @xcite . we describe its differences from wait-k and provide corresponding analyses in appendix e. dataset for the training dataset, we used must-c v2.0 @xcite for three language pairs: en-{ja, zh,-de} as the baseline, and simul-must-c, which is built upon on must-c v2.0, applying the salami technique. for evaluation, we used the tst-common from must-c v2.0. ## experimental results on wait-k policy en-ja figure 3 shows the results for en-ja. with a focus on comet-qe_atd, the latency gap in atd between must-c and simul-must-c widens as k increases, indicating that simul-must-c not only starts but also finishes translations faster compared to must-c. despite finishing translations faster, simul-must-c's translation quality, as shown by comet-qe, is better than must-c. in sist scenarios, where delays in translation can negatively impact subsequent inputs, simul-must-c enables faster completion of translations while maintaining the quality observed in the results. when evaluated offline using comet-qe, both models achieve similar quality. this suggests that comet-qe assesses performance directly from the source and target without requiring references, making it unaffected by offline translation style in the reference. however, when using reference-based metrics, a significant quality gap exists. specifically, with bleu, the quality difference between must-c and simul-must-c is around 5 points, suggesting that bleu may be strongly influenced by translation style in reference. this discrepancy between reference-free and reference-based metrics highlights the need for references better suited to simultaneous translation settings. en-zh figure 4 shows that simul-must-c outperforms must-c for en-zh. when focusing on bleu and bleurt, comet, comet-qe, the quality gap in bleu is larger than in bleurt, comet, comet-qe. since both the training and evaluation data originate from must-c, must-c might be expected to align more closely with the test, potentially enhancing bleu. however, the results show that simul-must-c achieves a closer surface-level match to the test than must-c across all k. with a focus on bleu, comet-qe_atd, translation by simul-must-c starts and ends faster while maintaining quality. this is the same trend we observed in en-ja, which is ideal for sist. when focusing on offline, the results are relatively similar, except that must-c performs better in bleu. however, simul-must-c outperforms must-c in all wait-k settings, indicating that simul-must-c is better suited for simultaneous translation, while must-c is better for offline translation. additionally, en-zh may also be affected by offline translation style in the reference, similar to the en-ja. this is because there is almost no quality gap in reference-free metrics, whereas a slight gap appears in bleu. however, compared to the en-ja, the quality gap between the two types of metrics is smaller, probably due to the lesser difference in word order. en-de figure 5 shows the results for en-de. focusing on bleu and comet-qr, simul-must-c shows a slight advantage, especially as k increases. this trend is consistent with our findings in en-zh. while surface-based evaluation metrics and semantic similarity evaluation metrics could show different tendencies sometimes, they correlates in this case. these results suggest that simul-must-c slightly but consistently outperforms must-c in quality. with a focus on atd, both must-c and simul-must-c achieve nearly the same latency level, indicating similar handling of the start and end timing of translation. this suggests that simul-must-c does not provide an improvement, as its results are comparable to must-c. in terms of offline quality, performance is relatively comparable, with must-c outperforming simul-must-c in bleu, while simul-must-c shows a slight advantage in comet-qe. however, in the simultaneous setting, simul-must-c consistently performs better. this pattern is also evident in en-ja and en-zh, though the quality gap in en-de is the smallest of the three language pairs, likely due to differences in word order. the word order gap is smallest in the en-de pair, which may explain why simul-must-c is effective, although its impact is limited, as reflected by the slight word order improvement shown in ## conclusion we proposed simul-must-c, a dataset, and a method to rearrange sentences into segmentationbased monotonic data for simultaneous speech translation using llms in en-{ja, zh, de}. this method, based on the salami technique used in conference interpreting, showed that simul-must-c improves quality and latency, especially in grammatically distant language pairs, indicating a correlation between word order monotonicity and quality-latency improvement. using llms is costeffective and helps address the scarcity of such datasets, which require extensive human labor. future work will expand this dataset to end-to-end speech-to-speech translation.