Word Order in E nglish- J apanese Simultaneous Interpretation: Analyses and Evaluation using Chunk-wise Monotonic Translation (2024)

Abstract: This paper analyzes the features of monotonic translations, which follow the word order of the source language, in simultaneous interpreting (SI). Word order differences are one of the biggest challenges in SI, especially for language pairs with significant structural differences like English and Japanese. We analyzed the characteristics of chunk-wise monotonic translation (CMT) sentences using the NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset and identified some grammatical structures that make monotonic translation difficult in English-Japanese SI. We further investigated the features of CMT sentences by evaluating the output from the existing speech translation (ST) and simultaneous speech translation (simulST) models on the NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset as well as on existing test sets. The results indicate the possibility that the existing SI-based test set underestimates the model performance. The results also suggest that using CMT sentences as references gives higher scores to simulST models than ST models, and that using an offline-based test set to evaluate the simulST models underestimates the model performance.

## introduction simultaneous interpreting (si) is the task of translating speech from a source language into a target language in real time. si is cognitively demanding, and human simultaneous interpreters employ such strategies as segmentation, summarization, and generalization @xcite . maintaining word order in a source language is another important strategy, especially for language pairs whose word order differs (e.g., english and japanese), to shorten delays and reduce cognitive load. because of these features, si sentences are different from offline translation sentences, although most automatic si studies @xcite @xcite have used offline translation corpora (e.g., @xcite for both training and evaluatng models due to the limited amount of simultaneous interpretation corpora (sics). for english-japanese language pairs, several sics have been constructed @xcite @xcite . based on the naist simultaneous interpretation corpus (naist-sic; @xcite , @xcite foot_0 created an automaticallyaligned parallel si dataset: naist-sic-aligned. since its sentences are aligned at the sentence level, they can be used for model training. actually, @xcite and @xcite trained si models using si data from naist-sic-aligned. their model performances were evaluated through automatic evaluation metrics such as bleu @xcite using a small test set curated based on si sentences generated by professional human simultaneous interpreters. although the scores reported in @xcite and @xcite were relatively low, the test set used in both studies might have underestimated the model performance. since human simultaneous interpreters use such strategies as summarization and generalization, phrases that do not affect the main idea are not necessarily translated into the target language. if an si model generates translations for phrases that a human interpreter did not, the output sentence might not be evaluated properly, even when it is a correct translation. @xcite pointed out the difficulty for si models to learn which phrases in source speech are less important and advocated constructing si models that only employ a strategy that maintains the word order in a source language. as a first step, they created the naist english-to- @xcite have investigated the quality of the cmt sentences in their dataset through human evaluation, although they have not analyzed its characteristics. nor have they conducted any evaluation experiments in which model outputs are evaluated on their dataset. in this paper, we qualitatively and quantitatively analyze cmt sentences in the naist english-to-japanese chunk-wise monotonic translation evaluation dataset. in the process of generating cmt sentences for the dataset, it was allowed to repeat, defer, and omit phrases in the source sentences to maintain the translation's fluency. we assume the presence of factors (e.g., syntactic structures) that prevent monotonic translation if phrases were repeated, deferred, or omitted in the cmt sentences since they were translated without time constraints. in addition, we evaluate the output from 2 https://dsc-nlp.naist.jp/data/naist-sic/ aligned-chunk_mono-ej/ 3 cmt refers to the task of segmenting a source sentence into chunks and translating it in the order of the chunks. a cmt sentence is a target sentence generated through cmt. 4 precisely, omissions that maintained the fluency of the sentence were allowed. see section 3.1 for the details about the dataset. an existing speech translation (st) model and two simultaneous speech translation (simulst) models (see 5.2). both the st foot_2 and simulst models are used to investigate the differences in scores when evaluating translations with different characteristics. the contributions of this paper are as follows: ## chunk-wise monotonic translation in si between language pairs with different sentence structures, interpreters segment source sentences into chunks and translate them from chunk to chunk foot_3 @xcite . this section describes the details of the naist english-to-japanese chunk-wise monotonic translation evaluation dataset, which is used in our analyses. ## evaluation using cmt sentences to investigate the impact of using cmt sentences for evaluating translation quality, we evaluated the output from existing st and simulst models using the naist english-to-japanese chunk-wise monotonic translation evaluation dataset as well as existing test sets. ## conclusion this paper focused on monotonic translations in english-japanese si. our analyses revealed some grammatical structures that make monotonic translations difficult and that human interpreters/translators address these challenges by repeating or deferring some phrases in source language in the subsequent chunks. the grammatical structures that might cause delays would be useful information for developing segmentation or decoding policies for simultaneous machine translation systems. one possible direction would be predicting whether a phrase in a chunk is the head of a phrase in subsequent chunks. we also evaluated the output from the existing st and simulst models on the naist englishto-japanese chunk-wise monotonic translation evaluation dataset as well as on existing si-based and offline-based test sets. the bleu, bleurt, and bertscore results supported using cmt sentences for evaluating simulst models trained using si data, although the results with comet were different. further analysis across various evaluation metrics is necessary. analyzing how the source and target sentences are aligned monotonically on different types of translations (e.g., @xcite would also be useful. this paper investigated the impact of using cmt sentences for evaluation purposes. a future study would involve using monotonic translation sentences for developing simulst models @xcite 16 . it could potentially address the problem that simulst models trained using si sentences suffered from under-translation @xcite . however, cmt sentences tend to be long. investigating the trade-offs between longer cmt sentences and the potential cognitive load on listeners/readers might provide further insights.