Findings of the Second Workshop on Automatic Simultaneous Translation (2021)

Abstract: This paper presents the results of the shared task of the 2nd Workshop on Automatic Simultaneous Translation (AutoSimTrans). The task includes two tracks, one for text-to-text translation and one for speech-to-text, requiring participants to build systems to translate from either the source text or speech into the target text. Different from traditional machine translation, the AutoSimTrans shared task evaluates not only translation quality but also latency. We propose a metric “Monotonic Optimal Sequence” (MOS) considering both quality and latency to rank the submissions. We also discuss some important open issues in simultaneous translation.

## introduction simultaneous translation is to translate concurrently with the speech in the source language, aiming to obtain high translation quality with low latency. the concurrent comprehension and production process makes simultaneous translation an extremely challenging task for both human experts and machines. as a combination of machine translation (mt), automatic speech recognition (asr), and text-to-speech synthesis (tts), simultaneous translation still facing many problems to be studied in the research and application. to promote the development in this cutting-edge field, we conduct a shared task at the 2nd workshop on automatic simultaneous translation. this year, we focus on chinese-english simultaneous translation and set up two tracks: we provide bstc @xcite (baidu speech translation corpus) as the training data, which consists of about 68 hours of mandarin speeches, together with corresponding transcripts, asr results, and translations. in addition, participants can also use bilingual corpus provided by ccmt (china conference on machine translation) 1 . we will describe the data in detail in section 2. one objective of the shared task is to explore the performance of state-of-the-art simultaneous translation systems. traditional evaluation metrics, such as bleu, only measure the translation quality, while recently proposed metrics, such as consecutive wait (cw) @xcite and average lagging (al) @xcite focus on latency. so far as we know, there is no metric that evaluates both quality and delay. we ask the participants to submit systems under different configurations to produce multiple translation results with varying latency. then we plot each result in a quality-latency coordinate. normally, a system is regarded as the best if all of its points are above others (figure 1(a) ). however, in most cases, their lines of points intersect with each other (figure 1(b) ). to consider both quality and latency in ranking, we propose a ranking metric, monotonic optimal sequence (mos) (section 3). the idea is to first corpus train dev test bstc audio (hours) 64.6 1.6 1.5 #talks 215 16 6 #utterances 37,901 956 975 ccmt #sentence pairs 9.1m 2,000 / ## shared task we first introduce the data sets used in the shared task and the setup of the two tracks. ## system evaluation unlike text translation evaluation that only takes one indicator (i.e., translation quality), simultaneous translation evaluation needs to consider quality and latency at the same time. the evaluation based on two criteria brings difficulties to ranking the systems. however, the two indicators are not easy to merge into one. to rank the submissions better, we propose a ranking algorithm called iterative monotonic optimal sequence (i-mos). specifically, we define an optimal point as the result of the best translation quality at each latency. our algorithm iteratively finds sets of optimal points to construct an optimal curve called monotonic optimal sequence (mos), then each team's proportion of points on the mos curve is calculated to measure the performance. the overall process is illustrated in figure 2 . in the following sections, we first introduce the commonly used metrics of quality and latency (section 3.1), then propose the monotonic optimal sequence (section 3.2) and elaborate our i-mos algorithm (section 3.3). ## systems results we received 6 systems submitted by four teams from four universities: algorithm 1: iterative monotonic optimal sequence (i-mos) input: number of teams n input: teams submission: t i contains all results submitted by team i output: teams score s: s i is the score of team i for ranking 1 @xmath0, 0, ..., 0] ⊲ initialize teams level 2 ⊲ tl[i] denotes the level of team i 3 k ← 1 ⊲ start from level 1 4 while n @xmath1, 2, ..., n do 6 if tl[i] = 0 then 7 s[i] ← s[i] + 1 8 calculate mos-k ⊲ the k th level mos for @xmath2, 2, ..., n do 9 if t i has at least one point on mos-k then 10 tl[i] ← k 11 s[i] ← n (p * t i )/n (p t i ) 12 k ← k + 1 we test each docker system with our testset, which contains 1.5 hours of 6 mandarin talks. all the systems are run on v100 gpu. we plot the evaluation results in figure 3 and rank them according to the i-mos algorithm. their ranking results are shown in ## discussion most recent studies on simultaneous translation focused on methods to balance translation quality and latency. besides this, we will discuss some other important challenges for simultaneous translation. ## conclusion this paper presents the results of the @xmath3. the shared task includes two tracks, the text-to-text track (track1) and the speech-to-text track (track2). six systems were submitted to the shared task, four to track1 and two to track2. we propose an evaluation method "monotonic optimal sequence" (mos) to evaluate both translation quality and time latency. we report the results and further discuss some important open issues of simultaneous translation. regrettably, the number of submissions is less than expected, especially for the speech-to-text track. in fact, there are more than 300 teams registered. however, most of them did not submit their results. the possible reason may be that the interdisciplinary task is not easy for participants. we hope to see more participants in the future.