Findings of the Third Workshop on Automatic Simultaneous Translation (2022)

Abstract: This paper reports the results of the shared task we hosted on the Third Workshop of Automatic Simultaneous Translation (AutoSimTrans). The shared task aims to promote the development of text-to-text and speech-to-text simultaneous translation, and includes Chinese-English and English-Spanish tracks. The number of systems submitted this year has increased fourfold compared with last year. Additionally, the top 1 ranked system in the speech-to-text track is the first end-to-end submission we have received in the past three years, which has shown great potential. This paper reports the results and descriptions of the 14 participating teams, compares different evaluation metrics, and revisits the ranking method.

## introduction simultaneous translation (st), which aims to perform translation from source language speech into the target language with high quality and low latency, is widely used in many scenarios, such as international conferences, live broadcasts, etc. generally, the research of st falls into two categories: the cascade method, and the end-to-end method. a typical cascade st system consists of an automatic speech recognition (asr) system that transcribes the source speech into streaming text @xcite @xcite , a machine translation (mt) system that translates the text into the target language, and a policy module lies in between them to decide when to start translation @xcite @xcite @xcite . another branch of work proposed end-to-end st methods that attempt to translate from source speech to target text directly without transcribing the source speech @xcite @xcite . we host a shared task at the third autosimtrans workshop to promote the exploration of advanced st approaches. the shared task is built on the past two editions @xcite . we set up three tracks this year: the objective of st systems is to achieve high translation quality with low latency. during the evaluation period, each participant can submit once a day. to examine their quality-latency trade-off ability, the submission of each track is required to contain multiple folders with different policies and varying latency. our platform supports automatic evaluation and plots the result of each folder to one point on a latency-quality diagram. we've received 24 submissions from 14 teams this year, 4 times as many as last year. the 14 participants are listed in we will introduce the details of the three tracks (section 2), report and analyze the submissions (section 3), and finally compare and analyze evaluation and ranking metrics (section 4). ## shared task we first introduce the corpora used in the shared task, then describe the system evaluation method, as well as the differences compared with the past editions. ## discussion we first carry out experiments to compare different translation quality evaluation metrics (section 4.1), then discuss a controversial ranking dilemma of i-mos algorithm in the ranking algorithm (section 4.2). recently, many quality estimation metrics have been proposed to better imitate human evaluation @xcite , such as ruse @xcite , yisi @xcite , bertscore @xcite , bleurt @xcite , etc. these metrics are proven to be superior to traditional quality evaluation metrics like bleu @xcite ) in text translation. however, to the best of our knowledge, no work has conducted experiments in the st scenario, and almost all st work still takes bleu as the criterion for translation quality evaluation. to keep consistent with previous work, we still used the document-level bleu 7 for evaluation in the shared task this year. now we conduct experiments to compare it with sentence-level bleu, bertscore 8 , and bleurt 9 . ## conclusion and future work this paper presents the results of the simultaneous translation shared task we hosted at the 3 rd workshop on automatic simultaneous translation work-shop. the shared task includes three tracks, two text-to-text tracks in different languages, and one speech-to-text track. we analyze the submissions from 14 participating teams and have the following inspirations for future st work: