SIMULEVAL : An Evaluation Toolkit for Simultaneous Translation (2020)

Abstract: Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario where the model starts translating before reading the complete source input. Evaluating simultaneous translation models is more complex than offline models because the latency is another factor to consider in addition to translation quality. The research community, despite its growing focus on novel modeling approaches to simultaneous translation, currently lacks a universal evaluation procedure. Therefore, we present SimulEval, an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation. A server-client scheme is introduced to create a simultaneous translation scenario, where the server sends source input and receives predictions for evaluation and the client executes customized policies. Given a policy, it automatically performs simultaneous decoding and collectively reports several popular latency metrics. We also adapt latency metrics from text simultaneous translation to the speech task. Additionally, SimulEval is equipped with a visualization interface to provide better understanding of the simultaneous decoding process of a system. SimulEval has already been extensively used for the IWSLT 2020 shared task on simultaneous speech translation. Code will be released upon publication.

## introduction simultaneous translation, the task of generating translations before reading the entire text or speech source input, has become an increasingly popular topic for both text and speech translation @xcite @xcite @xcite @xcite . simultaneous models are typically evaluated from quality and latency perspective. note that the term latency is overloaded and sometimes refers to the actual system speed. in this paper, latency refers to the simultaneous ability, which is how much partial source information is needed to start the translation process. while the translation quality is usually measured by bleu @xcite , a wide variety of latency measurements have been introduced, such as average proportion (ap) @xcite , continues wait length (cw) @xcite , average lagging (al) @xcite , differentiable average lagging (dal) @xcite , and so on. unfortunately, the latency evaluation processes across different works are not consistent: 1) the latency metric definitions are not precise enough with respect to text segmentation; 2) the definitions are also not precise enough with respect to the speech segmentation, for example some models are evaluated on speech segments @xcite ) while others are evaluated on time duration @xcite ; 3) little prior work has released implementations of the decoding process and latency measurement. the lack of clarity and consistency of the latency evaluation process makes it challenging to compare different works and prevents tracking the scientific progress of this field. in order to provide researchers in the community with a standard, open and easy-to-use method to evaluate simultaneous speech and text translation systems, we introduce simuleval, an open source evaluation toolkit which automatically simulates a real-time scenario and evaluates both latency 1 and translation quality. the design of this toolkit follows a server-client scheme, which has the advantage of creating a fully simultaneous translation environment and is suitable for shared tasks such as the iwslt 2020 shared task on simultaneous speech translation foot_1 or the 1st workshop on automatic simultaneous translation at acl 2020 foot_2 . the server provides source input (text or audio) upon request from the client, receives predictions from the client and returns different evaluation metrics when the translation process is complete. the client contains two components, an agent and a state, where the former executes the system's policy and the latter keeps track of information necessary to execute the policy as well as generating a translation. simuleval has built-in support for quality metrics such as bleu @xcite , ter @xcite and meteor @xcite , and latency metrics such as ap, al and dal. it also support customized evaluation functions. while all latency metrics have been defined for text translation, we discuss issues and solutions when adapting them to the task of simultaneous speech translation. additionally, simuleval users can define their own customized metrics. simuleval also provides an interface to visualize the policy of the agent. an interactive visualization interface is implemented to illustrate the simultaneous decoding process. the initial version of simuleval was used to evaluate submissions from the first shared task on simultaneous speech translation at iwslt 2020 @xcite . in the remainder of the paper, we first formally define the task of simultaneous translation. next, latency metrics and their adaptation to the speech task are introduced. after that, we provide a high-level overview of the client-server design of simule-val. finally, usage instructions and a case study are provided before concluding. ## task formalization an evaluation corpus for a translation task contains one or several instances, each of which consists of a source sequence @xmath0, . similar to an offline model, the quality is measured by comparing the hypothesis y to the reference y * after the translation process is complete. on the other hand, the latency measurement involves considering partial hypotheses. the latency metrics are calculated from a function which takes a sequence of delays ## conclusion in this paper, we introduced simuleval, a general and easy-to-use evaluation toolkit for simultaneous speech and text translation. it simulates a real-time scenario with a server-client scheme and automatically evaluates simultaneous translation given a user-defined agent, both for text and speech. furthermore, it provides a visualization interface for the user to track the online decoding process. we introduced example use cases of the toolkit and showed that its general design allows evaluation on different frameworks. we encourage future research on simultaneous speech and text translation to make use of this toolkit in order to obtain an accurate and standard comparison of the latency between different systems.