SLTEV : Comprehensive Evaluation of Spoken Language Translation (2021)

Abstract: Automatic evaluation of Machine Translation (MT) quality has been investigated over several decades. Spoken Language Translation (SLT), esp. when simultaneous, needs to consider additional criteria and does not have a standard evaluation procedure and a widely used toolkit. To fill the gap, we develop SLTev, an open-source tool for assessing SLT in a comprehensive way. SLTev reports the quality, latency, and stability of an SLT candidate output based on the time-stamped transcript and reference translation into a target language. For quality, we rely on sacreBLEU which provides MT evaluation measures such as chrF or BLEU. For latency, we propose two new scoring techniques. For stability, we extend the previously defined measures with a normalized Flicker in our work. We also propose a new averaging of older measures. A preliminary version of SLTev was used in the IWSLT 2020 shared task. Moreover, a growing collection of test datasets directly accessible by SLTev are provided for system evaluation comparable across papers.

## introduction spoken language translation (slt), i.e. translation of human speech across languages, is an application at least as important as machine translation (mt). many approaches have been examined so far, ranging from translation of transcript chunks @xcite to fully end-to-end, speech-to-speech neural systems, @xcite . in recent years, simultaneous translation systems aim at behavior similar to human interpreters, digesting and producing an infinite sequence of words. some systems @xcite @xcite @xcite @xcite do not consider any revision of their outputs and can be evaluated in two main criteria: quality and latency, allowing users to trade a bigger delay (including waiting for more input text) for a more accurate translation. simultaneous translation systems aimed at automatic subtitling @xcite @xcite ) may revise their outputs, demanding a new evaluation measure: the stability, i.e. the amount of revision. trading these qualities for one another is again possible: it is obvious that if a system creates the translations with a longer delay or revises them more (higher flicker), the quality of the final translation (i.e., the output text) can be better. given the existence of three evaluation criteria and a multitude of possible definitions for each of them, the need for some robust and standard metrics to evaluate slt is inevitable. recently, the mt community tackled a similar problem (i.e., the inconsistency in the reporting of bleu scores) by introducing a tool named sacre-bleu @xcite with a canonical implementation of the widely user metric. in this work, we propose sltev, foot_0 an open-source tool to calculate the quality of slt systems based on three different criteria: translation quality, latency, and stability, in a standardized way. furthermore, we complement sltev with a growing collection of freelyavailable test sets for automatic speech recognition (asr), mt and slt for a number of languages, so that these technologies can be evaluated in comparable settings, similarly to what the wmt news test sets @xcite offer in mt. the delay measures of @xcite were criticised by @xcite when they introduced their wait-k model. they defined a measure average lag (al), which measures how far, in words, the translation is behind an ideal wait-k model. since wait-k does not allow corrections, they do not need a stability measure. al was improved by @xcite , with differentiable average lag (dal), which not only is differen-tiable, but fixes some undesirable behaviour of al around sentence boundaries. however dal, like al, is defined in terms of word count and on segmented text (although it could be extended to fix these shortcomings). @xcite extended the retranslation model of @xcite , and so needed a measure of stability. for evaluation, they check the output of asr and the output of the mt system as recorded over time in their simple logging system. they assess the quality, latency, and stability (i.e., flicker). the quality is estimated using bleu after mwersegmenter re-segmentation. for the assessment of latency (translation lag) and stability (the number of erased tokens in temporary translations per final target token, "normalized erasure" in the paper), they do not use any segmentation at all and instead calculate the scores for ten-minute long audio chunks. the closest to our work is simuleval @xcite , a client-server toolkit measuring the latency of slt including any network effects between the evaluated system (client) and the mock user (simuleval server). simuleval offers a nice visualization interface but the required clientserver approach may be unsuitable for research prototypes solving slt only partially. most importantly, updates of output (flicker) are not supported and no test set for reproducible scoring is provided. ## input formats sltev can evaluate separate asr and mt systems as well as cascaded and end-to-end slt systems. we focus on slt here. three input files are used for slt evaluation: a time-stamped golden transcript in the source language (section 3.1), a reference translation (or translations in multi-reference format; section 3.2) and candidate output (section 3.3) in the target language. the intermediate asr output can be provided as the fourth input file to calculate the accuracy of the asr system if it was part of the cascade (section 3.3). ## proposed metrics in this section, all evaluation metrics and strategies introduced in our evaluation framework are described. our evaluations are based on three criteria: latency, stability, and the quality of mt outputs. ## a growing test set to allow for continued and comparable evaluation of slt by the research community, we create and keep extending a publicly available dataset which contains source audio, time-stamped golden transcripts and reference translations for different types of inputs called elitr-testset. 8 the dataset currently focuses on european languages, as needed by the elitr project @xcite , but it is designed to be easily extensible in both languages and domains. with the help of commit ids, full reproducibility of evaluations is ensured, even as the dataset will be growing. in simple words, the elitr-testset is an assorted collection of documents, with inputs and expected outputs for asr, mt and/or slt systems. we expect our users to pick a relevant subset of these documents depending on their application needs and evaluate on this subset. for comparability, we standardize some of these selections by introducing the concept of "indices". each index is simply a file list of documents and it is also versioned in the elitr-testset. for example, we provide indices of documents which are good for purposes like: (1) slt of english into czech/german in the auditing domain, (2) english asr in the computational linguistics domain, and (3) czech/german asr, regardless of the domain. another feature of elitr-testset is a collection of automatic checks that verify formal integrity of the documents (e.g. character encoding, line ends, number of lines) before every commit. all datasets included in elitr-testset are free for public use but some indices include confidential files. sltev can be used as a stand-alone tool to evaluate asr, mt or slt using source, candidate and reference files you provide, or it can be used very conveniently with elitr-testset. running sltev -g index-name will provide you with input files that your system should process and a second run of sltev will report your system's scores for the given index. ## conclusion in this paper, we introduced sltev, a framework for comprehensive and fine-grained evaluation of the output of simultaneous slt systems, i.e., systems for live speech translation, and their components (asr, mt). in contrast to text translation systems, simultaneous slt systems cannot be judged just based on translation quality. for example, if the system waits for the whole sentence to be analyzed and processed, the translation quality will likely be better but the high latency may not be acceptable to the end user. sltev evaluates quality, latency, and stability (the number of corrections the system makes). in order to tackle the problem of the output segmentation, we proposed a new time-based segmentation method, in addition to the classical re-segmentation strategy of mwersegmenter. we complement the release of sltev with elitr-testset, a publicly available dataset of source speech and reference translations, so that truly comparable evaluations are available for the research community. sltev directly accesses this growing collection for easy and comparable scoring of your systems in various domains. we used a preliminary version of sltev to evaluate systems in one of the iwslt 2020 shared tasks.