Findings of the WMT 2021 Shared Task on Efficient Translation (2021)

Abstract: The machine translation efficiency task challenges participants to make their systems faster and smaller with minimal impact on translation quality. How much quality to sacrifice for efficiency depends upon the application, so participants were encouraged to make multiple submissions covering the space of trade-offs. In total, there were 53 submissions by 4 teams. There were GPU, single-core CPU, and multi-core CPU hardware tracks as well as batched throughput or single-sentence latency conditions. Submissions showed hundreds of millions of words can be translated for a dollar, average latency is 5–17 ms, and models fit in 7.5–150 MB.

## introduction the efficiency task complements the collocated news task by challenging participants to make their machine translation systems computationally efficient. this is the fourth edition of the task, expanding upon previous editions @xcite @xcite . participants built @xmath0. for translation quality measurement, we use the same news-focused wmt21 test set and human evaluation protocol as the news task. however, human assessment was conducted separately from the evaluation of the news task submissions. submissions are made as docker containers so we can consistently measure their performance in terms of quality, speed, memory usage, and disk space. we run the containers in three different hardware environments: one gpu, one cpu core, and multiple cpu cores. systems were tested for throughput by providing 1 million sentences upfront to allow batching and parallelization. we also tested for latency with a program that drip-feeds one input sentence, waits for the translation, and then provides the next input sentence. there were five conditions in total: gpu batch (for throughput), gpu latency, 1 cpu core batch, 1 cpu core latency, and 36 cpu cores batch. we did not measure latency in a multi-core cpu setting because the test hardware has 36 cores and overhead for 36 threads is larger than the cost of arithmetic for the small tensors in optimized models. participants were free to choose which conditions to participate in. the condition was passed to the docker container as command line arguments. machine translation is used in a range of settings where users might choose different trade-offs between quality and efficiency. for example, a highfrequency trading system might prefer the lowest latency at the expense of quality given that the output will only be read by a machine. conversely, in a post-editing scenario the personnel costs outweigh many computational costs. therefore there is not a single best system, but a range of options that trade between quality and efficiency. we emphasize the pareto frontier: the fastest systems at each level of quality, or the smallest systems at each level of quality. to explore the pareto frontier, participants were encouraged to make multiple submissions covering the range of trade-offs. in total, 53 combinations of models, hardware, and batching were benchmarked. ## hardware we chose modern hardware to encourage exploiting new hardware features. the gpu is an nvidia a100 from the oracle cloud bm.gpu4.8 instance. the instance has eight gpus and we limited docker to using only one gpu. the gpu machine has an amd epyc 7542 cpu with all cores allowed. in practice, most submissions used only one core while niutrans's submissions used the cpu cores to parallelize preprocessing and postprocessing. the cpu-only condition used a dual-socket intel xeon gold 6354 from oracle cloud bm.optimized3.36 with a total of 36 cores. for the single-core cpu track, we reserved the entire machine then ran docker with -cpuset-@xmath1. in the 36-core cpu track, participants were free to configure their own cpu sets and affinities. the oracle cloud machines are bare metal servers, meaning there was no shared tenancy, no virtualization, and the test machines were otherwise quiescent. ## input text to amoritize loading time, avoid starving highly parallel submissions, and reduce the ability to cheat, we benchmark systems on 1 million sentences of input. the test set is hidden inside these 1 million sentences, shuffled with filler sentences. many filler sentences are drawn from parallel corpora to check that systems are in fact translating all sentences, though we do not consider scores on noisy corpora reliable enough to report. the composition of this set changes each year and is decided after the submission deadline. filler data was gathered from parallel corpora and gender bias challenge sets: wmt news test sets from 2008 through 2021 @xcite , the additional test inputs in wmt 2021, khresmoi summary test v2 @xcite , iwslt 2019 @xcite , simplegen @xcite , winomt @xcite , ted 2020 @xcite , and tilde rapid 2019 @xcite . we capped sentence lengths at 150 space-separated tokens, except for the wmt 2021 test set to preseve the ability to evaluate with it. because wmt 2020 includes excessively long segments that are actually concatenated sentences, we also added sentence split versions of wmt 2020 and wmt the input file has 1,000,000 lines, 19,951,184 space-separated words, and 124,257,215 bytes (most of which are characters since the file is english in utf-8). this is an average of 20 words per sentence compared to 15 words per sentence the previous year @xcite due to raising the cap from 100 to 150 tokens per sentence and the lengthy text in the rapid corpus. teams were responsible for their own tokenization and detokenization. we provided raw utf-8 english input text with one sentence per line. ## results all submissions are shown in the system ranking based on the standard da is presented in ## conclusion and future tasks using the highest quality system in this evaluation, translating 124,257,215 characters took 140 seconds on an a100 gpu that costs $3.05/hr in a cloud. the one-month gap between the news task dead-line and the efficient task deadline was too short and some teams noted this reduced the conditions they participated in. in addition, scaffolding would reduce the barrier to participation. this could take the form of providing a trained high-quality model, providing distilled @xcite training data, or even optimized models where only the toolkit code is optimized. providing this scaffolding would effectively require the organizers to perform the full task before releasing it to participants. if the training and test data are renewed each year as a countermeasure to overfitting and a participant that cheated, this would require more time between the news task and release of the news test set references. german is a high resource language, which raises the computational cost of participation. a medium resource language would generally reduce training costs and explore whether results apply in this data condition. the next task should aim to recruit more participants and perhaps separate the organization from one of the participants.