Findings of the WMT 2022 Shared Task on Efficient Translation (2022)

Abstract: The machine translation efficiency task challenges participants to make their systems faster and smaller with minimal impact on translation quality. How much quality to sacrifice for efficiency depends upon the application, so participants were encouraged to make multiple submissions covering the space of trade-offs. In total, there were 76 submissions from 5 teams. The task covers GPU, single-core CPU, and multi-core CPU hardware tracks as well as batched throughput or single-sentence latency conditions. Submissions showed hundreds of millions of words can be translated for a dollar, average latency is 3.5–25 ms, and models fit in 7.5–900 MB.

## introduction the efficiency task complements the collocated news task by challenging participants to make their machine translation systems computationally efficient. this is the fifth edition of the task, expanding upon previous editions @xcite @xcite . participants built @xmath3. the data condition follows the constrained 2021 workshop on machine translation news translation task. this year, to reduce the barrier to entry, organisers provided an ensemble of teacher systems, as well as cleaned data and distilled output from the teacher ensemble. participants were required to use the provided teacher systems, but were free to distil additional data from the constrained condition. the sentencepiece vocabulary used by the teachers was also made available. for translation quality measurement, we use the news-focused wmt22 dataset, and the systems are ranked according to the comet @xcite automatic metric. we also evaluate systems on bleu and chrf for additional reference. submissions are made as docker containers so we can consistently measure their performance in terms of quality, speed, memory usage, and disk space. we run the containers in three different hardware environments: one gpu, one cpu core, and multiple cpu cores. systems were tested for throughput by providing 1 million sentences upfront to allow batching and parallelization. we also tested for latency with a program that drip-feeds one input sentence, waits for the translation, and then provides the next input sentence. there were four conditions in total: gpu throughput, gpu latency, 1 cpu core latency, and 36 cpu cores throughput. we did not measure latency in a multicore cpu setting because the test hardware has 36 cores and overhead for 36 threads is larger than the cost of arithmetic for the small tensors in optimized models. we also did not measure throughput on a single cpu core as we found that setting to be a somewhat unrealistic real world scenario. participants were free to choose which conditions to participate in. the condition was passed to the docker container as command line arguments. machine translation is used in a range of settings where users might choose different trade-offs between quality and efficiency. for example, a highfrequency trading system might prefer the lowest latency at the expense of quality given that the out-put will only be read by a machine. conversely, in a post-editing scenario the personnel costs outweigh many computational costs. therefore there is not a single best system, but a range of options that trade between quality and efficiency. we emphasize the pareto frontier: the fastest systems at each level of quality, or the smallest systems at each level of quality. to explore the pareto frontier, participants were encouraged to make multiple submissions covering the range of trade-offs. in total, 76 combinations of models, hardware, and batching were benchmarked. ## hardware we chose modern hardware to encourage exploiting new hardware features. the gpu is an nvidia a100 from the oracle cloud bm.gpu4.8 instance. the instance has eight gpus and we limited docker to using only one gpu. the gpu machine has an amd epyc 7542 cpu with all cores allowed. the cpu-only condition used a dual-socket intel xeon gold 6354 from oracle cloud bm.optimized3.36 with a total of 36 cores. for the single-core cpu track, we reserved the entire machine then ran docker with -cpuset-@xmath4. in the 36-core cpu track, participants were free to configure their own cpu sets and affinities. the oracle cloud machines are bare metal servers, meaning there was no shared tenancy, no virtualization, and the test machines were otherwise quiescent. ## input text to amortize loading time, avoid starving highly parallel submissions, and reduce the ability to cheat, we benchmark systems on 1 million sentences of input. the test set is hidden inside these 1 million sentences, shuffled with filler sentences. many filler sentences are drawn from parallel corpora to check that systems are in fact translating all sentences, though we do not consider scores on noisy corpora reliable enough to report. the composition of this set changes each year and is decided after the submission deadline. the filler data was gathered from parallel corpora and gender bias challenge sets: wmt news test sets from 2008 through 2022 @xcite , the additional test inputs in wmt 2021, khresmoi summary test v2 @xcite , iwslt 2019 @xcite , simplegen @xcite , winomt @xcite , ted 2020 @xcite , and tilde rapid 2019 @xcite . we limit sentence lengths to 150 space-separated tokens. because wmt 2020 includes excessively long segments that are actually concatenated sentences, we also added sentence split versions of wmt 2020 and wmt 2021, though the difference on wmt 2021 was minor. source sentences were concatenated, deduplicated, and shuffled. the tilde rapid corpus was clipped to make a total of 1 million deduplicated lines. counts are shown in input text and tools to extract test sets from system outputs are available at https://data. statmt.org/wmt22/efficiency-task/ wmt22-testdata.tar.xz . the input file is 1,000,000 lines, consisting of 19,926,744 space-separated words, or 124,186,772 bytes of english text in utf-8. this is a mean of 19.9 words per sentence and is comparable to the previous year @xcite . teams were responsible for their own tokenization and detokenization; for this they were permitted to use the sentencepiece vocabulary provided with the teacher system, or to implement an alternative. we provided raw utf-8 english input text with one sentence per line. ## results the results of the task evaluation for the latency scenario are presented in figure 1 shows the trade-off between quality and speed of batched translation submissions separated by hardware environment. each plot shows the pareto frontier as a black staircase to highlight the best combinations of quality and speed. while gpu systems (figure 1a ) achieve higher throughput compared to cpu systems (figure 1b ), this ignores pricing differences between these compute options. in figure 2 , we combine gpu and 36 core cpu speed by using oracle cloud pricing. despite the less expensive per-hour pricing of cpu, gpu is cheaper for throughput-oriented tasks that allow batching. the all-hardware latency pareto frontier is shown in figure 3 . this year all participants submitted systems to the latency task. this year, for the first time, the semi-autoregressive gpu system by royalflush dominates the lower quality settings of the latency pareto frontier, with edinburgh gpu systems having won on some higher quality systems. model sizes at rest on disk appear in figures 4a. participants were allowed to compress their models using their own tools and standard tools like xz. the pareto frontier consists of almost entirely edinburgh submissions, with huaweitsc producing several systems on the lower quality settings, due to their 4-bit compression models. docker image sizes, which include model and software, appear in figure 4b , where the pareto frontier is dominated by edinburgh submissions. conversely, some others opted to optimize other metrics and included large linux installations. we compressed all docker images with xz before measuring. memory (ram) consumption appears in figure 5 . gpu memory consumption reflects batch size and some participants set a large batch size to maximize speed. optimizing speed for multisocket cpu machines implies having a copy of the model in ram close to each socket, so memory consumption is larger beyond simply having temporary space for more batches. finally, participants may have sorted the entire 118 mb input file in ram to form batches of equal length sentences. royalflush is the clear winner on the gpu latency ram consumption, and huaweitsc is the winner of cpu latency ram consumption. ## conclusion using the highest quality system in this evaluation, translating 124,186,772 characters took 283 nvidia a100 gpu latency 104 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0 20 40 60 80 100 120 140 160 180 200 comet million words per dollar edinburgh cpu ecnu cpu cuni cpu huaweitsc cpu gpu gpu gpu figure 2: cost of batched translation for an a100 gpu at @xmath02.7/hr on oracle cloud. for readability, we omit systems with a comet score less than 0.2. seconds on an a100 gpu that costs @xmath10.002/million characters. by comparison, google translate's cost is @xmath2 spent, the gpu submissions are better value for money, provided that enough sentences can be fed to the gpu continuously. the gpu latency track had been intended to attract non-autoregressive machine translation submissions in their ideal condition with a large gpu and no batch to parallelize. for the first time this year, we had a mix of autoregressive, semiautoregressive and non-autoregressive systems: the royalflush system proves extremely well suited to the gpu latency task, dominating the pareto frontier in the lower quality setting, even outperforming cpu systems, which have traditionally won this task. finally, we note that in semi-autoregressive models and non-autoregressive models, a small drop in bleu results in a large drop in comet compared to an autoregressive system, as evidenced by all teams who submitted any form of nonautoregressive mt to the task. this corroborates the findings of @xcite where the large discrepancies between bleu and comet were noted. we urge participants in future editions of the task to examine manually the output of their non-autoregressive systems. german is a high-resource language, which raises the computational cost of participation. we would be interested in also potentially including a medium resource language for distillation so that we can see if the methods that work on high-resource languages generalize well to lowerresource languages, or languages with more morphological complexity. last year @xcite the organisers suggested that an efficient training shared task would be an interesting natural extension to the efficient translation shared task, however it has proven difficult to set up in practice: we are conscious that the validity of such a task can be easily undermined by participants finding a favorable random seed that fits the training data, or more egregiously by including evaluation data in their training data. we are looking for potential solutions to these problems and we are open to suggestions for next year's edition of the task.