HW - TSC ’s Submission for the WMT 22 Efficiency Task (2022)

Abstract: This paper presents the submission of Huawei Translation Services Center (HW-TSC) to WMT 2022 Efficiency Shared Task. For this year’s task, we still apply sentence-level distillation strategy to train small models with different configurations. Then, we integrate the average attention mechanism into the lightweight RNN model to pursue more efficient decoding. We tried adding a retrain step to our 8-bit and 4-bit models to achieve a balance between model size and quality. We still use Huawei Noah’s Bolt for INT8 inference and 4-bit storage. Coupled with Bolt’s support for batch inference and multi-core parallel computing, we finally submit models with different configurations to the CPU latency and throughput tracks to explore the Pareto frontiers.

## introduction transformer and its variants @xcite @xcite have become benchmark models for machine translation. a lot of innovations and engineering optimizations @xcite in this area are based on transformer. however, with the increase of bilingual and monolingual data sizes used for training, the size of the model expands and the requirement of computing ability become higher. taking t5 @xcite , gpt3 @xcite and a series of subsequent large models @xcite as examples, although they have achieved very good performances, it is still difficult for ordinary practitioners to reproduce or use these models for research and industry application. especially in scenarios where hardware capability is limited, models that balance size, quality and power consumption is urgently needed. the wmt efficiency task is performed under such constraints. we employ knowledge distillation @xcite to train small models. the teacher models and distillation data come from official website. we only perform simple data cleaning, and all of our experiments are conducted based on fariseq @xcite . deep encoder and shallow deocder models can balance quality and inference speed @xcite . we follow this configuration for pursuing extreme efficient decoding. inspired by sru++ @xcite and aan @xcite , we integrate the average attention mechanism with a lightweight rnn for more efficient decoding. we retrain our 8-bit quantisation model @xcite , then compare its result with that of direct post-quantization @xcite model. we finally find that in the distillation scenario, the difference between the two is not obvious. we apply 4-bit storage to obtain an extremely small model size. although our training and inference strategies ensure basically the same model quality, the gap in overall quality is large and the model needs to be further optimized. we still use huawei noah's bolt as the inference library. this year, we implement batch inference and parallel computing on multi-core cpus for the throughput track. finally, after performing some necessary engineering optimizations, we submit four models with different configurations to explore the pareto frontiers. ## inference optimizations we use bolt acceleration library as cpu optimization backend to build the high-performance translation engine. bolt has a standalone c++ runtime, therefore it can perform fast inference without any third-party dependencies. we use bolt v1.4.0, which will be available in october 2022.