HW - TSC ’s Participation in the WMT 2021 Efficiency Shared Task (2021)

Abstract: This paper presents the submission of Huawei Translation Services Center (HW-TSC) to WMT 2021 Efficiency Shared Task. We explore the sentence-level teacher-student distillation technique and train several small-size models that find a balance between efficiency and quality. Our models feature deep encoder, shallow decoder and light-weight RNN with SSRU layer. We use Huawei Noah’s Bolt, an efficient and light-weight library for on-device inference. Leveraging INT8 quantization, self-defined General Matrix Multiplication (GEMM) operator, shortlist, greedy search and caching, we submit four small-size and efficient translation models with high translation quality for the one CPU core latency track.

## introduction transformer and its variants @xcite @xcite have become benchmark models in the domain of machine translation. a lot of innovations and engineering optimizations @xcite in this area are based on transformer. in general, to train a high-quality translation model, a large amount of data is required. expensive training and deployment costs pose great challenges to scenarios where hardware are limited or the deployment environment is complex. this task aims to explore a solution that balances efficient decoding and highquality translation. we focus on the one cpu latency track, which can better demonstrate the capability of our model and inference framework. we explore a balance between speed and quality, and ensure efficient memory usage and light-weight inference framework capability at the same time. we finally submit four models of different sizes. we use knowledge distillation @xcite to train small models. the teacher mod-1 https://github.com/huawei-noah/bolt els come from our wmt 2021 news shared task. for the sake of efficient decoding, our models have only 1-layer decoder. however, the number of encoding layers vary. such settings lead to a great increase of inference efficiency while ensuring the translation quality @xcite . all of our experiments are conducted based on fariseq @xcite , including the training of teacher and student models, as well as the generation of distillation data. we use huawei noah's bolt as the inference library. bolt is a universal deep learning library featuring light weight and high speed. for the cpu task, we realize int8 quantization inference and efficient gemm operator, which is faster than intel onednn foot_0 . with other engineering optimization strategies, we achieve a significant improvement in terms of inference efficiency. section 2 describes the teacher-student knowledge distillation process. section 3 introduces how we optimize inference for this task. section 4 presents the final result of our submissions. ## teacher to student knowledge distillation sentence-level distillation @xcite have been demonstrated effective for machine translation tasks. first of all, we train a large teacher model that emphasizes translation quality. then, we translate the source side of the training data and generate a synthetic parallel corpus, as synthetic data is easier for model fitting than real parallel data. finally, we train student models using the synthetic data, hoping to minimize model sizes while ensuring equal translation quality as the teacher model. we use kd refer to knowledge distillation. ## inference optimizations for cpu optimization, we use bolt v1.3.0. bolt is a standalone open-source deep learning acceleration library. v1.3.0 will be available in september 2021. ## optimization results the latency track we participated in is defined as providing one sentence on standard input and flushing then waiting for your system to provide a translation on its standard output (and flush) before providing the next sentence. so we don't use techniques such as batch. we believe such strategy can better demonstrate the capability our model and inference framework. after the preceding optimizations, the inference speed is significantly improved. in our small model setting, the translation quality of the 12-1 model is not significantly improved compared with the 9-1 model, but the inference speed decreases by about 25%. perhaps under the small model setting, the addition of three encoding layers does not bring significant changes to the model quality. compared with the tiny.2 model, the size of our small.6 model doubles, resulting in an increase of 2.5 bleu. however, the inference speed are almost the same. in addition, the speed of our tiny model is 30% faster than our tiny.2 model by dropping a decoder layer. our result demonstrates that the number of decoding layers has greater impacts on decoding efficiency. as a result, we submit four models: base.12, small.9, small.6 and tiny. the above tests on inference speed are performed with intel(r) xeon(r) gold 6278c cpu @ 2.60ghz. ## conclusion in order to produce a translation system with high inference efficiency, we explore sentence-level distillation techniques and train student models with a trade-off between speed and quality by leveraging deep-encoder and shallow-decoder models. in terms of inference, we use huawei noah's bolt library. using a series of optimization techniques, such as int8 inference and custom efficient gemm operators, we accelerate inference speed by 2 to 3 times. by using shortlist, greedy search and caching, we submit four models with different settings to the efficiency one cpu core latency task, realizing efficiency improvement under different circumstances.