M obile NMT : Enabling Translation in 15 MB and 30ms (2023)

Abstract: Deploying NMT models on mobile devices is essential for privacy, low latency, and offline scenarios. For high model capacity, NMT models are rather large. Running these models on devices is challenging with limited storage, memory, computation, and power consumption. Existing work either only focuses on a single metric such as FLOPs or general engine which is not good at auto-regressive decoding. In this paper, we present MobileNMT, a system that can translate in 15MB and 30ms on devices. We propose a series of principles for model compression when combined with quantization. Further, we implement an engine that is friendly to INT8 and decoding. With the co-design of model and engine, compared with the existing system, we speed up 47.0x and save 99.5% of memory with only 11.6% loss of BLEU. Our code will be publicly available after the anonymity period.

## introduction as a classic subfield of natural language processing, neural machine translation (nmt) has achieved great success in recent years. most of the studies focus on improving the accuracy of large machine translation systems, ignoring whether such models are easy to be deployed in real-world scenarios. here we adopt four metrics to evaluate whether an nmt model is deployment-friendly. (1) model size is the most important metric in model compression @xcite . (2) floating-point operations (flops) is commonly used to evaluate computational complexity in neural architecture design. (3) memory or memory mapped i/o (mmi/o) reflects the memory requirements of the real running system. (4) decoding speed depends on many realistic factors such as engine implementation and the power of avaliable processors. our contributions are summarized as follows: ## architecture design principles for model compression and acceleration, most studies focus on a single metric such as model size or flops, without considering the real-world applications. in this section, we consider four metrics including model size, flops, memory usage, and decoding speed, and then propose three design principles for parameter-limited mt models. we choose transformer (appendix a) as our baseline because of its great success in machine translation. ## the engine this section introduces the detailed implementations of our inference engine. ## conclusion we propose mobilenmt, a transformer-based machine translation system that can translate in 15mb and 30ms. it uses existing resources efficiently and can be easily deployed in real-world scenarios. we develop a mobile inference engine with gemm and memory optimization, hoping that it can bridge the gap between theoretical research and real-world applications on efficient machine translation.