The Green KNIGHT : Green Machine Translation with Knowledge-Distilled, Narrow, Inexpensive, Greedy, Hybrid Transformers (2025)

Abstract: State-of-the-art neural machine translation (NMT) models deliver high-quality translations at the expense of high inference latency and energy consumption, requiring vast GPU fleets and contributing significantly to carbon emissions. To democratize and “green” NMT, we introduce the Green KNIGHT, a hardware-agnostic collection of recipes to optimize translation speed and energy consumption, with only a moderate trade-off in quality. On high-resource En→De and En→Ko benchmarks, we achieve up to 117× CPU speedup and 98.2% energy savings with 9% relative BLEU decrease. On WMT 2014 En→De and En→Fr benchmarks, we obtain up to 140× speedup with 98.7% energy savings, while staying within 10–12% relative BLEU decrease. Our results demonstrate that efficient and environmentally conscious NMT can be realized through optimizations built on well-understood, off-the-shelf techniques with no custom low-level code required, making our approach immediately deployable in real-world translation pipelines.

## introduction neural machine translation (nmt) has rapidly become the standard for automated language transfer, achieving human-competitive fluency and adequacy across dozens of language pairs with transformer architectures @xcite . most research aims at improving model performance, which is typically achieved with larger and in particular deeper models whose inference time and energy consumption grow significantly due to the quadratic dependence on target length of the autoregressive decoder and high cost of beam search. as shown in figure 1 , the conventional transformer 'big' model can spend over 95% of its batch runtime in the decoder. this imbalance not only throttles throughput but also drives up energy * equal contribution. ## transformer models since the introduction of the transformer @xcite , this network architecture is the de facto standard for machine translation. the model can be described as being composed of two main parts; an encoder which compresses the input source sentence, and a decoder which autoregressively-i.e. word-by-word-generates the hypothesis in the target language. both, encoder and decoder consist of a stack of layers which transform dense vector-representations of a fixed model dimension d. the encoder as well as the decoder layers consist of a self-attention component, which scales quadratically in the input sequence length, as well as a feedforward sub-layer. in addition, the decoder also has a cross-attention component of similar complexity as the self-attention. despite encoder and decoder having a comparable total number of floating-point operations (roughly weighted 2:3 due to the cross-attention in the decoder), the computations of the encoder can be parallelized while the decoder in this architecture is inherently autoregressive. this effect is further amplified by beam search, and in our case leads to 95% of the total computation time being spent in the decoder, as presented in figure 1 . ## inference optimizations we commence with optimizing inference as it is independent of any system, i.e. it introduces no retraining burden and can be adopted with minimal engineering effort. ## architectural optimizations as our baseline, we adopt the state-of-the-art transformer 'big' model, in which the encoder and decoder share the same depth. as previously shown in figure 1 , the decoder dominates the encoder in inference runtime. hence, our focus lies on applying architectural modifications that shift computation towards the encoder and thus streamline the decoder, as illustrated in figure 2 . ## training optimizations after improving translation speed and energy consumption by optimizing inference and the model architecture, we address the question of how to optimize the training of our models. here, the goal is different, as we aim to obtain better models in terms of translation quality rather than making them faster and more energy efficient. to achieve this goal, we apply sequence-level knowledge distillation (kd; @xcite using the baseline model as the teacher model p t , which aids the training of the student model p s using the aforementioned optimized architecture. however, instead of using the cross-entropy between student and teacher, we use the kullback-leibler divergence d kl (•, •) to compute the additional training loss l kd : l ce is the cross-entropy loss of the student model and α, β are corresponding hyper parameters. ## experiments we evaluate our findings on four tasks in total: two general domain translation tasks resembling a industry-typical, unconstrained data condition for @xmath0, as well as the wmt 2014 datasets for @xmath1. we provide detailed intermediate results for both unconstrained settings, however moved the intermediate @xmath2. the overall goal is to build a system that preserves 90% translation quality of the baseline, while yielding as much translation speed and energy savings as possible. ## conclusion in this work, we introduced green knight, an easy-to-cook recipe that substantially accelerates inference and reduces energy consumption of nmt models while incurring only minimal loss in translation quality. in contrast to specialized low-level or hardware-specific optimizations, green knight achieves these gains through widely used and wellunderstood tools and methods, making it immediately adoptable in production nmt pipelines. our experiments on two language pairs show that this recipe achieves up to 140× speed-ups and up to 98.7% energy reduction, while maintaining at least 92.9% comet and 88.0% bleu scores relative to the baseline. crucially, the final models achieve the a 18-fold throughput on cpu in comparison to the throughput of the baseline model on gpu, significantly contributing to democratizing nmt. when applied on gpu, our model achieves a 47× speeup over the baseline model while reducing energy consumption by 98.4%.