Findings of the WMT 2025 Shared Task on Model Compression: Early Insights on Compressing LLM s for Machine Translation (2025)

Abstract: We present the results of the first edition of the Model Compression shared task, organized as part of the 10th Conference on Machine Translation (WMT25). The task challenged participants to compress Large Language Models (LLMs) toward enabling practical deployment in resource-constrained scenarios, while minimizing loss in translation performance. In this edition, participants could choose to compete in either a constrained track, which required compressing a specific model (Aya Expanse 8B) evaluated on a limited set of language pairs (Czech→German, Japanese→Chinese, and English→Arabic), or an unconstrained track, which placed no restrictions on the model and allowed submissions for any of the 15 language directions covered by the General MT task (GenMT). We received 12 submissions from three teams, all in the constrained track. They proposed different compression solutions and covered various language combinations: all targeted Czech→German, while one covered all the language pairs. Evaluation was conducted separately for each language, measuring translation quality using COMET and MetricX, model size, and inference speed on an Nvidia A100 GPU.

## introduction large language models (llms) have demonstrated exceptional performance across a wide range of tasks. however, efforts to enhance their capabilities, by expanding language coverage, integrating multimodal data, and improving task generalization, have led to a dramatic increase in both model size and computational demands @xcite . this rapid growth poses significant challenges for real-world deployment, particularly in resource-constrained environments such as mobile devices, embedded systems, and edge computing platforms, where low-latency, on-device processing is often required. compressing foundation models is therefore more than a technical pursuit: it is a strategic priority with implications for global ac-cessibility 1 and the sustainability of computational and environmental costs. striking the right balance between performance, compactness, and efficiency is thus essential to make llms truly ubiquitous and beneficial for everyone, regardless of location or access to high-end infrastructure. it is with this long-term goal in mind that, following the analogous task for speech translation in the iwslt 2025 campaign @xcite , the new model compression shared task was introduced at wmt 2025. 2 this initiative follows three editions of the shared task at the workshop on machine translation and generation @xcite @xcite and two editions of the shared task on efficient translation @xcite . it revives earlier focus on the efficiency of machine translation, while updating it to reflect the current ai landscape with the rise of general-purpose llms. in this context, our aim is to provide a timely evaluation of compression techniques for generalpurpose llms within the specific task of machine translation. this setting offers a valuable opportunity to explore key research questions, such as: the goal of the model compression task is to reduce the size of a general-purpose llm while preserving a strong balance between compactness and mt performance. this section provides a brief overview of how the first round of the task was structured, focusing on the proposed tracks, data conditions, and evaluation methodology. ## participants three teams submitted systems to the task, as summarized in ayaq foot_3 this participation employs gptq 4-bit quantization @xcite with a group size of 32 to enable efficient and scalable llm inference. the wmt dataset is used as calibration data to guide the quantization process, ensuring the compressed model retains high accuracy on language understanding and generation tasks. the quantized models are integrated through the llm compressor framework (ai and vllm project, 2024), which streamlines conversion and metadata management. the setup is fully compatible with vllm @xcite , a high-throughput inference engine optimized for gpu deployment, enabling fast and memory-efficient execution with minimal performance loss. this approach demonstrates how structured quantization, targeted calibration, and system-level integration can enable practical, production-ready llm deployment. leanaya this participation is based on lean-quant (loss-error-aware network quantization, @xcite ), an accurate, versatile, and scalable quantization method. existing iterative loss-error-based quantization techniques typically rely on min-max affine grids, which often degrade model quality due to outliers in the inverse hessian diagonals. leanquant overcomes this limitation by learning loss-error-aware quan-tization grids instead of using fixed, non-adaptive ones. this approach not only improves accuracy but also supports a wider range of quantization schemes, including both affine and non-uniform, enhancing compatibility across diverse deployment frameworks. tcd-kreasof @xcite this participation employs iterative layer pruning to incrementally identify and remove layers that contribute least to translation quality, one at a time. layer importance is assessed by measuring translation performance with each layer individually removed. after pruning the least critical layer, the evaluation is repeated on the remaining ones until the target pruning level is reached. the resulting pruned model was then fine-tuned on 100k sentences from the news commentary dataset. this process produced three submissions: the primary one is a 24layer model with 6.28b parameters, while the two contrastive submissions are 20-layer and 16-layer models, with 5.41b and 4.54b parameters, respectively. vicomtech @xcite this participation employs gelaco @xcite , an evolutionary approach to llm compression based on layer merging operations. models are compressed at three ratios (0.25, 0.50, and 0.75), representing the proportion of original layers collapsed through differential weight merging. to recover performance after compression, over 3 million translation instructions (1 million per language) from a subset of wmt25 translation data are used. for the 0.25 and 0.50 compression levels, models are fine-tuned on this data, while the 0.75 model is trained using general knowledge distillation (gkd @xcite ). additionally, post-training quantization is applied using the bitsandbytes library foot_4 to further reduce model size to 8-bit and 4-bit precision. the primary submission (gelaco-0.75_gkd_q4) combines evolutionary layer collapse, knowledge distillation, and quantization to achieve substantial size reduction while maintaining reasonable translation performance. baselines as a reference system (see ## results we evaluate systems' performance on the official wmt25 genmt test sets, which comprise between 332 and 456 paragraphs for each of the three considered language pairs. because we received submissions to the constrained track only and most submissions focused on @xmath0, we primarily report results for this language pair unless mentioned otherwise. we noticed that most of the systems may suffer from hallucinated content, which affects both translation quality and decoding speed, and complicates system comparisons. to minimize the impact of hallucinations, we segment the original paragraphlevel test data at newline characters and remove empty lines. for czech-german, this results in a test set with 2,868 segments. to further investigate the potential impact of hallucinations, we also evaluate on a subset of the test set that only includes 1,928 segments where none of the systems exhibits hallucinations under any batch setting. this subset makes it possible to assess system performance in terms of inference speed more accurately, by excluding the distortions that hallucinations would introduce into cross-system comparisons. for brevity, analysis on the hallucination-free test data is presented in appendix a. below, we briefly discuss the results in terms of quality, model size and speed. the detailed results across more benchmark settings are provided in appendix b. ## conclusion and future directions the final results of the wmt25 shared task on model compression are summarized in figure 2 shows pareto front comparisons across evaluation criteria. the key findings can be summarized as follows: overall, despite the moderate participation in this shared task limited the breadth of exploration, several submissions showed promising results. the results of this evaluation campaign highlight that task-specific compression of llms still warrants more research efforts, especially at high compression rates required for running systems on edge devices. the smallest model still requires almost 3gb of disk space, which is incompatible with many edge devices that are equipped with a few hundred mb of memory @xcite . additionally, high compression rates result in a significant performance drop. we believe that pushing the boundaries of compression rates and reducing the quality degradation in such settings represent the most interesting challenges for future research on the topic and for participants of the future editions of the task. looking ahead, future iterations of this task could benefit from expanding the evaluation to more language pairs, aligning more tightly with the evaluation benchmark at the genmt task, and including human assessments of the outputs.