Vicomtech@ WMT 2025: Evolutionary Model Compression for Machine Translation (2025)

Abstract: We describe Vicomtech’s participation in the WMT 2025 Shared Task on Model Compression. We addressed all three language pairs of the constrained task, namely Czech to German, English to Arabic and Japanese to Chinese, using the Aya Expanse 8B model as our base model. Our approach centers on GeLaCo, an evolutionary method for LLM compression via layer collapse operations, which efficiently explores the compression solution space through population-based search and a module-wise similarity fitness function that captures attention, feed-forward, and hidden state representations. We systematically evaluated compression at three different ratios (0.25, 0.50, and 0.75) and applied targeted post-training techniques to recover performance through fine-tuning and knowledge distillation over translation instructions. Additionally, we explored quantization techniques to achieve further model size reduction. Our experimental results demonstrate that the combination of evolutionary layer compression, targeted post-training, and quantization can achieve substantial model size reduction while maintaining competitive translation quality across all language pairs.

## introduction the remarkable success of large language models (llms) across diverse natural language processing tasks @xcite @xcite has established them as powerful tools for language understanding and generation. beyond their general capabilities, llms have also demonstrated remarkable effectiveness in machine translation tasks, often matching or exceeding the performance of dedicated neural machine translation systems @xcite @xcite . simultaneously, recent work has focused on the development of specialized multilingual llms designed specifically for translation and crosslingual tasks, such as aya expanse @xcite , @xcite , and tower @xcite . however, these advances come at the cost of substantial computational requirements. modern llms, ranging from billions to trillions of parameters, demand considerable memory and processing power for both training and inference, with associated environmental impacts that raise serious sustainability concerns @xcite . these computational requirements create barriers to widespread deployment and usage where reduced memory footprint and efficient inference are essential for practical adoption. in this work, we describe vicomtech's participation in the constrained track of the wmt 2025 model compression shared task @xcite . this task focuses specifically on making llms suitable for deployment in machine translation within resource-constrained environments. the task evaluates compression techniques across multiple dimensions: model size reduction, translation quality preservation, and inference speed optimization. participants were tasked to compress the aya expanse 8b model while maintaining competitive translation performance across three language pairs: czech-german, english-arabic, and japanese-chinese. to address these challenges, we employed gelaco @xcite , an evolutionary algorithm for llm compression that builds upon the layer collapse operations of laco @xcite . gelaco efficiently explores the compression solution space through population-based search and a module-wise similarity fitness function that captures attention, feed-forward, and hidden state representations. we systematically applied this approach across multiple compression figure 1 : overview of our model compression pipeline. starting from aya expanse 8b (32 layers, 8b parameters), we apply gelaco compression at three ratios (0.25, 0.50, 0.75), reducing to 24, 16, or 8 layers respectively. compressed models undergo post-training via supervised fine-tuning or generalized knowledge distillation for performance recovery, followed by optional 8-bit or 4-bit quantization for additional size reduction. ratios (0.25, 0.50, and 0.75) for all three language pairs. we explored targeted post-training techniques, including continued pre-training and knowledge distillation, to recover translation performance after compression. additionally, we used quantization methods to achieve further size reduction while maintaining translation quality. our experimental results demonstrate that the combination of evolutionary layer compression, targeted post-training, and quantization can achieve substantial model size reduction while preserving competitive translation capabilities across diverse language pairs. figure 1 presents an overview of our compression pipeline. ## results we present our experimental results analyzing the impact of compression ratios, post-training approaches, and quantization on translation quality, 5 https://huggingface.co/unbabel/wmt22-comet-da model size, and inference speed. ## conclusions this work presented our approach to the wmt 2025 model compression shared task, focusing on compressing the aya expanse 8b model for machine translation across czech-german, english-arabic, and japanese-chinese language pairs within the constrained setting of the task. we employed gelaco, an evolutionary algorithm for layer collapse operations, combined with posttraining techniques and quantization, to achieve substantial model size reduction while maintaining competitive translation performance. our experimental results demonstrated that compressed models can be successfully recovered through targeted post-training techniques. generalized knowledge distillation consistently outperformed traditional fine-tuning for performance recovery across all three language pairs at the 0.75 compression ratio where it was applied. the combination of layer compression with 4-bit quantization achieved an 81% reduction in model size (from 14.96 gib to 2.79 gib) while preserving reasonable translation quality, making such models viable for resource-constrained scenarios.