Chasing COMET : Leveraging Minimum B ayes Risk Decoding for Self-Improving Machine Translation (2024)

Abstract: This paper explores Minimum Bayes Risk (MBR) decoding for self-improvement in machine translation (MT), particularly for domain adaptation and low-resource languages. We implement the self-improvement process by fine-tuning the model on its MBR-decoded forward translations. By employing COMET as the MBR utility metric, we aim to achieve the reranking of translations that better aligns with human preferences. The paper explores the iterative application of this approach and the potential need for language-specific MBR utility metrics. The results demonstrate significant enhancements in translation quality for all examined language pairs, including successful application to domain-adapted models and generalisation to low-resource settings. This highlights the potential of COMET-guided MBR for efficient MT self-improvement in various scenarios.

## results the statistical significance of the evaluation results is assessed using a paired bootstrap resampling test @xcite , involving 1000 resampling trials to confirm the statistical significance of the model improvements (@xmath0.05). ## conclusion this study demonstrated the effectiveness of model self-improvement through mbr decoding in improving translation quality. this approach proves beneficial for both high and lowresource languages, offering versatility in its application across diverse scenarios. examples include domain-specific translation and the enhancement of general translation models. we conducted experiments with various sample sizes for mbr decoding, using two decoding algorithms: beam search and top-k. the aim was to find a balance between automatic metric gains and time efficiency. our experiments have shown that the beam search algorithm with a beam size set to 50 is the optimal choice. in the field of high-resource english-to-german biomedical translation, we investigated the impact of domain adaptation using various self-improvement approaches on mbr-decoded forward-translated data. experiments showed that all mbr-based fine-tuning, regardless of the domain of the test set, improved performance compared to the baseline model. this finding highlights the robustness of the self-improvement technique. experiments on the czech-ukrainian language pair revealed that fine-tuning the mt model on mbr-decoded translations of the training data set significantly improves translation performance. applying this process iteratively improves quality, but further iterations yield diminishing gains and at some point, the quality may even degrade due to overfitting to the mbr decoding utility metric. in the english-hausa experiments, we employed two models for mbr decoding: wmt22 comet and africomet. both models yielded comparable and significant improvements in automatic metrics, indicating their effectiveness in guiding the self-improvement process. while africomet, specifically trained on african languages, might intuitively seem favourable for this language pair, the performance of the mbr-comet model highlights the potential of utilizing more widely applicable metrics like wmt22 comet even for low-resource settings.