Investigating Techniques for a Deeper Understanding of Neural Machine Translation ( NMT ) Systems through Data Filtering and Fine-tuning Strategies (2023)

Abstract: In the context of this biomedical shared task, we have implemented data filters to enhance the selection of relevant training data for fine- tuning from the available training data sources. Specifically, we have employed textometric analysis to detect repetitive segments within the test set, which we have then used for re- fining the training data used to fine-tune the mBart-50 baseline model. Through this approach, we aim to achieve several objectives: developing a practical fine-tuning strategy for training biomedical in-domain fr<>en models, defining criteria for filtering in-domain training data, and comparing model predictions, fine-tuning data in accordance with the test set to gain a deeper insight into the functioning of Neural Machine Translation (NMT) systems.

## introduction the objective of our contribution to the biomedical shared task this year is to gain deeper insights into the nmt training pipeline, assess the factors influencing performance, and evaluate the robustness of the training system. our training strategy was to build tailor-made finetuning data with regard to the test data. we calculated repeated segments @xcite in the test data and used a selection of them to extract the corresponding data set from the available training data (as outlined in section 2.1.1). in order to highlight the "reproducibility", we consistently adhered to the same pipeline with minimum settings presented in section 2. while our experience encountered various technical obstacles that likely affected our system's performance, these challenges prompted us to prioritize explainability and comparability. this became especially important when the system produced irrelevant results and hallucinations. we elaborated on this in sections 3 and 4. the remaining sections of the paper are organized as follows: section 2 outlines our filtering pipeline, section 3 delves into our results, and section 4 provides a discussion of these results. section 5 explores related research and outlines future work. ## optimizing fine-tuning data selection and pipeline for this shared task, we exclusively fine-tuned the "mbart-50 large" baseline model @xcite over 3-epoch, 5-epoch, and 10-epoch training cycles. our approach prioritized reproducibility and involved a clear distinction between a "horizontal" dimension (facilitating inter-system analysis, especially in comparison with e-translation foot_0 ) and a "vertical" dimension (examining differences in training strategies within the same system). the fine-tuned setting was established with the following parameters set as a minimum: "num_train_epochs", "train_file", "validation_file", "test_file", "per_device_train_batch_size" foot_1 . ## related research and future work corpus filtering is discussed in many previous and recent works for training data preparation with different approaches : perplexity threshold of text segments @xcite , metric evaluations of raw nmt models' outputs @xcite , acceptability of filtering evaluated by mulitlingual bert classifier @xcite , etc. our ap-proach aims at data relevance between a given test set and in-domain training data. ## conclusion in this paper, we have described the translation systems used for the submissions in the wmt23 biomedical task6 (our data are available at: https://github.com/lichaozhu/wmt23 ). nevertheless, due to certain hardware constraints, we were unable to pinpoint the exact reasons for the model's underperformance. we also considered our previous participation in the biomedical task. since 2021, we have recognized that having scores provided in advance and reference texts used for score computation can significantly facilitate our work. these resources enable a more critical evaluation of the translations we generate. to address the absence of reference translations and evaluation results, translations can undergo spot checks. in our work, these checks involved the use of qualitative examples to assess the model's successes and failures. additionally, textometric browsing helped to unveil distinctive features within multiple machine translation outputs.