Rule-Based, Neural and LLM Back-Translation: Comparative Insights from a Variant of L adin (2024)

Abstract: This paper explores the impact of different back-translation approaches on machine translation for Ladin, specifically the Val Badia variant. Given the limited amount of parallel data available for this language (only 18k Ladin-Italian sentence pairs), we investigate the performance of a multilingual neural machine translation model fine-tuned for Ladin-Italian. In addition to the available authentic data, we synthesise further translations by using three different models: a fine-tuned neural model, a rule-based system developed specifically for this language pair, and a large language model. Our experiments show that all approaches achieve comparable translation quality in this low-resource scenario, yet round-trip translations highlight differences in model performance.

## introduction in recent years, a variety of methods have been developed to apply neural machine translation (nmt) also in low-resource scenarios @xcite @xcite . the back-translation technique has shown to be particularly effective in such settings @xcite , offering the potential for substantial improvements in translation quality. this work investigates the influence of the backtranslation model selection for a low-resource language. we do this, by comparing the results obtained by fine-tuning a pre-trained multilingual nmt model using synthesised translations generated by (i) a nmt system fine-tuned on the available parallel data, (ii) a rule-based machine translation (rbmt) system developed for the specific language pair, and (iii) a large language model (llm) prompted to translate given texts, accompanied by 8 exemplary samples. the quality of the synthesised data, which in turn is determined by the underlying models used to generate it, matters @xcite . in our case, the synthesised translations originate from three models based on a different paradigm. thus, the synthesised data is characterised by the specific strengths and weaknesses of the respective paradigms. rule-based systems are robust and computationally lightweight, but may face challenges in dealing with ambiguity. moreover, they lag behind at the grammatical level. neural models show a high ability to adapt to provided texts, but perform less well when confronted with out-of-domain data @xcite . in contrast, language model-based approaches (llms) are praised for their ability to produce fluent, coherent texts, but they are prone to hallucinations @xcite . it is therefore an interesting question to investigate how this affects the quality of the nmt models trained on this data. this comparative analysis sheds light on the nuanced contrasts inherent in the different mt methods. our results show that in this low-resource scenario the back-translation model does not have a significant impact, and the performance of the models converges to similar results in terms of bleu/chrf++ points. this assertion is supported by an empirical analysis carried out on the val badia variant of ladin. our main contributions are: in section 2 we describe our data collection and corpus creation process. in section 3 we present the three different methods used for the back-translation of monolingual ladin data into italian. section 4 gives an overview of the conducted experiments and section 5 presents the obtained results. section 6 discusses related work and similar approaches. in section 7, we summarize and discuss future work. the ladin language ladin is an officially recognised minority language, and thus taught in schools, used in the media, and employed in public administration. for this reason, an effective machine translation system could make a significant contribution to facilitating and supporting communication in this language. however, ladin is still an unexplored language in the field of machine translation. indeed, nearly no parallel data foot_0 is publicly available for this language, except for a few hundred samples on opus @xcite . this language, spoken by around 30,000 people in the northern italian dolomite regions, exhibits significant diversity across its five main variants (val badia, gherdëina, fascia, fodom, anpezo), each shaped uniquely by its development in different valleys. this diversity is not only evident in the spoken language but has also resulted in distinct standards for written communication. the first author of the paper originates from the val badia and is a native speaker of ladin. therefore, in this work we concentrate on the standard written language of this valley. in the rest of the paper, we will use lvb as language code to refer to this variant of ladin, and ita for italian. ## data this section gives an overview of the linguistic resources available for the ladin language and describes the method employed to collect data for the specific val badia variant of ladin. ## back-translation strategies the so-called back-translation, first introduced in @xcite , refers to the process of automatic translation of monolingual texts in the target language to the source language. this method of enriching additional training data in the sourceto-target translation direction (where the target side remains authentic) has proven to be particularly effective and is particularly valuable in low-resource scenarios. in this section we present the three different back-translation strategies used in our research to translate monolingual ladin texts into italian. ## experiments we used the opus-mt-ine-ine foot_15 model as base model for the experiments. in the rest of the paper, we use bm to refer to this model. we fine-tuned bm with the various data sets using the transformers library @xcite , specifically leveraging the seq2seqtrainer module. we always trained a single model for both directions using the corresponding prefixes. we configured the training to process batches of 16 samples, and restricted the input and output sequences to a maximum of 128 tokens to ensure manageable computation loads. the models were evaluated each 16,000 steps. as a stopping criterion, we used three consecutive evaluations resulting in an improvement of less than 0.2 chrf points on the validation set. for training, we utilised an nvidia titan rtx graphics card with 24 gb. in total, we have trained 15 models: we refer to the models n1, . . . , n5 that were trained with nmt backtranslated data as n-models. analogously, we use the term r-models and lmodels to refer to rbmt and llm models, respectively. models n4/r4/l4 illustrate the gains achieved through iterative back-translation @xcite . additionally, models n5/r5/l5 demonstrate potential improvements achievable with synthetically generated forward-translation data. we evaluated these models on the 3 test sets presented in section 2.5. the results are presented and analysed in the following section. ## results and discussion the results of the various experiments conducted are presented in additionally, as recommended in @xcite , in several findings can be deduced from these results, and will be discussed below. in general, there is evidence that augmenting the training data with monolingual data through back-translation is effective. n1 shows that fine-tuning the model with only authentic training data substantially improves the results in both directions (compared to bm) in terms of bleu/chrf++ points. this shows on the one hand that the training is effective and on the other hand that the available data is adequate. however, it is also evident that the model generates less fluent text, as indicated by the perplexity scores which increase for this model. the results reveal a progression in difficulty among the test sets, where testset 3 emerges as the most challenging one. on this test set, all approaches achieve similar low scores, suggesting the presented approach may face limitations with more complex texts. in the translation direction @xmath0, the best results were achieved by combining the different back-translations, as model a2 results indicate. this emphasises the importance of a broad and diversified dataset. remarkably, the a2 model is also competitive in the reverse translation direction (ita a comparison of the models n1, r1 and l1 suggests that the llm generates more fluent texts (low perplexity) but perhaps does not always accurately reproduce the meaning, as attested by the performance on testset 3 (low perplexity but also low bleu score). in this assessment, the rbmt system r1 also performs better than the fine-tuned nmt model n1. one of the reasons for the high perplexity values of n1 is that this model tends to hallucinate because it has been fine-tuned with a small data set. however, this does not seem to affect the performance as the models trained on this data do not perform considerably worse. the performance of the llm varies significantly, with pronounced differences between the three test sets in both directions of translation. the significant difference observed between testset 1 and testset 2 in the translation direction from @xmath1. it remains unclear to what extent the llm benefits from the given examples in the prompt. however, by providing an example, the propensity for errors was minimised, resulting in fewer mistakes during execution. even though llms are not (yet) suitable for generating texts in low-resource languages out-ofthe-box (see performance of l1 in the inclusion of forward translations in the training data did not consistently improve the models, with the exception of the r-models for @xmath3. this suggests that these synthesised texts introduce too much noise. however, model a2 was able to benefit from this data in the translation direction @xmath4. filtering this data could slightly improve the model. as the models achieve similar scores on the test data, we also examined the quality of round-trip translations to gain additional insights. for this, we used 10k sentences from the monolingual ladin and italian data (which were also used for training, hence the high scores), translated them into the other language and then back-translated them. this concept of so-called round-trip translation is a suitable evaluation method @xcite . we used the r4/n4/l4 models for this purpose, ap ## related work data augmentation such as back-translation @xcite and transfer learning @xcite are established strategies to improve mt systems. these concepts are discussed in @xcite ; @xcite , with a focus on low-resource scenarios. the fact that the synthesised data plays a critical role in the quality of the systems trained on it, as it also introduces a certain degree of noise, was discussed extensively in @xcite ; @xcite . it was shown that tagging synthetic data can be beneficial in the training process @xcite . in our work we do not apply advanced techniques to differentiate synthetic from real translations in training. the fact that rbmt systems can still be valuable for lowresource languages and can even help to achieve better results was also demonstrated for northern 18 https://doi.org/10.57967/hf/2695 19 https://doi.org/10.57967/hf/2693 20 https://doi.org/10.57967/hf/2694 sámi @xcite . in our experiments, we could also observe this in the translation direction @xmath5. this could be due to the ability of the rbmt system to provide general knowledge that is not available in the relatively limited parallel training datasets @xcite . the use of llms for mt and different prompting techniques was investigated in @xcite and their performance in the machine translation of low-resource languages has already been analysed in @xcite . even if they struggle to generate texts in low-resource languages @xcite , it has already been claimed that they can contribute to advances in machine translation of such languages. our work is an example of how llms can be used in machine translation of a low-resource language; however, further prompt engineering is needed to make better use of such models. ## conclusion in this work, we conducted a detailed comparison of rbmt, nmt and llms for back-translation in a low-resource scenario. we have tested various back-translation approaches and evaluated them for a previously unexplored language in the field of machine translation. our current methodology involved the exclusion of numerous ladin monolingual sentences. however, this filtering would be less important for the translation direction @xmath6. this previously discarded data could be re-incorporated to improve the performance of the models in this particular translation direction. the round-trip translation scores indicate that the initial back-translation with the rbmt system leads to more robust models. improving the ambiguity resolution of this rule-based translation system could lead to even better results. the simplicity of the prompts used to feed the llms provides a further starting point for investigations. in particular, the question arises as to whether the results can be improved by further prompt engineering, e.g., by including the meaning for the distinct words occurring in a text using the available dictionary. investigating the effects of prompt optimisation could provide new insights into maximising the efficiency of llms in machine translation, especially in low-resource scenarios. we plan to address these research questions in our future work.