Fine-tuning MBART -50 with F rench and F arsi data to improve the translation of F arsi dislocations into E nglish and F rench (2023)

Abstract: In this paper, we discuss the improvements brought by the fine-tuning of mBART50 for the translation of a specific Farsi dataset of dislocations. Given our BLEU scores, our evaluation is mostly qualitative: we assess the improvements of our fine-tuning in the translations into French of our test dataset of Farsi. We describe the fine-tuning procedure and discuss the quality of the results in the translations from Farsi. We assess the sentences in the French translations that contain English tokens and for the English translations, we examine the ability of the fine- tuned system to translate Farsi dislocations into English without replicating the dislocated item as a double subject. We scrutinized the Farsi training data used to train for mBART50 (Tang et al., 2021). We fine-tuned mBART50 with samples from an in-house French-Farsi aligned translation of a short story. In spite of the scarcity of available resources, we found that fine- tuning with aligned French-Farsi data dramatically improved the grammatical well-formedness of the predictions for French, even if serious semantic issues remained. We replicated the experiment with the English translation of the same Farsi short story for a Farsi-English fine-tuning and found out that similar semantic inadequacies cropped up, and that some translations were worse than our mBART50 baseline. We showcased the fine-tuning of mBART50 with supplementary data and discussed the asymmetry of the situation, adding little data in the fine-tuning is sufficient to improve morpho-syntax for one language pair but seems to degrade translation to English.

## introduction farsi (persian) is shown to be the language having least datasets in a survey of neural machine translation for low-resource languages @xcite . 1 previous research on neural machine translation for farsi shows clear limitations of existing systems @xcite and highlights the scarcity of nlp resources @xcite for farsi, as shown in the contributions of the proceedings of the workshop on nlp solutions for under-resourced languages @xcite .moreover, previous research on using mbart50 @xcite , a multilingual model trained on 50 languages, has shown its limitations for the translations of farsi into english and even more so for the translations into french where hallucinations @xcite and english words were observed in the translated texts @xcite . this paper is a follow-up on this initial series of observations and reports our first series of experiments to fine-tune mbart50 for the translation of farsi into english and french. farsi being an under-resourced language, we found that the translation into french gave way to hallucinations and english words, whereas most of the challenge set created for the occasion was translated into grammatical sentences from farsi into english, but nevertheless failed to capture what we called 'pragmatic adequacy'. the translations into english like 'i hate the heat' followed the english svo canonical order and lost the pragmatic intention ('as to the heat, i hate it'). focusing on a specific syntactic phenomenon, dislocation, may provide us with a better perspective on the nature of pragmatic inadequacies. dislocation is a syntactic phenomenon that is productive in farsi. left dislocation constructions, like in french and, less frequently, in english, can have specific semantic and pragmatic functions . they can be used to promote a topic @xcite or to focus on a specific constituent of the sentence. several constructions are available for dislocations in farsi. @xcite studied left dislocated construction as a marked construction in the framework of construction grammar @xcite . they suggest that "a syntactic two-place construction is responsible for preposing the oblique to the sentence-initial position by leaving a pronominal enclitic coreferential with it in its original place", show that it may apply to monovalent, divalent and trivalent verbs. in the construction they identify, almost all participant roles can be left-dislocated except agents and experiencers. the left-dislocated element can be marked with "-ra." in this example, the object (garma) is initially stated, and it can transfer a semantic effect based on the context. ## previous research to improve multilingual models, previous methods include adding data with back-translation from monolingual data @xcite , data from multimodal input (see for instance pictures and their descriptions for bengali @xcite , data from languages of the same language family @xcite , using denoisers to add other languages incrementally @xcite or fine-tuning on small datasets @xcite . while other researchers have worked on more "massively multilingual nmt" @xcite adding data to an initial subset of the ted talks for no less than 103 languages (including one million examples in farsi), we have focused on controlling parallel data for farsi to english and farsi to french translations. among the research questions we had is the effect of language interference or at least the fact that we found english tokens in the translation of farsi into french, probably because of this bootstrapping effect of the co-presence of the different languages. we were more interested in the change in one language, namely potentially english, when we added more data in french and especially we wanted to see if adding more french data would change the number of english tokens that we found in the translations into french. ## materials and methods for mbart50, the system does not include bilingual french and farsi data, but the 25 and then 50 language pairs of languages with english as a pivot so that the system manages to learn from the different language pairs including english and another language and enable translation from one language to the other even if there are no training data for a specific language pair outside english. as indicated in @xcite , in mbart50, french belongs to the first group of training data, containing more than 10 millions of training data in bitext pairs and farsi corresponds to the middle tier (100k to 1m tokens). the appendix mentions 14,4895 sentences for farsi (ted 58) used for training and 3,930 for validation and 4,490 for test. the training data for french is made up of the wmt14 training data, resulting in 36,797,950 (train), 3,000 (validation) and 3,003 (test) sentences. ## results this section summarises some of our findings in terms of the quality of the translations. 5 by paying attention to our distinction between pragmatic adequacy and syntactic adequacy, we discuss the differences in translation that were more satisfactory from a pragmatic standpoint. ## conclusion in this paper we have reported a small-scale experiment trying to fine-tune mbart50 for the translations from french into farsi. we have not tried yet to fine-tune on the facebook implementation of cross-lingual language model pretraining @xcite , which has farsi as one of the 100 languages used for the multilingual training. the baseline results may be more satisfactory for this model than what we observed with mbart50. our paper is just a small case study that contributes to cross-lingual language understanding (xlu) of these multilingual models. it remains to be seen whether this xlm-100 model developed for crosslingual representation learning (xlr), which outperforms mbert @xcite , would display similar tendencies, since more data was collected for farsi (13,259 m tokens) than for french (9,870 m tokens) on common crawl and wikipedia. our main finding is that even a small amount of fine-tuning drastically reduces the number of foreign words in the translations and the number of hallucinations. an important drawback is that our fine-tuned model with french-farsi parallel data only translates into french, as a direct consequence of the hug-gingface implementation. more research is needed to investigate whether more aligned data in the fine-tuning phase would allow predictions to be semantically more adequate. taking into account what we have managed to learn with so little data, we would like to be able to evaluate what a system is capable of learning during fine-tuning, and at what cost (at least in terms of the number of examples needed for learning, and -if we have a good idea to avoid that -in terms of "forgetting"). more specifically, we would like to compare the ability of fine-tuning to change : in both cases, the idea would be to use artificial examples to control exactly what is present in the fine-tuning data and how it impacts the resulting translations. for lexical capacity, we could, for example, create sentences with new words and watch when the system is able to produce them, or "change" the meaning of a word (typically, replace all "green" tokens with "peanuts" in a parallel corpus) and watch when the system's output changes. the assumption is there might be thresholds of frequency in the fine-tuning data to observe this. for syntactic capacity, we could start with the translation of dislocation: we would create an artificial corpus with well-translated dislocations and look at the size of the set that needs to be provided for the system to be able to translate dislocations correctly. to clarify what we mean by "artificial corpus", we have the intuition that we could define a "parallel pattern" like the pattern used to investigate gender bias (the n has finished pron work) @xcite to "model" the translation of a dislocation (in the same way a pattern like det adj noun is translated into det noun adj) that we would simply instantiate on a lexicon (without taking into account the semantics at the source sentence level, but guaranteeing that the "bilingual links" of the syntactic structure are well preserved). for both experiments, it would be interesting to see what happens on eng-fas, fra-fas, eng-fra pairs to see the impact of the pre-trained model. we have identified the parallel translations of the ted talks which could be used as a starting point for this triangulation.