Investigating Code-Mixed M odern S tandard A rabic- E gyptian to E nglish Machine Translation (2021)

Abstract: Recent progress in neural machine translation (NMT) has made it possible to translate successfully between monolingual language pairs where large parallel data exist, with pre-trained models improving performance even further. Although there exists work on translating in code-mixed settings (where one of the pairs includes text from two or more languages), it is still unclear what recent success in NMT and language modeling exactly means for translating code-mixed text. We investigate one such context, namely MT from code-mixed Modern Standard Arabic and Egyptian Arabic (MSAEA) into English. We develop models under different conditions, employing both (i) standard end-to-end sequence-to-sequence (S2S) Transformers trained from scratch and (ii) pre-trained S2S language models (LMs). We are able to acquire reasonable performance using only MSA-EN parallel data with S2S models trained from scratch. We also find LMs fine-tuned on data from various Arabic dialects to help the MSAEA-EN task. Our work is in the context of the Shared Task on Machine Translation in Code-Switching. Our best model achieves 25.72 BLEU, placing us first on the official shared task evaluation for MSAEA-EN.

## introduction recent year have witnessed fast progress in various areas of natural language processing (nlp), including machine translation (mt) where neural approaches have helped boost performance when translating between pairs with especially large amounts of parallel data. however, tasks involving a need to process data from different languages mixed together remain challenging for all nlp tasks. this phenomenon of using two or more languages simultaneously in speech or text is referred to as code-mixing @xcite and is (1) msaea . ## related work a thread of research on code-mixed mt focuses on automatically generating synthetic code-mixed data to improve the downstream task. this includes attempts to generate linguistically-motivated sequences @xcite . some work leverages sequence-to-sequence (s2s) models @xcite to generate code-mixing exploiting an external neural mt system, while others @xcite use a recurrent neural network along with data generated by a sequence generative adversarial network (seqgan) and grammatical information such as from a part of speech tagger to generate code-mixed sequences. these methods have dependencies and can be costly to scale beyond one language pair. arabic mt. for arabic, some work has focused on translating between msa and arabic dialects. for instance, @xcite studied the impact of combined dialectal and msa data on dialect/msa to english mt performance. @xcite uses msa as a pivot language for translating arabic dialects into english. @xcite investigate the effect of sentence-level dialect identification and several linguistic features for msa/dialect-english translation. @xcite propose an neural machine translation (nmt) system for arabic dialects using a vanilla recurrent neural networks (rnn) encoder-decoder model for translating algerian arabic written in a mixture of arabizi and arabic characters into msa. baniata et al. (2018) present an nmt system to translate levantine (jordanian, syrian, and palestinian) and maghrebi (algerian, moroccan, tunisia) to msa, and msa to english. @xcite , propose unsupervised dialectal nmt, where the source dialect is not represented in training data. this last problem is referred to as zero-shot mt @xcite . ## experiments and settings in this section, we describe the different ways we fine-tune and evaluate our models. ## evaluation on dev data we report results of all our models under different settings in bleu scores @xcite . discussion. we inspect output translations from our models on test data. we observe that even though s2st performs better than the two language models on test data, both of these models are especially able to translate egyptian arabic tokens such as in example (1) in returning to our three main research questions, we can reach a number of conclusions. for rq1, we observe that models trained from scratch on purely msa data fare reasonably well on the codemixed msaea data (i.e., zero-shot ea setting). this is due to lexical overlap between msa and ea. for rq2, we also note that language models such as mt5 and mbart do well under the codemixed condition, more so than models trained from scratch when inference data involve more ea. this is the case even though these language models in our experiments are fine-tuned with significantly 62 less data (i.e., ∼ 250k pairs) than the from-scratch s2st models (which are trained on 55m msa + 250k da pairs). for rq3, our results show that training on data from various arabic dialects helps translation in the msaea code-mixed condition. this is in line with previous research @xcite showing that exploiting data from various dialects can help downstream translation on a single dialect dialect in the zero-shot setting. ## conclusion we described our contribution to the shared tasks on mt in code-switching. 7 our models target the msaea to english task under the unsupervised condition. our experiments show that training models on msa data is useful for the msaeato-english task in the zero-shot ea setting. we also show the utility of pre-trained language models such as mt5 and mbart on the code-mixing task. our models place first in the official shared task evaluation. in the future, we intend to apply our methods on other dialects of arabic and investigate other methods such as backtranslation for improving overall performance.