Exploring Segmentation Approaches for Neural Machine Translation of Code-Switched E gyptian A rabic- E nglish Text (2023)

Abstract: Data sparsity is one of the main challenges posed by code-switching (CS), which is further exacerbated in the case of morphologically rich languages. For the task of machine translation (MT), morphological segmentation has proven successful in alleviating data sparsity in monolingual contexts; however, it has not been investigated for CS settings. In this paper, we study the effectiveness of different segmentation approaches on MT performance, covering morphology-based and frequency-based segmentation techniques. We experiment on MT from code-switched Arabic-English to English. We provide detailed analysis, examining a variety of conditions, such as data size and sentences with different degrees of CS. Empirical results show that morphology-aware segmenters perform the best in segmentation tasks but under-perform in MT. Nevertheless, we find that the choice of the segmentation setup to use for MT is highly dependent on the data size. For extreme low-resource scenarios, a combination of frequency and morphology-based segmentations is shown to perform the best. For more resourced settings, such a combination does not bring significant improvements over the use of frequency-based segmentation.

## introduction code-switching (cs), i.e. the alternation of language in text or speech, has been gaining worldwide popularity, due to several reasons, including globalization and immigration. while this has been met with a growing interest in the nlp field to build systems that can handle such mixed input, work on cs machine translation (mt) is still considered in its infancy, where only a few language pairs have been investigated @xcite @xcite @xcite ). * work done while at the university of stuttgart. cs sentence ‫ﺑﺼﺮاﺣﺔ‬ ‫ب#ﺻﺮاﺣﺔ‬ ‫ﺑﺎﻟﻨﺴﺒﺎﻟﻲ‬ ‫ب#اﻟﻨﺴﺒﺎ#ل#ي‬ ‫ع‬ ‫ع‬ ‫ال‬ ‫ال‬ it depends ‫ال‬ ‫ع‬ ‫ﺑﺎﻟﻨﺴﺒﺎﻟﻲ‬ ‫ﺑﺼﺮاﺣﺔ‬ situation it depends situation for me it honestly depends on the situation it it it depends depends depend#s honestly for me on the situation situation situation bsrahp balnsbaly e al bsrahp b#srahp balnsbaly b#alnsba#ly e e al al translation word translation segmentation figure 1 : an example sentence with code-switching (cs) between english and egyptian arabic. the words are contrasted with their segmentations and english translations. arabic words are paired with their transliterations in the buckwalter scheme @xcite . in this work, we focus on the cs egyptian arabic (egy)-english (en) language pair, as we observe its usage is becoming more common. besides being prevalent amongst egyptian migrant communities, it is also commonly used in egypt due to the increase in international schooling systems and educational advancements. we identify three main challenges for cs mt. first is data sparsity, a challenge common to many cs language pairs because of limited parallel corpora containing commissioned translations of cs text @xcite @xcite @xcite . second is egyptian arabic morphological richness, which further exacerbates the data sparsity situation @xcite . third, since the matrix language (egy) is morphologically rich, cs occurs at three cs levels: on the boundaries of sentences (inter-sentential cs), between words (intrasentential cs), and within words, i.e., morphological code-switching (mcs). this mix of types of cs raises the question of how to handle them all in the same system. these challenges are further illustrated in figure 1 . a common solution to handle data sparsity for mt of morphologically rich languages is morphological segmentation @xcite @xcite . however, this has not been investigated for cs. in this paper, we explore a wide range of segmentation approaches, covering unsupervised morphologybased segmenters, unsupervised frequency-based segmenters, and supervised morphology-based segmenters. this work aims to answer the following research questions (rqs): while our results show that there is no correlation between correct morphological segmentation and mt performance, we find that the performance ranking between the mt systems varies across different training data sizes and sentence types (monolingual vs. code-switched). we show that applying a combination of supervised morphology-based and unsupervised frequency-based segmentations consistently gives best results, with statistical significance under low data sizes. while commonwisdom suggests that byte-pair encoding (bpe) is the best approach, our experiments highlight the importance of integrating morphological knowledge in the case of extreme low-resource settings. we believe that the insights and methodology we follow will be useful to researchers working with low-resource languages. an additional contribution of our research is the creation of a gold standard morphologically annotated cs egyptian arabic-english dataset which we make publicly available. 1 the paper is organized as follows. section 2 discusses related work. section 3 describes the dataset we annotated. section 4 describes and evaluates the different segmenters used. section 5 describes and evaluates the various mt systems. in section 6, we answer our research questions. ## related work several researchers have investigated the effect of applying different morphological and agnostic segmentation approaches on the mt performance for monolingual languages. @xcite ; @xcite show that unsupervised morphology-based segmentation like linguistically motivated vocabulary reduction (lmvr) @xcite , morfessor @xcite , and flatcat @xcite for nepali-english, sinhala-english, kazakh-english, and inuktitut-english language pairs show either no improvement or no significant improvement over the agnostic bpe segmentation @xcite in translation tasks. meanwhile, @xcite and @xcite show that for polysynthetic and highly agglutinative languages, unsupervised morphology-based segmentation outperforms bpes @xcite in mt tasks in both directions. nevertheless, applying bpes on top of morphology-based segmentation for turkish-english, uyghur-chinese, and arabic-english has shown to bring improvements over solely using bpes or morphology-based segmentation for neural mt task @xcite . a similar result was achieved by @xcite , using a morphological guided bpe for polysynthetic languages. however, @xcite show that such an approach is beneficial in the case of statistical machine translation (smt), and does not improve results for neural machine translation (nmt). for other natural language processing (nlp) tasks, @xcite show that utilizing word segmented arabic dataset leads to improvements in text classification task over utilizing unsegmented dataset in terms of accuracy, precision, recall, and f-measure. as for work on cs mt, there are many efforts @xcite @xcite @xcite @xcite @xcite . to the best of our knowledge, none of these efforts presented an extensive comparison covering different segmentation techniques. with regards to the languages covered, only @xcite worked on cs arabic-english. however, since they used carefully edited un documents, the text only included the modern standard arabic variety, and contained limited types of cs. with regards to similar corpora, balabel et al. (2020) annotated cs egyptian arabic-english data @xcite with tokenization (canonical segmentation), lemmatization, and pos tags. however, their corpus does not contain translations. ## machine translation experiments since no previous research investigates the best segmentation technique for nmt of the code-switched egyptian arabic-english language pair, we explore training nmt models using the various segmentation setups discussed in section 4 to answer rq1. moreover, we analyze the performance of the topperforming mt systems on different types of cs sentences to answer rq2. afterward, we compare the mt scores against the emma f1 scores discussed in section 4.3 to answer rq3. ## discussion we revisit the rqs we outlined in our introduction. rq1 -which segmentation setup performs the best in the downstream mt task across different training sizes? results show that frequencybased segmentation applied on top of morphologybased segmentation outperforms the other segmentation techniques, with statistical significance on lower resource settings. the superiority of this approach is seen across sentences with varying morphological richness, percentage of english words, and across sentences with different cs types. we believe the strength of the combination is because it exploits complementarity of both methods. on one hand, supervised morphology-based segmenters bring in high correctness; however, they are not always robust, having high oov rates. on the other hand, while bpe segmentation is not necessarily morphologically correct, it achieves high robustness. the robustness of bpe is consistent with the findings in @xcite . rq2 -does the effectiveness of the different segmenters in the mt task differ according to the cs type of the source sentence? we observe that the effectiveness of the different segmenters on mt performance is consistent across two categories of cs sentences; those with and without mcs. however, when comparing their effectiveness on monolingual arabic vs. cs sentences, we observe that the rankings between segmenters are not consistent. in the case of constrained data size settings (25% and 50% of data), we observe a clear pattern where mt at b outperforms mt bp e on monolingual sentences, while mt bp e outperforms mt at b on cs. in the case of using 100% of the training data, mt at b+bp e outperforms mt bp e on cs sentences; however, mt bp e outperforms mt at b+bp e on monolingual arabic sentences. since our test and dev sets are dominated by cs sentences (61.5% and 63.8%, respectively), we believe that the overall ranking is more greatly affected by the systems' performance on cs sentences, thus reflecting the same ranking on the overall evaluation set as that across cs sentences. rq3 -is there a correlation between a more morphologically correct segmentation and mt performance? for unsupervised morphologybased segmenters, a segmenter with a better segmentation emma f1 score also scores better in the downstream mt task. however, we cannot hypothesize that a better segmentation score implies a better translation system, as counter examples exist. for example, while we notice that mdmr at b gives the best segmentation in terms of emma f1 score, it does not outperform any of the top-performing mt systems. we hypothesize that despite mdmr at b 's capability of generating morphologically correct segmentations, it can generate infrequent morphemes due to the out-ofdomain data which it is trained on. this may not only increase the sentence length which worsens mt performance as shown in @xcite , but may also be one of the contributing factors to the 9.70% oov percentage found in mt at b . on the contrary, bpe joint performs the worst in the segmentation task as we expect, since it is designated for agnostic-based segmentations; however, it surpasses the top-performing mt models. we believe this is due to its capability to generate semicorrect segmentation and to reduce oov rates. ## conclusion and future work in this paper, we study the impact of a comprehensive set of morphological and frequency-based segmentation methods on mt, where the source is a code-switched text. the experiments are performed on code-switched arabic-english to english. we found that the supervised morphological segmenter achieved the best performance on the segmentation task, followed by unsupervised morphological methods, and finally, unsupervised frequency-based. afterward, we train 18 different mt systems with different source and target side segmentations. we find that the rank of the segmenters is reversed, as bpe's could not be outperformed (significantly) by any morphologicalinspired segmentation method. however, combining morphology-based and frequency-based segmentations has shown to give improvements, which are statistically significant in lower resource settings, where the training data size is reduced to 25% and 50%. for future work, we plan to apply our different mt setups on other low resource and code-switched language pairs. specifically, we plan to explore languages with different typologies, to study whether or not the relation between the data size and choice of the segmentation setup (frequency-based, morphology-based, or a mix) is based on morphological features and data size rather than the language itself. moreover, we plan to extend our annotated dataset, arzenseg, by adding further details to allow evaluating different schemes.