Investigating Lexical Replacements for A rabic- E nglish Code-Switched Data Augmentation (2023)

Abstract: Data sparsity is a main problem hindering the development of code-switching (CS) NLP systems. In this paper, we investigate data augmentation techniques for synthesizing dialectal Arabic-English CS text. We perform lexical replacements using word-aligned parallel corpora where CS points are either randomly chosen or learnt using a sequence-to-sequence model. We compare these approaches against dictionary-based replacements. We assess the quality of generated sentences through human evaluation and evaluate the effectiveness of data augmentation on machine translation (MT), automatic speech recognition (ASR), and speech translation (ST) tasks. Results show that using a predictive model results in more natural CS sentences compared to the random approach, as reported in human judgements. In the downstream tasks, despite the random approach generating more data, both approaches perform equally (outperforming dictionary-based replacements). Overall, data augmentation achieves 34% improvement in perplexity, 5.2% relative improvement on WER for ASR task, +4.0-5.1 BLEU points on MT task, and +2.1-2.2 BLEU points on ST over a baseline trained on available data without augmentation.

## introduction code-switching (cs) is the alternation of language in text or speech. cs can occur at the levels of sentences (inter-sentential cs), words (intrasentential cs/code-mixing), and morphemes (intraword cs/morphological cs). given that cs data is scarce and that collecting such data is expensive and time-consuming, data augmentation serves as a successful solution for alleviating data sparsity. in this paper, we investigate lexical replacements for augmenting cs dialectal arabic-english data. researchers have investigated approaches that do not require parallel data, including translating source words into target language with the use of dictionaries @xcite , machine translation @xcite , and word embeddings @xcite , as well as relying on parallel data and performing substitutions of words/phrases using alignments @xcite @xcite . as will be discussed in section 2, most of the previous studies on this front have focused on one augmentation technique without exploring others, or reported results using only one type of word alignments configuration, or evaluated effectiveness of augmentation on only one downstream task. we attempt to provide a comprehensive study where we systematically explore the use of neuralbased models to decide on cs points for performing replacements using word-aligned parallel corpora versus randomly-chosen cs points, along with the interaction of different alignment configurations. we compare these approaches against dictionary-based replacements. we provide a rigorous evaluation of the different settings, where we assess the quality of the generated cs sentences through human evaluation as well as the impact on language modeling (lm), automatic speech recognition (asr), machine translation (mt), and speech translation (st) tasks. our human evaluation study shows that for the purpose of generating high-quality cs sentences, learning to predict cs points and integrating this information in the augmentation process improves the quality of generated sentences. on the downstream tasks, we report that performing alignmentbased replacement outperforms dictionary-based replacement. for alignment-based replacement, utilizing a predictive model to decide on where cs points should occur as opposed to replacing at random positions both lead to similar results for asr, mt, and st tasks. for both approaches, we investigate different word alignment configurations, and we report that performing segment replacements using symmetrized alignments outperforms word-replacements using intersection alignments on both human evaluation and extrinsic evaluation. we also investigate controlling the amount of generated data, to eliminate the effect of random producing more data over the predictive model. under the constrained condition, using a predictive model outperforms the random approach on the mt task. in this work, we tackle the following research questions (rqs): in the direction of lexical replacements, appicharla et al. (2021) source word is replaced by the aligned word(s). they evaluate their approach against unigram and bigram random replacements, and test its effectiveness on mt task for cs hindi-english. @xcite use data augmentation for mt task for cs spanish-english and french-english. symmetrized alignments are used to identify small aligned phrases (minimal alignment units) and phrase replacements are performed randomly. we also notice that in literature, human evaluation of generated cs data is mainly used to evaluate the synthetic data produced by the best model, rather than comparing different techniques. such a comparison was provided by @xcite , where a large-scale human evaluation was presented comparing different linguistic-driven and lexical replacement techniques. however, the study was focused on human evaluation without exploring the effectiveness of those techniques on downstream tasks. ## data augmentation for generating synthetic cs data, we investigate the use of word-aligned parallel sentences as well as dictionary-based replacements. in the latter approach, monolingual arabic sentences are augmented by replacing words at random locations with their english glossary entry. in the former approach, utilizing monolingual arabic-english parallel corpora, we inject words from the target side to the source side, where replacements are performed at random locations or using a cs point predictive model. as shown in figure 1 , the augmentation process consists of two main steps: (1) cs point prediction: identifying the target words to be borrowed, and (2) cs generation: performing the replacements. in sections 3.1 and 3.2, we will elaborate on the methodology for both steps. examples src . academic life i love i was a junior ta ← tgt and i was a junior ta for a period of time so i have tried this and i love the academic life a bit . output 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 0 0 src city expect ← tgt i wasn 't expecting to see such a city in the first place . output 0 0 0 1 0 0 0 0 1 0 0 0 0 0 ## results in order to evaluate our augmentation techniques, we provide intrinsic evaluation, extrinsic evaluation, as well as human evaluation. 9 according to human evaluation, the synthetic data generated using a cs predictive model is perceived as more natural. however, our extrinsic evaluation shows that both aligned-based approaches (random replacements and relying on a predictive model) perform equally on downstream tasks. we observe that using a predictive model generates less data than the random approach. when controlling for size, we observe that using a predictive model brings improvements on the mt task. both aligned-based approaches outperform dictionary-based replacements on human evaluation and extrinsic evaluation. regarding the effect of word alignment configurations, the improvements of using n-n alignments versus 1-1 alignments is confirmed in both human evaluation and extrinsic evaluation. ## discussion in this section, we revisit our rqs: rq1 -can a model learn to predict cs points using limited amount of cs data? as shown in the intrinsic evaluation, the model learns to predict cs points to some extent, as shown in the improvements in accuracy, precision, and f1 scores over random predictions. this is also observed where the pos distribution of the cs predictions using a predictive model has higher correlation to the distribution found in natural cs sentences compared to random predictions. rq2 -can this information be used to generate more natural synthetic cs data? yes, this was confirmed though human evaluation, where annotators reported higher scores for understandability and naturalness using the predictive model over using random replacements. rq3 -would higher quality of synthesized cs data necessarily reflect in performance improvements in downstream tasks? in the scope of our experiments, such an entailment does not necessarily hold. we believe two limitations are affecting the performance of the predictive model. first of all, the mappred approach is based on the assumption that the data provided to the predictive model is representative enough of the cs phenomenon and includes all cs patterns. due to the scarcity of cs corpora and the dynamic behaviour of @xcite , this point presents a challenge and could be restricting the potential power of this model, and it could be the case that maprand is able to cover more cs patterns. this is supported by the pos distribution analysis in section 5.1. secondly, random has the power of generating more data as opposed to using a predictive model. when we control for size, we observe improvements in mt using the predictive model. in the future, we plan to work on improving the predictive approach to generate more cs sentences. for asr, both approaches perform equally. it was also shown in @xcite that random lexical replacement outperforms the use of equivalence constraint linguistic theorem for asr. therefore, we believe further research is needed to draw strong conclusions about the relation between the quality of generated cs data and the improvements on different downstream tasks. ## conclusion and future work in this paper, we investigate data augmentation for cs egyptian arabic-english. we utilize parallel corpora to perform lexical replacements, where cs points are either selected randomly or based on predictions of a neural-based model that is trained on a limited amount of cs data. we investigate word replacements using intersection alignments as well as segment replacements using symmetrized alignments. we compare both aligned-based replacements with dictionary-based replacements. we evaluate the effectiveness of data augmentation on lm, mt, asr, and st tasks, as well as assess the quality through human evaluation. across all evaluations, we report that segment replacements outperform word replacements, and aligned-based replacements outperform dictionary-based replacements. the human evaluation study shows that utilizing predictive models produces augmented data of highest quality. for the downstream tasks, random and predictive techniques achieve similar results, both outperforming dictionary-based replacements. we observe that random has the advantage of generating more data. when controlling for the amount of generated data, the predictive technique outperforms random on the mt task. our best models achieve 34% improvement in perplexity, 5.2% relative improvement on wer for asr task, +4.0-5.1 bleu points on mt task, and +2.1-2.2 bleu points on st task.