Low-resource Machine Translation for Code-switched K azakh- R ussian Language Pair (2025)

Abstract: Machine translation for low-resource language pairs is a challenging task. This task could become extremely difficult once a speaker uses code switching. We present the first code-switching Kazakh-Russian parallel corpus.Additionally, we propose a method to build a machine translation model for code-switched Kazakh-Russian language pair with no labeled data. Our method is basing on generation of synthetic data. This method results in a model beating an existing commercial system by human evaluation.

## introduction code-switching presents a significant challenge in natural language processing due to its unpredictability, variability, and the lack of available corpora, especially for low-resource languages. there were no publicly available codeswitched kazakh-russian parallel dataset, thus we present one in this work. the sample from the dataset is presented in tab. 1. this dataset contains only 618 parallel sentences, so it can be used only for evaluation and not for training. we propose a method for training a machine translation model for code-switching task. in our method we use several publicly available kazakh-russian datasets, but since these datasets do not address code-switching problem, we generate additional training data by translating relevant monolingual corpus and show the effectiveness of this approach. we augment the data to address challenge of codeswitching. to do so we developed a novel text transformation method based on simalign @xcite . we train several machine translation models on the augmented dataset resulting in 3.09 likert score for the best baseline model, while yandex commercial model shows 2.80 likert score. these experimental results suggest that our method is able to improve the performance of machine translation systems on real code-switching data and jump start for those language pairs that do not have collected code-switched data. the following paper is structured as follows: section 2 describes the work on code-switching done for other language pairs alongside with studies devoted to russian-kazakh language pair; section 3 presents the description of the existing public datasets for the mentioned language pair and the description of a newly introduced dataset with code-switching phenomenon captured; section 4 contains the details regarding our proposed augmentation method; section 5 describes the baselines, their training process, and the achieved results, while section 9 concludes the paper. the contribution of this work is three-fold: (i) we present the first kazakh-russian codeswitching dataset; foot_0 (ii) we present an evaluation of the existing models on this dataset; @xcite we propose a novel data augmentation for not codeswitched datasets, which allowed us to fine-tune the existing open models achieving almost on par performance with an available commercial system. ## related work recent progress in nlp has spurred the development of technologies capable of handling code-switched data. despite the initiation of code-switching research several years ago, progress within the research community has been sluggish. the primary challenge to address this issue arises from the insufficient availability of data @xcite . a limited number of languages, such as spanish-english original казахстанский гендерлiк теңдiк саласындағы 12 халықаралық құжаттарды бекiттi . фискал көзқарастан гөрi либералдандыру жақсы corrected қазақстан гендерлiк теңдiк саласындағы 12 халықаралық құжаттарды бекiттi . фискалдық көзқарастан гөрi либералдандыру жақсы russian казахстаном ратифицировано 12 международных документов в сфере гендерного равенства . лучше либерализация , чем фискальный подход augmented қазақстан гендерного теңдiк саласындағы 12 халықаралық құжаттарды бекiттi . ## datasets training datasets consist of a dataset collected by nazarbayev university and described in @xcite , we refer to this dataset as nu below; a dataset collected by al farabi university and described in @xcite (kaznu); translated domain adaptation dataset, which is based on russian tweet corpus described in (рубцова, 2012) (rtc). we provide more details on domain adaptation in section 8. these three datasets are the main sources of training data, in addition we use several smaller datasets. to acquire these datasets we used mtdata tool described in @xcite . we combine all the datasets in a single one and apply deduplication. we call this dataset "all data" below. we provide the statistics for all the training datasets in appendix b. evaluation dataset we use kazakh-russian code-switching dataset (krcs) as our evaluation dataset. the krcs dataset consists of 618 colloquial kazakh sentences from social media which include some russian phrases with corresponding ground truth translations to grammatically correct kazakh and russian labeled by annotators. we had two annotators, both of them were natively bilingual in kazakh and russian, both of them are working in academia. the annotation were done as part of their academic duties. number of sentences 618 # in an original kazakh sentence 11.95 russian # in an original sentence 2.77 # in a corrected kazakh sentence 12.27 # in a russian sentence 13.64 the descriptive statistics of the collected corpus is provided in tab. 2. in tab. 1 we provide a sample from krcs dataset. ## dataset augmentation code-switching emulation method in the previous section we described the training datasets, nevertheless we need to state clearly that that datasets are not consider codeswitching phenomenon and thus cannot be used effectively in our setup. therefore we decided to make code-switching data artificially, using specific techniques for data augmentation. first, we prepare the data. for it we follow the m2m100 recipe provided in fairseq repository which is an official implementation of @xcite . namely, we filter out sentences with more than 50% of punctuation, remove the duplicates, and discard sentences with more than 50% of symbols that are not common for a given language. next, we take kazakh processed sentences and augment them. we chose cs-5 method for augmentation: replace a kazakh word with a russian word aligned using simalign @xcite . preliminary, we tried several augmentation techniques, their description and evaluation can be found in section 7. for cs-5 minimal aligned units (mau) are extracted following an approach described in @xcite : the small billingual phrase pairs (a, b) extracted from symmetrical alignment such that for every word in a there exists a link to word in b and vise versa. next, we replace 15% of tokens/maus in the kazakh sentence at random foot_1 . sentences with length of less than 7 tokens have one replacement following @xcite . we provide a sample of augmented sentence in tab. 1. we also provide additional linguistic analysis and justification for each method in appendix d. ## evaluation baselines there are several baselines which are used in our experiments. we use identity baseline, which simply copying its input to the output. this baseline is obviously not trained. there are two trained from scratch baselines, namely, the first one is transformer-600, which is described below. the architecture of the model follows nllb one, specifically the 600m parameters variant. the details of implementation can be found in appendix a. the second trained from scratch baseline is a reproduced approach from @xcite . we call this baseline transformer-nu. the next three baselines are using pretrained machine translation models and finetune them on our training data. these baselines are mbart, a model family described in @xcite , we use specifically mbart-large-50-many-to-many-mmt variant; m2m100, a model family described in @xcite , specifically facebook/m2m100_1.2b; and nllb-600, a model family described in (costa-jussà et al., 2022), specifically facebook/nllb-200-distilled-600m. the last fine-tuned baseline is nllb-3.3b from the same model family as the previous one, but it is facebook/nllb-200-3.3b variant. we do not fully fine-tune this model, instead we use pissa @xcite , a peft approach. metrics in our work we are using three standard metrics: bleu score @xcite , which is basically a token accuracy; chrf++ score @xcite , which is character level f-score; and comet score @xcite , which is a transformer-based model trained to compare translations. for the last metric we use specifically unbabel/wmt22cometkiwi-da model, described in @xcite . ## results for this evaluation we use all the baselines with cs-5 augmentation, since it is the best in our setup as it shown in previous section. in ## augmentation study we experiment with 5 augmentation types, namely: cs-1: replace a kazakh word with a russian one in normal form; cs-2: replace a kazakh word with a russian one's stem with kazakh ending, extracted from a kazakh word by excluding stem from it; cs-3: replace a kazakh word with a russian one in random form; cs-4: replace a kazakh word with a russian word aligned using fastalign @xcite ; cs-5: replace a kazakh word with a russian word aligned using simalign @xcite . for cs-1, cs-2, and cs-3 we employ a publicly available kazakh-russian dictionary from work (rakhimova, 2020). for cs-4 minimal aligned units are extracted as for cs-5. for all augmentation methods, the replacement is done as for cs-5. we provide samples for all the augmentation types in tab. 5. ## domain adaptation as one can conclude from section 3, there is a domain mismatch for the available training data and collected evaluation data. we provide a visualization of this mismatch in fig. 1 . it is a tsne projection of labse embeddings @xcite of the kazakh sentences from the training datasets and russian sentences from russian tweet corpus. one can see that centroid of russian tweet corpus is closer to the centroid of krcs dataset than any other one of another dataset. this observation drove us to conclusion that we might need a domain adaptation. since russian tweet corpus is a monolingual russian language dataset, we translated it to kazakh using publicly available machine translation model nllb-200-distilled-600m from nllb model family described in @xcite . our choice of the model was driven by the fact that it shows the best quality in standard russian-kazakh translation. ## conlusion in conclusion, the proposed method demonstrates a viable approach to tackling machine translation challenges for low-resource, codeswitched language pairs, specifically kazakh-russian. by utilizing synthetic data generation, the method circumvents the need for labeled training data, which is typically scarce for such language pairs. furthermore, the introduction of the first code-switching kazakh-russian parallel corpus represents a significant contribution to the field, providing a valuable resource for future research and development. the empirical results indicate that the system's performance surpasses that of an existing commercial translation system, as evidenced by superior human evaluation outcomes. this highlights the effectiveness and potential of the proposed approach for improving machine translation in similar low-resource, code-switched contexts. ## limitations synthetic data dependence: the approach relies heavily on the generation of synthetic data, which may not perfectly capture the nuances and complexities of natural code-switching in kazakh-russian speech. evaluation scope: while achieving a bleu score of 16.48 is promising, the evaluation is limited to specific criteria and doesn't necessarily account for all aspects of translation quality, such as fluency and contextual accuracy. corpus size and diversity: the newly presented code-switching kazakh-russian parallel corpus may still be limited in size and diversity, potentially impacting the generalizability of the model to broader linguistic contexts or different dialects. commercial system comparison: the performance comparison to an existing commercial system is based on certain benchmarks and human evaluations, which might not cover all practical use cases and scenarios where the commercial system might excel. scalability and adaptability: the method's scalability to other low-resource, code-switched language pairs is not addressed, raising questions about its broader applicability and adaptability to different linguistic environments. long-term sustainability: there is no discussion on the long-term sustainability and maintenance of the synthetic data generation process and how it might evolve with changes in the language pair dynamics or increased data availability. by acknowledging these limitations, future research can focus on addressing these gaps to further enhance the robustness and applicability of machine translation models for codeswitched languages. sholpan k zharkynbekova and valeria e chernyavskaya. 2022. kazakh-russian bilingual practice: code-mixing as a resource in communicative interaction. rudn journal of language studies, semiotics and semantics, 13(2):468-482. ю рубцова. 2012. автоматическое построение и анализ корпуса коротких текстов (постов микроблогов) для задачи разработки и тренировки тонового классификатора. инженерия знаний и технологии семантического веба, 1:109-116.