Machine Translation of O mani A rabic Dialect from Social Media (2023)

Abstract: Research studies on Machine Translation (MT) between Modern Standard Arabic (MSA) and English are abundant. However, studies on MT between Omani Arabic (OA) dialects and English are very scarce. This research study focuses on the lack of availability of an Omani dialect parallel dataset, as well as MT of OA to English. The study uses social media data from X (formerly Twitter) to build an authentic parallel text of the Omani dialects. The research presents baseline results on this dataset using Google Translate, Microsoft Translation, and Marian NMT. A taxonomy of the most common linguistic errors is used to analyze the translations made by the NMT systems to provide insights on future improvements. Finally, transfer learning is used to adapt Marian NMT to the Omani dialect, which significantly improved by 9.88 points in the BLEU score.

## introduction in the era of social media and worldwide communication, machine translation (mt) has become essential in lowering or eliminating the language barrier between people @xcite . using artificial intelligence, users can translate any post from any language without human involvement. recently mt underwent a remarkable evolution thanks to deep learning and artificial neural network models @xcite . although mt research attempted to produce high-quality translations of the most widely used languages, which are well documented with abundant sources, it still has a long way to go in terms of languages that are not as well documented, such as arabic dialects. over the past decade, the arabic language has drawn much interest from the mt community. however, most mt contributions focus on modern standard arabic (msa), while the translation of arabic dialects is still in its early stages. arabic is the world's fifth most widely used language, with almost 450 million speakers in 22 countries. classical arabic (ca) and msa are the standard arabic varieties recognized by western linguists. the quran, classical texts, and old arabic literature are written in ca. msa is a modern form that is based on the syntactic, morphological, and phonological structures of ca. msa is the primary form of official communication in the arab world that is used in education, business, news, and legislation @xcite . arabic dialects are used informally in day-to-day conversations throughout the arab world. arabic dialects are primarily spokenonly languages; however, in the last decade, these dialects have become increasingly prevalent in social media, text messages, tv shows, and other forms of informal communication. nowadays, arabic dialects are being used increasingly in written format for informal communication online @xcite . @xcite despite the extensive use of arabic dialects, they are considered low-resource language which hinder mt development. arabic dialects vary from msa in terms of phonology, semantics, morphology, and syntax @xcite . they simplify many standard arabic rules while simultaneously introducing new sets of rules that add additional complications. therefore, most msa resources and tools cannot be easily adapted to translate arabic dialects @xcite . the lack of standard orthography is one of the fundamental challenges associated with arabic dialects. arabic dialects have diglossia, a linguistic phenomenon in which the speakers mix two or more varieties of the same language (e.g., standard official language and local dialect) within the same context @xcite . it is worth noting that arabic has a diverse range of colloquial varieties, with over 27 variations existing worldwide. these varieties exhibit varying degrees of mutual understanding, highlighting arabic's nuanced and diverse nature @xcite . figure 1 provides a basic overview of the geographic distribution of these dialects. there are two primary ways to approach colloquial arabic mt. the first involves translation between msa and colloquial arabic dialects then to foreign language therefore msa acting as an intermediate language. the second approach involves translation of arabic dialects into foreign languages directly @xcite . it is worth noting that all contributions in this field are primarily related to the english language. this research focuses on the mt of omani arabic (oa). oman's location, surrounded by the indian subcontinent, persia, arabia, and east african coasts, played a significant role in shaping its history and the languages spoken by its people. despite oman's small population, its linguistic context is diverse. some omanis speak multiple indigenous languages, such as jibbali, shahri, and mehri, each with thousands of speakers, in addition to bathari, harsusi, and hobyot, with a few hundred speakers each @xcite . additionally, some omanis speak non-indigenous languages, including persian, aajmi, kumzari from iran, baluchi from baluchistan, zidjali from pakistan, kojki/luwati from india, and swahili from east africa @xcite . the impact of various languages on oa is particularly evident in its vocabulary, featuring words borrowed from hindi, such as guniyyah (meaning sack) and bigli (referring to an electric torch), as well as persian words like drishah (window) and saman (stuff). english has also contributed words such as sekal (bicycle), batri/betri (battery), swik (switch), and beb (pipe), while portuguese brings in banderah (flag) and mez (table). the prevalent dialect in oman differs from that dominant in the rest of the arabian gulf. it is mostly in the form of the hadari (sedentary) dialect rather than a bedouin one @xcite . the hadari dialect is prevalent in the northern part of oman, including the capital muscat, and is also used in most tv shows. limited research is available on translating colloquial arabic dialects, particularly omani dialects. while most prior works group oa dialects with other gulf dialects, more research is necessary. it is important to note that while the gulf region may share cultural similarities, it cannot be assumed that they share linguistic homogeneity. moreover, oa datasets used in prior works are not publicly available. this research aims to close this gap by creating an authentic omani arabic-english parallel corpus that is available for public use. the dataset will be used to adapt an existing arabic neural machine translation (nmt) system to the omani dialect. ## related works in this section, we will explore the literature on dialectical datasets and the mt of arabic dialects. ## omani parallel corpus this study aims to translate omani arabic, utilizing original data of language in use on social media posts. x (formerly twitter) was utilized to collect text representatives of the omani dialect. x is a leading social media platform that contains trending news and topics and has a very large user base. therefore, it offers a valuable resource for conducting large-scale text analysis. furthermore, api allows users to execute complex queries, such as retrieving all text related to a particular topic or extracting a specific user's posts. this chapter will explain the full process of creating the omani parallel corpus. after the completion of the corpus, we will present a translation baseline results on this corpus obtained using google translate, microsoft translation, and marian nmt systems. ## error analysis in omani arabic mt the most used measures for translation accuracy are automated. however, it can be difficult to establish a direct correlation between these measures and the actual errors present in the translations. a comprehensive analysis of errors is crucial for any natural language processing task, as it can reveal valuable insights into what went wrong and guide future research directions @xcite . for the error classification, we adopted a simplified version of the taxonomy @xcite shown in figure 2 . error analysis can be a time-consuming task that requires linguistic expertise. therefore, we conducted an analysis of a sample of sixty source sentences to identify errors generated by google translate and marian nmt. out of the 60 sentences translated by both mt engines, google had incorrect translations in 54 cases, while marian had errors in 57. google's total number of translation errors was 112, while marian had 150 errors. in ## transfer learning to implement this approach, we start with a pretrained nmt model that has been trained on a large parallel corpus (msa-en). we then use this model to initialize a new nmt model called the child model. this model is then trained on the domain dataset with a limited parallel data. using the pretrained parent model, the child model commences with established weights inherited from the parent model rather than starting with random weights. this method is particularly useful since the omani parallel corpus is limited, and the msa corpus provides a strong prior distribution over language vocabularies. marian nmt @xcite is used as a neural translation system for the parent model. marian is a highly efficient nmt framework that is built on pure c++, requiring minimal dependencies. the framework was mainly developed by the adam mickiewicz university and the university of edinburgh. it is currently utilized in varmarian has found its niche in the growing world of open-source nmt toolkits due to two key aspects: it is built entirely on c++ which makes it very efficient. additionally, it is self-contained with its own back-end that enables reverse-mode automatic differentiation using dynamic graphs. marian nmt model follows the original transformer architecture with six encoders and six decoders with eight attention heads in each layer @xcite . language technology research group at the university of helsinki, finland, trained marian nmt on many language pairs from the opus-mt datasets. these models have been converted to pytorch foot_1 using the transformers library by hugging face foot_2 . the arabic-english foot_3 translation model was trained with a parallel dataset of 126.6m arabic-english sentence pairs (see the omani parallel corpus was split into a train set, a validation set, and a test set. the training set contains 70% of the whole corpus, and the remaining 30% is divided equally between the validation and test sets. the training was done using seq2seqtraining script from hugging face and activation function is adamw. before jumping to the model results, having a look at the validation and training loss is a good practice to ensure models are generalized and there is no over-fitting. both validation and training loss decreased up until the fifth epoch. however, after the fifth epoch, the validation loss showed a slight increase. hence, we have decided to proceed with five epochs for training. in order to compare the fine-tuned model with various translation systems, we calculated the bleu score for the validation set translated by google, microsoft, marian nmt, and the transfer learning model. in figure 6 , the results indicate that google achieved the highest bleu score of 34.98, but it is noteworthy that microsoft and our model were not far behind, scoring 34.26 and 34.11, respectively. on the other hand, marian nmt scored the lowest with 24.22. fine-tuning marian nmt closed the performance gap between marian and other translation engines (google and microsoft). marian's initial bleu score was 24.22, but after completing transfer learning in the oa training set, it increased significantly to 34.11, representing an improvement of 9.88 points. our results outperformed @xcite , which achieved a maximum improvement of 7.5 points. the training process is significantly influenced by the closeness of the parent model language to the child's model language. although the transfer learning model has displayed positive outcomes in marian nmt, it has yet to surpass the msa-english systems of google and microsoft. it would be advantageous to implement transfer learning in these systems, but they do not offer open-source models. ## conclusion using machine translation (mt) is an effective way to overcome language barriers in communication. while there are numerous research studies on mt from modern standard arabic (msa) to english, there is a significant lack of studies on translating omani dialects. in this study, we aim to establish a first baseline targeted at the automatic translation of the written text of omani dialects from social media. our initial step was to thoroughly analyze the literature on the translation of colloquial arabic dialects and identify the datasets that contained an oa corpus. only one source was available for reseacrh use and it may not be representative of the omani dialect. we collected messages from social media to create an authentic omani dialect corpus. then we translated a total of 2906 messages. this corpus has been used to conduct a baseline study on existing mt models' performance in omani dialect translation, where we found that google and microsoft translation engines got higher bleu scores reaching 33%, compared to marian nmt, which scored 22.3%. we conducted a manual evaluation to identify google and marian nmt errors. after linguistically classifying the errors, we discovered that the most common error made by both nmts was choosing the wrong word sense. we enhanced the translation of oa by utilizing transfer learning with marian nmt. this resulted in a significant improvement of 9.88% in the bleu score. the main contribution of this research can be summarized as follows: we faced challenges because we had limited resources and time constraints. we didn't have the funds or time to hire professional translators for the corpus, and we couldn't review every sentence to select them for translation. additionally, we utilized transfer learning with the available open-source model. in the future, we hope to enhance the translation of the oa corpus by collaborating with linguistic experts, increasing the quantity of translated sentences, and providing multiple translations for each sentence to ensure an accurate evaluation.