Neural Machine Translation for S inhala- E nglish Code-Mixed Text (2021)

Abstract: Code-mixing has become a moving method of communication among multilingual speakers. Most of the social media content of the multilingual societies are written in code-mixed text. However, most of the current translation systems neglect to convert code-mixed texts to a standard language. Most of the user written code-mixed content in social media remains unprocessed due to the unavailability of linguistic resource such as parallel corpus. This paper proposes a Neural Machine Translation(NMT) model to translate the Sinhala-English code-mixed text to the Sinhala language. Due to the limited resources available for Sinhala-English code-mixed(SECM) text, a parallel corpus is created with SECM sentences and Sinhala sentences. Srilankan social media sites contain SECM texts more frequently than the standard languages. The model proposed for code-mixed text translation in this study is a combination of Encoder-Decoder framework with LSTM units and Teachers Forcing Algorithm. The translated sentences from the model are evaluated using BLEU(Bilingual Evaluation Understudy) metric. Our model achieved a remarkable BLEU score for the translation.

## introduction before 1990, translation was considered a difficult task due to many reasons such as ambiguity, translation mismatch, co-reference, translation divergence and development of language over time @xcite but machine translation(mt) since 1990 has been a vast and successful research area in natural language processing. machine translation has been given importance in the research field because it is used to translate texts for military authorities to track enemies, foreign business collaborations, marketing, etc. @xcite . expressing the thoughts of the personal interests, daily life etc., of a person in social media networks has become a trending activity among people. texts extracted from social media lead to measure the social dynamics of several societies @xcite . contentbased search engines, personalized advertisements, recommendation systems, etc. use the user-generated content from social media to increase their value and to provide more accurate results to the users @xcite . processing a content in standard language(without code-mixing) is considered as an easy task, where the text with code-mixing has been considered as a road block to extract the needed information. code-mixing is considered as an invention of bilingualism and multilingualism. the capability of speaking in two languages, is called bilingualism and more than two languages is called multilingualism. most of the srilankans are bilingual. the user-generated content such as posts, comments,reviews etc., in srilankan social media are mostly in secm text. the main focus of this study is to translate the secm text to sinhala language. to understand the necessity of processing secm text, a survey study was conducted among 82 native sinhala speaking citizens as a part of our research study to collect information about the usage of secm text in sri-lanka. figure 1 shows the results of a few essential questions from the survey. the majority of the people have stated that they use secm text in social media rather than their native language. in secm texts there are several problems identified related to morphology, syntax and the semantic structure of the text. the challenges found in secm texts are, • inconsistency in the transliteration • spelling mistakes • code-switching • words combined with suffixes of another language this paper is divided into six sections. section 2 elaborates on the works related to machine translation. section 3 describes the secm texts and corpus creation. the section 4 explains the methodology, which explains about the model implementation and prediction. section 5 describes the experimental setting of the model and the result gained. as the final section the paper discusses the conclusion of the research study. ## related work the high demand for machine translation is increased due to many reasons such as business in overseas, tracking down the information of another language for military services, high usage of social media, etc. machine translation was initiated by warren weaver in 1955. the research combined statistical machine translation(smt) with claude shannon's information theory @xcite . smt models output the degree of similarity between the source and target sentences @xcite . the structure of a sentence, feature engineering and design are considered as valuable factors in smt. also smt approach is noted as 'not suitable' for generalized sentence pair, sentences with hidden details @xcite and language pairs with different word orders @xcite . @xcite and @xcite , showed that features focused in smt are not helpful to track the long-distance dependency of a sentence like in recurrent neural network. @xcite explained, how semantic, synthetic and morphological similarities are captured better when there is continues representation of words that carry taskdependent knowledge. @xcite introduced the recurrent continues translation model, which has two parts, convolutional sentence model and recurrent language model. convolutional sentence model, uses a convolutional n-gram approach on source sentences in the encoder. the sentences are mapped into semantic vectors. the recurrent language model is applied in the decoder. a similar approach is proposed by @xcite , where the recurrent neural network(rnn) is used for translation. sentences needing translation are encoded into a sequence with a fixed length vector and decoded with another sequence of symbols. in this approach, the encoder and the decoder are jointly trained to increase the conditional probability of phrase pairs in the sentence using rnn. some research studies on translation are based on a monolingual dataset. a semisupervised approach is proposed by @xcite with labelled and unlabeled corpus. labelled corpus is a parallel corpus with source and target sentences of the chinese-english dataset. unlabeled corpus contains the monolingual dataset. in the semi-supervised setting, parallel and monolingual corpus are joined to learn bidirectional nmt(source to target and target to source models). @xcite proposed two approaches to translate monolingual datasets. in the first approach the monolingual corpus is matched with dummy inputs to construct the parameters of the encoder with attention model @xcite and the second approach utilizes a pre-trained nmt model. using nmt for translating one standard language to another standard language has been a success over the decade @xcite . however, it is not experimented well with the code-mixed text due the to lack of resources. in the domain of translating code-mixed text, very few researches have been carried on. a combined approach of statistical modelling @xcite and knowledge translation @xcite approach is introduced by @xcite for cross-language social media texts. @xcite introduce an approach for translating codemixed text, where words in sentences are categorized as dominant and non-dominant languages. words from dominant language are labelled as matrix language and non-dominant language are labelled as embedded language. the first task before the translation was wordlevel language identification. next, the data is applied to a current translator to translate the words to another language. dhar et al. (2018 highest bleu score of 22.46. ## secm text and corpus creation sinhala is the native language of the majority of the people in sri lanka. but in srilankan social media, sinhala-english code-mixed text is used frequently because of the multi-lingual users. in 2001 international organization for standardization published the iso15919 standard. it is an international standard for romanization which includes many languages including sinhala language . @xcite used the ipa(international phonetic alphabet) format to represent sinhala letters in their research study. iso15919 and ipa both present sinhala letters with english alphabets, which is called transliterated format or romanized text format @xcite . @xcite propose a conventional tag set, which uses the 26 alphabets of english to present the phonetic sound of sinhala letters using the festival framework. code mixing is described as a way of writing roman script @xcite . even though the standard tag sets from @xcite is defined as roman representation, the romanization of actual code-mixing used by multilingual societies is different from the standard tag sets. figure 3 and figure 4 shows us how the standard romanization defined for sinhala letters differs from the roman representation used in singlish text. according to figure 3 and figure 4 , phonetic tagset(pt) and romanized tagset(rt) are almost similar. but there is a huge difference between these two tagset and code-mixed text representations of sinhala letters. due to no consistency in the pattern of singlish text, it is not easy to translate the sinhala code-mixed text without a parallel corpus. to achieve a good outcome most machine translation systems needs a sufficient amount of parallel sentences in the corpus. we collected the secm sentences from public facebook posts, comments and reviews. for the translation of secm to sinhala, implementing the parallel corpus is an important part for our research study. each secm sentence was human translated by a linguistic expert of sinhala language to create the parallel corpus. in the parallel corpus secm is considered the source sentence and sinhala is considered the target sentence. the human translator was advised to follow the singlish to sinhala mapping provided in the research study of @xcite as the guide to maintain the consistency in the translation. the corpus contains around 1500 parallel sentences of secm and it's translated sinhala sentences. after the human translation the dataset was checked to see whether the translated sentences are fc (fully correct) or cr (correction required). if a parallel sentence was annotated with a correction required tag by the annotator, the same annotator would provide the alternate translation as well. each sentence in the corpus is annotated by two annotators. the annotators are people whose native language is sinhala and we made sure that they are fluent in the sinhala language. the annotators were provided with guidelines regarding the annotation process. the guidelines made sure that the annotators were checking whether there are any figure 4: difference between standard phonetic tags, romanized tags and singlish representation for sinhala vowels spelling mistakes, grammatical issues in the sinhala translation. as shown in figure 5 , when there is a sentence tagged with two different tags, for example one annotator annotated with fc tag and the other annotated with cr tag with an alternate translation, we considered the alternate translation. also when a sentence is annotated with cr tag from both the annotators, the alternate translations provided by both the reviewers are checked with a third annotator and the most suitable translation is selected. to evaluate the standard of the corpus, 100 randomly chosen sentences are provided to 3 experts of sinhala for ranking. the translations are ranked as good or bad, according to the meaningful translation, grammatical pattern and spelling errors. fleiss' kappa approach is used to calculate the reliability of the agreement between the raters @xcite . the overall fleiss' kappa score received for the translation is 0.88. ## methodology after corpus creation, the dataset was applied with several pre-processing steps. initially, all the sentences in the singlish corpus are converted into lower case and all the sinhala sentences in the target corpus are added with start and end tokens. the unique words from the corpus are extracted and unique numbers are allocated for each word according to the order of frequency of each word. these word-number arrays are called wordtoindex dictionaries. encoder-decoder framework is used as the base to initiate our model. encoder and decoder can be considered as two separate recurrent neural networks. in the encoder the secm sentence is fed as input. it produces the sentence with fixed-sized representation by encoding. each word from the input sentence provided into the encoder would be mapped into an integer using the wordtoindex dictionary and converted into one-hot encoding. the embedding layer maps the one-hot encoded representation into a smaller dimension. the word embedding would be the input to the next layer with long short term memory(lstm) as the basic unit. in lstm we have a cell state that is passed with each timestep. the lstm unit @xcite determines to neglect some unnecessary information and add some new information from the input fed to the current timestep. the significant information collected from the encoder would be passed into a context vector with the output and hidden states. only hidden states are passed as input to the decoder. decoder produces the target sentence using the significant information passed through the hidden state from the encoder and the input target word. each word from the target sentence is mapped into an integer using wordfigure 6: architecture of each timestep in encoder and decoder figure 7: encoder -decoder framework toindex dictionary and converted to one-hot encoding. the word embedding layer maps the embedding into a continuous representation which has a lot smaller dimension. figure 6 shows the architecture inside timesteps in encoder and decoder. teacher forcing algorithm @xcite is added in the decoder. teacher forcing algorithm inputs the expected output of previous timestep t-1 to the current timestep t. the advantage of using teacher forcing mechanism is the hidden state of the model would be updated with the correct expected outputs rather than the wrongly predicted output from the previous timestep. if the predicted output from previous timestep t-1 is fed to the next timestep t, the number of errors would be increased and the model would face difficulty in learning. combination of encoder and decoder is called as sequence to sequence model(seq2seq) as shown in figure 7 . final phase of the proposed architecture of the system is prediction. singlish sentence from the corpus is given as input, and output would be the predicted sinhala sentence. prediction phase is built with the sequence to sequence architecture. each timestep in decoder passes predicted output for the next timestep unlike the decoder in the training phase of the model. ## experimental setting and result from the corpus 70% of the data is allocated for training, and 30% of the data is allocated for testing. inputs for the encoder and the decoder are in the shape of 2d arrays. the shape of the encoder array is (10,27), where the batch size is ten and maximum length of source sentence is twenty seven. shape of the decoder array is (10,26), where the batch size is ten and maximum length of source sentence is twenty six. rmsprop is used as the optimizer and categorical cross entropy is used to calculate the loss. the rmsprop optimizer is used because it balances the step size and, decreases the no of steps for massive gradients to neglect the exploding and increases the number of steps for small gradient to avoid vanishing gradients issue. weights calculated after the training phase of the model are saved for the prediction phase. the model reached the training accuracy of 71.42% and testing accuracy of 37.17%. after the model's training, randomly selected hundred secm sentences from the corpus are inputted to predict the translated sinhala sentence. the predicted sinhala sentences are saved to calculate the bleu score. figure 8 shows examples of some predicted sinhala translations. bleu score is the evaluation metrics @xcite used to evaluate the translated sentences. bleu metric provides a score for the translation based on the predicted sentence and relevant reference sentence. for each sentence in the corpus, modified precision of unigram, bigram, trigram and four-gram are calculated. the weight of 0.25 has been given to each modified precision. equation 1, is used to calculate the bleu score. n is the number of n-grams and w n is the weight for each modified precision, p n is modified precision. bp is the brevity penalty to penalize short machine translations @xcite . the value of bp is decided according to the values of c and r. c is the number of unigrams in all the predicted sentences, and r is the best match length for each predicted sentence in the corpus. our model received a cumulative bleu4 score of 31.54. comparing to previous models proposed for code-mixed text translation @xcite our proposed approach with teacher forcing mechanism gives a remarkable bleu score for the translation. ## conclusion & future work this paper presents a deep analysis of sinhala-english code mixed texts. the difference between standard tagsets available for the romanization of sinhala letters and the romanization used in secm text are compared. the differences are discussed in this study. challenges in the pattern of secm sentences such as code-switching, spelling errors, improper usage of discourse marker etc., are also discussed in this paper. a parallel corpus is created containing secm sentences and the relevant sinhala sentences translated by a human translator who is a linguistic expert. the corpus is validated using annotators who are native sinhala language speakers. the parallel corpus introduced in this paper can be considered a useful resource for researches based on secm text. we combined teacher forcing algorithm with the sequence to sequence approach with lstm units to translate secm sentences to sinhala sentences. teacher forcing algorithm updates the hidden state of each timestep in the decoder with the expected output from the previous time step, which leads on providing more accurate results for the translation. the bleu score received for our model revealed that comparing the state of the art of other translation models for code-mixed texts, our model achieved significantly higher bleu score. the future work we would like extend this research to focus on sentiment analysis and entity extraction using the parallel corpus created in this research study.