Gui at M ix MT 2022 : E nglish- H inglish : An MT Approach for Translation of Code Mixed Data (2022)

Abstract: Code-mixed machine translation has become an important task in multilingual communities and extending the task of machine translation to code mixed data has become a common task for these languages. In the shared tasks of EMNLP 2022, we try to tackle the same for both English + Hindi to Hinglish and Hinglish to English. The first task dealt with both Roman and Devanagari script as we had monolingual data in both English and Hindi whereas the second task only had data in Roman script. To our knowledge, we achieved one of the top ROUGE-L and WER scores for the first task of Monolingual to Code-Mixed machine translation. In this paper, we discuss the use of mBART with some special pre-processing and post-processing (transliteration from Devanagari to Roman) for the first task in detail and the experiments that we performed for the second task of translating code-mixed Hinglish to monolingual English.

## introduction code mixing occurs when a multi-lingual individual uses two or more languages while communicating with others. it is the most natural form of conversation for multilinguals. it is often confused with code-switching but there is a slight difference between the two. both these phenomena include communicating in multiple languages but code switching usually takes place within multiple sentences while code mixing usually refers to words of different languages used in the same sentence. in code mixing, phrases, words and morphemes of one language may be embedded within an utterance of another language. code mixing is extensively observed on social media sites like facebook and twitter. with the rapid growth of social media and consequently, increase in the use of code-mixed data, it becomes important to develop systems to process such text. machine translation, also known as automated translation, is the process where a software translates text from one language to another without any human involvement. there are multiple forms of machine translation, however, over the past few years, neural machine translation has become extremely popular. the wmt shared task had two subtasks. the first subtask consisted of the translation of hindi-english parallel sentence pairs to hindi-english code mixed sentences through machine translation. the second subtask consisted of the translation of hindi-english code mixed sentences to english. ## background while there is a growing interest in code-mixed text analysis as a research problem, there is one bottleneck that has hindered the growth of such works, and that is the lack of data. due to this, there aren't many robust models for code-mixed text. to build standardized datasets of code-mixed text, we need to come up with ways of text generation of these code-mixed texts. these texts would be very helpful in training language models for various code-mixed pairs as language models only need unsupervised data. code mixed text generation is a relatively new problem, and so is its initial stage. one of the recent works in this field @xcite tried to use linguistic theories to synthetically build codemixed text using parallel monolingual corpora of two languages. the equivalence constraint theory @xcite says that code-mixing can only occur at parts of the text where the surface structures of two languages map onto each other. so in these parts, the grammatical rules of both languages are followed. the matrix language theory @xcite tries to solve this problem by separating the two languages into a base language and a second language. the grammatical rules of the base lan-guage are followed, and parts of the base language are replaced by the corresponding parts of the second language whenever it is grammatically feasible to do so. deep learning and neural networks have also been used to build systems for code mixed generation. in these systems, the problem of text generation has been posed as one of machine translation, where monolingual text is translated to code-mixed text. some of the early work involved using the then state of the art encoder-decoder models like pointer generator networks @xcite and gans @xcite to translate two sets of monolingual corpora into code mixed text. with the rise of multilingual models like mt5 @xcite , mbart, indicbart @xcite , etc. the task of translation has become much easier as these models understand both languages and this has been shown to outperform previous models in many workshops. mbart @xcite ) is a denoising autoencoder which has been trained on a very large dataset which contains text from 25 languages. it has the same transformer based architecture and training objective as bart, a denoising autoencoder which was shown to be one of the best performing sequence to sequence models at the time. it has been trained to reconstruct original text which has been corrupted as a way to add noise. it can perform various downstream sequence to sequence tasks like machine translation, text summarization, etc. mbart consists of 12 encoder layers and 12 decoder layers. there are 16 heads and a model dimension of 1024. another solution to circumvent the data problem is to create translation systems that can translate code-mixed text to monolingual text. this allows us to use robust nlp systems for various downstream tasks. while we have the above said top performing models at the moment, they are very heavy computational wise due to their large parameter sizes. with resource constraints, it is tough to replicate their performance. helsinki's opus-mt (tiedemann and thottingal, 2020) model was of comparabaly smaller size and focused on the initiative of supporting low-resource languages. it does accordingly have lower performance. we have attempted at utilizing this model in our case with further training on provided data to understand whether under the resource constraints, we can observe competitive data length synthetic (train) 3263 synthetic (validate) 396 human generated (train) 1800 human generated (validate) 376 ## experimental setup in task 1, we use the fairseq implementation of mbart as our base model which has been trained on 4 nvidia geforce rtx 2080 ti gpus. the model has been trained using label smoothed cross entropy as the loss criterion. the model uses an adam optimizer with polynomial decay learning rate scheduling, @xmath0.3, learning @xmath1, ϵ = 10 -6 , β 1 = 0.9 and β 2 = 0.98. the model was trained on 10000 steps with 2500 warm up steps and a batch size of 512 tokens. we validate the model on each epoch on a validation set and at the end we select the model with the lowest loss. in task2, for fine-tuning we use the salesken.ai's pretrained model provided on huggingface transformers. the model was trained on nvidia geforce rtx 2080 ti gpus.the model has been trained using label smoothed cross entropy as the loss criterion. the model uses an adam optimizer with learning @xmath2, ϵ = 10 -9 , β 1 = 0.9 and β 2 = 0.98. ## results and evaluation the test dataset consisted of 500 sentences. these sentences also had both english sentence and its corresponding hindi sentence. rouge-l score and wer score was considered for evaluation. rouge-l score considers longest common subsequence for its scoring. it counts the longest subsequence which is shared between both reference and the output. its different from precision as it only counts the ratio between longest subsequence matched and the number of words matched. it does not take all the words in the reference. the wer score represents the word error rate. total errors between the reference and output is considered for this score. it adds up all the substitution, addition and deletion required to convert the output to the reference sentence and treat it as total error of the output. it can be treated same as calculating levenshtein distance. so our aim was to maximise rouge-l score and minimize wer score. our score improved as we translitered the output from devanagari to roman using indicate library. the score increase significantly after we created a dictionary of words for transliterating most common hindi words and numbers. we achieved a rouge-l score of 0.61667 and wer score of 0.63342 after both the post processing steps. the test set provided for task 2 contained 1500 lines, which were processed as mentioned in 5 the results for the evaluation metrics we obtained for the test set provided for task 2 is available in 3. using the google transliterate api significantly improved the quality of the input data, and also the similarity of vocabulary with the dataset as mentioned earlier. the application of heuristics also bolstered the approach's performance. based on qualitative evaluation, it was observed that it struggled to get long sentence translations which can be attributed to the source of the dataset being of of tweets which have a noisy and inconsistent structure. this is alongside the lower parameter size and attention heads. ## conclusion in this paper, we approached code mixed machine translation problem from both the direction. we used mbart for our first task of translating english and corresponding hindi sentences to hinglish sentence. the results were significantly improved through transliterating the output from devanagari script to roman script. two different methods were used for the same. our model surpassed baseline in rouge-l and wer scores by a huge margin. for the second task of translating hinglish sentences to english sentence by fine-tuning salesken.ai's pre trained model. we cleared the baseline but their is still work to be done in that field as we think that it can be further improved. for the future work in this area we would like to work further on the second task in hand of translating codemixed language to a monolingual language. we need to retrieve information about both the languages from the code mixed sentence and try to give a output in a mono lingual langauge without disturbing the word order.