A Semi-supervised Approach to Generate the Code-Mixed Text using Pre-trained Encoder and Transfer Learning (2020)

Abstract: Code-mixing, the interleaving of two or more languages within a sentence or discourse is ubiquitous in multilingual societies. The lack of code-mixed training data is one of the major concerns for the development of end-to-end neural network-based models to be deployed for a variety of natural language processing (NLP) applications. A potential solution is to either manually create or crowd-source the code-mixed labelled data for the task at hand, but that requires much human efforts and often not feasible because of the language specific diversity in the code-mixed text. To circumvent the data scarcity issue, we propose an effective deep learning approach for automatically generating the code-mixed text from English to multiple languages without any parallel data. In order to train the neural network, we create synthetic code-mixed texts from the available parallel corpus by modelling various linguistic properties of code-mixing. Our codemixed text generator is built upon the encoder-decoder framework, where the encoder is augmented with the linguistic and task-agnostic features obtained from the transformer based language model. We also transfer the knowledge from a neural machine translation (NMT) to warm-start the training of code-mixed generator. Experimental results and in-depth analysis show the effectiveness of our proposed code-mixed text generation on eight diverse language pairs.

## introduction multilingual content is very prominent on social media handles, especially in the multilingual communities like the indian ones. code-mixing is a common expression of multilingualism in informal text and speech, where there is a switch between the two languages, frequently with one in the character set of the other language. this has been a mean of communication in a multi-cultural and multi-lingual society, and varies according to the culture, beliefs, and moral values of the respective communities. linguists have studied the phenomenon of codemixing, put forward many linguistic hypotheses @xcite @xcite , and formulated various constraints @xcite @xcite to define a general rule for code-mixing. however, for all the scenarios of code-mixing, particularly for the syntactically divergent languages @xcite , these limitations cannot be postulated as a universal rule. in recent times, the pre-trained language model based architectures @xcite have become the state-of the-art models for language understanding and generation. the underlying data to train such models comes from the huge amount of corpus, available in the form of wikipedia, book corpus etc. although, these are readily available in various languages, there is a scarcity of such amount of data in code-mixed form which could be used to train the state-of-the-art transformer @xcite based language model, such as bert @xcite , roberta @xcite , xlm @xcite etc. the existing benchmark datasets on various nlp tasks can also be transformed to the code-mixed environmental setup, and subsequently be leveraged to assess the model's flexibility under the multilingual framework. creating large-scale codemixed datasets for such tasks is expensive and time-consuming as it requires considerable human efforts and language expertise to generate these manually. therefore, it is necessary to build an automated code-mixed generation system capable of modeling intra-sentential language phenomenon. in this paper, we formulate the code-mixed phenomenon using the feature-rich and pre-trained lan-guage model assisted encoder-decoder paradigm. the feature-rich encoder assists the model to capture the linguistic phenomenon of code-mixing, especially to decide when to switch between the two languages. similarly, the pre-trained language model provides the task-agnostic feature which helps to encode the generic features. we adopt the gating mechanism to fuse the features of the pretrained language model and the encoder. additionally we also perform transfer learning to learn the prior distribution from the pre-trained nmt. the pre-trained nmt weights are used to initialize the code-mixed generation network. transfer learning guides the code-mixed generator to generate syntactically correct and fluent sentences. we summarize the contributions of our work below: (i). we propose a robust and generic method for code-mixed text generation. our method exploits the capabilities of linguistic feature-rich encoding and pre-trained language model assisted encoder to capture the code-mixed formation across the languages. our model is further tailored to generate the syntactically correct, adequate and fluent code-mixed sentences using the prior knowledge acquired by the transfer learning approach. (ii). to warm start the training, we devise a robust and generic technique to automatically create the synthetic code-mixed sentences by modeling the linguistic properties using the parallel corpus. to the best of our knowledge, this is the very first step where we attempt to propose a generic method that produces the correct and fluent code-mixed sentences on multiple language pairs. the generated synthetic dataset will be a useful resource for machine translation and multilingual applications. (iii). we demonstrate with detailed empirical evaluations the effectiveness of our proposed approach on eight different language pairs, viz. english-hindi (en-hi), english-bengali (en-bn), english-malayalam (en-ml), english-tamil (enta), english-telugu (en-te), english-french (en-fr), english-german (en-de) and english-spanish (enes). ## related work in the literature, there have been efforts for creating code-mixed texts by leveraging the linguistic properties. @xcite explored the equivalence constraint theory to construct artificial code-mixed data to reduce the perplexity of the rnn-based language model. @xcite proposed a multitask learning framework to address the issue of data scarcity in code-mixed setting. particularly, they leveraged the linguistic information using a shared syntax representation, jointly learned over part-of-speech (pos) and language modeling on codeswitched utterances. @xcite exploited seqgan in the generation of the synthetic codemixed language sequences. most recently, winata et al. (2019a) utilized the language-agnostic metarepresentation method to represent the code-mixed sentences. there are also other studies @xcite @xcite @xcite for code-mixed language modelling. there are some other nlp areas like parts-ofspeech @xcite @xcite , sentiment analysis @xcite , question answering @xcite , language identification @xcite @xcite , entity extraction @xcite @xcite , etc, where codemixing phenomena are explored and analyzed. in contrast to sthese existing works, firstly, we provide a linguistically motivated technique to create the code-mixed datasets from multiple languages with the help of parallel corpus (english to respective language). thereafter, we utilize this data to develop a neural based model to generate the code-mixed sentences from the english sentence. our current work has a wider scope as the underlying architecture can be used to harvest the code-mixed data for the various nlp tasks not only limited to the language modelling and speech recognition as it is generally been focused in the literature. in contrast to the previous studies, where only a few of the language pairs were considered for code-mixing, we propose an effective approach which shows its effectiveness in generating codemixed sentences for eight different language pairs of diverse origins and linguistic properties. ## synthetic code-mixed generation we follow the matrix language frame (mlf) @xcite theory to generate the code-mixed text. it is less restrictive and can easily be applied on many language pairs. according to mlf, a code-mixed text will have a language (l1) language (l2) code-mixed (l1-l2) en india's agriculture is their main strength. hi भारत क कृ ष उनक मु य ताकत है । india's कृ ष इसक main strength है । en especially valuable people like connor rooney. bn িবেশষত কনর িনর মেতা মূ ল বান ব ি । িবেশষ connor rooney মেতা valuable ব ি । en glasses and cups, whatever they are, can be turned upside down. ta க ணா க ம ேகா ைபக , அைவ எ வாக இ தா , தைல ழாக மா றலா . glasses ம cups அைவ எ வாக இ தா , தைல ழாக மா றலா en democracy and development go hand in hand. de demokratie und entwicklung gehen hand in hand. democracy und development gehen hand in hand. en we abolish national embassies. fr nous abolissons les ambassades nationales. nous abolissons les embassies national. we use the official implementation foot_2 of the fastalign algorithm @xcite to obtain the alignment matrix. the alignment matrix is used to construct the aligned phrases between the parallel sentences. we extract the pos (mainly adjective), named entity (ne) and noun phrase (np) from the english sentences, and insert them into the appropriate places of the sentences in the other language (i.e. the target language) counterparts. we use the stanford library foot_3 stanza @xcite to extract these linguistic features. we can extract multiple aligned phrases from the alignment matrix. however, in our proposed algorithm, we are interested in aligned words/phrases which are the nes of types 'person', 'location' and 'organization', noun phrases and adjective words. let us see an example of en-hi parallel sentence: the ne mahatma gandhi of type 'person' is mixed in en-hi foot_4 code-mixed sentence. the need of replacing the aligned noun phrases can be understood with the examples of parallel sentences shown in fig 1. in the given example, 'girl' and 'red umbrella' are the noun phrases foot_5 in the english sentence. to obtain the corresponding code-mixed sentence, their aligned phrases `लड़क ' and `लाल छाता' need to be replaced with english counterparts 'girl' and 'red umbrella', respectively. similarly, we can visualize the requirement of choosing the adjective words to be mixed in the code-mixed sentence by the following example: in the given example the adjective 'normal' is present in the english sentence. to make the corresponding code-mixed sentence the adjective word has to be inserted in the code-mixed sentence. in this case corresponding target (i.e. hindi here) word `सामा य' need to be replaced with the word 'normal' in the en-hi code-mixed sentence. we show some samples in ## methodology we depict the architecture of our proposed model in figure 2 . problem statement: given an english sentence e having m words e 1 , e 2 , . . . , e m , the task is to generate the code-mixed sentence ĉ having a sequence of n words ĉ = {y 1 , y 2 , . . . , y n }. ## results and analysis we evaluate the performance of our proposed approach on the synthetic code-mixed text from eight different language pairs. the datasets can be found here 8 . we compare the performance of our proposed code-mixed generation model with the (i) seq2seq @xcite , (ii) attentive-seq2seq @xcite and (iii) pointer generator @xcite baselines. ## conclusion in this paper, we have proposed a neural network based effective method coupled with the linguistic and pre-trained feature representation along with the transfer learning to generate the code-mixed sentences. to train and evaluate the proposed approach, we have introduced a linguistically motivated approach for code-mixed sentence generation using the parallel sentences of any particular language pair. our experimental results and indepth analysis show that the feature representation and transfer learning together effectively improve the model performance and the quality of the generated code-mixed sentence. we have shown the effectiveness of the proposed approach on eight different language pairs. in future work, we plan to explore the unsupervised neural approach for codemixed text generation.