CONFLATOR : Incorporating Switching Point based Rotatory Positional Encodings for Code-Mixed Language Modeling (2023)

Abstract: The mixing of two or more languages is called Code-Mixing (CM). CM is a social norm in multilingual societies. Neural Language Models (NLMs) like transformers have been effective on many NLP tasks. However, NLM for CM is an under-explored area. Though transformers are capable and powerful, they cannot always encode positional information since they are non-recurrent. Therefore, to enrich word information and incorporate positional information, positional encoding is defined. We hypothesize that Switching Points (SPs), i.e., junctions in the text where the language switches (L1 -> L2 or L2 -> L1), pose a challenge for CM Language Models (LMs), and hence give special emphasis to SPs in the modeling process. We experiment with several positional encoding mechanisms and show that rotatory positional encodings along with switching point information yield the best results.We introduce CONFLATOR: a neural language modeling approach for code-mixed languages. CONFLATOR tries to learn to emphasize switching points using smarter positional encoding, both at unigram and bigram levels. CONFLATOR outperforms the state-of-the-art on two tasks based on code-mixed Hindi and English (Hinglish): (i) sentiment analysis and (ii) machine translation.

## code-mixing: juxtaposition of two languages code-mixing is defined as the alternation of two or more languages during articulation. recently, code-mixing has gained a lot of attention in the area of nlp due to the prevalence of language mixing in multilingual societies such as india, europe, us, south africa, mexico, etc. in such societies, code-mixing is fairly commonplace, especially in informal conversations, where the native language * work does not relate to position at amazon. gaaye hi aur hi dance en kare hi english translation: sing and dance with the proliferation of code-mixing on the internet, it is important to study language processing and language modeling for code-mixed languages. while language modeling using neural networks has come a long way, replacing n-gram language models with distributed neural representations @xcite to recent large transformerbased pre-trained language models (lms) such as gpt-x @xcite , bert @xcite etc., code-mixed language modeling using state-of-the-art (sota) transformer-based models is still under-explored. the biggest hindrance in the adoption of sota transformer-based lms for code-mixing can be attributed to data scarcity. while transformer-based @xcite architectures such as bert and gpt have set new benchmarks in the domain of language modeling, they are infamous for their low sample efficiency. in other words, the voracious data appetite of transformers and the lack of substantial code-mixed datasets in the community is the primary reason for the technological hindrances in the area of code-mixed language modeling compared to vanilla language modeling. to corroborate the aforementioned arguments, we experiment with transformer-based models such as gpt-2 and bert for code-mixing. we empirically observe that these models perform poorly on tasks involving code-mixed data. our hypothesis is as follows: since information related to switch-ing point is a major component in the context of code-mixed content, it should thus be incorporated in downstream processing. switching points are a bottleneck for a model's processing of code-mixed data and the reason for poor performance using sota neural language models @xcite . switching points play a crucial factor when dealing with cm data. in the next few sections, we discuss various positional encoding approaches, switching points, and our approaches for language modeling on code-mixed data. our key contributions are: ## related work it is important to study code-mixing as it is a part of most multilingual societies and prevalent in social media. it is more complex to process codemixed text than monolingual text for nlp tasks @xcite . similar line of work was followed by @xcite and @xcite on the complexities of multi-languages on the basis of syntactics and grammar. the difficulties of processing code-mixed languages on social media is further exacerbated by unusual spellings, many unique ways of writing the same word, unnecessary capitalization etc @xcite . with the growing popularity on social media, various tasks like sentiment analysis @xcite , translation @xcite , hatespeech detection @xcite , pos tagging @xcite , etc. have been performed on code-mixed data. methods to handle code-mixing for text classification include the use of cnns @xcite , transformer or bert like models @xcite , ensemble models @xcite , focal loss @xcite etc. @xcite proposed transformers for neural language modeling using masked language modeling (mlm) and next sentence prediction, which achieved sota performance on many nlp tasks. @xcite released mbert, a model trained on multilingual corpus that includes 104 languages. a cross lingual language model xlm was proposed in @xcite which leveraged monolingual and crosslingual corpus for pretraining. @xcite present a bert pretrained on cm data. however, they do not make changes to their language model or technique to handle code-mixed data in particular. @xcite propose a hierarchical transformer based architecture that captures the semantic relationship among words and hierarchically learns the sentence level semantics of code-mixed data. @xcite were one of the first to incorporate switching point information in positional encoding. they utilize dynamic positional encodings whereas our method, conflator infuses switching point information in rotatory positional encodings and also uses both unigram and bigram tokens to get the final embedding. ## data extraction and strategies in this section, we discuss the details of code-mixed data extraction. our primary aim is to extract naturally distributed code-mixed data. ## the bottleneck of code-mixed language modeling: switching points formally, switching points (sps) are the tokens in text, where the language switches. for code-mixed languages, consisting of a pair of languages, there can be two types of switching points. suppose the two languages as part of the code-mixed language are l1 and l2, a switching point occurs when the language in the text changes from l1 to l2 or l2 to l1. to explain it better, let us consider the following sample in hinglish: gaana hi enjoy en kare hi english translation: enjoy the song. in the above example, when the language switches from hindi to english (gaana hi enjoy en ) a hi-en (hindi-english) switching point occurs. similarly, a en-hi(english-hindi) switching point occurs at -enjoy en kare hi . in the context of modeling code-mixed languages, switching points can be considered as ordinary bigrams, that occur with other monolingual bigrams in a corpus. it is easy to infer that particular sp bigrams will be relatively rare in a given corpus. hence, such sparse occurrences of switching point bigrams make it difficult for any language model to learn their probabilities and context. since the language changes at the switching point, lms are likely to find it difficult to process these tokens. in order to counter this challenge, we partition our code-mixed data into (i) switching points, and (ii) non-switching points. we then build lms specifically for switching points and non-switching points, as discussed in the following sections. ## positional encoding techniques as discussed, sps are a major bottleneck hence handling them separately is needed. positional encoding are necessary for language models to learn dependencies between tokens. positional embedding was first introduced by @xcite . the proposed sinusoidal positional encoding is composed of sine and cosine values with position index as inputs. the encoding techniques are further improved by @xcite where a dynamic function is introduced to learn position with gradient flow and @xcite learned positional representation of relative positions using a learnable parameter. we talk about different positional encoding techniques in detail in the following subsections. we experiment with several contemporary techniques and find that rotary positional encoding @xcite performs the best. ## incorporation of switching point information in cmlm positional encodings help the transformer learn dependencies between tokens at different positions of the input sequence. to enhance the positional encodings for code-mixed text, we modify the rotatory positional encoding to incorporate switching point information. ## experiments and results for our base models, each training step takes about 0.5 seconds. we train the base models for a total of 100,000 steps or 12 hours. for the big models like bigram and spm-based models, the step time is 1.0 seconds. the big models were trained for 250,000 steps (2 days). we use adam optimizer with β 1 = 0.9, β 2 = 0.98 and ϵ = 1e -9 . we use the method of varying the learning rate over the course of training from @xcite . we use two types of regularization during our training process: we apply dropout to the output of each encoder and decoder layer followed by normalization. in addition, we apply dropout and normalization to the sums of the word embeddings and the positional encodings in both the encoder and decoder layers. we use a rate of p @xmath0.2. ## conclusion & takeaways in this work, we report experiments on hinglish sentiment analysis and machine translation problems through the lens of language modeling. our contribution could be seen as following: (i) we introduce the idea of switching point based rotary positional encoding. whenever a switching point is encountered, we incorporate rotation change to learn the patterns of language mixing. (ii) we introduce conflator, a neural language modeling approach for code-mixed languages. conflator tries to learn better representations by means of switching point-based rotary positional encoding, initially at unigram level and then at bigram level. (iii) we empirically prove that conflator is learning the patterns of code-mixing which other models with different positional encodings prove unsuccessful, as shown in figure 5 . (iv) it is also noteworthy that conflator achieves comparable to sota results even without any pre-trained heavy language model.