C o M ix: Guide Transformers to Code-Mix using POS structure and Phonetics (2023)

Abstract: Code-mixing is ubiquitous in multilingual societies, which makes it vital to build models for code-mixed data to power human language interfaces. Existing multilingual transformer models trained on pure corpora lack the ability to intermix words of one language into the structure of another. These models are also not robust to orthographic variations. We propose CoMixCoMix is not a trademark and only used to refer to our models for code-mixed data for presentational brevity., a pretraining approach to improve representation of code-mixed data in transformer models by incorporating phonetic signals, a modified attention mechanism, and weak supervision guided generation by parts-of-speech constraints. We show that CoMix improves performance across four code-mixed tasks: machine translation, sequence classification, named entity recognition (NER), and abstractive summarization. It also achieves the new SOTA performance for English-Hinglish translation and NER on LINCE Leaderboard and provides better generalization on out-of-domain translation. Motivated by variations in human annotations, we also propose a new family of metrics based on phonetics and demonstrate that the phonetic variant of BLEU correlates better with human judgement than BLEU on code-mixed text.

## introduction code-mixing, i.e., embedding linguistic units of one language (embedded language l e ) into a sentence grammatically structured as per another language (matrix language l m ), is common in multilingual communities. growing mobile penetration coupled with the increased adoption of informal conversational interfaces is leading to further rise in such communication. currently, over 20% of user generated content from south asia and parts of europe is code-mixed @xcite . hinglish (code-mixed hindi-english) has nearly 1 comix is not a trademark and only used to refer to our models for code-mixed data for presentational brevity. 350 million speakers (gts, 2019) making it one of the most widely spoken languages. recent literature suggests that multilingual users associate codemixing with cultural affinity and prefer chatbots that can code-mix @xcite . code-mixed modeling is, thus, a foundational prerequisite for linguistic systems targeted towards such users. transformer models such as bart @xcite and bert @xcite have been successful across various nlp tasks. these models can readily capture code-mixing semantics if a large corpus was available for training. unfortunately, that is not true for most code-mixed languages. existing approaches rely on learning from a parallel corpus of embedded and matrix languages (e.g., english and hindi for hinglish). recent work @xcite , however, shows that multilingual models such as mbert trained on monolingual sources fail to effectively interleave words from topologically diverse languages. adapting transformers to code-mixed data requires addressing the following challenges: 1. divergent grammatical structure. for codemixed languages such as hinglish, where l e and l m have different parts-of-speech (pos) patterns, models trained on monolingual corpora do not yield similar representations for equivalent words across languages, which is needed to facilitate interleaving of l e and l m words. linguistic theories propose certain syntactic constraints for code-mixed generation @xcite , but these are not usually incorporated into the modeling. 2. code-mixing diversity. code-mixed languages also exhibit a wide diversity in the degree of code-mixing (e.g., ratio of l e to l m words). fig 1 shows multiple hinglish constructions for a given sentence in english. accounting for this variation in code-mixing is necessary for high fidelity modeling. 3. orthographic variations. the informal nature of code-mixed interactions and lack of standardized transliteration rules leads to users employing adhoc phonological rules while writing code-mixed content. fig 1 shows hinglish sentences with similar sounding words and their variations ("kis", "kys"). contributions. in this paper, we adapt transformer models for code-mixed data by addressing the above challenges. to ensure applicability to multiple downstream tasks, we focus on pretraining. 1. we propose comix, a set of generic pretraining methods to improve code-mixed data representations that can be applied to any transformer model assuming the availability of pos-tagger and phonetic transcription tools. these include: (a) domain knowledge-based guided attention (dkga) mechanism that facilitates intermixing of linguistic units of l e into the structure of l m through a modified attention function, (b) weakly supervised generation (wsg) that generates code-mixed data for training in a controllable fashion driven by linguistic constraints, and (c) inclusion of phonetic signals to align embeddings of similar sounding with different orthographic representation. 2. we instantiate comix pretraining for bart and bert and demonstrate efficacy on multiple downstream nlp tasks, namely machine translation, ner, sequence classification, and abstractive summarization with relative improvements of up to 22%. comixbart and comixbert achieve new state-of-the-art (sota) results for english-hinglish translation and hinglish ner tasks on lince leaderboard @xcite , beating previous best mt5 @xcite and xlm-r (winata et al., 2021) models, despite having less than 0.5x and 0.1x model size respectively. 3. we evaluate out-of-domain code-mixed translation performance on two test sets, one created in-house and other one adapted from gupshup corpus @xcite , and show that comix generalizes better than other models. to the best of our knowledge, this is the first such evaluation for english-hinglish translation. we hope our benchmark will assist the community to improve out-ofdomain generalization of code-mixed translation, a critical need for low-resource regimes. 4. to address the limitations of existing metrics in handling orthographic variations in code-mixed data, we propose a new family of natural language generation (nlg) metrics based on phonetic adaptation of existing metrics. we observe that phobleu, the phonetic variant bleu, is better aligned to human judgement (+0.10 -0.15 on pearson correlation) than bleu on hinglish. 2 related work multilingual and code-mixed nlp. recent advances in large multilingual pre-trained models such as mbert @xcite and mbart @xcite have led to significant gains on many multilingual nlp tasks. however, evaluation of these models on code-mixed content for machine translation @xcite , sequence classification @xcite , summarization @xcite and other tasks @xcite points to their inability to intermix words from two languages since these are pretrained on monolingual text without any language alternation. our comix approach encourages the model to learn representations that allows appropriate embedding of words from one language into structure of another via domain knowledge guided attention and through weakly supervised code-mixed generation. prior work @xcite focuses on generating synthetic code-mixed data using constraints from linguistic theories followed by learning. we perform joint generation and learning using pretrained models that has dual benefit of data generation and improving model representations, and has been shown to be effective for anomaly detection in medical images @xcite . ## comix approach given a corpus of sentence pairs from l m and l e , our goal is to adapt transformer models such as bart and bert to overcome the key challenges in modeling code-mixed data. to ensure applicability to multiple downstream tasks, we focus on the pretraining phase. we assume access to pos tagging and phonetic transcription tools foot_0 which is true for many languages (see section 7). below we summarize our approach for each of the challenges. p1 -divergence in pos structure of l e and l m : to enable transformer models to extrapolate from l e and l m to code-mixed data, we rely on linguistic constraints. we observe that coarse groups of pos labels of concepts are preserved across translation (see section 7) and that code-mixed sequences often retain the pos structure of l m sequence. assuming access to pos labels the above constraints provide token-level correspondence for parallel training sentences, which can be used to augment the transformer attention mechanism and lead to representations that facilitate accurate interleaving of l e and l m words. [section 3.1] p2 -variations in the level of code-mixing: to accurately model variations in code-mixed data such as the mixing propensity, we propose a weakly supervised approach that combines pos constraints with a control on the code-mixing probability to generate code-mixed sequences from parallel monolingual corpora foot_1 for training. [section 3.2] p3 -orthographic variations: to align similar sounding words with orthographic variations, we incorporate phonetic signal as an additional input channel. we modify the transformer architecture to include two multi-head self-attention layers, one each for text and phoneme channels. [section 3.3] ## phonetic sentence comparison metrics lack of standardized transliteration rules is a key challenge not only for modeling but also for evaluating code-mixed or multi-lingual nlg tasks, e.g., english to hinglish translation human annotators employ orthographic variations and are also inconsistent in the use of punctuation and upper-lower casing as shown in fig 1 . most nlg evaluation metrics such as bleu, do not account for these variations, which leads to pessimistic and inaccurate estimates of the performance of nlg systems. to address this gap, we propose a new family of metrics based on the phonetic representation. let s(•, •) be any metric such as bleu and rouge @xcite that facilitates comparison of a word sequence against a reference one. given a pair of sentences (x, y ), we define the phonetic metric as p ho-s(x, y ) = s(x ph , y ph ), where x ph , y ph are the phonetic sequences. in this paper, we limit our focus to phobleu and @xcite and indicbert @xcite are competitive relative to mbart and mbert respectively on indic languages. further, since we initialize our models comixbart and comixbert with weights from indicbart and indicbert, we consider these as strong baselines for our evaluation of generative and classification tasks respectively. ## extensibility of comix the proposed ideas of domain-knowledge guided attention, weak supervision, and phonetic representations are not specific to hinglish and readily generalize to any language pair where we have a parallel corpus of embedded and matrix language content and tools for pos tagging and phonetic transcription. below we discuss these requirements along with other assumptions on the pos structure that permits extensions of our methodology to most common code-mixed language pairs. assumption 1: availability of parallel corpora. most common code-mixed languages happen to include english among the language pair which is typically transcribed in latin script that permits easy phonetic transcription through tools such as pyphonetics. currently, there also exist multiple large parallel corpora (e.g. flores, ccmatrix, samanantar) where sentences in english are paired with that of multiple other languages. there are also many ongoing initiatives for creating such parallel corpora even for low-resource languages. hence, the requirement of a large parallel corpus of matrix and embedded language content is satisfied by most common code-mixed pairs. assumption 2: availability of pretrained multilingual models. with the proliferation of massively multilingual foundational models (e.g., mbart (50 languages), mbert (104 languages), t5 (101 languages)) including advances in synthetic data augmentation, our assumption on the availability of pretrained llms or datasets to pretrain those models is also a reasonable one. we choose to work with indicbart and indicbert which support 11 and 12 indic languages respectively because they provide stronger baselines for indic languages and are faster to experiment with because of their smaller size, but the proposed ideas can be readily applied with any pre-trained transformer model. assumption 3: languages of l m and l e share the same pos set and access to pos tagging utilities. @xcite proposed universal pos tagset comprising 12 categories that exist across languages and had developed a mapping from 25 language-specific tagsets to this universal set. they demonstrated empirically that the universal pos categories generalize well across language boundaries and led to a open community initiative by universaldependencies.org on creating universal pos tags @xcite for 90 languages. in our work, we use these universal pos tags to build three coarse groups (nouns-pronouns, adjectivesverbs-adverbs, rest) of pos tags (see fig 7 ). note that even though we utilize pos-tagging, the structural constraints are imposed with respect to these three coarse groups. fig 7 in a.1 lists the pos tags from the universal pos tags website, which we use in our work. further, stanza provides universal pos-tagging utilities for around 66 languages. assumption 4: equivalent word pair from l m and l e share the same coarse pos group. we assume that equivalent words in an l m and l e pair share the same coarse pos group (from fig 7 ) and not necessarily the same pos tag. a small-scale empirical analysis of 50 hindi-english-hinglish sentences from the hood dataset (sec 5.2) indicates this assumption is true in 88.6% of the cases. pos-tags provide complementary (weak) supervision for intermixing (in dkga) and generation (in wsg) in addition to word semantics already captured in embeddings. further, even though our current guiding function f dkga assumes a hard constraint on the word pairs to be in the same coarse pos group, our methodology is general and can be extended to the case where the two languages have different pos tag sets. in particular, given empirical probabilities that a matrix token with pos tag a maps to an embedded token with pos tag b for all possible pairs of pos tags (a, b), we can define the guiding function's value f dkga ij ## conclusion we presented comix, a pretraining approach for code-mixed data that combines (a) domain knowledge guided attention (dkga), (b) weakly supervised code-mixed generation based on posstructure constraints, and (c) transformer encoder modification to include phonetic signal. we showed that comix yields improvements across multiple code-mixed tasks, achieving new sota result for eng-hinglish translation and hinglish ner on lince leaderboard with superior performance on out-of-domain translation. our approach is applicable to code-mixing with all languages where pos tagging and phonetic transcription is possible. motivated by gaps in current nlg evaluation metrics for code-mixed data, we proposed a new family of metrics based on phonetic representation and show that phobleu is better correlated with human judgement than bleu on hinglish. in future, we plan to extend the applicability of dkga and wsg to other settings that can benefit from domain knowledge, and explore new metrics for code-mixed nlg with a large scale evaluation.