Elevating Code-mixed Text Handling through Auditory Information of Words (2023)

Abstract: With the growing popularity of code-mixed data, there is an increasing need for better handling of this type of data, which poses a number of challenges, such as dealing with spelling variations, multiple languages, different scripts, and a lack of resources. Current language models face difficulty in effectively handling code-mixed data as they primarily focus on the semantic representation of words and ignore the auditory phonetic features. This leads to difficulties in handling spelling variations in code-mixed text. In this paper, we propose an effective approach for creating language models for handling code-mixed textual data using auditory information of words from SOUNDEX. Our approach includes a pre-training step based on masked-language-modelling, which includes SOUNDEX representations (SAMLM) and a new method of providing input data to the pre-trained model. Through experimentation on various code-mixed datasets (of different languages) for sentiment, offensive and aggression classification tasks, we establish that our novel language modeling approach (SAMLM) results in improved robustness towards adversarial attacks on code-mixed classification tasks. Additionally, our SAMLM based approach also results in better classification results over the popular baselines for code-mixed tasks. We use the explainability technique, SHAP (SHapley Additive exPlanations) to explain how the auditory features incorporated through SAMLM assist the model to handle the code-mixed text effectively and increase robustness against adversarial attacks.

## introduction the proliferation of code-mixed content on social media platforms among multilingual commu-nities around the globe has been widely observed in recent years. it has been established that handling code-mixed content for information retrieval or classification poses a unique set of challenges. these challenges become even more prominent when a language is written in a different script during code-mixing. since there are no formal spelling standards for a word in a different script, there can be large variations in spellings (eg: ' ' (yes) in hindi can be written as 'haan @xcite . these spelling variations depend on many sociocultural factors, such as dialect, accent, and region @xcite . it has been noted that a significant portion of the code-mixed content present on social media platforms is romanized, which presents a challenge in terms of processing and analysis due to the lack of following a standardized romanization method. this lack of standard leads to many complexities and is one of the major roadblocks in training a reliable and robust code-mixed nlp system @xcite . managing such variations within text data are typically achieved through pre-processing techniques, such as data augmentation and normalization @xcite , which necessitates the utilization of human-annotated dictionaries and can entail a significant investment in manual annotation efforts. it has been observed that traditional techniques for processing and analysis of code-mixed content may prove ineffective in cases where the spelling of a word varies from those present in the corpus or dictionary @xcite . although transformer based pre-trained models @xcite have proven to be largely effective for most of the tasks in natural language processing (nlp) @xcite , it has been shown that even such models are not robust enough to handle small perturbations in spelling @xcite . such perturbations have been used to perform adversarial attacks on even the transformers based language models. adversarial attack entails making small human imperceptible perturbations to the input to mislead the models. the first study in this direction proposed three adversarial attacks based on phonetic perturbations to test the limits of a codemixed text classifier. in this study, it was found that the bert @xcite model was vulnerable to such phonetic perturbations. @xcite found that phonetically similar spelling variations of a word are often imperceptible to humans. for example, words acha (meaning 'okay'), acchha, and achha have similar sounds when spoken. these properties of words are known as sms property (similar sound, similar meaning, different spellings) @xcite . in this paper, we focus on incorporating the auditory phonetic (ap) features of words along with their semantic features in language models. we hypothesize that a model trained by utilizing these features would be agnostic to subtle spelling variations. these variations are often found to be the achilles heel of deep learning systems and such variations are exploited during adversarial attacks. incorporating these features would also lead to building better and more robust classifiers for code-mixed input. to obtain the ap features, we utilize the soundex algorithm @xcite . this algorithm encodes the sms property of words. in this encoding, the words acha (ok), achha, and acchha have the same encoding vectors (a200). to embed these phonetic properties, we propose two novel language modeling approaches named soundex language modelling (smlm) and soundex aligned masked language modeling (samlm) that are able to map between the semantic and auditory properties of words in a text. we use these approaches to pre-train bert and roberta models. we then fine-tune our pretrained models on downstream classification tasks based on code-mixed hinglish (hindi+english) and benglish (bengali+english) datasets. we perform phonetic perturbation-based attacks following @xcite and find that our smlm and samlm pre-trained models are more robust to such adversarial attacks. we observe a lower drop in performance in both models when compared to the base bert and roberta models after the attack. additionally, we also observe a improvement in classification scores on the downstream tasks of code-mixed text classification in both languages. however, these models lack transparency, which makes it difficult to understand their actual decision process. hence, we exploit the model explainability to analyze the decision process of our models by extracting the terms responsible for the final prediction. for this purpose, the explainability technique, shap (shapley additive explanations) (lundberg and lee, 2017) is used. to the best of our knowledge, this is the very first attempt towards utilizing ap properties to enhance the robustness of models while dealing with code-mixed datasets. the key contributions of this work are as follows: ## related work transformers-based pre-trained models have achieved remarkable success in a wide range of nlp tasks @xcite @xcite . however, several studies have shed light on vulnerabilities of these models @xcite . @xcite propose a black-box algorithm to attack bert model with the help of closet synonyms. but it can lead to unnatural sentences because the synonym may not fit the context of the sentence. to overcome this limitation, authors in @xcite @xcite proposed to use a masked language model (bert or roberta) for replacements or insertions. there are numerous studies to enhance adversarial robustness using data augmentation, adversarial training (morris et al., 2020), etc. data augmentation requires manual human efforts and adversarial training requires re-training models on adversarial data which is costly. however, all these attempts are for the high-resource english language except @xcite . increasing phenomena of code-mixing on social media platforms have also motivated researchers to analyze the adversarial robustness of code-mixed models. authors in @xcite exposed the vulnerability of code-mixed classifiers by performing an adversarial attack based on subword perturbations, character repetition, and word language change. however, there is no attempt to enhance the adversarial robustness of code-mixed text classifiers against these perturbations. this motivated us to develop a robust model to handle adversarial perturbations for code-mixed text. researchers analyzed the behaviour of pretrained language models (pmlm) for different languages and attempted to enhance their performance on the downstream tasks. for example, @xcite conducted experiments on tamil, kannada, and malayalam scripts and observed that multilingual models perform better than monolingual models. @xcite proposed a multilingual framework to fine-tune bert in shared private fashion to transfer knowledge between code-mixed and english languages. @xcite performs adapter-based fine-tuning of pmlms for code-mixed text classification. however, their focus is not on handling phonetic perturbations based adversarial attacks. there are a few attempts to enrich the representation of pre-trained models like bert in the speech domain. for example, @xcite proposed a bert-style language model, referred to as phonemebert that learns a joint language model with phoneme sequence and audio speech recognition (asr) errors to learn phonetic-aware representations that are robust to asr errors. they introduced noise to speech (noise related to door opening, aircaraft, etc.) and handled them using phoneme sequences. however, our task is different from above in the following aspects: (i). our focus is to enhance adversarial robustness of code-mixed classifiers against adversarial attacks; (ii). our proposed approach is tuned to handle textual perturbations in code-mixed data rather than perturbations in speech signals. ## threat model our target models are bert and roberta based code-mixed text classifiers due to their huge suc-cess in many nlp tasks @xcite . an adversary attempts to mislead the target models by generating adversarial samples to make wrong classification decision. adversary's goal: given an input sentence s, consisting of n tokens w 1 , w 2 , w 3 , . . . , w n , with ground truth label y, and a target model m (s) = y, the goal of the adversary is to perform an untargeted attack, i.e., find an adversarial sample s adv , causing m to perform misclassification, i.e., m (s)! = y. adversaries attack the model using phonetic perturbations in line with the prior work @xcite . design goals: based on the aforementioned adversary model, our proposed framework (smlm and samlm) must meet the robustness and accuracy requirements. ## methodology our objective is to equip the pre-trained models to increase their robustness against adversarial attacks and handle phonetic spelling variations in code-mixed datasets. the detailed flow of our proposed approach is shown in figure 1 . there are 3 main components, viz. pre-training, fine-tuning, and model explainability. first, we pre-train the models (bert and roberta) to incorporate auditory features, followed by task-specific fine-tuning. finally, the model explainability component explains the decision process of our proposed approach and illustrate the effectiveness of our proposed approach. it analyzes how the adversarial attacks and phonetic spelling variations are handled by our proposed models qualitatively. ## experimental setup and results we use bert-base and roberta-base as target models for each task. to access the experimental evaluation of our proposed approach, we conduct extensive experiments on code-mixed hinglish and benglish language datasets. for hinglish, we conduct experiments on two benchmark datasets related to offensive @xcite and sentiment analysis @xcite . for benglish, we conduct experiments on aggression analysis data @xcite . similarly, for pretraining task, we use a total of 33,014 hinglish sentences and 6,149 benglish sentences. 3 5.1 baselines vanilla classifiers (vc): we fine-tune the vanilla bert and roberta (henceforth referred to as vcbert and vcroberta) pre-trained models on the downstream tasks with only the word sequences as input. vanilla masked language modelling pretrained classifiers (vmlm): we pre-train bert and roberta on real code-mixed hinglish/benglish datasets (we henceforth refer to these pre-trained models as vmlmbert and vmlmroberta). after pre-training, the model is fine-tuned on their respective language datasets for the downstream tasks. during fine-tuning, only word sequences are considered as input. phonemlm classifiers: we pre-train bert and roberta on word and phoneme sequences. phoneme sequences are appended at the end of word sequence separated by '[sep]' token following sundararaman et al. @xcite . next, each model is fine-tuned on the downstream task using both word and phoneme sequences as input. phoneme sequences are generated using phonemizer tool foot_2 . ## qualitative analysis this section analyzes the actual decision process of the proposed framework for the classification tasks by extracting the terms responsible for predicting the final output class. we explain the behaviour of different bert-base models for hinglish sentiment dataset. ## conclusion in this paper, we propose two novel pre-training steps, smlm (soundex masked language modeling) and samlm (soundex aligned masked language modeling), to incorporate the auditory phonetic (ap) features into popular classification models, bert and roberta. our approach effectively handles spelling perturbations, a common form of attack in code-mixed languages like hinglish and benglish. we perform phoneticbased adversarial attacks on models trained using our technique and find that the performance decrease is significantly less than multiple baselines. additionally, incorporating the ap features leads to improvement in classification scores on different tasks in both hinglish and benglish as compared to models trained only on semantic features. in summary, the novel pre-training steps of smlm and samlm provide an effective way to incorporate ap features into nlp models, leading to improved robustness and performance on code-mixed text classification tasks. in future work, we plan to extend our approach to other code-mixed languages and evaluate its performance on more nlp tasks. we believe that our approach can have a significant impact on the robustness of nlp models, especially in the context of code-mixed languages.