Sociolinguistically Informed Interpretability: A Case Study on H inglish Emotion Classification (2024)

Abstract: Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression,especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and languages, but it remains to be seen whether these models learn and are robust to the differences in emotional expression across languages. Sociolinguistic studies have shown that Hinglish speakers switch to Hindi when expressing negative emotions and to English when expressing positive emotions. To understand if language models can learn these associations, we study the effect of language on emotion prediction across 3 PLMs on a Hinglish emotion classification dataset. Using LIME and token level language ID, we find that models do learn these associations between language choice and emotional expression. Moreover, having code-mixed data present in the pre-training can augment that learning when task-specific data is scarce. We also conclude from the misclassifications that the models may overgeneralise this heuristic to other infrequent examples where this sociolinguistic phenomenon does not apply.

## related work code-mixing previous works in emotion classification and sentiment analysis have demonstrated that processing code-mixed text is more difficult than monolingual text @xcite @xcite . this is in part due to the complexities of processing emotion from two different languages with varying sociocultural and grammatical structures at play @xcite @xcite . in this context, @xcite published a survey on the linguistic and social perspectives on code-mixing for language technologies. they emphasized the importance of incorporating the social context of a code-mixed language pair into systems processing code-mixed text. interpretability with lime lime (local interpretable model-agnostic explanations) @xcite is a popular tool for interpretability that is model agnostic and employable for classification tasks. it learns a linear classifier locally around a model's prediction, leveraging token weights (also learned by the linear classifier) to assign a "lime score" between 1 and -1 to each token. a positive score indicates that the token influenced the model towards the predicted label, and a negative score indicates that the token influenced the model to not predict that label. we leverage lime due to its availability and easy-to-use implementations, for instance in the language interpretability tool (lit; @xcite ), which we used for this work. previous work has also indicated the accuracy of its approximation of the models and its ability to provide human-friendly explanations @xcite . ## methodology dataset we utilize a dataset for sentiment analysis of code-mixed tweets by @xcite , later annotated with emotion labels by @xcite . each example is annotated with the six basic ekman emotions @xcite ) -joy, sadness, fear, surprise, disgust and anger. when an example does not fit any of these emotions, or expresses no emotion, it is labelled as others. this dataset contains 14,000 examples in the train set, 3,000 in the validation, and 3,000 in the test set. for this work, we randomly sample 1,000 examples from the validation set to enable manual verification of the automatic token tagging described below, maintaining the default distribution across labels (see appendix b). ## results and analysis first, to answer whether the models learn to meaningfully distinguish between languages for emotion prediction (rq1), we examine the distributions of lime scores across each language id tag (english, hindi, other). concretely, we inspect the frequency with which tokens received a positive or a negative lime score in our sample, for each language. we then conduct a χ 2 test of independence to determine whether these two variables have some dependency. next, we examine this dependency on a more granular level to determine whether the presence of english tokens influence the hinglish emotion classification models to predict more positive emotions (rq2), and whether hindi tokens influence them to predict negative emotions (rq3). we observe the distribution of language id across lime scores for examples that the models predicted as joy, anger, and sadness. these labels were selected in particular as they have the most examples in the dataset (after others), and provide the positive (joy) and negative (anger and sadness) polarity discussed in the sociolinguistics literature. ## discussion from the section above, it can be concluded that the models are able to distinguish patterns of speaker preference detailed by @xcite when predicting emotion for code-mixed data. english tokens influence the models more towards predicting a positive emotion, and hindi tokens influence the models more towards predicting a negative emotion. an example of this is provided in figure 3 , where all the models exhibit a strong degree of influence from the english tokens in their prediction of the joy label. at the same time, most of the tokens assigned a negative lime score come from hindi. for sadness, we surmise that a shortage of training data is responsible for models' failure to learn meaningful differences across the languages. about 10% of the entire dataset consists of examples labelled sadness. in contrast, joy is 30% and anger is about 20% (see appendix b). even with less data, however, we still observe a dependency between language and lime score with hin-groberta. it is the only model we examine with code-mixed data present in the pre-training. thus, when there is less data for a model to learn these associations, it can help to have code-mixed data in the pre-training. @xcite found that language models can adapt to heuristics that are valid for frequent cases and fail on the less frequent ones. in a similar vein, we investigate whether these sociolinguistic associations learnt by the models overgeneralise to the less frequent examples where this phenomenon is not seen. we examine instances where the models have misclassified examples labelled as joy and anger, highlighted in figure 4 . for both joy and anger, the models generally either predict another label of the same emotional polarity (for example, disgust instead of anger), or they predict them as others. the dataset is highly imbalanced, and thus we can say that although the models can discern the polarity difference between positive and negative emotion labels (as seen in figure 4 where the values in the lower left and upper right quadrants are low), they struggle with granular distinctions between them. ## conclusion in this work, we use sociolinguistics theories to understand what plms learn when training emotion classifiers for code-mixed data. we found that the models indeed learn the differences in language use and emotional expression detailed in the sociolinguistics literature. concretely, these are the associations of english tokens with positive emotions, and hindi tokens with negative emotions. adding code-mixed data to the pre-training can help augment this learning when task-specific data is scarce. however, the models can overgeneralise this learning to infrequent examples where it does not apply. in future work, it would be interesting to see if this understanding can be leveraged to help improve systems designed for code-mixed languages. aqsa younas, raheela nasim, saqib ali, guojun wang, and fang qi. 2020. sentiment analysis of codemixed roman urdu-english social media text using deep learning approaches. in 2020 ieee 23rd international conference on computational science and engineering (cse), pages 66-71. evi yulianti, ajmal kurnia, mirna adriani, and yoppy setyo duto. 2021. normalisation of indonesian-english code-mixed text and its effect on emotion classification. international journal of advanced computer science and applications, 12(11). george-eduard zaharia, george-alexandru vlad, dumitru-clementin cercel, traian rebedea, and costin chiru. 2020. upb at semeval-2020 task 9: identifying sentiment in code-mixed social media texts using transformers and multi-task learning. in proceedings of the fourteenth workshop on semantic evaluation, pages 1322-1330, barcelona (online). international committee for computational linguistics.