Towards Code-switched Classification Exploiting Constituent Language Resources (2020)

Abstract: Code-switching is a commonly observed communicative phenomenon denoting a shift from one language to another within the same speech exchange. The analysis of code-switched data often becomes an assiduous task, owing to the limited availability of data. In this work, we propose converting code-switched data into its constituent high resource languages for exploiting both monolingual and cross-lingual settings. This conversion allows us to utilize the higher resource availability for its constituent languages for multiple downstream tasks. We perform experiments for two downstream tasks, sarcasm detection and hate speech detection in the English-Hindi code-switched setting. These experiments show an increase in 22% and 42.5% in F1-score for sarcasm detection and hate speech detection, respectively, compared to the state-of-the-art.

## introduction code-switching is the juxtaposition of speech belonging to two different grammatical systems or subsystems within the same speech exchange. in other words, it denotes a shift from one language to another within the same utterance but not limited to beyond mere insertion of borrowed words, fillers, and phrases. it often involves morphological and grammatical mixing @xcite . codeswitching of speech is a common phenomenon in the multilingual communities @xcite , especially among peers who have similar fluency in multiple languages. code-switching among languages like spanish-english, hindi-english, tamil-english is often seen to be used by bilingual speakers of these languages. code-switching is likely to be found in informal settings, including speech and social media, and semi-formal such as navigation instructions. this phenomenon leads us to acknowledge codeswitching as a legitimate communication form deserving careful linguistic analysis and processing. processing code-switched communication would enhance user experience in various industrial settings, including advertising, healthcare, education, and entertainment @xcite . access to code-switched data is challenging and limited. this phenomenon makes the analysis and information extraction from code-switched languages a less explored and challenging task. most code-switching studies focus on pairs of a high resource and a lower resource language, often making these studies data starved. also, code-switched language is mostly used for informal and semiformal settings, making it less archived, resulting in its limited availability in terms of context and volume. traditionally, most works in academia have limited themselves to language-independent methodologies like svm, fasttext, and cnn @xcite for modeling code-switched languages. this can be attributed to the shortage of data in code-switched settings, limiting the exploitation of predictive performance achieved via transfer learning using pretrained contextualized word embeddings. we propose a generic methodology for modeling hindi-english code-switched data revolving around its translation and transliteration. we propose two approaches that exploit a) transliteration to convert this low-resource task into a highresource task using cross-lingual learning, and b) translation to utilize monolingual resources of each of the constituent languages. the importance of this methodology is that we enable the exploitation of pretrained transformer-based embeddings that are available in the monolingual or cross-lingual setting but not code-mixed setting due to the scarcity of data for pretraining. we then benchmark both the approaches for the tasks of hate speech detection and sarcasm detection using the current state-of-the-art and language-independent supervision learning as the baseline. we further provide insights into the proposed approaches and compare them analytically. ## related works the previous works on sarcasm and humor detection contain a myriad of methods employed over the years on multiple datasets, including statistical and n-gram analysis on spoken conversations from an american sitcom @xcite , word2vec combined with knn human centric features on the pun of the day and 16000 one-liner datasets @xcite , and convolutional neural networks on datasets with distinct joke types in both english and chinese @xcite . recently, transformer-based architectures have been used to detect humor on multiple datasets collected from reddit, kaggle, and pun of the day dataset @xcite . the prior works exploring hate speech detection employ several machine learning-based methods like linguistic features with character n-grams @xcite , svm with a set of features that includes n-grams, skip-grams, and clustering-based word representations @xcite . neural networks like lstm @xcite , fasttext @xcite and @xcite have also been used for hate speech detection. recently, transformer-based architectures using bert, al-bert, and roberta have been used to detect hate speech @xcite . there has been considerable work in codeswitching, as is well documented by @xcite . the use of code-switched data has been explored for the tasks of part-of-speech tagging @xcite , named entity recognition @xcite , dependency parsing @xcite , and text classification @xcite . @xcite introduced an english-hindi code-switching dataset for sarcasm detection in social media text. their dataset contains tweets annotated as sarcastic and non-sarcastic in the form of their constituent language-identified tokens. they further provide a feature-based model as a base-line. we use their dataset for our sarcasm detection experiments. @xcite introduced a dataset of english-hindi social media text, demonstrating code-switching for the task of hate speech detection. they also conduct benchmarking experiments using a feature-based model. similar to @xcite , their benchmarks only contain accuracies for their baseline systems and do not take f1-scores into account. we use their dataset for our hate speech detection experiments. there have been attempts to exploit monolingual datasets for enhancing downstream tasks in a code-switched setting. @xcite explored the use of monolingual taggers for spanish and english for part-of-speech tagging in an english-spanish code-switched dataset. they further show that their best results were obtained by using the output of monolingual pos taggers as input features. @xcite explore using monolingual and bilingual resources for the task of question answering in an english-hindi codeswitched dataset. for text classification involving code-switched text, traditional methodologies like svm, fasttext, cnn has been used @xcite . however, most works are unable to exploit the recent advances in pretrained models, including roberta and xlm-roberta @xcite due to data scarcity and lack of normalization in the code-switched setting. to overcome this, @xcite attempt to use multilingual bert for code-switched sentiment analysis of social media text using a transliteration based methodology. however, they failed to beat the traditional baselines, including fasttext. ## approach this section outlines our approach for analyzing code-switched language by converting it into its respective high resource languages and fine-tuning monolingual and cross-lingual contextual word embeddings. we highlight the data preparation followed by the contextual embedding used for each of the variants. figure 1 illustrates our proposed approach through an example code-switched sentence from one of the datasets. ## conclusion in this work, we show that converting codeswitched data into its constituent high resource language enables us to exploit the performance of the pretrained models for those languages on downstream tasks. we perform experiments on the english-hindi code-switching pair and benchmark our approach for sarcasm detection and hate speech detection. the experiments show an improvement of up to 42.5% in f1-score compared to strong baselines like cnn. our findings pave the way for further research to utilize monolingual resources for code-switched data for multiple downstream tasks and extend this methodology for other codeswitched language pairs.