T eam_ S wift at S em E val-2020 Task 9: Tiny Data Specialists through Domain-Specific Pre-training on Code-Mixed Data (2020)

Abstract: Code-mixing is an interesting phenomenon where the speaker switches between two or more languages in the same text. In this paper, we describe an unconventional approach to tackling the SentiMix Hindi-English challenge (UID: aditya_malte). Instead of directly fine-tuning large contemporary Transformer models, we train our own domain-specific embeddings and make use of them for downstream tasks. We also discuss how this technique provides comparable performance while making for a much more deployable and lightweight model. It should be noted that we have achieved the stated results without using any ensembling techniques, thus respecting a paradigm of efficient and production-ready NLP. All relevant source code shall be made publicly available to encourage the usage and reproduction of the results.

## introduction with the rapid growth in the number of active internet users, there comes a commensurate need to analyze their usage patterns. there may be several reasons to do so, including content moderation, cyber-abuse detection and even for market research purposes to enhance the user experience. however, considering the petabytes of data available online, it becomes a mammoth task that is nearly unachievable through manual means. for instance, social media platforms like twitter have millions of tweets generated by users every day. by abstracting away tasks like sentiment analysis and content moderation, natural language processing (nlp) techniques can play an instrumental role in this kind of data analysis. sentimix hindi-english @xcite ) is a code-mixed sentiment analysis task. code-mixing is a phenomenon where the author arbitrarily switches between two or more languages in the same snippet of text. code-mixing, along with the inherently noisy nature of social media text make its analysis a nontrivial task. we experimented with a wide range of techniques that included contemporary transformer architectures @xcite , word embedding pre-training and language model pretraining. our experiments provide an interesting inference that near state-of-the-art performance need not require massive model sizes. this analysis would especially be helpful for those trying to deploy machine learning models in production, where computational power and inference speed may be a constraint. ## background in this section, we provide a short discussion of the algorithms and technologies that are a part of our implementations. ## dataset description the organizers have provided a hindi/english code-mixed sentiment analysis dataset consisting of samples labeled as negative, neutral and positive. the dataset was provided with language labels per token (en-english, hi-hindi, o-universal). the distribution is as shown in the dataset consists of informal code-mixed text with the widespread use of slang words and colloquial references. this, along with some label noise makes training the model an especially interesting task. the dataset also contains emojis. the above-mentioned attributes make for a dataset that reliably represents the realistic scenario of analyzing noisy twitter data. ## pre-training/fine-tuning corpus generation while fasttext still provides an embedding for oov tokens from the hinglish text, the embeddings so generated would be of suboptimal quality due to a large number of unseen tokens of the hinglish text. to address this problem, a hinglish corpus is generated by scraping publicly available tweets from twitter. this corpus is also used in our alternate approach of fine-tuning the xlm-r language model to be subsequently used for classification. additional efforts are made to generate a corpus that covers domains relevant to the sentimix dataset by utilizing features such as popular user ids in the training sets. a small minority of tweets present in pure devanagiri script were removed to retain only hinglish tweets in roman script. thus, we generated a corpus consisting of approximately 5.7 million tokens (≈ 306259 tweets). ## proposed approach for the said task, we attempted multiple techniques and their combinations. in this paper, we have limited our discussion to the xlm-r and the hitext bi-lstm related approaches. ## conclusion through this paper, we have demonstrated how task-specific pre-training of very small models can give a performance that is comparable to classifier fine-tuning on massive generic publicly available models. one should also realize the advantages that small models would have concerning infrastructure costs and scalability when deployed in production. with inference speed nearly twenty times using such small scale models, we have shown how simple models can provide good deployability without compromising on performance. all relevant source code concerning dataset generation, preprocessing and model training has been made publicly available for the usage and reproduction of our results. 1