H inglish NLP at S em E val-2020 Task 9: Fine-tuned Language Models for H inglish Sentiment Detection (2020)

Abstract: Sentiment analysis for code-mixed social media text continues to be an under-explored area. This work adds two common approaches: fine-tuning large transformer models and sample efficient methods like ULMFiT. Prior work demonstrates the efficacy of classical ML methods for polarity detection. Fine-tuned general-purpose language representation models, such as those of the BERT family are benchmarked along with classical machine learning and ensemble methods. We show that NB-SVM beats RoBERTa by 6.2% (relative) F1. The best performing model is a majority-vote ensemble which achieves an F1 of 0.707. The leaderboard submission was made under the codalab username nirantk, with F1 of 0.689.

## introduction code-mixing or code-switching refers to the use of two or more languages or speech variants together @xcite . this is commonly observed in informal conversations, especially those on social media, e.g. twitter @xcite . while a small body of work does exist on code-mixing detection, this task focuses on polarity detection (sentiment analysis). we demonstrate the impressive performance of transfer learning for the task of sentiment detection in code-mixed context. we discuss limitations of existing deep learning pre-trained models which are trained on "monolingual" textwhich has sentences in only one language. in this work, we demonstrate how it is beneficial to fine-tune the language model (lm) for a code-mixed setting like hinglish, when hindi is written in the roman script. the labelled data for the sentiment classifier is from the task paper - @xcite . it consists of a total of 17000 tweets. the sentiment labels are positive, negative, or neutral, and the code-mixed languages are english-hindi. the train data after the split has 14k tweets. the validation and test data contain 3k tweets each. ## data we used two primary datasets. the data used for fine-tuning lm is from twitter stream. the tweet stream data was used which contains ∼1.9 million hinglish tweets. we manually sampled 3k tweets to verify what fraction of them are hinglish. this dataset consists of ∼86.5% hinglish tweets. the 17k tweet sentimix data @xcite was used to further fine-tune the sentiment classifier. the sentimix data contains 14,000 tagged tweets marked as positive, negative and neutral for training, and 3,000 for development and testing each. ## results the result for the experiments are summarized in there are three main incremental directions of improvements: data, methods, adopting techniques from text classification. for instance, initial tweet data had a lot of truncated tweets, using tweet ids to get an entire tweet would enrich our inputs. the training data can also be augmented in a wide variety of ways such as using vector similarity @xcite . we can investigate other methods which might help in understanding missed case. sentence embeddings for hinglish, similar to infersent @xcite or universal sentence encoding @xcite may be promising, in addition to skip thought or other sentence vectorisation methods, as well as exploring the performance of models which do not focus on transfer learning like r-cnn, and lstms. lastly, a wide variety of deep learning tricks and methods could be used, such as label smoothing @xcite , which can help in generalising better beyond the small training sample. ## conclusion we demonstrate that ensembles of classical machine learning models, even nb-svm exhibit competitive performance and can in fact be better than some transformer baselines. it is still worthwhile to implement simple classical baselines. additionally, we hope that the released dataset and models foot_1 will encourage readers to investigate this further.