Reed at S em E val-2020 Task 9: Fine-Tuning and Bag-of-Words Approaches to Code-Mixed Sentiment Analysis (2020)

Abstract: We explore the task of sentiment analysis on Hinglish (code-mixed Hindi-English) tweets as participants of Task 9 of the SemEval-2020 competition, known as the SentiMix task. We had two main approaches: 1) applying transfer learning by fine-tuning pre-trained BERT models and 2) training feedforward neural networks on bag-of-words representations. During the evaluation phase of the competition, we obtained an F-score of 71.3% with our best model, which placed 4th out of 62 entries in the official system rankings.

## introduction the internet today has a vast collection of data in various forms, including text and images. a big part of this data comes from various social media platforms, which enable users to share thoughts about their daily experiences. the millions of tweets, updates, location check-ins, likes, and dislikes that users share every day on different platforms form a large bank of opinionated data. extracting sentiment from this data, though immensely useful, is also challenging, giving rise to the nlp task known as sentiment analysis. several models have been proposed to perform this task over the years, such as those built on top of the recursive neural tensor network @xcite or the more recent bert model @xcite . although most of these language technologies are built for the english language, a lot of internet data comes from multilingual speakers who combine english with other languages when they use social media. thus, it is also important to study sentiment analysis for this so-called 'code-mixed' social media text. in this paper, we explore sentiment analysis on code-mixed hinglish (hindi-english) tweets, specifically as a participant in task 9 of semeval-2020 @xcite . we had two main strategies. we began by fine-tuning pre-trained bert models to our target task. during this initial phase, we observed that the accuracies yielded by bert-base, bert-multilingual and bert-chinese on the validation data were approximately the same. this made us hypothesize that the pre-trained weights were mostly unhelpful for the task. to test this, we attempted to recreate the bert fine-tuning results only using feedforward neural networks trained on a bag-of-words (bow) representation. we found that the the results of fine-tuning bert-large (24 layers) could be approximated by a 2-layer bow feedforward neural network, as our best-performing fine-tuned model had an accuracy of 63.9% and our best-performing bag-of-words model had an accuracy of 60.0% on the validation corpus. in the evaluation phase of the competition, our fine-tuned model had an f-score of 69.9% without bagging @xcite and 71.3% with bagging. once the official system rankings were published on april 6, 2020 1 , our best submission (codalab username: gopalanvinay) placed 4th out of 62 entries. all code needed to recreate our results can be found here 2 . ## background the organizers of sentimix simultaneously organized a hinglish and an analogous spanglish (code-mixed spanish-english) task @xcite . we participated in the hinglish track. the task was to predict the sentiment of a given code-mixed tweet. entrants were provided with training and validation data. these datasets were comprised of hinglish tweets and their corresponding sentiment labels: positive, negative, or neutral. besides the sentiment labels, the organizers also provided language labels at the word level. each word is tagged as english, hindi, or universal (e.g. symbols, mentions, hashtags). systems were evaluated in terms of precision, recall and f 1 -measure. all data was provided in tokenized conll format. in this format, the first line (prefaced with the term "meta") provides a unique id and the sentiment (positive, negative or neutral) of the tweet. each tweet is then segmented into word/character tokens and each token is given a language id, which is either hin for hindi, eng for english and o if the token is in neither language. below is an example for a positive-sentiment tweet with id 173: the data contains tokens like usernames (e.g. @beingsalmankhan) and urls, which we deemed unhelpful for sentiment analysis. prior to any training, we converted the conll format into a simpler format, consisting of the reconstituted tweet (omitting usernames and urls) and the sentiment. for the above example, the converted tweet would be: note that we ignore the token-level language ids. ## experimental setup for the experimental results reported in this paper (except for the final system results, which use the official competition test set), we trained our systems using the provided training set of 14k tweets and report results on the provided validation set of 3k tweets. since the task is a relatively balanced three-way classification task, we used simple accuracy as our hillclimbing metric. we used pytorch v1.2.0 foot_1 and python 3.7.1 foot_2 . all additional details needed to replicate our results are provided in the readme of our project's code repository foot_3 . ## fine-tuned bert for our baseline experiments, we used the pytorch-based implementation of bert @xcite provided by huggingface's transformers package @xcite . we adapted the sst-2 task of the run glue.py script. the original task was configured for the binary classification of sentences from the stanford sentiment treebank @xcite , as administered by the general language understanding evaluation (glue) benchmark @xcite . we adapted the task for three-way (positive, negative, neutral) classification. ## bag-of-words models the fine-tuning results suggested two possibilities: to distinguish between these possibilities, we attempted to replicate the performance of the fine-tuned systems using classical bag-of-words (bow) models @xcite . to create a bow system, we iterated through the training data and stored the frequency of each word in the corpus. we then created a vocabulary @xmath0, v 2 , ..., v n }, which was the set of words that appeared with frequency at least k in the training data, for some positive frequency threshold @xmath1. after removing stop words from the vocabulary, we used this vocabulary to transform each tweet into an n-length vector whose j th element equaled 1 if word v @xmath2. we then trained a simple classifier using these bow vector representations of the tweets. for our classifier, we used a standard feedforward neural network with m layers and hidden size h. we built and trained the nns using pytorch @xcite . in order to classify tweets into positive, negative and neutral, the final layer applies the softmax function to the 3-dimensional output of the nn, which normalizes the output into a probability distribution over the three possible sentiments of the input tweet. we used a cross-entropy loss function for training. ## final system submissions because we obtained slightly better performance with the fine-tuned bert model, we used this as the basis for our competition system. to improve its performance, we experimented foot_5 with various hyperparameter settings, including learning rate, weight decay, max grad norm and adam epsilon. our best validation result of 63.8% was yielded by bert-large-cased using a learning rate of 2 × 10 -5 , a weight decay of 0, a max grad norm of 1, and an adam epsilon parameter of 1 × 10 -8 . this system became our first submitted system, achieving an f-score of 69.9% according to the official scoring script. for our second submission, we used bagging @xcite to make our bert classifier more robust. we created 10 bootstrap samples from the given training data. we then trained 10 instances of the bert-large-cased model on these bootstrap samples, and then combined the results of all the models by voting, i.e., if a plurality of the 10 models predicted a particular tweet as positive (negative/neutral), then the tweet's label is deemed as positive (negative/neutral). our bagged system obtained an f-score of 71.3%, placing us 4 th overall in the competition out of 62 entrants. ## conclusion in this paper we investigated sentiment analysis on code-mixed hinglish (hindi-english) tweets as participants of task 9 of the semeval 2020 competition. we implemented two main approaches: 1) applying transfer learning by fine-tuning pre-trained models like bert and 2) training feedforward neural networks on bag-of-words representations. we found that the results of fine-tuning bert-large-cased (24 layers) could be approximated by a 2-layer bow feedforward nn. during the evaluation phase of the competition, our top system obtained an f-score of 71.3%, placing 4 th out of 62 entries in the official system rankings. the fact that we managed fourth place with a system that was little better than a bag-of-words classifier suggests that existing pre-trained vector representations are not particularly good models of code-mixed language. one obvious reason is the difference between the pre-training domain (english wikipedia) and the target domain (code-mixed tweets). while we were initially hopeful that the multilingual bert model released by google might be more effective than the english-only models, it was not. it remains unresolved whether this is because of the formality differences between the two domains, because of the lack of romanized hindi in google's training data, or because language models simultaneously pre-trained on multiple languages do not generalize sufficiently to properly understand code-mixed data.