S em E val-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets (2020)

Abstract: In this paper, we present the results of the SemEval-2020 Task 9 on Sentiment Analysis of Code-Mixed Tweets (SentiMix 2020). We also release and describe our Hinglish (Hindi-English)and Spanglish (Spanish-English) corpora annotated with word-level language identification and sentence-level sentiment labels. These corpora are comprised of 20K and 19K examples, respectively. The sentiment labels are - Positive, Negative, and Neutral. SentiMix attracted 89 submissions in total including 61 teams that participated in the Hinglish contest and 28 submitted systems to the Spanglish competition. The best performance achieved was 75.0% F1 score for Hinglish and 80.6% F1 for Spanglish. We observe that BERT-like models and ensemble methods are the most common and successful approaches among the participants.

## introduction the evolution of social media texts such as blogs, micro-blogs (e.g., twitter), and chats (e.g., whatsapp and facebook messages) has created many new opportunities for information access and language technologies. however, it has also posed many new challenges making it one of the current prime research areas in natural language processing (nlp). current language technologies primarily focus on english @xcite ), yet social media platforms demand methods that can also process other languages as they are inherently multilingual environments. 2 besides, multilingual communities around the world regularly express their thoughts in social media employing and alternating different languages in the same utterance. this mixing of languages, also known as code-mixing or code-switching, 3 is a norm in multilingual societies and is one of the many nlp challenges that social media has facilitated. ## related work linguists @xcite @xcite studied the phenomena of code-mixing and intrasentential code-switching and found that processing code-mixed language is much more complicated than monolingual text. code-mixing is often found on social media which contains a lot of nonstandard spellings of words and unnecessary capitalization @xcite , making the task more difficult. naturally, the difficulty will increase as the amount of code-mixing increases. to quantify the level of code-switching between languages in a sentence, @xcite introduced a measure called code mixing index (cmi) which considers the number of tokens of each language in a sentence and the number of tokens where the language switches. finding the sentiment from code-mixed text has been attempted by some researchers. @xcite used svm-based classifiers to detect sentiment in tweets and text messages using semantic information. @xcite proposed a skip-gram based word representation model that classifies the sentiment of tweets and provides an extensive vocabulary list for language. @xcite trained lexicon-based document vectors, word embedding, and hybrid systems with the polarity of words to classify the sentiment of a tweet. @xcite attempted shallow parsing of code-mixed data obtained from online social media, and @xcite tried word-level identification of code-mixed data to classify the sentiment. some researchers also tried normalizing the text with lexicon lookup for sentiment analysis of code-mixed data @xcite . to advance research in code-mixed language processing, few workshops have also been conducted. four successful series of mixed script information retrieval have been organized at the forum for information retrieval evaluation (fire) @xcite @xcite . three workshops on computational approaches to linguistic code-switching (calcs) have been conducted which included shared tasks on language identification and named entity recognition (ner) in code-mixed data @xcite @xcite . for our sentimix spanglish dataset, we adopt the sentistrength @xcite annotation mechanism and conduct the annotation process over the unified corpus from the three calcs workshops. ## task description although code-mixing has received some attention recently, properly annotated data is still scarce. we run a shared task to perform sentiment analysis of code-mixed tweets crawled from social media. each tweet is classified into one of the three polarity classes -positive, negative, neutral. each tweet also has word-level language marking. we release two datasets -spanglish and hinglish. we used codalab foot_0 , foot_1 to release the datasets and evaluate submissions. initially, the participants had access only to train and validation data. they could check their system's performance on the validation set on a public leaderboard. later, a previously unseen test set was released, and the performance on the test set was used to rank the participants. only the first three submissions on the test set by each participant were considered, to avoid over-fitting on the test set. the ranking was done based on the best out of the three submissions. there was no distinction between constrained and unconstrained systems, but the participants were asked to report what additional resources they have used for each submitted run. we release 20k labeled tweets for hinglish and ≈ 19k labeled tweets for spanglish. in both the datasets, foot_2 in addition to the tweet level sentiment label, each tweet also has a word-level language label. the detailed distribution is provided in ## dataset the datasets consist of tweets labeled into one of the three classes: both the hinglish and spanglish datasets are released using the previous sentiment label scheme. however, each dataset has been annotated separately as the studies were independent before the organization of this competition. we provide the data collection and annotation details in the following subsections. ## baseline we develop our baseline system using the pre-trained multilingual bert (m-bert; @xcite ). m-bert was trained on 104 languages' entire wikipedia dump and the wordpiece @xcite vocabulary of this model contains 110k sub-word tokens from these 104 languages. to balance the risk of low-resource languages being under-represented or over-fitted due to small training resources during pretraining, exponentially smoothed weighting was performed on the data during pre-training data creation and vocabulary creation. although m-bert was trained on monolingual data from different languages, it is capable of multilingual generalization in code-switching scenarios @xcite . we use the transformers @xcite library to implement our framework and we fine-tune the pre-trained bert-base, multilingual cased model separately for each of the two languages. based on our observation on the training split for each dataset, we set the highest sequence length to 40 and 56 tokens for spanglish and hinglish, respectively. then, we fine-tune the model for three epochs using adamw @xcite optimizer (η = 2e -5 ). ## participation and top performing systems we received an overwhelming response for both hinglish and spanglish. 61 teams submitted their systems for hinglish and 28 teams submitted their systems for spanglish. 16 teams submitted to both hinglish and spanglish. we received 33 system description papers in total. the embeddings and techniques used by the participants are tabulated in ## results and analysis in the previous section, we briefly described the top systems. here, we group and summarize various techniques used by the systems (codalab usernames are mentioned in parentheses) : xlp (liangzhao) used cross-lingual embeddings which could an interesting way for code-mixed language processing where we have scarcity of annotated data. upb (eduardgzaharia, clementincercel) used capsule network with bigru and showed promising results. the use of capsule networks in nlp tasks need further exploration. uld@nuig (koustava) explored an interesting way to phoneme based generative morphemes learning approach. sub-word based embedding is an interesting new way in the nlp community, but what is the best sub-word unit to choose is still unresolved. morpheme based approach could be a good alternative, especially for highly spelling variant code-mixed data. ## conclusion and future work sentimix, sentiment analysis of code-mixed tweets at semeval 2020 received an overwhelming response for both hinglish and spanglish. 61 teams submitted their systems for hinglish and 28 teams submitted their systems for spanglish. the best performance achieved was 75.0 % f1 score for hinglish and 80.6% for spanglish. we received a total of 33 system description papers. bert-like models were the most successful among participants. although the sentimix task mainly focused on sentiment analysis, the data will serve the nlp community or whoever is interested in the code-mixing problem for these particular languages and in general. properly annotated code-mixed data is still scarce. the success of sentimix motivates us to go further and organize similar events in the future. we plan to add more languages, especially from regions that have a high percentage of bi-or multilingual speakers. we also plan to enrich our datasets with annotations for other tasks (ner, emotion recognition, translation etc). we strongly believe that codemixing is a new horizon of interest in the nlp community and needs to be further explored in the future.