LT 3 at S em E val-2020 Task 9: Cross-lingual Embeddings for Sentiment Analysis of H inglish Social Media Text (2020)

Abstract: This paper describes our contribution to the SemEval-2020 Task 9 on Sentiment Analysis for Code-mixed Social Media Text. We investigated two approaches to solve the task of Hinglish sentiment analysis. The first approach uses cross-lingual embeddings resulting from projecting Hinglish and pre-trained English FastText word embeddings in the same space. The second approach incorporates pre-trained English embeddings that are incrementally retrained with a set of Hinglish tweets. The results show that the second approach performs best, with an F1-score of 70.52% on the held-out test data.

## introduction the emergence of web 2.0 has allowed people to easily share their opinion on a variety of topics. whereas in the past companies and policy makers used to conduct surveys to know the opinion of people on certain products, services or policies, they now have access to a wide range of easily accessible data to gather the public's sentiment @xcite . to automatically derive opinions from text, researchers have designed the task of sentiment analysis (sa), which deals with "the computational study of opinions, sentiments and emotions expressed in text" @xcite . an important challenge when applying sentiment analysis to usergenerated data is caused by code-mixing and non-standard language use. in linguistics, code-mixing traditionally refers to the embedding of linguistic units (phrases, words, morphemes) into an utterance of another language @xcite . the phenomenon of code-mixing frequently occurs in spoken languages, such as the combination of english with hindi (so-called "hinglish"), or english with spanish (so-called "spanglish"). more recently, code-mixing is increasingly being used in written text as well, as non-native english speakers often combine english with their mother tongue when using social media. in the case of hinglish, an additional challenge is added because people do not only mix languages, but also use english phonetic typing to write hindi words instead of using the devanagari script. in order to investigate sentiment analysis for code-mixed social media text, patwa et al. ( 2020 ) have organized the sentimix task, which consists in predicting the sentiment of a given code-mixed tweet. the sentiment labels are positive, negative, or neutral, and the code-mixed languages are english-hindi and english-spanish. besides the sentiment labels, the authors also provide language tags at the word level, being en (english), spa (spanish), hi (hindi), mixed, and univ (e.g., symbols, @ mentions, hashtags). this paper presents our research performed for the "hinglish" (english-hindi) sentiment analysis subtask of semeval-2020 task 9. the remainder of this paper is structured as follows. in section 2, we provide an overview of the related research. section 3 describes the data used to train and evaluate the system. section 4 introduces the two approaches developed to perform hinglish sentiment analysis, while section 5 discusses the results obtained for the task. section 6 concludes this paper. ## related research a first line of research for sentiment analysis applies supervised machine learning approaches @xcite . these approaches, however, require large amounts of labeled data, which are often lacking for low(er)-resourced languages. another important line of research uses machine translation systems to (1) map subjectivity lexicons to other languages @xcite or to (2) transfer sentiment information from a high-resource source language to a low-resource target language @xcite . @xcite use annotation projection to project supervised labels from the source languages to the target language and a direct transfer approach to develop sa systems. more recently, researchers have started to investigate cross-lingual embeddings for the task of sa. the idea of these embeddings stems from the idea of @xcite that vector spaces in different languages share a certain similarity. by creating monolingual spaces and then learning a projection from one language to another, there is no need for large parallel corpora. @xcite learn a linear mapping from one space to another and optimize the performance by using the most common words from both languages and by using a bilingual lexicon. as large bilingual lexicons are often not available, there was a need to either completely eliminate or drastically reduce the size of the required bilingual lexicon. to address this issue, @xcite propose a very simple self-learning approach that exploits the structural similarity of embedding spaces, and works with as little bilingual evidence as a 25 word dictionary or even an automatically generated list of numerals. research by @xcite attempts to learn bilingual sentiment embeddings, which jointly train the projection and the sentiment component to represent sentiment information in the source and target language. their method uses a bilingual lexicon, an annotated sentiment corpus in the source language and monolingual embeddings for the source and target language. their experimental results show the need for a dedicated high-quality sentiment lexicon in order to achieve a satisfactory performance. more recently, transformer-based approaches @xcite have been used for cross-lingual knowledge transfer. these approaches, however, require significant pretraining and a lot of low-resource languages are not accounted for in the pretrained models. applying sentiment analysis to code-mixed social media data, however, offers a number of challenges for standard nlp approaches. these approaches are usually trained on large monolingual corpora (e.g. english or hindi), and not on mixed data. in addition, social media language is characterized by informal language use (abbreviations, spelling mistakes, flooding, emojis, etc.), which causes a considerable drop in performance for standard nlp approaches that are trained on standard data @xcite . related research on computational models for code-mixing is scarce because of the lack of large code-mixed resources, which makes it hard to apply data-greedy approaches. seminal work in sentiment analysis (sa) of hindi text was done by @xcite , who introduce a hindi-english code-mixed dataset for sentiment analysis and propose a system to sa that learns sub-word level representations in lstm instead of character-or word-level representations. @xcite compare three bilingual word embedding approaches to perform code-mixed sentiment analysis and part-of-speech tagging. their results show that the applied bilingual embeddings do not perform well, and that multilingual embeddings might be a better solution to process code-mixed text. this is mainly because code-mixed text contains particular semantic and syntactic structures that do not occur in the respective monolingual corpora. recently, there is a lot of attention for nlp approaches on code-mixed data, as illustrated by the "fourth workshop on computational approaches to linguistic code-switching" foot_0 . in the proposed research, we experimented with two different approaches to tackle sentiment analysis for hinglish: (1) an approach using cross-lingual embeddings resulting from projecting hinglish and english embeddings in the same space, and (2) an approach incorporating pre-trained english embeddings that are incrementally retrained with hinglish information. ## system description hinglish is an amalgamation of english and transliterated hindi. however, since resources on code-mixed hindi are very limited, we have to find alternative ways to obtain supervision for understanding codemixed hindi text and ideally combine the information with already available resources for english. we approached the task of analysing hinglish code-mixed text from two different angles: ## results and discussion as can be seen from a more detailed overview of the precision and recall scores for the best performing cnn classifiers is presented in an error analysis has shown that there is still a lot of room for improvement. the fasttext embeddings are certainly not perfect due to the limited amount of tweets collected. frequent words like ham (english: we) and bharat (english: india) were well represented in the scraped tweets, whereas rarer words like abhigyaan (english: knowledge source) and kanoon (english: law) had few occurrences, thus diminishing the quality of the fasttext embeddings that were trained based on this corpus. ## conclusion in this paper we demonstrate that it is possible to create a sentiment analysis system for hinglish by a) treating it as an independent language and b) treating it as an extension of english with additional vocabulary. both models beat the task baseline convincingly. in addition, we achieve these results without using the language labels provided for every word in the task, thus demonstrating that these methods can be employed with real-world data and can be scaled to any code-mixed language in general. in future research, it would be interesting to evaluate the performance of these models on other code-mixed tasks. it would also be worthwhile to use contextual embeddings like bert and xlm, since these methods have significantly outperformed conventional word embeddings in all multilingual nlp tasks.