CS -Embed at S em E val-2020 Task 9: The Effectiveness of Code-switched Word Embeddings for Sentiment Analysis (2020)

Abstract: The growing popularity and applications of sentiment analysis of social media posts has naturally led to sentiment analysis of posts written in multiple languages, a practice known as code-switching. While recent research into code-switched posts has focused on the use of multilingual word embeddings, these embeddings were not trained on code-switched data. In this work, we present word-embeddings trained on code-switched tweets, specifically those that make use of Spanish and English, known as Spanglish. We explore the embedding space to discover how they capture the meanings of words in both languages. We test the effectiveness of these embeddings by participating in SemEval 2020 Task 9: Sentiment Analysis on Code-Mixed Social Media Text . We utilised them to train a sentiment classifier that achieves an F-1 score of 0.722. This is higher than the baseline for the competition of 0.656, with our team (codalab username francesita) ranking 14 out of 29 participating teams, beating the baseline.

## introduction sentiment analysis (sa), or opinion mining, aims to identify subjective information from text. sa plays a pivotal role in promoting and identifying reception of products in platforms such as amazon, or identifying the sentiment and opinions on numerous topics on social media platforms, such as twitter or facebook. as a consequence, sa is a popular task in the natural language processing (nlp) community. sentiment analysis of social media posts has been a popular research topic, and has naturally led to the sentiment analysis of posts in languages other than english, as well as code-switched posts. written text in which multiple languages co-exist, known as code-switching or code-mixing, is now more abundant thanks to the ample use of social media. recent research on code-switched text has focused on using multilingual word embeddings, however, these embeddings were not trained using code-switched text. in this work, we present word-embeddings trained on code-switched social media text, which serves as our main contribution for semeval 2020 task 9 1 . this task consisted of classifying tweets into one of three classes: positive, neutral, or negative. the dataset used for this task is the one provided by the competition organisers, who collected and annotated the corpus, @xcite . we trained a bilstm @xcite classifier to detect the sentiment of code-mixed tweets by creating our own code-switched embeddings. using the techniques outlined in section 3, large amounts of code-switched text can be collected to train code-switched embeddings for different nlp applications, such as semantic processing, dependency parsing @xcite , as well as sentiment analysis, all of which have been identified as challenges posed by code-mixing in nlp tasks @xcite . we release code for the code-switched sentiment classifier, crawled twitter data, and associated experimental data, including hyper-parameters 2 . ## background to our knowledge, there is a lack of literature and work on code-switched (cs) word embeddings. there is one paper on word embeddings using synthetic code-switched data @xcite . there is, however, considerably more literature on bilingual word embeddings created on parallel monolingual corpora. these are often used in cross-lingual and bilingual tasks @xcite . numerous methods have been introduced for creating bilingual word-embeddings. for instance, canonical correlation analysis (cca) finds the association or correlation between two vectors, @xcite to create bilingual word embeddings. bilingual canonical correlation analysis (bicca) was also developed, it is a technique in which monolingual embeddings are mapped to the same space by use of cca, (m. @xcite . not long after, bicca was extended to the learning of non-linear deep canonical correlation (dcca), which often outperforms linear cca @xcite . this method was followed by the introduction of bilingual skip-gram, in which bilingual representations of words are learned from scratch, @xcite . @xcite proposes learning bilingual sentiment word embeddings (bswe) for english-chinese sa by use of labelled documents and their translation rather than using parallel corpora with results that outperformed state of the art at the time. @xcite proposed to train embeddings individually, in their own monolingual space, and to map one monolingual embedding to the space of the other by use of a linear transformation. this technique makes use of bilingual dictionaries with as little as 25 words. however, recent progress shows that bilingual methods of creating word embeddings are not well-suited for code-mixed tasks, @xcite . indeed, there are grammatical structures that are not captured by monolingual text that exist when code-mixing occurs (j.m. @xcite ). it appears that bilingual embeddings created from monolingual or synthetic data may not be well suited for studies of code-mixed text. by training cs word embeddings, words that are usually used together in the same space when language switching occurs, will be clustered together @xcite . we believe code-switched embeddings will be more robust in classifying cs text than others because the patterns in language when code-switching occurs will be captured in the embeddings. for these reasons, we have decided to train word embeddings on code-switched data. ## system overview and experimental set-up we chose to train our own multilingual word embeddings using code-switched social media text because we hypothesise that these embeddings will be more effective in downstream tasks than combining monolingual embeddings from two different languages. to this end, we collected code-switched tweets in spanglish foot_0 . the external tools utilised for this work are gensim's word2vec model foot_1 , natural language toolkit (nltk) foot_2 , keras version 2.2.4 foot_3 , and tensorflow version 2.0 foot_4 . ## results as previously mentioned, our team decided to train word embeddings with code-switched text as we believed that these embeddings would be more effective than other multilingual embeddings for downstream tasks. our team results and ranking in task 9 are presented in ## conclusion in previous work, multilingual word embeddings have been used to classify code-switched text, however, these embeddings were not trained on code-switched data. in this work, we presented multilingual embeddings trained on code-switched social media text. 10 the results show that code-switched embeddings are able to capture the meanings of words when code-mixing occurs despite having a vocabulary much smaller in size. they also show that for the application of sentiment analysis, these embeddings are successfully employed to train a sentiment classifier, see