Voice@ SRIB at S em E val-2020 Tasks 9 and 12: Stacked Ensemblingmethod for Sentiment and Offensiveness detection in Social Media (2020)

Abstract: In social-media platforms such as Twitter, Facebook, and Reddit, people prefer to use code-mixed language such as Spanish-English, Hindi-English to express their opinions. In this paper, we describe different models we used, using the external dataset to train embeddings, ensembling methods for Sentimix, and OffensEval tasks. The use of pre-trained embeddings usually helps in multiple tasks such as sentence classification, and machine translation. In this experiment, we have used our trained code-mixed embeddings and twitter pre-trained embeddings to SemEval tasks. We evaluate our models on macro F1-score, precision, accuracy, and recall on the datasets. We intend to show that hyper-parameter tuning and data pre-processing steps help a lot in improving the scores. In our experiments, we are able to achieve 0.886 F1-Macro on OffenEval Greek language subtask post-evaluation, whereas the highest is 0.852 during the Evaluation Period. We stood third in Spanglish competition with our best F1-score of 0.756. Codalab username is asking28.

## introduction semeval task-9 sentimix @xcite is divided into two tasks, one for hinglish (hindi-english) and the other for spanglish (spanish-english) code-mixed subtasks. in the spanglish task, the dataset contains tweets in spanglish (spanish-english) code-mixed language, and it is labeled into three categories positive, negative, and neutral sentiments. the task is to classify codemixed tweets into these three sentiments. semeval task-12 offenseval @xcite is divided into different subtasks, english @xcite , danish (sigurbergsson and derczynski, 2020), arabic @xcite , turkish (c ¸öltekin, 2020), and greek @xcite languages. english task is divided into three subtasks-a, b, and c. subtask-a of offenseval is offensive language identification, subtask-b is categorization of offensive types into targeted and untargeted, and subtask-c is offensive target identification as individual, group, or other. in the last decade, there has been proliferation in the use of social media web sites. it has led to pervasive use of hate inducing speech and offensive language to express opinions. the use of profane language has been growing in face-to-face interactions as well as online communications in recent years. the anonymity provided by these websites and lack of stringent action has led to adoption aggressive behavior by people.youth who experienced cyberbullying, as either an offender or a victim, had more suicidal thoughts and were more likely to attempt suicide than those who had not experienced such forms of peer aggression @xcite . hence it's necessary to auto-remove offensive and profane language in an online environment. since the inflow of such type of content is huge, manual filtering is time-consuming and requires much manual labor; hence it becomes almost impractical to do manual filtering. due to this reason, researchers have proposed methods to automate filtering process by training machine learning models in pre-annotated datasets hate speech and offensive language by @xcite , cyberbullying @xcite and detection of racism by @xcite . in this work (team name is srib2020), we try to classify twitter tweets in different languages codemixed for sentimix tasks and monolingual tweets in different languages in the offenseval task into different classes. in offenseval tasks, tweets are classified as offensive or non-offensive, whereas in the sentimix task, tweets are classified as positive, negative, and neutral. in the sentimix task, the "neutral" class is bit ambiguous as many positive tweets in the dataset are labeled as neutral, and negative tweets are labeled as neutral. class "neutral" has a very thin boundary with the other two classes, "positive" and "negative". the generated representations are used as inputs to a dual encoder network, consisting of two different bilstms -the collective and specific encoder. the collective encoder captures the overall sentiment of the sentence, while the specific encoder utilizes an attention mechanism to focus on individual sentiment-bearing sub-words. @xcite have annotated the data, developed a language identifier, a normalizer, a part-of-speech tagger, and a shallow parser for sentiment analysis of code-mixed data. @xcite used a lexicon lookup approach to perform domain-specific sentiment analysis. @xcite introduce learning sub-word level representations in lstm (subword-lstm) architecture instead of character-level or word-level representations; this enables to learn the information about sentiment value of meaningful morphemes. @xcite uses the shared parameters of siamese networks to map the sentences of code-mixed and standard languages to a common sentiment space. they introduce a primary clustering-based preprocessing method to capture variations of code-mixed transliterated words. supervised learning techniques for hate detection, offensive detection, and target and sentiment classification on social media datasets have been explored in recent times. @xcite described a way of multi-class classification of offensive language and hate speech in tweets, using svm, random forest, naive bayes, and logistic regression. @xcite reported performance for a simple lstm classifier not better than an ordinary svm, when evaluated on a small sample of facebook data for only two classes (hate, no-hate), and three different levels of strength of hatred. @xcite propose a detection scheme that is an ensemble of recurrent neural network (rnn) classifiers. it incorporates various features associated with user-related information, such as the users' tendency towards racism or sexism. this paper can be summarised into five key points-1. applied a variation of focal loss by applying class weight along with gamma parameter in the loss function. ## experiments and results we performed our experiments on three deep-learning models cnn, lstm, and ensemble of cnn, lstm, and self-attention. in spanglish, we achieved f1-macro of 0.770, precision of 0.749, and recall of 0.803 with ensemble model on pre-processed data, whereas it was 0.709, 0.755 and 0.672 respectively for raw text without pre-processing of the data. in hinglish challenge, ensemble model out performed other models on pre-processed dataset. ensemble model on pre-processed data achieved f1-score of 0.682, precision of 0.695 and recall of 0.679 whereas same model when trained on raw data achieved 0.665, 0.681 and 0.665 respectively. from these results we can infer that cleaning steps involved helped to improve the results . we have also experimented our models with and without pretrained embeddings in hinglish task but it did not help in improving the scores. we trained hinglish embeddings by collecting code mixed hinglish data from various sources such as blogs and scraping twitter data using fasttext library @xcite . post evaluation on spanglish task was not performed since gold labels were not released after the competition. error analysis of hinglish and spanglish tasks is presented in appendix a and b respectively. bert multilingual and bert-uncased mode @xcite are trained and fine-tuned by adding three delta layers (dense) layers on top of pre-trained models. we trained bert model using both ways one by freezing bert pre-trained parameters and other by keeping parameters as trainable during the complete training process. from our experiments on hinglish and spanglish datasets using these models and techniques foot_6 we found that bert-uncased model performed better than the bert-multilingual model. keeping the pre-trained parameters of bert as trainable during the complete process performed better than freezing the bert parameters during fine-tuning. we attribute this behavior of bert to the difference in data distributions of bert pre-training and sentimix tasks. in offenseval tasks, we performed post-evaluation experiments by tuning only few hyper-parameters in the model like changing class-weights and changing loss function. in greek language subtask our model foot_7 ## conclusion and future work in this paper, we present description of the system that we have used in all offenseval and sentimix tasks. with our best model we were able to achieve third position in spanglish task in evaluation period. in post evaluation experiments our model is able to achieve f1-score more than the highest score in greek task evaluation period. from our experiments, we have found that pre-processing steps played a huge role in increasing f1-scores. since we have used deep learning models, our model could not perform very well in the tasks where dataset was small like in english subtask-c. in this work paper we present different data pre-processing steps that played important role. in english subtasks we experimented with pre-trained embeddings 11 trained in twitter corpus, and found that pre-trained embeddings helped to increase f1-score in english sub-tasks but did not help in hinglish task. the results obtained through our experiments in test data are lower than the results obtained in development set data. we inferred from our experiments that f1-score partly depends upon data distribution of different classes in training and development data which is used to tune hyper-parameters. in most of the tasks, data is not distributed equally among different classes. exploratory data analysis reveals that there is huge difference in class distribution in the datasets. our system presents a solid baseline for sentiment analysis of code-mixed languages and offensiveness detection in multiple languages. in our future work, we plan to add handcrafted features along with current features and train it on different machine learning models. we also plan to explore techniques of data augmentation as deep learning models need large amount of data to train. corpus used to train code-mixed language models and languages other than english is very small as compared to corpus used to train english language models. lot of research needs to be done in this direction.