Kk2018 at S em E val-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification (2020)

Abstract: Code switching is a linguistic phenomenon which may occur within a multilingual setting where speakers share more than one language. With the increasing communication between groups with different languages, this phenomenon is more and more popular. However, there are little research and data in this area, especially in code-mixing sentiment classification. In this work, the domain transfer learning from state-of-the-art uni-language model ERNIE is tested on the code-mixing dataset, and surprisingly, a strong baseline is achieved. And further more, the adversarial training with a multi-lingual model is used to achieved 1st place of SemEval-2020 Task9 Hindi-English sentiment classification competition.

## introduction in today's society, the use of social media has become a necessary daily activity. as a result, a large amount of text content has been produced. among those corpora, there are many code-mixed texts. code-mixing is a common phenomenon in societies in which two or more languages are used. it is the change of one language to another within the same utterance or in the same oral/written text @xcite . code-mixing poses several unseen difficulties in nlp tasks on lexical/syntax/semantic levels. in recent years, a lot of models based on transformer @xcite have been proposed such as bert @xcite , roberta @xcite and albert @xcite . those models take advantage of both available large corpora and computational power and have outperformed the traditional techniques in various sentiment analysis tasks @xcite . meanwhile, many multi-lingual models are proposed, e.g. @xcite . @xcite have tested the multilingual bert with hierarchical attentive network, shakeel and karim (2020) have tested bert for code-mixing informal short text classification. however, the code-mixed corpus is small. large models like bert @xcite and ernie @xcite have millions of parameters, and are prone to overfit. to tackle this problem, adversarial training can be used as a regularizer in the training phase @xcite . adversarial examples are a way of fooling a neural network to behave incorrectly @xcite . they are created by applying a small perturbation to the original inputs. the original algorithm was designed for use in the image field, and the perturbation added to the image was deliberately designed to be invisible to the naked eye, but causing neural networks to output a completely different response from the true one. by introducing those adversarial examples, which the neural nets make mistakes on, to the network during training, the performance can be improved. this is called "adversarial training" which acts as a regularizer to help the network generalize better @xcite . for natural language, it is not feasible to produce perturbed examples due to the nature of the text. as a solution, @xcite apply this perturbation to the embedding space and design a new loss for semi-supervised adversarial training. our contributions are twofold. first, the domain transfer learning method is applied for code-mixing classification, which surprisingly shows promising efficacy. and furthermore, the adversarial training is applied during the training process with a multilingual model. finally, this method has achieved the state-of-the-art score and rank 1st in semeval-2020 task 9 for code-mixing hindi-english sentiment classification. ## related work in this section, the existing research on multi-lingual sentiment classification and adversarial learning is discussed, which are closely related to this work. ## model in this section, we describe the two models that were used in this competition are proposed. the first one is ernie proposed by @xcite which uses the paradigm of continuous learning to pretrain language models which are adapted to sentiment classification using domain transfer learning. the other one is a multi-lingual model xlm-r and we have further trained it with adversarial training. ## experiments implementation details there are 14000/3000/3000 examples in the train/dev/test dataset respectively, for more detailed dataset information please refer to the work of @xcite . to train a better model, data cleaning strategies have been applied, e.g. the urls, hashtags, usernames are all removed. when processing the data, to increase the data diversity, we tried both retaining and discarding case information. to fully make use of the data provided by the competition, the spanish-english code-mixing dataset is combined with hindi-english dataset, however, it shows a negative effect with training those two datasets together. we performed all our experiments on a gpu (nvidia tesla v100) with 32 gb of memory. except for the code specific to our model, we adapted the codebase utilized by xlm-r. to carry out the ablation study of adversarial training, batches of 32 were specified. for optimization, the adam optimizer @xcite with a learning rate of 3e -5 was used. also, we have applied the linear warm strategy for the learning rate. we set α = 1 for the adversarial loss. the experiments are conducted for 5 epochs and we pick the best model using the dev set. improving generalization as the adversarial training can be a regularizer to model, it is expected to have a more robust generalization on both development and test set. from figure 1 it can be seen that the hyperparameter is important. @xmath0, we get the best f1 score for the development dataset, and furthermore, the scores for this parameter are better than the baseline most of the time (green line in figure 1 ). from ## conclusion in this paper, we tested the performance of ernie for domain transfer learning and applied adversarial to code-mixed sentiment analysis with xlm-r. the experiments show that ernie provides a strong baseline in the cross-lingual setting. after replacing the backbone model with the multilingual model xlm-r, the f1-score of sentiment classification was significantly improved, further improved after the adversarial learning was applied. in particular, we have carried out experiments on hyperparameters in adversarial learning. different parameters have a significant influence on the experimental results, but they all achieve the purpose of enhancing the generalization of the model. as a result, we have achieved the top rank for the competition of hindi-english code-mixed sentiment classification and our user name is kk2018. as future work, other white-box adversarial examples as well as black-box ones will be utilized to compare adversarial training methods for various sentiment analysis tasks.