Emoji-Based Transfer Learning for Sentiment Tasks (2021)

Abstract: Sentiment tasks such as hate speech detection and sentiment analysis, especially when performed on languages other than English, are often low-resource. In this study, we exploit the emotional information encoded in emojis to enhance the performance on a variety of sentiment tasks. This is done using a transfer learning approach, where the parameters learned by an emoji-based source task are transferred to a sentiment target task. We analyse the efficacy of the transfer under three conditions, i.e. i) the emoji content and ii) label distribution of the target task as well as iii) the difference between monolingually and multilingually learned source tasks. We find i.a. that the transfer is most beneficial if the target task is balanced with high emoji content. Monolingually learned source tasks have the benefit of taking into account the culturally specific use of emojis and gain up to F1 +0.280 over the baseline.

## introduction many natural language processing (nlp) tasks suffer from a lack of available data. this is especially true for sentiment tasks, such as hate speech (hs) detection, which depend on the availability of manually annotated data. when moving to languages other than english, many sentiment tasks quickly become very low-resourced. on the other hand, noisy social media content is available in abundance and many sentiment tasks are based on user comments on such platforms. emojis can be a valuable source for the distant supervision of sentiment tasks, as they correlate with the underlying emotion of a comment. in this study, we aim to exploit the emotional information encoded in emojis to improve the performance on various sentiment tasks using a transfer learning approach from an emoji-based source task (st) to a sentiment target task (tt). previous work has focused on the transfer from predicting single emojis @xcite or strictly pre-defined emoji-clusters @xcite . however, predefined emoji clusters do not take into account the culturally diverse usage of emojis @xcite . we therefore introduce datadriven supervised and unsupervised emoji clusters and compare these with single emoji prediction tasks. specifically, we analyze the efficacy of the transfer from a single emoji or (un)supervised emoji cluster prediction st to a sentiment tt under three conditions, i.e. i) low vs. high amount of emoji content present in tt, ii) balanced vs. unbalanced label distribution in tt and iii) monolingually or multilingually learned st. the first two conditions are based on typical qualities of sentiment corpora, which tend to be unbalanced in their label distribution with varying degrees of emoji content depending on the source of the data. the third condition is relevant for languages for which a tt is low-resource and which might benefit from a multilingually learned st. in section 2 we give an outline of related work, followed by the introduction of our method (section 3). the experimental setup in section 4 details the data and models used as well as the (un)supervised clusters generated. in section 5 we describe our results and conclude in section 6. ## related work emojis have been used as a type of distant supervision using pre-defined emotion classes based on psychological models @xcite , binary (positive/negative) classes @xcite or a set of single emojis @xcite . however, such pre-defined emoji classes often do not account for the culturally diverse use of emojis @xcite . in contrast, our work does not pre-define the emotion classes found in emojis and instead learns these classes, or clusters, from the data itself. while our and the above approaches focus on exploiting emojis as additional labelled data, e.g. in a transfer setting, emoji embeddings @xcite have been used as additional features in downstream tasks such as sarcasm detection @xcite . transfer learning has recently been driven by transformer-based @xcite language models (lm) such as bert @xcite or xlm-r @xcite . when learning a source task on these models, the representations in the encoder change to become informative to the task at hand. in a parameter transfer setting, a new but related target task then profits from the learned representations in the encoder. transfer learning has been applied to sentiment analysis (sa) using parameter transfer methods such as pretrained sentiment embeddings @xcite or machine translation-based context vectors @xcite . our approach forms part of the parameter transfer approach, as we use encoder representations learned using emoji-based source tasks and transfer these to sentiment target tasks. hate speech classification and sentiment analysis have in recent years been the object of many shared tasks @xcite @xcite @xcite . classification models for these tasks often rely on feature engineering and statistical methods such as naivebayes @xcite , logistic regression over subwords @xcite or neural approaches including convolutional neural networks @xcite or, as in our case, the representations of large lms @xcite . ## method: emoji-prediction for our parameter transfer, we rely on a single transformer-based lm which is shared among different tasks. a sequence @xmath0. a linear layer is then used as a predictive function f : @xmath1. a task @xmath2, f (x)} is then a set of labels y and the predictive function f over the instances in x. we follow a transfer learning approach, where source task t s is an emoji-based classification task, i.e. given a sequence, predict the emoji (class) that it originally contained. target task t t is a downstream task such as sa or hs (section 4.1). each task has its own set of instances x, labels y and predictive function f , while the feature-generating lm stays the same. the error of predictor f is back-propagated to the lm, which allows us to transfer learned parameters from t s to t t . ## experimental setup we describe the data used for the sts and tts respectively (section 4.1), followed by the specifications of the encoding lm (section 4.2) and the emoji cluster creation (section 4.3). ## results we train each model over 10 seeded runs and report the averaged macro f1 with standard error (figure 2 ). for each tt, we train a baseline, which is the same pre-trained bert-{de,m} model that is now fine-tuned directly on the tt classification task at hand, without prior training on the st. we compare these baselines with those models that have undergone a transfer from st to tt. we use the term equivalent to signify that two models lie within each others error bounds. ## summary we have evaluated and identified conditions under which the transfer from an emoji-based st is beneficial for a sentiment tt. in the experiments in section 5 we observed three major trends, namely i) tts with high amounts of emoji content benefit more from the transfer, ii) pmi-target tends to be detrimental to unbalanced tts and iii) monolingually learned sts tend to perform better than their multilingual counterparts, due to their improved representation of culturally unique emoji usages. the latter underlines the importance of taking into account cultural differences when exploiting the information encoded in emojis. from these results, we can draw conclusions about the conditions under which a given emojibased st is beneficial. due to the shared information between emojis and slurs, pmi-swear is beneficial to hs tasks when the data that can be generated from the swear word list is decently large. pmi-target is beneficial when the tt is balanced, otherwise it replicates the already existing class imbalance. unsupervised kmeans-{2,3} should be learned monolingually to be beneficial and ep is a safe choice for tts with high emoji content.