Multi-resolution Annotations for Emoji Prediction (2020)

Abstract: Emojis are able to express various linguistic components, including emotions, sentiments, events, etc. Predicting the proper emojis associated with text provides a way to summarize the text accurately, and it has been proven to be a good auxiliary task to many Natural Language Understanding (NLU) tasks. Labels in existing emoji prediction datasets are all passage-based and are usually under the multi-class classification setting. However, in many cases, one single emoji cannot fully cover the theme of a piece of text. It is thus useful to infer the part of text related to each emoji. The lack of multi-label and aspect-level emoji prediction datasets is one of the bottlenecks for this task. This paper annotates an emoji prediction dataset with passage-level multi-class/multi-label, and aspect-level multi-class annotations. We also present a novel annotation method with which we generate the aspect-level annotations. The annotations are generated heuristically, taking advantage of the self-attention mechanism in Transformer networks. We validate the annotations both automatically and manually to ensure their quality. We also benchmark the dataset with a pre-trained BERT model.

## introduction emojis have become crucial components of written language. emojis were initially designed to express emotions or feelings, e.g., for a smiley face, and they have grown to be a large family of over 2,000 icons over the years which can express not only emotions but a wide range of objects or actions, e.g., for a gift and for celebrations. compared to words, emojis have the merit of preserving information more densely. for example, carries the same meaning as the phrase "laughing with tears in eyes". additionally, the byte-level encoding of subtle linguistic expressions makes it easier to discriminate complicated feelings, e.g., the bond between and is clearly weaker than their phrasal explanations "laughing with tears in eyes" and "crying loudly" due to the similarity between "tear" and "crying". these characteristics of emojis aid in accurate summarization of text, thus benefiting natural language understanding (nlu) tasks. @xcite define the emoji prediction task by finding the most appropriate emoji(s) summarizing a piece of text. they also show with experiments that language representations learned on the emoji prediction task can boost the performance of emotion recognition, sentiment analysis, and sarcasm detection tasks. consequently, using emoji prediction as a bridge to solve other natural language processing (nlp) tasks appears to be effective and promising. however, the emoji prediction task is yet far from being well established. first and foremost, as a classification task, there is not a set of labels agreed upon by previous research. to the best of our knowledge, all the existing papers on emoji prediction use either a handcrafted emoji set @xcite or the most frequent emojis in their individual datasets @xcite . handcrafted emoji sets are usually limited in size and topics (usually limited to emotional emojis), while frequency-based emoji sets are dataset-specific. the lack of a standard label set makes it difficult to evaluate and compare emoji prediction models, hampering the research on emoji prediction and its interactions with other nlp tasks. to solve this problem, we use an emoji list from the unicode office 1 as the label set for the emoji prediction task. this emoji list includes 1,467 emojis in total, ordered by the median frequency of their use from multiple resources. we believe using this emoji list is good for standardizing the task since it is open to all researchers and is not influenced by how we sample the data. the second problem with emoji prediction is that existing labeled datasets are either too small in scale or not publicly available. this often results from the policy of social media platforms on using their data and the constantly changing nature of posts on these platforms, e.g., post deletion and edits. to address the problem of data unavailability or expiration, we annotate the pan-19 celebrity profiling corpus @xcite , a tweetbased corpus, which is large and available to all researchers. we provide three types of annotations in this paper. existing emoji prediction datasets are almost all annotated on the passage-level under the multi-class classification setting, which means each record contains exactly one tweet and one emoji. while we also release this type of annotation, we additionally provide passage-level multi-label and aspect-level multi-class classification annotations. annotations for the passage-level multi-label classification setting are similar to the multi-class setting, but with possibly multiple emojis in each record (i.e., a tweet could be associated with multiple emojis). we introduce aspect-level labels to the emoji prediction task to enable a finer-grained analysis of the functions of emojis in tweets. each emoji in these annotations points to a span of its corresponding text instead of the entire tweet. text fractions associated with different emojis in the same tweet may overlap with each other. given the large size of our dataset, all three types of annotations are generated automatically using heuristics or with the help of a transformer-based model. the assumption underlying the passagelevel annotations is that the text fully covers the meanings of emojis in a tweet. thus we extract the emojis appearing in the text as passage-level labels, as @xcite do. under the multiclass classification setting, a record is duplicated and assigned different emojis if it contains multiple emojis. the aspect-level annotations are created based on passage-level multi-class classification labels. since the attention maps in a transformerbased model reflect the interrelations of each word pair, we are able to evaluate the contribution of each word to a predicted emoji under the multi-class classification setting. we then combine the labels based on tweets to form the aspect-level multi-class annotations for the dataset. we will introduce the annotation methods in more detail in section 3.2. the contributions of this paper are three-fold. first, we provide a large emoji list to be used as a label set for the emoji prediction task. these emojis are all frequently-used and meaningful, benefiting further research on the emoji prediction task and its connections to other nlp tasks. second, we introduce a data annotation method based on the self-attention mechanism in transformer networks @xcite . the method is designed specifically for annotating aspect-based labels and can potentially be used on any nlp task. third, we provide three types of annotations for emoji prediction based on a publicly available tweet dataset. besides the commonly used tweet-level foot_1 multi-class classification labels, our annotations include passage-level multi-label and aspect-level multi-class classification labels for better understanding of the linguistic roles of emojis. we release a carefully curated (both manually and automatically) emoji prediction dataset based on the 64 top-ranked emojis in our emoji list. 3 2 related work the study of emoji usage in textual data has seen a rise in recent years. most related research are done over twitter, gab, or microblog data since the use of emojis is more common on social media. @xcite compared the way emojis were used in twitter and gab posts, and they claimed that emojis with negative sentiment scores were more frequently used on gab than the other. the use of emojis on twitter also appeared to be more balanced compared to gab in posts related to the same event, i.e., the most frequent emoji counts for 19.79% of total emoji usage on gab and 6.28% on twitter. we base our research on a twitter dataset for the balanced emoji usage and more neutral points of view. since the amount of unique emojis was large, early research treated emojis as special word-level tokens and examined the linguistic roles with coarse-grained classification objectives, for example, predicting whether an emoji was used redundantly in its context @xcite or classifying the linguistic purposes of emojis @xcite . with the help of recurrent neural networks (rnns), @xcite and @xcite could predict proper emojis from tweet posts. going one step further, @xcite combined image and text features for predicting emojis. due to the overly large amount of data required to train neural-networkbased classifiers with tens or even hundreds of labels, most emoji prediction datasets were labeled automatically, using heuristics or pre-defined rules. the logic previous researchers used to label these datasets were simple, assuming that every emoji appearing in a social media post qualified as a label of the text. our work also relies on this assumption since the data we use consists of tweets posted by authorized accounts, who are not likely to often use emojis arbitrarily or randomly. we also extend the annotation method to be able to generate more complex, aspect-level annotations automatically. additionally, it is commonly agreed that emojis are closely related to emotions, sentiments, sarcasm, irony, etc. @xcite showed that the language representations learned from the emoji prediction task were useful in emotion recognition, sentiment analysis, and sarcasm detection tasks. @xcite designed experiments to show the interconnections between emoji usage and ironic expressions. @xcite also evaluated the influence of emojis on irony detection and sentiment analysis tasks, but they replaced emojis with descriptive text in this process. based on previous knowledge about emojis and the emoji prediction task, we also use sentiment analysis, emotion recognition, and formality classification tasks to validate the quality of our annotations in this paper. ## annotation method we annotate tweets for the emoji prediction task in this paper. for clarity, we refer to each tweet in the dataset by @xmath0, w 2 , ..., w n } of length n and an emoji set @xmath1, e 2 , ..., e m } of size m, where w i is the i -th word in t and e j is the j -th emoji in e. the three settings of this task are formally defined as follows. passage-level multi-class classification: predict the best e j most closely related to t. passage-level multi-label classification: predict whether each e @xmath2. aspect-level multi-class classification: given t and p subsets of t, each denoted as s @xmath3, w 2 , ..., w k } where w @xmath4, predict the best e j mostly closely related to s q . this is an extension to our earlier work @xcite in which tweets are labeled on passage-level only, and no official emoji list is used to standardize the annotations. the passage-level classification datasets are annotated directly using the emojis appearing in each tweet. under the multi-class classification setting, we duplicate the tweets if they are bond to multiple emojis and use one emoji to label each copy. for the aspect-level annotations, each record is identified by a tweet and an aspect text piece. we generate the aspect-level annotations with the help of a transformer-based model. emojis are removed from the text in all three settings in case that neural models trained on our datasets directly copy and paste the emojis from text into predictions. ## emoji prediction dataset one of the goals of this paper is to annotate the pan-19 celebrity profiling dataset for emoji prediction. we refer to the newly-annotated dataset as multi-resolution emoji prediction (mrep) dataset since it contains both passage-level and aspect-level annotations. when releasing the data, we randomly split the dataset into train, dev, and test datasets with 80%, 10%, and 10% of the data amount, respectively. the random seed we use for this separation is 29936. ## conclusion and future work emoji prediction has become a popular task in the nlp community, but the lack of publicly available large-scale datasets with high-quality annotations remains a bottleneck for this task. in this paper, we annotated a publicly available twitter dataset for the emoji prediction task. we designed an annotation method for aspect-level annotations using the self-attention mechanism in transformer networks. this method showed great performance in labeling our dataset, and can potentially be used in other tasks as well. our dataset contains three types of annotations, namely the passage-level multi-class and multi-label classification labels, and the aspectlevel multi-class classification annotations. we validated our annotations both automatically and manually to ensure their quality. we also benchmarked our dataset using a pre-trained bert-largecased model. our labeled datasets are available upon request. there are two main paths for extending this work. first, the aspect-level annotation method can be applied to other nlp tasks. second, our annotations in the emoji prediction dataset can be enhanced by including an enriched label set.