Early research into emoji in textual communication has focused largely on highfrequency usages and ambiguity of interpretations. Investigation of a wide range of emoji usage shows these glyphs serving at least two very different purposes: as content and function words, or as multimodal affective markers. Identifying where an emoji is replacing textual content allows NLP tools the possibility of parsing them as any other word or phrase. Recognizing the import of non-content emoji can be a a significant part of understanding a message as well. We report on an annotation task on English Twitter data with the goal of classifying emoji uses by these categories, and on the effectiveness of a classifier trained on these annotations. We find that it is possible to train a classifier to tell the difference between those emoji used as linguistic content words and those used as paralinguistic or affective multimodal markers even with a small amount of training data, but that accurate sub-classification of these multimodal emoji into specific classes like attitude, topic, or gesture will require more data and more feature engineering. Emoji characters were first offered on Japanese mobile phones around the turn of the 21st century. These pictographic elements reached global language communities after being added to Unicode 6.0 in 2010, and then being offered within software keyboards on smartphones. In the ensuing half-decade, digitally-mediated language users have evolved diverse and novel linguistic uses for emoji. The expressive richness of emoji communication would, on its own, be sufficient reason to seek a nuanced understanding of its usage. But our initial survey of emoji on Twitter reveals many cases where emoji serve direct semantic functions in a tweet or where they are used as a grammatical function such as a preposition or punctuation. Early work on Twitter emoticons (Schnoebelen, 2012) pre-dated the wide spread of Unicode emoji on mobile and desktop devices. Recent work (Miller et al., 2016) has explored the crossplatform ambiguity of emoji renderings; (Eisner et al., 2016) created word embeddings that performed competitively on emoji analogy tasks; (Ljubešic and Fišer, 2016 ) mapped global emoji distributions by frequency; (Barbieri et al., 2017) used LSTMs to predict them in context. We feel that a lexical semantics of emoji characters is implied in these studies without being directly addressed. Words are not used randomly, and neither are emoji. But even when they replace a word, emoji are used for different purposes than words. We believe that work on emoji would be better informed if there were an explicit typology of the linguistic functions that emoji can serve in expressive text. The current project offered annotators a framework and heuristics to classify uses of emoji by linguistic and discursive function. We then used a model based on this corpus to predict the grammatical function of emoji characters in novel contexts. Although recognizing the presence of emoji characters is trivial, the linguistic distinctions we sought to annotate were ambiguous and seemed prone to disagreement. Therefore in our annotation guidelines we structured the process to minimize cognitive load and lead the annotators to in-136 tuitive decisions. This was aided somewhat by the observation that emoji are often used in contexts that make them graphical replacements for existing lexical units, and that such uses are therefore straightforward to interpret. Taking advantage of such uses, our flow presented annotators with a few simple questions at each step, to determine whether to assign a label or to move on to the next category. The high-level labels we defined for emoji uses were: • Function (func): stand-ins for a function word in an utterance. These had a type attribute with values prep, aux, conj, dt, punc, other. An example from our data: "I like u". • Content (cont): stand-ins for lexical words or phrases that are part of the main informative content of the sentence. These have natural parts of speech, which annotators could subtype as: noun, verb, adj, adv, other. "The to success is " • Multimodal (mm): characters that enrich a grammatically-complete text with markers of affect or stance, whether to express an attitude ("Let my work disrespect me one more time... "), to echo the topic with an iconic repetition ("Mean girls ", or to express a gesture that might have accompanied the utterance in face-to-face speech ("Omg why is my mom screaming so early "). Subtypes: attitude, topic, gesture, other. The POS tags we chose were deliberately coarse-grained and did not include distinctions such as noun sub-types. We wanted to capture important differences while knowing that we would have fewer instances for the function and content labels. For all three labels, annotators were asked to provide a replacement: a word or phrase that could replace the emoji. For func and cont, replacements were a criterion for choosing the label; for mm there was room for interpretation. Tweets were pulled from the public Twitter streaming API using the tweepy Python package. The collected tweets were automatically filtered to include: only tweets with characters from the Emoji Unicode ranges (i.e. generally U+1FXXX, U+26XX-U+27BF); only tweets labeled as being in English. We excluded tweets with embedded images or links. Redundant/duplicate tweets were filtered by comparing tweet texts after removal of hashtags and @men-tions; this left only a small number of cloned duplicates. After that, tweets were hand-selected to get a wide variety of emojis and context in a small sample size -therefore, our corpus does not reflect the true distribution of emoji uses or context types. Our guidelines gave annotators cursory background about emoji and their uses in social media, assuming no particular familiarity with the range of creative uses of emoji. In hindsight we noticed our assumption that annotators would have a fair degree of familiarity with modes of discourse on Twitter. The short-message social platform has many distinctive cultural and communicative codes of its own, not to mention subcultures, and continuously evolving trends combined with a long memory. As two of the authors are active and engaged users of Twitter, we unfortunately took it for granted that our annotators would be able to decipher emoji in contexts that required nuanced knowledge of InterNet language and Twitter norms. This left annotators occasionally bewildered: by random users begging celebrities to follow them, by dialogue-formatted tweets, and by other epigrammatic subgenres of the shorttext form. The analytical steps we prescribed were: • Identifying each emoji in the tweet • Deciding whether multiple contiguous emoji should be considered separately or as a group • Choosing the best tag for the emoji (or sequence) • Providing a translation or interpretation for each tagged span. Eliciting an interpretation serves two goals: first, as a coercive prompt for the user to bias them toward a linguistic interpretation. A replaceable phrase that fits with the grammar of the sentence is a different proposition than a marker that amounts to a standalone utterance such as "I am laughing" 137 or "I am sad". Secondly, one of the eventual applications of annotated corpus may be emoji-sense disambiguation (ESD), and mapping to a lexicalized expression would be useful grounding for future ESD tasks. The text field was very helpful during the adjudication process, clarifying the annotators' judgments and understanding of the task. For each tweet, annotators first read without annotating anything, to get a sense of the general message of the tweet and to think about the relationship between the emoji and the text. On subsequent readings, they are asked to determine whether the emoji is serving as punctuation or a function word; then if it is a content word; and if it is neither of those, then to examine it as a multimodal emoji. A key test, in our opinion, was asking annotators to simulate reading the message of the tweet aloud to another person. If a listener's comprehension of the core message seemed to require a word or phrase to be spoken in place of an emoji, then that would be a compelling sign that it should be tagged as function or content. For each step we provided examples of tweets and emoji uses that clearly belong in each category. These examples were not included in the data set. Uses that failed the first two tests were assigned the multimodal category. We provided guidance and examples for deciding between 'topic', 'attitude' or 'gesture' as subtypes of the multimodal category. Four annotators, all computational linguistics grad students, were given 567 tweets with 878 total occurrences of emoji characters; in the gold standard these amounted to 775 tagged emoji spans. For the first 200 tweets annotated ('Set 1' and 'Set 2' in Table 1 ), each was marked by four annotators. After establishing some facility with the task we divided annotators into two groups and had only two annotators per tweet for the remaining 367. There are two separate aspects of annotation for which IAA was relevant; the first, and less interesting, was the marking of the extent of emoji spans. Since emoji are unambiguously visible, we anticipated strong agreement. The one confounding aspect was that annotators were encouraged to group multiple emoji in a single span if they were a semantic/functional unit. The overall Krippendorff's α for extent markings was around 0.9. The more significant place to look at IAA is the labeling of the emoji's functions. Because we were categorizing tokens, and because these categories are not ordered and we presented more than two labels, we used Fleiss's κ. But Fleiss's κ requires that annotators have annotated the same things, and in some cases annotators did not complete the dataset or missed an individual emoji character in a tweet. In order to calculate the statistics on actual agreement, rather than impute disagreement in the case of an 'abstention', we removed from our IAA-calculation counts any spans that were not marked by all annotators. There are many of these in the first dataset, and progressively fewer in each subsequent dataset as the annotators become more experienced. A total of 150 spans were excluded from Fleiss' kappa calculations for this reason. Content words. Part-of-speech identification is a skill familiar to most of our annotators, so we were not surprised to see excellent levels of agreement among words tagged for part of speech. These content words, however, were a very small proportion of the data (51 out of 775 emoji spans) which may be problematically small. For datasets 3B and 4B, annotators were in perfect agreement. Multimodal. Agreement on multimodal sublabels was much lower, and did not improve as annotation progressed. Multimodal emoji may be inherently ambiguous, and we need a labeling system that can account for this. A smiley face might be interpreted as a gesture (a smile), an attitude (joy), or a topic (for example, if the tweet is about what a good day the author is having) -and any of these would be a valid interpretation of a single tweet. A clearer typology of multimodal emojis, and, if possible, a more deterministic procedure for labeling emoji with these subtypes, may be one approach. Worst overall cross-label agreement scores were for week one, but all following datasets improved on that baseline after the annotation guidelines were refined.
REF found that the multimodal category is the most frequently used; and we contend that emojis used in the multimodal function may also be most likely to demonstrate solidarity.
Varying Linguistic Purposes of Emoji in (Twitter) Context
ACL
Computer Science