Unleashing the Power of Emojis in Texts via Self-supervised Graph Pre-Training (2024)

Abstract: Emojis have gained immense popularity on social platforms, serving as a common means to supplement or replace text. However, existing data mining approaches generally either completely ignore or simply treat emojis as ordinary Unicode characters, which may limit the model’s ability to grasp the rich semantic information in emojis and the interaction between emojis and texts. Thus, it is necessary to release the emoji’s power in social media data mining. To this end, we first construct a heterogeneous graph consisting of three types of nodes, i.e. post, word and emoji nodes to improve the representation of different elements in posts. The edges are also well-defined to model how these three elements interact with each other. To facilitate the sharing of information among post, word and emoji nodes, we propose a graph pre-train framework for text and emoji co-modeling, which contains two graph pre-training tasks: node-level graph contrastive learning and edge-level link reconstruction learning. Extensive experiments on the Xiaohongshu and Twitter datasets with two types of downstream tasks demonstrate that our approach proves significant improvement over previous strong baseline methods.

## introduction emojis have gained significant attention due to their popularity in digital communication, especially on social media platforms. they convey emotions and reactions, supplementing or replacing text in posts, which adds nuance to conversations. for example, the heart emoji conveys love more strongly than words, and the face-with-tears-of-joy emoji effectively expresses humor. accurately modeling emojis can enhance natural language processing * both authors contributed equally to this research. current methods for processing emojis are varied. traditional statistical methods either ignore emojis or treat them as regular characters, failing to capture their semantics. methods like emoji2vec @xcite learn emoji representations based on textual descriptions but rely heavily on annotations, which fails to keep up with the evolving semantics of emojis in different contexts. selfsupervised learning, which doesn't require labeled data, is widely used for emojis. however, many self-supervised methods either use emojis only as annotations @xcite ) for text embeddings or focus solely on emoji embeddings, neglecting text embeddings. these models fail to fully utilize the relationship between text and emojis. to address these issues, we aim to better model the relationship between emojis and text, improving embeddings by learning their interactions. it can be observed that emojis in similar texts share semantics, and texts with similar emojis often have shared sentiments or popularity. given the ability of heterogeneous graphs to model interactive relations across information at different granularities, we construct a heterogeneous graph to represent the information that lies in social media posts. specifically, the graph contains emoji nodes and word nodes to capture the relationship between emojis and texts. to better absorb the global information of the whole post, another node type, i.e., post nodes, is also constructed in our graph. in this manner, the above similarity could be represented by the similarity between n-hop neighbors. in order to make information further flow between different nodes in the heterogeneous graph, we develop two self-supervised graph pre-training tasks that facilitate reciprocal text-emoji joint representation learning. (1) based on the observa-tions above, we construct node-level subgraph contrastive learning task. this task first randomly chooses a starting node, and then samples its neighbors through random walk, subsequently forming a sub-graph containing both the starting node and the neighbors. by repeating this process on different starting nodes, several sub-graphs are obtained. the task is used to guide the graph model to judge whether two sub-graphs are generated from the same starting node. this task could help capture the similarity among nodes. (2) to better capture the mutual relationships among these three types of nodes, we build the second task: edge-level link reconstruction learning. this task selects existing edges as positive edges and constructs negative edges from randomly added non-existing ones. then, the task requires the graph model to distinguish positive edges from negative ones. this task lets the graph model focus on the relationships among different node types (every edge links two nodes from different types). consequently, with the help of the above two tasks, graph models could learn to understand how posts, emojis and words interact with each other, which will benefit downstream tasks like popularity prediction. our main contributions are concluded as follows: • to better describe the relationship between posts, emojis and words in social media data, we construct a post-emoji-word heterogeneous graph. ## related work emojis play multiple roles in people's communication. they not only provide an expressive way to convey emotions or situational information but are also convenient for grasping implicit sentiments as well as emotions. the use of emojis on social media tends to affect the popularity of tweets. however, the importance of emojis in textual studies has not been fully realized by early work @xcite @xcite . recently, many studies have revealed the significance of emojis, and these studies could be summarized into four types of research fields: traditional pre-process, description mining, supervised learning, and self-supervised learning methods. traditional pre-process methods. these methods directly utilize standard pre-processing methods and then feed the pre-processed emojis to language models @xcite @xcite . specifically, (1) removing emoji strategy treats emojis as noise and remove them from the original text (pavan kumar and dhinesh babu, 2019). this strategy neglects the semantic information conveyed by emojis. ## methodology in this section, we first construct the heterogeneous graph with three types of nodes: posts, emojis and words, to capture the information and interactions among them ( § 3.1). then, based on the heterogeneous graph, we design a graph pre-training framework to model the comprehensive interaction among all types of nodes ( § 3.2). after that, we show how to use our proposed method in various downstream tasks, i.e., popularity prediction and sentiment classification( § 3.3). figure 1 illustrates the overview of our pre-training and fine-tuning stages. ## experiment in this section, we first introduce the implementation details of our proposed pre-training framework ( § 4.1), and then evaluate our framework on two downstream tasks, i.e., popularity prediction ( § 4.2) and sentiment classification ( § 4.3). ## conclusion in this paper, we propose a novel framework for joint pre-training on text and emojis. to model the relationship between posts, words and emojis, we first construct a heterogeneous graph comprising three node types corresponding to each element. subsequently, we propose a subgraph-level pre-training framework for representation learning, including node-level graph contrastive learning and edge-level link reconstruction learning. our experimental results on the two downstream tasks, i.e., popularity prediction and sentiment analysis, demonstrate that these pre-training tasks could leverage the acquired embeddings for downstream applications, and our approach yields substantial improvements over baseline methods.