NTULM : Enriching Social Media Text Representations with Non-Textual Units (2022)

Abstract: On social media, additional context is often present in the form of annotations and meta-data such as the post’s author, mentions, Hashtags, and hyperlinks. We refer to these annotations as Non-Textual Units (NTUs). We posit that NTUs provide social context beyond their textual semantics and leveraging these units can enrich social media text representations. In this work we construct an NTU-centric social heterogeneous network to co-embed NTUs. We then principally integrate these NTU embeddings into a large pretrained language model by fine-tuning with these additional units. This adds context to noisy short-text social media. Experiments show that utilizing NTU-augmented text representations significantly outperforms existing text-only baselines by 2-5% relative points on many downstream tasks highlighting the importance of context to social media NLP. We also highlight that including NTU context into the initial layers of language model alongside text is better than using it after the text embedding is generated. Our work leads to the generation of holistic general purpose social media content embedding.

## introduction understanding the social context is crucial to the semantic understanding of a social media post @xcite @xcite . this is especially true for short-text social media such as twitter where the textual content available for semantic understanding is inherently limited. as such, pretrained language models that ignore nontextual context can demonstrate sub-optimal performance when utilized for social-media nlp. fortunately, on social media, there are many available non-textual units (ntus), which provide social contexts for any written text. for example, the author of a post provides a social prior as to the content written by that author. additionally, the author may annotate their post with meta-data such as hashtags, user mentions, or urls and other media. these units can frame the content of a post by providing social context, a stance, or additional supporting material. previous research has investigated augmenting pretrained language model representations with additional signals. these include enrichments by incorporating image features @xcite , better-segmented hashtags @xcite , url understanding @xcite , or temporal-spatial contexts @xcite . however, these existing works are type-specific and require a specialized technique to integrate just one type of non-textual signal (e.g., requiring an image encoder to extract image features). we claim that this added complexity makes it difficult to incorporate different non-textual signals and effectively train a joint model. in this paper, our ntu enriched language model (ntulm) can easily, without loss of generality, train and integrate graph embeddings @xcite for multiple types of ntus. ntulm can do this through the use of heterogeneous information network embeddings of ntus. this allows us to not only co-embed multiple ntu types, but also incorporate a variety of interaction types as edges in our network (e.g., authoring posts, favoriting hashtags, and co-mentioning users). this general embedding framework is simple and does not require specialized feature encoders for different ntu types. after obtaining the ntu knowledge embeddings, ntulm deeply integrates them with the language model at the token level and simply applies the default attention mechanism used in the bert encoder. to ensure our alignment with @xcite which allows only inclusion of a single context embedding to bert, we take the average of ntu embeddings and attach the unified embedding at the end of token embedding sequence. the framework of ntulm is shown in figure 1 . to ensure high coverage of the ntu vocabulary across tweets, we construct a large-scale heterogeneous ntu graph ensuring high overlap with all tweets. with the scalability of our graph embeddings, we can rapidly embed ntus ensuring high coverage across tweets. we state and analyze the problem in section 2, followed by our proposed solution involving ntu embedding and bert integration in section 3. in section 4 we evaluate our proposed solution compared to text-only baselines. we go over related works in section 6 and conclude in section 8. ## task formulation in this section, we formulate the task of enriching pretrained language models with additional ntu embeddings. ## ntu enriched language model the framework of ntulm is shown in figure 1 . in this section, we first introduce how we learn high-quality ntu embeddings by embedding an ntu-centric heterogenous social graph. we then describe how we principally integrate these ntu embeddings in a standard bert-style language model yielding tweet embeddings that utilize both text and ntu information. we will use the tweet in author: user1 tweet: our paper was accepted at @w n u t with @user2 @user3 #nlproc #socialmedia favorited by: user4, user5 ## experiments we conduct experiments on a variety of datasets and downstream tasks to highlight the utility of ntulm. additionally, we perform an ablation to measure the contribution of each type of ntu to the overall ntulm performance. ## limitations one major limitation of our work is the averaging of heterogenous embeddings. this approach works because the embeddings trained using transe lie in the same space but is less expressive as we are not including explicit information around which type of ntu an embedding is coming from. in future we plan to address this by including type specific embedding transformation before doing an averaging. however, given the results, this naive averaging of user+hashtag still works well across tasks it shows the utility of our approach. next, our training data is relatively small and less diverse with only 1m tweets as budgetary and computational constraints influenced our experimental setup. in this paper, our goal has been to demonstrate the effectiveness of our approach paving the way for future work that scales up the training and uses a much larger and more diverse dataset. finally, our results are on english specific datasets and models. while the utilization of ntu embeddings make our approach language agnostic, in future we plan to demonstrate its impact across multiple languages. ## conclusion in this paper we introduced ntu enriched language model (ntulm), a method of enriching a pretrained bert model by adding graph embeddings of non-textual units. we experimentally demonstrate that including ntu representations alongside text yields superior representations vs a text-only language model. on several downstream tasks, we show significant improvment using ntulm representations compared to bertbased sentence embeddings.