Re-entry Prediction for Online Conversations via Self-Supervised Learning (2021)

Abstract: In recent years, world business in online discussions and opinion sharing on social media is booming. Re-entry prediction task is thus proposed to help people keep track of the discussions which they wish to continue. Nevertheless, existing works only focus on exploiting chatting history and context information, and ignore the potential useful learning signals underlying conversation data, such as conversation thread patterns and repeated engagement of target users, which help better understand the behavior of target users in conversations. In this paper, we propose three interesting and well-founded auxiliary tasks, namely, Spread Pattern, Repeated Target user, and Turn Authorship, as the self-supervised signals for re-entry prediction. These auxiliary tasks are trained together with the main task in a multi-task manner. Experimental results on two datasets newly collected from Twitter and Reddit show that our method outperforms the previous state-of-the-arts with fewer parameters and faster convergence. Extensive experiments and analysis show the effectiveness of our proposed models and also point out some key ideas in designing self-supervised tasks.

## introduction online social media platforms are popular for individuals to discuss topics they are interested in and exchange viewpoints. however, a large number of online conversations are posted every day that hinder people from tracking the information they are interested in. as a result, there is a pressing demand for developing an automatic conversation management tool to keep track of the discussions one would like to keep engaging in. re-entry prediction @xcite is proposed to meet such demand. it aims to foresee whether a user (henceforth target user) will come back to a conversation they once participated in. nevertheless, the state-of-theart work @xcite ) mostly focuses on rich information in users' previous chatting history and ignores the thread pattern information @xcite . to this end, we study in re-entry prediction by exploiting the conversation thread pattern to signal whether a user would come back since the degree of repeated engagement of users can indicate their temporary interests in the ongoing conversation. self-supervised learning aims to train a model on labels that are automatically derived from the data itself. compared to previous generic selfsupervised methods (e.g., switch, replace, and mask), task-specific methods can achieve better performance @xcite , especially on medium-sized datasets, since task-oriented designs can better capture domain-specific features and thus achieve better performance for the target task. therefore, we propose a prediction model (main model) for re-entry prediction (main task) with three auxiliary self-supervised tasks (spread pattern prediction, repeated target user prediction and turn authorship prediction) to assist learning of main model for re-entry prediction. spread pattern prediction is inspired by expansionary and focused thread in @xcite , where thread pattern reflects the development of a conversation. we implement this task in a simplified but reasonable way to discriminate thread patterns based on the number of participated users. on the other hand, @xcite shows that target users who contribute two or more posts in a conversation have a higher probability of coming back. hence, we introduce a repeated target user prediction task to facilitate the learning of the main model by capturing the target user's behavior, i.e., whether the target user has posted more than one message in a given context. finally, we introduce the turn authorship prediction task, in figure 1: x-axis: thread pattern, e.g., "ab" represents thread where user a posts then b posts. y-axis: re-entry rate, e.g., the re-entry rate for "ab" is 27%, means that 27% of the target users (user "b") in this kind of conversations will come back. to better illustrate our motivation, figure 1 shows the re-entry rate of six representative thread patterns on reddit dataset. as we can see, the left three threads with user number "≤ 2" (focused) show a higher re-entry rate than the right three threads with user number "> 2" (expansionary). we can also see that although "abca" is expansionary, it has repeated target user "a", which leads to a higher re-entry rate than the other two expansionary threads. therefore, we can conclude that both spread pattern and repeated target user signals help predict re-entry behavior. furthermore, since more challenging tasks get better performance (mao, 2020), we propose turn authorship prediction, where we predict whether each turn's author is a target user or not. before the introduction of pretraining technique @xcite @xcite , researchers focused on developing complex models @xcite , such as key phrase generation with neural topic model @xcite and structured models for coreference resolution @xcite . thus models are time-consuming in training and testing. for this reason, we propose our compact main model, which consists of three parts, turn encoder, conversation encoder and prediction layer. in addition, the chatting history information of the target user is also applied to our model by initializing the beginning hidden state of the target turn. the main model is jointly trained with the three self-supervised tasks in the manner of multi-task training and outperforms the bertbased @xcite model which consists of large number of parameters and is time-consuming. in summary, our contributions are three-fold: the remainder of this paper is organized as follows. the related work is surveyed in section 2. section 3 and 4 present the proposed approach, including model architecture and designed self-supervised tasks. section 5 and 6 then present the experimental setup and results respectively. finally, conclusions are drawn in section 7. re-entry prediction. re-entry prediction @xcite @xcite aims to forecast whether the users will return to a discussion they once entered and @xcite achieves state-of-the-art performance by exploiting user's history context @xcite . re-entry prediction focuses on conversation-level response prediction @xcite . most of them adopt a complex framework @xcite and massive parameters (see figure 4(b) ) while our model is simple and effectively combines the current conversation and chatting history. self-supervised learning. self-supervised learning aims to train a network on an auxiliary task where the ground-truth label is automatically derived from the data itself @xcite @xcite . it has been applied to many tasks, such as text classification @xcite , neural machine translation @xcite , multi-turn response selection @xcite , summarization @xcite and dialogue learning @xcite . these auxiliary tasks can be categorized into word-level tasks and sentence-level tasks. in word-level tasks, nearby word prediction @xcite and next word prediction @xcite are widely explored in language modeling. masked language model @xcite is also in the line of word-level tasks. wang et al. (2019a) exploits mask, replace and switch for extractive summarization. wu et al. (2019) propose inconsistent order detection for dialogue learning. xie et al. (2020) exploit drop, replace, and tov (temporal order verification) for story cloze test. xu et al. (2020) also design several self-supervised tasks to improve the performance of response selection. most of the previous self-supervised tasks (both in word-level and sentence-level) focus on the general domain while our work is based on taskorientated supervised methods and achieves better performance. ## re-entry prediction framework this section describes our re-entry prediction framework. the left part of figure 2 shows our overall structure. in the following, we first introduce the input and output in section 3.1. then in section 3.2, we describe our prediction framework. finally, the learning objective of the entire model will be given in section 3.3. ## experimental results in this section, we first introduce the main comparison results in section 6.1. then the effects of our methods and how our methods make effects are given in section 6.2 and section 6.3 respectively. finally, section 6.4 yields further discussion on user history and error analysis. ## conclusion we present a basic model with three novel selfsupervised tasks for re-entry prediction. experiments on two newly constructed conversation datasets, twitter and reddit, show that our model outperforms the previous models with fewer parameters and faster convergence. further discussions provide more insights on how our model works and how to design task-oriented self-supervised tasks.