Bi-Directional Recurrent Neural Ordinary Differential Equations for Social Media Text Classification (2022)

Abstract: Classification of posts in social media such as Twitter is difficult due to the noisy and short nature of texts. Sequence classification models based on recurrent neural networks (RNN) are popular for classifying posts that are sequential in nature. RNNs assume the hidden representation dynamics to evolve in a discrete manner and do not consider the exact time of the posting. In this work, we propose to use recurrent neural ordinary differential equations (RNODE) for social media post classification which consider the time of posting and allow the computation of hidden representation to evolve in a time-sensitive continuous manner. In addition, we propose a novel model, Bi-directional RNODE (Bi-RNODE), which can consider the information flow in both the forward and backward directions of posting times to predict the post label. Our experiments demonstrate that RNODE and Bi-RNODE are effective for the problem of stance classification of rumours in social media.

## introduction information disseminated in social media such as twitter can be useful for addressing several realworld problems like rumour detection, disaster management, and opinion mining. most of these problems involve classifying social media posts into different categories based on their textual content. for example, classifying the veracity of tweets as false, true, or unverified allows one to debunk the rumours evolving in social media @xcite . however, social media text is extremely noisy with informal grammar, typographical errors, and irregular vocabulary. in addition, the character limit (240 characters) imposed by social media such as twitter make it even harder to perform text classification. social media text classification, such as rumour stance classification 1 (qazvinian et al., 1 rumour stance classification helps to identify the veracity 2011; @xcite can be addressed effectively using sequence labelling models such as long short term memory (lstm) networks @xcite @xcite @xcite @xcite . though they consider the sequential nature of tweets, they ignore the temporal aspects associated with the tweets. the time gap between tweets varies a lot and lstms ignore this irregularity in tweet occurrences. they are discrete state space models where hidden representation changes from one tweet to another without considering the time difference between the tweets. considering the exact times at which tweets occur can play an important role in determining the label. if the time gap between tweets is large, then the corresponding labels may not influence each other but can have a very high influence if they are closer. we propose to use recurrent neural ordinary differential equations (rnode) @xcite and developed a novel approach bidirectional rnode (bi-rnode), which can naturally consider the temporal information to perform time sensitive classification of social media posts. node @xcite is a continuous depth deep learning model that performs transformation of feature vectors in a continuous manner using ordinary differential equation solvers. nodes bring parameter efficiency and address model selection in deep learning to a great extent. rnode generalizes rnn by extending node for time-series data by considering temporal information associated with the sequential data. hidden representations are changed continuously by considering the temporal information. we propose to use rnode for the task of sequence labeling of posts, which considers arrival times of the posts for updating hidden representaof a rumour post by classifying the reply tweets into different stance classes such as support, deny, question, comment tions and for classifying the post. in addition, we propose a novel model, bi-rnode, which considers not only information from the past but also from the future in predicting the label of the post. here, continuously evolving hidden representations in the forward and backward directions in time are combined and used to predict the post label. we show the effectiveness of the proposed models on the rumour stance classification problem in twitter using the rumoureval-2019 @xcite dataset. we found rnode and bi-rnode can improve the social media text classification by effectively making use of the temporal information and is better than lstms and gated recurrent units (gru) with temporal features. ## background we consider the problem of classifying social media posts into different classes. let d be a collection of n posts, @xmath0. each post p i is assumed to be a tuple containing information such as textual and contextual features x i , time of the post t i and the label associated with the post y i , thus p @xmath1, t i , y i )}. our aim is to develop a sequence classification model which considers the temporal information t i along with x i for classifying a social media post. in particular, we consider the rumour stance classification problem in twitter where one classifies tweets into support, query, deny, and comment class, thus y @xmath2, query, deny, comment}. ## bi-directional recurrent node lstms are popular for sequence classification but only considers the sequential nature of the data and ignore the temporal features associated with the data in its standard setting. as the posts occur in irregular intervals of time, the nature of a new post will be influenced by the recent posts, influence will be inversely proportional to the time gap. in these situations, it will be beneficial to use a model where the number of transformations depend on the time gap. we propose to use rnode which considers the arrival time and accordingly the hidden representations are transformed across time. in rnode, the transformation of a hidden representation h(t i-1 ) at time t i-1 to h(t i ) at time t i is governed by an ode parameterized by a nn f (). unlike standard lstms where h(t i ) is obtained from h(t i-1 ) as a single nn transformation, rnode first obtains a hidden representation h ′ (t i ) as a solution to an ode at time t i with initial value h(t i-1 ). the number of update steps in the numerical technique used to solve this ode depends on the time gap t i -t i-1 between the consecutive posts. the hidden representation h ′ (t i ) and input post x i at time t i are passed through neural network transformation (rnncell()) to obtain final hidden representation h(t i ), i.e., h(t i ) = rnncell(h ′ (t i ), x i ). the process is repeated for every element (x i , t i ) in the sequence. the hidden representations associated with the elements in the sequence are then passed to a neural network (nn()) to obtain the post labels. using standard cross-entropy loss, the parameters of the models are learnt through backpropagation. figure 1 provides the detailed architecture of the bi-directional rnns @xcite such as bi-lstms @xcite were proven to be successful in many sequence labeling tasks in natural language processing such as pos tagging @xcite . they use the information from the past and future to predict the label while standard lstms consider only information from the past. we propose a bi-rnode model, which uses the sequence of input observations from past and from the future to predict the post label at any time t. it assumes the hidden representation dynamics are influenced not only by the past posts but also by the futures posts. unlike bi-lstms, bi-rnode considers the exact time of the posts and their inter-arrival times in determining the transformations in the hidden representations. bi-rnode consists of two rnode blocks, one performing transformations in the forward direction (in the order of posting times) and the other in the backward direction. the hidden representations h and h b computed by forward and backward rnode respectively are aggregated either by concatenation or by averaging appropriately to obtain a final hidden representation and is passed through a nn to obtain the post labels. bi-rnode is useful when a sequence of posts with their time of occurrence needs to be classified together. figure 2 provides an overview of bi-rnode model for post classification. for bi-rnode, an extra neural network f θ ′ () is required to compute hidden representations h b (t ′ i ) in the backward direction. training in bi-rnode is done in a similar manner to rnode, with cross-entropy loss and back-propagation to estimate parameters. ## experiments to demonstrate the effectiveness of the proposed approaches, we consider the stance classification problem in twitter and @xcite data set. this twitter data set consists of rumours associated with eight events. each event has a collection of tweets labelled with one of the four labels -support, query, deny and comment. we picked four major events charliehebdo, ferguson, ottawashooting and sydneysiege (each with approximately 1000 tweets per event) from rumoureval-2019 to perform experiments. features : for dataset preparation, each data point x i associated with a tweet includes text embedding, retweet count, favourites count, punctuation features, negative and positive word count, presence of hashtags, user mentions, urls etc. obtained from the tweet. the text embedding of the tweet is obtained by concatenating the word embeddings foot_0 . each tweet timestamp is converted to epoch time and min-max normalization is applied over the time stamps associated with each event to keep the duration of the event in the interval [0, 1]. ## conclusion we proposed rnode, bi-rnode models for sequence classification of social media posts. these models consider temporal information of the posts and hidden representation are evolved as solution to ode. through experiments, we show these models perform better than lstms on rumour stance classification problem in twitter