Autoregressive Affective Language Forecasting: A Self-Supervised Task (2020)

Abstract: Human natural language is mentioned at a specific point in time while human emotions change over time. While much work has established a strong link between language use and emotional states, few have attempted to model emotional language in time. Here, we introduce the task of affective language forecasting – predicting future change in language based on past changes of language, a task with real-world applications such as treating mental health or forecasting trends in consumer confidence. We establish some of the fundamental autoregressive characteristics of the task (necessary history size, static versus dynamic length, varying time-step resolutions) and then build on popular sequence models for words to instead model sequences of language-based emotion in time . Over a novel Twitter dataset of 1,900 users and weekly + daily scores for 6 emotions and 2 additional linguistic attributes, we find a novel dual-sequence GRU model with decayed hidden states achieves best results ( r = .66 ) significantly out-predicting, e.g., a moving averaging based on the past time-steps ( r = .49 ). We make our anonymized dataset as well as task setup and evaluation code available for others to build on.

## introduction with the growth of social media, natural language processing has increasingly turned toward problems in the social scientific domains. language has been used to predict personality disorders, political ideology, and mental health @xcite @xcite . however, such predictions are typically made cross-documents or -users rather than cross-time (forecasting the future). predictions across time can enable another level of applications for nlp in the social sciences. for example, knowing how someone's mood may change next week can be a valuable insight into preemptively preparing mental health assistance. we may be able to better predict a substance abuse relapse or suicide attempt, or provide individual insights into the precipitators of such events. 1 applications also exist in other domains, such as correlating mood and personal buying habits, or predicting the mood of performers (e.g. professional athletes) as insight into how well they will perform next. one of the challenges for doing temporal predictions based on language is the availability of timeseries data. most user-level datasets only provide one measurement, typically of a demographic or psychological trait (something that doesn't change frequently). on the other hand, document-level data (such as that of most sentiment analysis) may provide information across time, but they are typically divorced from the individual. alternatively, instead of relying on annotations or self-reported data, one can simply predict the future language-based measurement (i.e. as output from another model). here, we propose a method of forecasting any future dimension of language, based on their current and past language. formally, we define affective language forecasting as finding a function of past language (f ) which estimates the output of a function of future language (a): our contributions include: such architectures have been used extensively in the medical domain, by leveraging electronic health records, to forecast a patient's future physical health. for example, the work of @xcite designs a stacked gru architecture to forecast the diagnosis of a patient at their next doctor's visit, while jointly learning to predict the duration until the next visit(an example of a poison process). in the language domain the only similar work is that of @xcite . in which, their goal is to predict an online forum participants "emotional progression" measured by their negative emotion index, a simple metric that defines the polarity of a user's word count num(neg words)-num(pos words) num(total words) ## data and time-series we derived our data set from a collection of publicly posted tweets previously used for demographic prediction @xcite . as one of the first studies of language forecasting of any kind, we sought to primarily focus on a wellrepresented time resolution -weekly aggregates of tweets. this helped us focus on the fundamentals of the methods rather than missing data issues. 4 only tweets without urls (i.e. no retweets) were considered and users were filtered based on having 3+ original tweets per week, resulting in a dataset containing 850,000 tweets spanning 1,900 users. our dataset contains users who posted between the years of 2009 and 2014 and had 20 contiguous weeks of data. additionally, we treat the problem as that of predicticting future change in affective language as opposed to modeling the static scores. by focusing on the change, the data is essentially detrended as if mean centered (i.e. increasing stationarity in our time-series) while the static scores can still be derived. to determine the change, a difference is taken between neighboring time steps for all features. after which, 19 data points remain in our sequence which is further split into 14 weeks for training and 4 weeks for testing per user. ## methods we seek to model the affective score of future language (y t ; e.g. the positive/negative valence of the language) as a function of past affective language (y t-1 , y t-2 , ...) as well as other dimensions of past language (x t-1 , x t-2 , ...; e.g. vector embeddings): ## evaluation the first set of results showcases the chosen zero rule baselines compared to all deep learning models using a fixed order of 14 (the max). we show the results for the test set consisting of oos time users for target variable affective valence. the zero rule baselines perform quite well, showing that even outpredicting the past can be challenging. note that the deep averaging network is operating on the same information as the mean( 14 ) baseline, so we expect it to perform similarly. figure 3 : comparison of grus for fixed and variable order pure autoregressive approaches as more history is added to the training examples for target variable affective valence. gru-d always leverages 14 weeks of data (where anything older than the set order is marked as missing and leverages the mean instead). + denotes dynamic order and f denotes fixed order. additionally in practice there are many users who would be forecasted but would possibly not be part of the training data of any particular model. to emulate this we examine results for a set of held-out users for which our model has neither trained nor been tuned on. we compare the results across select models from the previous test set and this oos user test set in in figure 3 , we show the effect of order p on our autoregressive models with respect to correlation in a pure autoregressive setup. as mentioned, a benefit of using recurrent neural models for time-series modeling is the ability to implement dynamic unrolling on variable length sequences. to explore the effectiveness of our dynamic order technique for time-series, we directly compare the gru models using variable p against training with a fixed p. we find that applying the dynamic order to a model such as gru-d, where data can be marked as missing, gives clear benefits when compared against smaller fixed orders. however, as the order size increases this loses its effectiveness. with respect to the amount of p required to create a robust model, we typically find that most models perform best when utilizing 9 weeks of p before saturating. we believe this is due to the shrinking of the training set. for each extra time-step of history, there is a loss of 1 training example per user. in other words, each increase of 1 time-step of p reduces training model (order) uni(r) w2v(r) joint sequence gru (11+) .63 .63 gru-ds (13) .64 .64 gru-d (9+) .65 .65 dual sequence gru (11+) -.63 gru-ds (13) -.66 gru-d (9+) -.66 in we also highlight the differences in performance between using word2vec and bert representations in affective valence prediction is also examined using off-the-shelf models such as svr, gbr, and (ridge) autoregression. here, we show the best performing variants of each(their optimal p) and compare to our best gru model(dual gru-ds @xmath2, shown in a robustness check is performed to analyze sensitivity to seasonal trends. figure 4a shows the squared error averaged across all years of data separated by month for affective valence. notably, there is not a large increase in error during the winter months where one may expect emotions to be abnormal due to the many holidays that occur, but rather error seems more dependent on number of users present in that testing month, suggesting that this task scales well with respect to dataset size at the user level. robustness is also checked within user, where we visualize our model's forecasted weekly changes for a person's affective valence, shown in figure 4b . our model achieves great success at not only producing accurate predictions but also capturing general trends in mood. however, there is evidence of the model struggling when a user continually has large changes in mood week after week. results for affective attributes listed in section 3.1 are shown in bold results show significance with @xmath4.05. 14 ) and the ground truth labels. we find that our predictions are typically accurate. however, even when they fail the general trend is still present valence are seen across these attributes, implying there is an underlying structure of affective language in the temporal domain. finally, performance on the daily resolution dataset for affective valence is presented. a brief selection of results is shown in ## conclusion we presented a new subtask in nlp, self-supervised affective forecasting. the models proposed in this work cover a wide range from traditional machine learning to deep learning techniques. while traditional models perform well, grus with decayed hidden states provided more accurate results and offer a more flexible architecture. we find this most helpful when experimenting with extra context features, such as word embeddings, splitting the inputs into 2 separate forecasts for a dual sequence gru offers the best predictive power. we also show that our modeling choices are robust to both seasonal variation and multiple evaluation paradigms: oos time and oos user. lastly, we found a strong relationship between the order of our models and the accuracy of future predictions with a performance leveling out roughly at a history of 9 weeks for pure autoregressive and 13 weeks for dual gru-ds. affective forecasting can be leveraged for many domains, such as social sciences (mental health), business (consumer buying habits, targeted ads), and athletics (game performance). we encourage others to pick up this challenging task by releasing our novel dataset and task setup code.