Opinion Prediction with User Fingerprinting (2021)

Abstract: Opinion prediction is an emerging research area with diverse real-world applications, such as market research and situational awareness. We identify two lines of approaches to the problem of opinion prediction. One uses topic-based sentiment analysis with time-series modeling, while the other uses static embedding of text. The latter approaches seek user-specific solutions by generating user fingerprints. Such approaches are useful in predicting user’s reactions to unseen content. In this work, we propose a novel dynamic fingerprinting method that leverages contextual embedding of user’s comments conditioned on relevant user’s reading history. We integrate BERT variants with a recurrent neural network to generate predictions. The results show up to 13% improvement in micro F1-score compared to previous approaches. Experimental results show novel insights that were previously unknown such as better predictions for an increase in dynamic history length, the impact of the nature of the article on performance, thereby laying the foundation for further research.

## introduction sentiment analysis plays a key role in economic, social, and political contexts. companies can understand customer's opinions based on reviews and/or social media conversations to make fast and accurate product decisions. they can index unstructured customer data at scale based on broad sentiments such as positive, negative, or neutral. it also enables market research -explore new markets, anticipate future trends, and seek an edge on the competition. in a political context, sentiment analysis can be useful in understanding political homophily using tweets analysis @xcite . moreover, as social media has emerged as a new source of communication, governments can analyze people's reactions to issues such as police-related encounters, mob protests and anticipate responses before they turn violent. in this context, predicting people's opinion before expressing is an important next step in applying sentiment analysis to real-world applications. we are approaching this problem through sentiment prediction, especially on current news events, that can raise situational awareness, understand the future viewpoints of the citizenry on pressing social and political issues. recently fine-grained models generated a person's fingerprint, based on one's recent reading and response history, to predict response on an unknown event @xcite . in this work, we propose a novel architecture to create a dynamic fingerprint of a user that is contingent upon the target event. we choose to evaluate our models on the dataset used by @xcite , which contains newspaper articles as events and users' comments on them as opinions. those ground truth opinions are used as a basis for sentiment prediction in this work. our model consists of three main steps. in the first step, relevant articles are extracted from a user's reading history based on their similarity to target article. in the second step, the contextual embedding of these relevant articles, conditioned on the target article, is used to create a reading track. similarly, we create a writing (response) track with the contextual embedding of extracted articles with corresponding comments by the user. lastly, dynamic fingerprints are generated based on the temporal pattern of the reading and writing tracks. these dynamic fingerprint vectors for a particular user are then used to predict the user's sentiment on the target event. our main contributions in this paper are: ## related work opinion prediction based on the temporal pattern of sentiments is a relatively new research topic, but basic concepts such as sentiment analysis, question answering based on dialogue context have been explored in different communities and settings. @xcite proposed a technique to leverage topic-based twitter sentiments to predict the stock market using vector autoregression and dirichlet process mixture models. li et al., 2019 proposed a time+user dual attention-based lstm network to perform emotional analysis on chinese public opinion texts on social networking platform. but they did not use contextual embedding and explore the prospect of generating a unique user fingerprint before predicting sentiment. conversational question answering (cqa) is an emerging research area in the machine reading comprehension task (mrc). for single-turn mrc tasks, contextualized language representation using bert has obtained state-of-the-art scores on squad datasets @xcite . cqa is a multi-turn question answering task that includes passage comprehension, contextual understanding, and coreference resolution. zhu et al. have proposed sdnet @xcite to solve this problem by concatenating previous questions and answers as one query to fuse context into traditional mrc models by leveraging bert, attention, and rnn techniques. similarly, @xcite have proposed fine-tuning approach with bert in a multi-turn context by modeling the interaction between paragraph and dialogue history independently. however, these models cannot be applied to the present problem since they did not integrate the concepts of the sequential pattern of sentiments along with the unique fingerprint of each user, which can play a key role in predicting the future opinion of a user on different topics. ## proposed model and methodology in this section, we propose two classes of fingerprint embedding models (fpe) -static and dynamic -for the task of predicting the sentiment of a user u to a new article. in a narrow sense, we used the term static to refer to the approach of using recent comment history, which is independent of the nature of target article a(t), and dynamic to refer to the approach of using articles relevant to target article a(t) in the overall history of user's comments, which are dependent on the nature of target article. in a broad sense, static and dynamic terms distinguish the way target article a(t) is integrated with user's reading history to generate the fingerprint. we used the user's commenting history on articles that they read. we assume that we know the sentiment of each comment. (this can be obtained with one of the many sentiment analysis tools.) formally, we are given the articles, comments along with the sentiments of a user u, i.e., (a 1 , c 1 , s 1 ), (a 1 , c 2 , s 2 ),..., (a 2 , c j , s j ), ..., (a t-1 , c n , s n ), and the goal is to predict the sentiment s t of u's response to unseen article a t . in general, @xmath0. the overall architecture includes history selection, text embedding, fingerprint creation, and lastly sentiment prediction. we describe them below. ## experiments the main goals of our experiments are: f1-score) of both static and dynamic model variants in the prediction of the sentiment of a user to an unknown article 2. analysis of model performance by studying the impact of dynamic history length and nature of articles on prediction. ## results and discussion firstly, contextual embedding, in place of static bpe embedding in fpe, of articles and comments is a key factor behind the superior performance. specifically, the contextual embedding of article history with target article (reading track) and with corresponding comment history (writing track) has enabled us to generate a better representation of the input text. secondly, the dynamic fpe model, unlike the static fpe model, creates reading and writing fingerprints through gru networks separately before concatenating them for sentiment prediction. with an extra gru hidden layer in the reading track compared to the writing track, we have been able to create a higher level of understanding of the temporal relationship between target article and history articles. from these fingerprints, we also found that users with the closest fingerprints in the euclidean space are found to have higher prediction accuracy than that of farther fingerprint users. further, the electra-based model outperforms distilbert-based model in most of the cases, despite having only 20% of the number of parameters of distilbert-based model. this reiterates the result of @xcite that novel pretraining by a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not performs better than masked language modeling pretraining method of distilbert, even for a small model. other important inferences are: for complete details of these experiments, please refer to longer version of the paper . ## limitations and future work according to sutrop and laas-mikko, 2012, fingerprinting for predicting behavior is secondgeneration biometrics, which is different from firstgeneration biometrics that uses characteristics that are visible to the naked eye such as facial images, hand fingerprints. in this context, user fingerprinting in our model can be loosely classified under behavior fingerprinting. we did linguistic opinion and content-based user fingerprinting as a response history embedding for a user. in this section, we briefly discuss the limitations of the dataset and model, and future research direction. firstly, the newspaper articles, in general, may be biased in terms of story selection, tone, and organizing of the story. the users(readers) may also have an implicit bias -attitudes that unconsciously affect individual thoughts and actions-and confirmation bias -the tendency to support information that confirms their beliefs. to address these biases, we would like to extend our model to datasets that are not related to news articles. moreover, representations encoded in the models often inadvertently perpetuate undesirable social biases from the data on which they are trained. nlp models, especially neural embeddings, may perpetuate these biases towards race, religion, gender and disability @xcite @xcite . though the bert variant based sentence encoders exhibit less bias than previous models @xcite , we would also like to experiment with other sentence encoders to measure the bias in our predictions in future work. another limitation of our approach is that we used only article titles rather than whole content. this would be more critical when the title is misleading, for instance in satirical articles. moreover, we have not experimented with multilingual models of pretrained bert variants. for future research, these experiments can be extended for whole article content and use various attention mechanisms to generate better fingerprints and also generate author profiles based on their reading history. further, bert variants trained in other languages can also be used. ## conclusion in this paper, we propose a novel dynamic fingerprinting technique based on bert variants and rnn networks to predict a user's sentiment to an unseen article based on reading-writing history. two variants of our model extract relevant history in two different ways and create contextual embedding for articles read by a user conditioned with target article and also corresponding comments. finally, we used rnn to interpret temporal relationship and create a fingerprint, which is used to predict unseen target article. our models demonstrated state-of-the-art performance on a real-world dataset. from our experiments, we found that performance saturates after an optimum length of relevant history.