Finding Spoiler Bias in Tweets by Zero-shot Learning and Knowledge Distilling from Neural Text Simplification (2021)

Abstract: Automatic detection of critical plot information in reviews of media items poses unique challenges to both social computing and computational linguistics. In this paper we propose to cast the problem of discovering spoiler bias in online discourse as a text simplification task. We conjecture that for an item-user pair, the simpler the user review we learn from an item summary the higher its likelihood to present a spoiler. Our neural model incorporates the advanced transformer network to rank the severity of a spoiler in user tweets. We constructed a sustainable high-quality movie dataset scraped from unsolicited review tweets and paired with a title summary and meta-data extracted from a movie specific domain. To a large extent, our quantitative and qualitative results weigh in on the performance impact of named entity presence in plot summaries. Pretrained on a split-and-rephrase corpus with knowledge distilled from English Wikipedia and fine-tuned on our movie dataset, our neural model shows to outperform both a language modeler and monolingual translation baselines.

## introduction people who expose themselves to the process of satisfying curiosity expect to enhance the pleasure derived from obtaining new knowledge @xcite . conversely, induced revelatory information about a plot of a motion picture, tv program, video game, or book can spoil the viewer sense of surprise and suspense, and thus greatly diminish the enjoyment in consuming the media. as social media has thrived into a medium for self-expression, live tweets, opinion dumps, or even hashtags tend to proliferate within minutes of the media reaching the public, and hence the risk of uncovering a spoiler widens rapidly. spoilers on review websites may inevitably contain undesired information and disclose critical plot twists that evoke far less interest to users who consult online reviews first, and later engage with the media itself. social media platforms have placed elaborate policies to better guard viewers from spoilers. on the producer side, some sites adopted a strict convention of issuing a spoiler alert to be announced in the subject of the post @xcite . recently, twitter started to offer provisions for the tweet consumer to manually mute references to specific keywords and hashtags and stop displaying tweets containing them @xcite . but despite these intricate mechanisms that aim to preemptively ward off unwanted spoilers, the solutions proposed lack timely attraction of consumers and may not scale, as spoilers remain a first-class problem in online discourse. rather than solicit spoiler annotations from users, this work motivates automatic detection of spoiler bias in review tweets, of which spoiler annotations are unavailable or scarce. surprisingly, for the past decade spoiler detection only drew little notice and remained a relatively understudied subject. earlier work used machine learning techniques that incorporated humancurated features in a supervised settings, including a latent dirichlet allocation (lda) based model that combines simple bag-of-words (boa) with linguistic cues to satisfy spoiler detection in review commentary @xcite , baseline n-gram features augmented with binary metadata that was extracted from their review dataset showed dramatically improved performance of spoiler detection in text @xcite , and while frequent verbs and named entities play a critical role in identifying spoilers, adding objectivity and main sentence tense improved classification accuracy by about twelve percentage points @xcite . more recently researchers started to apply deep learning methods to detect spoiler sentences in review corpora. the study by @xcite pro-poses a model architecture that consists of a convolutional neural network (cnn) based genre encoder and a sentence encoder that uses a bidirectional gated recurrent unit (bi-gru) @xcite . a genre-aware attention layer aids in learning spoiler relations that tend to vary by the class of the item reviewed. their neural model was shown to outperform spoiler detection of machine-learning baselines that use engineered features. using a book review dataset with sentence-level spoiler tags, @xcite followed a similar architecture explored by @xcite and introduced spoil-ernet that comprises a word and sentence encoders, each realizing a bi-gru network. we found their error analysis interesting in motivating the rendition of spoiler detection as a ranking task rather than a conventional binary classification. although only marginally related, noteworthy is an end-toend similarity neural-network with a variance attention mechanism @xcite , proposed to address real-time spoiled content in time-sync comments that are issued in live video viewing. in this scenario, suppressing the impact of often occurring noisy-comments remains an outstanding challenge. in our approach we propose to cast the task of spoiler detection in online discourse as ranking the quality of sentence simplification from an item description to a multitude of user reviews. we used the transformer neural architecture that dispenses entirely of recurrence to learn the mapping from a compound item summarization to the simpler tweets. the contributions of this work are summarized as follows: ## related work text simplification is an emerging nlp discipline with the goal to automatically reduce diction complexity of a sentence while preserving its original semantics. inspired by the success of neural machine translation (nmt) models @xcite , sentence simplification has been the subject of several neural architectural efforts in recent years. @xcite addressed the simplification task with a long shortterm memory (lstm) @xcite based encoder-decoder network, and employed a reinforcement learning framework to inject prior knowledge and reward simpler outputs. in their work, @xcite propose to combine neural semantic encoders (nse) @xcite , a novel class of memory augmented neural networks which offer a variable sized encoding memory, with a trailing lstm-based decoder architecture. their automatic evaluation suggests that by allowing access to the entire input sequence, nse present an effective solution to simplify highly complex sentences. more recently, @xcite incorporated a hand-crafted knowledge base of simplification rules into the self-attention transformer network @xcite . this model accurately selects simplified words and shows empiri- ## dataset our task requires a dataset that pairs user reviews obtained from discourse on social media with a summarization record collected from an item specific domain. by and large, a user-item model implies distinct review authors for each item. in this study we chose the movie media as the item, or the subject to review, of which a plot description lets users read about the title before watching the film. to the extent of our knowledge, publicly accessible spoiler corpora to date include the dataset used in the work by @xcite that consists of four movies and a total of 2,148 review comments, and more recently, the imdb spoiler dataset, a large-scale corpus provided by kaggle. we constructed a new dataset for our experiments that comprises meta-data and a total of 15,149 user reviews across twenty three movies. 3 we selected movies released as early in 1939 till 2019 ( we viewed our movie corpus as a parallel drawn between a single elaborate plot summary and many simpler review tweets. a text simplification system learns to split the compound passage into several small text sequences further rewritten to preserve meaning @xcite . to perform lexical and syntactic simplification requires to sustain a reasonable span length ratio of the plot summary to a user review. this predominately motivated our choice of using review tweets that are extremely short-text sequences of 240 characters at most. distributions of span length in words for both plot summaries and user reviews are shown in figure 1c and figure 1d , respectively. review spans affecting at least 200 tweets are shown to range from 6 to 22 words and apportion around 46 percent of total reviews. still, the minimum summary span-length has 38 words that is larger than the maximum review span-length at 28 words. as is evidenced in ## model neural text simplification (nts) models prove successful to jointly perform compelling lexical simplification and content reduction @xcite . combined with sari @xcite , a recently introduced metric to measure the goodness of sentence simplification, nts models are encouraged to apply a wide range of simplification operations, and moreover, their automatic evaluation was shown empirically to correlate well with human judgment of simplicity. unlike bleu @xcite that scores the output by counting n-gram matches with the reference, i.e. the simplified version of the complex passage, sari principally compares system output against both the reference and input sentences and returns an arithmetic average of n-gram precisions and recalls of addition, copy, and delete rewrite operations. foot_1 sari that correctly rewards models like ours which make changes that simplify input text sequences has inspired us to cast the spoiler detection task as a form of sentence simplification. we hypothesized that the better simplification of a user review from a plot summary, the higher the likelihood of the tweet to contain a spoiler. moreover, our system is designed to not just confirm the presence or absence of a spoiler, but rather rank the plot revealing impact of a review tweet in a fine-grained scale. using a colon notation, we denote our collection of n movie objects m 1:@xmath0, . . . , m n }, each incorporating a plot summary s and k user reviews r 1:@xmath1, . . . , r k }. thus the input movie m i we feed our model consists of k (i) pairs {s, r j } (i) , where 1 ≥ @xmath2. given s, a linearly l-split text sequence s 1:@xmath3, . . . , s l }, our model predicts a simplified version r of an individual split s. figure 2: architecture overview of the transformer (encoder path shown in blue, decoder in brown). neural sentence simplification is a form of textto-text generation closely resembling the task of machine translation in the confines of a single language. in our study we used the self-attention transformer architecture that requires less computation to train and outperforms both recurrent and convolutional system configurations on many language translation tasks @xcite . in figure 2 , we present an overview of the transformer architecture. stacked with several network layers, the transformer encoder and decoder modules largely operate in parallel. in the text simplification framework, the input sequences to the transformer consist of the compound passage words x i , obtained from a summary split s, and the simple reference words y i from the current user review r j . to represent the lexical information of words x i and y i , we use pretrained embedding vectors that are shared across our entire movie corpus. the transformer network facilitates complex-to-simple attention communication and a softmax layer operates on the decoder hidden-state outputs to produce the predicted simplified output ŷi . our training objective is to minimize the cross-entropy loss of the reference-output review pairs. ## experimental setup in this section, we provide preprocessing steps we applied to our movie corpus and details of our training methodology. with a domineering father. depressed by his oppressive family life and feeling he has no future in a hostile in building our movie dataset, we limited the language of tweets to english. made of unstructured text, tweets tend to be noisy and often presented with incomplete sentences, irregular expressions, and out of vocabulary words. this involved scraped reviews to undergo several cleanup steps, including the removal of duplicate tweets and any hashtags, replies and mentions, retweets, and links. in addition, we pulled out punctuation symbols and numbers, and lastly converted the text to lowercase. 5 we used the microsoft translator text api for language detection and retained tweets distinguished as english with certainty greater than 0.1. at this stage, we excluded empty and single-word tweets from our review collection, and all in all this reduced the total number of posts from our initial scraping of 15,149 down to 7,928 tweets. in contrast, plot summaries are well-defined and only required the conversion to lowercase. respectively, vocabularies of plot summaries and user reviews consisted of 863 and 24,915 tokens. ## experiments our evaluation comprises both a spoiler classification and ranking components. first, we applied zero-shot learning @xcite to our unlabeled tweets and identified the presence or absence of a spoiler bias. then, our proposed nts model produced probability for each summary-post pair to rank the bias of spoilers in tweets. performance of our nts approach is further compared to a language modeling and monolingual translation baselines. moreover, our study analyzes the ## discussion given its proven record of state-of-the-art performance for various nlp tasks, we considered the use of the bert architecture @xcite a transformer based language model that was pretrained on a large-scale corpus of english tweets (bertweet) @xcite . however, the smaller number of 850m tweets bertweet employs compared to 2b posts in our corpus, along with the dearth of support for nts at the time of publication precluded further use of bertweet in our study. of great practical importance is the answer to how the genre of a movie or the year in which it was released weigh in on the amount of spoiler bias. average sari scores of spoiledness as a function of the primary movie genre (family, romance, and thriller considered secondary) are shown in figure 8a . somewhat surprisingly, biography and science fiction types came most vulnerable, as adventure and action titles scored lower by about 8 sari points, and documentary category appears the most immune to spoilers. in figure 8b , we reviewed bias impact of the year the movie was launched. most biased are movies of the seventies and eighties, as the more recent twenty first century movies are clustered in the middle, and west side story (1961) ranked the least biased. ## conclusions we proposed a new dataset along with applying advanced language technology to discover emerging spoiler bias in tweets that represent a diverse population with different cultural values. performing text simplification, our model principally drew on named entity learning in an item-user paradigm for media review and showed through correlational studies plausible performance gains. the results we presented in this study carve several avenues of future research such as minimizing spoiler bias before user comments reach their audience, and incorporating novel pretrained language models and training schemes to improve bias ranking quality. when supplied with the proper dataset, we foresee our generic method aid researchers to address a broader bias term in text.