Spoiler Detection as Semantic Text Matching (2023)

Abstract: Engaging with discussion of TV shows online often requires individuals to refrain from consuming show-related content for extended periods to avoid spoilers. While existing research on spoiler detection shows promising results in safeguarding viewers from general spoilers, it fails to address the issue of users abstaining from show-related content during their watch. This is primarily because the definition of a spoiler varies depending on the viewer’s progress in the show, and conventional spoiler detection methods lack the granularity to capture this complexity. To tackle this challenge, we propose the task of spoiler matching, which involves assigning an episode number to a spoiler given a specific TV show. We frame this task as semantic text matching and introduce a dataset comprised of comments and episode summaries to evaluate model performance. Given the length of each example, our dataset can also serve as a benchmark for long-range language models.

## introduction many online social platforms (e.g., reddit, discord) provide opportunities for fans to discuss a particular tv show and share their thoughts about details in episodes. however, engaging in such discussions comes with the risk of spoilers, which may lead to unsatisfactory viewing experiences @xcite . as a result, many viewers avoid these communities altogether until they have caught up with the latest episode of the show. as an attempt to resolve this problem, some platforms (such as reddit) have built-in functionality allowing users to tag their content as containing just watched ep. 5 remeo died! omg no!! juliet actually kissed him?? 15 replies 35 replies the music in ep1 is beautiful 5 replies spoiler predictions [hidden due to spoiler] juliet actually kissed him?? 15 replies 35 replies the music in ep1 is beautiful 5 replies irrelevant ep 2 ep 12 # i can discuss without worrying about spoilers! online community comments a spoiler. however, for long shows with a large number of episodes, this proves unsatisfactory: a user could be halfway through a show but will still be afraid to click on spoiler-tagged content for fear that it might contain spoilers for events later in the show when in reality, it might pertain to events that the viewer has already seen and with which they can engage. again, some websites allow users to tag spoilers with more granularity, but it is far from guaranteed that users will be both accurate and consistent in tagging their content. this highlights the need for automatic spoiler matching. unlike traditional spoiler detection @xcite @xcite , which determines whether a comment is a spoiler, spoiler matching aims to match a given spoiler to an episode number. a spoiler matching model working hand-in-hand with a spoiler detection model could provide much more fine-grained protection from spoilers. in this work, we consider the setting where the show is known, and we would like to determine the episode to which a comment is referring, as shown in figure 1 . specifically, we pose the problem as a semantic text matching @xcite task between comments and episode summaries. we obtain a high-quality evaluation dataset by manually manual labeling after data cleaning, we group the comments by show name and episode number while traditional spoiler detection classifies text as spoiler or non-spoiler, we at this step look to separate the irrelevant comments we have scraped from the relevant ones. we define a relevant comment as one that describes events from the episode (discussion thread) from which it is scraped. examples of irrelevant comments are discussions about music, acting quality, personal feelings about the episode, etc. automatic labeling to obtain a larger-scale training set, we split the 11,032 manually labeled examples into a small training, validation, and test set by the ratio of 7:2:1, to train an auto-labeler. to maximize data efficiency with this small dataset, we perform prompt-based fine-tuning to train roberta @xcite . specifically, we use lm-bff @xcite to fine-tune the model to predict if the comment is relevant. we evaluate lm-bff under two settings. for the first, we perform prompt-based fine-tuning on a set of hand-crafted templates, then select the best one according to validation set auc. for the second, we use the automatic template search (auto-t) to generate and rank 40 templates based on validation auc. under both settings, we do not perform the auto-l search and instead fix the label words as "relevant" and "irrelevant". from this ranking, the top 20 templates are selected, and the logits from the corresponding models are averaged. it is important to note that at this step, summaries are not concatenated with comments before being fed to the auto-labeler; the auto-labeler predicts relevant/irrelevant based on the words in the comment alone. this is because the job of the auto-labeler is to filter out generic irrelevant comments; unlike matching, episode-specific context is not required to perform this task. using our lm-bff ensemble auto-labeler, the 511,959 unlabeled comments are auto-labeled, separating 204,475 relevant comments from 307,484 irrelevant ones. using our test set, we estimate that about 12% of the relevant comments are actually irrelevant. as we will demonstrate later, this number is low enough that the auto-labeled comments still serve as an effective training set for fine-tuning a spoiler matching model. ## case study in this section, we analyze comments from two shows in the validation set, dr. stone and spy × family to understand behaviors of the models. 4 the second comment describes an event from the show but references a character from an entirely different show; it is drawing a comparison between two characters, one of them in the show, based on physical likeness. the reference is fairly well-known within the community, but without additional external information, it would be difficult for the model to understand this comment beyond just context clues. the third comment is challenging because it concerns predictions. comments on ongoing shows often contain predictions, and if they happen to be correct, will likely match better lexically to the future episode when the events occur than the current episode when they are foreshadowed/predicted. this is not necessarily a bad thing for the end user, but it poses a challenge for training and evaluating our models. for this example, it is visually hinted in episode 11 that the dog has the ability to see the future but not confirmed until episode 13, so the summary for episode 11 does not mention this ability explicitly but the summary for episode 13 does, posing a possible explanation for the model's behavior. taken together, these examples give a glimpse into the challenges posed by spoiler matching. the hope is that this analysis motivates new lines of work into the study. ## conclusion in this work, we define a new task of spoiler matching that formulates spoiler detection as a semantic text matching task and construct a large scale dataset with 223k comments and 496 episode summaries by mixing human and automatic labeling. we benchmark the performance of bm25 and four language models on the proposed dataset.