Billie-Newman at S em E val-2023 Task 5: Clickbait Classification and Question Answering with Pre-Trained Language Models, Named Entity Recognition and Rule-Based Approaches (2023)

Abstract: In this paper, we describe the implementations of our systems for the SemEval-2023 Task 5 ‘Clickbait Spoiling’, which involves the classification of clickbait posts in sub-task 1 and the spoiler generation and question answering of clickbait posts in sub-task 2, ultimately achieving a balanced accuracy of 0.593 and a BLEU score of 0.322 on the test datasets in sub-task 1 and sub-task 2 respectively. For this, we propose the usage of RoBERTa transformer models and modify them for each specific downstream task. In sub-task 1, we use the pre-trained RoBERTa model and use it in conjunction with NER, a spoiler-title ratio, a regex check for enumerations and lists and the use of input reformulation. In sub-task 2, we propose the usage of the RoBERTa-SQuAD2.0 model for extractive question answering in combination with a contextual rule-based approach for multi-type spoilers in order to generate spoiler answers.

## introduction clickbait involves a text, usually accompanied by a thumbnail or a link, specifically designed to invoke the interest or curiosity of a user. this is done with the intent of enticing the user into clicking the text in order to satisfy their created curiosity. in task 5 from semeval-2023, the clickbait challenge engages with the study of multi-class classification and extractive question answering in respect to these aforementioned clickbait posts @xcite . for this purpose, the task suggests the act of clickbait spoiling, which involves the generation of a short text that answers this supposed curiosity gap @xcite . while models engaging in clickbait spoiling can be used for the purpose of saving a user a click on clickbait posts, the application of models able to question answer a set of questions reliably extends to many more use cases such as answering search requests formed as questions in classic information retrieval, making insights gained from question answering models valuable. in order to facilitate the study, task 5 is divided into two subtasks: sub-task 1: spoiler type classification this sub-task involves the automated classification of clickbait posts into the category of • phrase: a post that can be answered with a single phrase such as persons, dates or organisations ## background for the clickbait challenge, task 5 provided participants with a high-quality annotated data set called the 'webis clickbait spoiling corpus 2022' that serves as the basis for the systems, essentially used for training and evaluating resulting models @xcite . it contains 5000 clickbait posts crawled from social media platforms such as facebook, reddit and twitter, accompanied by complete annotations and labels about their respective type of spoiler in sub-task 1 and the manually created spoiler answers for sub-task 2. in specifics, these clickbait posts contain manually cleaned excerpts from the articles behind the links and also classifies which exact passages classify as categorized spoilers. to aid participants in classifying and generating spoilers, all these posts and their respective spoilers are classified into the three types, including short phrase spoilers, long passage spoilers and multi spoilers, while the spoiler answers are mainly categorized as extractive spoilers, with abstractive, rephrased spoilers being the absolute minority. being kept private until the end of the shared task. the overall spoiler type distribution, categorized by their respective split, is further defined in table @xcite . each post contains various amount of fields that help identifying posts and their respective context and spoilers, as shown in detail in table @xcite , with the grand majority of present language of the posts being in the english language domain. in this section, we will discuss the various components that are included in the implementations of our systems for sub-task 1 and sub-task 2 in closer detail. ## experimental setup this section explains the experimental setups that were used for implementing our systems for subtask 1 spoiler classification and sub-task 2 spoiler generation. ## results in this section, we will discuss the performance of our submitted systems. it involves the roberta-ner system outlined in section 3.1 and both the v1 and the v2 systems outlined in section 3.2 for sub-task 1 and 2 respectively. all systems were submitted to the tira platform as executable docker images for the sake of reproducibility and were compared against the performance of the naive and transformer baselines as provided by the task organizers, where the naive baseline for sub-task 1 simply always predicts a 'passage' spoiler-type for each classification task and spoils each clickbait post with the title of the linked page in sub-task 2 3 . ## conclusion all in all, our systems did not manage to outperform either of the basic transformer baselines provided by the challenge organizers. in sub-task 1, the inclusion of features based on named entities, spoiler-title ratios, multi-enumeration checks and input reformulation led to a decrease in average accuracy in comparison to the basic transformer baseline, while in sub-task 2 we saw a more minor decrease in bleu score when compared to the provided baseline. the visible decrease of performance in sub-task 1 in comparison to the transformer baseline could possibly be explained by the implementation of the spoiler-title-ratio metric, which was not rounded after the second decimal. thus, it might have led the model to being trained and possibly overfitted on very precise numbers up to the 16th or further decimal, therefore causing the model to not perform well on the test set. for sub-task 2 we submitted two different systems: v1 without additional rule-based logic and v2 with additional rule-based logic for multi type spoilers. upon inspection of the final results, we see a minor performance increase in the latter v2 system on both the validation and test set, when inspecting and comparing the systems in respect to multi-type spoiler answers in detail, we can see an even larger performance difference between the v1 and v2 systems, overall suggesting that the addition of rulebased logic in combination with the transformer model approach had a positive effect on the observed performance scores. for future research, it would make sense to further delve into the positive insights gained in this paper. originally, our team planned to implement a complete rule-based approach for question answering covering all types of spoilers by following a simple logic routine explained in papers such as @xcite . the approaches in this paper, however, did not seem very suitable for the task at hand, as its context is centered around rule-based question answering approaches for elementary school types of questions involving reading comprehension test, making it an inherently different use case than the use case found in the semeval-2023 task 5 context involving news-and blog posts functioning as clickbait. seeing how the simple rule-based logic addition in sub-task 2 led to a small positive increase in performance across the board, would suggest that further research into implementing rule-based logic such as taking into account semantic or syntactic patterns, in conjunction with transformer-based approaches might be worth looking into for further research. finally, it might also be useful for further research to find ways to combine the features of sub-task 1 and sub-task 2 together in one unified system, where a reliable multi-class classification model predicts the spoiler type correctly, which then dictates how and which rule-based approach the question answering system uses for generating the appropriate spoiler answer.