C lark K ent at S em E val-2023 Task 5: SVM s, Transformers, and Pixels for Clickbait Spoiling (2023)

Abstract: In this paper we present an analysis of our approaches for the 2023 SemEval-2023 Clickbait Challenge. We only participated in the sub-task aiming at identifying different clikcbait spoiling types comparing several machine learning and deep learning approaches. Our analysis confirms previous results on this task and show that automatic methods are able to reach approximately 70\% accuracy at predicting what type of additional content is needed to mitigate sensationalistic posts on social media. Furthermore, we provide a qualitative analysis of the results, showing that the models may do better in practice than the metric indicates since the evaluate does not depend only on the predictor, but also on the typology we choose to define clickbait spoiling.

## introduction there are many posts in social media that are intended to allure the readers into visiting a specific web page. the phenomenon is commonly known as clickbait and it is usually achieved through sensationalistic formulations that strike curiosity into the readers mind. studies show that humans get dopamine rewards during information seeking processes driven by curiosity or pleasure of anticipation @xcite . clickbait titles are linguistic exploits that activate these mechanisms and therefore users tend to reduce the perceived information gap @xcite by clicking or accessing the advertisements. while these type of linguistic exploits may be used to steer people into making healthier choices such as avoiding unhealthy foods, making more exercises etc. @xcite , most often, the information overload of clickbait posts from social media poses serios mental health risks @xcite and additional risks of propagating misleading information for profit or by the far-right for questionable political purposes @xcite . clickbait spoiling implies rephrasing the original title or the introduction of additional content in the original post with the purpose of limiting the curiosity of the readers. a techno-solutionist pipeline to spoilling for social media can be described by the following steps: the first step, for detecting whether a post is clickbait or not, has been greatly explored in recent years including a widely participated clickbait challenge in 2017 @xcite and several individual studies mentioned here briefly @xcite @xcite @xcite . the task of identifying the different types of spoiler and generating the adequate information for clickbait spoiling (steps three and four) has been tackled more recently with the works of @xcite ; @xcite and @xcite , and the 2023 seveval task 5 on clickbait spoiling @xcite . our participation at this task is only focused on step number three -classifying different types of clickbait spoilers. in the following sections we will cover a comparison of several linear and deeplearning classifiers together with a quantitative and qualitative analysis of the data. ## background the semeval task organizers @xcite have provided a typology of spoilers comprising of three main classes: in according to ## system overview support vectors we experiment with several types of support vector machine trained without hyper-parameter tuning. the models use features extracted from the post, title or content of the linked documents: ## experimental setup to be consistent with the shared task organizers and to be able to compare against previous work hagen et al. (2022), we report balanced accuracy for each trained model. additionally we also measured mathews correlation coefficient (mcc) which appeared to be in-line with the balanced accuracy measures, showing objectively considerably low values between 0.2 and 0.51. in our experiments, we noticed that the absolute difference between the average matthews correlation coefficient (mcc) on 5-fold cross-validation and the mcc on the test and development sets was smaller compared to the differences between balanced accuracy scores. the table provided as an example ( this phenomenon somewhat confirms previous investigations @xcite ) that advocate for mcc being a superior metric over accuracy of f1 based scores. due to space constraints, we do not report the mcc scores in all our tables, but we indicate the reader to see our reproducible notebooks in the public repository 4 . we compared each model against a stratified 5fold cross-validation approach, where each split included a proportional number of examples from every class. we opted for 5-fold cross-validation to ensure that each split of the combined train (3200) and dev (800) sets from the shared task had similar sizes. we did not use the test set provided by the organizers for cross-validation purposes. we consider this method to be more appropriate for comparing the models, as it demonstrates their ability to generalize on new data. the svm model experiments do not benefit from hyperparameter tuning, as we maintained the default parameters specified in scikit-learn, with a regularization constant of @xmath0. we conducted experiments using various feature sources for both svm and deep learning models. based on our results with svm classifiers, we found that the original post texts are the most effective source for identifying the type of clickbait spoiling. this finding aligns with similar results reported in previous experiments on the task @xcite . we can conclude therefore, that the type of spoiler is more of a characteristic of the post than a pattern of the linked document or its title. additionally, we observed that combining the post text with title or document information only degrades the overall results. semeval-task5-2023 overall, for deep learning methods, due to limited access to compute resources, we only employ the post text for classification and not the document or the title. ## conclusion our investigations reveal that transformer models have a limited capacity at identifying the clickbait spoiler types, reaching around 70% accuracy. this raises the question of whether the typology itself is consistent. the majority of confusions occur in multipart examples, as this class is characterized by the presence of discontinuous passages and phrases in the linked document. upon examining the examples, we found that a significant portion of multipart examples contain very short phrases or passages, making it difficult even for humans to distinguish this category from the others without knowing the actual spoiler. lastly, we do not believe this to be the biggest obstacle in reaching the final goal: to diminish the amount of sensationalistic content on social media. after all, this classifier would only feed the decisions of text generator at the following steps. in the end, we believe that content creators and companies will be less enthusiastic to reduce their clickthrough rate of their social-media posts due to such spoilers and possibly new types of language mechanisms that include both images and texts will be used to both bypass the clickbait detectors and exploit the human psychology. in the face of such challenges, we require further research to prepare multilingual and lightweight models that can be easily adapted to a wider variety of media.