Predicting Clickbait Strength in Online Social Media (2020)

Abstract: Hoping for a large number of clicks and potentially high social shares, journalists of various news media outlets publish sensationalist headlines on social media. These headlines lure the readers to click on them and satisfy the curiosity gap in their mind. Low quality material pointed to by clickbaits leads to time wastage and annoyance for users. Even for enterprises publishing clickbaits, it hurts more than it helps as it erodes user trust, attracts wrong visitors, and produces negative signals for ranking algorithms. Hence, identifying and flagging clickbait titles is very essential. Previous work on clickbaits has majorly focused on binary classification of clickbait titles. However not all clickbaits are equally clickbaity. It is not only essential to identify a click-bait, but also to identify the intensity of the clickbait based on the strength of the clickbait. In this work, we model clickbait strength prediction as a regression problem. While previous methods have relied on traditional machine learning or vanilla recurrent neural networks, we rigorously investigate the use of transformers for clickbait strength prediction. On a benchmark dataset with ∼39K posts, our methods outperform all the existing methods in the Clickbait Challenge.

## introduction clickbait refers to those sensational, provocative or controversial posts which appear to be informative and objective, but are designed to entice its readers into clicking the link accompanying with the post. according to a survey of 53 stanford students, 96.2 percent of stanford students encounter clickbait articles on the internet at least once per day foot_1 . @xcite estimate that 19.46% of headlines were "clickbait" in 2014; 23.73% in 2015; and 25.27% in 2016. beyond the increased prevalence, clickbait is also a challenge across multiple modes of data, text, images and even videos 3 . the economic model of the contemporary online news industry @xcite incentivizes more content views. a report by the columbia journalism review highlighted the case of online magazine slant, which pays writers @xmath05 for every 500 clicks on their stories. this clearly motivates journalists to write catchy and suspenseful headlines. cognitively, human minds have a tendency to satisfy and bridge their curiosity gap by clicking on the link. marketing companies have been using clickbaits to attract and engage more number of users resulting in getting more page views. websites need more page views to promote their content or to create more opportunities to show advertisements which increase their revenue. moreoever, it is a well clickbaits play with human psychology and sometimes are a wastage of time. they create a curiosity gap for the users through the short post, but do not make judicious attempt to fill it in the clicked article, thereby creating an information void. it is quite annoying to have social feeds spammed by over-promising headlines that lead users to under-delivering half-stories. even for enterprises which use clickbaits for effective marketing, clickbaits are hurtful more than being helpful for the following reasons: (1) misleading clickbait damages brands and erodes user trust, (2) clickbaits attract wrong visitors rather than interested ones, (3) user interaction with clickbaits produces negative signals for ranking algorithms, (4) clickbait muddles the website's important data, and (5) sensationalism is now seen as more disappointing by smart users. ## dataset the webis clickbait corpus consists of 38517 tweets @xcite . restricting to englishlanguage publishers, @xcite obtain a ranking of the top-most retweeted news publishers from the newswhip social media analytics service foot_4 . taking the top 27 publishers, they used twitter"s api to record every tweet they published in the period from december 1, 2016, through april 30, 2017. they filtered and sampled from this collection of 459541 tweets to obtain a clean dataset of 38517 tweets. each of the tweets was annotated for clickbait intensity label by five different workers from amazon mechanical turk (amt). a 4-point likert scale was followed with these values: not clickbaiting (0.0), slightly clickbaiting (0.33), considerably clickbaiting (0.66), heavily clickbaiting (1.0). of this, 19538 (of which 4761 are clickbaits) tweets have been released for training with labels. the maximum post size is 25. the post lengths follow a normal distribution around a mean of 12. we split the labeled data into 80:20 ratio for training and validation. we perform 5-fold cross validation and compile the results on the validation set. the remaining 18979 (of which 4515 are clickbaits) tweets are used by the clickbait challenge server for testing the submissions. this test set is private and not accessible publicly. moreover, there are extremely limited number of test runs which can be submitted to the test server. empirical observations reveal that the field posttext (text of the post) in the given dataset contributes majorly to decide the intensity of the clickbait. hence, in spite of the availability of the tweets' metadata like the post media, the title of the target linked page, the content paragraphs and keywords of the target page, the time of the post and caption of every image in the target article, we use only the post text of the tweet to train a machine learning model to predict the clickbait intensity score of each tweet. we leave further exploration of other metadata fields as part of future work. ## evaluation metrics the goal for the clickbait intensity prediction task is to develop a model that can predict how click baiting a social media post is. the score is a real number between 0 and 1. mean squared error (mse) with respect to the mean judgments of the annotators is used as the primary evaluation metric. models whose preditions have the lowest mse would be ranked on the top. unlike other classification task, where f1 score or accuracy is the evaluation metric, this challenge focuses more on predicting the intensity of the title than classifying the title as clickbait or not. official evaluation is done on the platform called tira @xcite . this platform evaluates the predictions by running the code for predictions in a virtual machine in a sandboxed environment which ensures that the test data is kept private and not revealed to public. moreover, there are a limited number of times one can predict on the test data to ensure that the models are not trained to overfit the test data. the evaluation platform also computes secondary evaluation metrics such as the median absolute error (medae), the f1-score (f1) and accuracy (acc) with respect to the truth class. ## approach we formulate the problem of clickbait strength prediction as a regression problem. we build multiple regression models using pretrained word embeddings, pretrained transformer representations and finetuned transformer representations for clickbait strength prediction. we experiment with various regression algorithms and rigorously investigate the efficacy of these for clickbait strength prediction. ## experiments in this work, we try various kinds of approaches and investigate how they perform for the clickbait prediction task. we use multiple models to transform the text into features. we train multiple regression models on the above features and evaluate the efficacy of each of the pre-trained embeddings (bert, gpt2 or roberta) for the downstream clickbait prediction task. we experimented with the following regression techniques: (1) simple linear regression (lr), (2) ridge regression (rr), (3) gradient boosted regressor (gbr), (4) random forest regression (rfr), ( 5 ## conclusion in this paper, we proposed various methods for clickbait intensity prediction based on the title of the post. using a benchmark dataset from the clickbait challenge, we evaluate multiple models; we are the first to investigate effectiveness of transformer based models for this task. as of now, we rank at the top on the official leaderboard for the challenge. we plan to work on reducing the model size and improve runtime latency using popular knowledge distillation methods.