A Novel Contrastive Learning Method for Clickbait Detection on R o C li C o: A R omanian Clickbait Corpus of News Articles (2023)

Abstract: To increase revenue, news websites often resort to using deceptive news titles, luring users into clicking on the title and reading the full news. Clickbait detection is the task that aims to automatically detect this form of false advertisement and avoid wasting the precious time of online users. Despite the importance of the task, to the best of our knowledge, there is no publicly available clickbait corpus for the Romanian language. To this end, we introduce a novel Romanian Clickbait Corpus (RoCliCo) comprising 8,313 news samples which are manually annotated with clickbait and non-clickbait labels. Furthermore, we conduct experiments with four machine learning methods, ranging from handcrafted models to recurrent and transformer-based neural networks, to establish a line-up of competitive baselines. We also carry out experiments with a weighted voting ensemble. Among the considered baselines, we propose a novel BERT-based contrastive learning model that learns to encode news titles and contents into a deep metric space such that titles and contents of non-clickbait news have high cosine similarity, while titles and contents of clickbait news have low cosine similarity. Our data set and code to reproduce the baselines are publicly available for download at https://github.com/dariabroscoteanu/RoCliCo.

## introduction clickbait detection @xcite , the task of automatically identifying misleading news titles, has received a raising interest in recent years @xcite @xcite @xcite @xcite @xcite due to its importance in addressing concerns regarding the deceptive behavior of online media platforms aiming to increase their readership and revenue in an unethical manner. the task has been widely explored in multiple languages, such as english @xcite @xcite @xcite , chinese @xcite , hindi @xcite , and turkish @xcite , among others. however, for low-resource languages, such as romanian, there is a comparatively lower amount of research studying clickbait detection @xcite , primarily due to the lack of annotated resources. in this context, we introduce a novel romanian clickbait corpus (roclico) comprising 8,313 news samples which are manually annotated with clickbait and non-clickbait labels. to the best of our knowledge, roclico is the first publicly available corpus for romanian clickbait detection. the news articles were collected from six publicly available news websites from romania, while avoiding overlapping publication sources between our training and test splits. this ensures that the reported performance levels are not artificially increased because of models relying on factors such as authorship or publication source identification. it is important to clearly separate between satirical, fake and clickbait news. satirical news represents a type of fake news, where the intention of the author is to make the readers laugh. in satirical news, it is clear to the readers that the reported events are fictitious. fake news reports fake events or facts, where the intention of the author is to deceive the readers into believing that the fake events or facts are true. clickbait news represents a distinct category where the news title is deceptive with respect to the content. in some clickbait news, the content can present real events and true facts. in other clickbait news, the reported events or facts can be fake. hence, these three categories (satirical, fake, clickbait) are distinct, although their intersection is not necessarily empty. we perform experiments with several machine learning models. because some recent challenge reports @xcite showed that shal-low and handcrafted models can sometimes surpass deep models, we include both handcrafted and deep models as baselines for roclico. hence, two baselines are based on handcrafted features that are commonly used for clickbait detection, while the other three are based on deep learning architectures, such as a long short-term memory (lstm) network @xcite , and two versions of bidirectional encoder representations from transformers (bert) @xcite for romanian @xcite . among the considered baselines, we propose a novel bert-based contrastive learning model that learns to jointly encode news titles and contents into a deep metric space. the embedding space is constructed such that titles and contents of non-clickbait news have high cosine similarity, while titles and contents of clickbait news have low cosine similarity. thus, the similarity between the title and the content of an article is leveraged as a clickbait indicator. finally, the list of models evaluated on roclico is completed by an ensemble of the five individual baselines, which is based on weighted voting. in summary, our contribution is twofold: ## related work related corpora. clickbait detection was first studied for the english language, several english resources being available to date @xcite . with the rising interest towards solving the task, researchers started to compile and publish clickbait detection corpora for various languages, including chinese @xcite , hindi @xcite , hungarian (vincze and szabó, 2020), indonesian @xcite , thai @xcite and turkish @xcite . these resources vary in size, ranging from 180 data samples for hungarian (vincze and szabó, 2020) to 52,000 data samples for hindi @xcite . there are three corpora containing more than 30k samples, but these comprise only tweets. corpora formed of full news articles usually have less than 10k samples. with over 8k samples, roclico is similar in size with the resources available for other languages. however, most of the other corpora are based on distant supervision, labeling the news as clickbait based on the source platforms they belong to. in contrast, our corpus is based on manual annotations. we note that news articles from a certain source platform cannot be placed in the same bucket, although it is true that different platforms use clickbait in different percentages. for roclico, these percentages vary between 17% and 71%, supporting our previous statement. to the best of our knowledge, there are no publicly available data sets for romanian clickbait detection. we thus consider roclico as a useful resource for the low-resource romanian language. related methods. to date, researchers proposed various clickbait detection methods, ranging from models based on handcrafted features @xcite to deep neural networks @xcite @xcite . however, most of these approaches are based on standard learning objectives, little attention being dedicated to alternative optimization criteria, such as those based on contrastive learning @xcite . @xcite trained a variational auto-encoder to jointly predict labels from text and generate text from labels using a mixed contrastive loss, while @xcite employed contrastive learning to minimize the distances between headlines from the same category and maximize the distances between headlines from different categories. our model learns a metric space between titles and contents, while existing contrastive learning methods learn a metric space between titles, which is fundamentally different. to the best of our knowledge, we are the first to propose a contrastive learning objective based on title and content pairs. the novelty of our approach consists of directly transforming each news article into a pair, essentially eliminating the need of performing hard example mining @xcite . ## corpus roclico gathers news from six of the most prominent romanian news websites (cancan, digi24, libertatea, protv, wowbiz, viva), while only containing articles from the public web domain, being free of subscription. the corpus consists of 8,313 samples distributed over two classes: 4,593 non-clickbait and 3,720 clickbait. as the corpus is manually labeled, each sample contains a title, a body, and a label that deems it clickbait or not by taking into consideration the characteristics of both text and title. there are 3,942,790 tokens in our data set, the average number of tokens per article being roughly 474. the average title length is 21 tokens, while the average content length is 454 tokens. as expected, the titles are much shorter than the contents. in terms of the number of sentences, the news articles vary between 3 and 448. the average number of sentences per news article is 28. all data samples were labeled by annotators who received detailed instructions on the factors that should be taken into account when labeling news as clickbait or non-clickbait. the annotators are native romanian speakers, who graduated from their bachelor studies at romanian universities. we checked their reading comprehension skills prior to enrollment. each annotator was asked to read an instructive document written in romanian, where we provided definitions, guidelines and examples of clickbait and non-clickbait news articles. the labeling process, completed by three annotators, consists of filtering each sample through a set of criteria by identifying the properties of the title and the relationship between the body and the headline (title). as clickbait titles tend to misguide or attract readers by overemphasizing certain aspects, we recommended annotators to take into considerations the following characteristics: containing questions for the reader, teasers, emotionally evocative structures, suggesting the existence of a photo or video, being controversial, or being racist. we note that the title's inclusion in one of the following genres may also represent an indicator for its membership to the clickbait category: political, finance, celebrity. consequently, we instructed the annotators to be extra careful when they have to annotate a news article from the politics, finance, or celebrity categories. these genres are picked by studying romanian news websites and finding that the most common themes likely to contain clickbait are the ones above. to assign the non-clickbait category, even if one of the above criteria fits for the title, the body and its similarity with the headline determine the final separation line between the two classes. the final label for a sample is decided based on the majority vote of three annotators, which reduces labeling errors by a large extent. the cohen's kappa coefficient among our annotators is 0.73, indicating a substantial agreement between annotators. we confirm the consistency of the labels by checking the assigned labels for a subset of 200 samples. after completing the labeling process, we found that the clickbait ratio per source ranges between 17% and 71%, indicating that all the considered news sources seem to use clickbait titles to some extent. this is likely because the chosen sources do not use a subscription-based publication format, and the news publishers have to rely on advertisement to obtain earnings. we separate the news platforms between the training and test sets, keeping the news from cancan, protv and wowbiz for training and those from libertatea, viva and digi24 for testing. as such, the training set consists of 6,806 articles, while the test set comprises a total of 1,507 articles. the distribution of classes per subset is presented in without source separation between training and test, a model that overfits to certain source-specific aspects that are not related to clickbait (e.g. author style, preference towards certain subjects, and so on) will reach high scores on the test. however, these scores are unlikely to represent the actual performance of the model in a real-world scenario, where news can come from different sources without annotated samples to be used for training. thus, we consider that a more realistic evaluation is to separate the platforms. in the proposed setting, models that learn patterns related to author style, preferred themes and such will not be able to capitalize on features unrelated to clickbait detection. ## baselines rf and svm based on handcrafted features. our first baseline embodies a morphological, syntactical and pattern analysis model, similar to the recent approach of coste et al. ( 2020 ), which uses a set of engineered features based on part-of-speech tagging, scores (clscore, lix, and rix), and punctuation patterns. first, we lowercase the titles and strip down special characters. next, we extract features such as the number of question words, punctuation patterns in the headline, the lix and rix indexes @xcite , the clscore (coleman and liau, 1975), part-of-speech tags based on stanza @xcite , the number of common nouns, and the number of proper nouns. these features are then passed to two separate models, a random forest (rf) model and a support vector machines (svm) classifier. bilstm network. our second baseline approaches the problem via a recurrent architecture based on bidirectional lstm (bilstm) layers @xcite . there are two bilstm branches, one for titles and one for contents. each input is first preprocessed by the keras tokenizer. next, word embeddings are extracted using fast-text @xcite . the resulting embeddings are passed as input to the bilstm neural network. the outputs of the separate bilstm branches are pooled, concatenated, and further passed through two fully-connected layers and a softmax classification layer. fine-tuned ro-bert. our third baseline involves the fine-tuning of the romanian bert (dumitrescu et al., 2020), a transformer-based model trained exclusively on a comprehensive collection of romanian corpora. to fine-tune the model, the title and body of each news sample are concatenated using the separation token [sep] as a delimiter, and the result is encoded using the ro-bert encoder. the [cls] tokens returned by the last transformer block are used as sample embeddings in our classification task. the ro-bert embeddings are passed through a dropout layer, a fully-connected layer, and a classification layer. finally, the labels are predicted by applying argmax to the probabilities produced by the classification layer. contrastive ro-bert. as a novel technique to solve the clickbait detection problem, our contrastive learning approach employs siamese networks @xcite to learn the similarity between news headlines and contents. this is achieved by harnessing the distance between headlines and contents in a loss function that penalizes the clickbait pairs that are too close or the non-clickbait pairs that are too far apart. let @xmath0, x 2 , ..., x n } and @xmath1, y 2 , ..., y n } represent our training set of news samples and the set of corresponding labels, where x @xmath2, c i ) is a pair given by title t i and content c i , and y @xmath3, 1} is the class label (0 for clickbait and 1 for nonclickbait), for all @xmath4, 2, ..., n}. our approach applies a siamese ro-bert model to generate the normalized embeddings v t i and v c i for each title t i and body c i , respectively. we further compute the proposed contrastive loss, as follows: we set @xmath5. unlike our approach, related contrastive methods use (t i , t j ) pairs, i.e. a pair is composed of two titles of different news articles. such methods require hard example mining, resulting in an expensive training process. in contrast, our approach has the same training complexity as the standard bert. ensemble. as our baselines consist of relatively distinct approaches to the problem, we consider the idea of combining the results into an ensemble of classifiers worthy of investigation. we gather the soft labels predicted by the baselines presented so far and use a weighted voting mechanism to predict the label of each sample. validation set, which is distinct from the test data. the weight of a model is equal to the ratio between the number of correct labels and the number of validation samples. ## results in our empirical study, we perform binary classification experiments aimed at predicting whether a news is clickbait or non-clickbait. as evaluation metrics, we report the precision, recall and f 1 score for each of the two classes, on the official test set. the results obtained by the individual baselines, as well as their weighted ensemble, are shown in we evaluate the statistical significance of the proposed contrastive ro-bert with respect to the finetuned ro-bert using a paired mcnemar's test. the test indicates that the performance improvement brought by our contrastive learning method is statistically significant, with a p-value of 0.001. in figure 1 , we illustrate the precision-recall (pr) curves of the individual models on the test set. each curve is summarized via the average precision (ap) measure, which represents the area under the pr curve. in terms of ap, it looks like the best model is the fine-tuned ro-bert. although the ap score of our contrastive ro-bert model is lower, our approach generates the sweetest spot on the graph, i.e. the closest point to the top-right corner. this point corresponds to the best trade-off between precision and recall. the overall results of the models evaluated on the clickbait detection task suggest that predicting non-clickbait articles is fairly easy. however, clickbait articles are comparatively harder to detect, regardless of the employed model. this indicates that romanian clickbait detection is a challenging task, which remains to be solved in the future with the help of our new corpus. ## conclusion in this work, we introduced roclico, the first public corpus for romanian clickbait detection. we trained and evaluated four models and an ensemble of these models to establish competitive baselines for future research on our corpus. among these models, we proposed a novel approach, namely a bert model based on contrastive learning that uses the cosine similarity between title and content to detect clickbait articles. the empirical results indicate that there is sufficient room for future research on the romanian clickbait detection task.