S ci T ech B ait RO : C lick B ait Detection for R omanian Science and Technology News (2024)

Abstract: In this paper, we introduce a new annotated corpus of clickbait news in a low-resource language - Romanian, and a rarely covered domain - science and technology news: SciTechBaitRO. It is one of the first and the largest corpus (almost 11,000 examples) of annotated clickbait texts for the Romanian language and the first one to focus on the sci-tech domain, to our knowledge. We evaluate the possibility of automatically detecting clickbait through a series of data analysis and machine learning experiments with varied features and models, including a range of linguistic features, classical machine learning models, deep learning and pre-trained models. We compare the performance of models using different kinds of features, and show that the best results are given by the BERT models, with results of up to 89% F1 score. We additionally evaluate the models in a cross-domain setting for news belonging to other categories (i.e. politics, sports, entertainment) and demonstrate their capacity to generalize by detecting clickbait news outside of domain with high F1-scores.

## introduction and related work clickbait is a form of content used with the intention of attracting as many readers as possible through a type of content supported by a specific title, designed to attract as many clicks as possible. news media is no stranger to this way of attracting readers. furthermore, the technique has been used for more than 100 years under the name of "yellow journalism" or "yellow press" (britannica, 2024), i.e. that type of journalism that used shocking, "sensational" headlines to attract readers to buy the newspaper, without the news being interesting or at least partially supported by real facts. clickbait is used nowadays by news publications to promote articles on social networks (i.e. facebook, instagram) by engineering news titles to contain certain terms, words, and patterns that arouse curiosity or revolt, such as "incredible", "you must read this" or "it is outrageous" etc. this is to the detriment of the readers, who are being manipulated into clicking misleading links. given how widespread this practice is and the amount of news published daily, automatic solutions for detecting clickbait can be a welcome solution. some social media channels, such as facebook, have already implemented a protocol to reduce clickbait content. technology and science play a crucial role in shaping modern society, driving progress, and improving the quality of life. from medical advancements that extend and save lives to innovations in communication that connect people across the globe, the impact of science and technology is profound and far-reaching. socially, these fields are essential for addressing some of the most pressing challenges of our time, such as climate change, health crises, and sustainable development. by providing the tools and knowledge needed to understand and solve complex problems, science and technology empower societies to make informed decisions, promote economic growth, and enhance social equity. in essence, the advancement of technology and science not only drives progress but also ensures that society can adapt, thrive, and respond effectively to the evolving needs of its members. in the technology and science domain, disinformation can be especially harmful by twisting scientific results and the public's trust in science. recent examples such as research on the covid-19 virus and vaccines have shown that misinformation about scientific findings can easily spread through manipulation methods such as clickbait and can be profoundly harmful for society. one of the first studies on clickbait detection using machine learning techniques was published by @xcite , where standard ml models were used, including logistic regression (lr), naive bayes (nb), random forest (rf). the study was conducted on a compilation of a corpus of 2,992 english tweets, among which 767 were in the clickbait category. a novel contribution brought by this study is the generation of 215 independent variables that were further split into three types: teaser message aimed at capturing the characteristics of the clickbait teaser message (bag-of-words features, sentiment polarity, tweet's readability, use of terrier stopword list @xcite , list of the easy words dale and chall, use of contractions, punctuation use); link-based features (analysis of web pages concerning tweets); meta information (encoding the tweet's sender, attaching an image/video to the tweet, retweet, the part of the day the tweet was sent). @xcite investigated several lexical and syntactic features for clickbait, achieving remarkable performance. a set of 15,000 article titles were analyzed with various features created and divided into various categories: sentence structure (average word length, title length, etc.), word patterns (existence of special punctuation patterns), clickbait language (standard and classical phrases from clickbait domain, slang, jargon @xcite ), n-gram features. more recent studies use deep learning @xcite for classifying clickbait for the english language. few studies on clickbait detection were performed for languages other than english. one notable example includes studies on clickbait detection performed for turkish @xcite by forming and expanding a turkish language dataset -clickbaittr @xcite , as well as indonesian language, with the release of click-id @xcite . for the romanian language, on the other hand, the clickbait sphere has not been studied extensively. the only reserach in this direction is the very recently published study of @xcite introducing roclico, a general domain corpus specifically designed for clickbait in romanian. @xcite perform experiments on a previous version of our dataset, obtaining a 0.85 f1-score with a multi-layer perception classifier. our contributions in this paper include the release of an annotated corpus for the romanian language on clickbait in the science and technology area, scitechbaitro -the first corpus for clickbait detection for science and technology news, and the largest clickbait corpus for romanian, including approximately 11,000 samples. we apply various artificial intelligence algorithms in order to automatically predict clickbait titles, and show that detection is possible with an f1-score of 90%. section 2 describes the methodology used for building our corpus, starting from the details related to the dataset collection and annotation, to data analysis and duplicate detection. in section 3, we describe a series of machine learning experiments performed to automatically detect clickbait based on the constructed corpus. we use a variety of models from classical ml models to deep learning and pre-trained transformers. we create linguistic-based features and measure the performance of models using these features in comparison to the pre-trained models using simple word sequence features. section 4 reports the results obtained from the models, followed by experiments to measure the performance of the best model on out-of-domain news data (news belonging to other domains such as politics, economics) in section 5. finally, the last section (section 6) concludes the study and offers some perspectives on what could be studied further in this direction. ## corpus construction given the lack of annotated datasets in the field of news articles in the scientific-technology category, we release an annotated corpus of clickbait news in these domains for the romanian language. it represents the first dataset of this kind, including annotated clickbait news for the romanian language, scitechbaitro foot_0 . we collected a number of 10,867 articles from the scientific-technology area published on romanian news websites between 1.02.2021 and 1.02.2022 and manually annotated the type of article (clickbait or not). we obtained 5,464 titles identified as non-clickbait and the remaining 5,403 in the clickbait category. the following subsections discuss in more detail the methodologies used for collecting and annotating the data. ## methodology in this section we discuss the experiments performed for automatic detection of clickbait news based on our introduced dataset. we experiment with various types of supervised machine learning models in order to learn to predict clickbait, including classical ml models, deep learning, and pre-trained transformers. ## results this section relates the obtained results in all the different settings discussed in the previous section. all reported results were obtained using single runs. the main metric used for comparison was f1-score. results on additional metrics can be found in the appendix. ## conclusions and further work the main goal of this research is to introduce scitechbaitro, a new annotated corpus of clickbait news in romanian, a low-resource language, with a novel focus on the science and technology area., and to evaluate the feasibility of automatically detecting clickbait on these data. we experiment with various machine learning models and features in order to automatically detect the clickbait news, obtaining results upward of 89% f1-score. we additionally show that classifiers trained on our dataset can perform well on other examples of romanian online news from different domains as well (i.e. sports, politics, economics, roclico corpus), showing the models are able to generalize to other domains. while our results show that clickbait can be accurately detected with our methods, some future research could improve performance. incorporating the body of the news articles as well as using other models (such as hierarchical attention networks (han) @xcite or setfit) for clickbait news classification could be promising directions.