B angla C lick BERT : B angla Clickbait Detection from News Headlines using Domain Adaptive B angla BERT and MLP Techniques (2023)

Abstract: News headlines or titles that deliberately persuade readers to view a particular online content are referred to as clickbait. There have been numerous studies focused on clickbait detection in English language, compared to that, there have been very few researches carried out that address clickbait detection in Bangla news headlines. In this study, we have experimented with several distinctive transformers models, namely BanglaBERT and XLM-RoBERTa. Additionally, we introduced a domain-adaptive pretrained model, BanglaClickBERT. We conducted a series of experiments to identify the most effective model. The dataset we used for this study contained 15,056 labeled and 65,406 unlabeled news headlines; in addition to that, we have collected more unlabeled Bangla news headlines by scraping clickbait-dense websites making a total of 1 million unlabeled news headlines in order to make our BanglaClickBERT. Our approach has successfully surpassed the performance of existing state-of-the-art technologies providing a more accurate and efficient solution for detecting clickbait in Bangla news headlines, with potential implications for improving online content quality and user experience.

## introduction the internet has led to a surge in the use of online news media, which provides users with easy access to information at any time. however, news websites use clickbait headlines that can be misleading and frustrating to users. these headlines are designed to attract users and create suspense, containing exaggerated information that does not match the content. clickbait headlines aim to lure users into clicking on them but ultimately cause frustration. pengnate et al. concluded a research and found that clickbait headlines can lead to higher click-through rates, but may lead to negative user experiences such as frustration and disappointment. examples of clickbait headlines in bangla are in figure 1. the core differences between clickbait and non-clickbait is described in appendix a. the use of online news media has increased rapidly in bangladesh, with an estimated 66.3 million internet users foot_0 and 14 million online readers of prothom alo @xcite , one of the top newspapers in the country. however, the increasing number of clickbait titles on news websites has become a significant issue, leading to frustration and disappointment among users. while research has been conducted on clickbait detection in english, very little has been done in bangla, a language spoken by millions of people in bangladesh and other countries. in english, for the clickbait challenge 2017, webis clickbait corpus 2017 @xcite was created which had a total of 38,517 sentences from major us news publishers. in bangla, mahtab et al. have constructed a bangla clickbait detection dataset containing 15,056 labeled news articles and 65,406 unlabelled news articles. in this paper, we present banglaclick-bert, a pretrained model for clickbait detection in bangla news websites. we use the labeled dataset for training and validating our model and scrape clickbait-dense websites to gather more unlabelled news article headlines, increasing the number of unlabelled news headlines to around 1 million. we use this to pretrain the banglabert (bhattacharjee et al., 2022) model, which we then pretrain to create banglaclickbert. the main contributions of this paper can be summarized as follows: ## literature review the roots of clickbait can be found in tabloids, a form of journalism that has existed since the 1980s @xcite . the three primary sources from which clickbait identification attributes may be generally retrieved are (1) the related article that the post text wants the user to visit, (2) metadata for both, and (3) the connected article (munna and hossen, 2021). potthast et al. and biyani et al. additionally took into account metadata, related content, and handcrafted elements in addition to the post-text analysis. they used methods like gradient boosted decision trees (gbdt) and assessed the tf-idf similarity between the headline and article content. potthast et al. in another paper also mentioned the clickbait challenge 2017, which invited the affirmation of 13 detectors were presented as the clickbait detectors for screening, realizing considerable enhancements in detecting performance above the prior state of the art. zhou first used a self-attentive rnn to choose the crucial terms in the title before building a bigru network to encode the contextual information for the 2017 clickbait challenge. on the contrary, thomas used an lstm model for the clickbait challenge that included article content. to create the word embedding of clickbait titles, rony et al. applied the continuous skip-gram model. nevertheless, indurthi et al. were the first to study the use of transformer regression models in clickbait identification and won the clickbait challenge. additionally, hossain et al. produced the first dataset of bengali newspapers for bengali false news detection of around 50k bangla news articles in an annotated dataset. besides bangla, we have explored about clickbait detection techniques in news and social media in other languages. genç and surer used logistic regression (85% accuracy), random forest (86% accuracy), lstm (93% accuracy), ann (93% accuracy), ensemble classifier (93% accuracy), and bilstm (97% accuracy) on 48,060 headlines from news sources pulled from twitter for turkish clickbait detection. moreover, razaque et al. used long short-term memory, word2vec and compared their models with naive bayes classifier for clickbait detection on social media. bronakowski et al. achieved 98% accuracy in recognizing clickbait headlines by using thirty distinct types of semantic analysis and six different machine-learning approaches, both individually and in groups. the suggested models can be used as a model for creating useful programs that swiftly identify clickbait headlines. farhan et al. used gated recurrent unit (gru) and convolutional neural network (cnn)-based ensemble model for sarcasm detection for bangla language achieving 96% f1-score and accuracy. it gave us an insight on what type of work can be done using nlp and for gathering knowledge and examples related to our work. additionally, beltagy et al. created scibert which is a pretrained language model, based on bert used unsupervised pretraining on scientific articles, providing us knowledge about domain-adaptive bert which can help enhance efficiency on a range of scientific nlp tasks and produce cutting-edge results. moreover, jahan et al. created banglahatebert, which is a retrained version of the pre-existing banglabert model, and trained it having a widespread corpus of hostile, insulting, and offensive bengali language, and outperformed the generic pretrained language model in various datasets. so, to sum up with, we have analysed about the origins of clickbait, checked different datasets on different languages, learned about differnet nlp methods, and observed the potentials of specialized transformers models like scibert and banglahatebert. ## problem statement we approach the task of clickbait detection as a decision-making challenge; a binary classification task problem with two main categories @xmath0, non -clickbait}. given a set of bangla news headlines @xmath1, t 2 , t 3 , . . . , t n }, our objective is to predict labels @xmath2, y 2 , y 3 , . . . , y n } for these headlines. here, y i assumes the value 1 if title t i is classified as clickbait and 0 if it is classified as non-clickbait. ## dataset description the dataset @xcite we used consists of two sets: an annotated set and an unannotated set of clickbait news information. the information with our augmentation is shown in ## methodology we have used some statistical models and deep learning models and then we have implemented transformers models like banglabert, xlm-roberta and domain adaptive banglaclick-bert with several variations. based on these variation, we try to come up with the best model. ## results and analysis as depicted in transformer models; banglabert, xlm-roberta and banglaclickbert demonstrated consistent improvements over all other approaches. in particular, using only the last layer in conjunction with mlp yielded excellent results. notably, both banglabert base and banglabert large performed similarly, 73.08% and 73.38%, indicating that increasing the model's parameters did not contribute significantly to this specific clickbait detection task. on the other hand, the domain adaptive banglaclickbert exhibited better than that of banglabert and xlm-roberta, respectively, with f1 score of 74.24%. this underscores the effectiveness of pretraining the model with domainadaptive data for clickbait detection. considering the average of all layers proved to be both advantageous and disadvantageous as it captured more informative representations. the f1-score of banglabert and xlm-roberta decreased, whereas, it proved to be beneficial for domain adaptive banglaclickbert as taking the average of all its layer increased the f1-score by 1.01%. it again proves that pretraining the layers of banglabert has helped all its layers to understand more about clickbait sentences. model names attention weighted words banglabert [cls] এক িমস ##কল ## ◌ই মধু র স ক সব নাশ ত ণীর [unk] . দখু ন (িভিডও) [sep] banglaclickbert [cls] এক িমস ##কল ## ◌ই মধু র স ক সব নাশ ত ণীর [unk] . দখু ন (িভিডও) [sep] raw headline এক িমস কেলই মধু র স ক সব নাশ ত ণীর. দখু ন (িভিডও) translated headline one missed call destroys the sweet relationship of the young woman. @xcite moreover, concatenating the embeddings from two pretrained language models further enhanced performance, illustrating that combining related models could capture complementary information for clickbait detection in bangla. the combination of domain adaptive banglaclickbert and xlm-roberta achieved the highest f1-score of 75.51% for the clickbait class surpassing other models we discussed about including the gan-banglabert @xcite . in terms of precision, recall, f1-score, and accuracy, the domain adaptive banglaclickbert model proved to be more consistent to all other models. however, since the f1-scores were tightly clustered within the range of 0.74 to 0.75, to support our claims, we ran each model ten times with different seeds and conducted a statistical test. the model, labeled "domain adaptive banglaclick-bert + xlm-roberta + embeddings concatenated. before concatenating passed through one linear layer. followed by mlp" outperforms all other models, and the difference in performance is statistically significant (@xmath3.05) according to @xcite ." this finding is further supported by the t-sne visualization depicted in figure 5 and figure 6 . the t-sne visualization effectively shows how these models, even without fine-tuning of the training data, group their predictions. it becomes evident that banglaclickbert exhibits better clustering than banglabert, underscoring the idea that training banglaclickbert can enhance the learned representations and subsequently improve overall performance. this can be shown more prominently in figure 6 that banglaclickbert managed to cluster the embeddings of clickbait headlines better than banglabert. additionally, as illustrated in in conclusion, the results suggest that banglaclickbert, proves to be highly effective for clickbait detection in bangla. if more and better labelled data is used to finetune this, this approach will perform better than other approaches. ## conclusion in conclusion, this study represents a significant advancement in the field of clickbait detection, particularly for the bangla language, where research has been limited. while clickbait detection in english has been extensively studied, the bangla news headlines have been largely overlooked. to address this gap, we conducted a comprehensive analysis using state-of-the-art transformer models, such as banglabert, xlm-roberta, and the newly developed banglaclickbert. we enhanced the performance of these models by incorporating mlp methods to achieve the best results. to bolster the research, we augmented the dataset by including an additional 1 million unlabeled bangla news headlines, sourced from clickbait-dense websites. this expanded dataset significantly empowered the banglaclickbert model. through rigorous experimentation and testing, our approach showed better results compared to existing state-of-the-art techniques. our work not only contributes to the improvement of clickbait identification in bangla news headlines but also fills the void in research in this language domain. as clickbait continues to impact the way information is consumed, our findings will be valuable for media organizations, content creators, and platforms to promote responsible and reliable information dissemination in the bangla-speaking community.