Overview of the shared task on Fake News Detection from Social Media Text (2023)

Abstract: This document contains the instructions for preparing a manuscript for the proceedings of RANLP 2023. The document itself conforms to its own specifications and is therefore an example of what your manuscript should look like. These instructions should be used for both papers submitted for review and for final versions of accepted papers. Authors are asked to conform to all the directions reported in this document.

## introduction online social network (osn) platforms such as twitter, facebook, whatsapp, and instagram are extensively used by millions of users in this modern internet era to publish and spread the news about emergent events to many people more quickly and without any validation or verification. according to the @xcite the core ideas in cascading news and sensitive information are ingrained in truth notions and communication accuracy theories. according to a recent statistics report given in statistica 2023 foot_1 , facebook has 2.96 billion monthly active users, while the count for twitter has reached 556 million monthly active users, whereas what-sapp, as well as instagram, have more than 2 billion active users 1. a vast amount of news is shared and propagated among socially connected network users without knowing the authenticity of the news during the election campaign, trending events, and pandemic emergencies. misinformation creators are intentionally flooding falsified and unverified information for various political and commercial purposes. hence, a significant amount of misinformation and false news has proliferated over osns, negatively influencing readers, and causing numerous negative consequences on the economy, politics, and social security. therefore, fake news detection (fnd) is demanding in the current scenario. in @xcite the authors identified primary methods currently available to spot false news and how these methods might be used in various contexts by conducting a systematic literature review. a pertinent example, the difficulties, and the ideal setting in which to use a certain technique are all provided for some approaches. in general, fnd methods can be categorized into two types: social context-based and contentbased methods as shown in figure 1 . the former is more concerned with the user engagement data such as comments, reposts, and ratings, while the latter is associated with the article's news content like title, text, image, and video @xcite . the social context-based methods can be further divided into two categories: propagation structurebased and post-based methods. the propagation structure-based methods concentrate on propagation patterns or trends of fake news on social networks, while post-based methods examine the opinions or emotions expressed by the users in their posts. due to the unstructured nature of the data, these two types of social-context techniques face the following challenges: data collection and analysis, noisy data, and missing data. hence, the focus of this shared task is on content-based strategy. the content-based methods are more straightforward and convenient to detect fake news, particularly at an early stage. transfer learning-based fnd system is introduced in @xcite according to their perspective, bert is a bidirectional language model since it considers the context of both a word's left and right sides. in contrast, gpt and elmo are only trained in the right-to-left context and the left-to-right context, respectively. the local contextual features over space and the global semantic relation features over time are then extracted in the feature representation layer using multichannel cnn and stacked bilstm. the model may learn many characteristics from several viewpoints using a multichannel cnn. the model's various channels each extract features from the same input in their own unique ways, producing a more reliable representation @xcite . in @xcite , the authors applied machine learning-based ensemble methods with the help of textual properties to distinguish fake news from the original one. ## related work the researchers used pre-trained language models such as bert, and roberta for contextual word embedding and then used dl-based models to detect fake news. in @xcite , authors present the dl-based fnd framework in which roberta and ffn are used to extract contextual dependent features and to detect fake news respectively. similar to fnd there are numerous works published such as hope speech detection @xcite and @xcite in social media posts. the author in (chakravarthi, 2022) employs a dl-based hope speech detection model in which t5-sentence and indic-bert are used for word embedding to capture the contextual relationship among words. then the contextual features are sent as input to cnn to detect the hope speech comments @xcite . the proposed model's performance is evaluated on a multilingual dataset named hopeedi which is introduced in the shared task 2021 (chakravarthi, 2020). dhivya @xcite proposed a two-stage hope detection process in which the language detector identifies the language of the model, and the hope detector classifies the text into hope speech, non-hope speech, or not lang. the various pretrained language models and dl-based models are proposed, and their performance is evaluated on three hope speech datasets in english, tamil, and malayalam. ## task description the task aims to identify fake news from the posts or comments in dravidian languages such as malayalam which is collected from the youtube osn. each comment/post is annotated at the comment/post level and assigned the class labels fake or real. ## dataset description the dataset is balanced since the number of samples in fake and real classes is almost nearer. the dataset contains a total of 5,091 comments of which 2,512 are fake news and 2,579 are real news. the dataset is split into training, validation, and testing. the detail of the dataset is shown in ## methodology in this shared task, there are eight teams actively participated and implemented their models. they evaluated their model's performance on our fake news dataset. deepblueai (luo and wang, 2023): team deepblueai used a pre-trained language model such as xlm-roberta to identify fake news. the authors employed the xlm-roberta model to extract the context-aware features from the textual news. then the contextual feature vector is sent as input to the fully connected layer with softmax to classify fake or real news. their model achieves an f1-score of 0.90 in this task. abhipaw (bala and krishnamurthy, 2023): team abhipaw abhi presented a multilingual representations for indian languages (muril) for fnd. the f1-score of 0.87 is achieved with their proposed model. nitk-it-nlp (r l and kumar m, 2023): team nitk-it-nlp used a multilingual version of the transformer-based muril model for developing the fnd system. they also introduced focal loss as the loss function while training the model. the model achieves the f1-score of 0.87 for fnd. nlpt @xcite : team nlpt malayalam employed a pre-trained language model called xlm-roberta for fnd. the proposed model achieves an f1-score of 0.87 which is better than the ml-based models. the reasons for the improvement are a self-attention mechanism of transformers, a byte-level bpe, and a dynamic masking pattern during training. mucs (sharal coelho and shashirekha, 2023): team mucs proposed tf-idf to convert words into vectors based on the occurrence of the words. the extracted features of tf-idf are passed as input to the different ml-based classifiers to predict the given news as fake or real. the model achieves the f1-score of 0.83 for fnd. @xcite : team ml ai iiitranchi proposed an ensemble ml-based fnd system which uses bag-of-words and indic bert for the textual features extraction. then, ensemble ml-based classifiers such as random forest (rf) and adaboost are employed to detect the fake news. the f1-score of 0.78 is achieved with their proposed model for fnd. dlrg rr: team dlrg rr presents the ml-based fnd system in which tf-idf is used to transform the words into vectors and passive aggressive classifier (pac) is adopted for fnd. their model achieves the f1-score of 0.73. ## results and discussions the performance assessment of the proposed models by the participating teams was conducted using the macro f1 score metric, which is a widely used measure for evaluating classification models. the results of this evaluation are presented in securing the top position, the team "deep-blueai" which achieved the first rank demonstrated an impressive macro f1 score of 0.90. their accomplishment was attributed to the adept utilization of the xlm-roberta model. this pre-trained transformer model was fine-tuned by the team's authors to effectively discern fake comments. this achievement underscores the efficacy of leveraging powerful transformer-based architectures for addressing the task. moving on, teams ranked 2nd, 3rd, and 4th garnered identical macro f1 scores of 0.87, highlighting their consistent performance. among these, the team "abhipaw," positioned at the 2nd rank, strategically employed the multilingual representations for indian languages (muril) model. while muril was primarily designed as a multilingual language model for indian languages, the team harnessed its potential for the classification task. this innovative approach signifies the adaptability of pre-trained models across diverse downstream applications. in the lower rankings, the last two teams, securing the 7th and 8th positions, achieved a macro f1 score of 0.73. their methodology involved the implementation of the passive aggressive classifier (pac) coupled with the feature weighting technique known as term frequency-inverse document frequency (tf-idf). the teams meticulously experimented with various configurations of the maximum document frequency parameter in the tfidfvectorizer, leading to their placement at the 7th rank. similarly, the team "nlp ssn cse," positioned at rank 8, employed a similar approach, utilizing an array of pre-trained transformer models like bert, albert, xlnet, and mbert. despite their diverse model ensemble, their performance closely aligned with the team ranked 7th. overall, the ranking table provides a comprehensive overview of the distinct strategies and models adopted by each participating team. this evaluation sheds light on the varying degrees of success achieved by exploiting transformer-based models, language-specific architectures, and traditional classification techniques, all contributing to the advancement of fake news detection. ## conclusion this paper presents an overview of the fake news detection shared task conducted at dravidianlangtech-ranlp 2023, specifically focusing on the malayalam language. the task garnered participation from eight teams, each submitting predictions for evaluation. the methods employed by these teams varied, spanning from traditional tf-idf vectorizers with machine learning to contemporary pre-trained transformer models for data representation. an analysis of the methodologies revealed a consistent trend: transformer-based methods outperformed other techniques, as indicated by evaluation metrics such as classification accuracy and confusion matrices. this suggests the potency of transformer models in effectively capturing fake news detection performance. in summary, the paper summarizes the dravidianlangtech 2023 fake news detection shared task for malayalam, highlighting diverse strategies, and underscoring the prevalence of transformer-based methods for improved performance.