NITK - IT - NLP @ D ravidian L ang T ech: Impact of Focal Loss on M alayalam Fake News Detection using Transformers (2023)

Abstract: Fake News Detection in Dravidian Languages is a shared task that identifies youtube comments in the Malayalam language for fake news detection. In this work, we have proposed a transformer-based model with cross-entropy loss and focal loss, which classifies the comments into fake or authentic news. We have used different transformer-based models for the dataset with modifications in the experimental setup, out of which the fine-tuned model, which is based on MuRIL with focal loss, achieved the best overall macro F1-score of 0.87, and we got second position in the final leaderboard.

## introduction social media is becoming the most popular media through which people share information happening around them. online social networks such as twitter, facebook, instagram, weibo, and many others are major channels for people to obtain an enormous amount of information; meanwhile, there can be unauthentic or fake information among them. the central aspect of fake news is that it fully caters to the audience's curiosity, which is why it spreads faster @xcite . moreover, research proves that false information or fake news spreads faster than authentic news @xcite . thus it is very urgent to detect fake news and stop its propagation to prevent potential harm. fake news detection in dravidian languages is a shared task organized by dravidianlangtech, which focuses on detecting fake news from youtube comments annotated as fake and original. the dataset was retrieved from youtube comments which are in malayalam language. we have proposed a transformer-based model that used crossentropy and focal loss function. the rest of the paper is organized as follows. first, in section 2, some related works are explained. followed by the dataset description in section 3. then in section 4 we explain the different models and the methodology we have used, followed by data preprocessing and model description in section 5 and 6. the result and discussions are explained in section 7 followed by the conclusions and future scope in section 8 2 related works fake news detection is one of the popular problems in natural language processing tasks. we can see from the literature that an enormous amount of research has been carried out to tackle the propagation of fake news. in this section, we provide some of the works done on mitigating fake in the dravidian language, especially in malayalam. we have also included some recent works developed on fake news detection for other languages. @xcite has proposed a hindi fake news detection model that utilizes linguistic feature-based word embedding. they mainly focus on 24 key features which are extracted and derived for the successful detection of fake news. @xcite has proposed a method that uses both word embeddings and linguistic features for the identification of textual fake news. @xcite has created fact-checked data in 13 different indian languages covering different news stories from 2013 to 2020. the data were mainly retrieved from social media websites. they have also analyzed the characterization of multilingual, multimedia, and multidomain. @xcite has created a dataset for fake news detection in the tamil language. they have scrapped news snippets from various news media and annotated them into binary, fake, and real labels. the news articles scrapped cover different regions like sports, politics, and science, which they manually annotate. (hariharan and anand kumar, 2022) has studied the impact of different transformer-based models using the data they have created using translation. the dataset covers different domains retrieved from fakenewsamt and celebrity fake news, which are translated into tamil and malayalam languages using google translate. it is evident from the literature the number of works for the dravidian language, especially malayalam, is much less than the other languages, the main reason being the lack of proper annotated data resources. hence, we should develop a system that could address these challenges. ## dataset description the shared task provided data retrieved from youtube comments and annotated for fake news detection @xcite . the dataset was mainly in malayalam, and some had code-mixed content. the dataset was labeled into two classes, 'fake' and 'original', and the detailed statistics as shown in ## methodology the shared task was to classify the youtube comment into corresponding labels correctly. we were supposed to get a prediction for the test data with the training and validation data provided. moreover, we were allowed to use external data to train the model. we have employed transformer-based pre-trained models for training the proposed model. even though the task was for fake news detection in the dravidian language, and the dataset given was for malayalam, we had code-mixed samples. hence we used the multilingual version of the bert @xcite model for training the data. we further experimented with the multilingual version of roberta @xcite and the muril @xcite model. we used the binary cross entropy loss function as the loss function in all of our models and have experimented with focal loss. ## data preprocessing as we were using malayalam and code-mixed to train the model, we applied some basic preprocessing before training the model. the dataset from youtube comments was cleaned using the python library cleantext, which helps remove the unknown characters and do the ascii conversions. as mentioned previously, we were using a pretrained model based on the transformer; hence we had to tokenize the data according to how the model expected. we use the tokenization method from the hugging face library, corresponding to the underlying pre-trained model. ## model description we mainly used three pre-trained models for training the multilingual bert (m-bert) and muril from google, multilingual roberta (xlmroberta). the m-bert and xlmroberta were trained on multilingual data, and the muril model was trained specially for the indian context. the muril model has been pre-trained with multilingual representations from various indian languages, and they have augmented data that used the translated and transliterated pairs of documents for training. we used recommended hyperparameters for all the models whose particulars are given in ## results and discussion this section explains the different experiments we have conducted for the shared task and the results for the same. we used the pre-trained transformerbased models to build our system. we have finetuned the transformer-based models using the hyperparameters mentioned in table, using adamw as the optimizer. the experiments were conducted on tesla p100 16 gb gpu on the kaggle platform. we initially begin training the data with the m-bert model after the necessary preprocessing, as explained in the section. we used cross-entropy loss for the experiments. we used the experimental settings for training the model on xlmroberta and muril. the results are shown in