Comparison of Multilingual and Bilingual Models for Satirical News Detection of A rabic and E nglish (2024)

Abstract: Satirical news is real news combined with a humorous comment or exaggerated content, and it often mimics the format and style of real news. However, satirical news is often misunderstood as misinformation, especially by individuals from different cultural and social backgrounds. This research addresses the challenge of distinguishing satire from truthful news by leveraging multilingual satire detection methods in English and Arabic. We explore both zero-shot and chain-of-thought (CoT) prompting using two language models, Jais-chat(13B) and LLaMA-2-chat(7B). Our results show that CoT prompting offers a significant advantage for the Jais-chat model over the LLaMA-2-chat model. Specifically, Jais-chat achieved the best performance, with an F1-score of 80% in English when using CoT prompting. These results high- light the importance of structured reasoning in CoT, which enhances contextual understanding and is vital for complex tasks like satire detection.

## introduction satire is the act of making fun of someone or something intending to embarrass or discredit them (assiri and himdi, 2023) @xcite . satire is context-dependent, which is why satirical news can sometimes be mistaken for misinformation, even though there is no intention of misleading any parties, making satirical news prone to being misclassified as "false positive" misinformation @xcite . most existing methods focus on satire detection in a single language, with limited research on multilingual approaches. zero-shot prompting of large language models (llms) has been explored, but this technique struggles with satire detection due to a lack of context. this research investigates how cot prompting improves the accuracy of bilingual and multilingual models, using jais-chat @xcite foot_0 and llama-2chat @xcite . bilingual models like jais-chat are trained on only two languages, english and arabic in our case, while multilingual models like llama-2-chat are trained on more than two languages. our paper provides insight into how specialized, language-focused training compares to more general, multilingual training, particularly in the context of satire detection for english and arabic texts. this research aims to answer: i) how does the performance of a bilingual model compare to a multilingual model in detecting satire across languages? and ii) what impact does cot prompting have on accuracy? we evaluate jais-chat foot_1 and llama-2-chat foot_2 across two languages (english and arabic) using cot prompting. our results indicate that cot prompting outperforms zero-shot prompting for satire detection, particularly with the jaischat model, whereas llama-2-chat showed minimal improvements with cot, maintaining consistent performance across both prompting methods. our contributions include: the rest of the paper is organized as follows: section 2 reviews the prior research on satire detection. section 3 outlines our proposed methodology and experiment setup. section 4 presents the results of our experiments, and section 5 concludes the paper with a discussion of findings and future work. despite advancements in satire detection, challenges persist, especially with multilingual support and cot prompting. this paper tackles these issues by leveraging the jais-chat and llama-2-chat models, both trained on english and arabic, and integrating them with cot to enhance accuracy and nuance in satire detection. ## results as observed in it is worth noting that the llama-2-chat model achieved exceptional recall scores across all datasets, exceeding 97%. this suggests that while the model may struggle with precision, it is highly effective at identifying relevant instances, potentially indicating a tendency to classify more instances as positive. over-classifying instances as satirical risks dismissing legitimate information, while over-classifying instances as non-satirical could lead to the spread of false information as credible. both scenarios contribute to the spread of misinformation. therefore, the trade-off between recall and precision should be carefully considered in the context of satire detection. ## conclusion this study explores the efficacy of satire detection using multilingual models utilizing different prompting techniques, comparing the bilingual jaischat model with the multilingual llama-2-chat model. referring to the research questions, we observe that the multilingual llama-2-chat model produces consistently stable outcomes regardless of the prompting technique. in contrast, the bilingual jais-chat model demonstrates more variable results, showing significantly improved performance with cot prompting compared to zero-shot prompting. the results indicate that cot prompting improves or maintains performance depending on the model. future work should aim to refine these models, expand datasets, and include more languages to better address the complexities of satire in diverse cultural contexts. improving satire detection methodologies can enhance public understanding of media content and reduce the spread of misinformation in an increasingly complex information landscape.