Crafting Tomorrow’s Headlines: Neural News Generation and Detection in E nglish, T urkish, H ungarian, and P ersian (2024)

Abstract: In the era dominated by information overload and its facilitation with Large Language Models (LLMs), the prevalence of misinformation poses a significant threat to public discourse and societal well-being. A critical concern at present involves the identification of machine-generated news. In this work, we take a significant step by introducing a benchmark dataset designed for neural news detection in four languages: English, Turkish, Hungarian, and Persian. The dataset incorporates outputs from multiple multilingual generators (in both, zero-shot and fine-tuned setups) such as BloomZ, LLaMa-2, Mistral, Mixtral, and GPT-4. Next, we experiment with a variety of classifiers, ranging from those based on linguistic features to advanced Transformer-based models and LLMs prompting. We present the detection results aiming to delve into the interpretablity and robustness of machine-generated texts detectors across all target languages.

## introduction the remarkable power of current advances in natural language processing (nlp) has enabled the creation of text that closely resembles human-authored content @xcite . while these technological achievements offer both positive and negative prospects, a significant concern arises from the proliferation of neural-generated news, which may not always present accurate information. thus, in the pipeline of fake news detection, a pivotal stage can be authorship identification, either by a human or a machine. the task of news or story generation appeared to be a fascinating challenge already since the popularization of @xcite . indeed, it can be an excellent technology for editing support @xcite . then, the appearance of gpt-2 @xcite opened new horizons in neural news generation @xcite . however, generators still suffered from a english: tiktok is now facing a ban in the united states, a fate that has already befallen a string of american social media giants that tried to make it in china. turkish: türkiye'de tarım sektörü, yapay zeka teknolojisinin gelişimi ile büyük bir dönüşüm yaşıyor. yapay zeka destekli tarım uygulamaları, çiftçilere daha verimli, sürdürülebilir ve karlı tarım yöntemleri sunuyor. lack of human fluency and were not able to handle proper long text generation, forgetting about the context. as a result, in the news domain, such generative tasks as headlines generation @xcite and news summarization @xcite received more attention. in the past, several machine-generated text detection benchmarks were created in the domains of scientific papers @xcite , story-telling texts @xcite , question-answering datasets, and wikipedia articles @xcite . for the news domain, such datasets were generated with the grover model @xcite based on gpt-2. however, especially without evidence, e.g. historical facts, the nature of the task remains challenging, given the advances in the domain of nlp. moreover, all these datasets were mostly designed only for one language, english, and did not cover the news domain and recent models sufficiently. to this end, we chose turkish, hungarian, and persian (native languages of the authors) to create a proofof-concept to determine whether underrepresented languages can also thrive with the recent generative ai advances, specifically for news creation and classification of ai-generated content. thus, our contributions are the following: the constructed benchmark dataset and the bestperforming classifiers are available for public usage online. 1 the first dataset of generated texts that specifically focused on the news domain was created using grover @xcite . the grover model for neural news generation was based on gpt-2 @xcite and already created a strong benchmark for neural news detection in english. after that, a dataset of gpt-2-generated news headlines was also introduced in @xcite . @xcite , a benchmark dataset for generated scientific texts detection was presented. the authors utilized powerful modern lms including those specifically fine-tuned for scientific texts domain, like scigen @xcite and galactica @xcite . in the context of multilingual datasets, the m4 dataset @xcite covers seven languages from various domains like question-answering, wikipedia, news, and legal texts. the authors also benchmarked various baselines, ranging from linguistically informed approaches to transformerbased classifiers, and concluded with zero-shot prompting of language models (lms). additionally, the multilingual benchmark collection can be expanded with the hc3 corpus @xcite , which includes question-answering data for english and chinese. for russian, the ru-atd @xcite corpus provides 1 tum-nlp/neural-news-benchmark also, refer to appendix a for all models links. text summarization, paraphrasing, and translation data. however, previous studies have not addressed the examination of neural texts for underrepresented and complex languages such as turkish, hungarian, and persian. we are closing this gap with our work while also including a popular language, english. ## models for news generation the first step before training models for news generation was evaluating the pre-trained llms in producing news content akin to proficient news editors. therefore, the aim of the initial experiments was to assess the capability of existing models in generating news content across all four considered languages in a zero-shot setting and then fine-tune promising models for the downstream task of news generation. ## benchmark dataset after generative models selection, we constructed the novel benchmark dataset for neural/human authorship classification. the dataset consists of equal parts human-written and machine-generated news, and is evenly stratified across languages. the neural articles were generated by prompting diverse llms: the fine-tuned models described above, gpt4, and four additional open-source models. ## models for generated news detection for neural news detection, some classical machine learning benchmarks as well as the latest deep learning based on transformers encoder were assessed @xcite . ## results all the results are presented in the out-of-domain data put stress on the classifiers robustness. for the zero-shot prompted bloomz and llama, mbert and xlm-roberta gain the top places, however, with a drop in the performance. notably, xlm-r demonstrates near-perfect scores for persian and hungarian in bloomz, as well as across all languages in llama. at the same time, for bloomz for english and turkish, both models have a significant decline in the classification results. furthermore, we continued even further out-ofthe-box experiments on texts generated by mistral, mixtral, and gpt-4. for these models, a common trend emerges in the results: all classifiers show high detection scores for the persian language, possibly due to the shorter length of texts in this language, but with a significant drop of the score for all other languages. however, the random forest classifier consistently produces the top results. in other languages, linguistically informed detectors as well demonstrate more resilient performance. notably, for gpt-4 in english, random forest once again achieves the highest f1 score. these findings suggest a hypothesis that linguistically informed detectors for out task may offer the most promising, robust, and potentially explainable models compared to the black-box transformer-based classifiers. in the final phase, we experimented on a smaller test set with prompting llms to detect the generated texts from llms as well. these initial findings yield quite encouraging implications. thus, outof-the-box llama-2 can nearly perfectly detect generated texts in all languages from our llama-2 instance fine-tuned for the downstream task and even gpt-4. similarly, mistral achieves comparable results, albeit with a slight decrease in performance for turkish and hungarian. at the same time, gpt-4 show the best performance at the detection of english bloomz generated data, but shows flaws in the detection of its own generated texts. thus, llms indeed can be more robust detectors, however, requiring significantly greater resources compared to linguistic classifiers. this potentially means that mono-lingual or multi-lingual models may struggle to learn to write like a news agent dur- generative model discriminative model pre re f1 pre re f1 pre re f1 pre re f1 english turkish hungarian persian in-domain test set: samples from these generative models were the part of training data for the discriminators. fine-tuned bloomz-3b lr .96 .96 .96 .98 .98 .98 .98 .98 .98 .99 .99 .99 rf .98 .98 .98 .97 .97 .97 .97 .97 .97 1.00 1.00 1.00 svm .98 .98 .98 .99 .99 .99 .98 .98 .98 .99 .99 .99 mbert .99 .99 .99 1.00 1.00 1.00 .99 .99 .99 .99 .99 .99 xlm-r .99 .99 .99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 fine-tuned llama-2 chat-7b or llama-7b persian lr .97 .97 .97 .99 .99 .99 .98 .98 .98 .99 .99 .99 rf .97 .97 .97 1.00 1.00 1.00 .99 .99 .99 1.00 1.00 1.00 svm .98 .98 .98 1.00 1.00 1.00 .99 .99 .99 .99 .99 .99 mbert .99 .99 .99 1.00 1.00 1.00 .99 .99 .99 .99 .99 .99 xlm-r .99 .99 .99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 out-of-domain test set: samples from these generative models were unseen for discriminators. zero-shot bloomz-3b lr .70 .61 .56 .50 .50 .34 .75 .61 .56 .82 .79 .78 rf .70 .59 .52 .25 .49 .33 .68 .54 .43 .99 .98 .98 svm .74 .61 .55 .50 .50 .35 .77 .62 .57 .82 .77 .75 mbert .77 .53 .40 .77 .57 .47 .81 .70 .68 .96 .96 .95 xlm-r .79 .64 .59 81 .70 .67 98 .98 .98 1.00 1.00 1.00 zero-shot llama-2 chat-7b lr .82 .81 .80 .74 .57 .49 .77 .66 .61 .79 .77 .76 rf .85 .84 .84 .75 .51 .36 .68 .56 .47 .99 .98 .98 svm .86 .84 .84 .77 .57 .48 .80 .68 .64 .76 .72 .72 mbert .88 .84 .83 .99 .99 .99 .99 .99 .99 .90 .88 .88 xlm-r 1.00 1.00 1.00 .98 .98 .98 .99 .99 .99 1.00 1.00 1.00 zero-shot mistral lr .24 .46 .32 .70 .53 .40 .72 .60 .54 .83 .79 .78 rf .24 .48 .32 .75 .51 .34 .71 .54 .42 1.00 .99 .99 svm .25 .48 .33 .77 .57 .48 .75 .61 .56 .87 .83 .83 mbert .76 .55 .44 .76 .54 .42 .87 .83 .83 .93 .92 .92 xlm-r .76 .54 .42 .78 .53 .40 .93 .92 .93 .93 .92 .92 zero-shot mixtral lr .72 .60 .54 .75 .59 .52 .76 .54 .42 .85 .84 .84 rf .75 .67 .64 .77 .57 .48 .76 .53 .40 1.00 1.00 1.00 svm .73 .61 .55 .77 .58 .49 .76 .53 .39 .85 .80 .79 mbert .75 .50 .35 .77 .56 .45 .75 .50 .34 .93 .92 .92 xlm-r .76 .57 .47 .75 .52 .39 .75 .50 .35 .98 .98 .98 gpt-4 lr .55 .51 .41 .57 .51 .37 .57 .51 .37 .88 .85 .85 rf .56 .53 .45 .47 .50 .34 .55 .50 .35 .99 .99 .99 svm .59 .53 .44 .60 .51 .35 .64 .51 .35 .86 .82 .82 mbert .44 .49 .36 .65 .52 .38 .88 .85 .85 .99 .99 .99 xlm-r .75 .50 .34 .59 .52 .42 .25 .50 .33 .89 .85 .85 test subset for zero-shot prompting of out-of-the-box llms. fine-tuned bloomz-3b gpt4 .93 .93 .92 .98 .98 .98 .88 .83 .83 .98 .98 .98 mistral .90 .87 .87 .81 .69 .65 .99 .99 .99 1.00 1.00 1.00 llama-2 .90 .87 .87 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 fine-tuned llama-2 chat-7b gpt4 .93 .92 .92 .97 .96 .96 .88 .83 .83 .95 .95 .95 mistral 1.00 1.00 1.00 .91 .88 .88 .97 .97 .97 1.00 1.00 1.00 llama-2 .98 .98 .98 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 gpt4 gpt4 .93 .93 .92 .79 .73 .72 .86 .82 .82 .90 .88 .87 mistral .98 .98 .98 .90 .88 .87 .88 .84 .84 1.00 1.00 1.00 llama-2 .97 .96 .96 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 ## conclusion in this work, we presented the first of its kind benchmark dataset for machine-generated news detection covering four languages-english, turkish, hungarian, and persian. to gain machine-generated data, we assessed the most promising llms' ability of news generation for the target languages. our final choice was to explore bloomz, llama-2, mistral, mixtral, and gpt-4 in zero-shot setups as well as additionally fine-tune bloomz and llama-2 for the downstream generation task. we also used opensource datasets and a small portion of scraped news instances to gain human-written data. in the end, we use the generated news from fine-tuned bloomz and llama instances to construct the training split for classifiers and then utilize samples from other generators for out-of-domain experiments. then, based on our dataset, we trained and tested diverse classifiers-linguistically informed baselines like logistic regression, random forest, and support vector machine were compared with mbert and xlm-r. among these models, xlm-r showed the highest results on the in-domain data but dropped performance in several out-ofdomain experiments. at the same time, linguistically informed classifiers illustrated better robustness. finally, we also experimented with llms to detect llm-generated news. llama-2 showed a quite promising performance for all languages even with ability gpt-4 generated texts. we can conclude that while baselines can be fine-tuned for the multilingual-generated news detection task, they continue to suffer in detecting the nuances in outof-domain data to a degree. but, such models may offer greater explainability. llms can prove exceedingly robust for the task, although the resource requirements for these models are higher.