AI ‘News’ Content Farms Are Easy to Make and Hard to Detect: A Case Study in I talian (2024)

Abstract: Large Language Models (LLMs) are increasingly used as ‘content farm’ models (CFMs), to generate synthetic text that could pass for real news articles. This is already happening even for languages that do not have high-quality monolingual LLMs. We show that fine-tuning Llama (v1), mostly trained on English, on as little as 40K Italian news articles, is sufficient for producing news-like texts that native speakers of Italian struggle to identify as synthetic.We investigate three LLMs and three methods of detecting synthetic texts (log-likelihood, DetectGPT, and supervised classification), finding that they all perform better than human raters, but they are all impractical in the real world (requiring either access to token likelihood information or a large dataset of CFM texts). We also explore the possibility of creating a proxy CFM: an LLM fine-tuned on a similar dataset to one used by the real ‘content farm’. We find that even a small amount of fine-tuning data suffices for creating a successful detector, but we need to know which base LLM is used, which is a major challenge.Our results suggest that there are currently no practical methods for detecting synthetic news-like texts ‘in the wild’, while generating them is too easy. We highlight the urgency of more NLP research on this problem.

## introduction the modern large language models (llms) can generate increasingly fluent and plausible-sounding texts, which sparks concerns about their potential misuse by bad actors. one of the emerging problems is ai-driven news "content farms": news-like sites filled with synthetic texts that are not necessarily serving a misinformation campaign, but are plausible-looking enough to deceive the readers and generate web traffic. already in may 2023 newsguard reported that they found 49 such 'outlets' @xcite , and on june 4th 2024 their count foot_0 was 840. sometimes such sites publish original 'content', and sometimes they automatically 'rewrite' articles from real news outlets without attribution @xcite . they are primarily created for serving programmatic ads, and even major brands may unwittingly support such 'outlets' @xcite . they already operate in many languages. 2 the problem will likely only get worse with time, and it needs more attention both on the policy & regulation side and from the nlp researchers. we illustrate how easy it is to create 'content farm' models (cfms), and how hard to detect them, by considering a case that should be relatively tricky. in §3 we successfully turn llama, a relatively old llm mostly trained on english, into an italian news cfm -by fine-tuning it on only 40k italian news texts. for llama65b this turns out to be sufficient to mislead native speakers of italian, who identify synthetic texts with only 64% accuracy, vs 50% random guess (see §4). we also find that existing detection methods perform better than humans ( §5), but are impractical in the real world ( §7). in §6 we explore an alternative approach: finetuning another llm as a proxy for the real cfm and relying on its token likelihood scores as proxies for the scores of the real cfm. we find that this works with even a small amount of fine-tuning data (only 3% of the full fine-tuning dataset) but only given that we know which base llm was used as cfm. we also experiment with using these proxy scores to identify the base llm, but this method would also be impractical if there are dozens, if not hundreds possible alternatives. we hope that our findings will boost similar investigations for other languages, and highlight the urgency of developing model-agnostic methods for synthetic text detection. to facilitate future research on italian, we release (i) 15k news passages generated by models fine-tuned on the change-it dataset, an italian news dataset @xcite , (ii) ratings produced by 5 human annotators on 400 texts, with a balanced distribution of 50% humanwritten and 50% synthetic passages, and (iii) 600k synthetic alterations of both the original samples from the change-it dataset and the synthetic texts. the code and data are publicly available. 3 we will not publicly release our fine-tuned llms (since they are best suited to be used as cfms), but we welcome direct requests from researchers working on this problem. ## related work driven by the increasing number of strong openly available llms @xcite @xcite , several studies focused on the detection of synthetic text detection. @xcite identified two main groups of approaches: those based on token likelihood, and supervised classification. among the former, @xcite ; @xcite ; @xcite ; @xcite proposed detection methods that rely on language models token distribution. for the supervised detection, @xcite proposed an approach that relies on the availability of labelled datasets of synthetic and human-written texts. @xcite propose a 6-way split of synthetic text detection methodologies: (i) watermarking, (ii) perplexity estimation, (iii) burstiness estimation, (iv) negative log-likelihood curvature, (v) stylometric variation, and (vi) classifier-based approaches. after the release of chatgpt (openai, 2022) and @xcite , a lot of studies (mostly not yet peer reviewed) focused specifically on the detection of text generated by these models @xcite . since it is not possible to access token probabilities for a candidate generated text, some of this work relies on a proxy model @xcite , but most rely on supervised classification @xcite @xcite , including openai itself @xcite . sometimes classical machine learning techniques are reported to perform well: e.g. @xcite report a high performance on chemistry articles with xgboost classifier and 20 features extracted from paragraphs. recently, several benchmarks for the synthetic text detection task have been released, @xcite contribute a large multilingual, multidomain dataset that can be used to benchmark synthetic text detection systems. @xcite propose a benchmark for the detection of text generated by multilingual llms, while @xcite propose a very large benchmark for synthetic text detection, controlling for the temperature used to generated the text, which has been shown to be relevant for detection @xcite . the above works focus on the scenario where the effort of detecting synthetic texts is on the user side. the complementary direction on the developer side is watermarking: ensuring that the llm output creates some kind of statistical "signature" that would help to identify it. there are multiple proposals for how to do this @xcite @xcite @xcite @xcite , and some initial results suggesting that such techniques could be sufficiently robust to human and machine paraphrasing @xcite . however, most of the current 'open' llms are freely available without any watermarking, and as we will show, they are already sufficient to be used as cfms. for the problem of detecting synthetic text without watermarking, the current research focuses on either monolingual or multilingual-by-design llms, and most studies do not focus on a specific domain. we stress the importance of also investigating fine-tuned models because llms are expensive to both train and run inference on @xcite . hence, starting from a public, relatively small, but high-quality llm and fine-tuning it for a specific type of content is probably the most plausible scenario for 'content farms' that aim to produce texts cheaply. to the best of our knowledge, this is the first study to focus on the scenario where the 'cfm' is fine-tuned for news, and in a language that it was not meant for originally. moreover, except for the recent work by @xcite , existing resources do not cover italian. ## fine-tuning of llama as an italian 'content farm' model as our 'cfm', we choose the original llama model @xcite , in 7b and 65b parameter versions. since this is one of the first public highperforming llms of this size, our llama results serve as a lower bound for what could be expected from later llms, such as llama 2 @xcite and llama 3, 4 aya @xcite , and others. we do not suggest that it is possible to obtain good results with any language and any llm (see also §10): such a transfer depends on the similarity between languages and their coverage in the training data. but given that 'content farms' have already been identified in at least 16 languages (see §1), many others could follow. our choice of italian enables a lower-bound estimate of what could be expected of monolingual or multilingual llms with more exposure to the target language. llama is a 'mostly-english' model, not intended to be multilingual. the resources it was trained on, such as c4 corpus @xcite , made deliberate efforts to filter out non-english text. however, it was exposed to at least italian wikipedia @xcite . hence, wikipedia is likely the main, if not the only source of italian in llama. we experiment with the original llama baseline (65b pre-trained model with no extra training), and two versions of our llama fine-tuned on italian news, after 20k and 60k steps. the technical details for fine-tuning are provided in app. b.1. we remark that creating such a cfm now comes with very few technical or financial difficulties, 5 which 4 https://ai.meta.com/blog/meta-llama-3/ 5 renting gpus on cloud providers such as amazon is now relatively cheap (approx 15@xmath0 to replicate one of our llm training sessions and data generation. the technical barrier to fine-tuning llms is also low now, thanks to tools like huggingface's autotrain: https://huggingface.co/autotrain . we do not criticize open-sourcing such tools, but we hope that could increase the number of bad actors. ## detection of synthetic news in italian by native speakers methodology. to assess whether native italian speakers would be able to identify synthetic news texts generated by our cfm, we set up a crowdbased study following the general recommendations for human evaluation of automatically generated texts proposed by @xcite . specifically, we created 4 surveys, with 100 questions in each. to maintain the rater engagement, we administered the surveys in five sessions, each comprising 20 questions. the raters were anonymously recruited among italian native speakers via the prolific foot_3 online crowd-sourcing platform. five different raters participated in each survey session, with no limitations on the number of sessions a rater could undertake. the 20-question sessions took 8 mins 23 secs on average, and the raters were compensated at 9,68$ per hour. 7 the study involved a total of 93 different raters, with an average age of 32.01 (±10.76). each of the 4 surveys is designed to assess the texts generated by a different model: llama 7b and 65b, both with and without fine-tuning on italian news. each question required raters to read two texts, denoted as a and b, and answer the question "text b follows text a, do you think text b is written by a machine?". the answer was a rating on a 5-point likert scale, where 1 indicates 'certainly human-written' and 5 'certainly machinegenerated'. in 50% of the questions, both text a and b were human-written news articles coming from the original change-it test dataset. the topics included daily national political events (e.g. politicians' declarations), general news (e.g. climate catastrophes), and relevant international news (e.g. european leadership meetings). to estimate the accuracy of human raters on this task, we map the scores on the 5-point scale to a binary score by computing the average score assigned by the raters to a given sample. we interpret the average score above 3 as indicating that a given sample b was generally perceived as machinegenerated. we also experimented with a different our results would highlight the necessity of more research on synthetic text detection. ## conclusion in this work we have shown how creating a 'content farm' generative model for news-like text can be easy, even though we started with a relatively old llm and a language it was not originally meant for. after fine-tuning llama 65b on only 40k italian news texts, native speakers of italian have only ≈ 64% accuracy on the synthetic news text detection task. we show that the current approaches to automatic text detection, based on token likelihood and supervised classification, outperform human raters in the synthetic text detection, but they would all be impractical in the real world, since they require access to the token likelihood information or a large training dataset. we further consider the proxy model approach, and we find that it works well even with little data for fine-tuning, but only if it is known which base llm was used. our study highlights the urgency of further work on developing model-agnostic methods of synthetic text detection.