Fine-Tuning GPT -3 for Synthetic D anish News Generation (2023)

Abstract: While GPT-3 has garnered significant attention for its capabilities in natural language generation, research on its use outside of English is still relatively limited. We focus on how GPT-3 can be fine-tuned for generating synthetic news articles in a low-resource language, namely Danish. The model’s performance is evaluated on the dimensions of human and machine detection in two separate experiments. When presented with either a real or GPT-3 generated news article, human participants achieve a 58.1% classification accuracy. Contrarily, a fine-tuned BERT classifier obtains a 92.7% accuracy on the same task. This discrepancy likely pertains to the fine-tuned GPT-3 model oversampling high-likelihood tokens in its text generation. Although this is undetectable to the human eye, it leaves a statistical discrepancy for machine classifiers to detect. We address how decisions in the experimental design favoured the machine classifiers over the human evaluators, and whether the produced synthetic articles are applicable in a real-world context.

## introduction in recent years, rapid development in natural language processing, particularly in the area of pretrained language models, has led to significant advancements in various language tasks. state-of-theart models, such as gpt-3 @xcite and bert @xcite , have excelled in tasks such as classification of documents @xcite ), text completion @xcite , language translation @xcite ) and text summarization @xcite . these advances have even led some to suggest that we are currently experiencing a whole paradigm shift in nlp with the introduction of pretrained language models @xcite . however, most contemporary studies using gpt-3 focus on its performance in english. this is to be expected as the model was almost exclu-sively trained on english with less than 8% of training data being non-english (openai, 2020). still, the few investigations on gpt-3 for non-english text generation show promising results @xcite . this even holds for low-resource languages such as catalan @xcite . yet, the more prevalent approach in nlp for low-resource languages has been using smaller language-specific models or multilingual models such as mbert @xcite . this is despite multilingual models seemingly lacking in natural language generation tasks, especially for the nordic languages and other low-resource languages @xcite . in terms of language-specific models, this development has also occurred in danish nlp with several danish models appearing based on the likes of @xcite @xcite . nevertheless, such models are miniscule in size compared to the state-of-the-art language models. for instance, the danish bert model by møllerhøj (2021) is trained on 9.7 billion characters. comparatively, gpt-3's total training data corresponds to 1.1 trillion characters @xcite . in this paper, we seek to understand how well gpt-3 can perform for a low-resource language such as danish when optimized for that language through fine-tuning. to our knowledge, this is the first structured assessment of gpt-3's capabilities in a danish nlp task. concretely, we investigate whether gpt-3 can be fine-tuned to produce synthetic news articles that are indistinguishable to real news articles written by journalists. generating news articles with gpt-3 is a common task with previous work showing remarkable results in english @xcite . inspired by a similar study from @xcite , we make a two-fold evaluation of the model's performance: as human and machine detection methods presumably apply distinct techniques to spot synthetically generated text @xcite , a two dimensional evaluation provides a more nuanced insight into how gpt-3 performs on the task. our findings suggest that a fine-tuned gpt-3 can generate convincing danish synthetic news, deceiving human readers while being identifiable by a bert classifier. this demonstrates gpt-3's capacity to perform succesfully in the context of low-resource languages, but with the drawback of heightened machine-detectability due to an overuse of high-probability tokens. ## data the real news stories were all sourced from the danish news site tv2.dk. in october 2022, tv2's news platform boasted over 3 million unique users @xcite , which is more than half of denmark's population. hence, it makes an excellent representation of typical news content consumed by danes. these articles were obtained via two channels: directly scraping from tv2 and employing the danewsroom danish news database @xcite . in the selection process, only article bodies with a minimum length of 100 words were considered, and longer articles were shortened to a maximum of 150 words. although the exact threshold is somewhat arbitrary, it was kept in this range for two reasons. firstly, accumulating costs for generating articles with the fine-tuned gpt-3 necessitated that we kept the articles short. also, using longer articles would entail that each participant would evaluate fewer articles as their time was limited. in total, 1866 real danish news articles from tv2 were sourced and used for three purposes: fine-tuning gpt-3 (1209 real articles), providing training/validation data for machine classifiers (609 real articles), and serving as test data in the experiments (48 real articles). additionally, 657 synthetic articles were generated by the fine-tuned gpt-3 for training the classifiers (609 synthetic articles) and test data in the experiments (48 synthetic articles). ## experiment a: human detection experiment a is designed as a binary classification task where human participants must distinguish between real articles written by humans and gpt-3's synthetic ones. ## experiment b: machine detection experiment b explores whether it is possible to construct machine classifiers that are capable of distinguishing between real and synthetic articles. this is approached with logistic regression using bagof-words (bow) and tf-idf as baseline models. the more advanced language model, nb-bert-large, is then fine-tuned, tested and evaluated against the baselines and human participants. ## limitations a few limitations must be addressed in relation to these results. firstly, several design decisions presumably favoured the machine detectors over the human evaluators. whereas 78.3% of human participants had never seen gpt-3 produced texts before, all machine classifiers received extensive training on over 1000 labelled articles prior to the final testing. also, the zero temperature token sampling for generating synthetic articles created an overrepresentation of high-likelihood tokens. this may be identified by the machine detectors, whereas such patterns are are probably too subtle to notice for humans @xcite . also, @xcite show that higher temperatures are associated with gpt-3 making off-prompt errors. such errors would not be captured by the machine classifiers, whereas humans would more likely identify these more semantic shortcomings. moreover, it must be addressed that human classifications are possibly influenced from being conducted in an experimental setting. contrary to the machine classifiers, the human participants saw the headline and subheading for all articles. despite being repeatedly told not to evaluate them, it cannot be dismissed that these extra elements still could have influenced their decision-making process. for instance, a familiar headline could have evoked an intuition for the article being real before reading the article body. on the other hand, one could argue that this was beneficial for humans as they could improve assessments by comparing contents in the headline and subheading to the article body. still, these methodological decisions systematically favored the machine classifiers over the human evaluators. however, asserting that the machine superiority would evaporate based on these considerations is a reach considering how vast the performance gap was. another limitation relates to the generalizability of the synthetic news articles. due to experimen- greenland's government has decided not to apply for permission for further oil drilling in the coming year. this is announced by the greenlandic ministry of nature, environment and agriculture in a press release. "we have decided not to apply for oil drilling in 2023, because we want to spend time developing a new strategy for the greenlandic economy, which will form the basis for a new oil and gas strategy," it says. the government also emphasizes that it will maintain its "vision of a fossil-free greenland". the decision comes after a meeting on tuesday between the government's four parties. it is mainly the consideration for the climate that has led the government to drop further oil drilling. correct human bow tf-idf bert synthetic synthetic real real synthetic two photographers and a culture minister are now criticized by the press council for having participated in a photo series where they posed with weapons. the press council writes this in a press release. in the case against culture minister ane halsboe-jørgensen (s), the council has assessed that she has violated good press ethics by participating in the photo series 'the gun series'. "by participating in a photo series with weapons and ammunition, the culture minister has expressed that it is acceptable to carry weapons, whether it is in connection with artistic photography or not," the decision states. the decision against photographer rasmus flindt pedersen and jim lyngvild is more stringent. both have violated good press ethics by participating in the photo series, says the press council. article c correct human bow tf-idf bert real real real real synthetic consideration for endangered animal species such as hazel dormice, birch mice, and bats in denmark is now temporarily hindering a massive natural gas project that aims to supply poland with natural gas from norway. the environmental and food appeals board has annulled the project's environmental permit, thereby halting the construction work of the baltic pipe pipeline across denmark. "we are very disappointed with the decision," says marian kaagh, the deputy director of the company energinet, which is responsible for the construction work in denmark. in a press release, she says that energinet has been working on a number of initiatives to ensure good living conditions for the animals in the areas where the pipeline is being built. this was a requirement when the environmental protection agency granted the environmental permit for the baltic pipe project in 2019. however, according to the environmental and food appeals board, the conditions should have been thoroughly investigated before the permit was issued and construction work could begin. article d correct human bow tf-idf bert synthetic real real real real the upcoming super hospitals are meant to help improve the healthcare system in denmark. however, they will not be completed on time. on average, the 16 hospital constructions are almost two years delayed, according to a statement from the quality fund for hospital constructions, which tv 2 has obtained access to. it is an expression of "a number of challenges," as the fund's director, morten hjortenberg, puts it. "we had hoped for better results halfway through the construction period. it raises concerns and reflections on some of the decisions and priorities that were made during the projects," he says. the fund's task is to provide funding for hospital constructions and ensure high quality -that is, what is often called "quality funds." the total budget for the constructions is over 30 billion danish kroner -of which the state accounts for 23 billion and the regions' self-financing contribution is 9 billion. ## conclusion as advancements in natural language processing continue to progress rapidly, it is crucial to remember the importance of including and improving upon nlp in low-resource languages. this paper acknowledges this need by conducting a structured assessment of gpt-3's abilities for danish natural language generation when fine-tuned for the task. our study shows that gpt-3 can be fine-tuned to produce danish synthetic news articles that are virtually indistinguishable to real news articles for humans. however, this does not imply that the articles are actually indistinguishable as the human eye is not all-seeing. by constructing a fine-tuned bert model for the same discrimination task, we find that machine detection of the synthetic news articles was possible to a great extent. hence, there must have been underlying flaws in gpt-3's article generations, likely relating to an oversampling of high-likelihood words. the introduction of chatgpt and gpt-4 will likely impact the findings presented in this paper, lowering detection accuracies further for both humans and machines. although, as those models are closed-sourced, it would be troublesome to assess whether the testing articles are already part of the training data which poses a methodological challenge. regardless, as our findings for danish conform with similar studies in english, we encourage future work on low-resource languages to develop machine detectors which possibly stand the test when human evaluators are deceived.