Investigating Wit, Creativity, and Detectability of Large Language Models in Domain-Specific Writing Style Adaptation of R eddit’s Showerthoughts (2024)

Abstract: Recent Large Language Models (LLMs) have shown the ability to generate content that is difficult or impossible to distinguish from human writing. We investigate the ability of differently-sized LLMs to replicate human writing style in short, creative texts in the domain of Showerthoughts, thoughts that may occur during mundane activities. We compare GPT-2 and GPT-Neo fine-tuned on Reddit data as well as GPT-3.5 invoked in a zero-shot manner, against human-authored texts. We measure human preference on the texts across the specific dimensions that account for the quality of creative, witty texts. Additionally, we compare the ability of humans versus fine-tuned RoBERTa-based classifiers to detect AI-generated texts. We conclude that human evaluators rate the generated texts slightly worse on average regarding their creative quality, but they are unable to reliably distinguish between human-written and AI-generated texts. We further provide the dataset for creative, witty text generation based on Reddit Showerthoughts posts.

## introduction as large language models (llms) continue to advance, it becomes increasingly challenging for humans to distinguish ai-generated and humanwritten text. generated text may appear surprisingly convincing, inciting debates whether new forms of evaluating models are necessary @xcite . the high quality of llm outputs can benefit diverse use cases, while also increasing the risk of enabling more sophisticated spam, misinformation, and hate speech bots @xcite . llms are known to master various aspects of grammar and basic semantics. yet, one goal that still has proven non-trivial using llms is that of generating creative text @xcite , especially in the realm of humour @xcite . ## data compilation to create the showerthoughts dataset, we used the publicly available pushshift api @xcite to extract submissions from the showerthoughts subreddit from april 2020 to november 2022, resulting in an initial collection of 1.3 million posts. 4 we discard posts that have been deleted or removed (often due to rule violation) as well as those that contain images or additional explanations in their body text (as the community's rules require the full showerthought to be contained in the title). accordingly, we only use each post's title for our experiments, resulting in a dataset of 411,189 showerthoughts. an analysis of the most frequent choices of words reveals that they are often about people, life, common objects, and the world in general. a frequent word analysis indicates that they often compare things using, e.g., "more", "other", "old", "good". in order to obtain a ground truth about the lexical characteristics of the dataset and later compare them with the generated texts, we conducted several tests on 5,000 randomly selected examples, focusing on sentence complexity, length, grammar, and vocabulary, the results of which are summarized in ## experimental setup in the following, we detail our experimental setup for addressing our three research questions. we explain our process for generating reddit showerthoughts-like texts with differently sized selected llms. these texts are subsequently evaluated through a survey, assessing several textual aspects. additionally, we compare the ability of humans and fine-tuned bert-based classifiers in detecting originality. an overview of this experimental setup is given in figure 1 . 5 tests are conducted with the textstat, language_tool_python, and nltk libraries. ## results this section presents our experimental results. section 5.1 compares lexical characteristics, showing that the llms come close to human quality. next, section 5.2 explores the survey results, providing insights into crucial showerthought attributes such as logical validity and creativity. lastly, section 5.3 reports on our authorship identification, including patterns to distinguish between human-written and ai-generated showerthoughts. ## conclusion in this study, we demonstrate that relatively small, gpt-based llms can be fine-tuned to replicate the writing style of short texts of high creative quality, using the showerthoughts subreddit as an example. while it remains to be investigated to what extent the creativity stems from observations encountered in the pretraining corpus as opposed to novel creations, we find that large numbers of diverse texts can be produced with great ease. human raters confirm that the generated texts exhibit wit, creativity, and humour. this paves the way for diverse applications in productivity, creative work, and entertainment, and is relevant for practitioners deploying small llms to be cost-efficient. we find that human evaluators rate the generated texts on average slightly lower regarding creativity, humour, cleverness. this does not seem to aid in authorship detection ("i believe this showerthought has been written by a real person"), as we find that evaluators could not reliably distinguish ai-generated texts from human-written ones. additionally, the quality of human-written showerthoughts varies, with bad ones often being mislabeled as ai-generated. nonetheless, the possibility to abuse these models to produce spam, misinformation, or other harmful content is a growing concern. our robertabased authorship identification classifiers performs well after fine-tuning, revealing interesting hidden patterns that help in detecting the texts generated by specific llms. while ml classifiers can currently detect ai-generated texts (when fine-tuned for the task), we can assume that the text generation quality of llms will further improve, making this task more difficult. additionally, differently designed models may pursue other strategies for generating texts, necessitating their inclusion when training general-purpose classifiers. our work extends existing work that llms can learn to generate specific types of texts (when fine-tuned on high-quality data) to the domain of creative and witty texts, as exhibited by showerthoughts, but not limited to those. for example, practitioners who would like to utilize such a llm for marketing or copy-writing, could not only prompt it for general showerthoughts about a random topic, but also add the start of a text or topic to their prompt for the llm to complete. alternatively, generated texts can be clustered by topic to identify the right topics for a specific use case. simultaneously, we strongly recommend further research on detection mechanisms -while training detection models using generated texts of known llms and those fine-tuned on known datasets seems feasible, the task becomes more difficult when there is an exceedingly high number of llms to consider and even more so if the author-llm's architecture or the training dataset is not known.