The Pure Poet: How Good is the Subjective Credibility and Stylistic Quality of Literary Short Texts Written with an Artificial Intelligence Tool as Compared to Texts Written by Human Authors? (2022)

Abstract: The application of artificial intelligence (AI) for text generation in creative domains raises questions regarding the credibility of AI-generated content. In two studies, we explored if readers can differentiate between AI-based and human-written texts (generated based on the first line of texts and poems of classic authors) and how the stylistic qualities of these texts are rated. Participants read 9 AI-based continuations and either 9 human-written continuations (Study 1, N=120) or 9 original continuations (Study 2, N=302). Participants’ task was to decide whether a continuation was written with an AI-tool or not, to indicate their confidence in each decision, and to assess the stylistic text quality. Results showed that participants generally had low accuracy for differentiating between text types but were overconfident in their decisions. Regarding the assessment of stylistic quality, AI-continuations were perceived as less well-written, inspiring, fascinating, interesting, and aesthetic than both human-written and original continuations.

## introduction artificial intelligence (ai) is increasingly used to provide support in creative domains such as the * this paper was published previously by proceedings of cogsci 2022 (44st annual meeting of the cognitive science society) composition of emotional film trailers @xcite or the ideation in fashion design @xcite . as part of this trend, advanced tools for human-ai co-creative processes have been developed in recent years. for instance, in a visual arts context, an empathic ai-tool has been developed that provides help in portrait drawing by means of embodied conversational interaction @xcite . another example from the field of music composition is an ai-tool enabling computational melodic harmonization (chameleon) that has been developed by @xcite . when evaluating this tool with experienced and inexperienced music composers engaging in human-ai co-creative processes it turned out that this tool was particularly helpful for less experienced students to better express their ideas. in this paper we will focus on using ai-tools in an even more complex creative domain then music, namely the production of literary texts such as short stories or poems. this domain can be seen as providing harder challenges than music composition or drawing due to the complexity of its underlying semantic structure and the embodied grounding of the symbols used to express it (cf. @xcite @xcite @xcite . the pure poet: how good is the subjective credibility and stylistic quality of literary short texts written with an artificial intelligence tool as compared to texts written by human authors? * vivian emily gunser 1 , steffen gottschling 1 , birgit brucker 1 , sandra richter 2 , dîlan canan çakir 2 , and peter gerjets 1 1 leibniz-institut fuer wissensmedien, schleichstr. 6, 72076 tuebingen, germany {v.gunser, s.gottschling, b.brucker, p.gerjets}@iwm-tuebingen.de 2 deutsches literaturarchiv marbach, schillerhoehe 8-10, 71672 marbach am neckar, germany {sandra.richter, dilan.cakir,}@dla-marbach.de