Analyzing Large Language Models’ pastiche ability: a case study on a 20th century R omanian author (2025)

Abstract: This study evaluated the ability of several Large Language Models (LLMs) to pastiche the literary style of the Romanian 20th century author Mateiu Caragiale, by continuing one of his novels left unfinished upon his death. We assembled a database of novels consisting of six texts by Mateiu Caragiale, including his unfinished one, six texts by Radu Albala, including a continuation of Mateiu’s novel, and six LLM generated novels that try to pastiche it. We compared the LLM generated texts with the continuation by Radu Albala, using various methods. We automatically evaluated the pastiches by standard metrics such as ROUGE, BLEU, and METEOR. We performed stylometric analysis, clustering, and authorship attribution, and a manual analysis. Both computational and manual analysis of the pastiches indicated that LLMs are able to produce fairly qualitative pastiches, without matching the professional writer performance. The study also showed that ML techniques outperformed the more recent DL ones in both clusterization and authorship attribution tasks, probably because the dataset consists of only a few literary archaic texts in Romanian. In addition, linguistically informed features were shown to be competitive compared to automatically extracted features.

## introduction the llms' capacity to imitate art is ever increasing in all creative domains. in literature, their ability to mimic the style of an author, of a character, of a literary genre, or of an epoch constitutes a vibrant research area with intriguing topics such as role-play @xcite , storytelling @xcite , creative writing @xcite . since machine generation of literary pastiches of human authors raises ethical concerns due to the possibility of llm-generated texts to pass as the work of human writers, @xcite , research on the llms' ability to imitate a given author's style is much needed. the term pastiche has a long history. it originates from the italian pasticcio, meaning a mixture of meat and pasta turned into a pie. this food analogy suggests that the pastiche involves mixing available (recognizable) elements into a new thing, but without a new substance @xcite . until the 20th century, the term had a negative connotation of a lack of creativity. later, in theories of postmodernist literature, the term acquires its current meaning of an homage of past styles in the form of a deliberate imitation or blending of prior works of art, such as painting, architecture, design, sculpture, movie, music, poetry, or literature @xcite . it consists of acknowledged borrowings of style, words, phrases, or motifs of previous authors, genres, or periods. the intention of pastiche is not mockery or forgery, but rather an open reference to the original @xcite , most often paying it a tribute. some examples of literary pastiches are: extending a series when an author has died (like the sherlock holmes series, produced long after sir arthur conan doyle's death) or allowing fans to play with the narrative as in the case of fan fiction (like e l james' "fifty shades of grey", the fanfic inspired by stephenie meyer's "twilight"). in this paper, we investigate the llm's capacity to pastiche an author style. to do so, we propose a case study on an intriguing literary pastiche case from romanian 20th century literature. this choice was motivated by the existence of a pastiche novel authored by a professional writer who tried to imitate the style of another author, which naturally constitutes a golden standard for comparison of the machine-generated pastiches. the original novel was written by mateiu caragiale , a romanian symbolist and decadent writer recognized for his role in modernizing the romanian literary language, through his unique voice, stylistic innovation, lexical baroque richness, elaborate syntax, poetic language, and focus on mood over plot. in his last seven years of life, he authored the novel sub pecetea tainei (under the seal of secrecy) without finishing it. some decades later, in the 1970s, the rumor that the continuation of the novel was found spread in romanian literary circles, passing for a short time as a possibly genuine ending of mateiu's last novel, due to its very similar writing style. the debate was settled by radu albala, the actual author of the continuation entitled în deal, pe militari (on the militari hill). he revealed that his goal was precisely to continue the original novel in a style so similar to the original writer, as to pass as mateiu's text for human experts. radu albala was one of the closest stylistic followers of mateiu caragiale, among others like eugen bȃlan and alexandru george, who also wrote continuations of the unfinished novel of mateiu, as a stylistic exercise @xcite . the rest of the paper is structured as follows: the next section presents related work; the next one describes the data in detail. the analysis section is divided in two subsections: one for the computational analysis, comprising evaluation metrics between the original and the pastiches, stylometric analysis, and automatic methods such as pastiche clustering and authorship predictions, and the other focusing on human interpretation. we summarize the findings of our study in the conclusions section. ## related work a thorough survey of stylometry or authorial style, comprising techniques, tools, and algorithms can be found in @xcite . the methodology of stylometry centered on authorship or style debates of old texts is well established and used in numerous recent research, like @xcite , who performed stylometric analysis based on pos and n-grams on amadís de gaula and its sequel sergas de esplandián medieval spanish chivalric romances, or @xcite who focuses on authorship attribution with pos and n-grams stylistic features on 15th century tirant lo blanc, or @xcite , who analyses the (word embeddings) semantic similarity and intertextuality of the vedic sanskrit corpus. in the field of the more recent llm-generated texts, a comprehensive literature review of authorship attribution @xcite categorizes four representative problems: human-written text attribution, llm-generated text detection, llm-generated text attribution, and human-llm coauthored text attribution. llms can be prompted to generate any kind of creative text, in any manner. for instance, @xcite prompted chatgpt to forge a novel and not the author's style. another example is the prolific domain of creative writing. to give a very recent instance, @xcite evaluated the creative writing abilities of three llms and ten humans, instructing them to create a story based on a prompt that included the summary of a novel. the results showed that the llms performed worse than humans. also, they used llms to asses the quality of the generated tests, but their evaluations correlated poorly with human judgment. @xcite and @xcite compared llms and humans writing style on news articles, finding that there are relevant distinctive features between the two. durward and thomson (2024) investigate vocabulary usage for ai and human-generated text in news articles and creative writing, noting thematic differences between them. @xcite identified systematic differences between llms and humans on different register texts. chen and moscholios (2024) explored llms capacities of imitating a person's language style. @xcite proposed the task of emulating human style with llms on blog posts. previous work on romanian 20th century writers mateiu caragiale and radu albala @xcite focused on authorship identification for albala's pastiche of mateiu's unfinished novel, using stop words rankings. another similar research @xcite measured the style similarities between mateiu's writing and the writing of his followers, who tried to mimic or pastiche him (albala, agopian, bȃlan, and iovan), finding that they are closer in style to each other than to mateiu. ## data we obtained the six original novels by mateiu caragiale, published as volume chapter of the book "craii de curtea veche", from wikisource. for radu albala, we obtained the six novels from a publishing house, for research purposes. the pastiches generated by the llms were obtained by few-shot prompting, providing them with the last unfinished novel written by mateiu caragiale, sub pecetea tainei. we used the following prompt to ask the llms to generate a pastiche that continues it: you are mateiu caragiale, a ro-manian writer, son of i.l. caragiale. continue the plot with 18000 characters from the short story sub pecetea tainei! here is an example of how mateiu wrote: "...". the choice of the generated text length is motivated by the intention to match the length of 18528 characters of albala's pastiche in deal pe militari that continued mateiu's sub pecetea tainei, so as to directly compare the llms generated texts with the professional writer's pastiche. we used six publicly available llms for this pastiche generation task: chatgpt4o foot_0 , claude haiku foot_1 , gemini 1.5 pro foot_2 , qwen 2.5 72b instruct foot_3 , wizzard lm2 8x22b, and llama 3.1 70b turbo (both accessed via deepinfra chat platform foot_4 ). for gemini, we deactivated all safety settings, as this feature was available and since negative sentiments have been shown to correlate with artistic creativity @xcite . we did not change any other parameters of the models, like top-p or temperature, as we focused on their default generative capacities. we manually inspected the texts and cleaned them accordingly. we removed any special characters. we standardized the dialogue marker, since in some texts a small dash was used and some of the llms used the english standard quotation marks, replacing them all by the standard romanian emdash. we also cleaned any page number, footnote mention, or others. the data set is well balanced, in terms of the number of examples per author and of the text length. we give the name of all the novels by human authors and the data statistics in ## conclusions in general, llms generated fairly good pastiches, although without matching the quality of the human written pastiche. this is supported by all scores and methods used: similarity scores, stylometry, language style matching scores, clusterization, prediction, and manual inspection. overall, traditional ml methods outperformed more recent dl ones. this happened because our data consisted in a few literary archaic text in romanian, this kind of dataset being typical of dh. nevertheless, a study focused on contemporary english could show bet-