Detecting, Generating, and Evaluating in the Writing Style of Different Authors (2025)

Abstract: In recent years, stylometry has been investigated in many different fields. Hence, in this work, we are going to tackle this problem, detecting, generating, and evaluating textual documents according to the writing style by leveraging state-of-the-art models. In the first step, the sentences will be extracted from several different books, each belonging to a different author, to create a dataset. Then the selected models will be trained to detect the author of sentences in the dataset. After that, generator models are utilized to generate sentences based on the authors’ writing styles with unpaired samples in the dataset. Finally, to evaluate the performance of the generators, the previously trained models will be used to assess the generated sentences and to compare the distribution of various syntactic features between the original and generated sentences. We hope the result shows that models can be achieved to detect and generate textual documents for the given authors according to their writing style.

## introduction stylometry is a linguistic discipline that applies statistical analysis to literature based on the assumption that each author has an unconscious aspect to their style @xcite . generally and simply, stylometry is a field of study that statistically analyzes authorship attribution @xcite . as the production of digital documents increases, the importance of stylometry grows as well. the increasing attention to stylometry has been reflected in @xcite : "as non-handwritten communications become more prevalent, such as blogging, text messaging, and emails, there is a growing need to identify writers not by their written script, but by analysis of the typed content". in this work, after demonstrating the existence of distinguishable patterns between different authors' writing styles, we aim to train generator models without paired samples to generate and then evaluate the generated sentences in different writing styles. leveraging these models opens new advances in generating stylistic text, further enriching applications such as authorship verification, creative writing, forensic linguistics, legal systems, and criminology. it is important to note that one of the key issues in this work lies in evaluating the generated sentences. first, while well-known metrics like accuracy or f1 score are valuable, they cannot adequately reflect how accurately the model detects and mimics writing styles. moreover, these metrics do not provide clear insights into the performance of the model in each of the writing style categories. on the other hand, relying on expert human evaluations presents significant challenges. for example, gathering experts who specialize in all five authors' writing styles in the dataset is nearly impossible. hence, we are going to use an ai-evaluate-ai technique to assess the generated sentences. we will train a detector capable of classifying sentences with high performance and use it to evaluate the generated sentences. furthermore, we will incorporate feature-based evaluation by comparing the features extracted from both the original sentences and the generated sentences to measure their alignment. given the importance of stylometry, and the challenges mentioned above, this study has been focused on answering three main research questions in this area: rq1 (detection): given the differences in authors' writing styles, can the proposed model extract related features and accurately detect the authors for a given sentence? rq2 (generation): is it possible to train a generator to produce sentences in the writing style of a specific author without using paired training data? rq3 (evaluation): can detector models be used to evaluate generated sentences by generator models? in @xcite , the authors propose a novel generative model for sentence style transfer that modifies the style of a given sentence based on categorical attributes. the architecture comprises an rnn-based encoder-decoder that generates sentences consistent with the input's content and specified attributes. also, in @xcite , the researchers propose a text style transfer model using an attentional auto-encoder and a binary style classifier, ensuring content preservation by minimizing the distance between the pos-tagged structure of input and output sentences. the approach focuses on maintaining noun consistency, incorporating a language model for fluency and a style classifier to guide the generator in producing sentences with the desired style. for the text style transferring task, other researchers in @xcite propose a gan-based framework for non-parallel text style transfer that integrates a seq-to-seq encoder-decoder with attention, word-level conditional mechanisms, and dual discriminators (cnn and rnn) to balance content preservation and style transformation. authors in @xcite propose a deep generative model that enhances variational autoencoders (vaes) with structured latent variables and holistic discriminators to generate text with specified attributes while ensuring disentanglement. their approach, which incorporates a wake-sleep algorithm for collaborative optimization, effectively learns interpretable latent representations from minimal supervision, enabling controlled text generation with potential applications in nlp and content creation. in @xcite , researchers introduced schema-guided natural language generation (sg-nlg), a task that generates natural language prompts based on rich schemata, repurposing a dataset from dialog state tracking to train seq2seq, cvae, and gpt-2 models. their findings show that leveraging schema information enhances semantic quality and diversity, with seq2seq and cvae excelling in reference similarity and gpt-2 performing best in diversity and human evaluation. the stable style transformer (sst) presented in lee (2020), introduces a model-agnostic text style transfer approach using the delete and generate framework, where a pre-trained classifier extracts attribute markers without relying on a dictionary or attention scores, and a transformer-based encoderdecoder generates the transferred sentence while preserving content. this method, trained on nonparallel datasets, demonstrates robust performance in handling long dependencies and offers a stable, effective solution for text style transfer. cterm-gan @xcite addresses the common limitation of nlg models that focus solely on syntax by incorporating both syntactic and semantic aspects through a relational memorybased generator and dual discriminators. experimental results show that it maintains or improves syntactic accuracy while significantly enhancing semantic coherence, demonstrating its potential for generating text conditioned on various inputs, including writing styles. authors in @xcite , developed diffusion-lm, a non-autoregressive language model leveraging continuous diffusion processes for controllable text generation, where gradient-based manipulation of latent variables during denoising enables finegrained style control, outperforming prior plugand-play models and achieving competitive results against fine-tuned autoregressive baselines. the paper by @xcite explores the application of diffusion models for fine-grained text style transfer, demonstrating that their approach, trained solely on the styleptb dataset without external resources, achieves state-of-the-art performance across 13 tasks, including compositional style transfers. their results highlight the potential of diffusion-based models for controllable text generation in low-resource settings, while also suggesting future directions such as integrating pre-trained embeddings and exploring alternative architectures. and evaluation. the outline of the procedure has been shown in figure 1 , which illustrates each phase and the data flow using different colors: blue for phase 1 (detection), green for phase 2 (generation), and purple for phase 3 (evaluation). also, this shows that three different subsets of the dataset were extracted and flowed in different paths, one for the detection phase and two for the text generator model. in order to create the dataset, we used project gutenberg to collect books by different authors. we picked five authors, charles dickens, mark twain, herman melville, jane austen, and louisa may alcott. they all belong to the 18th century and provide a good balance of male and female authors as well as british and american authors in our analysis. we also aim to cover a broad range of topics, as twain and melville generally wrote for men, whereas austen and alcott typically wrote about women. dickens, meanwhile, mostly addressed social conditions, such as poverty and wealth, rather than focusing specifically on men or women. the total number of extracted sentences is 115,471. in datasets with paired samples, there exist at least two different styles for the same content, like formal and informal datasets. as mentioned before, in our dataset, there are no identified sentences from different authors expressing the same content, which makes it much more challenging for the model to understand the differences or, in the next steps, to transfer one sentence from an author to another author's writing style. sentences from the dataset along with their labels are used to train the bert classification model. here, bert functions as a classifier to determine the author of the given sentences. the expectation is that a highly accurate classifier not only demonstrates that there are distinguishable features among authors, making the generation phase possible but also will provide a reliable model for evaluating newly generated sentences, verifying whether they align with the writing style of the intended author. on the other hand, as illustrated in the detection phase, there is also a syntactic feature extraction path. this path aims to perform a similar function as bert but relies on syntactic features. we expect these syntactic features, comprising both low-level and high-level features, to clearly demonstrate differences between various writing styles. the generator model will be trained on sentences concatenated with their labels at the beginning. the main idea behind this approach is that, since we don't have paired samples, we explicitly add the label of each sentence to help the model understand patterns shared by sentences with similar labels. after training the model, new sentences are generated by providing seeds with different labels and randomly extracted words. the generator then completes these sentences based on the initial labels. finally, the generated sentences must be preprocessed to evaluate their quality and remove tags. as mentioned in the introduction, evaluating generated sentences with common techniques has several challenges. hence, to make the evaluation more systematic and practical, we will use the aievaluate-ai technique. in addition to using a large language model like bert for evaluation, we employ a feature-based evaluation to further assess the quality of the generated text. as demonstrated in the next section, we will show that extracting syntactic features can help highlight stylistic differences between authors. for example, in prior studies @xcite , we showed how the syntax in speeches by u.s. presidents differed from the syntax in their written works. similarly, we will extract high-level and low-level syntactic features using stanford corenlp @xcite to compare the generated sentences with the original dataset. in summary, our proposed workflow combines detection, generation, and evaluation techniques to accurately create sentences in different writing styles and assess them in a meaningful way. the goal is to ensure that the generated sentences not only reflect the stylistic features of the target authors but also maintain the consistency and fluency of any generated sentences.