HANSEN : Human and AI Spoken Text Benchmark for Authorship Analysis (2023)

Abstract: \textit{Authorship Analysis} , also known as stylometry, has been an essential aspect of Natural Language Processing (NLP) for a long time. Likewise, the recent advancement of Large Language Models (LLMs) has made authorship analysis increasingly crucial for distinguishing between human-written and AI-generated texts. However, these authorship analysis tasks have primarily been focused on \textit{written texts} , not considering \textit{spoken texts} . Thus, we introduce the largest benchmark for spoken texts - {\sf HANSEN} ( \underline{H} uman \underline{AN} d ai \underline{S} poken t \underline{E} xt be \underline{N} chmark). {\sf HANSEN} encompasses meticulous curation of existing speech datasets accompanied by transcripts, alongside the creation of novel AI-generated spoken text datasets. Together, it comprises 17 human datasets, and AI-generated spoken texts created using 3 prominent LLMs: ChatGPT, PaLM2, and Vicuna13B. To evaluate and demonstrate the utility of {\sf HANSEN} , we perform Authorship Attribution (AA) & Author Verification (AV) on human-spoken datasets and conducted Human vs. AI text detection using state-of-the-art (SOTA) models. While SOTA methods, such as, character n-gram or Transformer-based model, exhibit similar AA & AV performance in human-spoken datasets compared to written ones, there is much room for improvement in AI-generated spoken text detection. The {\sf HANSEN} benchmark is available at: https://huggingface.co/datasets/HANSEN-REPO/HANSEN

## introduction authorship analysis is a longstanding research area in nlp that has garnered significant attention over the years. two tasks are at the core of authorship analysis -authorship attribution (aa) and authorship verification (av). aa is the process of identifying the author of a text document @xcite . similarly, av determines whether two documents are written by the same author (halteren, prompt: create a post match interview with tennis player roger federer who . . . <rest of the prompt>. the reporter will ask exactly the following questions: q:. . . . . . . . . . . . . . . . . . . . . . . . . . . q: what do you think the williams sisters' rivalry with one another has meant to the sport? human: i don't know. i'd say the players themselves have been great for the sport. i don't know if the matches themselves is what people remember that much. chatgpt: the williams sisters have been amazing for tennis, and their rivalry has been a huge part of that. they've both achieved so much in their careers, and they've pushed each other to be better. palm2: the williams sisters' rivalry with one another has been one of the most compelling and enduring in tennis history. the rivalry has meant a great deal to the sport of tennis. vicuna13b: the williams sisters' rivalry has been incredible for the sport. it's been amazing to see the two of them compete against each other and push each other to be their best. 2007). both have been extensively researched as text classification problems due to their substantial impacts on various applications, such as author profiling, author forensic analysis, resolving copyright disputes, and similar issues @xcite . currently, most of the research in nlp text classification focuses on written text due to vast amounts of text data available for training and evaluation @xcite . however, "text" is also inherent in spoken language as a textual representation of what individuals say, commonly known as spoken text @xcite . although spoken language has always existed before written language (i.e., considering human history of language as a twelve-inch ruler, written language has only existed for the "last quarter of an inch" @xcite ), it has not received much attention from the nlp community. simultaneously, recent advancements in speech-to-text technology have ## authorship analysis on hansen we conduct three authorship analysis tasks using hansen and evaluate how existing sota methods perform in the context of "spoken" texts. details on aa, av, & tt methods are provided in tables 13 , 14 , 15 in the appendix section. we run the experiments five times and report the average. ## discussion character n-gram dominates in aa but struggles in av: while character n-gram performs best in aa for most datasets, it underperforms in the av task. more intricate dl models, such as adhominem, excel in av, consistent with findings in written text datasets @xcite . larger dl models tend to outperform smaller models when the datasets have more words per class, leading to better av performance since it only has two classes @xcite . character n-grams outperform word n-grams by a notable margin (5%-10% in general), emphasizing the potential enhancement of aa performance through the inclusion of informal words (e.g., "eh," "err," "uhh"). aa and av performance is dataset specific: our study highlights significant performance variations in aa & av across datasets, influenced by factors such as dataset type, domain, and modality. daily-life conversation datasets (base, bnc, pan) generally yield high f1 scores, except for msu, which comprises simulated conversations with predefined topics, potentially limiting speakers' natural speaking styles. conversely, talk-showtype datasets (usp, ftn) exhibit poor aa & av performance due to a skewed distribution of samples from show hosts and the influence of communication accommodation theory @xcite , suggesting speech adaptation to host styles. individuals written & spoken samples are vastly different: our results affirm the distinctions between individuals' written and spoken texts, consistent with prior corpus-based research @xcite . we also conduct an ablation study on sec and pan datasets, training on written texts only and testing on spoken texts (and vice versa). the results in ai-generated spoken and written texts have different characteristics: how close are llm-generated spoken texts to humans? the aa-10(l) and aa-20() experiments reveal a decrease in the f1 score compared to aa with all human speakers, indicating that llmgenerated spoken texts exhibit greater similarity to other speakers in the experiments. it highlights the potential for further research in training llms to replicate individual spoken styles accurately. which llm is the winner? identifying the best llm for spoken text generation remains an open question, influenced by multiple factors. palm2 demonstrates a lower gruen & div score and is more easily detectable than other llms in all datasets. on the contrary, chatgpt exhibits lower mauve scores, indicating its generation outside of human distribution, but its higher average recall value exhibits that it is difficult to detect. ## conclusion we present hansen, a benchmark of human and ai-generated spoken text datasets for authorship analysis. our preliminary analysis suggests that existing sota aa & av methods behave similarly for spoken texts in most scenarios. however, the stylistics difference in spoken & written for the same individuals and ai-generated spoken texts show different characteristics than existing notions, emphasizing the need for a more nuanced understanding of the spoken text.