Improving Long-Text Authorship Verification via Model Selection and Data Tuning (2023)

Abstract: Authorship verification is used to link texts written by the same author without needing a model per author, making it useful to deanonymizing users spreading text with malicious intent. In this work, we evaluated our Cross-Encoder system with four Transformers using differently tuned variants of fanfiction data and found that our BigBird pipeline outperformed Longformer, RoBERTa, and ELECTRA and performed competitively against the official top ranked system from the PAN evaluation. We also examined the effect of authors and fandoms not seen in training on model performance. Through this, we found fandom has the greatest influence on true trials, and that a balanced training dataset in terms of class and fandom performed the most consistently.

## introduction as more people turn to various online sources for their information, the ability to discriminate and discern authorship characteristics is critical to counter misinformation, plagiarism, and inappropriate aggregation. understanding authorship is also essential to detecting individuals who make use of the anonymity afforded by the internet to engage in harassment, impersonation, or criminal activities. conversely, such technologies could also be applied in unethical ways such as the deanonymization of whistle-blowers, for example. additionally, the identification of bots and information operation campaigns is essential in the areas of cyber and national security. authorship analysis includes multiple tasks that address different use cases. the goal of author identification/attribution is to identify if a document was written by one of a known set of authors and, if yes, specify the individual. author verification compares two documents to determine if they were written by the same author, without directly identifying or providing author information. advancements in authorship analysis have been furthered by efforts in the community. pan foot_0 is a yearly series of shared tasks on important text forensics topics, including authorship. the pan author verification task from 2020 @xcite and 2021 @xcite uses a fanfiction dataset. fanfiction has many interesting traits with respect to its use for automated authorship verification: the documents are long-text, authors can write stories in different fandoms (e.g., harry potter or star trek), and authors may emulate a certain style when writing within a specific fandom. traditional approaches to authorship recognition often focus on modeling lexical choice (i.e. word usage) or stylometry separately. modern, deep learning-based approaches are much more expressive. new developments in transformers and other large language models have been incredibly impactful in natural language processing and have been used to great success in tasks such as machine translation, text generation, and question and answering. open-source communities make using these models easy and accessible. for author recognition, these more expressive models have the potential to learn both lexical and stylometric information at the same time. one challenge in realizing this potential, however, is context length. authorship is a subtle task and the more information the model can integrate at one time, the more of this subtlety can be captured. further, diversity in training data can also play a large role in the quality and robustness of trained models. accordingly, in this paper, we evaluate transformer models for author verification on the longtext fanfiction data from pan and attempt to understand the influence of topic and data tuning on performance. our main contributions are the following: in the following sections, we first discuss work in the area of transformers for authorship attribution and verification; outline the creation/tuning and statistical breakdown of our datasets; present our cross-encoder approach for verification; and then describe and discuss our experiments and results regarding the relative performance of multiple transformer backbones, the fandom effect, and the performance of our bigbird cross-encoder on the official pan20/21 test sets and how it is influenced by the dataset tuning. ## related work previous approaches for author identification focused on traditional machine learning models with lexical information or stylometry, such as burrow's delta @xcite . deep learning approaches, like transformers, have shown promising results for author identification. fabien et al. (2020) developed a bert approach and incorporated stylistic and hybrid features into their model to improve performance. @xcite combined a multi-headed classifier (mhc) with pretrained language models to evaluate their system's performance for cross-topic and cross-domain author verification (e.g., essays versus emails). they showed both the elmo and bert versions of their system outperformed a recurrent neural network (rnn) baseline for cross-topic. further in barlas and stamatatos (2021), they introduced transfer learning and evaluated an additional cross-fandom author identification scenario. this is different than our work, where we are using cross-fandom as our cross-topic scenario. in these experiments, their elmo and ulmfit systems outperformed their rnn baseline but was not sota for the target dataset. although pre-trained models overall appear promising for authorship analysis tasks, some work has been done that highlights possible limitations of these approaches. in @xcite , the authors focused on the effect of topic and proposed a topic confusion task, where author and topic pairs are swapped between the train and test datasets. transformers' use in author verification has mixed results. @xcite evaluated several bert-like models for author verification using the pan20/21 data with good success. however, their experiments also indicated these models relied on topical information rather than authorship characteristics. as in our work, the authors investigated how data partitioning affected model performance. however, this work was limited to dataset tuning based on disjoint authors or fandoms. @xcite used longformer for the pan20 challenge but had very different results on their test splits and the official pan test set. their model performed worse than the baselines provided by pan. conversely, in @xcite , the authors used a bert-based model for pan21 (openset scenario) and had promising results when compared to other models trained on the small dataset. these works inspired us to explore how pretrained transformer models performed for author verification. pan20/21 is a great source of longtext data, so we compared general transformers with ones specialized for long-text. we also studied how fandoms and datasets affect performance. ## datasets pan offered data for closed and open-set author verification tasks. we used four training/validation sets and four test sets, with all training sets and two test sets derived from the pan training data. ## system approach we proposed a transformer-based cross-encoder model setup for authorship verification that allowed us to evaluate several transformer backbones and compared them to the baseline from pan. this baseline (called "naïve" in the pan official results and "cosine" in ours) is based on term frequency-inverse document frequency (tf-idf) cosine similarity computed over word tokens. ## experimental results and discussion our approach was to evaluate our cross-encoder model using multiple transformer backbones on datasets with different types of tuning, and then compare its performance to the pan baseline systems. we first optimized the hyper-parameters for each backbone, then examined the model's perfor-mance and effect of fandom. after identifying bigbird as the backbone with the best performance, we evaluated the cross-encoder model using the metrics from the pan challenge across all combinations of the multiple dataset variants. ## conclusion we compared several transformer backbones with our cross-encoder systems and found the choice in backbone dramatically impacted the feasibility of our cross-encoder model for long-text authorship verification. bigbird outperformed another long-text transformer (longformer) and two general transformers that use different pre-training approaches (distilroberta and electra). our experiments show that longformer and electra are both sensitive to the tuning and preparation of training data. our longformer results were consistent with @xcite ; this sensitivity to datasets makes longformer and electra nonideal candidates for this task. we found that fandom (which we considered equivalent to topic) is particularly important for tts. tts that are between fandom were significantly more difficult for our system to correctly predict than those that were within fandom. this fandom effect was seen to a lesser extent for fts but was eliminated in bigbird cross-encoder. this visible fandom effect indicates that there is still room for future work to improve the model's ability to learn features of the author and reduce reliance on fandom. our bigbird cross-encoder performed very competitively with the official pan20/21 scores and outperformed the top system for both the closed-and open-set verification tasks with only a naïve approach to leaving hard trials unanswered. these results show that bigbird may have great potential for author recognition work. the bigbird cross-encoder performed well on pan20/21 test sets without extra tuning, but different data tuning approaches affect system performance on test sets. for example, the minimally processed pan20 shuffled did not work the best for the pan20 curated test sets. matching the distribution of trials, authors, and fandoms between train and test data led to the best performance, but this approach is not necessarily feasible for real-world applications. we found that the pan20 equal training data, which was tuned for equal trial types, performed consistently across all the test sets. more research is needed to determine what aspects of this tuning actually affects performance, and if pan20 equal is also generalizable to other test sets or the approach to other types of data.