Whodunit? Learning to Contrast for Authorship Attribution (2022)

Abstract: Authorship attribution is the task of identifying the author of a given text. The key is finding representations that can differentiate between authors. Existing approaches typically use manually designed features that capture a dataset’s content and style, but these approaches are dataset-dependent and yield inconsistent performance across corpora. In this work, we propose to learn author-specific representations by fine-tuning pre-trained generic language representations with a contrastive objective (Contra-X). We show that Contra-X learns representations that form highly separable clusters for different authors. It advances the state-of-the-art on multiple human and machine authorship attribution benchmarks, enabling improvements of up to 6.8% over cross-entropy fine-tuning. However, we find that Contra-X improves overall accuracy at the cost of sacrificing performance for some authors. Resolving this tension will be an important direction for future work. To the best of our knowledge, we are the first to integrate contrastive learning with pre-trained language model fine-tuning for authorship attribution.

## introduction authorship attribution (aa) is the task of identifying the author of a given text. aa systems are commonly used to identify the authors of anonymous email threats @xcite and historical texts @xcite , and to detect plagiarism @xcite . with the rise of neural text generators that are able to create highly believable fake news @xcite , aa systems are also increasingly employed in machine-generated-text detection @xcite . when performed on texts generated by human and machine writers, aa can also act as a type of turing test for natural language generation @xcite . contrastive learning is a technique that pulls similar samples close and pushes dissimilar samples apart in the representation space @xcite . it has proven useful in tasks that require distinguishing subtle differences @xcite . this makes it highly suited to encouraging the learning of distinct author subspaces. however, no prior work has investigated its relevance to the aa task. to this end, we seek to under-stand its impact on the learning of author-specific features under the supervised learning paradigm. to achieve this, we integrate contrastive learning with cross-entropy fine-tuning (contra-x) and demonstrate its efficacy via evaluation on multiple aa datasets. unlike previous aa work, we evaluate the proposed approach not only on human writing corpora but also on machine-generated texts. there are three major reasons. first, this can show that our approach is generic to writer identity and dataset composition. second, performing aa on human and machine authors reflects the increased importance of identifying machine-generated text sources. third, this potentially reveals information about how differently machines write compared to humans. in addition, we study the performance of our method under different data regimes. we find contra-x to consistently improve model performance and yield distinct author subspaces. finally, we analyze the performance gains vis-à-vis a number of aa-specific stylometric features. to the best of our knowledge, we are the first to incorporate contrastive learning into large language model fine-tuning for authorship attribution. we note that feature-based approaches are investigated in two streams of work. one stream benchmarks on public datasets such as imdb62 @xcite and blog @xcite . the various features proposed include term frequency-inverse document frequency (tf-idf) @xcite , letter and digit frequency @xcite , and character n-grams @xcite . the other stream is the pan shared task of authorship identification. these methods typically use multiple features such as n-grams @xcite @xcite @xcite in an ensemble. the two streams share similar technical ideas and developments. however, feature-based approaches require dataset-specific engineering @xcite and their performance does not scale with more data in contrast, learning-based approaches learn representations completely from data. bertaa @xcite shows that a simple fine-tuning of pretrained language models can outperform classical approaches by a clear margin. however, purely cross-entropy fine-tuning may not directly address the challenge of learning distinctive representations for different authors. thus, we propose to incorporate contrastive learning, which explicitly enforces distance constraints in the representation space. contrastive learning. contrastive learning aims to learn discriminative features by pulling semantically similar samples close and pushing dissimilar samples apart. this encourages the learning of highly separable features that can be easily operated on by a downstream classifier. unsupervised contrastive learning has been used to improve the robustness and transferability of speech recognition @xcite and to learn semantically meaningful sentence embeddings @xcite . it has also been combined with supervised learning for intent detection @xcite , punctuation restoration @xcite , machine translation @xcite , and dialogue summarization @xcite . however, to the best of our knowledge, we are the first to study its efficacy and limitations on authorship attribution. ## human authorship attribution we first investigate the impact of contrastive learning on models for human authorship attribution. ## synthetic text authorship attribution we investigate our proposed models on aa datasets with machine-generated text. this is to show how our method performs consistently across different dataset qualities and writers. performing aa on human and machine authors together also reflects the increased importance of identifying machinegenerated text sources. ## discussion in this section, we study the following questions: to do this, we construct 4 subsets of the blog10, blog50, and turingbench datasets with stratified 2 results of previous methods are from turingbench @xcite . for consistency, we report results to 2 decimal places. for full results for other metrics, i.e., precision, recall, and f1-score, see appendix f. ## conclusion successful authorship attribution necessitates the modeling of author-specific characteristics and idiosyncrasies. in this work, we made the first attempt to integrate contrastive learning with pretrained language model fine-tuning on the authorship attribution task. we jointly optimized the contrastive objective and the cross-entropy loss, demonstrating improvements in performance on both human-written and machine-generated texts. we also showed our method is robust to dataset sizes and consistently improves upon cross-entropy fine-tuning under different data regimes. critically, we contributed analyses of how and when contra-x works in the context of the aa task. at the dataset level, we showed qualitatively that contra-x creates a tighter representation spread of each author and increased separation between authors. within each dataset, at the author level, we found that contra-x is able to distinguish between highly similar author pairs at the cost of hurting its performance on other authors. this points to a potential direction for future work, as resolving it would lead to better overall improvement and increased fairness of the final representation.