Topic-Regularized Authorship Representation Learning (2022)

Abstract: Authorship attribution is a task that aims to identify the author of a given piece of writing. We aim to develop a generalized solution that can handle a large number of texts from authors and topics unavailable in training data. Previous studies have proposed strategies to address only either unseen authors or unseen topics. Authorship representation learning has been shown to work in open-set environments with a large number of unseen authors but has not been explicitly designed for cross-topic environments at the same time. To handle a large number of unseen authors and topics, we propose Authorship Representation Regularization (ARR), a distillation framework that creates authorship representation with reduced reliance on topic-specific information. To assess the performance of our framework, we also propose a cross-topic-open-set evaluation method. Our proposed method has improved performances in the cross-topic-open set setup over baselines in 4 out of 6 cases.

## introduction authorship attribution is a task that aims to identify the authors of anonymous texts. applications of this task include academic and forensic ones, such as finding the authors of literary works, historical writings @xcite @xcite or threatening online messages @xcite @xcite . ## proposed method authorship representation learning has shown to be effective for large-scale open-set authorship attribution @xcite . however, these approaches have not been explicitly designed to help with generalization toward unseen topics. we hypothesize that we can improve generalization by reducing the topic information of an authorship representation. therefore, we propose a solution based on the concept of supervised contrastive learning @xcite and confidence regularization (utama et al., 2020). our framework can be applied to remove bias from any text encoder model regardless of the architecture. we propose authorship representation regularization (arr), a framework to obtain topicregularized authorship representation, i.e., an embedding that can be used to compare writing style similarity with minimum topic influence. step a: base model construction. first, we train a base encoder g on an authorship representation learning objective. we then freeze all the parameters of the base model. at target encoder training time, we sample a minibatch represented by a set of texts n . we calculate the probability score p g (i,j) from the cosine similarity score of each pair (i, j) ∈ n × n to use in the next step. we define n × @xmath0, j) : @xmath1. for each (i, j), we compute cosine similarity of encoded representation of text i and another text j from encoder g. we denote the l2 normalized representation of text i computed from g as g i and denote variable τ as the temperature scaling hyperparameter. we also use eq. 1 to derive probability score p h @xcite ) from text pairs encoded by topic bias model h and p f (i,j) from target model f . we only calculate scores for text pairs where i ̸ = j. ## evaluation method as stated in section 1, we want our solution to handle a large number of classes as well as deal with texts from both unseen authors and topics. this section describes the strategies to assess our method as follows. dataset. to assess the capability to handle a large number of authors, we choose three datasets that contain thousands of authors from three heterogeneous genres: amazon reviews @xcite , reddit @xcite , and fanfiction @xcite . ## experimental results we conducted experimental studies according to the evaluation method described in section 3. tables 2 and 3 show results from the cross-topic and in-distribution-topic studies, respectively. cross-topic. discussion. results from the cross-topic study reveal that arr is effective in 4 out of 6 cases. such results show the effectiveness of our method in reducing the influence of topical information. however, the method also reveals performance penalties in scenarios where topic shortcut seems beneficial, as shown in in-distribution topic experiments. this result is expected since our method reduces the usage of topical information in a text representation. furthermore, we have also observed performance penalties in some cases from crosstopic experiments (reddit and amazon). we hypothesize that the resemblance between these experiments is caused by topic information leakage. ## conclusion in conclusion, we propose authorship attribution solutions that can handle large amount of unseen authors and topics. firstly, we present authorship representation regularization, a self-distillation framework that helps authorship representation to generalize toward unseen topics and authors at scale. secondly, we propose studies in authorship attribution with a cross-topic-open-set environment to assess our method. our experimental results show that our framework can improve recall@8 and mrr over baselines in 4 out of 6 cases in cross-topic environments. however, our method's effectiveness is diminished in the in-distribution topic (or topic leaked) scenarios where models can still use topic-related features to help discriminate the text's writing styles. in future works, it is interesting to investigate the cross-topic data split that can prevent the topic information leakage issue. such investigation should help create a more challenging evaluation method for cross-topic authorship attribution, as well as help create an authorship attribution method that is robust toward various real-world applications.