Addressing Topic Leakage in Cross-Topic Evaluation for Authorship Verification (2024)

Abstract: Authorship verification (AV) aims to identify whether a pair of texts has the same author. We address the challenge of evaluating AV models’ robustness against topic shifts. The conventional evaluation assumes minimal topic overlap between training and test data. However, we argue that there can still be topic leakage in test data, causing misleading model performance and unstable rankings. To address this, we propose an evaluation method called Heterogeneity-Informed Topic Sampling (HITS), which creates a smaller dataset with a heterogeneously distributed topic set. Our experimental results demonstrate that HITS-sampled datasets yield a more stable ranking of models across random seeds and evaluation splits. Our contributions include: 1. An analysis of causes and effects of topic leakage; 2. A demonstration of the HITS in reducing the effects of topic leakage; and 3. The Robust Authorship Verification bENchmark (RAVEN) that allows topic shortcut test to uncover AV models’ reliance on topic-specific features.

## introduction authorship verification (av) is a task that aims to predict whether a pair of texts is written by the same author. a common research problem in av is to develop a model that performs well across unseen topics, domains, or genres @xcite @xcite . our study focuses on the unseen topic problem for two reasons: first, it is not realistic to assume that an author will always write on the same topics. second, the unseen texts might be written on topics with unseen keywords or themes. unlike typical domain adaptation scenar- * corresponding authors. ios, a topic shift can also be subtle and unrecognized. we think it is useful for av systems to be able to recognize authors' writing styles regardless of whether the topics of texts change or not. to develop such systems, cross-topic benchmarks are necessary to assess the model's performance in handling topic shifts and compare the effectiveness of various methods. existing crosstopic av evaluations assume that different topic categories hold dissimilar information. consequently, the topic shift is commonly simulated by separating training and test data across two different sets of topics. for example, two topics are automatically considered dissimilar if they come from two different domain categories. in this paper, we challenge the conventional practice of cross-topic split by viewing topic similarity as a value on a continuous spectrum. in other words, some pairs of topics may be more similar, sharing common attributes or characteristics than the rest. we argue that performing a cross-topic train-test split without considering topic similarity can cause topic leakage. topic leakage has been suggested @xcite to exist when documents in cross-topic test data unintentionally contain topical information similar to those in training data. furthermore, we explain that topic leakage can cause uncertainties in model evaluation and selection. for example, some models may rely on only learning shortcuts from topic-specific features. such models can demonstrate inflated performances in test data with topic leakage. to address the issue of topic leakage, we propose an evaluation framework that takes into account the similarity of topics in a dataset. the crux of our method lies in the similarity-based sampling technique that can create a smaller but more topically heterogeneous version of any existing dataset. this ensures that each topic category is less overlapping in information, thus helping reduce topic leakage. our experimental results demonstrate that our evaluation method can help prevent misleading cross-topic performance by exposing models relying on topic shortcuts and improving model ranking stability compared to comparable-sized datasets with randomly distributed topics. we summarize our contribution as follows. in particular, the pan 2020 and 2021 av task @xcite is considered one of the largest benchmarks for cross-topic av that compares the effectiveness of many av approaches. the pan organizers use a dataset collected from fanfiction.com comprising over 4,000 topics. additionally, @xcite introduced a cross-topic setup (denoted as open unseen fandoms) as one of their evaluation splits that extend the experiment setups of pan competitions. @xcite also attempted to compare various attribution and verification methods on the 15 datasets, but only four were cross-topic evaluations. in more extreme cases, the study by @xcite tests authorship attribution models' behavior by deliberately inverting the topic-author relationship between training and test data, a setup considered more challenging than regular cross-topic train-test splits. other than benchmarks, numerous studies have also attempted to develop av systems to handle topical changes in texts. those studies conducted experiments with many different variations on datasets and problem formulations. for example, @xcite studied the features' topic independence using greek newspaper articles. in addition, @xcite has proposed using text distortion to mask topic-specific terms, having experimented with datasets collected from guardian news articles. more recently, @xcite have proposed a hybrid neural-probabilistic system, achieving state-of-the-art performance on the pan2021 cross-topic av competition. furthermore, @xcite and @xcite studied applying representation learning in authorship verification, evaluating in test sets with both unseen topics and unseen authors. despite advances in cross-topic av research, few studies have questioned the limitations of existing cross-topic evaluation. the most closely related study we have found was conducted by @xcite . the study suggests that current av training objectives either do not control content or only use domain/topic labels to approximate topic differences. the study then proposed a contrastive style representation learning task that controls the topic similarity of each text pair to help force models to favor learning writing styles rather than content. in our understanding, content control is a broader concept similar to cross-topic evaluation. however, the key difference is that their work primarily addresses content control between text pairs to improve style representation learning of av models. on the other hand, our study aims to control the topic information at the dataset level to enhance the reliability of cross-topic evaluation. to the best of our knowledge, in studies proposing either cross-topic benchmarks or cross-topic av methods, we notice that these studies are still limited to the assumption that each labeled topic category is mutually exclusive. our study argues that such an assumption might lead to an overlooked issue of topic information leakage and its consequences toward model evaluation and selection, which we will further describe in section 3. figure 1 : an illustration of two different scenarios of how texts in various topics are distributed on a topic embedding space. in randomly distributed datasets, some topics are more similar to each other than other topics. when performing a cross-topic train-test split from this data, some topics are leaked into the test data. on the other hand, a topic similarity-controlled dataset removes topics that are similar to each other, reducing the degree of leakage. ## topic leakage in cross-topic av studies, it is essential to simulate an environment comprising texts from unseen topics to assess models' behavior when applied to topic-shifted scenarios. however, we argue that there is an issue that might diminish the effectiveness of cross-topic evaluation: topic leakage. we define topic leakage as a phenomenon where some topics in test data unintentionally share information with topics in training data. a topic in training data and another in test data might share common topical attributes despite being labeled as different topics. consequently, the test data may include texts intended to represent unseen topics but are not ''unseen'' regarding topic content. with topic leakage, the ''cross-topic'' property of test sets is diminished. ## proposed evaluation method we hypothesize that a more controlled, topicheterogeneous dataset is less prone to topic information leakage, regardless of the train-test split method. the reason behind this hypothesis is that if there is less overlap in information between each topic category, there would be a higher degree of distribution shift in cross-topic evaluation splits. we present heterogeneity informed topic sampling (hits), an evaluation framework involving a subsampling technique that ensures that the resulting subsampled dataset has less topic similarity and, thus, less topic leakage. our framework aims to process a full, original dataset (denoted as d) into a smaller but more topic-heterogeneous subset, d . to ensure that the resulting subset has low topic similarity, we use an iterative process that selects each candidate topic based on its similarity with previously selected topics. the pipeline of the hits method is illustrated in figure 2 . ## experimental setup we conducted a number of experimental studies to reveal the consequences of topic leakage described in section 3: misleading model evaluation and unstable model rankings. ## experimental results this section illustrates the effectiveness of controlling topic similarity in mitigating topic leakage. we present a number of experiments comparing datasets sampled using the hits method and randomly sampled datasets to see the difference between datasets with heterogeneous and randomly distributed topic sets. we present hits results and randomly subsampled datasets, both with m (number of topics) = 70. for random datasets, we subsampled five datasets, each from a different random seed. the results for random datasets in this section are reported with a mean average across five randomly subsampled datasets from different random seeds, each with a different topic set. on the other hand, since the hits method is deterministic, there is only one subsampled dataset for the hits dataset. evaluation results. first, we assess the performance of the baseline and state-of-the-art av models on both setups. we present the mean average metrics across ten validation folds in ranking stability. we also assess the ranking stability of the hits-sampled dataset in ## component design analysis in this section, we perform additional experiments to study the impact of varying the following components in the hits method: 1. the number of topics, 2. whether to discard the unselected topics and 3. the choice of topic representation encoder. ## raven benchmark we propose the robust authorship verification benchmark (raven) created with our hits framework. the objective of our benchmark is to assess the robustness of authorship verification models by uncovering the topic bias, or their reliance on topic-specific features. ## limitations it is important to address the limitations of our hits evaluation method and the raven benchmark. first, the hits method assumes a large number of topics and samples to still have sufficient data after removing some of the topics. one would also need to consider tuning the parameter m, which is the number of topics in the target subsampled dataset, to balance the trade-off between the dataset size and the degree of topic similarity. second, the hits methods assume existing topic labels for each sample in the dataset. our experiments use the topic label provided in the fanfiction dataset. when applying the hits method to other datasets without such labels, one needs to perform topic modeling methods to obtain the topics. however, the scope of our experiments does not cover the outcome of performing hits on the automatically extracted topics. moreover, the score calibration used in some of the baselines in our experiments does not explicitly handle class imbalance, which might affect metrics such as the f1 score. when applying these baselines to other datasets, post-hoc calibration methods such as the one described by @xcite might be more suitable. we recommend exploring these calibration methods in future work to enhance the robustness of the score adjustments. furthermore, due to the subsampling process, the raven benchmark is still limited in the number of topics, authors, and text samples. therefore, this benchmark only simulates smaller real-world applications where a domain shift between training and inference, such as historical or literary texts, is expected. future efforts can be made to improve the dataset size, which might better simulate other av applications in large corpora. ## conclusion in conclusion, we describe the topic leakage issue in the conventional cross-topic evaluation of authorship verification systems. we illustrate how topic leakage can cause misleading evaluation and unstable model rankings. to tackle these issues, we present hits, an evaluation method that can create a dataset with heterogeneous topic sets from existing datasets. our experimental results show that a heterogeneous topic set can help reduce topic information leakage, thus improving ranking stability in evaluating authorship verification models. furthermore, we present raven, a benchmark created using the hits method on the fanfiction dataset. the benchmark is designed to uncover the degree of topic bias of authorship verification models to select the most robust one. one can also use the hits method on their datasets to create a similar benchmark.