Small-Scale Cross-Language Authorship Attribution on Social Media Comments (2021)

Abstract: Cross-language authorship attribution is the challenging task of classifying documents by bilingual authors where the training documents are written in a different language than the evaluation documents. Traditional solutions rely on either translation to enable the use of single-language features, or language-independent feature extraction methods. More recently, transformer-based language models like BERT can also be pre-trained on multiple languages, making them intuitive candidates for cross-language classifiers which have not been used for this task yet. We perform extensive experiments to benchmark the performance of three different approaches to a smallscale cross-language authorship attribution experiment: (1) using language-independent features with traditional classification models, (2) using multilingual pre-trained language models, and (3) using machine translation to allow single-language classification. For the language-independent features, we utilize universal syntactic features like part-of-speech tags and dependency graphs, and multilingual BERT as a pre-trained language model. We use a small-scale social media comments dataset, closely reflecting practical scenarios. We show that applying machine translation drastically increases the performance of almost all approaches, and that the syntactic features in combination with the translation step achieve the best overall classification performance. In particular, we demonstrate that pre-trained language models are outperformed by traditional models in small scale authorship attribution problems for every language combination analyzed in this paper.

## introduction in cross-language authorship attribution, the true author of a previously unseen document must be determined from a set of candidate authors after training a machine learning model with documents from those candidates in a different language. applications for this research include plagiarism detection or other forensic analyses, where the authorship of an incriminating document must be determined, but ground truth texts for comparison of selected suspects are only available in different languages. the language gap imposes difficulties on the machine learning setup, as the training and testing documents have fewer common features. for example, while some languages may share common words, others use completely different alphabets or writing systems. therefore, this problem requires one of three general strategies to solve: (1) use machine learning features that don't depend on language, (2) use a model that is inherently capable of solving multilingual problems, or (3) transform one feature space into the other to enable the use of languagedependent features (e.g., by using machine translation). in this paper, we explore these three approaches for the case of cross-language authorship attribution, depicted in figure 1 . en de svm language independent features: univ. pos tags dt-grams train test (a) language-independent features train test en de mbert (b) language-independent model (mbert) en de en svm bert distilbert roberta train test char. n-grams word n-grams machine translation sliding window sequences (c) translation figure 1 : the three approaches for cross-language classification models tested in this paper. de represents any of the other languages from the datasets listed in for the first approach, we use universal part-of-speech (pos) tag n-grams @xcite as well as dt-grams @xcite as language-independent features, paired with a support vector machine as a classifier. for the second approach, we utilize a pre-trained multilingual bert model, which doesn't require a separate translation pre-processing step, and fine-tune it using training part of the attribution problem. to the best of our knowledge, our study is the first to analyze the performance of this type of model to cross-language authorship attribution problems, representing our first contribution. finally, we utilize the publicly available marian nmt machine translation system @xcite , and perform several experiments with well-established single-language authorship attribution models, including character n-grams in combination with support vector machines, and also more recent approaches including bert or roberta. for these experiments, we make use of datasets consisting of reddit social media comments compiled by @xcite . we select bilingual foot_1 authors that write documents in english as well as one of german, spanish, french, dutch and arabic. by using these small-scale datasets, we provide valid and realistic scenarios for forensic application, are able to skip additional human translation steps required in previously used translation-based datasets, and generalize approaches from previous studies by applying them on a different type of texts and authors. this represents our second contribution. to ensure the reproducibility of our results and promote future research, all of our code is published online foot_2 . ## related work cross-language text classification problems require different strategies to solve. some problems allow the usage of parallel corpora for training the model, which enables straightforward transformations of output classes from one language to another @xcite . however, parallel corpora are not available for many language combinations, and not suitable for many tasks where the output classes can't be mapped between languages directly. similarly, other approaches include creating a shared low-dimensional embedding space across languages in an unsupervised pre-processing step @xcite , but it has been shown that these approaches usually can be outperformed by adding a small amount of supervised cross-language training data @xcite . transformer-based pre-trained language models have provided many state-of-the-art results in natural language processing (nlp) in general, and models pre-trained on multiple languages show promising performances in a wide variety of nlp tasks @xcite , specifically also in document classification @xcite . however, to the best of our knowledge, these models have not yet been tested on cross-language authorship attribution problems. when focussing on authorship attribution, few cross-language studies remain. llorens and delany (2016) use differently sized windows in which vocabulary richness measurements are aggregated, requiring very large documents. @xcite use a variety of different features including the frequency of universal pos tags on attribution, and also utilize machine-translation followed by traditional attribution techniques, providing their best results. however, the dataset that they use consists of translated documents in a single language pair (spanish -english). more recently, @xcite have shown that classifying bilingual authors of social media comments by using universal (language-independent) pos tags can be improved by including dependency grammar information. in this study, we take inspiration from the latter two studies and compare the performance of grammar-based features to translation-based approaches as well as multilingual language models, which have not been applied to this task. in contrast to similar efforts by @xcite , who classify novels by professional authors, we use small-scale datasets consisting of social media comments, which provide untranslated data and focus on a different text and author type. ## datasets authorship attribution is the task of determining the authorship of an unknown document given a set of candidate authors. an important difference to other text classification problems is that obtaining more data from a specific target (author) is often not possible. a cross-language setup requires multiple documents from multiple bilingual authors who write in the same two languages, further increasing the difficulty of obtaining large quantities of data. for this reason, some previous studies have used translated corpora as an alternative means @xcite to solve the availability issue. there, the author wrote all novels in one language, and translated versions of some of them are used as a source of a different language. while this translation does not fully obfuscate the original authorship @xcite , it represents a different scenario as the original authors of those novels did not write them in more than one language. instead, we want to focus on cross-language features originating from the same author and hence use the corpus presented by @xcite , which consists of comments form the reddit social media platform, written by bilingual authors. here, no additional translation step lies between the originally written documents and the classification model. we further add arabic as a language from a different group of languages to increase the linguistic diversity. the size of any classification dataset can be divided into two parts; the number of target classes (authors) and the number of training samples (documents) per target class. as both of these numbers differ significantly across the datasets in to address the first imbalance and make the results across different language combination directly comparable, we select 10 random authors from each dataset. this number is difficult to increase as it is hard to find bilingual authors in general (e.g., the languages in regarding the second imbalance, we select 10 random documents from each author for training, and repeat all experiments five times (each time, choosing 10 random documents) to accommodate for this imbalance. this way, each author receives the same number of training documents. we choose 10 as this is the lower bound of how many documents an author has written (in the arabic dataset). note that we do not restrict the number of documents used for testing, as it does not influence the training process of the machine learning models, but rather helps to increase the confidence of the evaluation results. we want to emphasize that having few authors and few documents per author is a valid and realistic scenario for many applications, and therefore, the small size of the datasets is a challenging and central corner stone of our work, rather than a limitation. ## methodology we test three different approaches for cross-language attribution, which are depicted in figure 1 . we follow the same evaluation strategy with each approach: for each dataset, we perform all experiments in two directions, (1) train with the english part of the dataset and test with the respective other language, and (2) the other way around. each subsection will discuss any (hyper)parameters of the respective model, the full list of these parameters is shown in ## results the results of the language-independent features show that while adding dependency information to the universal pos tag features increases the f1 score for all language pairs consistently, the extent of this increase differs and is most clearly visible for the english/german dataset. the language-independent model mbert outperforms the dt-grams+svm model for some language combinations, but not for the english/german and english/dutch dataset. the dt-grams exclusively capture grammatical features while the mbert model incorporates content-based properties, making the two feature categories largely independent from each other. we therefore suspect them to be suitable candidates for ensemble models, which we aim to pursue in future work. both the grammar-based features in combination with the support vector machine as well as the multilingual bert model are outperformed by the translation approach using the characterand word-based n-grams. the former confirms the results of @xcite , while the latter is a novel result showing that a multilingual bert model is less efficient for such small datasets. in total, the english-specific syntactic features of the machine-translated documents show the best average performance consistently across many different languages. only for the arabic dataset, the word n-grams produce the best results. machine-translation also improves the performance of the models using languageindependent features. the relevant part of our results in this regard is visible in the combined part of while the multilingual bert model is able to compete with the other language-independent features, it's performance is well below all methods using machine-translation. in general, all language models are outperformed by syntactic and lexicographic features in the respective approaches, signaling that the datasets are too small for fine-tuning them sufficiently. we observe an amplification of this effect on distilbert, which suggests that models produced by knowledge distillation are more susceptible to smaller datasets than their original teacher model. summarized, our findings suggest that for cross-language authorship attribution at a small scale, machine-translation is a highly efficient first step in every case, and syntactic features are a promising candidate for datasets of this size. ## limitations in general, the different datasets show a varying performance, where documents in some languages (e.g., german) are easier to attribute than others (e.g., arabic). we attribute this effect to the linguistic distance between the language pairs (i.e., german and english are closer related to each other than arabic and english). more datasets containing additional language pairs are required for more comprehensive comparisons in this regard. by design, the results using machine translation in the fashion presented in this paper depend on the quality of these translation models. especially for low-resource languages, this means that differences in translation quality between different language pairs are likely to influence the final attribution results. ## conclusion in this paper, we have demonstrated different approaches to the problem of cross-language authorship attribution for bilingual authors writing in both english and one of arabic, german, spanish, french, and dutch. we have analyzed language-independent syntactic features, using multilingual pre-trained language models as well as performing machine translation followed by several single-language solutions. eventually, we show that for small-scale problems with very few training documents, using machine translation followed by models using lexicographic and syntactic features yields the best results for all languages analyzed in this work. in the near future, we want to focus on the influence of the dataset size on the pre-trained language models to see how much data is required for these models to succeed in authorship attribution tasks. also, we want to investigate recent work suggesting that small translation dictionaries represent a suitable substitution for full translation @xcite , which is a time and resource-consuming process.