SYSML : S t Y lometry with S tructure and M ultitask L earning: I mplications for D arknet Forum Migrant Analysis (2021)

Abstract: Darknet market forums are frequently used to exchange illegal goods and services between parties who use encryption to conceal their identities. The Tor network is used to host these markets, which guarantees additional anonymization from IP and location tracking, making it challenging to link across malicious users using multiple accounts (sybils). Additionally, users migrate to new forums when one is closed further increasing the difficulty of linking users across multiple forums. We develop a novel stylometry-based multitask learning approach for natural language and model interactions using graph embeddings to construct low-dimensional representations of short episodes of user activity for authorship attribution. We provide a comprehensive evaluation of our methods across four different darknet forums demonstrating its efficacy over the state-of-the-art, with a lift of up to 2.5X on Mean Retrieval Rank and 2X on Recall@10.

## introduction crypto markets are "online forums where goods and services are exchanged between parties who use digital encryption to conceal their identities" @xcite . they are typically hosted on the tor network, which guarantees anonymization in terms of ip and location tracking. the identity of individuals on a crypto-market is associated only with a username; therefore, building trust on these networks does not follow conventional models prevalent in ecommerce. interactions on these forums are facilitated by means of text posted by their users. this makes the analysis of textual style on these forums a compelling problem. stylometry is the branch of linguistics concerned with the analysis of authors' style. text stylometry was initially popularized in the area of forensic linguistics, specifically to the problems of author profiling and author attribution @xcite . traditional techniques for authorship analysis on such data rely upon the existence of long text corpora from which features such as the frequency of words, capitalization, punctuation style, word and character n-grams, function word usage can be extracted and subsequently fed into any statistical or machine learning classification framework, acting as an author's 'signature'. however, such techniques find limited use in short text corpora in a heavily anonymized environment. advancements in using neural networks for character and word-level modeling for authorship attribution aim to deal with the scarcity of easily identifiable 'signature' features and have shown promising results on shorter text @xcite . @xcite drew upon these advances in stylometry to propose a model for building representations of social media users on reddit and twitter. motivated by the success of such approaches, we develop a novel methodology for building authorship representations for posters on various darknet markets. specifically, our key contributions include: first, a representation learning approach that couples temporal content stylometry with access identity (by levering forum interactions via meta-path graph context information) to model and enhance user (author) representation; second, a novel framework for training the proposed models in a multitask setting across multiple darknet markets, using a small dataset of labeled migrations, to refine the representations of users within each individual market, while also providing a method to correlate users across markets; third, a detailed drill-down ablation study discussing the impact of various optimizations and highlighting the benefits of both graph context and multitask learning on forums associated with four darknet markets -black market reloaded, agora marketplace, silk road, and silk road 2.0 -when compared to the state-of-the-art alternatives. recent work @xcite @xcite has levered the notion of a heterogeneous information network (hin) embedding to improve graph modeling, where different types of nodes, relationships (edges) and paths can be represented through typed entities. @xcite used a hin to model marketplace vendor sybil 1 accounts on the darknet, where each node representing an object is associated with various features (e.g. content, photography style, user profile and drug information). similarly, @xcite proposed a multiview unsupervised approach which incorporated features of text content, drug substances, and locations to generate vendor embeddings. we note that while such efforts @xcite are related to our work, there are key distinctions. first, such efforts focus only on vendor sybil accounts. second, in both cases, they rely on a host of multi-modal information sources (photographs, substance descriptions, listings, and location information) that are not readily available in our setting -limited to forum posts. third, neither effort exploits multitask learning. authorship attribution of short text: kim (2014) introduced convolutional neural networks (cnns) for text classification. follow-up work on authorship attribution @xcite leveraged these ideas to demonstrate that cnns outperformed other models, particularly for shorter texts. the models proposed in these works aimed at balancing the trade-off between vocabulary size and sequence length budgets based on tokenization at either the character or 1 a single author can have multiple users accounts which are considered as sybils word level. further work on subword tokenization @xcite , especially byte-level tokenization, have made it feasible to share vocabularies across data in multiple languages. models built using subword tokenizers have achieved good performance on authorship attribution tasks for specific languages (e.g., polish @xcite ) and also across multilingual social media data @xcite . non-english as well as multilingual darknet markets have been increasing in number since 2013 @xcite . our work builds upon all these ideas by using cnn models and experimenting with both character and subword level tokens. multitask learning (mtl): mtl @xcite , aims to improve machine learning models' performance on the original task by jointly training related tasks. mtl enables deep neural network-based models to better generalize by sharing some of the hidden layers among the related tasks. different approaches to mtl can be contrasted based on the sharing of parameters across tasks -strictly equal across tasks (hard sharing) or constrained to be close (soft-sharing) @xcite . such approaches have been applied to language modeling (howard and ruder, 2018), machine translation @xcite , and dialog understanding @xcite . ## sysml framework motivated by the success of social media user modeling using combinations of multiple posts by each user @xcite , we model posts on darknet forums using episodes. each episode consists of the textual content, time, and contextual information from multiple posts. a neural network architecture f θ maps each episode to combined representation @xmath0. the model used to generate this representation is trained on various metric learning tasks characterized by a second set of we design the metric learning task to ensure that episodes having the same author have similar embeddings. figure 1 describes the architecture of this workflow and the following sections describe the individual components and corresponding tasks. note that our base modeling framework is inspired by the social media user representations built by andrews and bishop (2019) for a single task. we add meta-path embeddings and multitask objectives to on darkweb markets @xcite . to identify different user accounts across markets that correspond to the same author, we follow a two-step process. first, we select the posts containing a pgp key, and then pair together users who have posts containing the same pgp key. following this, we still have a large number of potentially incorrect matches (including scenarios such as information sharing posts by users sharing the pgp key of known vendors from a previous market). we manually check each pair to identify matches that clearly indicate whether the same author or different authors posted them, leading to approximately 100 reliable labels, with 33 pairs matched as migrants across markets. ## evaluation while ground truth labels for a single author having multiple accounts are unavailable, individual models can still be compared by measuring their performance on authorship attribution as a proxy. we evaluated our method using retrieval-based metrics over the embeddings generated by each approach. denote the set of all episode embeddings as @xmath1, . . . e n } and let @xmath2, q 2 , . . . q κ } ⊂ e be the sampled subset. we computed the cosine similarity of the query episode embeddings with all episodes. let r @xmath3, r i2 , . . . r in denote the list of episodes in e ordered by their cosine similarity with episode q i (excluding itself) and let a(.) map an episode to its author. the following measures are computed. mean reciprocal rank: (mrr) the rr for an episode is the reciprocal rank of the first element (by similarity) with the same author. mrr is the mean of reciprocal ranks for a sample of episodes. recall@k: (r@k) following andrews and bishop (2019), we define the r@k for an episode e i to be an indicator denoting whether an episode by the same author occurs within the subset r i1 , . . . , r ik . r@k denotes the mean of these recall values over all the query samples. baselines we compare our best model against two baselines. first, we consider a popular short text authorship attribution model @xcite based on embedding each post using character cnns. while the method had no support for additional attributes (time, context) and only considers a single post at a time, we compare variants that incorporate these features as well. the second method for comparison is invariant representation of users @xcite . this method considers only one dataset at a time and does not account for graph-based context information. results for episodes of length 5 are shown in