MATCHED : Multimodal Authorship-Attribution To Combat Human Trafficking in Escort-Advertisement Data (2025)

Abstract: Human trafficking (HT) remains a critical issue, with traffickers increasingly leveraging online escort advertisements to advertise victims anonymously. Existing detection methods, including text-based Authorship Attribution (AA), overlook the multimodal nature of these ads, which combine text and images. To bridge this gap, we introduce MATCHED, a multimodal AA dataset comprising 27,619 unique text descriptions and 55,115 unique images sourced from Backpage across seven U.S. cities in four geographic regions. This study extensively benchmarks text-only, vision-only, and multimodal baselines for vendor identification and verification tasks, employing multitask (joint) training objectives that achieve superior classification and retrieval performance on in-sample and out-of-data distribution datasets. The results demonstrate that while text remains the dominant modality, integrating visual features adds stylistic cues that enrich model performance. Moreover, text-image alignment strategies like CLIP and BLIP2 struggle due to low semantic overlap and vague connections between the modalities of escort ads, with end-to-end multimodal training proving more robust. Our findings emphasize the potential of multimodal AA to combat HT, providing Law Enforcement Agencies with robust tools to link advertisements and disrupt trafficking networks.

## introduction human trafficking (ht) is a pervasive crime exploiting individuals of all ages and genders, with sex trafficking being particularly prevalent. traffickers coerce victims into commercial sex through violence, threats, deception, and debt bondage, mostly affecting women and girls (europol, 2020; undoc, 2020; @xcite . furthermore, the rise of digital platforms has enabled traf-fickers to exploit online advertisements (ads) for anonymity, overwhelming manual tracking efforts and leaving many cases undetected @xcite . while end-to-end classification methods show promise in detecting ht @xcite @xcite , reliance on expert-generated labels risks overfitting and poor generalization. therefore, law enforcement agencies (leas) and researchers have developed ht indicators for identifying suspicious ads @xcite @xcite . however, these indicators require grouping ads linked to individuals or networks. traditional methods rely on phone numbers and email addresses @xcite ), yet research reports that only 37% of ads contain such identifiers @xcite . supervised @xcite @xcite and unsupervised techniques @xcite @xcite often depend on explicit similarities (e.g., names, phrases, or near-duplicates), limiting effectiveness when vendors alter details to evade detection. authorship attribution (aa) offers a more holistic approach by identifying unique language patterns and stylistic features across ads from the same vendor or group. nlp-based aa methods have successfully linked ads by analyzing subtle written expressions, even when explicit markers differ @xcite . however, existing aa research largely overlooks the multimodal nature of escort ads, which typically include text (title, description) and images. integrating visual cues can enhance aa by capturing stylistic consistencies, locations, or poses that uniquely characterize a vendor's profile. for instance, vendors in larger networks may reuse images with varying text or pair similar text with different images. while current aa methods require at least five ads per vendor @xcite , leveraging multimodal aa (maa) can improve performance for vendors with fewer ads by utilizing the multiple images typically present in each ad. this work aims to support leas in building aa-driven knowledge graphs and enabling targeted investigations across extensive collections of escort ads by making the following contributions: ## related research aa in nlp has advanced from basic stylometric analysis @xcite to sophisticated models detecting distinct linguistic patterns across text segments @xcite @xcite . aa applications span forensic linguistics, aiding attributing authorship in legal contexts @xcite @xcite , to cybersecurity, where it tracks malicious actors and criminal activity across platforms @xcite . however, applying aa to online criminal markets presents unique challenges: conventional models struggle to capture the specialized jargon, coded language, and noise prevalent in illicit environments like illegal criminal marketplaces @xcite . this gap highlights the need for finetuned models that adapt to the nuanced linguistic and stylistic shifts in these contexts. therefore, ardakani (2020) proposed supervised neural networks for aa on backpage escort ads, uncovering stylistic consistencies even when explicit identifiers are altered. similarly, @xcite leverage transformer-based models for vendor identification and verification, effectively linking ads across 41 cities. in addition to text, images in criminal markets can also reveal recurring stylistic patterns, such as backgrounds, lighting, or object placement, complementing linguistic cues when text data is sparse or inconsistent @xcite . multimodal aa (maa) approaches leverage these text and images, enhancing accuracy by merging stylistic patterns across media and creating comprehensive vendor profiles @xcite . this research introduces a novel multimodal dataset, matched, of escort ads collected from seven u.s. cities across four geographical regions. using a multitask training approach on the matched dataset, we establish benchmarks for text, vision, and multimodal domains in escort market ads, laying a foundation for future maa research. our models optimize vendor identification (classifying ads to specific vendors) and verification (assessing if two ads are from the same vendor) through this unified training objective. this enables leas to identify known vendors in closedset environments and link emerging vendors across out-of-data distribution ads in open-set scenarios. finally, our multimodal approach leverages textual and visual cues, enabling leas to track ht networks more precisely across various online markets and platforms, laying the groundwork for advanced aa research. integrating this multimodal data, especially for vendors with limited text ads, further enhances model performance by creating multiple samples per ad. table lugo-graulich and meyer (2021) provides compelling evidence linking backpage escort advertisements to ht, motivating our focus on backpage ads. we curate a dataset of 28,513 ads, comprising 27,619 unique text descriptions and 55,115 unique images associated with 3,549 vendors. approximately 56% of the images feature an escort's face, while the remaining 44% display partial body images (without faces). to establish ground truth for aa tasks, we follow @xcite , extracting phone numbers using @xcite and leveraging networkx @xcite to form vendor communities. each community is assigned a unique vendor label, enabling robust aa analysis. since the vendor label generation process is based on existing literature, detailed steps for phone number extraction and vendor label creation are provided in appendices a.2-a.3. the dataset spans seven major u.s. cities-chicago, houston, detroit, dallas, san francisco, new york, and atlanta-representing four geographic regions: south, midwest, west, and northeast. these regions group ads by city, with average text sequence lengths of 125, 118, 113, and 132 tokens, respectively. detailed statistics, including vendor overlap between regions, text and image ad similarity, sentence and character lengths, and the frequency of text, image, and multimodal ads per vendor, are provided in appendix a.2 (figure 2b ). the south region dataset, containing the largest number of text and image ads, is the primary dataset for training and in-distribution evaluation. the midwest, west, and northeast datasets are used as ood datasets to evaluate model generalization. notably, many vendors appear across multiple regions, meaning the ood datasets include ads from vendors present in the south dataset as well as additional region-specific vendors. ## experimental setup our experiments address 2 aa tasks critical for disrupting ht networks: vendor identification (closedset classification) and vendor verification (open-set metric learning). vendor identification determines whether an ad originates from a known vendor in a predefined candidate set. in contrast, vendor verification assesses whether two ads belong to the same vendor, including vendors unseen during training. we evaluate these tasks using text-only, visiononly, and multimodal baselines on the south region dataset and test ood generalization on midwest, west, and northeast datasets. complete implementation details are provided in appendix a.4. (i). vendor identification task: for vendor identification, we perform multi-class classification using pre-trained backbones with a classification head on the south region dataset. we optimize models with cross-entropy (ce) loss @xcite and a multitask joint objective combining ce with supervised contrastive (supcon) @xcite and triplet losses @xcite . these multitask joint training objectives, referred to as ce+supcon and ce+triplet, enhance feature discrimination by aligning representations of ads from the same vendor while separating those from different vendors. (ii). vendor verification task: since the vendor verification task aims to compare vendor ads based on content similarity, we employ contrastive learning with triplet and supcon losses @xcite to learn discriminative ad embeddings. these embeddings cluster ads from the same vendor while separating those from different vendors, enabling retrieval of all ads linked to a vendor-including those outside the training set-via faiss-based similarity search @xcite . (iii). baselines: following @xcite ), text-only baselines utilizes style-embedding @xcite and declutr-small @xcite backbones, whereas vision-only baselines utilizes vgg-16 @xcite , resnet-50 @xcite , densenet-121 @xcite , inceptionnetv3 @xcite , efficientnetv2 @xcite , convnext-small @xcite , and vit-base-patch16-244 @xcite backbones. the text-only and vision-only baselines are finetuned with ce, ce+triplet, and ce+supcon objectives for vendor identification tasks and triplet or supcon objectives for vendor verification tasks. the multimodal baselines utilize visualbert @xcite , vilt @xcite , and a custom declutr-vit backbone (combining declutr for text and vit for images) with four fusion strategies-concatenation @xcite , mean pooling @xcite , self-attention @xcite , and adaptive auto fusion via a neural network @xcite , enabling nuanced cross-modal interactions by combining complementary signals. finally, we employ the declutr-vit backbone to also perform image-text alignment pre-training task on the combined dataset from all regions, applying three alignment strategies: image-text contrastive (itc, aka clip) @xcite , itc+itm (image-text contrastive and image-text matching) @xcite , and blip2 @xcite . these alignment techniques ensure that text and images from the same ad are represented closely in the latent space, particularly when a single text ad is associated with multiple images. once the pre-training is completed, these backbones are finetuned similarly to other baselines with ce and ce+supcon for vendor identification on the southregion dataset. (iv). evaluation: all the baselines in our research are evaluated for classification and retrieval tasks. due to the class imbalance in our datasets (figure 2b ), we evaluate our classifiers on the macro-f1 metric. additionally, we evaluate all our models on a retrieval task focused on assessing the model's ability to find stylometric similarities between writing and photometric styles in our escort ads. to perform retrieval, the dataset is split into training ("documents") and test ("queries") sets, with text, image, and multimodal embeddings generated by trained models to compute cosine similarity via faiss-based similarity-search operation. text-only and vision-only baselines extract embeddings directly from their respective encoders, while multimodal baselines, including the clip baseline with itc objective, combine text and vision embeddings from the declutr-vit backbone using a mean pooling strategy. for itc+itm and blip2based baselines, we take these image embeddings from the qformer encoder. the retrieval tasks are categorized as text-to-text, image-to-image, or multimodal based on whether query and document embeddings are derived from text, vision, or pooled multimodal representations. all the retrieval tasks are evaluated using r-precision@x, which measures precision when the number of retrieved items equals the number of relevant ads per vendor, with higher scores reflecting more accurate representations of vendor activity @xcite . additionally, mean reciprocal rank (mrr@10) evaluates the average ranking position of the first ten correctly retrieved ads for each query, with scores closer to 1 indicating higher relevance ranking, thereby reducing manual search efforts for leas @xcite . lastly, macro-f1@x independently calculates and averages f1 scores for each vendor class, ensuring equal weight for all vendors regardless of sample size. in macro-f1@x and r-precision@x, x represents the cutoff, defined as the number of relevant items per vendor. ## results this section evaluates text-only, vision-only, and multimodal baselines for vendor identification (classification) and verification (retrieval) tasks. given the space constraints, we only compare the best-performing baselines in our manuscript. however, an extensive analysis of results from all the baselines is provided in appendix tables 4 5 6 7 8 9 10 11 . amongst the multimodal baselines, the end-toend declutr-vit backbone with mean pooling fusion achieves the highest macro-f1 (0.9670), surpassing visualbert (0.9355) and vilt (0.7369). when fine-tuned, alignment baselines (clip, itc+itm, blip2) underperform compared to the end-to-end baseline, though blip2-pretrained declutr-vit backbone comes closest to matching this performance (0.9420). when trained with the joint ce+supcon objective, the declutr-vit backbone performs exceptionally (0.9802) in capturing multimodal relationships. this performance is attributed to the dataset's structure, where each text ad is paired with multiple images and vice versa, ensuring the model encounters diverse combinations during training. (ii). retrieval task: the retrieval task evaluates the effectiveness of metric learning (triplet and supcon losses) and joint-objective classifiers in clustering ad representations by vendor-specific stylometric patterns. the zero-shot (zs) average reflects retrieval performance across datasets without task-specific training, and the ood average measures the generalization of south-trained models to unseen regions. figure 1(a ) compares the text-to-text retrieval performance of text-only pre-trained (•), fine-tuned, and multimodal baselines. fine-tuning on the south region dataset significantly improves performance across all metrics. among text-only baselines, the declutr backbone trained with the joint ce+supcon objective (■) outperforms the ce-only baseline (■) and performs on-par with the supcon-only baseline (■) on ood avg score, while surpassing it on the training dataset. given the consistent performance of the declutr backbone with ce+supcon objective on the classification and retrieval task, we establish it as the benchmark for text-only modality. this benchmark is further compared against the text representations from the multimodal declutr-vit backbone trained end-to-end with ce+supcon ( ) and the fine-tuned declutr-vit backbone, pretrained for text-image alignment task using blip2 objective ( ). the multimodal backbone trained end-to-end with ce+supcon consistently outperforms all baselines on training and ood datasets. figure 1(b ) highlights image-to-image retrieval performance, comparing vision-only pretrained (•), fine-tuned, and multimodal baselines. fine-tuning on image ads also improves retrieval performance. amongst vision-only baselines, the vit backbone trained with the ce+supcon objective (■) achieves superior performance over other baselines on both training and ood datasets, establishing itself as the benchmark for the vision-only modality. despite the better performance of the vit backbone with ce+triplet objective (■) on classification, it underperforms on the retrieval task. we further compare this vision benchmark against the vision representations from the multimodal declutr-vit backbone trained end-to-end with ce+supcon ( ) and the fine-tuned declutr-vit backbone, pre-trained for text-image alignment task using blip2 objective ( ). the end-to-end multimodal backbone with ce+supcon objective consistently outperforms other baselines on ood datasets. however, it underperforms the fine-tuned blip2 baseline on r-precision and macro-f1 metrics for the training dataset. figure 1(c ) compares retrieval performance among multimodal baselines, evaluating the multimodal representation from the end-to-end multimodal declutr-vit backbone trained endto-end with ce+supcon ( ) and the fine-tuned declutr-vit backbone, pre-trained for textimage alignment task using blip2 objective ( ). the end-to-end multimodal backbone with ce+supcon objective consistently outperforms the other baseline across the training and ood datasets. our analysis (appendix given the dual objective of achieving insample and ood distribution performance, the ce+supcon joint objective consistently outperforms or matches other training objectives, demonstrating robustness and generalization. this dual focus enables models to effectively address closed-set vendor identification (linking ads to known vendors in lea databases) and open-set vendor verification (connecting ads from emerging vendors on new platforms). while some baselines excel at one task, our benchmarks are established based on their ability to perform well across both objectives, ensuring practical utility for leas in tracking known and emerging ht networks. (iii). multimodal integration significantly enhances aa performance by leveraging complementary textual and visual features to capture richer authorship patterns. beyond the quantitative improvements shown in ## discussion this research introduces a novel multimodal dataset and conducts extensive benchmarking to demonstrate that multitask joint objectives and multimodal data integration enhance aa performance on both in-sample and ood distribution datasets. by linking escort ads through these techniques, we aim to assist researchers, investigators, and leas study ht indicators. due to space constraints, the main manuscript focuses on critical claims and experimental results, while additional insights and detailed analyses are provided in the appendix. specifically, appendix sections a.2 and a.3 provide detailed information on data-specific statistics, preprocessing steps, label creation, and a datasheet following @xcite . due to the sensitive nature of our research, we cannot display data samples, publicly release our models, or provide model cards. given the explicit sexual content in images and associated privacy concerns, qualitative examples cannot be provided in the paper. however, we conduct extensive qualitative and statistical analyses into model insights and learning in appendix a.6. further details on architectural design, training setup, and computational considerations are presented in appendix a.4, while comprehensive performance metrics for all baselines are available in appendix a.5. lastly, appendix a.7 explores the practical application of aa tasks in building knowledge graphs to support investigative efforts. by structuring our paper this way, we balance clarity and depth. the main manuscript provides a concise overview, while the supplementary material ensures transparency, rigor, and accessibility, enabling domain experts and practitioners to derive actionable insights from our work. ## conclusion through this research, we demonstrate the potential of maa in addressing the complexities of vendor identification and verification within online escort markets. using our novel matched dataset, we extensively benchmark text-only, vision-only, and multimodal approaches, showcasing the advantages of ce+supcon multitask training objectives. our analysis reveals that this dual-objective consistently outperforms single-task approaches across in-sample and ood distribution datasets, enabling leas to identify known vendors while linking emerging ones in new markets. additionally, multimodal integration significantly enhances model performance by capturing complementary patterns across text and images. while text remains the dominant modality, integrating image data along text descriptions adds stylistic cues that enrich the model's capabilities. among text, vision, and multimodal representations, text representations from the declutr-vit backbone emerge as the most effective for retrieval tasks, achieving the best results across all modalities. while pretrained text-image alignment strategies like clip and blip2 fail to establish meaningful cross-modal connections due to low semantic overlap and ineffective use of stylistic features, end-to-end multitask training is a more robust approach for leveraging multimodal data in aa tasks. finally, the performance gap between pre-trained checkpoints and fine-tuned baselines highlights the importance of domain-specific adaptations and task-specific training, providing a strong foundation for future research. by addressing real-world challenges and emphasizing scalability, we aim to equip leas with actionable tools to uncover and disrupt trafficking networks effectively.