D ream D rug - A crowdsourced NER dataset for detecting drugs in darknet markets (2021)

Abstract: We present DreamDrug, a crowdsourced dataset for detecting mentions of drugs in noisy user-generated item listings from darknet markets. Our dataset contains nearly 15,000 manually annotated drug entities in over 3,500 item listings scraped from the darknet market platform “DreamMarket” in 2017. We also train and evaluate baseline models for detecting these entities, using contextual language models fine-tuned in a few-shot setting and on the full dataset, and examine the effect of pretraining on in-domain unannotated corpora.

## introduction this paper describes the construction of dreamdrug, a dataset of drug-related product listings from darknet marketplaces with human-annotated drug entities, suitable for training and evaluating named entity recognition (ner) systems. we provide a detailed description of all steps of annotation and dataset construction, and we also optimize and evaluate standard ner architectures on the new corpus. our experiments include training models in a few-shot setting and using the full dataset, as well as a study of the effect of domain-adaptive pre-training using unannotated datasets of similar topic and genre. all software used in producing dreamdrug and for reproducing our experiments is published under an mit license along with sample data files foot_0 , the full dataset is available upon request 2 . our main contributions are the following: the paper is structured as follows. in section 2 we introduce the task and provide an overview of recent related work on information extraction from noisy user-generated text. section 3 describes the corpus construction process. section 4 and section 5 present our baseline experiments and results, respectively, and section 6 provides a brief conclusion. ## background extracting information from text concerning illicit drug consumption or trade is a non-trivial task, since such information is rarely available in a structured form. patients who misuse their medication are more likely to discuss this on social media than with medical professionals @xcite , while drug dealers intentionally misspell names of drugs on social media to prevent automatic detection @xcite . drug-specific named entity recognition (ner) models are most often developed for the medical domain and cannot handle such misspellings, the usage of slang, and other characteristics of noisy user-generated texts such as lack of punctuation and non-standard grammar @xcite . many of these issues are peruanse 92 cocain very good snife and for people tghe want cooking out very good smills good frech from the block orginail good good good aaa. illustrated by the example listing from a darknet marketplace in figure 1. still, many organizations are dependent on extracting information about drugs or similiar goods from noisy online sources. public health organizations study drug use patterns in society such as the opioid epidemic in the united states @xcite , pharmaceutical companies can collect evidence of adverse drug reactions from social media data @xcite , and law enforcement agencies can track illicit trading activities in darknet markets. each of these use-cases can be supported by the automatic detection of drug entities in noisy user-generated text. to enable the development of such systems we create a dataset for training and evaluation by annotating drug-related item listings of darknet marketplaces. an example of such a listing is shown in figure 2 . ## experiments we train a set of baseline transformer-based models, examine the effects of domain-adaptive pretraining using multiple datasets, and also evaluate top-performing models in a few-shot training scenario. ## results tables 5 and 6 present the perfomance of top baseline configurations in the few-shot setting and when using the full training set. the bert-based systems achieved higher scores in the few-shot setup, while roberta-based setups performed better when trained on the full dataset. for both setups we include the performance of the top system with and without dropout as well as with domain-adaptive pretraining (dapt) on each of the additional datasets. for both the few-shot setup and the full training set we also include the best-performing configuration of the other language model (roberta for the few-shot setting and bert for the full dataset) and the performance of the best flair model. the effect of domain adaptation using in-domain unannotated text is especially pronounced in the few-shot setting, where it achieves an 8 point increase in f-score, 6.5 of which can be achieved using only the dm dataset, which is the one most similar to the listings in dreamdrug. we believe this shows the strong potential of pretraining on unannotated text that is characteristic of both the domain and the genre of a novel task, especially in settings where the availability of human-annotated data is limited. finally, ## conclusion we have constructed the novel dataset dreamdrug for the task of recognizing drug entities in darknet market listings, used it to train and evaluate baseline ner models, and demonstrated the effectiveness of domain-adaptive pretraining on large unannotated datasets. as one of the first entity-annotated corpora of darknet text, and the first task-specific one, dreamdrug will facilitate the study of named entity recognition and related information extraction tasks in noisy user-generated texts. additionally, it can also be extended with several types of annotation, including entities from multiple classes, relations such as "drug-quantity" or "drug-strength", or links from entity mentions to a knowledge base.