Extracting Person Names from User Generated Text: Named-Entity Recognition for Combating Human Trafficking (2022)

Abstract: Online escort advertisement websites are widely used for advertising victims of human trafficking. Domain experts agree that advertising multiple people in the same ad is a strong indicator of trafficking. Thus, extracting person names from the text of these ads can provide valuable clues for further analysis. However, Named-Entity Recognition (NER) on escort ads is challenging because the text can be noisy, colloquial and often lacking proper grammar and punctuation. Most existing state-of-the-art NER models fail to demonstrate satisfactory performance in this task. In this paper, we propose NEAT (Name Extraction Against Trafficking) for extracting person names. It effectively combines classic rule-based and dictionary extractors with a contextualized language model to capture ambiguous names (e.g penny, hazel) and adapts to adversarial changes in the text by expanding its dictionary. NEAT shows 19% improvement on average in the F1 classification score for name extraction compared to previous state-of-the-art in two domain-specific datasets.

## introduction there are approximately 4.8 million people being trafficked around the world for commercial sex, a global industry estimated to be worth $99 billion usd @xcite . technology has been a critical tool for traffickers to recruit, advertise and exploit victims (on drugs and crime, 2020) and the majority of human trafficking (ht) victims are advertised on online escort websites @xcite . recently, there have been multiple efforts to leverage ai techniques for analyzing the online advertising market and providing law enforcement with actionable intelligence to counter human trafficking @xcite @xcite . one of the key correspondence to: pratheeksha.nair@mila.quebec tasks in this domain is extracting information from the online escort ads e.g. names, phone numbers, locations, prices and activity types, which are critical for higher level analysis and modus operandi detection. this information needs to be extracted from the text of the ads, and it plays a vital role in tasks such as identifying groups of related ads and finding links between them. however, this is a challenging task since in order to avoid detection, this data is made: • noisy and obscured e.g. using 'rose' symbols as a proxy for dollar sign and spelling variations such as 'cathy' and 'kathy.' • evolving and adversarial e.g. traffickers are actively introducing new patterns and intentional misspellings to avoid detection, e.g. adapting new phrases to indicate underage victims such as 'amber alert' or intentionally misspelling 'miss' as 'mizz.' names have a particular importance. trafficking is an organized activity @xcite and multiple people being involved in an escort ad is a strong indicator of human trafficking. also, the more victims involved in a case, the higher priority of investigation it needs to be given in order to minimize the harm. hence, accurately retrieving all the names in an advertisement is critical for further analysis and action. thus, while the general task of named entity recognition (ner) usually includes diverse entities such as person names, organizations, and geolocations, in this paper, we focus on person name extraction from escort ads for combating human trafficking. most advanced ner models are trained on annotated structured text corpora and/or rely on contextual information for identifying entities. however, escort advertisements are usually colloquial and consist of segmented phrases instead of continuous sentences. the state-of-the-art extractor in this domain @xcite is rule-based but has limited dictionaries and has difficulty dealing with ambiguity such as distinguishing person names from location names. therefore, simple rulebased extractors or machine learning models that are context-based alone will fail in several instances in this domain. to better illustrate this, we compared the results from tjbatch and transformerbert (a baseline nlp model) on two example ads collected from an escort advertisement website: ## related work there have been several surveys and systematic reviews of the problem of named entity recognition (ner) from text @xcite @xcite which all tend to outline three broad techniques: rule-based, learning-based and hybrid approaches @xcite . we group the related works as: rule-based ner: rule-based approaches involve predefined lists, dictionaries and/or rules based on syntactic-lexical patterns which text snippets are matched against. these methods have the obvious limitation of missing out on uncommon rules and never-seen-before dictionary keys. these methods tend to fail especially in applications where text may be informal and unstructured and need to be fortified with additional modules. statistical learning models for ner: some of the methods used for the general ner problem include hidden markov models (hmm) @xcite , support vector machines (svm) @xcite , conditional random fields (crf) @xcite , maximum entropy markov models (memm) @xcite and logistic regression based systems @xcite . in semi-supervised approaches, a small set of the training data is first used for extracting word and context features and the rest of the data is used for training a learning algorithm in a supervised fashion @xcite . however, these methods require large quantities of labelled data for training which is difficult to achieve in many real world applications, including our domain of interest. embedding models for ner: embedding methods such as word2vec @xcite , glove @xcite , and fasttext @xcite generate fixed vectors for each input token and more recent methods such as elmo @xcite and bert @xcite and its variations give context-aware embeddings and have been shown to improve entity extraction. a more recent work, luke @xcite , based on a transformer model that treats not only words but also entities as independent tokens. it computes intermediate and output representations for all tokens, and reports the state-of-the-art results in general ner. other popular architectures for ner include bidirectional lstm + crf @xcite , enhanced with character-level contextualized representations @xcite and contextualized word representations (cwr) based on a bidirectional transformer @xcite . applying pre-trained models directly on escort ads performs poorly because of the very limited amount of labeled data available for training. noncontext-aware models are limited in this domain due to a significant number of names that are also common nouns which may occur with an abnormal frequency due to the adversarial and pseudonymous nature of the domain (e.g. amber, joy, angel...). ner for noisy text: kumar et al. (2020) conducted experiments to explore the sensitivity of bert to synthetic noise (such as spelling mistakes) in text data and showed that performance decreases drastically with increase in noise. @xcite introduced a linear crf model called twitterner that uses random feature dropout and a modified encoding scheme in combination with semi-supervised learning from unlabelled data to handle noisy and unstructured user-generated text such as tweets. however, the low f1-score in ht datasets, as shown in ## problem definition in this section, we formally define the problem of person name extraction in the human trafficking domain both at the word-level and at the ad-level. word-level definition: given an advertisement @xmath0, a 2 , a 3 , ..., a n ), where a i represents a sequence of words in the given ad, the ner task is to output a sequence with labels @xmath1, y 2 , y 3 , ..., y n ) where y @xmath2, 1} (¬person name, person name). since our focus is in correctly retrieving all the names in a given escort ad, we also consider a slightly reformulated problem: ad-level definition: given an advertisement @xmath3, a 2 , a 3 , ..., a n ), we want to find a list of names that have appeared in this ad. the corresponding ner task is to output a set of words the language used in a typical escort ad may be both unstructured and ambiguous. the sentences are usually short and segmented using white spaces or special characters, making it harder to tokenize. it has a free choice in syntax and many person names appear in the text without a proper context. in the second example in figure 1 the names 'rose', 'tiffany' and 'camilla' are not particularly in context. additionally, some names may be similar to location names ('jane' in figure 1 ) or adjectives (olive, hazel). such names cannot simply be omitted from the dictionary either without hurting the recall. foot_0 ## proposed method we propose a person name recognition system (figure 2 ) for combating human trafficking. this system consists of three modules -a rule-based extractor, a disambiguation layer and a dictionary expansion procedure. the first two modules handle named-entity recognition and disambiguation of names respectively. the third module is designed to adapt the system to evolving changes in the advertisement text like newly introduced (pseudo)names. preprocessing: due to its colloquial nature, the input text needs to undergo preprocessing. contractions related to a person name context are expanded (e.g "i'm" is changed to "i am") to fit the rule-based extractor, and all special characters and emojis are removed. the text is also true-cased as the extractor relies on correct part-of-speech tags. rule-based extraction: our base extractor draws on the rule-based extraction process of tj-batch @xcite . it consists of 2 parts -regular expression (regex) rule matching and gazetteer matching (person name dictionary). for the regex rule matching, we manually created 15 rules, including common titles for women (e.g. 'miss') and patterns like 'my name is + nnp'. for the gazetteer matching, we collected a list of common female names as the gazetteer. the rule-based extractors find exactly matched word tokens from the input text. ## results and discussion name extraction: we evaluate the performance of the baseline models and neat for name extraction from text and display the results in tables 2, 3 and 4. we see that neat gives the best overall f1 score performance in the word-level evaluation ( it's worth mentioning that neat has better performance on ht2k than on ht1k. this outcome is not surprising because our weight assignments are trained with ht2k training set explicitly. since advertisements from different websites have different language features and it is unrealistic for our tool to be retrained on every new website we crawl, it's important for our proposed system to do effective transfer learning. results in conversely, both neat and tjbatch perform worse on the non-ht datasets. this is likely because their rule-based components are specifically designed to capture patterns common in escort ads but less so in generic data. we found that using our regex-rule extractor alone, without the rest of our model, neat would only extract 3 words from all of conll and 10 from all of wnut17, confirming the rules do not fit these datasets. the dictionary component of our model also lacks many traditional male names and last names of all genders. we include an ablation study to review the effectiveness of modules. results from neat-base and neat-fixed performs well on the domain datasets but poorly on the generic datasets due to the reason mentioned above. neat-fixed in general performs better than neat-base because of the additional disambiguation module. finally, we see that ht-bert, as well as the other bert-based models, gives mediocre performance in this domain. this shows that standard methods for domain adaptation of language models (i.e. unsupervised training on a domain-specific corpus, and supervised fine-tuning) are insufficient to produce optimal results. an approach that can leverage stronger domain adaptation, like ours, is needed. parameter tuning: we investigate the consequence of adding weight components and experiment with different parameter combinations (of threshold and k) on the ht2k dataset. the threshold is the minimum score a word extracted by neat needs to have to be considered a candidate in the list of words predicted by ht-bert 'fill_mask'. the dataset is split into train, validation and test sets. m base represents the performance measures of the base extractors alone without weights. m weighted denotes the performance measures of the proposed weighted extractor. the results in ## conclusions we presented neat, which addresses the problem of name extraction from noisy, adversarial escort advertisement text. neat consists of two main components -a rule-based extractor (combining a dictionary of names and regex matching) and a disambiguation layer. neat significantly improves on the previous state-of-the-art for this task, with around 19 percentage points at the word-level and 9 percentage points at the ad-level improvements in f1 score on two datasets. both our method and the previous state-of-the-art outperform generic methods, highlighting the continued need for and benefit of domain-driven approaches. by modifying the dictionary and regex matching, our pipeline can be adapted to other domains and tasks to a greater extent than is possible with many generic models and methods. in future work, we plan to investigate this further, by examining other tasks (such as extracting locations as well as person names) and domains (such as more generic but still noisy, twitter data). reproducibility: our code is made available online at https://github.com/tudou0002/ neat . the domain related data used in this paper contains person identifying information and can be made available for research purposes only, based on a data-sharing agreement. please reach out to the authors of the paper for getting access to this data.