SWEET - Weakly Supervised Person Name Extraction for Fighting Human Trafficking (2023)

Abstract: In this work, we propose a weak supervision pipeline SWEET: Supervise Weakly for Entity Extraction to fight Trafficking for extracting person names from noisy escort advertisements. Our method combines the simplicity of rule-matching (through antirules, i.e., negated rules) and the generalizability of large language models fine-tuned on benchmark, domain-specific and synthetic datasets, treating them as weak labels. One of the major challenges in this domain is limited labeled data. SWEET addresses this by obtaining multiple weak labels through labeling functions and effectively aggregating them. SWEET outperforms the previous supervised SOTA method for this task by 9% F1 score on domain data and better generalizes to common benchmark datasets. Furthermore, we also release HTGEN, a synthetically generated dataset of escort advertisements (built using ChatGPT) to facilitate further research within the community.

## introduction over 6.3 million people worldwide are victims of forced sexual exploitation or human trafficking (ht) on any given day (international labour organization), and the majority of the victims have been advertised online @xcite through escort websites @xcite . ht is an organized crime and traffickers tend to advertise multiple victims simultaneously. hence, finding connections between online escort advertisements hints towards them being posted by the same individual and strongly indicates organized activity. several methods have previously taken the approach of uncovering connections between ads by looking for repeated phrases, phone numbers, locations, prices, service types, etc., in the text @xcite @xcite . hence, efficient entity extractors must extract accurate and relevant information from ad text @xcite @xcite . however, this is very challenging because the text is often noisy, ungrammatical, and obscured. information related to the names of individuals, services, locations, etc, often have letters replaced by symbols or emojis. the writing style constantly evolves and is made adversarial to avoid detection by online filters and moderators. based on discussions with domain experts, including criminologists, law enforcement, and other groups fighting trafficking, it was determined that extracting advertised names from the ads is particularly important. traffickers tend to control, on average, four to six victims and often post advertisements on their behalf @xcite . hence, multiple people being advertised in the same or similar ads is a strong indicator of trafficking @xcite . other entities such as locations, phone numbers and email addresses could also be informative. however, most websites have explicit fields for these entities which makes them more straightforward to extract. the same is not true for person names which are usually embedded inside the text of the ads and are much more complex to extract. traditional extraction methods, such as the embedding-based flair @xcite and spacy nlp models, do not do well on this type of data because they are sensitive to noise and tend to decrease in performance as more noise is injected into the data @xcite . neat @xcite stands as the current sota name extractor in this domain and uses an ad-hoc combination of a name dictionary, a rule dictionary, and a bert-based disambiguation layer. moreover, neat relies on a sample of manually labeled data to provide optimal results. this is a challenge because the relevant datasets for this task are typically curated in-house and not shared publicly due to their sensitive nature. similar constraints also apply to methods such as crowd-sourcing. to address these challenges, we propose a weakly supervised method sweet (supervise weakly for entity extraction to fight trafficking) 1 , that efficiently extracts person names from escort ads and is independent of training labels. we define label-independence in the scope of our work as not requiring domain or task-specific labels. sweet uses a novel combination of fine-tuned language models (elucidated in section 3), antirules, and an hmm-based method of aggregating them. our contributions are: ## background in this section, we cover the background information and related works necessary to understand this paper's context better. named entity recognition named entity recognition (ner) is the task of identifying elements in the text that correspond to high-level categories like person, organization, lo-cation, dates, and other concrete concepts that can be explicitly named. the ner task aims to extract all occurrences of such entities. in this paper, the focus is on human names because they are most associated with an ht victim and names are the most common entity type to appear in escort ads. ## methodology in this paper, we introduce a hybrid weakly supervised skweak-based model that uses both rulebased approaches (such as "antirules" detailed below) and statistical-based approaches (such as de-berta and roberta models trained on different ner datasets) as labeling functions and combine them into a single, more effective predictor. our methodology pipeline is in figure 1 . the proposed weak supervision pipeline in sweet consists of 2 main types of lfs: using the skweak framework, we fit an hmm on the annotated train set and apply the fitted model on the annotations to obtain a final set of aggregated labels. the hmm's initial parameters are obtained using a majority voter that treats each lf as a voter of equal weight to get a predicted label. we also experimented with using the three components of previous sota neat @xcite as lfs but ultimately found it did not boost performance (see neat (weakly supervised) in ## datasets to fine-tune sweet lfs, we used 4 open-source datasets, namely the training sets of conll2003, wnut2017, fewnerd-l1, and wikiner-en, and 2 escort ad datasets, namely htunsup and htgen. to evaluate sweet, we used the test sets of conll2003, wnut2017, btc, tweebank and an escort ad dataset called htname. we also generated a domain dataset htgen whose train set was used for fine-tuning lfs and test set was used for evaluating sweet. we provide the dataset statistics in ## conclusion in this paper, we introduced sweet, a weak supervision pipeline that extracts person names from escort ads without the need for manual labeling. the experimental evaluations show that sweet achieves sota results on the htname dataset with an f1 score of 0.87, outperforming the previous sota by 9% while also generalizing better to conll2003 and wnut2017 datasets. moreover, sweet maintains or improves this performance on removing any lfs trained on relevant domain data. sweet does not require any human annotators and labeling which is a very important improvement over previous methods because it allows it to be applied easily to real-world datasets. it can also be easily adapted to other domains by designing specific new labeling functions. we also release a new public escort ad dataset htgen that uses chatgpt for both data generation and labeling, which will improve reproducibility and benchmarking in this domain where most data cannot be shared publicly, as it contains personal information.