S lot GAN : Detecting Mentions in Text via Adversarial Distant Learning (2022)

Abstract: We present SlotGAN, a framework for training a mention detection model that only requires unlabeled text and a gazetteer. It consists of a generator trained to extract spans from an input sentence, and a discriminator trained to determine whether a span comes from the generator, or from the gazetteer. We evaluate the method on English newswire data and compare it against supervised, weakly-supervised, and unsupervised methods. We find that the performance of the method is lower than these baselines, because it tends to generate more and longer spans, and in some cases it relies only on capitalization. In other cases, it generates spans that are valid but differ from the benchmark. When evaluated with metrics based on overlap, we find that SlotGAN performs within 95% of the precision of a supervised method, and 84% of its recall. Our results suggest that the model can generate spans that overlap well, but an additional filtering mechanism is required.

## introduction detecting mentions of entities in text is an important step towards the extraction of structured information from natural language sources. mention detection (md) components can be found frequently in systems for named entity recognition (ner) @xcite , entity linking @xcite , relationship extraction @xcite , and coreference resolution @xcite @xcite , where accurately modeling mentions is crucial for downstream performance. the md task is often subsumed under ner, where most effective approaches employ supervised learning with exhaustively annotated datasets. these methods become less feasible in cases where we need to rapidly build md systems, for example, when moving to a domain with incompatible ner classes; or when there are not enough resources to create a labeled dataset. in contrast, we assume that we have access to an unlabeled corpus, and a list of known entity names (i.e. a gazetteer). we propose slotgan-a framework for detecting mentions that uses a generator to extract spans conditioned on some input text, and a discriminator that determines whether a span comes from the generator, or from the gazetteer (see fig. 1 ). in contrast with distant supervision methods that require training with false negatives @xcite @xcite , slotgan relies on the discriminator to learn patterns that are not likely to be names of entities (such as verb phrases, or very long spans, which rarely occur in a gazzetteer), thereby improving the generator's ability to detect valid mentions. we evaluate the method in a md task using the conll 2003 english dataset @xcite . we observe that the absence of strong supervision in slotgan results in different, yet valid notions of what constitutes an entity. for instance, while in the sentence "on the road to tripoli airport..." the word tripoli is selected as a gold mention, slotgan selects tripoli airport. in this case, exact match metrics for ner underestimate performance, assigning zero precision and recall. to account for this, we introduce overlap-based metrics into the evaluation. when using exact boundary match metrics, slot-gan exhibits lower performance compared to different baselines. when evaluating overlap, precision (how much of the predicted span overlaps with the gold span) is within 95% of the performance of the supervised baseline, while recall (how much of the gold span is actually predicted) is within 84%. we observe that slotgan tends to generate more and longer spans than those in the benchmark, and in some cases it relies only on capitalization. our contributions are the following: 1) a framework towards distantly-supervised md that avoids explicit training with false negatives, and an imple- the nobel committee in norway sweden emb emb pad gazetteer figure 1: slotgan consists of a generator g trained to extract spans from an input sentence. we represent spans as matrices containing embeddings of words in a span, padded with zeros to a fixed length l. true spans are generated from a gazetteer. a discriminator d is trained to determine if a span was generated from g or from the gazetteer. ## slotgan in the md task, we are given a sentence from a corpus as a sequence of words (w 1 , w 2 , ..., w n ). the output of the system is a set of spans that contain a mention, and each span is a tuple (i s , i e ) where i s is an integer indicating the position where the span starts, and i e the position where it ends. additionally, we have access to a gazetteer @xmath0, e 2 , ..., e n ) containing names of entities relevant to a particular domain. slotgan is a method for md based on generative adversarial networks @xcite . it consists of a generator trained to extract spans from a sentence, and a discriminator that determines whether a span comes from the generator or from the gazetteer. we define the embedding of a sentence @xmath1, ..., w n ) as a matrix emb(w) ∈ r d×n , where emb is a function that maps words to d-dimensional pretrained embeddings (for example, from the input embedding layer of bert @xcite ). we represent each mention span in a sequence as a matrix in a space @xmath2, where l is the length of the sequence. for a span (i s , i e ), the matrix contains the embeddings of the words within the span, from column i s to column i e , and is zero in the remaining columns. the generator takes as input the embedding matrix emb(w) of a sentence, and assigns each of its columns to one of k slots. the output is a sequence of k span representations (s i ) k @xmath3, such that the j-th column of s i contains the j-th column of the input matrix, if it was assigned to slot i. unused columns of s i are filled with zeros. when sampling a name e of an entity in the gazetteer, we embed it as emb(e) and then add zero padding via a pad function until reaching a maximum length l, to obtain a span representation in s. the amount of padding is added randomly to both sides of an entity name, with the purpose of emulating how in a sentence, a mention can appear at an arbitrary position. the discriminator takes as input span representations in s, and outputs a score that should be high for samples from the gazetteer, and low for samples from the generator. denoting as p w the distribution used to sample sentences from the corpus, and as p e the distribution for sampling names from the gazetteer, the generator and discriminator are trained via gradient descent using the w-gan @xcite minimax optimization objective: to allow also not extracting any mentions when not required, we randomly introduce empty spans in the gazetteer, and we reformulate the generator objective with an equality constraint. following @xcite , we define the constraint in terms of a differentiable function c such that c(g(emb(w) i ) counts the number of transitions from zero to non-zero, and vice versa, in a span representation. for valid spans, this should be equal to 2. we solve the problem introducing a lagrange multiplier λ, and the term in eq. 1 that depends on the generator becomes at test time, we can use the spans produced by the generator as predictions for mentions. alternatively, we can balance precision and recall by leveraging the discriminator, by only keeping spans with a score d(s i (w)) > t where t is a threshold. we implement the generator using bert (devlin et al., 2019), followed by a modified slot attention layer @xcite to model discrete selections of distinct spans. the discriminator is a temporal cnn. for more details on the architecture, we refer the reader to appendix a. ## related work the task of md has been addressed under ner effectively via supervised learning @xcite @xcite @xcite . some works address the lack of labeled data in a target domain by applying adaptation techniques from a source domain with labeled data @xcite @xcite . in this work we focus on the case where annotations are not available. closer to our work are methods for weakly or distantly supervised learning, where heuristics and domain-specific rules are used to generate a noisy training set, often using external sources like gazetteers @xcite @xcite @xcite . these methods are limited by false negatives that reduce recall in md. furthermore, even though rules can be used to annotate a dataset at a large scale, the process of devising these rules in the first place can be tedious, and might require domain expert knowledge. @xcite recently introduced a fully unsupervised method for ner that uses a pipeline of clustering over word embeddings, a generative model, and reinforcement learning to solve the ner task without any labels or external sources. these elements are optimized separately, whereas slotgan provides an end-to-end architecture. ## conclusion we have presented slotgan, a method for training a mention detector that only requires unlabeled text and a list of entity names, that relies on implicit supervision provided by a discriminator that is also optimized during training. this results in spans that overlap well with gold spans, but also a tendency towards generating more and longer spans, and relying on capitalization only. this suggests that spans predicted by slotgan are likely to be correct, but an additional mechanism is needed to filter them. this can be enforced via tighter constraints on generated spans, or a stronger discriminator. even though its performance is close to a supervised model according to overlap-based metrics, it cannot match other methods that also use a gazetteer or are unsupervised. in spite of this, we consider slotgan a promising framework for other ie tasks with less supervision, for example, where relations between slots could be induced. the end-to-end architecture also presents an opportunity for fine-tuning with gold labels, which we plan to explore in future work.