Partially-supervised Mention Detection (2020)

Abstract: Learning to detect entity mentions without using syntactic information can be useful for integration and joint optimization with other tasks. However, it is common to have partially annotated data for this problem. Here, we investigate two approaches to deal with partial annotation of mentions: weighted loss and soft-target classification. We also propose two neural mention detection approaches: a sequence tagging, and an exhaustive search. We evaluate our methods with coreference resolution as a downstream task, using multitask learning. The results show that the recall and F1 score improve for all methods.

## introduction mention detection is the task of identifying text spans referring to an entity: named, nominal or pronominal @xcite . it is a fundamental component for several downstream tasks, such as coreference resolution @xcite , and relation extraction @xcite ; and it can help to maintain coherence in large text generation @xcite , and contextualized machine translation @xcite . previous studies tackled mention detection jointly with named entity recognition @xcite @xcite . there, only certain types of entities are considered (e.g., person, location), and the goal is to recognize mention spans and their types. in this study, we are interested in discovering entity mentions, which can potentially be referred to in the text, without the use of syntactic parsing information. our long term objective is to have a model that keeps track of entities in a document for word disambiguating language modeling and machine translation. data from coreference resolution is suitable for our task, but the annotation is partial in that it contains only mentions that belong to a coreference chain, not singletons. nevertheless, the missing mentions have approximately the same distribution as the annotated ones, so we can still learn this distribution from the data. figure 1 shows an example from ontonotes v.5 dataset @xcite where "the taxi driver" is annotated in sample 1 but not in 2. thus, we approach mention detection as a partially supervised problem and investigate two simple techniques to compensate for the fact that some negative examples are true mentions: weighted loss functions and soft-target classification. by doing this, the model is encouraged to predict more false-positive samples, so it can detect potential mentions which were not annotated. we implement two neural mention detection methods: a sequence tagging approach, and an exhaustive search approach. the first method is novel, whereas the other is similar to previous work @xcite . we evaluate both techniques for coreference resolution by implementing a multitask learning system. we show that the proposed techniques help the model increase recall significantly with a minimal decrease in precision. in consequence, the f1 score of the mention detection and coreference resolution improves for both methods, and the exhaustive search approach yields a significant improvement over the baseline coreference resolver. our contributions are: this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . ii we propose a sequence tagging method for mention detection that can model nested mentions. the rest of the paper is organized as follows. sections 2 and 3 describe the two mention detection approaches we use in our experiments. section 4 presents the proposed methods to deal with partially annotated mentions. we use coreference resolution as a proxy task for testing our methods which is described in section 5. section 6 contains the experimental setting and the analysis of results. section 7 contains related work to this study. finally, the final conclusion is drawn section 8. ## sequence tagging model several studies have tackled mention detection and named entity recognition as a tagging problem. some of them use one-to-one sequence tagging techniques @xcite , while others use more elaborate techniques to include nested mentions @xcite . here, we propose a simpler yet effective tagging approach that can manage nested mentions. we use a sequence-to-sequence model, which allows us to tag each word with multiple labels. the words are first encoded and contextualized using a recurrent neural network, and then a sequential decoder predicts the output tag sequence. during decoding, the model keeps a pointer into the encoder, indicating the word's position, which is being tagged at each time step. the tagging is done using the following set of symbols: {[, ], +, -} . the brackets "[" and "]" indicate that the tagged word is the starting or ending of a mention respectively, the symbol "+" indicates that one or more mention brackets are open, and "-" indicates that none mention bracket is open. the pointer into the encoder moves to the next word only after predicting "+" or "-"; otherwise, it remains in the same position. figure 2 shows a tagging example indicating the alignments of words with tags. given a corpus of sentences @xmath0, ..., x m ), the goal is to find the parameters θ which maximize the log likelihood of the corresponding tag sequences @xmath1, ..., y t ): the next tag probability is estimated with a softmax over the output vector of a neural network: figure 2: tagged sentence example where the decoder is initialized with the last states of the bidirectional encoder, d 0 = h m . the i-th word pointed to at time t is given by: at decoding time, we use a beam search approach to obtain the sequence. the complexity of the model is linear with respect to the number of words. it can be parallelized at training time, given that it uses ground-truth data for the conditioned variables. however, it cannot be parallelized during decoding because of its autoregressive nature. ## span scoring model our span scoring model of mention detection is similar to the work of @xcite for solving coreference resolution, and to @xcite for nested named mention detection, as both are exhaustive search methods. the objective is to score all possible spans m ij in a document, where i and j are the starting and ending word positions of the span in the document. for this purpose, we minimize the binary cross-entropy with the labels y: the probability of detection is estimated as: the complexity of this model is quadratic with respect to the number of words. however, it can be parallelized at training and decoding time. @xcite uses an attention function over the embeddings instead of an average. that approach is less memory efficient and requires the maximum length of spans as a hyperparameter. also, they include embeddings of the span lengths which are learned during training. as shown in the experimental part, these components do not improve the performance of our model. ## partially annotated data the partial annotation of coreference data for mention detection means that not labeled spans may be true mentions of entities. thus, the approach of treating spans without mention annotations as true negative examples would be incorrect. on the other hand, the ideal solution of sampling all possible mention annotations, which are consistent with the given partial annotation, would be intractable. we want to modify the model's loss function in such a way that, if the system predicts a false-positive, the loss is reduced. this encourages the model to favor recall over precision by predicting more mention-like spans, even when they are not labeled. we assume that it is possible to learn the true mention distribution using the annotated mention samples by extrapolating the non-annotated mentions, and we propose two ways to encourage the model to do so. weighted loss function: we use a weighted loss function with weight @xmath2, 1} for negative examples only. the sequence tagging model makes word-wise decisions; thus, we consider words tagged as "out of mention", y @xmath3, as negative examples, while the rest are positives. although this simplification has the potential to increase inconsistencies, e.g., having non-ending or overlapping mentions, we observe that the lsmt-based model can capture the simple grammar of the tag labels with very few mistakes. for span scoring, the distinction between negative and positive examples is clear, given that the decisions are made for each span. soft-target classification: soft-targets allow us to have a distribution over all classes instead of having a single class annotation. thus, we applied soft-targets to negative examples to reflect the probability that they could actually be positive ones. for sequence tagging, we set the target of negative examples, y @xmath4, to (ρ, ρ, ρ, 1 -3ρ) corresponding to the classes ([, ], +, -). for span scoring, we change the target of negative examples to y @xmath5. in both cases, ρ is the probability of the example being positive. ## coreference resolution we use multitask learning to train the mention detection together with coreference resolution. the weights to sum the loss functions of each task are estimated during training, as in @xcite . the sentence encoder is shared, and the output of mention detection serves as input to coreference resolution. we use the coreference resolver proposed by @xcite . it uses a pair-wise scoring function s between a mention m k and each of its candidate antecedents m a , defined as: for the sequence tagging approach, the function s m serves as a bias value and it is calculated as: for the span scoring approach, s m is a function of the probability defined in eq. 8, scaled by a parameter v learned during training. instead of the end-to-end objective of @xcite , we use a multitask objective, which adds the loss function of mention detection. we do not prune mentions with a maximum length, nor impose any maximum number of mentions per document. we use the probability of the mention detector with a threshold of τ for pruning. ## experiments and results we evaluate our model on the english ontonotes set from the conll 2012 shared-task @xcite , which has 2802 documents for training, 343 for development, and 348 for testing. the setup is the same as @xcite for comparison purposes, with the hyper-parameters ρ, w, τ optimized on the development set. we use the average f1 score as defined in the shared-task @xcite for evaluation of mention detection and coreference resolution. ## conclusion we investigate two simple techniques to deal with partially annotated data for mention detection and propose two methods to approach it: a weighted loss function and a soft-target classification. we evaluate them on coreference resolution and mention detection with a multitask learning approach. we show that the techniques effectively increase the recall of mentions and coreference links with a small decrease in precision, thus, improving the f1 score. in the future, we plan to use these methods to maintain coherence over long distances when reading, translating, and generating large text, by keeping track of abstract representations of entities.