M ULTI -G RAINED E NTITY P ROPOSAL N ETWORK FOR N AMED E NTITY R ECOGNITION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we focus on a new Named Entity Recognition (NER) task, i.e., the Multi-grained NER task.
This task aims to simultaneously detect both ﬁnegrained and coarse-grained entities in sentences.
Correspondingly, we develop a novel Multi-grained Entity Proposal Network (MGEPN).
Different from traditional NER models which regard NER as a sequential labeling task, MGEPN provides a new method that proposes entity candidates in the Proposal Network and classiﬁes entities into different categories in the Classiﬁcation Network.
All possible entity candidates including ﬁne-grained ones and coarse-grained ones are proposed in the Proposal Network, which enables the MGEPN model to identify multi-grained entities.
In order to better identify named entities and determine their categories, context information is utilized and transferred from the Proposal Network to the Classiﬁcation Network during the learning process.
A novel
Entity-Context attention mechanism is also introduced to help the model focus on entity-related context information.
Experiments show that our model can obtain state-of-the-art performance on two real-world datasets for both the Multi-grained
NER task and the traditional NER task.
I NTRODUCTION
In order to really understand the semantic meanings of natural languages, the ﬁrst step is to identify meaningful entities from the raw text effectively.
Such a process is usually called Named Entity
Recognition (NER), which is one of the fundamental tasks in natural language processing (NLP).
A typical NER takes an utterance as the input and outputs identiﬁed entities, such as person names, locations, and organizations.
The extracted named entities can beneﬁt various subsequent NLP tasks, including syntactic parsing (Koo & Collins, 2010), question answering (Krishnamurthy & Mitchell, 2015) and relation extraction (Lao & Cohen, 2010).
However, accurately recognizing representative entities remains challenging.
Recent work treats NER as a sequence labeling problem.
For example, Peters et al.
(2017) achieves the state-of-the-art performance on sequence labeling problem by incorporating deep recurrent neural networks (RNN) (Hochreiter & Schmidhuber, 1997) with conditional random ﬁeld (CRF) (Lafferty et al., 2001).
However, a critical problem that arises by treating NER as a sequence labeling task is that it can only recognize non-overlapping entities.
It fails to detect entities when they are over-lapping with each other: e.g. a ﬁne-grained entity and a coarse-grained entity as shown in Figure 1.
Traditional NER approaches can only identify either “Belgain Grand Prix” or “Grand Prix”, but not both of them unless two NER models with different granularities are trained seperately.
Here, we deﬁned the NER task that needs to detect both ﬁne-grained entities and coarse-grained entities as the Multi-grained Named Entity Recognition (MGNER) problem.
To solve the MGNER problem, we propose a novel deep neural network named Multi-grained Entity Proposal Network (MGEPN).
Essentially, the idea of MGEPN is to ﬁrst propose entity candidates and then classify these candidates into different categories.
The proposed MGEPN model is composed of two sub-modules: the Proposal Network that proposes entities and the Classiﬁcation
Network that classiﬁes entities.
In the Proposal Network, all possible entity candidates including ﬁne-grained ones and coarse-grained ones will be proposed like (Lee et al., 2017b; He et al., 2018).
Consequently, our model is able to detect both ﬁne-grained entities and coarse-grained entities without explicitly assuming that entities are non-overlapping or totally nested (Finkel & Manning, 2009).
To improve the model performance as well as to speed up the model learning process, context information learned in the Proposal Network is transferred into the Classiﬁcation Network.
In the Classiﬁcation Network, a novel Entity-Context attention mechanism is proposed to help the model focus on entity-related context for classiﬁcation.
In summary, the contributions of this work are:
• We study the Multi-grained Named Entity Recognition (MGNER) problem, which aims to detect both coarse-grained and ﬁne-grained name entities in a single system.
• We propose a novel deep neural network, the Multi-grained Entity Proposal Network (MGEPN), which is composed of a Proposal Network and a Classiﬁcation Network for MGNER.
• MGEPN utilizes context information to detect entities boundaries in the Proposal Network, which is transferred to the Classiﬁcation Network during the NER process.
In the Classiﬁcation Network, a novel Entity-Context attention mechanism is proposed to help the model concentrate on entity-related context information for entity type classiﬁcation.
P ROPOSED
[Equation] ODEL
[Figure]
EntityRepresentation EntityClassiﬁcation WordRepresentation The Classiﬁcation Network
[Figure]
Entity
[Figure]
Attention
Attentive Context
Sentence Features
ELMo
Category Probabilities
Sentence LSTM
Fully Connected
Hidden States Sentence Representation
Word Representation
Entity Proposal
Char level
Character Embedding
Character LSTM
Word level
Word LSTM
The Proposal Network
German German Input Input Output
EU rejects German …
Proposal Probabilities
Output To detect both ﬁne-grained entities and coarse-grained entities, we propose a novel NER approach, called Multi-grained Entity Proposal Network (MGEPN).
The overall framework of MGEPN is illustrated in Figure 2.
Speciﬁcally, MGEPN consists of two modules: the Proposal Network and the Classiﬁcation Network.
The Proposal Network is a deep neural network that proposes potential named entities including ﬁne-grained entities and coarse-grained entities based on an input utterance.
The second module, Classiﬁcation Network, aims at classifying entities proposed in the Proposal
Network into pre-deﬁned entity categories.
Sentence-level semantic features are ﬁrst trained in the
Proposal Network and then transferred into the Classiﬁcation Network to introduce and utilize the context information.
A novel attention mechanism is also involved in the proposed MGEPN model with a special focus on the Entity-Context relationship.
It helps the model capture and utilize entityrelated context information.
It is worthy to mention that in order to improve the learning speed as well as the model performance of MGEPN, the Proposal Network and the Classiﬁcation Network are trained with a series of shared input features, including the pre-trained word embeddings and the pre-trained language model features.
We provide the key building blocks and the properties of the
Proposal Network in Section 3.1 and the Classiﬁcation Network in Section 3.2, respectively.
2. T HE P ROPOSAL N ETWORK
The Proposal Network aims at generating possible entity proposals for each utterance.
It takes an utterance as input and outputs a set of entity candidates.
Essentially, we use a semi-supervised neural network inspired by Peters et al.
(2017) to model this process.
The architecture of the Proposal Network is illustrated in the left part of Figure 2.
Three major components are contained in the Proposal
Network: Word Representation, Sentence Representation and Entity Proposal.
More speciﬁcally, pre-trained word embeddings and character-level word information are used for generating semantic meaningful word representations.
Word representations and the language model embeddings, ELMo (Peters et al., 2018), are concatenated together to produce context aware sentence representations.
For the entity proposal task, possible named entities are proposed, and sentence representations are fed into a fully connected layer to output the probability of a proposal being a real entity.
2.1. W ORD R EPRESENTATION
Given an input utterance with K tokens ( t 1 , ..., t K ) , each token t k (1 ≤ k ≤ K ) is represented as x k using a concatenation of a word-level embedding w k and a character-level word information c k : x k = [ w k ; c k ] .
The pre-trained word embedding w k with dimension D w is obtained from GloVe (Pennington et al., 2014), and the character-level word information
c k is obtained with a bidirectional LSTM (Hochreiter & Schmidhuber, 1997) layer to capture the morphological information.
The hidden size of this character LSTM is set as D cl .
As shown in the bottom of Figure 2, character embeddings are fed into the character LSTM.
Those character embeddings are randomly initialized and learned within the Proposal Network.
The ﬁnal hidden states from the forward and backward character LSTM are concatenated as the character-level word information c k .
2.1. S ENTENCE R EPRESENTATION
To learn the context information from each sentence, another bidirectional LSTM, named word
LSTM, is applied to sequentially encode the utterance.
For each token, the forward hidden states → h k and the backward hidden states ← h k are concatenated into the hidden states h k .
The dimension of the hidden states of the word LSTM is set as D wl .
[Equation]
Besides, we also utilize the language model embeddings pre-trained in the Elmo (Peters et al., 2018) fashion which is an unsupervised way.
The pre-trained Elmo embeddings are concatenated to the hidden states h k in the word LSTM.
Hence, the concatenated hidden states h k for each token can be reformulated as:
[Equation]
where ELMo k is the ELMo embeddings for token t k .
A three-layer hierarchy bi-LSTM neural network is trained as the language model.
Since the lower-level LSTM hidden states have the ability to model syntax properties and higher-level LSTM hidden states can capture context information,
Elmo computes the language model embeddings as a weighted combination of all the bidirectional
LSTM hidden states:
[Equation]
where γ is a task-speciﬁed scale parameter which indicates the importance of the entire ELMo vector to the NER task.
L is the number of layers used in the pre-trained language model, the vector u =
[ u 0 , · · · , u L ] represents softmax-normalized weights that combines different layers.
h LMk,l are the language model hidden states of layer l at time step k .
A sentence bidirectional LSTM layer with hidden dimension D sl is employed on the top of the concatenated hidden states h k .
The forward and backward hidden states in this sentence LSTM are concatenated for each token as the ﬁnal representation f k ∈ R 2 D sl .
2.1. E NTITY P ROPOSAL
Using the semantic meaningful features obtained in f k , we can identify representative entity proposals for each utterance.
The strategy of ﬁnding entity candidates is to ﬁrst generate all possible entity proposals and then estimate the probability of being an entity or not for each proposal.
Different types of entity proposals are generated surrounding each token position, and a sliding window is applied among the sequence to output all possible ones.
In order to limit the number of generated proposals, we set the maximum length of an entity proposal to R .
Thus, there will be at most
R types of different entity proposals generated at each position.
An example of how the proposed method generates different types of entity proposals for the token t 3 is illustrated in Figure 3.
Specifically, assuming that an input utterance consisting of a sequence of ﬁve tokens ( t 1 , t 2 , t 3 , t 4 , t 5 ) , and the Proposal Network takes a maximum length of proposals R of 5.
Here at token t 3 , we employ the following ﬁve proposals: ( t 3 ) , ( t 3 , t 4 ) , ( t 2 , t 3 , t 4 ) , ( t 2 , t 3 , t 4 , t 5 ) , ( t 1 , t 2 , t 3 , t 4 , t 5 ) .
Proposal 1:
[Figure]
Proposal 2:
[Figure]
Proposal 3:
[Figure]
Proposal 4:
[Figure]
Proposal 5:
[Figure]
For each sliding-window location (token), we simultaneously estimate the probability of a proposal being an entity or not, for all types of entity proposals.
A fully connected layer with a two-class softmax function is used to determine the quality of entity proposals:
[Equation]
where W p ∈ R 2 D sl × 2 R and b p ∈ R 2 R are the weights for the entity proposal layer; s k contains 2 R scores including R scores for being an entity and R scores for not being an entity at position k .
In the Proposal Network, we employ the cross-entropy loss as follows:
[Equation]
where y rk is the label for proposal type r at position k and s rk is the probability of being an entity for proposal type r at position k .
It is worth mentioning that, most entity proposals are negative proposals.
Thus, to balance the inﬂuence of positive proposals and negative proposals in the loss function, we keep all positive proposals and use down-sampling for negative proposals when calculating the loss L p .
For each batch, we ﬁx the number of the total proposals, including all positive proposals and sampled negative proposals, used in the loss function as N b .
In the inference procedure of the Proposal Network, an entity proposal will be recognized as an entity candidate if its score of being an entity is higher than score of not being an entity.
2. T HE C LASSIFICATION N ETWORK
The Classiﬁcation Network aims to classify entity candidates obtained from the Proposal Network into different categories.
Here we use both sentence-level context information and the proposed
Entity-Context attention to help the model focus on entity-related context tokens.
The framework of the Classiﬁcation Network is shown in the right part of Figure 2.
Basically, it consists of three modules: Word Representation, Entity Representation and Entity Classiﬁcation.
2.2. W ORD R EPRESENTATION
The network architecture used for learning word representations in the Classiﬁcation Network is the same as that in the Proposal Network.
It is a concatenation of the pre-trained word-level embedding from (Pennington et al., 2014) and a character-level word information learned from a bidirectional character LSTM (Hochreiter & Schmidhuber, 1997).
2.2. E NTITY R EPRESENTATION
According to the proposal type illustrated in Figure 3, we can obtain words contained in the entity candidates from the utterances.
Similar to the sentence representation in the Proposal Network, we concatenate the word representation and the ELMo language model embeddings together as the entity features.
A bidirectional LSTM with hidden size D el is applied to the entity feature to capture sequence information among the entity words.
The last hidden states of the forward and backward
Entity LSTMs are concatenated as the entity representation e ∈ R 2 D el .
. E NTITY C LASSIFICATION
A same word in different context may have different semantic meanings.
Thus, in our model, we take the context information into consideration when learning the semantic representations of entity candidates.
We capture the context information from other words in the same utterance.
Denote c as the context feature vector for these context words, and it can be extracted from the sentence representation in the Proposal Network.
The sentence features trained in the Proposal Network is directly transferred to the Classiﬁcation Network to improve the speed of model learning.
An easy way to model context words is to concatenate all the word representations or average them.
However, this naive approach may fail when there exists a lot of unrelated context words.
To select high-relevant context words and learn an accurate context representation, we propose an EntityContext attention mechanism, where the goal is to simulate the relatedness between the context and the entity.
The attention module takes the entity representation e and all the context features
C =
[ c 1 , c 2 , ..., c N ] as the inputs and outputs a vector of attention weights a :
[Equation]
where W ∈ R 2 D sl × 2 D el is a weight matrix for the attention layer, and a is the Entity-Context attention weight on different context words.
To help the model focus on entity-related context, the attentive vector C att is calculated as the attention-weighted context:
[Equation]
The lengths of the attentive context C att varies in different contexts.
However, the goal of the
Classiﬁcation Network is to classify entity candidates into different categories, and thus it requires a ﬁxed embedding size.
We achieve that by adding another LSTM layer.
An Attention LSTM with the hidden dimension D ml is used and the concatenation of the last hidden states in the forward and backward LSTM layer as the context representation m ∈ R 2 D ml .
Hence the shape of the context representation is aligned.
We concatenate the context representation and the entity representation together as the feature to classify entity candidates: o = [ m ; e ] .
A two-layer fully connected neural network is used to classify candidates into pre-deﬁned categories:
[Equation]
where W c1 ∈ R (2 D ml +2 D el )
× D h , b c1 ∈ R D h , W c2 ∈
R D c 1 × ( D t +1) , b c2 ∈ R D t +1 are the weights for this fully connected neural network, and D t is the number of entity types.
Actually, this classiﬁcation function classiﬁes entity candidates into ( D t + 1) types since we add one type for the entity candidates which are not real entities.
Finally, the hinge-ranking loss is adopted in the
Classiﬁcation Network:
[Equation]
[Equation]
where p w is the probability for the wrong labels y w , p r is the probability for the right label y r , and ∆ is a margin.
The hinge-rank loss urges the probability for the right label higher than the probability for the wrong labels and improves the classiﬁcation performance.
E XPERIMENTS 3. D ATASETS
To evaluate the performance of the proposed MGEPN model for NER, we conduct the experiments on two different datasets: the CoNLL-2003 dataset (Tjong Kim Sang & De Meulder, 2003) and the
OntoNotes 5.0 dataset (Hovy et al., 2006; Pradhan et al., 2013).
Speciﬁcally, there are four types of named entities in the CoNLL-2003 dataset like including locations, persons, and organizations.
The OntoNotes 5.0 dataset is larger and more challenging compared with the CoNLL-2003 dataset.
Essentially, it contains 18 types of entities, such as persons, time, money and languages.
For the
Multi-grained task
, we generate two datasets which contained multi-grained entities, named MG
CoNLL-2003 and MG OntoNotes 5.0, based on the CoNLL-2003 dataset and the OntoNotes 5.
dataset, respectively.
We label coarse-entities that are contained in the ﬁne-grained entities and add them to the multi-grained datasets.
An overview of these four datasets are illustrated in Table 1.
MG CoNLL-2003 and MG OntoNotes
[Table]
0 datasets are used for
[Table]
3. I MPLEMENTATION D ETAILS
We employ the Adam optimizer (Kingma & Ba, 2014) with learning rate decay for all the experiments.
The learning rate is set as 0.001 at the beginning and exponential decayed by multiply 0.9 after each epoch.
The batch size is set to 20 for CoNLL-2003 and 10 for Ontonotes 5.0, respectively.
To alleviate over-ﬁtting, we add dropout regularizations after all LSTM layers in our proposed MGEPN model with a dropout rate 0.5.
In addition, we employ the early stopping strategy when there is no performance improvement on the development dataset after three epochs.
The pre-trained word embeddings are from GloVe (Pennington et al., 2014), and the word embedding dimension D w is 300.
Besides, the ELMo 5.5B data 1 is utilized in the experiment for the language model embedding.
Moreover, the character embedding size is 100, and the hidden size of the Character LSTM D cl is also 100.
The hidden size of the Entity LSTM D el and the Attention LSTM D ml are 300, respectively.
We set the maximum length of entity proposals R to 5.
The hidden dimension of the classiﬁcation layer D h is 50.
The margin ∆ in the hinge-ranking loss for the entity category classiﬁcation is set to 5.
The hidden dimension of the Word LSTM layer D wl is 300 for CoNLL2003 and 1024 for OntoNotes 5.0.
The hidden dimension of the sentence LSTM layer D sl isfor CoNLL-2003 and 128 for OntoNotes 5.0.
The ELMo scale parameter γ used in the Proposal
Network is 3.35 for CoNLL-2003 and 3.05 for OntoNotes 5.0.
The ELMo scale parameter γ used in the Classiﬁcation Network is 3.05 for CoNLL-2003 and 2.95 for OntoNotes 5.0, respectively.
R ESULTS
Multi-grained NER Task .
The advantage of the proposed MGEPN is to detect multi-grained named entities.
In order to validate this advantage, we compare MGEPN with a classical sequence labeling model LSTM-LSTM-CRF for the Multi-grained NER task.
The LSTM-LSTM-CRF baseline combines LSTM with CRF.
It uses one bidirectional LSTM to capture character information and another bidirectional LSTM to capture sentence information.
Experiment results of the Multi-grained NER task on the MG CoNLL-2003 dataset and the MG
OntoNotes 5.0 dataset are reported in Table ?? .
We can observe from Table ?? that, our proposed model MGEPN achieves signiﬁcant improvements compared with the baseline approach.
For both two datasets, our model outperforms the basic LSTM-LSTM-CRF model for more than 10% in terms of precision, recall, as well as the F1 score.
For sentences with over-lapping entities, we generate a sequence label for each annotation and feed them to sequence labeling based methods during both training and testing.
The ﬁrst step in MGEPN is to propose entity candidates in the Proposal Network, where the effectiveness of proposing correct entity candidates immediately affects the performance of the whole model.
To this end, we provide the experiment results of proposing multi-grained named entities [Table] in Table 3.
As shown in Table [Table] on the MG CoNLL-2003 dataset
[Table]
MGEPN obtains pretty high F score for proposing multi-grained named entities on both the development set and the test set.
[Table]
Traditional NER Task .
We also evaluate the proposed MGEPN model on traditional NER tasks comparing with a series of classical NER models to show the effectiveness of MGEPN.
Speciﬁcally, those baselines include: 1) Lample et al.
(2016), which adopts the LSTM-CRF structure; 2) Ma & Hovy (2016), which uses a LSTM-CNNs-CRF architecture; 3) Chiu & Nichols (2016), which proposes a CNN-LSTM-CRF model; 4)
Peters et al.
(2017), which adds semi-supervised language model embeddings; and 5) Peters et al.
(2018), which utilizes the state-of-the-art ELMo language model embeddings.
[Table]
Table 4 shows the F1 scores of different approaches for NER on two experimental datasets.
It can be observed from Table 4 that the proposed MGEPN model outperforms all the baselines on the CoNLL-2003 dataset and achieves comparable performance on the OntoNotoes 5.0 dataset.
To study the contribution of different modules in MGEPN, we also report the performance of two reduced variations of the proposed MGEPN at the bottom of Table 4.
MGEPN w/o attention is a variation of MGEPN which reducing the Entity-Context attention mechanism, and MGEPN w/o context is another variation which reduces all the context information.
By purely adding the context information, the F1 score on the CoNLL-2003 test set improves from 92.23 to 92.26, and by adding the attention mechanism, the F1 score improves to 92.28.
Also, we show the performance for proposing entity candidates in the NER task in Table 5.
[Table]
3. C ASE S TUDY
To demonstrate the beneﬁt of the proposed MGEPN, we conduct the following case study.
We ﬁrst select three utterances that contain multi-grained entities from the MG OntoNotes 5.0 dataset, then run the proposed MGEPN, and ﬁnally show the results in Figure 4.
In Figure 4, ﬁne-grained entities are marked with red boxes like “Academia Sinica ’s institute of Biomedical Science”, and coarse-grained entities are highlighted in yellow like “the Financial Times”.
The baseline model
LSTM-LSTM-CRF only recognizes one coarse-grained entity at the selected area, but the proposed
MGEPN model can successfully detect both entities in each example.
Dr. Konan Peck , an assistant research fellow at Academia Sinica 's Institute of Biomedical Sciences … ….
insurance services for the business of the Gansu Company .
James Drummond , correspondent for the Financial Times of London speaking to us from Cairo .
R ELATED W ORK
Existing approaches for recognizing named entities usually cast the NER task to the problem of sequence labeling, and thus several sequence labeling models are proposed, including linear statistical models such as Conditional Random Fields (CRF) (Ratinov & Roth, 2009), and deep neural networks like recurrent neural networks or convolutional neural networks (CNN).
(Hammerton, 2003) is the ﬁrst work to use LSTM for NER.
Collobert et al.
(2011) employ a CNN-CRF structure, which obtains competitive results to statistical models.
Santos & Guimaraes (2015) introduce a character
CNN to augment the CNN-CRF model.
Most recent work leverages an LSTM-CRF architecture.
Huang et al.
(2015) use hand-crafted spelling features; Ma & Hovy (2016) and Chiu & Nichols (2016) utilize a character CNN to represent spelling characteristics;
Lample et al.
(2016) employ a character LSTM instead.
The attention mechanism is also introduced to NER to dynamically decide how much information to use from a word or character level component (Rei et al., 2016).
External resources have been used to further improve the NER performance including Peters et al.
(2017) and semi-supervised learning.
(Peters et al., 2017) adds pre-trained context embeddings from bidirectional language models to NER.
ELMo (Peters et al., 2018) learns a linear combination of internal hidden states stacked in a deep bidirectional language model to utilize both higher-level states which capture context-dependent aspects and lower-level state which model aspects of syntax.
Other directions for NER related work are transfer learning and multi-task learning.
Transfer learning models for NER transfer information from a source task with plentiful annotations to improve the performance on a target task (Luan et al., 2018; Yang et al., 2017; Lee et al., 2017a; Zirikly & Hagiwara, 2015).
These models can be transferred between different datasets or even different languages.
Multi-task learning models jointly learn the NER task with other tasks like intent prediction (Goo et al., 2018) to obtain better semantic frame results by the global optimization, or ﬁne-grained named entity categorization (Aguilar et al., 2017) for more generalized feature representations.
Several works (Xu et al., 2017; Katiyar & Cardie, 2018; Ju et al., 2018; Wang & Lu, 2018) have been proposed to address the task of nested NER where entities may have overlaps.
To tackle this issue, we propose MGEPN, which can accurately detect both ﬁne-grained and coarse-grained entities by ﬁrst proposing entity candidates with the designed Proposal Network and then classifying entities using the proposed Classiﬁcation Network.
C ONCLUSIONS
In this work we propose a novel Named Entity Recognition approach, called Multi-grained Entity
Proposal Network (MGEPN), which consists of two modules: the Proposal Network and the Classiﬁcation Network.
The Proposal Network aims to output all correct entity candidates.
Among these candidates, both ﬁne-grained and coarse-grained entities are proposed which enables the proposed
MGEPN to identify multi-grained entities.
For better understanding the entities, context information learned in the Proposal Network is then transferred to the Classiﬁcation Network.
In the Classiﬁcation Network, We develop a novel Entity-Context attention mechanism to help the model focus on entity-related context information.
Experiments on two real-world datasets show the effectiveness of the proposed approach for both the NER task and the Multi-grained NER task