Combining Long Short Term Memory and Convolutional Neural Network for Cross-Sentence n -ary Relation Extraction
Abstract
A combined Long Short Term Memory and Convolutional Neural Networks ( LSTM CNN + WF + PF ) that exploits word embeddings and positional embeddings is proposed for cross-sentence n -ary relation extraction.
The proposed model brings together the properties of LSTM s and CNN s, to simultaneously exploit long-range sequential information and capture the most informative features, essential for cross-sentence n -ary relation extraction.
The LSTM CNN + WF + PF model was evaluated using standard datasets for cross-sentence n -ary relation extraction, where the model signiﬁcantly outperforms baseline CNN and LSTM model and a combined CNN LSTM model.
The paper also shows that the LSTM CNN model outperforms the current state-of-the-art methods on cross-sentence n -ary relation extraction.
1. Introduction
Research in the ﬁeld of relation extraction has largely focused on identifying binary relations that exist between two entities in a single sentence, known as intra-sentence relation extraction [Bach and Badaskar, 2007].
However, relations can exist between more than two entities that appear across consecutive sentences.
For example, in the text span comprising the two consecutive sentences given in L ISTING 1, there exists a ternary relation response across three entities: EGFR , L858E , geﬁtnib .
This relation extraction task, focusing on identifying relations between more than two entities – either appearing in a single sentence or across sentences, is known as cross-sentence n -ary relation extraction .
L ISTING 1: T EXT SPAN OF TWO CONSECUTIVE SENTENCES
“ The deletion mutation on exon-19 of EGFR gene was present in 16 patients, while the L858E point mutation on exon-21 was noted in 10.
All patients were treated with geﬁtnib and showed a partial response. ”
This paper focuses on the cross-sentence n -ary relation extraction task.
Formally, let { e 1 , .., e n } be the set of entities in a text span S containing t consecutive sentences.
For example, in the text span comprising 2 sentences ( t = 2 ) given in L ISTING 1 above, the relation that can be extracted is that cancer patients with mutation v (EGFR) in gene g (L858E) demonstrated response to drug d (geﬁtnib).
Thus, a ternary relation response ( EGFR , L858E , geﬁtnib ) exists among the three entities spanning across the two sentences in L ISTING 1.
The entities e 1 , .., e n in a text span can either appear in a single sentence ( t = 1 ) or multiple sentences ( t > 1 ).
Thus, given an instance deﬁned as a combined sequence of m tokens x = x 1 , x 2 , ..., x m in t consecutive sentences and a set of entities { e 1 , .., e n } , the cross-sentence n -ary relation extraction task is to identify the n -ary relation (if it exists) among the entities in x .
Cross-sentence n -ary relation extraction is particularly challenging compared to intra-sentence relation extraction for several reasons.
Lexico-syntactic pattern-based relation extraction methods [Hearst, 1992, Brin, 1998, Agichtein and Gravano, 2000] have been shown to be highly effective for intra-sentence relation extraction.
Unfortunately, such pattern-based relation extraction methods cannot be readily applied to cross-sentence n -ary relation extraction because it is difﬁcult to match lexico-syntactic patterns across longer text spans covering multiple sentences.
Features extracted from the dependency parse trees for individual sentences [Culotta and Sorensen, 2004, Bunescu and
Mooney, 2005, Fundel et al., 2006, Xu et al., 2015, Miwa and Bansal, 2016] have also been found to be extremely useful for intra-sentence relation extraction.
However, it is not clear how best to merge dependency parse trees from different sentences to extract path-based features for cross-sentence relation extraction.
Moreover, difﬁculties in coreference resolution and discourse analysis, further complicate the problem of detecting relations among entities across sentences [Elango, 2005].
The principal challenges for cross-sentence n -ary relation extraction arise from: (a) difﬁculties in handling long-range sequences resulting from combining multiple sentences, (b) modeling the contexts of words related to different entities present in different sentences, and (c) the problem of representing a variable-length text span containing an n -ary relation using a ﬁxed-length representation.
To address these issues, we propose a combined model consisting of a Long Short-Term
Memory unit and a Convolutional Neural Network ( LSTM CNN ) that exploits both word embedding and positional embedding features for cross-sentence n -ary relation extraction.
The LSTM is used as the ﬁrst layer to encode the combined set of sentences representing an n -ary relation, thereby capturing the long-range sequential information.
The hidden state representation obtained from the
LSTM is then used with the CNN to further identify the salient features for relation classiﬁcation.
Our main contributions in this paper can be summarised as follows:
a. An LSTM CNN + WF + PF model that exploits word embedding and position embedding features is proposed for cross-sentence n -ary relation extraction.
We compare the performance of the proposed model against several baseline models such as CNN , LSTM and a combined CNN LSTM model and show that the proposed model signiﬁcantly outperforms the evalauted baselines.
b.
An evaluation of the proposed model against State-Of-The-Art (SOTA) for cross-sentence n -ary relation extraction on two different benchmark datasets is presented.
Results show that the proposed model signiﬁcantly outperforms the current SOTA methods for cross-sentence n -ary relation extraction.
2. Related Work
There is a large body of research on intra-sentence relation extraction [Bach and Badaskar, 2007].
However, our main focus in this paper is on cross-sentence relation extraction.
Therefore, we will limit our discussion below to cross-sentence relation extraction.
Most of the studies on cross-sentence relation extraction have limited their focus on extracting binary relations present across sentences [Swampillai and Stevenson, 2010, Quirk and Poon, 2016,
Moschitti et al., 2013, Nagesh, 2016].
However, recently, Peng et al.
2017 proposed graph-LSTMs not only to consider binary relations across sentences, but also identify n -ary relations involving n entities both within a single sentence and across sentences.
Speciﬁcally, the study considered relations with n =2 (binary relations) and n =3 (ternary relations), as they found it difﬁcult to obtain relation instances where n >
[Peng et al., 2017].
To date, this is one of the large datasets that is available for evaluating n -ary cross-sentence relation extraction.
We therefore evaluate our proposed model on the dataset developed by Peng et al. 2017.
In addition to the Peng et al. 2017 dataset, we also evaluate our model on a chemical-induced disease relation extraction dataset [Li et al., 2016] that comprises instances with binary relations in single sentences and across two sentences.
The task of cross-sentence relation extraction has largely used features drawn from dependency trees [Swampillai and Stevenson, 2010, Quirk and Poon, 2016, Peng et al., 2017].
The dependency features are useful particularly in providing connections between sentences.
Tree kernel features [Moschitti et al., 2013, Nagesh, 2016] are also used to avoid manual feature engineering, and provide more efﬁcient and comprehensive feature sets.
Recently Peng et al.
2017 proposed the idea of graph-LSTMs, which employ document graphs to capture intra-and inter-sentence dependencies to target cross-sentence n -ary relation extraction.
Graph-LSTMs, learn by partitioning the document graph into two Directed Acyclic Graphs (DAGs) and are shown to outperform linguistic features and also neural network based systems such as CNNs, LSTMs and BiLSTMs.
Although graphLSTMs are useful in modelling rich linguistic knowledge by exploiting graph edges, Song et al.
2018 proposed the graph-state LSTM model that uses a parallel state to model each word, recurrently enriching state values via message passing.
Graph-state LSTMs are shown to outperform graph-LSTMs for both binary and ternary relations across sentences.
Although graph-LSTMs and graph-state LSTMs are useful to model n -ary relations across sentences, the process of creating
DAGs covering words in multiple sentences is complex and error-prone.
It is not obvious how to connect two parse trees; also the parse errors will compound during the graph creation step.
Moreover, the co-reference resolution and discourse features used by Peng et al.
2017 do not always improve performance of cross-sentence relation extraction.
In comparison to the above featurebased systems, we present in this paper a simple neural network-based approach that does not rely on heavy syntactic features such as dependency trees, co-reference resolution or discourse features for cross-sentence n -ary relation extraction.
Speciﬁcally, we propose a combined lstm cnn+wf+pf network model that simply takes as input the combined sequence of sentences containing n -ary relations.
The combined model simultaneously exploits the ability of LSTMs to generate features that preserve long-range relations among words in the combined sequence of sentences and CNNs to generate different weighted combinations of those features to select the most informative ones via pooling.
Further, although several studies have recently explored combining CNNs and RNNs for text classiﬁcation [Lai et al., 2015, Lee and Dernoncourt, 2016, Hsu et al., 2017, Zhang et al., 2016] and sentiment analysis [Wang et al., 2016], to the best of our knowledge, we are the ﬁrst to propose a combined LSTM CNN model for cross-sentence n -ary relation extraction.
Zhang et al.proposed a dependency sensitive convolution neural network (DSCNN) model as a general-purpose classiﬁcation system for both sentences and documents, very similar to our proposed model.
However, the major difference between our model and the DSCNN model, is that the DSCNN model does not employ position embeddings crucial for relation extraction.
The sentence-level represen- tation in the DSCNN model is obtained by ﬁrst passing the input sentence to an LSTM layer to obtain the hidden-state vector, which is then provided as an input to a CNN.
Finally, the maxpooling layer is then employed to obtain the salient features.
The document-level representation is simply obtained by learning from individual sentences.
Thus, the DSCNN model does not employ position embeddings to differentiate the input words w.r.t the entities between which a semantic relation holds either in a single sentence or across sentences.
Such position features are not useful for text classiﬁcation and thus the DSCNN model simply does not use them.
This renders the
DSCNN model of little use for relation extraction.
In comparison to DSCNN model, the proposed model employs position embeddings to capture the relative distance between the respective word and the n entities in the text.
We show that the use of positional embeddings signiﬁcantly improves performance of cross-sentence n -ary relation extraction.
3. Cross-Sentence n -ary Relation Extraction
The architecture of the proposed LSTM CNN + WF + PF model - combined LSTM CNN using word features ( WF ) and positional features ( PF ) for cross-sentence n -ary relation extraction, is shown in
Figure 1.
We describe the different components of the proposed model in the following sections.
3.1 Input Representation
The input to the LSTM CNN + WF + PF model is the combined sequence of tokens in a text span S comprising t consecutive sentences where an n -ary relation exists between n entities.
The sequence of tokens is transformed into a combination of word embeddings and position embeddings as described below.
3.1.1 W ORD E MBEDDINGS
The transformation of words into lower dimensional vectors are observed to be useful in capturing semantic and syntactic information about words [Mikolov et al., 2013, Pennington et al., 2014].
Thus, each of the words in the combined sequence x = x 1 , x 2 , ..., x n is mapped to a k − dimensional embedding vector using a look-up matrix W ∈ R | V |× k where | V | is the number of unique words in the vocabulary.
3.1.2 P OSITION F EATURES
Following Zeng et al.
2014, Positional Features ( PF s) are used to encode the position of entities for n -ary cross-sentence relation extraction.
The ( PF s) provide relative distance of the current word w.r.t the entities in the sequence.
For instance, in the sequence of tokens provided in L ISTING 1, the relative distance of the word ‘mutation’ to the left of the entity EGFR is − 4 and the same word ‘mutation’ appearing to the right of EGFR is 10 , respectively 1 .
Each relative distance is mapped to a vector of dimension d b , initialized randomly.
The position features help CNN to keep track of how close the words are to the target entities.
Given entity mentions e 1 , .., e n in the sequence x = x 1 , x 2 , ..., x n , Although n PF s can be deﬁned based on n entities, the proposed model, speciﬁcally considers only e 1 and e n to create position embeddings for the input sequence, as preliminary experiments demonstrated a decrease in performance with having n PF s in the model.
Thus, the model deﬁnes two sets of PF s P F 1 and P F n for the entities e 1 and e n , respectively, as a combination of the relative distances from the current word to the respective entity.
The position embedding matrices are randomly initialised and the relative distance of words w.r.t entities are transformed into real valued vectors by looking up the position embedding matrices.
Thus, the vector representation for models using position features, transforms an instance into a matrix S ∈ R s × d by combining the word embeddings and position embeddings, where s is the sentence length and d = d a + d b × 2 ( d a and d b are the dimensions of word and position embeddings, respectively).
3.2 LSTM Layer
Although RNNs are useful in learning from sequential data, these networks are observed to suffer from the problem of exploding or vanishing gradient, which makes it difﬁcult for RNNs to learn long distance correlations in a sequence [Hochreiter and Schmidhuber, 1997, Hochreiter et al., 2001].
To speciﬁcally address this issue of learning long-range dependencies, LSTM [Hochreiter et al., 2001] was proposed, which maintains a separate memory cell that updates and exposes the content only when deemed necessary.
Given the long-range sequential information resulting from the combined set of sentences expressing an n -ary relation, LSTM is an excellent choice to learn long-range dependencies.
Thus, as shown in Figure 1, the transformed vector representation combining word embeddings and position features is provided as input to the LSTM layer.
The LSTM units at each time step t is deﬁned as a collection of vectors in R l and comprises the following components: an input gate i t , a forget gate f t , an output gate o t ,
a memory cell c t and a hidden state h t .
l is number of LSTM units and the entries of the gating vectors i t , f t and o t are in [0 , 1] .
The three adaptive gates i t , f t and o t depend on the previous state h t − 1 and the current input x t (Equations 1-3).
The candidate update vector g t (Equation 4) is also computed for the memory cell.
[Equation]
The current memory cell c t is a combination of the previous cell content c t − 1 and the candidate content g t , weighted respectively by the input gate i t and forget gate f t (Equation 5).
[Equation]
The hidden state h t , which is the output of the LSTM units is computed using the following equation:
[Equation]
[Equation] Here σ denotes a sigmoid function and (cid:12) denotes element-wise multiplication.
3.3 CNN Layer
Let h i ∈
R l be the l -dimensional hidden state vector corresponding to the i -th token in the combined sequence x .
The combined hidden state vectors in the sequence of length m is represented as:
[Equation]
where ⊕ denotes vector concatenation.
In general, let h i : i + j refer to the concatenation of hidden state vectors
h i , h
i +1 , ..., h i + j .
The convolution operation involves a ﬁlter w ∈ R pl , which is applied to a window of p hidden state vectors to generate a new feature.
For instance, a feature c i is generated from a window of hidden state vectors
h
i : i + p − 1 calculated as follows:
[Equation]
Here b ∈ R is the bias term and f is a non-linear function such as the rectiﬁed linear unit (ReLU).
This ﬁlter is applied to each possible window of hidden state vectors in the combined sequence h 1: p , h 2: p +1 , . . .
, h m − p
+1: n to produce a feature map c ∈ R m − p +1 given by:
[Equation]
Max-pooling is applied over the feature map to take the maximum value ˆ c = max { c } as the feature corresponding to a particular ﬁlter.
The use of multiple ﬁlters facilitates selection of the most important feature (one with the highest value) for each feature map.
This helps in capturing different semantics within the sentence.
Finally, the use of multiple ﬁlters with varying window sizes result in a ﬁxed length vector g ∈ R fw , where f is the number of ﬁlters and w is the number of different window sizes.
3.4 Predicting n -ary Relations
The task of predicting n -ary relations is modeled both as a binary and multi-class classiﬁcation problem.
The output feature vector g obtained from the convolution and max-pooling operation is passed to a softmax layer, to obtain the probability distribution over relation labels.
Dropout [Srivastava et al., 2014] is used on the output layer to prevent over-ﬁtting.
Thus, given a set of instances, with each instance being a text span S i comprising t consecutive sentences (combined sequence of tokens x = x 1 , x 2 , ...x m ), entity mentions e 1 , ..., e n and having an n -ary relation r , the cross-entropy loss for this prediction is deﬁned as follows:
[Equation]
where s indicates the total number of text spans and θ indicates the parameters of the model.
3.5 Implementation details
The proposed model was implemented using Tensorﬂow [Abadi et al., 2016] and will be made publicly available upon paper acceptance.
The hyper-parameters of the models were set based on preliminary experiments using an independent development dataset.
Training was performed following mini-batch gradient descent (SGD) with a batch size of 10.
The models were trained for at most 30 epochs, which was sufﬁcient to converge.
The size of the hidden vectors for the LSTM was set to 300.
The window sizes for the CNN was set to 3,4 and 5, and experiments were conducted with different number of ﬁlters set to 10 and 128.
Word embeddings were initialised using publicly available 300-dimensional Glove word vectors trained on a 6 billion token corpus from Wikipedia and web text [Pennington et al., 2014].
The dimension for position embeddings was set to 100 and was initialised randomly between [-0.25, 0.25].
4. Experiments 4.1 Datasets
We conduct experiments using the following datasets.
4.1.1 Q UIRK AND P OON ( QP ) D ATASET
We use the dataset 2 developed by Quirk and Poon 2016 and Peng et al.
2017 for the task of crosssentence n -ary relation extraction.
Distant supervision was adopted to extract relations involving drug, gene and mutation triples from the biomedical literature available in PubMed Central 3 .
The idea of minimal span [Quirk and Poon, 2016] was used to avoid co-occurrence of the same entity triples and also to obtain spans with ≤ 3 consecutive sentences to avoid candidates where triples are far apart in the span.
A total of 59 drug-gene-mutation triples was used to obtain 3,462 ternary relation instances and 3,192 binary relation instances (involving drug-mutation entities) as positive examples.
The dataset has instances with ternary and binary relations, either appearing in a single sentence or across sentences.
Each instances is labeled using four labels: ‘resistance’, ‘resistance or non-response’, ‘response’, and ‘sensitivity’.
The label ‘none’ is used for negative instances.
Negative samples were generated by randomly sampling co-occurring entity triples without known interactions, following the same restrictions used for obtaining positive samples.
Negative examples were sampled to match the same number of positive samples to develop a balanced dataset.
4.1.2 C HEMICAL I NDUCED D ISEASE ( CID ) D ATASET
We also evaluated the proposed model using the CID dataset 4 , which provides binary relation instances between chemicals and related diseases.
We followed the methodology of Gu et al.to obtain relation instances from the corpus.
Accordingly, a total of 1206, 1999 and 1330 positive instances were obtained for binary relations in single sentences and a total of 702, 788 and 786 positive instances were binary relations across sentences, respectively.
Negative instances were created following the same restrictions, however without any known interactions between entities.
4.2 Evaluation Metrics
We conduct ﬁve-fold cross-validation and report average test accuracy on held-out folds experiments using the Q & P DATASET , as prior work reported in Peng et al.
2017 adopted similar evaluation approach.
To avoid training and test contamination, held-out evaluation was conducted in each fold, based on categorizing instances related to speciﬁc entity pairs (binary relations) or entity triples (ternary relations).
For example, for binary relations, the instances relating to the ﬁrst 70% of the entity pairs drawn from a unique list of entity pairs was used as the training set.
Instances relating to the next 10% and last 20% were used as development set and test set, respectively.
For CID
DATASET , the Precision, Recall and F-score on the test set is reported, since the corpus is already divided in training, development and test sets and also for comparison, as previous studies [Gu et al., 2016, 2017, Zhou et al., 2016] used a similar evaluation strategy.
4.3 Baseline models
The proposed LSTM CNN + WF + PF model was evaluated against the following baseline models: (a)
CNN + WF : a CNN model using word features alone; (b) CNN + WF + PF : a CNN model using word features and positional features; (c) LSTM + WF : an LSTM model using word features alone; (d)
LSTM + WF + PF : an LSTM model using word features and positional features; (e) CNN LSTM + WF : a model that begins with a CNN layer followed by an LSTM layer and uses word features only; (f) CNN LSTM + WF + PF : model that begins with a CNN layer followed by LSTM layer and employs word features and position features; (g) LSTM CNN + WF : model that begins with an LSTM layer followed by a CNN layer and employs word features only.
4.4 Results and Discussion 4.4.1 P ERFORMANCE OF THE PROPOSED MODEL .
The performance of the proposed LSTM CNN + WF + PF model for cross-sentence n -ary relation extraction on the Q & P DATASET is shown in Tables 1 and 2.
As seen from Tables 1 and 2, the
LSTM CNN + WF + PF model achieves statistically signiﬁcant accuracy ( p ≤ 0 .
Friedman Test) against all baseline models such as CNN + WF , CNN + WF + PF , LSTM + WF , LSTM + WF + PF ,
CNN LSTM + WF , CNN LSTM + WF + PF and LSTM CNN + WF , for both cross-sentence ternary and binary relation extraction.
The results for the performance of the combined LSTM CNN model are better than CNN and LSTM models when used in isolation, indicating the usefulness of such combined models for relation extraction.
Combining LSTM and CNN helps in bringing together the strength of LSTM s to learn from long sequences (input sequences) and the ability of CNN s to identify salient features from the hidden-state output sequences from LSTM for cross-sentence n -ary relation extraction.
Given the above results, it is interesting that a combined model of LSTM and CNN using word features ( WF ) and positional features ( PF ), outperformed the baseline models.
Interestingly, the use of WF alone already helps the combined model ( LSTM CNN ) in achieving higher performance against other baselines, particularly for extracting binary relations in single sentences and across sentences, and also ternary relations in single sentences (Tables 1 and 2 with nf = 128 ).
However, it is the addition of PF that helps to drastically improving the performance for relation extraction.
The PF clearly helps the combined LSTM CNN model by providing a useful encoding of the position of words w.r.t entities in the text span and thus providing for a higher accuracy.
Further, the higher performance achieved in extracting both ternary and binary relations, particularly from cross-sentence text spans which are longer in sequence, indicates that the LSTM CNN + WF + PF model is highly suitable for extracting relations from longer sequences.
Furthermore, the LSTM CNN + WF + PF model’s superior performance for extracting ternary and binary relations from single sentences also indicates the suitability of the LSTM CNN + WF + PF model for relation extraction in single sentences.
4.4.2 W HERE EXACTLY DOES LSTM CNN MODEL SCORE ?
To assess the contribution of LSTM CNN + WF + PF against the baseline models, we divided each dataset into three groups based on the distance between entity e 1 and e n in the text span.
Speciﬁcally, we calculated the average number of tokens ( µ ) between e 1 and e n and the standard deviation ( σ ) over different lengths of tokens between e 1 and e n in the dataset.
Thus, if k is the total number of tokens between e 1 and e n , the dataset was divided into the following three groups: (a) shortdistance spans ( k ≤ µ − σ ); (b) medium-distance spans ( µ − σ < k < µ + σ ); (c) long-distance spans ( k ≥ µ + σ ).
Analysing the performance of models on different groups of spans divided in the above manner would provide insights into the model’s performance on different sequence lengths and the contribution of different features for relation extraction.
The performance of the various models on three groups of sentences, divided according to the number of tokens between entities e 1 and e n in the text span is provided in Table 3.
As can be seen from the Table, the proposed LSTM CNN + WF + PF model scores higher, particularly for mediumdistance spans ( µ − σ < k < µ + σ ) and long-distance spans ( k ≥ µ + σ ).
For short-distance and long-distance spans involving ternary relations across sentences, the LSTM CNN + WF + PF model predicts ternary relations correctly for 81 .
3 % and 82 .
9 % spans, respectively.
Similarly, the percent- single cross sentence sentences nf = nf = nf = nf = CNN + WF 72. CNN + WF + PF 73. LSTM + WF † - 75.
LSTM + WF + PF † - 74.
CNN LSTM + WF 77.
CNN LSTM + WF + PF 72.
LSTM CNN + WF 78.
LSTM CNN + WF + PF 73.
single cross
[Table]
sentences nf = nf = nf = nf =
CNN + WF 68.
CNN + WF + PF 74.
LSTM + WF † - 75.
LSTM + WF + PF † - 74.
CNN LSTM + WF 71.
CNN LSTM + WF + PF 74.
LSTM CNN + WF 74. LSTM CNN + WF + PF 85. age of correct predictions for binary relation extraction in single sentences and across sentences is signiﬁcantly higher than the performance of other models. These results clearly indicate that the combined LSTM CNN + WF + PF model is more useful compared to using the CNN and LSTM models in isolation for cross-sentence n -ary relation extraction, particularly where the distance between the ﬁrst ( e 1 ) and the last entity ( e 2 ) is large.
In other words the combined LSTM CNN models are more useful in extracting relations from larger spans of consecutive sentences.
Further, the highest margin between LSTM CNN + WF + PF and the baselines is recorded for binary interactions in single sentences and across sentences with an accuracy of 85.8 and 88.6, respectively (Table 2).
This is followed by ternary interactions in single sentences and across sentences with an accuracy of 79.6 and 82.9, respectively (Table 1).
It is interesting to note that the average length of tokens ( µ ) between entities in text spans in the datasets relating to binary and ternary interactions in single sentences and across sentences is of the order 19, 29, 34 and 44, respectively.
Based on these results, it can be broadly concluded that the contribution of PF decreases with the increase in the distance between entities in the text span.
4.4.3 LSTM CNN VS . CNN LSTM .
The results shown above clearly indicate that it is more useful to start with an LSTM layer followed by CNN layer ( LSTM CNN model) than having a CNN LSTM model for cross-sentence n -ary relation extraction.
As seen from Tables 1 and 2, the LSTM CNN models perform signiﬁcantly higher than
CNN LSTM models both for ternary and binary relations in single sentences and across sentences.
A
LSTM CNN model is more useful in that, it initially learns from the sequential information available in the input, which is further exploited by the CNN max-pooling layer to identify salient features.
However, in the CNN LSTM model, although the use of a CNN layer with max-pooling as the ﬁst component helps in identifying salient features from the input, the output from the CNN layer does not retain the sequential information.
The CNN output feature vector without sequential information when fed to LSTM layer, results in poor performance.
This indicates that an LSTM CNN model is more useful than CNN LSTM model for cross-sentence n -ary relation extraction.
Further, as the results show, the addition of position embeddings in the CNN LSTM model ( CNN LSTM + WF + PF ) results in poor performance in comparison to the use of word embeddings alone ( CNN LSTM + WF ).
This is particularly true for ternary relation extraction (Table 1).
Further as seen in Table 1, the use of a higher number of ﬁlters combining word embeddings and position embeddings, dramatically lowers the performance.
This indicates that position embeddings along with a higher number of ﬁlters are not useful for CNN LSTM models.
However, it is worth noting that, as seen from Table 3, the CNN LSTM + WF + PF model extracts ternary relations in single sentences for the higher number of long-distance spans (88.8%), indicating that CNN LSTM models are useful in certain cases.
4.4.4 CNN AND LSTM MODELS .
The results provided above clearly show that, when used in isolation, LSTM -based models are more useful for cross-sentence n -ary relation extraction, compared to CNN -based models.
Interestingly, the use of PF helps only longer sequences (accuracy of 78.9 ( LSTM + WF + PF ) vs. 78.2 ( LSTM + WF ) and 80.8 LSTM + WF + PF ) vs. 80.3 ( LSTM + WF + PF ) scored for ternary relations in drug-mutationgene (Table 1) and drug-mutation (Table 2), respectively).
However, for shorter sequences, the use of PF results in a decrease in accuracy (accuracy of 74.5 ( LSTM + WF + PF ) vs. 75.0 ( LSTM + WF ) and 74.4 LSTM + WF + PF ) vs. 75.4 ( LSTM + WF + PF ) scored for binary relations in drug-mutation-gene (Table 1) and drug-mutation (Table 2), respectively).
The contribution of WF in the CNN model signiﬁcantly improves with the use of higher number of ﬁlters, so much so that the model performs better than combining WF and PF .
This is particularly true for extracting ternary relations in single sentences (Table 1).
4.4.5 n - POSITIONAL EMBEDDINGS .
Given entities e 1 , ..e n in the text span, the proposed LSTM CNN + WF + PF model employed only eand e n to create positional embeddings.
However, we could also create n -positional embeddings for each of the n entities in the text span.
To this end, we evaluated the LSTM CNN + WF + PF model using n -positional embeddings.
The use of n -positional embeddings resulted in a lower accuracy of 80.
[Table] and 77.9 (compared to 82.5 and 79.6 using position embeddings for e 1 and e n ) for ternary relation extraction across sentence and single sentences, respectively.
This indicates that using positional embeddings for e 1 and e n is more useful for cross-sentence relation extraction.
4.4.6 C OMPARISON AGAINST THE STATE - OF - THE - ART .
We compare in this section, the performance of LSTM CNNW - WF + PF model against SOTA methods for cross-sentence n -ary relation extraction using the Q & P DATASET and CID DATASET .
Q & P DATASET .
The performance of LSTM CNNW - WF + PF against different methods using
Q & P DATASET is provided in Table 4.
As can be seen in the Table, the LSTM CNNW - WF + PF clearly achieves a signiﬁcantly higher performance against all compared SOTA methods particularly for binary (involving drug-mutation relations) cross-sentence relation extraction.
However, for ternary (involving drug-gene-mutation relations) cross-sentence relation extraction, the LSTM CNNW - WF + PF achieves a comparable performance against SOTA methods.
More speciﬁcally, the proposed model clearly outperform feature-based models [Quirk and
Poon, 2016], RNN-based networks such as BILSTM [Miwa and Bansal, 2016] and TREE - LSTM for both binary and ternary relations, indicating the superiority of the LSTM CNNW - WF + PF model.
The superior performance of the combined CNN and LSTM against the performance of models used in isolation ( BILSTM and TREE - LSTM ), clearly indicates that the combination of CNN and LSTM is useful for improving performance.
Further, the improved performance of LSTM CNNW - WF + PF model against SOTA graph-LSTM
[Peng et al., 2017] indicates that the proposed model is more useful, particularly in overcoming problems arising with graph-based models, which includes difﬁculties in connecting multiple sentences due to parsing errors and absence of an established mechanism to connect different parse trees.
This is also true for graph-state LSTMs [Song et al., 2018] for binary cross-sentence relation extraction.
With regard to cross-sentence ternary relation extraction, the proposed model achieves a comparable performance with SOTA methods.
The graph-state LSTM model [Song et al., 2018] achieves a slightly higher performance against the proposed method.
Given the comparison of the proposed model against SOTA, the strength of the proposed model comes from the fact that previous SOTA methods heavily rely on syntactic features such as dependency tress, co-reference and discourse features, which are time-consuming and less accurate particularly in the biomedical domain.
However, in contrast, the proposed LSTM CNN + WF + PF model does not use any such sophisticated features, but employs much simpler features such as
WF and PF .
The ability to provide signiﬁcantly higher performance with much simpler features make the proposed LSTM CNN + WF + PF model an attractive choice for cross-sentence n -ary relation extraction.
CID DATASET .
The performance of the LSTM CNN + WF + PF model on the CID DATASET is provided in Table 5.
From the Table, it can be seen that the LSTM CNN + WF + PF model achieves a statistically signiﬁcant performance for extracting binary relations from text spans with two sentences ( t = 2 ) against methods based on supervised learning using linguistic features and maximum entropy models.
The LSTM CNN + WF + PF model also performs well when extracting binary relations in single sentences ( t = 2 ).
The combined LSTM CNN + WF + PF model achieves a higher F-score (0.63) compared with various SOTA methods 5 on the CID DATASET as shown in Table 5.
The combination of LSTM CNN provides a slight increase over using CNN and LSTM separately on the CID
DATASET .
The CNN -based models proposed by Nguyen and Verspoor 2018, although achieve a high recall, tend to score a lower precision.
The same is the case with CNN + ME + PP
[Gu et al., 2017] and
CNN [Zhou et al., 2016].
On the other hand, LSTM s achieve higher precision, but suffer from poor recall ( LSTM , LSTM + SVMP
[Zhou et al., 2016]).
In comparison to CNN models and LSTM models, [Table] the combined LSTM CNN achieves a higher precision and at the same time does not lose on recall, resulting in a higher F-score using the CID DATASET .
5. Conclusion
To conclude, we presented in this paper a combined LSTM CNN + WF + PF model that exploits both word embeddings and position embeddings for the task of cross-sentence n -ary relation extraction.
The evaluation of the proposed model against various baseline models presented in the paper clearly establish that combining LSTM s and CNN s are quite useful for cross-sentence n -ary relation extraction.
The strength of the combined model comes from bringing together the ability of LSTM s to learn from longer sequences and the usefulness of CNN s to learn salient features, which is very important for cross-sentence n -ary relation extraction, as it mainly involves handling longer sequences.
The evaluation of the proposed model against SOTA methods for cross-sentence n -ary relation extrac- Model
P R F
Single sentences (text span where t =1 )
[Table]
[Gu et al., 2016] 0.67 0.68 0.
CNN [Gu et al., 2017] 0.59 0.55 0. LSTM CNN + WF + PF (proposed model) 0.69 0.70 0. Across sentences (text span where t =2 ) LINGUISTIC
[Table]
[Gu et al., 2016] 0.51 0.29 0.
MAXIMUM ENTROPY [Gu et al., 2017] 0.51 0.07 0.
LSTM CNN + WF + PF (proposed model) 0.57 0.57 0.57* Across sentences (text span where t ≤ 2 ) [Table] + ME
[Gu et al., 2016] 0.62 0.55 0.
CNN + ME
[Gu et al., 2017] 0.60 0.59 0.
CNN + ME + PP
[Gu et al., 2017] 0.55 0.68 0.
CNN [Zhou et al., 2016] 0.41 0.55 0.
LSTM
[Zhou et al., 2016] 0.54 0.51 0.
LSTM + SVMP
[Zhou et al., 2016] 0.64 0.49 0.
LSTM + SVM + PP
[Zhou et al., 2016] 0.55 0.68 0.
SVM [Xu et al., 2016] 0.55 0.68 0.
CNN 0.54 0.69 0.
CNN + CNNCHAR 0.57 0.68 0.
CNN + LSTMCHAR [Nguyen and Verspoor, 2018] 0.56 0.68 0. LSTM CNN + WF + PF (proposed model) 0.63 0.63 0. tion, clearly demonstrate the superiority of the proposed model, particularly in avoiding reliance on heavy syntactic features such as dependency, co-reference and discourse relations.