Abstractiﬁed Multi-instance Learning (AMIL) for Biomedical Relation Extraction
Abstract
Relation extraction in the biomedical domain is a challenging task due to a lack of labeled data and a long-tail distribution of fact triples.
Many works leverage distant supervision which automatically generates labeled data by pairing a knowledge graph with raw textual data.
Distant supervision produces noisy labels and requires additional techniques, such as multi-instance learning (MIL), to denoise the training signal.
However, MIL requires multiple instances of data and struggles with very long-tail datasets such as those found in the biomedical domain.
In this work, we propose a novel reformulation of MIL for biomedical relation extraction that abstractiﬁes biomedical entities into their corresponding semantic types.
By grouping entities by types, we are better able to take advantage of the beneﬁts of MIL and further denoise the training signal.
We show this reformulation, which we refer to as abstractiﬁed multi-instance learning (AMIL), improves performance in biomedical relationship extraction.
We also propose a novel relationship embedding architecture that further improves model performance.
1. Introduction
Relation extraction (RE) is a key facet of information extraction in large bodies of unstructured textual data.
RE is particularly important in the biomedical domain where extracting relationships between pairs of biomedical entities, also known as “fact triples”, can produce new insights into complicated biological interactions.
For instance, with the near-exponential growth of microbiome research [Sa’ed et al., 2019], advanced RE methods may help discover important links between gut microbiota and diseases.
It is in this context that we motivate our work.
RE within the biomedical domain comes with two inherent challenges: there are more thanmillion scientiﬁc articles, with hundreds of thousands of articles published every year, and there is a corresponding lack of labeled data.
To resolve these challenges, many have leveraged distant supervision techniques which pair knowledge graphs with raw textual data to automatically generate labels to train deep-learning models [Gu et al., 2019, Su et al., 2019, Junge and Jensen, 2019].
We seek to improve distantly supervised biomedical RE methods in this work.
We use the Uniﬁed
Medical Language System (UMLS) Metathesaurus [Bodenreider, 2004] for our knowledge graph and pair it with raw textual data from PubMed [Canese and Weis, 2013].
To automatically generate labels, distantly supervised RE methods rely on a simple yet powerful assumption: any singular sentence that contains a pair of entities also expresses a relationship, as determined by the accompanying knowledge graph, between those entities [Mintz et al., 2009].
However, this assumption leads to a noisy training signal with many false positives as not all sentences express a relationship between an entity pair.
To combat this, many works have leveraged multiinstance learning (MIL)
[Riedel et al., 2010, Hoﬀmann et al., 2011, Zeng et al., 2015] where, instead of assessing single sentences, MIL assesses positive and negative bags of sentences that contain the same entity pair.
Grouping sentences into bags greatly reduces noise in the training signal since a bag of sentences is more likely to express a relationship than a single sentence.
This enables the model to better classify relationships between unseen entity pairs.
However, similar to many NLP tasks, biomedical RE suﬀers from a long-tail distribution of fact triples, where many entity pairs are only supported by a few sentences of evidence.
After processing the PubMed corpus, we observe that a majority ( ∼ 52%) of extracted triples are supported by fewer than three sentences.
Creating bags of sentences for such entity pairs requires heavy up-sampling.
For example, if a pair of entities is only supported by one sentence and a bag size is equal tosentences, the single sentence is duplicated 15 times to ﬁll the bag.
This erases the beneﬁt of MIL.
To counter this issue, we introduce abstractiﬁed multi-instance learning (AMIL) where, instead of grouping entity pairs by name, we group entities by the corresponding semantic type as determined by UMLS.
UMLS categorizes each entity with a semantic type within the UMLS semantic network.
The UMLS semantic network is curated by human experts for decades and provides a rich ontology of biomedical concepts which we leverage to group multiple diﬀerent entity pairs within a single MIL bag, reducing the need to up-sample sentences.
For example, consider two sentences: (1) a sentence containing the entity pair ( ﬁbula , tibia ) and (2) a second sentence containing the entity pair ( humerus , ulna ).
With distant supervision, we assume each sentence expresses the relationship linking both pairs, namely articulates with .
Despite expressing the same relationship, without abstraction, these sentences are placed into separate MIL bags since bags are grouped by distinct entity pairs.
By introducing abstractiﬁed multi-instance learning, the entities ﬁbula , tibia , humerus , and ulna are grouped by their corresponding UMLS semantic type—“ Body Part, Organ, or Organ Component .”
This allows us to place the aforementioned sentences into the same MIL bag based on their entity type, creating a heterogeneous bag of entity pairs that express the same relationship.
With this reformulation, bags containing a single duplicated sentence are reduced by half.
AMIL produces better overall performance for biomedical RE with signiﬁcant performance gains for “rare” triples.
Here, we deﬁne “rare” triples as triples that are supported by fewer than eight sentences.
These triples make up roughly 80% of the long-tail distribution of triples.
We also take inspiration from Soares et al.(2019) and conduct a suite of experiments with variations of relationship embedding architectures.
Such experiments are underexplored in the biomedical domain and many are novel to the general task of relationship classiﬁcation.
Soares et al.
report the best RE performance using a relationship representation consisting of embedded entity start markers—special span tokens that denote the beginning of an entity.
We test this RE architecture in the biomedical domain and also test the performance of entity end markers.
Moreover, we introduce a novel relationship representation, namely the middle mention pool, which pools word pieces between head and tail entities.
This embedding architecture is inspired by the observation that context between two biomedical entities in a sentence often contains the information-rich and relationship-relevant signal.
Our best performing relationship embedding architecture results from the combination of both entity end markers and the middle mention pool.
We observe that this architecture further increases the performance of our relation classiﬁcation model.
In this paper, we make the following contributions:
• We introduce abstractiﬁed multiple-instance learning (AMIL), which achieves new state-of-theart performance for biomedical relationship extraction.
We also report signiﬁcant performance gains for rare fact triples.
• We propose an improved relationship representation for biomedical relation extraction.
We show that concatenating embedding tokens from entity end markers with the middle mention pool produces the best performing model.
• We make all our code, saved models, and pre-processing scripts publicly available 1 to facilitate future biomedical RE eﬀorts.
Pre-processing scripts can impact model performance and are important to prepare an up-to-date, ready-for-RE dataset from ever-growing PubMed and UMLS.
Our results in Section 5 show that using updated pre-processing tools can improve model performance by ∼ 10%.
2. Related Work
Early works combining distant supervision with relation extraction [Bunescu and Mooney, 2007,
Craven and Kumlien, 1999] relied on the strong assumption claiming that, if a relationship between two entities exists, then all sentences containing those entities express the corresponding relationship.
This assumption was relaxed by Riedel et al.
with the introduction of multi-instance learning (MIL) which claimed that if a relationship exists between two entities, at least one sentence that contains the two entities may express the corresponding relation.
Hoﬀmann et al.
(2011) build on the work of Riedel et al.
by allowing for overlapping relations.
Zeng et al.
(2015) extend distantly supervised RE by combining MIL with a novel piecewise convolutional neural network (PCNN).
Lin et al.
(2016) made further improvements by introducing an attention mechanism that attends to relevant information in a bag of sentences.
This sentence-level attention mechanism for MIL inspired numerous subsequent works [Luo et al., 2017, Han et al., 2018a, Alt et al., 2019].
Han et al.
(2018a) propose a joint training RE model that combines a knowledge graph with an attention mechanism and MIL.
Dai et al.
(2019) extend the work by Han et al.
into the biomedical domain and use a PCNN for sentence encoding.
Amin et al.(2020) propose an RE model that uses BioBERT [Lee et al., 2019], a pretrained transformer based on bert [Devlin et al., 2019], for sentence encoding.
They leverage MIL with entity-marking methods following R-BERT [Wu and He, 2019] and achieve the best performance when the directionality of extracted triples is matched to the directionality from the UMLS knowledge graph.
Notably, the model proposed by Amin et al.(2020) does not beneﬁt from sentence-level attention.
We choose the model proposed by Amin et al.(2020) for our baseline model as it achieves the current state-of-the-art (SOTA) in biomedical RE.
We also conduct a suite of experiments with variations of relationship embedding architectures.
These experiments are inspired by Soares et al.
(2019) who conduct experiments with six diﬀerent embedding architectures and report performance on general-domain RE datasets.
They show that constructing a relationship embedding with special entity start markers outperforms other architectures.
We build on this work by (1) conducting similar experiments in the biomedical domain and (2) by proposing numerous novel architectures for a total of seventeen alternatives for a comprehensive comparison on the biomedical RE task.
3. Datasets
UMLS Metathesaurus and Semantic Network : The UMLS Metathesaurus and Semantic
Network is a knowledge graph of biomedical entities and their corresponding relationships.
Following numerous previous works in biomedical relation extraction [Zhang and Wang, 2015, Dai et al., 2019,
Amin et al., 2020], we only extract fact triples that contain a relationship other than “synonymous”, “narrower”, or “broader”.
These general relationships make up the majority of relationships in the
UMLS Semantic Network and we exclude them to focus on more substantive relationships.
Using this ﬁlter, we extract 7,025,733 triples from the 2019AB UMLS release.
PubMed : For our textual data, we use abstracts from the 2019 PubMed corpus 2 .
The corpus contains 34.4M abstracts which we segment into 158,848,048 unique sentences using Sci-Spacy [Neumann et al., 2019]. 4. Method
Problem statement: given knowledge graph G and text corpus C , sentences s from C that contain exactly two distinct entities, ( e i 1 , e i 2 ), that are linked via relationship r
i as determined by G , are grouped into bags B i = (cid:8) s i 1 , . . . , s im (cid:9) based on their corresponding entity types ( e i 1 − Type , e i 2 − Type ) where e i 1 − Type , e i 2 − Type ∈ E T and E T represents the set of all UMLS semantic types. Our goal is to predict the relationship r i that is expressed in each bag
B i , forming a fact triple T
= { ( e i 1 , r i , e i 2 ) } .
For simplicity, indices for bags, sentences, and entities are omitted if not required for clarity.
4.1 Pre-processing
Although static benchmark test sets are typically available for general-domain RE tasks (e.g., [Zhang et al., 2017, Han et al., 2018b]), there are no such test sets for biomedical RE.
Biomedical RE relies on large datasets that are constantly updated (e.g. PubMed and UMLS) and, part of the task involves developing a pre-processing pipeline.
To best compare the performance of our approach, we model our pre-processing steps after those used by Amin et al.
Entities within sentences are found using the UMLS Metathesaurus which contains every UMLS concept and their corresponding surface form variations.
Sentences are retained and considered “positive” examples if they meet the following criteria: (1) they contain exactly two distinct entities and (2) those entities are linked by a relationship in the UMLS knowledge graph.
Each sentence is grouped by its corresponding fact triple T = { ( e 1 , r, e 2 ) } , where e 1 , e 2 ∈ E and E represents the set of UMLS entities and r ∈ R where R is the set of UMLS relation types.
The UMLS knowledge graph provides directionality for relationships (i.e. directed edges) and that directionality preserved in the fact triples extracted from sentences regardless of the order entities appear in a sentence.
Amin et al.
[2020] show that preserving directionality further denoises the training signal and leads to better predictive performance.
Negative examples are generated by randomly replacing either a head or tail entity within a positive sentence such that the newly formed entity pair is not linked by a relationship in the UMLS knowledge graph.
The entity pairs extracted from negative sentences are assigned the negative relationship label “NA” to form negative triples.
Triples from the negative class are chosen randomly and the size of the negative class is set to 70% of the largest positive relationship class.
Span markers denoting the start and end of entity spans are then inserted into each sentence.
Head entities ( e 1 ) are marked with ‘ ∧ ’ and tail entities ( e 2 ) are marked with ‘$’.
Soares et al.
show that bert achieves the best sentence-level relationship extraction performance when entity spans are denoted with special start and end tokens.
We construct random train/dev/test splits based on extracted triples and ensure no triples or sentences overlap between the splits.
We use 20% of the data for a test set.
With the remaining 80% of data, 10% is used for a development set and the rest is used for training.
These steps mirror those used by Amin et al.(2020) but our splits contain diﬀerent sets of triples and sentences.
To ensure fair comparison, we trained the Amin et al.(2020) model, AMIL, and all AMIL variations on identical data from our randomized splits.
Lastly, to train AMIL, entities are abstracted using their corresponding entity-types (immediate hypernyms), which are determined using the UMLS Semantic Network, and grouped into more general entity-type triples, { ( e 1 − Type , r, e 2 − Type ) } .
Bags of sentences are then formed using the abstracted entity types and sentences are randomly up-sampled to ﬁll any bags that fall short of the set bag size of 16 sentences.
An example of an abstractiﬁed bag is provided in Figure 1(b).
[Table]
4.2 Training
We use a pretrained transformer, namely BioBERT [Lee et al., 2019], to produce low-dimensional relationship embeddings with the following hyper-parameters:
• Transformer Architecture : 12 layers, 12 attention heads, 786 hidden size • Weight Initialization : BioBERT • Activation : GELU [Hendrycks and Gimpel, 2016] • Learning Rate : 2e-5 with Adam • Batch Size :• Max sequence length:• Total Parameters: 110M
Each bag of sentences containing entity markers is passed through the transformer to obtain an encoded sentence.
For our baseline AMIL model, we match our relationship representation architecture to that used by Amin et al.
We ﬁrst condense each encoded head and tail entity via average pooling, where ( j, k ) is the span containing the head entity e 1, ( l, m ) is the span containing the tail entity e 2, and an encoded sentence of length n is denoted as [( h 0 , ..., h n )]: h e1 = 1 + k − j k (cid:88) i = j h i h e2 = 1 + m − l m (cid:88) i
= l h i
Pooled entities are then concatenated with the [CLS] embedding to form the relationship representations for each sentence in the bag r = (cid:104)
h
CLS | h e 1 | h e 2 (
cid:105) ∈ R 3 d , where (cid:104) x
| y (cid:105) denotes the concatenation of x and y .
The representations are then aggregated via average pooling and sent through a tanh activation, a dropout layer, and, ﬁnally, a fully connected (2304 × 2304) linear layer.
We use cross-entropy to compute the loss and train the model over 300 epochs with early-stopping on the best F1-score from the development set.
All models were trained on an NVIDIA Tesla V100 and completed with an average training time of 7 hours 59 minutes.
4.3 Relationship Representations
We present a suite of experiments with various relationship embedding architectures.
We draw inspiration from Soares et al.
and conduct 17 experiments to empirically determine the most eﬀective relationship representation architecture for the task of biomedical relationship classiﬁcation.
We expand on their experiments with 12 novel relationship embedding architectures (types ‘F’ through ‘Q’) which we describe in the following section.
The following relation embedding experiments are grouped into pairs that feature the same relationship embedding with and without the reserved [CLS] token from bert .
In all experiments, [Figure] the models are trained using the hyper-parameters and methods described in Section 4.2 (hiddendimension d = 786).
All experiments that involve pooling are conducted using average pooling.
Figure 1(a) provides a visual representation of each relation embedding architecture and Figure 1(b) illustrates the ﬂow of data using AMIL and an abstractiﬁed bag of sentences.
4.3.1 Types of Relationship Representations
A — [CLS] Token : The [CLS] token from bert acts as a representation of the entire input sequence and, thus, serves as a baseline for our experiments with various relationship representation architectures.
[Equation]
B, C — Entity Mention Pool : For this architecture, each entity’s word pieces are pooled via average pooling and then concatenated together.
Architecture type ‘C’, which contains the entity mention pool concatenated [CLS] token, is the architecture used by both our baseline AMIL model and the model proposed by Amin et al.(2020)
It achieves the current SOTA in biomedical relation classiﬁcation.
[Equation]
D, E — Entity Start Markers : Soares et al. report the best relation classiﬁcation performance using concatenated entity start markers.
We recreate this top-performing architecture to test its performance in the biomedical domain.
[Equation]
F, G — Entity End Markers : We propose representing the relationship by using the entity end markers to determine if there is any beneﬁt over the entity start markers.
[Equation]
H, I — Entity Start and End Marker : For this experiment, we concatenate the representations for both the entity start markers and the entity end markers.
[Equation]
J, K — Middle Mention Pool : Here, we propose using the middle mention pool which is the pooled word pieces between the head and tail entities.
This embedding architecture is inspired by a pattern we observe in relationship-containing sentences where, often, the context between two entities in a sentence contains the most information-rich and relationship-relevant signal. [Equation] R
[Equation]
L, M — Middle Mention Pool and Entity End Markers : This architecture concatenates the middle mention pool with the entity end markers.
[Equation]
N, O — Middle Mention Pool and Entity Start and End Markers : We form this representation by concatenating entity start markers, entity end markers, and the middle mention pool.
This architecture results in the highest dimensional relationship representation and will help us determine if the added information is beneﬁcial to model performance.
[Equation]
P, Q — Complete Sequence Pool : We obtain the average of all the output tokens to form the complete sequence pool.
This is diﬀerent from the [CLS] token, which is a learned representation of a sequence, in that it averages all the encoded word piece tokens in a sequence.
All other relationship representations consist of subsets of the output tokens.
By including all tokens, this representation acts as a type of baseline experiment that will allow us to validate or invalidate the use of subsets in other architectures.
[Equation]
4.4 Evaluation
All evaluations between AMIL and Amin et al.(2020) are conducted using identical sets of triples and sentences.
To properly evaluate AMIL, we ﬁrst de-abstract the triples in a bag of sentences and evaluate performance on the original set of triples.
Our pre-processing steps vary from Amin et al.(2020) in that we segment sentences using Sci-Spacy [Neumann et al., 2019] instead of NLTK [Loper and
Bird, 2002].
Sci-Spacy is speciﬁcally tuned to process biomedical texts where as NLTK is tuned for general English.
Using Sci-Spacy, we observe a 30% reduction in extracted sentences due to fewer extracted sentence fragments.
We train the Amin et al.(2020) model using our improved preprocessing steps which results in a higher performance than reported in their original paper ( ∼ 10% increase in AUC).
We were unable to attain the code and data used by Dai et al.(2019). Ideally, we would have trained and tested the Dai et al.(2019) model with the same data we used for our other experiments. Since this was not an option, we provide the results of the Dai et al.(2019) model as reported by the authors.
Without access to data from their experiments, we believe a direct comparison is not fair; however, the precision @ k indicates the model’s overall ability to extract true triples from a hold-out set of triples found in a test corpus. Because we use similar data (e.g. the UMLS knowledge graph with raw text from PubMed abstracts), we believe this metric allows for a good, but not perfect, comparison.
Weevaluate our model using both corpus-level and sentence-level evaluation:
Corpus-level Evaluation : The benchmarks for biomedical RE are set using a corpus-level evaluation [Mintz et al., 2009] which evaluates model performance on a hold-out set of triples.
Using this method, we sort predictions of triples from the test set based on their softmax probability and compare the set of predicted triples to the ground truth triples contained in the test corpus.
We then report AUC, F1, and precision on the top K predictions.
Sentence-level Evaluation : To better understand model performance, we decompose triples into two groups: (1) “rare” triples and (2) “common” triples and conduct a sentence-level evaluation.
Extracted triples follow a heavy-tailed Pareto distribution.
Using the Pareto principle, we deﬁne rare triples as triples that make up the lower 80% of the long-tail distribution of extracted triples.
These are triples that are supported by seven or fewer sentences.
Common triples are deﬁned as triples that make up the upper 20% of the long-tail distribution of triples and are supported by eight or more sentences.
Sentence-level evaluation relies on standard precision, recall, and F1-score metrics.
It allows us to more holistically assess model performance since sentence-level metrics do not obscure performance on low-conﬁdence predictions.
A relationship predicted between an entity pair that matches the ground truth relationship, as determined by the UMLS knowledge graph, is a true positive.
A relationship predicted between an entity pair that is not linked in the UMLS knowledge graph is a false positive.
A false negative occurs when the model predicts “NA” and the ground truth is something other than “NA.”
Note, sentence-level evaluation is not a good estimate of overall model performance since, for this task, we are primarily interested in knowledge-graph completion.
We only include sentence-level evaluation to evaluate AMIL’s ability to predict rare triples.
Corpus-level evaluation is more appropriate for assessing overall model performance.
5. Results
Table 2 compares the performance of our baseline AMIL model to other distantly supervised biomedical relation extraction models.
We also include performance from our top-performing relationship embedding architecture—relationship representation architecture type ‘L’.
Here, we observe AMIL achieves signiﬁcantly higher scores for AUC and F1.
It also outperforms on each subset of precision.
AMIL using relationship representation architecture type ‘L’ makes additional gains in each metric.
Table 3 compares sentence-level performance on the rare and common subsets of triples.
We also include sentence-level performance on all triples for comparative purposes.
Again, AMIL outperforms
Amin et al.(2020) in all metrics.
AMIL with relation representation type ‘L’ attains an additional performance boost in all metrics but recall for common triples.
Table 4 compares the 17 variations of relationship embedding architectures described in Section 4.3 using a corpus-level evaluation.
We observe that relationship embedding architecture type ‘L’ outperforms other architectures in all metrics. [Table] [Table]
Sentence-level performance on rare triples (lower 80% of the long-tail distribution of triples) and common triples (upper 20% of the long-tail distribution).
Rare triples are supported by seven or fewer sentences and common triples are supported by eight or more sentences.
AMIL with relationship type ‘L’ is deﬁned by Equation (1) in Section 4.3.1.
6. Discussion
The large performance gains reported in Table 2 conﬁrm that abstractiﬁed multi-instance learning is successful in further denoising the training signal for distantly supervised relation extraction.
By grouping entities by entity type, we are able to better leverage the beneﬁts of multi-instance learning on long-tail datasets.
We hypothesize that AMIL as a denoising strategy will have the greatest impact on rare triples.
Rare triples and their corresponding sentences require the most up-sampling to ﬁll a bag of sentences and, thus, should receive a greater beneﬁt compared to common triples when grouped into larger entity-type bags.
The results in Table 3 conﬁrm this hypothesis.
Compared to Amin et al.(2020) which uses standard MIL, AMIL gains 10.5 F1 percentage points on rare triples compared to a 4.
point gain on common triples.
From table 4, the high performance of relationship representation types ‘L’ and ‘J’, both of which contain the middle mention pool, conﬁrms our hypothesis that the context between two entities
Relationship Representation
F
AUC
P@20k
A: [CLS] .
B: entity mention pool .
C:
[CLS] + entity mention pool .
D: e 1 Start + e 2 Start .
E: [CLS] + e 1 Start + e 2 Start .
F: e 1 End + e 2 End .
G: [CLS] + e 1 End + e 2 End .
H: e 1 Start + e 1 End
+ e 2 Start + e 2 End .
I: [CLS] + e 1 Start + e 1 End + e 2 Start + e 2 End .
J: middle mention p.
K: [CLS] + middle mention p.
L: e 1 End + middle mention pool + e 2 End .
M: [CLS] + e 1 End + middle mention p. + e 2 End .
N: e 1 Start + e 1 End + middle mention p. + e 1 End + e 2 End .
O:
[CLS] + e 1 Start + e 1 End + middle mention p. + e 1 End + e 2 End .
P: entire sequence avg .
Q: [CLS] + [Table] in a sentence provides an information-rich and relationship-relevant signal. Soares et report relationship representation type ‘D’, the entity start markers, as their top-performing as tested on general-domain data.
However, our tests in the biomedical domain show that this architecture fails to reach high performance compared to other architectures.
This points to the potential need for domain-speciﬁc relationship architectures. Comparing the performance of entity start markers ‘D’ with entity end markers , we see that the model beneﬁts from the of the end entity markers indicating that, although bert is bidirectional, the of embedded special markers informs the ability to relationships. Abstractified
We constructed pairs of experiments with and without the special [CLS] token from bert to determine the eﬀect of the [CLS] token on the model’s ability to predict relationships.
Interestingly, we observe mixed eﬀects.
There is an even split between the performance of architectures with and without a concatenated [CLS] token.
Experiment pairs (B, C), (H, I), (L, M), and (P, Q) beneﬁt from the [CLS] token while experiment pairs (D, E), (F, G), (J, K), and (N,O) are hindered by the [CLS] token.
In the experiment pairs where the [CLS] token was beneﬁcial, the average increase on the AUC is 0 .
On experiment pairs that resulted in hindered performance, the average decrease on the AUC is − 0 .
Lastly, the performance of the entire sequence average (representations P and Q) justiﬁes the need for subset representations.
The model performs best when an information-rich subset of tokens, such as the middle mention pool and/or the entity end markers, are used to construct a relationship representation.
7. Conclusion
In this work, we propose abstractiﬁed multi-instance (AMIL), a novel denoising method that increases the eﬃcacy of multi-instance learning in the biomedical domain.
With it, we improve performance on biomedical relationship extraction and report signiﬁcant performance gains on rare fact triples.
We also propose a novel relationship embedding architecture which further increases model performance.
Forfuture work, we will explore combining AMIL with more advanced bag aggregation methods.
We will also explore applying our novel relationship embedding architectures to relationship extraction tasks using general-domain datasets. Acknowledgments
Thank you to the anonymous reviewers for their thoughtful comments and corrections.
This work is supported by IBM Research AI through the AI Horizons Network.