DARE: Data Augmented Relation Extraction with GPTAnonymous ACL submission
Abstract
Real-world Relation Extraction (RE) tasks are challenging to deal with, either due to limited training data or class imbalance issues.
In this work, we present Data Augmented Relation Extraction (DARE), a simple method to augment training data by properly ﬁne-tuning GPT-2 to generate examples for speciﬁc relation types.
The generated training data is then used in combination with the gold dataset to train a BERT-based RE classiﬁer.
In a series of experiments we show the advantages of our method, which leads in improvements of up to 11 F1 score points against a strong baseline.
Also, DARE achieves new state of the art in three widely used biomedical RE datasets surpassing the previous best results by 4.7 F points on average.
Our code and data are available at https://github_to_be_added .
Introduction
Relation Extraction (RE) is the task of identifying semantic relations from text, for given entity mentions within it.
This task, along with Named
Entity Recognition, has recently become increasingly important due to the advent of knowledge graphs and their applications.
In this work, we focus on supervised RE (Zeng et al., 2014; Lin et al., 2016; Wu et al., 2017; Verga et al., 2018), where relation types come from a set of predeﬁned categories, as opposed to Open Information Extraction approaches that represent relations among entities using their surface forms (Banko et al., 2007;
Fader et al., 2011).
RE is inherently linked to Natural Language Understanding in the sense that a successful RE model should manage to adequately capture language structure and meaning.
So, almost inevitably, the latest advances in language modelling with
Transformer-based architectures (Radford et al., 2018a; Devlin et al., 2018; Radford et al., 2018b) have been quickly employed to also deal with RE tasks (Soares et al., 2019; Lin et al., 2019; Shi and
Lin, 2019; Papanikolaou et al., 2019).
These recent works have mainly leveraged the discriminative power of BERT-based models to improve upon the state of the art (SOTA).
In this work we take a step further and try to assess whether the text generating capabilities of another language model, {+GPT-
+}
(Radford et al., 2018b), can be applied to augment training data and successfully deal with class imbalance and small-sized training sets.
Speciﬁcally, given a RE task we ﬁne-tune one pretrained GPT-2 model per relation type and then use the resulting ﬁne-tuned models to generate new training samples.
We then combine the generated data with the gold dataset and ﬁne-tune a pretrained BERT model (Devlin et al., 2018) on the resulting dataset to perform RE.
We conduct extensive experiments, studying different conﬁgurations for our approach and compare
DARE against two strong baselines and the SOTA on three well established biomedical RE benchmark datasets.
The results show that our approach yields signiﬁcant improvements against the rest of the approaches.
To the best of our knowledge, this is the ﬁrst work augmenting training data with GPT-2 for RE.
In Table 1 we show some generated examples with GPT-2 models ﬁne-tuned on the datasets that are used in the experiments (refer to Section 4).
In the following, we provide a brief overview of related works in Section 2, we then describe our approach in Section 3, followed by our experimental results (Section 4) and the conclusions (Section 5).
Related Work
Relation Extraction is usually modelled as a text classiﬁcation task. Therefore most methods to deal with class imbalance or limited data in RE follow
Dataset(relation type)
Generated sentences
CDR(Induce)
DISEASE was the most common adverse reaction ( 21 % ) reported for DRUG, and occurred in approximately 50 % of patients .
DDI2013(Effect)
DRUGA may enhance the effects of alcohol, barbiturates, DRUGB, and other cns depressants.
DDI2013(Advise) caution should be observed when DRUGA and DRUGB are coadministered.
DDI2013(Mechanism) co-administration of DRUGA decreased the oral bioavailability (48%) of
DRUGB, a substrate for cyp2d6.
ChemProt(Activate)
DRUG enhances PROTEIN sensitivity via activation of the pi3k / akt signaling pathway.
ChemProt(Inhibit)
DRUG, a novel orally bioavailable xanthine PROTEIN inhibitor,
ChemProt(Product) the enzyme PROTEIN catalyzes the two-electron reduction of DRUG to produce acetyl groups.
the respective methods from text classiﬁcation.
In the following, we describe the different approaches that have been followed in the literature.
One approach is to deal with imbalance at the classiﬁer level, by penalizing misclassiﬁcation errors differently for each class, depending on the class frequency (Lewis et al., 2004; Zhou and Liu, 2005) or by explicitly adjusting prior class probabilities (Lawrence et al., 1998).
Another popular approach relies on either undersampling the majority class(es) or oversampling the minority one(s), transforming the training data with the aim of balancing it.
One of the simplest approaches, random majority undersampling, simply removes a random portion of examples from majority classes so that per class training examples are roughly equal (Japkowicz and Stephen, 2002).
An improved version of the previous method, balanced bagging (Hido et al., 2009), employs an ensemble of classiﬁers that have been trained with random majority undersampling.
Oversampling approaches for textual data have been somehow limited as opposed to those for image data (Wong et al., 2016; Fawzi et al., 2016;
Wang and Perez, 2017; Frid-Adar et al., 2018), since text semantics depend inherently on the exact order or structure of word tokens.
A simple approach is to replace words or phrases with their synonyms (Zhang et al., 2015).
Chen et al.
(2011) employed topic models to generate additional training examples by sampling from the topic-word and document-topic distributions.
Ratner et al.
(2016) proposed a data augmentation framework that employs transformation operations provided by domain experts, such as a word swap, to learn a sequence generation model.
Kaﬂe et al.
(2017) used both a template-based method and an LSTM-based approach to generate new samples for visual question answering.
A similar method to our approach was proposed by Sun et al. (2019a) who presented a framework to successfully deal with catastrophic forgetting in language lifelong learning (LLL).
Speciﬁcally and given a set of tasks in the framework of LLL, they ﬁne-tune GPT-2 to simultaneously learn to solve a task while generating training samples for it.
When dealing with a new task, the model is trained on the generated training samples from previous tasks alongside the data of the new task, therefore avoiding catastrophic forgetting.
Our work falls into the oversampling techniques for text, but our focus is RE.
Importantly, we do not need any domain expertise, templates, synonym thesaurus or to train a model from scratch, which makes our approach easily adaptable to any domain, with relatively low requirements in resources.
Methods
In this section we present brieﬂy the GPT-2 model and before giving a detailed introduction to our approach.
3. GPT-
GPT-2 (Radford et al., 2018b) is a successor of the
GPT language model (Radford et al., 2018a).
Both models are deep neural network architectures using the Transformer (Vaswani et al., 2017), pre-trained on vast amounts of textual data.
Both models are pre-trained with a standard language modelling objective, which is to predict the next word token given k previously seen word tokens.
This is achieved by maximizing the following likelihood:
[Equation]
where Θ are the neural network parameters.
The authors have gradually provided publicly four different ﬂavours of GPT-2, with 124M, 355M, 774M and 1558M parameters respectively.
In our experiments we use the second largest model (774M), since it seems to represent a good compromise between accuracy and hardware requirements 1 .
3. Data Augmented Relation Extraction
Let D =
[ s 0 , ...
s d ] be a RE dataset containing d sequences.
Furthermore, we assume that each sequence s =
[ w 0 , ...w n ] will be a sequence of n word tokens and that e 1 = [ w e 1 i , ...
w e 1 j ] and e 2 =
[ w e 2 k , ...
w e 2 l ] will represent a pair of entity mentions in s .
Furthermore, let L =
[ l 1 , ..., l c ] be a set of c relation types.
Then, RE is the task of learning a function that maps each triple ( s i , e 1 , e 2) to L , i.e.,
[Equation]
where Θ are the parameters of the model.
In this work we employ a RE classiﬁer based on a pretrained BERT language model.
This classiﬁer follows the same principle followed by Devlin et al.
(2018), using a special token (CLS) for classiﬁcation.
The only modiﬁcation is that we mask entity mentions with generic entity types, i.e., $ ENTITY A $ or $ ENTITY B $ .
It should be noted that the method that we introduce here is not classiﬁer speciﬁc, so any other classiﬁer can be used instead.
To generate new training data, we split the D dataset into c subsets where each D c subset contains only examples from relation type c .
Subsequently, we ﬁne-tune GPT-2 on each D c for ﬁve epochs and then prompt each resulting ﬁne-tuned model to generate new sentences, ﬁltering out sentences that do not contain the special entity masks or that are too small (less than 8 tokens).
The generated sequences are combined for all relation types into a dataset Dsynth .
Subsequently, we build an ensemble of RE classiﬁers, each of them being ﬁne-tuned on a subset of Dsynth and the whole D , such that the perrelation type generated instances are equal to the number of gold instances for that relation, multiplied by ratio , i.e., | Dsynth (cid:48) c | = | D c | ∗ r .
In our experiments we have set r = 1 .
(refer to Section 4.6 for a short study of its inﬂuence).
Algorithmillustrates our method.
ALGORITHM
Input: D , L for each relation type c ∈ L do
D c = { s | s ∈ D, rel type ( s )
= c } ; ﬁne-tune GPT-2 on D c ; generate Dsynth c with GPT-2; end
Dsynth =
Dsynth 1 ∪ ... ∪ Dsynth c ; for each classiﬁer in ensemble do
Dsynth (cid:48) ∼ Dsynth s.t.
| Dsynth (cid:48) c | = r ∗ | D c | and
Dsynth (cid:48)
=
Dsynth (cid:48) 1 ∪ ... ∪ Dsynth (cid:48) c ; train RE classiﬁer on D ∪ Dsynth (cid:48) ; end predict on Dtest with majority voting over the ensemble;
We would like to note that in early experiments, we also experimented with ﬁne-tuning over the whole D , by adding a special token to the beginning of each sentence that encoded the relation type, e.g., < 0 > : or < 1 > :.
Then during generation, we would prompt the model with the different special tokens and let it generate a training instance from the respective relation type.
However, this approach did not prove effective leading to worse results than just using gold data, primarily because frequent classes ”inﬂuenced” more GPT-2 and the model was generating many incorrectly labeled samples.
Experimental Evaluation
In this section we present the empirical evaluation of our method.
We ﬁrst describe the experimental setup, the datasets used, the baselines against which we evaluate DARE and subsequently present the experiments and report the relevant results.
Dataset | L |
Training
Development
Test
[Table]
4. Setup
In all experiments we used the second-largest GPT2 model (774M parameters).
All experiments were carried out on a machine equipped with a GPU
V100-16GB.
For the implementation, we have used HuggingFace’s
Transformers library (Wolf et al., 2019).
To ﬁne-tune GPT-2 we employed Adam as the optimizer, a sequence length of 128, a batch size of 4 with gradient accumulation over 2 batches (being equivalent to a batch size of 8) and a learning rate of 3 e − 5 .
In all datasets and for all relation types we ﬁne-tuned for 5 epochs.
For generation we used a temperature of 1 . 0 , ﬁxed the top-k parameter toand generated sequences of up to 100 word tokens.
An extensive search for the above optimal hyperparameter values is left to future work.
Since all of our datasets are from the biomedical domain, we found out empirically (see Section 4.
for the relevant experiment) that it was beneﬁcial to ﬁrst ﬁne-tune a GPT-2 model on 500k PubMed abstracts, followed by a second round of ﬁne-tuning per dataset, per relation type.
In all cases, we used a pre-trained BERT model (the largest uncased model) as a RE classiﬁer, which we ﬁne-tuned on either the gold or the gold+generated datasets.
We used the AdamW optimizer (Loshchilov and Hutter, 2017), a sequence length of 128, a batch size of 32 and a learning rate of 2 e − 5 , We ﬁne-tuned for 5 epochs, keeping the best model with respect to the validation set loss.
Also, we used a softmax layer to output predictions and we assigned a relation type to each instance si as follows:
[Equation]
where c ∈ L and 0 < t < 1 is a threshold that maximizes the micro-F score on the validation set.
For DARE, in all experiments we train an ensemble of twenty classiﬁers, where each classiﬁer has been trained on the full gold set and a sub-sample of the generated data.
In this way, we manage to alleviate the effect of potential noisy generated instances.
4. Datasets
To evaluate DARE, we employ three RE datasets from the biomedical domain, their statistics being provided in Table 2.
The BioCreative V CDR corpus (Li et al., 2016) contains chemical-disease relations.
The dataset is a binary classiﬁcation task with one relation type, chemical induces disease , and annotations are at the document level, having already been split into train, development and test splits.
For simplicity, we followed the work of Papanikolaou et al.
and considered only intra-sentence relations.
We have included the dataset in our GitHub repository to ease replication.
In the following, we dub this dataset as CDR .
The DDIExtraction 2013 corpus (Segura Bedmar et al., 2013) contains MedLine abstracts and
DrugBank documents describing drug-drug interactions.
The dataset has four relation types and annotations are at the sentence level.
The dataset is provided with a train and test split for both MedLine and DrugBank instances.
Following previous works, we concatenated the two training sets into one.
Also, we randomly sampled 10% as a development set.
In the following this dataset will be referred to as DDI2013 .
The
BioCreative
VI-ChemProt corpus (Krallinger et al., 2017) covers chemical-protein interactions, containing ﬁve relation types, the vast majority of them being at the sentence level.
The dataset comes with a train-development-test split.
In the following we will refer to it as ChemProt .
4. Baselines
The above datasets suffer both from class imbalance and a limited number of positives. For example the rarest relation type in DDI2013 has onlyinstances in the training set, while the respective GPTPrecision Recall
F
Vanilla 0. ﬁne-tuned 0. one in ChemProt has only 173 data points. Therefore, we consider two suitable baselines for such scenarios, the balanced bagging approach and the class weighting method, both described in Section 2.
Both baselines use the base classiﬁer described in Section 4.1.
Also, in both cases we consider an ensemble of ten models 2 .
Finally, for the class weighting approach we set each class’s weight as weight c = freq min freq
[Equation]
with min being the rarest class.
4. Fine-tuning GPT-2 on In-domain Data
Since all our datasets come from the biomedical domain, we hypothesized that a ﬁrst round of ﬁnetuning GPT-2 on in-domain data could be beneﬁcial instead of directly employing the vanilla GPTmodel.
We designed a short experiment using the
CDR dataset to test this hypothesis.
To clarify, any of the two models (i.e, the vanilla and the one ﬁnetuned in in-domain data) would then be ﬁnetuned per relation type to come up with the ﬁnal GPT-2 models that would generate the new training examples.
Table 3 illustrates the results of this experiment.
As we expect, this ﬁrst round of ﬁne-tuning proves signiﬁcantly favourable.
We note that when inspecting the generated examples from the vanilla GPTmodel, generated sentences often contained a peculiar mix of news stories with the compound-disease relations.
4. DARE on Imbalanced Datasets
In this experiment, we wanted to evaluate the effect of our method when dealing with great imbalance, i.e., datasets with very few positive samples.
To that end, we considered the CDR dataset and sampled different numbers of positive examples from
# positive samples
F
Balanced Bagging
DARE
Dataset
BB
DARE 0.
all(1453) 0.
the dataset (50, 250, 500, 1000 and all positives) and combined them with all the negative instances. The resulting ﬁve datasets were used to train either a balanced bagging ensemble or DARE. In Figure 1, we show the results, averaging across ﬁve different runs.
In all cases, our approach has a steady, signiﬁcant advantage over the balanced bagging baseline, their difference reaching up to 11 F1 score points when only few positives ( ≤ 250 ) are available.
As we add more samples, the differences start to smooth out as expected.
These results clearly illustrate that DARE can boost the predictive power of a classiﬁer when dealing with few positive samples, by cheaply generating training data of arbitrary sizes.
4. Effect of Generated Data Size
Our next experiment focuses in studying the effect of different sizes of generated data on DARE’s performance.
As explained, our method relies on ﬁne-tuning
GPT-2 to generate examples for each relation type that will, ideally, come from the same distribution as the ones from the gold training data.
Nevertheless, we should expect that this procedure will not be perfect, generating also noisy samples.
As mentioned previously, we try to alleviate this effect by training an ensemble of classiﬁers, each trained on
[Figure]
ratio
F
DARE
Balanced Bagging the whole gold and a part of the generated dataset. An important question that arises therefore, is how to determine the optimal ratio of generated examples to include in each classiﬁer. If too few, the improvements will be insigniﬁcant, if too many we risk to have the model being inﬂuenced by the noise.
In order to gain empirical insight into the above question we design a short experiment using the
CDR dataset, for different sizes of generated data.
As gold set, we consider a random subset of 1, positive examples and all negatives, to make more prominent the effect of class imbalance.
In Figure 2 we show the results for ﬁve different generated data sizes.
Interestingly, adding more data does not necessarily boost classiﬁer performance, since the noisy patterns in the generated data seem to inﬂuence more the classiﬁer than those in the gold data.
In the following, we choose a ratio = 1 , adding for each relation type a number of generated instances equal to the number of gold instances.
It should be noted that we are not limited in the total generated data that we will use since we can ﬁne-tune an arbitrary number of classiﬁers on combinations of the gold data and subsets of the generated data.
4. DARE against the SOTA and Baselines
Taking into account the previous observations, we proceed to compare DARE against the SOTA and the two previously described baselines.
Tabledescribes the results.
For the multi-class datasets we report the micro-F score in order to make our results comparable with previous works.
Also, in the Supplementary Material we report the per class results for DARE against the SOTA and the class weighting baseline, for the two multi-class datasets in order to ease comparison with past or future works.
Comparing DARE against the SOTA, we observe a steady advantage of our method across all datasets, ranging from 3 to 8 F1 points.
These results are somehow expected, since we employ BERT-large as our base classiﬁer which has proven substantially better than Convolutional (CNN) or Recurrent neural networks (RNN) across a variety of tasks (Devlin et al., 2018).
In CDR, Papanikolaou et al.
(2019) have used BioBERT(Lee et al., 2019) which is a BERT base (cased) model pre-trained on PubMed, while we use BERT large (uncased), in ChemProt, Peng et al.
(2018) use ensembles of SVM, CNN and RNN models while in DDI2013 Sun et al.
(2019b) have used hybrid CNN-RNN models.
When observing results for the baselines, we notice that they perform roughly on par.
DARE is better from 2 to 5 F1 points against the baselines, an improvement that is smaller than that against the SOTA, but still statistically signiﬁcant in all cases.
Overall, and in accordance with the results from the experiment in Section 4.5, we observe that
DARE manages to leverage the GPT-2 automatically generated data, to steadily improve upon the SOTA and two competitive baselines.
Conclusions
We have presented DARE, a novel method to augment training data in Relation Extraction.
Given a gold RE dataset, our approach proceeds by ﬁnetuning a pre-trained GPT-2 model per relation type and then uses the ﬁne-tuned models to generate new training data.
We sample subsets of the synthetic data with the gold dataset to ﬁne-tune an ensemble of RE classiﬁers that are based on BERT.
Through a series of experiments we show empirically that our method is particularly suited to deal with class imbalance or limited data settings, recording improvements up to 11 F1 score points over two strong baselines.
We also report new SOTA performance on three biomedical RE benchmarks.
Our work can be extended with minor improvements on other Natural Language Understanding tasks, a direction that we would like to address in future work. Dataset Conﬁguration Precision
Recall F CDR SOTA (Papanikolaou et al., 2019) 0.
BERT+class weighting 0.
BERT+balanced bagging 0.
DARE 0.
ChemProt
SOTA (Peng et al., 2018) 0.
BERT+class weighting 0.
BERT+balanced bagging 0.
BERT+DARE 0.
DDI
SOTA (Sun et al., 2019b) 0.
[Table]
A Supplemental Material
In this section we present additionally the results per class for ChemProt and DDI2013, for DARE against the class weighting baseline and the SOTA. [Table] [Table] advise
[Table]
effect
[Table]