On Data Augmentation and Consistency-based Semi-supervised Relation Extraction
Abstract
To improve the sample efficiency of the Relation extraction (RE) models, semisupervised learning (SSL) methods aim to leverage unlabelled data in addition to learning from limited labelled data points.
Recently, strong data augmentation combined with consistency-based semi-supervised learning methods have advanced the state of the art in several SSL tasks.
However, adapting these methods to the
RE task has been challenging due to the difficulty of data augmentation for RE.
In this work, we leverage the recent advances in controlled text generation to perform high quality data augmentation for the RE task.
We further introduce small but significant changes to model architecture that allows for generation of more training data by interpolating different data points in their latent space.
These data augmentations along with consistency training result in very competitive results for semi-supervised relation extraction on four benchmark datasets.
Introduction
Relation extraction (RE) task aims to discover the semantic relation between a given head entity and tail entity based on the context in the input sentence.
For example, given a sentence " The battle led to panic on the frontier and settlers in the surrounding counties fled. ", the goal is to extract the
Cause-Effect relation between the head entity ‘ battle ’ and the tail entity ‘ panic ’.
The models for RE task typically require large amounts of labelled data to give production-ready performance.
A common strategy to improve the sample efficiency of machine learning models is semi-supervised learning methods which leverage easily accessible unlabelled data to improve the overall performance.
While there are several paradigms of semi-supervised learning methods, consistency training based methods have advanced the state of the art in several SSL tasks Ghosh and Thiery (2021).
These methods can typically reach performances that are comparable to their fully supervised counterparts while using only a fraction of labelled data points.
Recently strong data augmentation combined with consistency training algorithms have shown great success, even surpassing fully supervised models, in low-data settings of various tasks
Xie et al.
Adapting these methods to the task of relation extraction has been challenging due to the difficulty of data augmentation for RE task.
This is because, in addition to the input sentence, each data point also consists of a head entity and a tail entity contained in the input sentence.
Typical data augmentation techniques used in NLP such as back-translation, synonym-replacement, languagemodel based augmentation, etc.
Feng et al.
(2021) can not be easily applied to such ‘structured’ input as they do not guarantee the integrity of either a) the entities in the input sentence or, b) the meaning of the input sentence itself.
Figure 1 in the Appendix shows that using synonym-replacement and vanilla back-translation (BT) methods Sugiyama and Yoshinaga (2019) the entities themselves could be paraphrased or replaced.
Matching the new and the old entities is a whole problem in itself.
In the language-model based augmentation method Anaby-Tavor et al.
(2020), the semantic meaning of the Input sentence: The battle led to panic on the frontier, and settlers in the surrounding counties fled. Synonym-replacement:
The struggle cause scare on the frontier, and settlers in the surrounding counties fly.
LM-based augmentation: The battle reduced panic on the frontier, and settlers in the surrounding counties relaxed. Vanilla BT: The war caused panic at the border, and residents of the nearby counties fled. Our constrained BT: The battle sparked panic at the border, with residents fleeing in surrounding counties.
Existing methods replace the head/tail entities, change the original meaning or do not fluent input sentence changes altogether, which makes it difficult to employ consistency training.
Present work . In this work, we leverage the recent advances in controlled text generation to perform high quality data augmentation for the relation extraction task that not only keeps the meaning and the head/tail entities intact but also produces fluent and diverse data points.
In particular, we modify back-translation to leverage lexically constrained decoding strategies Post and Vilar (2018); Hu et al. (2019) in order to obtain paraphrased sentences while retaining the head and the tail entities.
We further propose novel modifications to the widely popular relation extraction model architecture, that allows for generation of more samples by interpolating different data points in their latent space, a trick that has been very successful in other domains and tasks Berthelot et al.
; Chen et al. (2020b,a).
Additionally, we leverage the entity types of the head and the tail entities, when available, in a way that effectively exploits the knowledge embedded in pre-trained language models.
These data augmentations, when applied to unlabelled data, let us employ consistency training techniques to achieve very competitive results for semi-supervised relation extraction on four benchmark datasets.
To the best of our knowledge, this is the first study to apply and show the merit of data augmentation and consistency training for semi-supervised relation extraction task.
Proposed approach
In this work, we focus on the sentence-level relation extraction task, i.e., given a relation statement x : ( s , e h , e t ) consisting of a sentence, s , a head entity, e h , and a tail entity, e t (both the entities are mentioned in the given sentence s ), the goal is to predict a relation r ∈ R ∪ { NA } between the head and the tail entity, where R is a pre-defined set of relations.
If the sentence does not express any relation from the set R between the two entities, then the relation statement x is accordingly labelled
NA.
This is typically done by learning a relation encoder model F θ : x (cid:55)→ h r that maps an input relation statement, x , to a fixed length vector h r that represents the relation expressed in s between e h and e t .
This relation representation, h r , is then classified to a relation r ∈ R ∪ { NA } via an MLP classifier.
In our approach we build on the base model architecture described in CITEand implement two different data augmentation techniques and employ consistency training to obtain state-of-the-art performance.
Constrained back-translation .
Back-translation Edunov et al.
(2018) generates diverse and fluent augmentations while retaining the global semantics of the original input.
Specifically, one translates a given text into an intermediate language, say, German, and translates it back to the source language, say English.
Applying this back-translation technique in a vanilla fashion is not possible for RE task because one has little control over the retention of the head and tail entities (Figure 1).
Thus, when translating back to the source language from the intermediate language we perform lexicallyconstrained decoding Hu et al.
(2019), i.e., force the inclusion of pre-specified words and phrases– positive constraint set–in the output.
In our case the original head and tail entity words/phrases make up this positive constraints set.
Latent-space interpolation .
Here, we adapt a mixup-based data augmentation technique to the RE task.
As done in previous works (Chen et al., 2020b,a), we sample two random data points: ( x , y ) and ( x ′ , y ′ ) , where x and y denote the relation statement and the corresponding relation label.
We then follow Chen et al.
(2020b) to linearly interpolate latent representations of x and x ′ and obtain the encoder output representation of a virtual input ˜ x , i.e., ˜ h
L = F θ (˜ x ) .
We denote this whole mix-up operation as ˜ h
L :=
MixRE ( x , x ′ ) .
Now, for the RE task we need to extract a fixed-length relation representation from the encoder output representation of the entire input sequence.
The traditional way to do this for RE task is by concatenating the encoder output representations of the entity marker tokens [E1] and [E2] .
However, it is challenging to do this for a virtual sample, ˜ x , as the entity markers are not clearly defined in this case.
We thus modify the relation representation to be the encoder output representation of the [CLS] token.
However, Baldini Soares et al.
(2019) have shown this choice to be sub-optimal compared to concatenation of marker tokens.
This is because the marker token representations provide direct access to the contextual information of the respective entities.
Although the [CLS] token, in theory, has access to the entire context of the sentence, it might be difficult to capture the nuances like the head entity type, tail entity type, and the contextual information around the two entities all in a single vector.
On the other hand, entity type information is easily accessible in most RE benchmarks 2 .
So, to compensate for the sub-optimal choice of using [CLS] token representation as the relation representation, we modify how we represent the entity spans in the input token to more effectively use the easily accessible entity type information.
In particular, we note that the entity type labels can trivially be mapped to tokens from any pre-trained language model’s vocabulary.
For example, entity types like PERSON and STATE_OR_PROVINCE can be tokenized into a word/phrase like ‘person’ and ‘state or province’, respectively.
Instead of using special marker tokens like [E1] and [E2] , we prepend the entity spans in the input sequence with the word/phrases corresponding to their respective types and enclose these ‘type-words’ in punctuation marks
Zhou and Chen (2021).
We train our model with these mixup-based augmentations, back-translated augmentations, and consistency training following the same training procedure as described in Chen et al.
(2020b).
Detailed description of the whole training algorithm is provided in Appendix 5.
Experiments
We perform experiments on four benchmark datasets for sentence-level RE: SemEval 2010 Task(SemEval) Hendrickx et al.
(2010), the TAC Relation Extraction Dataset (TACRED) Zhang et al. (2017), RE-TACRED Stoica et al.
(2021), and KBP37 Zhang and Wang (2015).
We compare MixRE with three state-of-the-art models that are representative of the existing class of methods for SSRE:
MRefG Li et al.
MetaSRE Hu et al.
(2021a), and GradLRE Hu et al.
(2021b).
MetaSRE and
GradLRE are two of the strongest methods in the widely adapted self-training methods for SSRE.
Detailed description of the datasets and baselines is provided in Appendix 6.
Implementation details .
We follow the established setting to use stratified sampling to divide the training set into various proportions of labelled and unlabelled sets so that the relation label distribution remains the same across all subsets.
Following existing work, we sample 5%, 10%, and 30% of the training set as labelled data for the SemEval and KBP37 datasets, and 3%, 10%, and 15% of the training set as labelled data for TACRED and RE-TACRED datasets.
For all datasets and experiments, unless otherwise specified, we sample 50% of the training set as the unlabelled set.
For
TACRED and SemEval datasets we take the performance numbers of all baseline models reported by
Hu et al.
(2021b).
For other datasets, we re-run the models with their best configuration as provided in their respective implementations.
Full implementation details can be found in Appendix 6.
3.1 Main Results
Table 1 shows F1 results of all baseline models and our proposed model, MixRE, on the four datasets when leveraging various amounts of labelled data and 50% unlabelled data. We report the mean and [Table] standard
[Table] [Table] different seeds [Table]
training [Table] testing. MixRE gives state-of-the-art performance on 10 out of 12 different configurations across all four datasets.
This reinforces the importance of consistency regularization beyond the currently popular self-training methods for SSRE.
Interestingly, the performance gains are significantly higher for TACRED and
RE-TACRED datasets–we see an average improvement of as much as 17% when trained on 3% labelled data.
This can be attributed to the fact that entity type information is available for these datasets and entity type markers are very effective in exploiting the knowledge embedded in the pre-trained language models.
We revisit this observation in our ablation studies (Section 3.2) where we concretely establish the benefits of using entity type markers.
3.2 Analysis and discussion
We first conduct experiments to empirically demonstrate the effectiveness of three components of our proposed model: i) data augmentation by latent space interpolation (Mix-DA), ii) data augmentation by constrained back-translation (BT-DA), and iii) entity type markers (ET).
In Table 2, we report the mean F1 score of five different runs for different variations of our model by removing a certain combination of these components. As can be seen from Table 2, each of these components, contributes to the overall success of MixRE. For contribution of just the Mix-DA: we compare i) row a v/s row c , and ii) row b v/s row d . All comparisons show positive improvement.
For contribution of just the BT-DA: we compare i) row a v/s row b , and ii) row c v/s row d .
We note that BT-DA results only in marginal improvements in most cases.
Upon closer inspection we note that the constraineddecoding algorithms we implement for BT-DA are actually not perfect, especially when combined with translation models.
It sometimes misses the constraints and sometimes falls into repetitive loops in an attempt to satisfy the constraint.
With the ever-improving language generation capabilities, we believe the quality of data augmentation will only improve with time and result in more significant performance improvements.
For contribution of both DA techniques together: we compare row a v/s row d .
All comparisons show significant improvements with data augmentation.
The contribution of entity type markers can be noted in TACRED and RE-TACRED datasets.
We see an average drop of 5.4% in F1 score across all 8 comparisons.
Next, we examine the effect of using different amounts of unlabelled data.
In Figure 2, we report the average F1 score for different models trained with different amounts of unlabelled data and 10% labelled data.
MixRE outperforms the baselines in all settings except on SemEval dataset, and, interestingly, the performance only marginally changes with the change in the amount of unlabelled data.
Note that we train the models until the performance on the validation set stops improving for more than 5 epochs.
Hence, MixRE generates, in principle, an infinite amount of unlabelled data via the mixup strategy.
Coupled with the fact that the label distribution remains the same in all settings, adding more unlabelled data does not seem to add a lot of new information.
This explains why the model performance is relatively insensitive to changing amounts of unlabelled data.
This also implies that MixRE can leverage low amounts of unlabelled data better than the baselines. [Table]
[Figure] Next [Figure] in Figure 3 we show how the performance of MixRE changes with a change in the mean of the
Beta distribution from which λ is sampled on each iteration.
Note that a value near 0 and 1 for λ means the augmented virtual data point will be closer to one of the underlying data points.
As we get closer to 0.5 the virtual data points get further from the original data manifold and become more ‘novel’.
On TACRED and RE-TACRED datasets the performance peaks at E ( λ ) = 0 .
15( or 0.85 ) and drops in the mid-values.
This can be interpreted as: adding datapoints far from the original data manifold is detrimental for these datasets.
Interestingly, on KBP37 and SemEval the pattern inverts, i.e., the performance increases as E ( λ ) towards 0 . 5 , implying that more ‘novel’ augmentations help for these datasets.
[Figure]
Conclusion
In this paper, we propose a consistency training based semi-supervised algorithm for relation extraction and empirically show the merit of this class of methods in comparison to the current state-of-the-art self-training class of methods.
This shows there is promise in a potential future direction where one could bootstrap the self-training methods with consistency training as done in some previous works on vision tasks Pham et al.
Additionally, we show how the entity type information, when available, can result in massive performance boosts in the semi-supervised scenario.
This is important because in most practical use cases when entities have already been identified, the entity type information is easy available and could be effectively leveraged in the proposed fashion.
Acknowledgements
The author would like to thank the reviewers for their thoughtful feedback, the entire VAiR team at
Vanguard for their support and especially David Landsman for their valuable input and discussion on the initial drafts of this work.
Disclaimer
This material is provided for informational purposes only and is not intended to be investment advice or a recommendation to take any particular investment action.
{+©
+}
The Vanguard Group, Inc.
All rights reserved.
Proposed approach
Task formulation .
In this work, we focus on the sentence-level relation extraction task, i.e., given a relation statement x : ( s , e h , e t ) consisting of a sentence, s , a head entity, e h , and a tail entity, e t (both the entities are mentioned in the given sentence s ), the goal is to predict a relation r ∈ R ∪ { NA } between the head and the tail entity, where R is a pre-defined set of relations.
If the sentence does not express any relation from the set R between the two entities, then the relation statement x is accordingly labelled NA.
This is typically done by learning a relation encoder model F θ : x (cid:55)→ h r that maps an input relation statement, x , to a fixed length vector h r that represents the relation expressed in s between e h and e t .
This relation representation, h r , is then classified to a relation r ∈ R ∪ { NA } via an MLP classifier.
Base model architecture .
Most recent methods for RE use a Transformer-based architecture Devlin et al.
; Vaswani et al.
(2017) for the relation encoder model, F θ .
To represent the head and tail entities in the input to the encoder, the widely accepted strategy is to augment the input sentence s with entity marker tokens– [E1], [/E1] , [E2], [/E2] –to mark the start and end of both entities.
Concretely, an input sentence like " Lebron James currently plays for LA Lakers team. "
when augmented with entity marker tokens becomes [E1]
Lebron James [/E1] currently plays for [E2]
LA Lakers [/E2] team.
This modified text is input to the Transformer-based sequence encoder.
Next, the encoder output representations 3 of the tokens [E1] and [E2] are concatenated to give the fixed length relation representation, h r =
[ h [ E 1] ⊕ h
[ E 2] ] .
This fixed length vector is in turn passed through an MLP classifier, p ϕ ( h r ) , to give a probability vector, y , over the relation set R ∪ { NA } .
In our approach we build on the base model architecture described above and introduce additional model design elements that are necessary to obtain an improved performance in semi-supervised relation extraction (SSRE) task.
We first describe the two data augmentation techniques we perform, and the model architectural changes we introduce that facilitate these augmentations.
Then, we describe the training procedure we follow to leverage unlabelled data and achieve state-of-the-art performance on three out of four benchmark datasets for SSRE.
5.1 Constrained back-translation
Back-translation Edunov et al.
(2018) generates diverse and fluent augmentations while retaining the global semantics of the original input.
Specifically, one translates a given text into an intermediate language, say, German, and translates it back to the source language, say English.
Using different intermediate languages and temperature based sampling results in a diverse set of paraphrases.
Applying this back-translation technique in a vanilla fashion is not possible for RE task because one has little control over the retention of the head and tail entities (Figure 1).
Thus, when translating back to the source language from the intermediate language we perform lexically-constrained decoding Hu et al.
(2019), i.e., force the inclusion of pre-specified words and phrases–positive constraint set–in the output.
In our case the original head and tail entity words/phrases make up this positive constraints set.
We use German and Russian as intermediate languages and use the pre-trained WMT’19 EnglishGerman and English-Russian translation models (in both directions) and their implementations provided by Ott et al.
This methodology generates diverse data augmentations for a given sentence.
For example, the sentence " The battle led to panic on the frontier, and settlers in the surrounding counties fled. " is converted to " The battle sparked panic at the border, with residents fleeing in surrounding counties " when back-translated via German, and to " The battle caused panic on the border and settlers in nearby counties fled. "
when done via Russian.
This strong dataaugmentation technique for RE can be applied to both labelled and unlabelled data opening the doors to consistency training Xie et al.
(2020) as we will see in Section 5.3.
5.2 Latent-space interpolation
Here, we adapt a mixup-based data augmentation technique to the RE task by making necessary modifications to the base model architecture we described in Section 2.
As done in previous works
Chen et al.
(2020b,a), we sample two random data points– ( x , y ) and ( x ′ , y ′ ) , where x and y denote the relation statement and the corresponding relation label–from the training data and separately compute the respective latent representations, h m and h ′ m , upto the layer m of the relation encoder
F θ as follows:
[Equation]
where h l is the latent representation of all tokens in the sentence x at the l th layer of the encoder.
Next, the latent representations of each token in x at the m th layer are linearly interpolated:
[Equation]
where λ is the mixing coefficient which is sampled from a Beta distribution, i.e., λ ∼ Beta ( α, β ) .
Then, the interpolated latent representation is passed through the rest of the encoder layers:
[Equation]
This final encoder output representation, ˜ h L , can be interpreted as the encoder output representation of a virtual input ˜ x , i.e., ˜ h
L = F θ (˜ x ) .
We denote this whole mixup operation 4 as ˜ h
L :=
MixRE ( x , x ′ ) .
The label for this augmented virtual sample is given by the linear interpolation of the respective labels, y and y ′ , with the same mixing coefficient λ i.e., ˜ y :
= mix ( y , y ′ )
=
λ y + (1 − λ ) y ′ .
This virtual data point, (˜ x , ˜ y ) , is the augmented data point and can be used as additional training data.
Proposed architecture change .
Now, for the RE task we need to extract a fixed-length relation representation from the encoder output representation of the entire input sequence.
The traditional way to do this for RE task is by concatenating the encoder output representations of the entity marker tokens [E1] and [E2] .
However, it is challenging to do this for a virtual sample, ˜ x , as the entity markers are not clearly defined in this case.
We thus modify the relation representation to be the encoder output representation of the [CLS] token.
However, Baldini Soares et al.
(2019) have shown this choice to be sub-optimal compared to concatenation of marker tokens.
This is because the marker token representations provide direct access to the contextual information of the respective entities.
Although the [CLS] token, in theory, has access to the entire context of the sentence, it might be difficult to capture the nuances like the head entity type, tail entity type, and the contextual information around the two entities all in a single vector.
On the other hand, entity type information is easily accessible in most RE benchmarks 5 .
So, to compensate for the sub-optimal choice of using [CLS] token representation as the relation representation, we modify how we represent the entity spans in the input token to more effectively use the easily accessible entity type information.
In particular, we note that the entity type labels can trivially be mapped to tokens from any pre-trained language model’s vocabulary.
For example, entity types like PERSON and STATE_OR_PROVINCE can be tokenized into a word/phrase like ‘person’ and ‘state or province’, respectively.
Instead of using special marker tokens like [E1] and [E2] , we prepend the entity spans in the input sequence with the word/phrases corresponding to their respective types and enclose these ‘type-words’ in punctuation marks
Zhou and Chen (2021).
The modified input to the transformer along with the [CLS] token looks as follows:
[Equation]
CLS
[Equation]
Lebron James [Equation] plays [Equation]
We use different punctuation symbols to distinguish between subject and object entities.
This representation helps leverage the knowledge already contained in the pre-trained large-language model about the type of the entity and offset some of the downside of using a simplified relation representation in the [CLS] token.
Zhou and Chen (2021) recently showed the success of this method in the fully supervised setting.
Here we use it in conjunction with a simplified relation representation and show its merit in semi -supervised RE setting
5.3 Consistency training for SSRE
Let the given limited labelled set be X l = { x l 1 , ..., x ln } , with their relation labels Y l = { y l 1 , ..., y ln } , where y li ∈ { 0 , 1 } |R∪{ NA }| is a one-hot vector and R is the set of pre-defined relations.
Let
X u = { x u 1 , ..., x um } be a large unlabelled set.
The goal is to apply both the data augmentation techniques described above and train a model with consistency loss to effectively leverage unlabelled data along with the limited labelled data.
We largely adapt the semi-supervised training techniques introduced by Chen et al. (2020b).
For each x ui in the unlabelled set X u , we generate K augmentations x ai,k , k ∈ { 1 , 2 , ..., K } using the constrained back translation technique with different intermediate languages 6 .
These augmentations make up the set X a = { x ai,k } .
For a given unlabelled data point x ui and its K augmentations x ai,k the label is given by the average of current model’s predictions on all K + 1 data points:
[Equation]
where y ui is a probability vector.
This not only enforces the constraint that the model should make consistent predictions for different augmentations but also makes the predictions more robust by ensembling all the predictions.
We merge the unlabelled set and the augmented set into X ua = X u ∪
X a and the corresponding labels are given by Y ua =
Y u ∪ Y a , where Y u = { y ui } , Y a =
{ y ai,k } , and y ai,k =
y ui ∀ k ∈ { 1 , 2 , ..., K } , i.e., all the augmented data points share the same label as the original unlabelled data point.
Given this cumulative set X ua and their generated labels Y ua as additional training data, we employ the MixRE augmentation technique to generate arbitrary amounts of training data.
In particular, we randomly sample two data points x ua s , x ua t ∈ X ua , and compute the encoder output representation of a new virtual data point with MixRE ( x ua s , x ua t ) and the corresponding target label with mix ( y ua s , y ua t ) .
Additionally, while computing the final unsupervised loss in each training iteration we filter out the unlabelled data points with prediction confidence below a certain threshold γ Xie et al.
Finally, to encourage low-entropy predictions on unlabelled data, we sharpen the predictions with a sharpening coefficient T :
[Equation]
Everything put together, the final unsupervised loss in each training iteration with mini-batch size B is computed as:
[Equation]
where
[Equation]
Here, I ( . ) is an indicator function and m ( . ) denotes the confidence masking function which filters out the low-confidence datapoints.
In our implementation p ϕ ( . ) is a two-layer MLP classifier on top of the relation encoder model.
CE denotes the cross entropy loss function.
Note that we only apply the augmentation techniques on the unlabelled data set.
Initial experiments applying these to the labelled data set resulted in only marginal improvements and even performance deterioration in some cases, likely due to introduction of too much noise into an already limited labelled set.
This combined with the traditional supervised loss, L sup = (cid:80) B x i ∼ X l CE ( y l i || p ϕ
( F θ ( x i ))) , constitutes the total loss:
[Equation]
where γ m is a parameter which controls the trade-off between supervised and unsupervised loss
[Table]
Experiment details 6.1 Datasets
We use two widely popular relation extraction benchmark datasets: SemEval 2010 Task 8 (SemEval)
Hendrickx et al.
(2010), and the TAC Relation Extraction Dataset (TACRED)
Zhang et al.
SemEval is a standard benchmark dataset for evaluating relation extraction models containingexamples in total.
Each sentence is annotated with a pair of untyped nominals (concepts; example in
Figure 1) that are related via one of 19 semantic relation types (including no_relation ).
TACRED is a large-scale crowd-sourced relation extraction dataset with 106264 examples which is collected from all the prior TAC KBP relation schema.
Unlike SemEval, sentences in TACRED are labelled with pairs of typed-entities that are related via one of 42 person- and organization-oriented relation types (including no_relation ).
In addition to these standard benchmark datasets, we also show results on two more datasets: RE-TACRED Stoica et al.
(2021) and KBP37 Zhang and Wang (2015).
RE-TACRED is a re-annotated version of the original TACRED dataset using an improved annotation strategy to ensure high-quality labels.
Zhou and Chen (2021) provide a compelling analysis and recommend using this as the evaluation benchmark for sentence-level RE.
KBP37 is another sentencelevel RE dataset with 21046 total examples collected from 2010 and 2013 KBP documents as well as
July 2013 dump of Wikipedia.
In terms of size, this falls between SemEval and TACRED.
Similar to
SemEval the entity types are not available in this dataset, however the 37 relation types are person- and organization-oriented like in TACRED.
This dataset is thus a good segue between the two standard benchmarks.
The statistics of these datasets is given in Table 3.
The sources of all the datasets are given in Table 4.
We use the given train/validation/test splits for TACRED, RE-TACRED and KBP datasets.
For SemEval dataset, we use the same splits as all the baselines, i.e., we split the original training set into 90% training set and 10% validation set.
6.2 Baseline models
We compare MixRE with three state-of-the-art models that are representative of the existing class of methods for SSRE:
MRefG Li et al.
MetaSRE Hu et al.
(2021a), and GradLRE Hu
et al. (2021b).
MRefG leverages the unlabelled data by semantically or lexically connecting them to labelled data by constructing reference graphs, such as entity reference or verb reference.
This approach heavily leverages the linguistic structure of the data and is the only existing method that falls outside the self-training class of methods.
MetaSRE generates pseudo labels on unlabelled data by learning from the mistakes of the classification model as an additional meta-objective.
GradLRE on the other hand generates pseudo label data to imitate the gradient descent direction on labelled data and bootstrap its optimization capability through trial and error Hu et al.
(2021b).
MetaSRE and
GradLRE are two of the strongest methods in the widely adapted self-training methods for SSRE.
6.3 Implementation details
To be consistent with all the baselines we initialize the text encoder of MixRE with the bert-base-cased model architecture and pre-trained weights.
Following Chen et al.
(2020b), [Table] we use {7, 9, 12} for the mixup layer set; this layer subset contains most of the syntactic and semantic information as suggested by Jawahar et al.
We use the BERT tokenizer and set maximum sequence length as 256 to pre-process all datasets.
We use the AdamW optimizer Loshchilov and
Hutter (2019) with 5e-5 learning rate and 0.1 warmup ratio.
We sweep over the following hyperparameters: sharpening coefficient T , confidence threshold γ , the Beta-distribution parameters ( α , β ) 7 , and the unsupervised loss weight γ m .
We perform incremental grid search to get the best performing configuration based on the F1 score on validation set.
Table 5 shows the set of values we use for each parameter.
Table 6 shows the best parameter values on each dataset and configuration.
[-https://catalog.ldc.upenn.edu/LDC
-]
[Table]
[Table]
We train each model on a single NVIDIA Tesla T4 GPU with 16GB memory.
We employ mixed precision training and gradient checkpointing techniques for faster and memory-efficient training.
Note that we train the models until the performance on the validation set plateaus.
The full MixRE model roughly takes about 6 hours to train on TACRED, 5 hours in RE-TACRED, 1 hour in KBP37, and about 30 minutes on SemEval.
Note that the training time slightly vary ( ± 30 minutes) depending on the percentage of labelled and unlabelled data we use.
The number of parameters in all our models are largely dominated by the bert-base-cased that we use as the text encoder.
The relatively negligible varying component is the MLP classifier that varies with the varying number of relations in each dataset.