relbert/roberta-large-semeval2012-mask-prompt-b-nce-classification-conceptnet-validated
RelBERT fine-tuned from roberta-large on
relbert/semeval2012_relational_similarity.
Fine-tuning is done via RelBERT library (see the repository for more detail).
It achieves the following results on the relation understanding tasks:
- Analogy Question (dataset, full result):
- Accuracy on SAT (full): 0.5080213903743316
- Accuracy on SAT: 0.5192878338278932
- Accuracy on BATS: 0.6653696498054474
- Accuracy on U2: 0.45614035087719296
- Accuracy on U4: 0.5393518518518519
- Accuracy on Google: 0.84
- Lexical Relation Classification (dataset, full result):
- Micro F1 score on BLESS: 0.9132138014163026
- Micro F1 score on CogALexV: 0.8502347417840377
- Micro F1 score on EVALution: 0.6852654387865655
- Micro F1 score on K&H+N: 0.9604228976838005
- Micro F1 score on ROOT09: 0.9022250078345346
- Relation Mapping (dataset, full result):
- Accuracy on Relation Mapping: 0.856984126984127
Usage
This model can be used through the relbert library. Install the library via pip
pip install relbert
and activate model as below.
from relbert import RelBERT
model = RelBERT("relbert/roberta-large-semeval2012-mask-prompt-b-nce-classification-conceptnet-validated")
vector = model.get_embedding(['Tokyo', 'Japan']) # shape of (1024, )
Training hyperparameters
The following hyperparameters were used during training:
- model: roberta-large
- max_length: 64
- mode: mask
- data: relbert/semeval2012_relational_similarity
- split: train
- data_eval: relbert/conceptnet_high_confidence
- split_eval: full
- template_mode: manual
- template: Today, I finally discovered the relation between and : is 's
- loss_function: nce_logout
- classification_loss: True
- temperature_nce_constant: 0.05
- temperature_nce_rank: {'min': 0.01, 'max': 0.05, 'type': 'linear'}
- epoch: 27
- batch: 128
- lr: 5e-06
- lr_decay: False
- lr_warmup: 1
- weight_decay: 0
- random_seed: 0
- exclude_relation: None
- exclude_relation_eval: None
- n_sample: 640
- gradient_accumulation: 8
The full configuration can be found at fine-tuning parameter file.
Reference
If you use any resource from RelBERT, please consider to cite our paper.
@inproceedings{ushio-etal-2021-distilling-relation-embeddings,
title = "{D}istilling {R}elation {E}mbeddings from {P}re-trained {L}anguage {M}odels",
author = "Ushio, Asahi and
Schockaert, Steven and
Camacho-Collados, Jose",
booktitle = "EMNLP 2021",
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
}
- Downloads last month
- 21
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Dataset used to train research-backup/roberta-large-semeval2012-mask-prompt-b-nce-classification-conceptnet-validated
Evaluation results
- Accuracy on Relation Mappingself-reported0.857
- Accuracy on SAT fullself-reported0.508
- Accuracy on SATself-reported0.519
- Accuracy on BATSself-reported0.665
- Accuracy on Googleself-reported0.840
- Accuracy on U2self-reported0.456
- Accuracy on U4self-reported0.539
- F1 on BLESSself-reported0.913
- F1 (macro) on BLESSself-reported0.910
- F1 on CogALexVself-reported0.850