PubMedBert-PubMed200kRCT

This model is a fine-tuned version of microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext on the PubMed200kRCT dataset. It achieves the following results on the evaluation set:

• Loss: 0.2833
• Accuracy: 0.8942

Model description

More information needed

Intended uses & limitations

The model can be used for text classification tasks of Randomized Controlled Trials that does not have any structure. The text can be classified as one of the following:

• BACKGROUND
• CONCLUSIONS
• METHODS
• OBJECTIVE
• RESULTS

The model can be directly used like this:

from transformers import TextClassificationPipeline
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("pritamdeka/PubMedBert-PubMed200kRCT")
tokenizer = AutoTokenizer.from_pretrained("pritamdeka/PubMedBert-PubMed200kRCT")
pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer, return_all_scores=True)
pipe("Treatment of 12 healthy female subjects with CDCA for 2 days resulted in increased BAT activity.")

Results will be shown as follows:

[[{'label': 'BACKGROUND', 'score': 0.0028450002428144217},
  {'label': 'CONCLUSIONS', 'score': 0.2581048607826233},
  {'label': 'METHODS', 'score': 0.015086210332810879},
  {'label': 'OBJECTIVE', 'score': 0.0016815993003547192},
  {'label': 'RESULTS', 'score': 0.7222822904586792}]]

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 2.0

Training results

Framework versions

• Transformers 4.18.0.dev0
• Pytorch 1.10.0+cu111
• Datasets 1.18.3
• Tokenizers 0.11.6

Citing & Authors