bert-paper-classifier
This model is a fine-tuned version of microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract on the dataset with titles and abstracts of papers from arXiv.
Intended uses & limitations
This model is intended to predict the arXiv taxonomy terms given the paper title (and optionally its abstract) — this is relevant for papers in mathematics, computer science, physics, etc. For example, it is likely to predict cs.CL (computation and language) as the mostl likely category for the original transformers paper.
Training procedure
The code for the model fine-tuning can be found in the respective notebook.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 128
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3
Framework versions
- Transformers 4.28.1
- Pytorch 2.0.0+cu117
- Datasets 2.11.0
- Tokenizers 0.13.3
- Downloads last month
- 3