|
--- |
|
datasets: |
|
- wikimovies |
|
|
|
language: |
|
- English |
|
|
|
thumbnail: |
|
|
|
tags: |
|
- roberta |
|
- roberta-base |
|
- masked-language-modeling |
|
|
|
license: cc-by-4.0 |
|
|
|
--- |
|
# roberta-base for MLM |
|
|
|
``` |
|
model_name = "thatdramebaazguy/roberta-base-wikimovies" |
|
pipeline(model=model_name, tokenizer=model_name, revision="v1.0", task="Fill-Mask") |
|
``` |
|
## Overview |
|
**Language model:** roberta-base |
|
**Language:** English |
|
**Downstream-task:** Fill-Mask |
|
**Training data:** wikimovies |
|
**Eval data:** wikimovies |
|
**Infrastructure**: 2x Tesla v100 |
|
**Code:** See [example](https://github.com/adityaarunsinghal/Domain-Adaptation/blob/master/shell_scripts/train_movie_roberta.sh) |
|
|
|
## Hyperparameters |
|
``` |
|
num_examples = 4346 |
|
batch_size = 16 |
|
n_epochs = 3 |
|
base_LM_model = "roberta-base" |
|
learning_rate = 5e-05 |
|
max_query_length=64 |
|
Gradient Accumulation steps = 1 |
|
Total optimization steps = 816 |
|
evaluation_strategy=IntervalStrategy.NO |
|
prediction_loss_only=False |
|
per_device_train_batch_size=8 |
|
per_device_eval_batch_size=8 |
|
adam_beta1=0.9 |
|
adam_beta2=0.999 |
|
adam_epsilon=1e-08, |
|
max_grad_norm=1.0 |
|
lr_scheduler_type=SchedulerType.LINEAR |
|
warmup_ratio=0.0 |
|
seed=42 |
|
eval_steps=500 |
|
metric_for_best_model=None |
|
greater_is_better=None |
|
label_smoothing_factor=0.0 |
|
``` |
|
## Performance |
|
|
|
perplexity = 4.3808 |
|
|
|
Some of my work: |
|
- [Domain-Adaptation Project](https://github.com/adityaarunsinghal/Domain-Adaptation/) |
|
|
|
--- |
|
|
