Trained using https://github.com/jfia0107/PyTorch-Lightning-NLP-lab and these parameters:
freeze_backbone: True
unfreeze_top_n: 2
scheduler: linear_warmup
warmup_ratio: 0.1
pretrained_model: distilroberta-base
attn_implementation: flash_attention_2
head_num_filters: 64
head_input_size: 300
head_filter_sizes: [1, 2, 3, 4]
dropout: 0.3
batch_size: 2048
max_epochs: 5
optimizer: torch.optim.AdamW
bbone_lr: 0.0001
head_lr: 0.0003
weight_decay: 0.01
minority_ratio: 0.05

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support