|
--- |
|
license: apache-2.0 |
|
language: |
|
- ru |
|
- en |
|
library_name: transformers |
|
--- |
|
|
|
# RoBERTa-base from deepvk |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
Pretrained bidirectional encoder for russian language. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
Model was pretrained using standard MLM objective on a large text corpora including open social data, books, Wikipedia, webpages etc. |
|
|
|
|
|
- **Developed by:** VK Applied Research Team |
|
- **Model type:** RoBERTa |
|
- **Languages:** Mostly russian and small fraction of other languages |
|
- **License:** Apache 2.0 |
|
|
|
## How to Get Started with the Model |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("deepvk/roberta-base") |
|
model = AutoModel.from_pretrained("deepvk/roberta-base") |
|
|
|
text = "Привет, мир!" |
|
|
|
inputs = tokenizer(text, return_tensors='pt') |
|
predictions = model(**inputs) |
|
``` |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
500gb of raw texts in total. Mix of the following data: Wikipedia, Books, Twitter comments, Pikabu, Proza.ru, Film subtitles, |
|
News websites, Social corpus. |
|
|
|
### Training Procedure |
|
|
|
#### Training Hyperparameters |
|
|
|
| Argument | Value | |
|
|--------------------|----------------------| |
|
| Training regime | fp16 mixed precision | |
|
| Training framework | Fairseq | |
|
| Optimizer | Adam | |
|
| Adam betas | 0.9,0.98 | |
|
| Adam eps | 1e-6 | |
|
| Num training steps | 500k | |
|
|
|
Model was trained using 8xA100 for ~22 days. |
|
|
|
#### Architecture details |
|
|
|
Standard RoBERTa-base parameters: |
|
|
|
| Argument | Value | |
|
|-------------------------|----------------| |
|
|Activation function | gelu | |
|
|Attention dropout | 0.1 | |
|
|Dropout | 0.1 | |
|
|Encoder attention heads | 12 | |
|
|Encoder embed dim | 768 | |
|
|Encoder ffn embed dim | 3,072 | |
|
|Encoder layers | 12 | |
|
|Max positions | 512 | |
|
|Vocab size | 50266 | |
|
|Tokenizer type | Bete-level BPE | |
|
|
|
## Evaluation |
|
|
|
Russian Super Glue dev set. |
|
|
|
Best result across base size models in bold. |
|
|
|
| Модель | RCB | PARus | MuSeRC | TERRa | RUSSE | RWSD | DaNetQA | Результат | |
|
|------------------------------------------------------------------------|-----------|--------|---------|-------|---------|---------|---------|-----------| |
|
| [vk-roberta-base](https://huggingface.co/deepvk/roberta-base) | 0.46 | 0.56 | 0.679 | 0.769 | 0.960 | 0.569 | 0.658 | 0.665 | |
|
| [vk-deberta-distill](https://huggingface.co/deepvk/deberta-v1-distill) | 0.433 | 0.56 | 0.625 | 0.59 | 0.943 | 0.569 | 0.726 | 0.635 | |
|
| [vk-deberta-base](https://huggingface.co/deepvk/deberta-v1-base) | 0.450 |**0.61**|**0.722**| 0.704 | 0.948 | 0.578 |**0.76** |**0.682** | |
|
| [vk-bert-base](https://huggingface.co/deepvk/bert-base-uncased) | 0.467 | 0.57 | 0.587 | 0.704 | 0.953 |**0.583**| 0.737 | 0.657 | |
|
| [sber-bert-base](https://huggingface.co/ai-forever/ruBert-base) | **0.491** |**0.61**| 0.663 | 0.769 |**0.962**| 0.574 | 0.678 | 0.678 | |
|
| [sber-roberta-large](https://huggingface.co/ai-forever/ruRoberta-large)| 0.463 | 0.61 | 0.775 | 0.886 | 0.946 | 0.564 | 0.761 | 0.715 | |
|
|
