File size: 1,846 Bytes
d34f966 967fbbf d34f966 967fbbf d34f966 967fbbf d34f966 967fbbf |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
---
language:
- ru
- myv
tags:
- erzya
- mordovian
- fill-mask
- pretraining
- embeddings
- masked-lm
- feature-extraction
- sentence-similarity
license: cc-by-sa-4.0
datasets:
- slone/myv_ru_2022
---
This is an Erzya (`myv`, cyrillic script) sentence encoder from the paper "The first neural machine translation system for the Erzya language".
It is based on [sentence-transformers/LaBSE](https://huggingface.co/sentence-transformers/LaBSE) ([license here](https://tfhub.dev/google/LaBSE/2)), but with updated vocabulary and checkpoint:
- Removed all tokens except the most popular ones for English or Russian;
- Added extra tokens for Erzya language;
- Fine-tuned on the [slone/myv_ru_2022](https://huggingface.co/slone/myv_ru_2022) corpus using a mixture of tasks:
- Cross-lingual distillation of sentence embeddings from the original LaBSE model, using the parallel `ru-myv` corpus;
- Masked language modelling on `myv` monolingual data;
- Sentence pair classification to distinguish correct `ru-myv` translations from random pairs.
The model can be used as a sentence encoder or a masked language modelling predictor for Erzya, or fine-tuned for any downstream NLU dask.
Sentence embeddings can be produced with the code below:
```python
import torch
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("slone/LaBSE-en-ru-myv-v1")
model = AutoModel.from_pretrained("slone/LaBSE-en-ru-myv-v1")
sentences = ["Hello World", "Привет Мир", "Шумбратадо Мастор"]
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
with torch.no_grad():
model_output = model(**encoded_input)
embeddings = model_output.pooler_output
embeddings = torch.nn.functional.normalize(embeddings)
print(embeddings.shape) # torch.Size([3, 768])
```
|