File size: 2,927 Bytes
058adc6 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
---
language: sa
---
# RoBERTa trained on Sanskrit (SanBERTa)
**Mode size** (after training): **340MB**
### Dataset:
[Wikipedia articles](https://www.kaggle.com/disisbig/sanskrit-wikipedia-articles) (used in [iNLTK](https://github.com/goru001/nlp-for-sanskrit)).
It contains evaluation set.
[Sanskrit scraps from CLTK](http://cltk.org/)
### Configuration
| Parameter | Value |
|---|---|
| `num_attention_heads` | 12 |
| `num_hidden_layers` | 6 |
| `hidden_size` | 768 |
| `vocab_size` | 29407 |
### Training :
- On TPU
- For language modelling
- Iteratively increasing `--block_size` from 128 to 256 over epochs
### Evaluation
|Metric| # Value |
|---|---|
|Perplexity (`block_size=256`)|4.04|
## Example of usage:
### For Embeddings
```
tokenizer = AutoTokenizer.from_pretrained("surajp/SanBERTa")
model = RobertaModel.from_pretrained("surajp/SanBERTa")
op = tokenizer.encode("इयं भाषा न केवलं भारतस्य अपि तु विश्वस्य प्राचीनतमा भाषा इति मन्यते।", return_tensors="pt")
ps = model(op)
ps[0].shape
```
```
'''
Output:
--------
torch.Size([1, 47, 768])
```
### For \<mask\> Prediction
```
from transformers import pipeline
fill_mask = pipeline(
"fill-mask",
model="surajp/SanBERTa",
tokenizer="surajp/SanBERTa"
)
## इयं भाषा न केवलं भारतस्य अपि तु विश्वस्य प्राचीनतमा भाषा इति मन्यते।
fill_mask("इयं भाषा न केवल<mask> भारतस्य अपि तु विश्वस्य प्राचीनतमा भाषा इति मन्यते।")
ps = model(torch.tensor(enc).unsqueeze(1))
print(ps[0].shape)
```
```
'''
Output:
--------
[{'score': 0.7516744136810303,
'sequence': '<s> इयं भाषा न केवलं भारतस्य अपि तु विश्वस्य प्राचीनतमा भाषा इति मन्यते।</s>',
'token': 280,
'token_str': 'à¤Ĥ'},
{'score': 0.06230105459690094,
'sequence': '<s> इयं भाषा न केवली भारतस्य अपि तु विश्वस्य प्राचीनतमा भाषा इति मन्यते।</s>',
'token': 289,
'token_str': 'à¥Ģ'},
{'score': 0.055410224944353104,
'sequence': '<s> इयं भाषा न केवला भारतस्य अपि तु विश्वस्य प्राचीनतमा भाषा इति मन्यते।</s>',
'token': 265,
'token_str': 'ा'},
...]
```
### It works!! 🎉 🎉 🎉
> Created by [Suraj Parmar/@parmarsuraj99](https://twitter.com/parmarsuraj99) | [LinkedIn](https://www.linkedin.com/in/parmarsuraj99/)
> Made with <span style="color: #e25555;">♥</span> in India
|