File size: 776 Bytes
1abd917 731f316 1abd917 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
---
language:
- dv
license: apache-2.0
---
# BERT base for Dhivehi
Pretrained model on Dhivehi language using masked language modeling (MLM).
## Tokenizer
The *WordPiece* tokenizer uses several components:
* **Normalization**: lowercase and then NFKD unicode normalization.
* **Pretokenization**: splits by whitespace and punctuation.
* **Postprocessing**: single sentences are output in format `[CLS] sentence A [SEP]` and pair sentences in format `[CLS] sentence A [SEP] sentence B [SEP]`.
## Training
Training was performed over 16M+ Dhivehi sentences/paragraphs put together by [@ashraq](https://huggingface.co/ashraq). An Adam optimizer with weighted decay was used with following parameters:
* Learning rate: 1e-5
* Weight decay: 0.1
* Warmup steps: 10% of data
|