---
language:
- dv
license: apache-2.0
---

# BERT base for Dhivehi

Pretrained model on Dhivehi language using masked language modeling (MLM).

## Tokenizer

The *WordPiece* tokenizer uses several components:

* **Normalization**: lowercase and then NFKD unicode normalization.
* **Pretokenization**: splits by whitespace and punctuation.
* **Postprocessing**: single sentences are output in format `[CLS] sentence A [SEP]` and pair sentences in format `[CLS] sentence A [SEP] sentence B [SEP]`.

## Training

Training was performed over 16M+ Dhivehi sentences/paragraphs put together by [@ashraq](https://huggingface.co/ashraq). An Adam optimizer with weighted decay was used with following parameters:

* Learning rate: 1e-5
* Weight decay: 0.1
* Warmup steps: 10% of data