|
--- |
|
language: id |
|
datasets: |
|
- oscar |
|
--- |
|
# IndoBERT (Indonesian BERT Model) |
|
|
|
## Model description |
|
IndoBERT is a pre-trained language model based on BERT architecture for the Indonesian Language. |
|
|
|
This model is base-uncased version which use bert-base config. |
|
|
|
## Intended uses & limitations |
|
|
|
#### How to use |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
tokenizer = AutoTokenizer.from_pretrained("sarahlintang/IndoBERT") |
|
model = AutoModel.from_pretrained("sarahlintang/IndoBERT") |
|
tokenizer.encode("hai aku mau makan.") |
|
[2, 8078, 1785, 2318, 1946, 18, 4] |
|
``` |
|
|
|
|
|
## Training data |
|
|
|
This model was pre-trained on 16 GB of raw text ~2 B words from Oscar Corpus (https://oscar-corpus.com/). |
|
|
|
This model is equal to bert-base model which has 32,000 vocabulary size. |
|
|
|
## Training procedure |
|
|
|
The training of the model has been performed using Google’s original Tensorflow code on eight core Google Cloud TPU v2. |
|
We used a Google Cloud Storage bucket, for persistent storage of training data and models. |
|
|
|
## Eval results |
|
|
|
We evaluate this model on three Indonesian NLP downstream task: |
|
- some extractive summarization model |
|
- sentiment analysis |
|
- Part-of-Speech Tagger |
|
it was proven that this model outperforms multilingual BERT for all downstream tasks. |
|
|
