language: id
tags:
- indonesian-roberta-base-indonli
license: mit
datasets:
- indonli
widget:
- text: Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.
model-index:
- name: w11wo/indonesian-roberta-base-indonli
results:
- task:
type: natural-language-inference
name: Natural Language Inference
dataset:
name: indonli
type: indonli
config: indonli
split: test_expert
metrics:
- name: Accuracy
type: accuracy
value: 0.6072386058981233
verified: true
- name: Precision Macro
type: precision
value: 0.6304330508019023
verified: true
- name: Precision Micro
type: precision
value: 0.6072386058981233
verified: true
- name: Precision Weighted
type: precision
value: 0.6320495884503851
verified: true
- name: Recall Macro
type: recall
value: 0.6127303344145852
verified: true
- name: Recall Micro
type: recall
value: 0.6072386058981233
verified: true
- name: Recall Weighted
type: recall
value: 0.6072386058981233
verified: true
- name: F1 Macro
type: f1
value: 0.6010566054103847
verified: true
- name: F1 Micro
type: f1
value: 0.6072386058981233
verified: true
- name: F1 Weighted
type: f1
value: 0.5995456855334425
verified: true
- name: loss
type: loss
value: 1.157181739807129
verified: true
Indonesian RoBERTa Base IndoNLI
Indonesian RoBERTa Base IndoNLI is a natural language inference (NLI) model based on the RoBERTa model. The model was originally the pre-trained Indonesian RoBERTa Base model, which is then fine-tuned on IndoNLI's dataset consisting of Indonesian Wikipedia, news, and Web articles [1].
After training, the model achieved an evaluation/dev accuracy of 77.06%. On the benchmark test_lay subset, the model achieved an accuracy of 74.24% and on the benchmark test_expert subset, the model achieved an accuracy of 61.66%.
Hugging Face's Trainer class from the Transformers library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.
Model
| Model | #params | Arch. | Training/Validation data (text) |
|---|---|---|---|
indonesian-roberta-base-indonli |
124M | RoBERTa Base | IndoNLI |
Evaluation Results
The model was trained for 5 epochs, with a batch size of 16, a learning rate of 2e-5, a weight decay of 0.1, and a warmup ratio of 0.2, with linear annealing to 0. The best model was loaded at the end.
| Epoch | Training Loss | Validation Loss | Accuracy |
|---|---|---|---|
| 1 | 0.989200 | 0.691663 | 0.731452 |
| 2 | 0.673000 | 0.621913 | 0.766045 |
| 3 | 0.449900 | 0.662543 | 0.770596 |
| 4 | 0.293600 | 0.777059 | 0.768320 |
| 5 | 0.194200 | 0.948068 | 0.764224 |
How to Use
As NLI Classifier
from transformers import pipeline
pretrained_name = "w11wo/indonesian-roberta-base-indonli"
nlp = pipeline(
"sentiment-analysis",
model=pretrained_name,
tokenizer=pretrained_name
)
nlp("Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.")
Disclaimer
Do consider the biases which come from both the pre-trained RoBERTa model and the IndoNLI dataset that may be carried over into the results of this model.
References
[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). IndoNLI: A Natural Language Inference Dataset for Indonesian. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Author
Indonesian RoBERTa Base IndoNLI was trained and evaluated by Wilson Wongso. All computation and development are done on Google Colaboratory using their free GPU access.