|
--- |
|
language: id |
|
license: mit |
|
tags: |
|
- indonesian-roberta-base-indonli |
|
datasets: |
|
- indonli |
|
widget: |
|
- text: Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih. |
|
model-index: |
|
- name: w11wo/indonesian-roberta-base-indonli |
|
results: |
|
- task: |
|
type: natural-language-inference |
|
name: Natural Language Inference |
|
dataset: |
|
name: indonli |
|
type: indonli |
|
config: indonli |
|
split: test_expert |
|
metrics: |
|
- type: accuracy |
|
value: 0.6072386058981233 |
|
name: Accuracy |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzY4NDkwNTdlYjI4MzY3ZTk3NmZjYjA1MjE2YWQ5MjJjMGM3NTc1NWVjODQzNTc1ZTYyZWVmYmY5NTI3NWY1ZSIsInZlcnNpb24iOjF9.Aeo_Id90j2JtyApv3LvJHkQtHz-9wO4SNvTdb8O_pp0KFQGfWXnkgX2t2hafIUxSKmZbETIte-FaPbZ9AGZSDA |
|
- type: precision |
|
value: 0.6304330508019023 |
|
name: Precision Macro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2I3Y2RkNjU0NzlkYmJiNWYyZjZhNzIzZGE5ODU4NzYxYmQ0NTYxYzZkM2JiNTQwZTdkMmYxOTRmMDlmOGFkMiIsInZlcnNpb24iOjF9.iEt7Mq6a3TubFQfdC3OAxAiZDXp0bPGhN9JPzSfKl89_dxnKzDp0IrVzkt1HNLHR_S22Q75Tevqh3_G8Pp05Dg |
|
- type: precision |
|
value: 0.6072386058981233 |
|
name: Precision Micro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmI4ZmEwOTY0NTViNTM1ZjcwY2E2ODRmNGJiMTg2ZDJmZTgyNGUxM2UwNjZjYzVmYTcxZGY4OGY3OTI4MzcyMSIsInZlcnNpb24iOjF9.Jn1OPD1ZxkblCqKT1CfeUYOt5Xb6CL6C2ZENLmvfYNzh-p0oHcIBgapfbCHc89oMSR-FhjQk_ME8f8A3eyy6CA |
|
- type: precision |
|
value: 0.6320495884503851 |
|
name: Precision Weighted |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYWU1NmI5MTk3ODY4Y2M5ZjYyNzMwZjRlZTAzNjFmNmUxNzgxYjVhODNjZDAwOTQyNDBlZDJkNDYyNzRlYzBmOSIsInZlcnNpb24iOjF9.ItCi8SouqOtM3P7c0KN5ifRpGOr1090aqo4zX4aVSlVOTq0iQj9_c3z0B_UAzFcr0qW7ObnvuiD8D5d-9EzkBg |
|
- type: recall |
|
value: 0.6127303344145852 |
|
name: Recall Macro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTZmMGEzMGZjYTI2MzA0OGQ1Nzk5ZDRmNDNhMTRhOGMyM2I4Zjc2NzMzNmM2NjQ0YzVjZDY1Mzk1ZWI5Zjg2NiIsInZlcnNpb24iOjF9.fWCCNatB50nCptCbXopRjwxbWic6BvWIG6frUo_iXJVFXsi3Q_ik91_70fLgZc9NfhIpewpNoe4ETn0Gmps4Cw |
|
- type: recall |
|
value: 0.6072386058981233 |
|
name: Recall Micro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNGMwOWVmOWU4YzM2ZjMyY2NjNjljMDdjZTZmMjU5MjRiNDU0NmVhYWEzZmQ5MzUzMmRhMjdmZjhkNDU4ZTM2ZCIsInZlcnNpb24iOjF9.Sy2c29OhxT-x4UBSr9G7rfwtyqzYOX4KNRe2blonfOdKrqSfSEORY01Y67WweDiKdRvbECzI-DemJUXVtx-QCg |
|
- type: recall |
|
value: 0.6072386058981233 |
|
name: Recall Weighted |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODgwODAwMDNkZWJlMzYyYmQwYzNmMGZiMzhjZjVjNGYyMTg3OWVjNjZmMzFhMDczNGEyMDAyODkwZTZhZWM4NSIsInZlcnNpb24iOjF9.8IxdbjDQHzcNW71RAMtKHzlviweLTQvYVQ4JlrqoZsV-8gyzxpyYOmDjUm3n6uQNfRLRpyvsT-E8ysLHPyMqDg |
|
- type: f1 |
|
value: 0.6010566054103847 |
|
name: F1 Macro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmMwMjg2YjZmYzMzYzlmNTk3OGIyMDc5NGUxNTFlNmNkNmU3MzU2ZWMyZjY0OGFhYmY5YTUyNDkxNjJiODIwNSIsInZlcnNpb24iOjF9.r1ylajrOC-Qu4QNdNnXzisjGlczTF_9tYpNEr8LYdTtdmJtRjNtNmElINneuaWX7XGN9TExdzmg7OWTwutjsAg |
|
- type: f1 |
|
value: 0.6072386058981233 |
|
name: F1 Micro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDdhMGU1NDNjMzgxN2NjNWYzNTk5OWY5OTZhZWRjOGZkNjI4ZDA1YjI5ZWYxOWNmNDc4NmVhNjllMjUyMTFkMSIsInZlcnNpb24iOjF9.5G1km-a2_ssO_b3WTD8Ools29e6h8X8rjpClFN5Q_I4ADbPxKI2QbCfd5vl89CMHclignQ1_H6vqYbdTL9usDQ |
|
- type: f1 |
|
value: 0.5995456855334425 |
|
name: F1 Weighted |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMjYyNTZjOTUzNmMxOTY3MzQzNjMxZGNhYTY3NTQ4Mjg3NWRlMjc2NmY1NjMxOWY0NTFiODlhZjA3ZTEzNGQ3MSIsInZlcnNpb24iOjF9.3iTI9IieFa3WJFr7ovDvO24IPScGB7WQk3Pw_Qxh32zKx5QyOwmZf_p2bgbEG6hBeCkR0KaMDvIiZXnbW6DBDQ |
|
- type: loss |
|
value: 1.157181739807129 |
|
name: loss |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmIxNTQyYWRkMjgxMTZkN2JhZTg5NDFiMDRlZGEzOGE5ZDIwYTE5NTU4YmU0NDUxOTE1MDQwMzFlMjQ5MDQ2YSIsInZlcnNpb24iOjF9.M-U6Dp-I-DEXZ3qSGLxYrCdQjgXi6DotHDgz1acjWnHIZWKPApy-n2194FZik1Tpv2AcJVe45tDRLxNSW3zVBg |
|
--- |
|
|
|
## Indonesian RoBERTa Base IndoNLI |
|
|
|
Indonesian RoBERTa Base IndoNLI is a natural language inference (NLI) model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`IndoNLI`](https://github.com/ir-nlp-csui/indonli)'s dataset consisting of Indonesian Wikipedia, news, and Web articles [1]. |
|
|
|
After training, the model achieved an evaluation/dev accuracy of 77.06%. On the benchmark `test_lay` subset, the model achieved an accuracy of 74.24% and on the benchmark `test_expert` subset, the model achieved an accuracy of 61.66%. |
|
|
|
Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless. |
|
|
|
## Model |
|
|
|
| Model | #params | Arch. | Training/Validation data (text) | |
|
| --------------------------------- | ------- | ------------ | ------------------------------- | |
|
| `indonesian-roberta-base-indonli` | 124M | RoBERTa Base | `IndoNLI` | |
|
|
|
## Evaluation Results |
|
|
|
The model was trained for 5 epochs, with a batch size of 16, a learning rate of 2e-5, a weight decay of 0.1, and a warmup ratio of 0.2, with linear annealing to 0. The best model was loaded at the end. |
|
|
|
| Epoch | Training Loss | Validation Loss | Accuracy | |
|
| ----- | ------------- | --------------- | -------- | |
|
| 1 | 0.989200 | 0.691663 | 0.731452 | |
|
| 2 | 0.673000 | 0.621913 | 0.766045 | |
|
| 3 | 0.449900 | 0.662543 | 0.770596 | |
|
| 4 | 0.293600 | 0.777059 | 0.768320 | |
|
| 5 | 0.194200 | 0.948068 | 0.764224 | |
|
|
|
## How to Use |
|
|
|
### As NLI Classifier |
|
|
|
```python |
|
from transformers import pipeline |
|
|
|
pretrained_name = "w11wo/indonesian-roberta-base-indonli" |
|
|
|
nlp = pipeline( |
|
"sentiment-analysis", |
|
model=pretrained_name, |
|
tokenizer=pretrained_name |
|
) |
|
|
|
nlp("Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.") |
|
``` |
|
|
|
## Disclaimer |
|
|
|
Do consider the biases which come from both the pre-trained RoBERTa model and the `IndoNLI` dataset that may be carried over into the results of this model. |
|
|
|
## References |
|
|
|
[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). [IndoNLI: A Natural Language Inference Dataset for Indonesian](https://arxiv.org/abs/2110.14566). _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics. |
|
|
|
## Author |
|
|
|
Indonesian RoBERTa Base IndoNLI was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access. |
|
|
