Create README.md

README.md

@@ -0,0 +1,66 @@
+---
+language: id
+tags:
+  - indonesian-roberta-base-indonli
+license: mit
+datasets:
+  - indonli
+widget:
+  - text: "Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih."
+---
+
+## Indonesian RoBERTa Base IndoNLI
+
+Indonesian RoBERTa Base IndoNLI is a natural language inference (NLI) model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`IndoNLI`](https://github.com/ir-nlp-csui/indonli)'s dataset consisting of Indonesian Wikipedia, news, and Web articles [1].
+
+After training, the model achieved an evaluation/dev accuracy of 77.06%. On the benchmark `test_lay` subset, the model achieved an accuracy of 74.24% and on the benchmark `test_expert` subset, the model achieved an accuracy of 61.66%.
+
+Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.
+
+## Model
+
+| Model                             | #params | Arch.        | Training/Validation data (text) |
+| --------------------------------- | ------- | ------------ | ------------------------------- |
+| `indonesian-roberta-base-indonli` | 124M    | RoBERTa Base | `IndoNLI`                       |
+
+## Evaluation Results
+
+The model was trained for 5 epochs, with a batch size of 16, a learning rate of 2e-5, a weight decay of 0.1, and a warmup ratio of 0.2, with linear annealing to 0. The best model was loaded at the end.
+
+| Epoch | Training Loss | Validation Loss | Accuracy |
+| ----- | ------------- | --------------- | -------- |
+| 1     | 0.989200      | 0.691663        | 0.731452 |
+| 2     | 0.673000      | 0.621913        | 0.766045 |
+| 3     | 0.449900      | 0.662543        | 0.770596 |
+| 4     | 0.293600      | 0.777059        | 0.768320 |
+| 5     | 0.194200      | 0.948068        | 0.764224 |
+
+## How to Use
+
+### As NLI Classifier
+
+```python
+from transformers import pipeline
+
+pretrained_name = "w11wo/indonesian-roberta-base-indonli"
+
+nlp = pipeline(
+    "sentiment-analysis",
+    model=pretrained_name,
+    tokenizer=pretrained_name
+)
+
+nlp("Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.")
+```
+
+## Disclaimer
+
+Do consider the biases which come from both the pre-trained RoBERTa model and the `IndoNLI` dataset that may be carried over into the results of this model.
+
+## References
+
+[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). [IndoNLI: A Natural Language Inference Dataset for Indonesian](https://arxiv.org/abs/2110.14566). _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
+
+## Author
+
+Indonesian RoBERTa Base IndoNLI was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.