Create README.md

README.md

@@ -0,0 +1,71 @@
+---
+language: id
+tags:
+  - indonesian-roberta-large
+license: mit
+datasets:
+  - oscar
+widget:
+  - text: "Budi telat ke sekolah karena ia <mask>."
+---
+
+## Indonesian RoBERTa Large
+
+Indonesian RoBERTa Large is a masked language model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. It was trained on the [OSCAR](https://huggingface.co/datasets/oscar) dataset, specifically the `unshuffled_deduplicated_id` subset. The model was trained from scratch and achieved an evaluation loss of 4.801 and an evaluation accuracy of 29.8%.
+
+This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by HuggingFace. All training was done on a TPUv3-8 VM, sponsored by the Google Cloud team.
+
+All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/flax-community/indonesian-roberta-large/tree/main) tab, as well as the [Training metrics](https://huggingface.co/flax-community/indonesian-roberta-large/tensorboard) logged via Tensorboard.
+
+## Model
+
+| Model                      | #params | Arch.   | Training/Validation data (text)            |
+| -------------------------- | ------- | ------- | ------------------------------------------ |
+| `indonesian-roberta-large` | 355M    | RoBERTa | OSCAR `unshuffled_deduplicated_id` Dataset |
+
+## Evaluation Results
+
+The model was trained for 10 epochs and the following is the final result once the training ended.
+
+| train loss | valid loss | valid accuracy | total time |
+| ---------- | ---------- | -------------- | ---------- |
+| 5.19       | 4.801      | 0.298          | 2:8:32:28  |
+
+## How to Use
+
+### As Masked Language Model
+
+```python
+from transformers import pipeline
+
+pretrained_name = "flax-community/indonesian-roberta-large"
+
+fill_mask = pipeline(
+    "fill-mask",
+    model=pretrained_name,
+    tokenizer=pretrained_name
+)
+
+fill_mask("Budi sedang <mask> di sekolah.")
+```
+
+### Feature Extraction in PyTorch
+
+```python
+from transformers import RobertaModel, RobertaTokenizerFast
+
+pretrained_name = "flax-community/indonesian-roberta-large"
+model = RobertaModel.from_pretrained(pretrained_name)
+tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)
+
+prompt = "Budi sedang berada di sekolah."
+encoded_input = tokenizer(prompt, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+## Team Members
+
+- Wilson Wongso ([@w11wo](https://hf.co/w11wo))
+- Steven Limcorn ([@stevenlimcorn](https://hf.co/stevenlimcorn))
+- Samsul Rahmadani ([@munggok](https://hf.co/munggok))
+- Chew Kok Wah ([@chewkokwah](https://hf.co/chewkokwah))