---
language: jv
tags:
- javanese-bert-small-imdb-classifier
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Dhuh Gusti, film iki elek banget. Aku getun ndelok !!!"
---

## Javanese BERT Small IMDB Classifier
Javanese BERT Small IMDB Classifier is a movie-classification model based on the [BERT model](https://arxiv.org/abs/1810.04805). It was trained on  Javanese IMDB movie reviews.

The model was originally [`w11wo/javanese-bert-small-imdb`](https://huggingface.co/w11wo/javanese-bert-small-imdb) which is then fine-tuned on the [`w11wo/imdb-javanese`](https://huggingface.co/datasets/w11wo/imdb-javanese) dataset consisting of Javanese IMDB movie reviews. It achieved an accuracy of 76.37% on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.

## Model
| Model                                 | #params  | Arch.          | Training/Validation data (text) |
|---------------------------------------|----------|----------------|---------------------------------|
| `javanese-bert-small-imdb-classifier` |   110M   |   BERT Small   | Javanese IMDB (47.5 MB of text) |

## Evaluation Results
The model was trained for 5 epochs and the following is the final result once the training ended.

| train loss | valid loss |  accuracy  | total time |
|------------|------------|------------|------------|
|    0.131   |    1.113   |   0.763    |    59:16   |

## How to Use
### As Text Classifier
```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-bert-small-imdb-classifier"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Film sing apik banget!")
```

## Disclaimer
Do consider the biases which came from the IMDB review that may be carried over into the results of this model.

## Author
Javanese BERT Small IMDB Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: id
tags:
  - indonesian-roberta-base-sentiment-classifier
license: mit
datasets:
  - indonlu
widget:
  - text: "Jangan sampai saya telpon bos saya ya!"
---

## Indonesian RoBERTa Base Sentiment Classifier

Indonesian RoBERTa Base Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indonlu`](https://hf.co/datasets/indonlu)'s `SmSA` dataset consisting of Indonesian comments and reviews.

After training, the model achieved an evaluation accuracy of 94.36% and F1-macro of 92.42%. On the benchmark test set, the model achieved an accuracy of 93.2% and F1-macro of 91.02%.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                                          | #params | Arch.        | Training/Validation data (text) |
| ---------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-sentiment-classifier` | 124M    | RoBERTa Base | `SmSA`                          |

## Evaluation Results

The model was trained for 5 epochs and the best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.342600      | 0.213551        | 0.928571 | 0.898539 | 0.909803  | 0.890694 |
| 2     | 0.190700      | 0.213466        | 0.934127 | 0.901135 | 0.925297  | 0.882757 |
| 3     | 0.125500      | 0.219539        | 0.942857 | 0.920901 | 0.927511  | 0.915193 |
| 4     | 0.083600      | 0.235232        | 0.943651 | 0.924227 | 0.926494  | 0.922048 |
| 5     | 0.059200      | 0.262473        | 0.942063 | 0.920583 | 0.924084  | 0.917351 |

## How to Use

### As Text Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-sentiment-classifier"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Jangan sampai saya telpon bos saya ya!")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and the `SmSA` dataset that may be carried over into the results of this model.

## Author

Indonesian RoBERTa Base Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: id
tags:
  - indonesian-roberta-base-indolem-sentiment-classifier-fold-2
license: mit
datasets:
  - indolem
widget:
  - text: "Pelayanan hotel ini sangat baik."
---

## Indonesian RoBERTa Base IndoLEM Sentiment Classifier

Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).

A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.

On **fold 2**, the model achieved an F1 of 83.05% on dev/validation and 82.89% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-2` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |

## Evaluation Results

The model was trained for 10 epochs and the best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.563300      | 0.436885        | 0.786967 | 0.568528 | 0.700000  | 0.478632 |
| 2     | 0.307800      | 0.301497        | 0.864662 | 0.763158 | 0.783784  | 0.743590 |
| 3     | 0.178900      | 0.287950        | 0.899749 | 0.818182 | 0.873786  | 0.769231 |
| 4     | 0.093100      | 0.345517        | 0.894737 | 0.820513 | 0.820513  | 0.820513 |
| 5     | 0.054500      | 0.434557        | 0.899749 | 0.826087 | 0.840708  | 0.811966 |
| 6     | 0.030900      | 0.487250        | 0.904762 | 0.827273 | 0.883495  | 0.777778 |
| 7     | 0.021700      | 0.494885        | 0.899749 | 0.830508 | 0.823529  | 0.837607 |
| 8     | 0.011500      | 0.580564        | 0.897243 | 0.814480 | 0.865385  | 0.769231 |
| 9     | 0.006000      | 0.604532        | 0.894737 | 0.820513 | 0.820513  | 0.820513 |
| 10    | 0.005000      | 0.616260        | 0.897243 | 0.822511 | 0.833333  | 0.811966 |

## How to Use

### As Text Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.

## Author

Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: jv
tags:
- javanese-gpt2-small
license: mit
datasets:
- wikipedia
widget:
- text: "Jenengku Budi, saka Indonesia"
---

## Javanese GPT-2 Small
Javanese GPT-2 Small is a language model based on the [GPT-2 model](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). It was trained on the latest (late December 2020) Javanese Wikipedia articles.

The model was originally HuggingFace's pretrained [English GPT-2 model](https://huggingface.co/transformers/model_doc/gpt2.html) and is later fine-tuned on the Javanese dataset. Many of the techniques used
are based on a [notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb)/[blog](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787) shared by [Pierre Guillou](https://medium.com/@pierre_guillou), where Pierre Guillou fine-tuned the English GPT-2 model on a Portuguese dataset.

Frameworks used include HuggingFace's [Transformers](https://huggingface.co/transformers) and fast.ai's [Deep Learning library](https://docs.fast.ai/). PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.

## Model
| Model                 | #params | Arch.       | Training /Validation data (text)    |
|-----------------------|---------|-------------|-------------------------------------|
| `javanese-gpt2-small` |  124M   | GPT-2 Small | Javanese Wikipedia (319 MB of text) |

## Evaluation Results
Before fine-tuning, the English GPT-2 model went through a validation step just to see how the model fairs prior to training.

| valid loss | perplexity |
|------------|------------|
|   10.845   |  51313.62  |

The model was then trained afterwards for 5 epochs and the following are the results.

| epoch | train loss | valid loss | perplexity | total time |
|-------|------------|------------|------------|------------|
|   0   |    4.336   |    4.110   |    60.94   |    22:28   |
|   1   |    3.598   |    3.543   |    34.58   |    23:27   |
|   2   |    3.161   |    3.331   |    27.98   |    24:17   |
|   3   |    2.974   |    3.265   |    26.18   |    25:03   |
|   4   |    2.932   |    3.234   |    25.39   |    25:06   |

## How to Use (PyTorch)
### Load Model and Byte-level Tokenizer
```python
from transformers import GPT2TokenizerFast, GPT2LMHeadModel
pretrained_name = "w11wo/javanese-gpt2-small"
tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_name)
tokenizer.model_max_length = 1024
model = GPT2LMHeadModel.from_pretrained(pretrained_name)
```

### Generate a Sequence
```python
# sample prompt
prompt = "Jenengku Budi, saka Indonesia"
input_ids = tokenizer.encode(prompt, return_tensors='pt')
model.eval()

# generate output using top-k sampling
sample_outputs = model.generate(input_ids,
                                pad_token_id=50256,
                                do_sample=True,
                                max_length=40,
                                min_length=40,
                                top_k=40,
                                num_return_sequences=1)

for i, sample_output in enumerate(sample_outputs):
    print(tokenizer.decode(sample_output.tolist()))
```

## Disclaimer
Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.

## Credits
Major thanks to Pierre Guillou for sharing his work, which did not only enable me to realize this project but also taught me tons of new, exciting stuff.

## Author
Javanese GPT-2 Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
license: apache-2.0
tags:
- automatic-speech-recognition
- kresnik/zeroth_korean
- generated_from_trainer
datasets:
- zeroth_korean_asr
model-index:
- name: ''
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

#

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the KRESNIK/ZEROTH_KOREAN - CLEAN dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0996
- Wer: 0.1022
- Cer: 0.0345

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:------:|
| 5.1239        | 0.72  | 500   | 4.8180          | 1.0    | 1.0    |
| 4.4011        | 1.44  | 1000  | 4.3980          | 0.9967 | 0.9385 |
| 4.1539        | 2.16  | 1500  | 4.1499          | 1.0171 | 0.7633 |
| 2.9662        | 2.88  | 2000  | 1.8843          | 0.9233 | 0.4671 |
| 2.4398        | 3.6   | 2500  | 1.1572          | 0.8259 | 0.3531 |
| 2.2207        | 4.32  | 3000  | 1.0066          | 0.7262 | 0.2824 |
| 2.1795        | 5.04  | 3500  | 0.8836          | 0.7523 | 0.3046 |
| 2.1027        | 5.75  | 4000  | 0.8822          | 0.6886 | 0.2540 |
| 2.0327        | 6.47  | 4500  | 0.8347          | 0.6906 | 0.2662 |
| 1.9621        | 7.19  | 5000  | 0.7796          | 0.6506 | 0.2385 |
| 1.9346        | 7.91  | 5500  | 0.7973          | 0.6873 | 0.2731 |
| 1.8845        | 8.63  | 6000  | 0.7259          | 0.6216 | 0.2169 |
| 1.8333        | 9.35  | 6500  | 0.6754          | 0.5926 | 0.2023 |
| 1.7733        | 10.07 | 7000  | 0.6137          | 0.5754 | 0.1977 |
| 1.721         | 10.79 | 7500  | 0.5678          | 0.5513 | 0.1827 |
| 1.6651        | 11.51 | 8000  | 0.5269          | 0.5148 | 0.1652 |
| 1.6084        | 12.23 | 8500  | 0.4882          | 0.4912 | 0.1602 |
| 1.5919        | 12.95 | 9000  | 0.4909          | 0.4928 | 0.1577 |
| 1.5654        | 13.67 | 9500  | 0.5053          | 0.5556 | 0.2023 |
| 1.5249        | 14.39 | 10000 | 0.4497          | 0.4645 | 0.1465 |
| 1.4697        | 15.11 | 10500 | 0.4128          | 0.5210 | 0.1800 |
| 1.4           | 15.83 | 11000 | 0.3737          | 0.4780 | 0.1609 |
| 1.3423        | 16.55 | 11500 | 0.3608          | 0.4059 | 0.1269 |
| 1.3118        | 17.27 | 12000 | 0.3304          | 0.4227 | 0.1382 |
| 1.2893        | 17.98 | 12500 | 0.3249          | 0.3644 | 0.1030 |
| 1.2752        | 18.7  | 13000 | 0.3277          | 0.3577 | 0.1053 |
| 1.2229        | 19.42 | 13500 | 0.3165          | 0.3778 | 0.1140 |
| 1.1846        | 20.14 | 14000 | 0.2839          | 0.3834 | 0.1320 |
| 1.165         | 20.86 | 14500 | 0.2809          | 0.4138 | 0.1662 |
| 1.119         | 21.58 | 15000 | 0.2561          | 0.3615 | 0.1289 |
| 1.1039        | 22.3  | 15500 | 0.2508          | 0.3173 | 0.1018 |
| 1.1019        | 23.02 | 16000 | 0.2687          | 0.3490 | 0.1204 |
| 1.0304        | 23.74 | 16500 | 0.2409          | 0.3303 | 0.1461 |
| 1.0399        | 24.46 | 17000 | 0.2501          | 0.3425 | 0.1229 |
| 0.9987        | 25.18 | 17500 | 0.2136          | 0.3334 | 0.1268 |
| 0.965         | 25.9  | 18000 | 0.2071          | 0.3247 | 0.1153 |
| 0.9424        | 26.62 | 18500 | 0.2079          | 0.3280 | 0.1238 |
| 0.9386        | 27.34 | 19000 | 0.2279          | 0.3286 | 0.1194 |
| 0.9087        | 28.06 | 19500 | 0.2040          | 0.3449 | 0.1399 |
| 0.8975        | 28.78 | 20000 | 0.1919          | 0.3399 | 0.1425 |
| 0.8631        | 29.5  | 20500 | 0.1790          | 0.3064 | 0.1238 |
| 0.8247        | 30.22 | 21000 | 0.1846          | 0.2307 | 0.0665 |
| 0.8254        | 30.93 | 21500 | 0.1870          | 0.2019 | 0.0534 |
| 0.8118        | 31.65 | 22000 | 0.1654          | 0.1915 | 0.0538 |
| 0.7761        | 32.37 | 22500 | 0.1626          | 0.2107 | 0.0705 |
| 0.7669        | 33.09 | 23000 | 0.1590          | 0.2342 | 0.0857 |
| 0.7305        | 33.81 | 23500 | 0.1490          | 0.1616 | 0.0434 |
| 0.7219        | 34.53 | 24000 | 0.1458          | 0.1578 | 0.0439 |
| 0.7024        | 35.25 | 24500 | 0.1457          | 0.1518 | 0.0421 |
| 0.6868        | 35.97 | 25000 | 0.1336          | 0.1527 | 0.0423 |
| 0.6761        | 36.69 | 25500 | 0.1290          | 0.1400 | 0.0403 |
| 0.6637        | 37.41 | 26000 | 0.1314          | 0.1439 | 0.0405 |
| 0.6312        | 38.13 | 26500 | 0.1303          | 0.1500 | 0.0473 |
| 0.6217        | 38.85 | 27000 | 0.1279          | 0.1439 | 0.0463 |
| 0.6213        | 39.57 | 27500 | 0.1254          | 0.1326 | 0.0387 |
| 0.5993        | 40.29 | 28000 | 0.1235          | 0.1729 | 0.0695 |
| 0.6075        | 41.01 | 28500 | 0.1230          | 0.1222 | 0.0356 |
| 0.5816        | 41.73 | 29000 | 0.1211          | 0.1646 | 0.0667 |
| 0.5619        | 42.45 | 29500 | 0.1175          | 0.1720 | 0.0752 |
| 0.541         | 43.17 | 30000 | 0.1114          | 0.1229 | 0.0395 |
| 0.5436        | 43.88 | 30500 | 0.1098          | 0.1216 | 0.0395 |
| 0.5441        | 44.6  | 31000 | 0.1116          | 0.1121 | 0.0340 |
| 0.5261        | 45.32 | 31500 | 0.1083          | 0.1093 | 0.0328 |
| 0.5184        | 46.04 | 32000 | 0.1053          | 0.1062 | 0.0336 |
| 0.506         | 46.76 | 32500 | 0.1046          | 0.1054 | 0.0353 |
| 0.5029        | 47.48 | 33000 | 0.1008          | 0.1025 | 0.0351 |
| 0.4856        | 48.2  | 33500 | 0.1011          | 0.1045 | 0.0353 |
| 0.496         | 48.92 | 34000 | 0.0995          | 0.1010 | 0.0345 |
| 0.4813        | 49.64 | 34500 | 0.0991          | 0.1021 | 0.0350 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


---
language: id
tags:
  - indonesian-roberta-base-indolem-sentiment-classifier-fold-4
license: mit
datasets:
  - indolem
widget:
  - text: "Pelayanan hotel ini sangat baik."
---

## Indonesian RoBERTa Base IndoLEM Sentiment Classifier

Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).

A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.

On **fold 4**, the model achieved an F1 of 84.21% on dev/validation and 85.36% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-4` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |

## Evaluation Results

The model was trained for 10 epochs and the best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.565600      | 0.441932        | 0.771930 | 0.542714 | 0.658537  | 0.461538 |
| 2     | 0.303300      | 0.302146        | 0.869674 | 0.785124 | 0.760000  | 0.811966 |
| 3     | 0.175300      | 0.337158        | 0.879699 | 0.769231 | 0.879121  | 0.683761 |
| 4     | 0.084000      | 0.343521        | 0.894737 | 0.815789 | 0.837838  | 0.794872 |
| 5     | 0.036700      | 0.511235        | 0.889724 | 0.821138 | 0.782946  | 0.863248 |
| 6     | 0.020300      | 0.628374        | 0.897243 | 0.812785 | 0.872549  | 0.760684 |
| 7     | 0.019400      | 0.609241        | 0.909774 | 0.842105 | 0.864865  | 0.820513 |
| 8     | 0.005400      | 0.623904        | 0.897243 | 0.825532 | 0.822034  | 0.829060 |
| 9     | 0.005300      | 0.641695        | 0.904762 | 0.836207 | 0.843478  | 0.829060 |
| 10    | 0.003900      | 0.645317        | 0.904762 | 0.836207 | 0.843478  | 0.829060 |

## How to Use

### As Text Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.

## Author

Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: id
tags:
  - indonesian-roberta-base-posp-tagger
license: mit
datasets:
  - indonlu
widget:
  - text: "Budi sedang pergi ke pasar."
---

## Indonesian RoBERTa Base POSP Tagger

Indonesian RoBERTa Base POSP Tagger is a part-of-speech token-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indonlu`](https://hf.co/datasets/indonlu)'s `POSP` dataset consisting of tag-labelled news.

After training, the model achieved an evaluation F1-macro of 95.34%. On the benchmark test set, the model achieved an accuracy of 93.99% and F1-macro of 88.93%.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                                 | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-posp-tagger` | 124M    | RoBERTa Base | `POSP`                          |

## Evaluation Results

The model was trained for 10 epochs and the best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Precision | Recall   | F1       | Accuracy |
| ----- | ------------- | --------------- | --------- | -------- | -------- | -------- |
| 1     | 0.898400      | 0.343731        | 0.894324  | 0.894324 | 0.894324 | 0.894324 |
| 2     | 0.294700      | 0.236619        | 0.929620  | 0.929620 | 0.929620 | 0.929620 |
| 3     | 0.214100      | 0.202723        | 0.938349  | 0.938349 | 0.938349 | 0.938349 |
| 4     | 0.171100      | 0.183630        | 0.945264  | 0.945264 | 0.945264 | 0.945264 |
| 5     | 0.143300      | 0.169744        | 0.948469  | 0.948469 | 0.948469 | 0.948469 |
| 6     | 0.124700      | 0.174946        | 0.947963  | 0.947963 | 0.947963 | 0.947963 |
| 7     | 0.109800      | 0.167450        | 0.951590  | 0.951590 | 0.951590 | 0.951590 |
| 8     | 0.101300      | 0.163191        | 0.952475  | 0.952475 | 0.952475 | 0.952475 |
| 9     | 0.093500      | 0.163255        | 0.953361  | 0.953361 | 0.953361 | 0.953361 |
| 10    | 0.089000      | 0.164673        | 0.953445  | 0.953445 | 0.953445 | 0.953445 |

## How to Use

### As Token Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-posp-tagger"

nlp = pipeline(
    "token-classification",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Budi sedang pergi ke pasar.")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and the `POSP` dataset that may be carried over into the results of this model.

## Author

Indonesian RoBERTa Base POSP Tagger was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: ko
license: apache-2.0
tags:
  - automatic-speech-recognition
  - generated_from_trainer
  - robust-speech-event
datasets:
  - kresnik/zeroth_korean
model-index:
  - name: Wav2Vec2 XLS-R 300M Korean
    results:
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Zeroth Korean
          type: kresnik/zeroth_korean
          args: clean
        metrics:
          - name: Test WER
            type: wer
            value: 29.54
          - name: Test CER
            type: cer
            value: 9.53
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Robust Speech Event - Dev Data
          type: speech-recognition-community-v2/dev_data
          args: ko
        metrics:
          - name: Test WER
            type: wer
            value: 76.26
          - name: Test CER
            type: cer
            value: 38.67
---

# Wav2Vec2 XLS-R 300M Korean

Wav2Vec2 XLS-R 300M Korean is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the [Zeroth Korean](https://huggingface.co/datasets/kresnik/zeroth_korean) dataset.

This model was trained using HuggingFace's PyTorch framework and is part of the [Robust Speech Challenge Event](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.

All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean/tensorboard) logged via Tensorboard.

## Model

| Model                        | #params | Arch. | Training/Validation data (text) |
| ---------------------------- | ------- | ----- | ------------------------------- |
| `wav2vec2-xls-r-300m-korean` | 300M    | XLS-R | `Zeroth Korean` Dataset         |

## Evaluation Results

The model achieves the following results on evaluation:

| Dataset                          | Loss   | WER    | CER    |
| -------------------------------- | ------ | ------ | ------ |
| `Zeroth Korean`                  | 0.2089 | 29.54% | 9.53%  |
| `Robust Speech Event - Dev Data` | N/A    | 76.26% | 38.67% |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:

- `learning_rate`: 7.5e-05
- `train_batch_size`: 8
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 4
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 2000
- `num_epochs`: 50.0
- `mixed_precision_training`: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    19.7138    | 0.72  |  500  |     19.6427     |  1.0   |  1.0   |
|    4.8039     | 1.44  | 1000  |     4.7842      |  1.0   |  1.0   |
|    4.5619     | 2.16  | 1500  |     4.5608      | 0.9992 | 0.9598 |
|     4.254     | 2.88  | 2000  |     4.2729      | 0.9955 | 0.9063 |
|    4.1905     |  3.6  | 2500  |     4.2257      | 0.9903 | 0.8758 |
|    4.0683     | 4.32  | 3000  |     3.9294      | 0.9937 | 0.7911 |
|     3.486     | 5.04  | 3500  |     2.7045      | 1.0012 | 0.5934 |
|     2.946     | 5.75  | 4000  |     1.9691      | 0.9425 | 0.4634 |
|     2.634     | 6.47  | 4500  |     1.5212      | 0.8807 | 0.3850 |
|    2.4066     | 7.19  | 5000  |     1.2551      | 0.8177 | 0.3601 |
|    2.2651     | 7.91  | 5500  |     1.0423      | 0.7650 | 0.3039 |
|    2.1828     | 8.63  | 6000  |     0.9599      | 0.7273 | 0.3106 |
|    2.1023     | 9.35  | 6500  |     0.9482      | 0.7161 | 0.3063 |
|    2.0536     | 10.07 | 7000  |     0.8242      | 0.6767 | 0.2860 |
|    1.9803     | 10.79 | 7500  |     0.7643      | 0.6563 | 0.2637 |
|    1.9468     | 11.51 | 8000  |     0.7319      | 0.6441 | 0.2505 |
|    1.9178     | 12.23 | 8500  |     0.6937      | 0.6320 | 0.2489 |
|    1.8515     | 12.95 | 9000  |     0.6443      | 0.6053 | 0.2196 |
|    1.8083     | 13.67 | 9500  |     0.6286      | 0.6122 | 0.2148 |
|     1.819     | 14.39 | 10000 |     0.6015      | 0.5986 | 0.2074 |
|    1.7684     | 15.11 | 10500 |     0.5682      | 0.5741 | 0.1982 |
|    1.7195     | 15.83 | 11000 |     0.5385      | 0.5592 | 0.2007 |
|    1.7044     | 16.55 | 11500 |     0.5362      | 0.5524 | 0.2097 |
|    1.6879     | 17.27 | 12000 |     0.5119      | 0.5489 | 0.2083 |
|     1.656     | 17.98 | 12500 |     0.4990      | 0.5362 | 0.1968 |
|    1.6122     | 18.7  | 13000 |     0.4561      | 0.5092 | 0.1900 |
|    1.5919     | 19.42 | 13500 |     0.4778      | 0.5225 | 0.1975 |
|    1.5896     | 20.14 | 14000 |     0.4563      | 0.5098 | 0.1859 |
|    1.5589     | 20.86 | 14500 |     0.4362      | 0.4940 | 0.1725 |
|    1.5353     | 21.58 | 15000 |     0.4140      | 0.4826 | 0.1580 |
|    1.5441     | 22.3  | 15500 |     0.4031      | 0.4742 | 0.1550 |
|    1.5116     | 23.02 | 16000 |     0.3916      | 0.4748 | 0.1545 |
|    1.4731     | 23.74 | 16500 |     0.3841      | 0.4810 | 0.1542 |
|    1.4647     | 24.46 | 17000 |     0.3752      | 0.4524 | 0.1475 |
|    1.4328     | 25.18 | 17500 |     0.3587      | 0.4476 | 0.1461 |
|    1.4129     | 25.9  | 18000 |     0.3429      | 0.4242 | 0.1366 |
|    1.4062     | 26.62 | 18500 |     0.3450      | 0.4251 | 0.1355 |
|    1.3928     | 27.34 | 19000 |     0.3297      | 0.4145 | 0.1322 |
|    1.3906     | 28.06 | 19500 |     0.3210      | 0.4185 | 0.1336 |
|     1.358     | 28.78 | 20000 |     0.3131      | 0.3970 | 0.1275 |
|    1.3445     | 29.5  | 20500 |     0.3069      | 0.3920 | 0.1276 |
|    1.3159     | 30.22 | 21000 |     0.3035      | 0.3961 | 0.1255 |
|    1.3044     | 30.93 | 21500 |     0.2952      | 0.3854 | 0.1242 |
|    1.3034     | 31.65 | 22000 |     0.2966      | 0.3772 | 0.1227 |
|    1.2963     | 32.37 | 22500 |     0.2844      | 0.3706 | 0.1208 |
|    1.2765     | 33.09 | 23000 |     0.2841      | 0.3567 | 0.1173 |
|    1.2438     | 33.81 | 23500 |     0.2734      | 0.3552 | 0.1137 |
|    1.2487     | 34.53 | 24000 |     0.2703      | 0.3502 | 0.1118 |
|    1.2249     | 35.25 | 24500 |     0.2650      | 0.3484 | 0.1142 |
|    1.2229     | 35.97 | 25000 |     0.2584      | 0.3374 | 0.1097 |
|    1.2374     | 36.69 | 25500 |     0.2568      | 0.3337 | 0.1095 |
|    1.2153     | 37.41 | 26000 |     0.2494      | 0.3327 | 0.1071 |
|    1.1925     | 38.13 | 26500 |     0.2518      | 0.3366 | 0.1077 |
|    1.1908     | 38.85 | 27000 |     0.2437      | 0.3272 | 0.1057 |
|    1.1858     | 39.57 | 27500 |     0.2396      | 0.3265 | 0.1044 |
|    1.1808     | 40.29 | 28000 |     0.2373      | 0.3156 | 0.1028 |
|    1.1842     | 41.01 | 28500 |     0.2356      | 0.3152 | 0.1026 |
|    1.1668     | 41.73 | 29000 |     0.2319      | 0.3188 | 0.1025 |
|    1.1448     | 42.45 | 29500 |     0.2293      | 0.3099 | 0.0995 |
|    1.1327     | 43.17 | 30000 |     0.2265      | 0.3047 | 0.0979 |
|    1.1307     | 43.88 | 30500 |     0.2222      | 0.3078 | 0.0989 |
|    1.1419     | 44.6  | 31000 |     0.2215      | 0.3038 | 0.0981 |
|    1.1231     | 45.32 | 31500 |     0.2193      | 0.3013 | 0.0972 |
|     1.139     | 46.04 | 32000 |     0.2162      | 0.3007 | 0.0968 |
|    1.1114     | 46.76 | 32500 |     0.2122      | 0.2982 | 0.0960 |
|     1.111     | 47.48 | 33000 |     0.2125      | 0.2946 | 0.0948 |
|    1.0982     | 48.2  | 33500 |     0.2099      | 0.2957 | 0.0953 |
|     1.109     | 48.92 | 34000 |     0.2092      | 0.2955 | 0.0955 |
|    1.0905     | 49.64 | 34500 |     0.2088      | 0.2954 | 0.0953 |

## Disclaimer

Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.

## Authors

Wav2Vec2 XLS-R 300M Korean was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.

## Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


---
language: jv
tags:
- javanese-roberta-small-imdb
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Aku bakal menehi rating film iki 5 <mask>."
---

## Javanese RoBERTa Small IMDB
Javanese RoBERTa Small IMDB is a masked language model based on the [RoBERTa model](https://arxiv.org/abs/1907.11692). It was trained on Javanese IMDB movie reviews.

The model was originally the pretrained [Javanese RoBERTa Small model](https://huggingface.co/w11wo/javanese-roberta-small) and is later fine-tuned on the Javanese IMDB movie review dataset. It achieved a perplexity of 20.83 on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.

## Model
| Model                         | #params  | Arch.             | Training/Validation data (text) |
|-------------------------------|----------|-------------------|---------------------------------|
| `javanese-roberta-small-imdb` |   124M   |   RoBERTa Small   | Javanese IMDB (47.5 MB of text) |

## Evaluation Results
The model was trained for 5 epochs and the following is the final result once the training ended.

| train loss | valid loss | perplexity | total time  |
|------------|------------|------------|-------------|
|    3.140   |    3.036   |   20.83    |   2:59:28   |

## How to Use
### As Masked Language Model
```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-roberta-small-imdb"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Aku mangan sate ing <mask> bareng konco-konco")
```
### Feature Extraction in PyTorch
```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "w11wo/javanese-roberta-small-imdb"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```

## Disclaimer
Do consider the biases which came from the IMDB review that may be carried over into the results of this model.

## Author
Javanese RoBERTa Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: zh-HK
license: apache-2.0
tags:
  - automatic-speech-recognition
  - generated_from_trainer
  - robust-speech-event
datasets:
  - common_voice
model-index:
  - name: Wav2Vec2 XLS-R 300M Cantonese (zh-HK) LM
    results:
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice
          type: common_voice
          args: zh-HK
        metrics:
          - name: Test CER
            type: cer
            value: 24.09
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 7
          type: mozilla-foundation/common_voice_7_0
          args: zh-HK
        metrics:
          - name: Test CER
            type: cer
            value: 23.10
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 8
          type: mozilla-foundation/common_voice_8_0
          args: zh-HK
        metrics:
          - name: Test CER
            type: cer
            value: 23.02
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Robust Speech Event - Dev Data
          type: speech-recognition-community-v2/dev_data
          args: zh-HK
        metrics:
          - name: Test CER
            type: cer
            value: 56.86
---

# Wav2Vec2 XLS-R 300M Cantonese (zh-HK) LM

Wav2Vec2 XLS-R 300M Cantonese (zh-HK) LM is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the `zh-HK` subset of the [Common Voice](https://huggingface.co/datasets/common_voice) dataset. A 5-gram Language model, trained on multiple [PyCantonese](https://pycantonese.org/data.html) corpora, was then subsequently added to this model.

This model was trained using HuggingFace's PyTorch framework and is part of the [Robust Speech Challenge Event](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.

All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-lm-v2/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-lm-v2/tensorboard) logged via Tensorboard.

As for the N-gram language model training, we followed the [blog post tutorial](https://huggingface.co/blog/wav2vec2-with-ngram) provided by HuggingFace.

## Model

| Model                             | #params | Arch. | Training/Validation data (text) |
| --------------------------------- | ------- | ----- | ------------------------------- |
| `wav2vec2-xls-r-300m-zh-HK-lm-v2` | 300M    | XLS-R | `Common Voice zh-HK` Dataset    |

## Evaluation Results

The model achieves the following results on evaluation without a language model:

| Dataset                          | CER    |
| -------------------------------- | ------ |
| `Common Voice`                   | 31.73% |
| `Common Voice 7`                 | 23.11% |
| `Common Voice 8`                 | 23.02% |
| `Robust Speech Event - Dev Data` | 56.60% |

With the addition of the language model, it achieves the following results:

| Dataset                          | CER    |
| -------------------------------- | ------ |
| `Common Voice`                   | 24.09% |
| `Common Voice 7`                 | 23.10% |
| `Common Voice 8`                 | 23.02% |
| `Robust Speech Event - Dev Data` | 56.86% |

## Training procedure

The training process did not involve the addition of a language model. The following results were simply lifted from the original automatic speech recognition [model training](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-v2).

### Training hyperparameters

The following hyperparameters were used during training:

- `learning_rate`: 0.0001
- `train_batch_size`: 8
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 4
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 2000
- `num_epochs`: 100.0
- `mixed_precision_training`: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    69.8341    | 1.34  |  500  |     80.0722     |  1.0   |  1.0   |
|    6.6418     | 2.68  | 1000  |     6.6346      |  1.0   |  1.0   |
|    6.2419     | 4.02  | 1500  |     6.2909      |  1.0   |  1.0   |
|    6.0813     | 5.36  | 2000  |     6.1150      |  1.0   |  1.0   |
|    5.9677     |  6.7  | 2500  |     6.0301      | 1.1386 | 1.0028 |
|    5.9296     | 8.04  | 3000  |     5.8975      | 1.2113 | 1.0058 |
|    5.6434     | 9.38  | 3500  |     5.5404      | 2.1624 | 1.0171 |
|    5.1974     | 10.72 | 4000  |     4.5440      | 2.1702 | 0.9366 |
|    4.3601     | 12.06 | 4500  |     3.3839      | 2.2464 | 0.8998 |
|    3.9321     | 13.4  | 5000  |     2.8785      | 2.3097 | 0.8400 |
|    3.6462     | 14.74 | 5500  |     2.5108      | 1.9623 | 0.6663 |
|    3.5156     | 16.09 | 6000  |     2.2790      | 1.6479 | 0.5706 |
|     3.32      | 17.43 | 6500  |     2.1450      | 1.8337 | 0.6244 |
|    3.1918     | 18.77 | 7000  |     1.8536      | 1.9394 | 0.6017 |
|    3.1139     | 20.11 | 7500  |     1.7205      | 1.9112 | 0.5638 |
|    2.8995     | 21.45 | 8000  |     1.5478      | 1.0624 | 0.3250 |
|    2.7572     | 22.79 | 8500  |     1.4068      | 1.1412 | 0.3367 |
|    2.6881     | 24.13 | 9000  |     1.3312      | 2.0100 | 0.5683 |
|    2.5993     | 25.47 | 9500  |     1.2553      | 2.0039 | 0.6450 |
|    2.5304     | 26.81 | 10000 |     1.2422      | 2.0394 | 0.5789 |
|    2.4352     | 28.15 | 10500 |     1.1582      | 1.9970 | 0.5507 |
|    2.3795     | 29.49 | 11000 |     1.1160      | 1.8255 | 0.4844 |
|    2.3287     | 30.83 | 11500 |     1.0775      | 1.4123 | 0.3780 |
|    2.2622     | 32.17 | 12000 |     1.0704      | 1.7445 | 0.4894 |
|    2.2225     | 33.51 | 12500 |     1.0272      | 1.7237 | 0.5058 |
|    2.1843     | 34.85 | 13000 |     0.9756      | 1.8042 | 0.5028 |
|      2.1      | 36.19 | 13500 |     0.9527      | 1.8909 | 0.6055 |
|    2.0741     | 37.53 | 14000 |     0.9418      | 1.9026 | 0.5880 |
|    2.0179     | 38.87 | 14500 |     0.9363      | 1.7977 | 0.5246 |
|    2.0615     | 40.21 | 15000 |     0.9635      | 1.8112 | 0.5599 |
|    1.9448     | 41.55 | 15500 |     0.9249      | 1.7250 | 0.4914 |
|    1.8966     | 42.89 | 16000 |     0.9023      | 1.5829 | 0.4319 |
|    1.8662     | 44.24 | 16500 |     0.9002      | 1.4833 | 0.4230 |
|    1.8136     | 45.58 | 17000 |     0.9076      | 1.1828 | 0.2987 |
|    1.7908     | 46.92 | 17500 |     0.8774      | 1.5773 | 0.4258 |
|    1.7354     | 48.26 | 18000 |     0.8727      | 1.5037 | 0.4024 |
|    1.6739     | 49.6  | 18500 |     0.8636      | 1.1239 | 0.2789 |
|    1.6457     | 50.94 | 19000 |     0.8516      | 1.2269 | 0.3104 |
|    1.5847     | 52.28 | 19500 |     0.8399      | 1.3309 | 0.3360 |
|    1.5971     | 53.62 | 20000 |     0.8441      | 1.3153 | 0.3335 |
|     1.602     | 54.96 | 20500 |     0.8590      | 1.2932 | 0.3433 |
|    1.5063     | 56.3  | 21000 |     0.8334      | 1.1312 | 0.2875 |
|    1.4631     | 57.64 | 21500 |     0.8474      | 1.1698 | 0.2999 |
|    1.4997     | 58.98 | 22000 |     0.8638      | 1.4279 | 0.3854 |
|    1.4301     | 60.32 | 22500 |     0.8550      | 1.2737 | 0.3300 |
|    1.3798     | 61.66 | 23000 |     0.8266      | 1.1802 | 0.2934 |
|    1.3454     | 63.0  | 23500 |     0.8235      | 1.3816 | 0.3711 |
|    1.3678     | 64.34 | 24000 |     0.8550      | 1.6427 | 0.5035 |
|    1.3761     | 65.68 | 24500 |     0.8510      | 1.6709 | 0.4907 |
|    1.2668     | 67.02 | 25000 |     0.8515      | 1.5842 | 0.4505 |
|    1.2835     | 68.36 | 25500 |     0.8283      | 1.5353 | 0.4221 |
|    1.2961     | 69.7  | 26000 |     0.8339      | 1.5743 | 0.4369 |
|    1.2656     | 71.05 | 26500 |     0.8331      | 1.5331 | 0.4217 |
|    1.2556     | 72.39 | 27000 |     0.8242      | 1.4708 | 0.4109 |
|    1.2043     | 73.73 | 27500 |     0.8245      | 1.4469 | 0.4031 |
|    1.2722     | 75.07 | 28000 |     0.8202      | 1.4924 | 0.4096 |
|     1.202     | 76.41 | 28500 |     0.8290      | 1.3807 | 0.3719 |
|    1.1679     | 77.75 | 29000 |     0.8195      | 1.4097 | 0.3749 |
|    1.1967     | 79.09 | 29500 |     0.8059      | 1.2074 | 0.3077 |
|    1.1241     | 80.43 | 30000 |     0.8137      | 1.2451 | 0.3270 |
|    1.1414     | 81.77 | 30500 |     0.8117      | 1.2031 | 0.3121 |
|     1.132     | 83.11 | 31000 |     0.8234      | 1.4266 | 0.3901 |
|    1.0982     | 84.45 | 31500 |     0.8064      | 1.3712 | 0.3607 |
|    1.0797     | 85.79 | 32000 |     0.8167      | 1.3356 | 0.3562 |
|    1.0119     | 87.13 | 32500 |     0.8215      | 1.2754 | 0.3268 |
|    1.0216     | 88.47 | 33000 |     0.8163      | 1.2512 | 0.3184 |
|    1.0375     | 89.81 | 33500 |     0.8137      | 1.2685 | 0.3290 |
|    0.9794     | 91.15 | 34000 |     0.8220      | 1.2724 | 0.3255 |
|    1.0207     | 92.49 | 34500 |     0.8165      | 1.2906 | 0.3361 |
|    1.0169     | 93.83 | 35000 |     0.8153      | 1.2819 | 0.3305 |
|    1.0127     | 95.17 | 35500 |     0.8187      | 1.2832 | 0.3252 |
|    0.9978     | 96.51 | 36000 |     0.8111      | 1.2612 | 0.3210 |
|    0.9923     | 97.85 | 36500 |     0.8076      | 1.2278 | 0.3122 |
|    1.0451     | 99.2  | 37000 |     0.8086      | 1.2451 | 0.3156 |

## Disclaimer

Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.

## Authors

Wav2Vec2 XLS-R 300M Cantonese (zh-HK) LM was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.

## Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


---
language: id
tags:
- indo-roberta-small
license: mit
datasets:
- wikipedia
widget:
- text: "Karena pandemi ini, kita harus <mask> di rumah saja."
---

## Indo RoBERTa Small
Indo RoBERTa Small is a masked language model based on the [RoBERTa model](https://arxiv.org/abs/1907.11692). It was trained on the latest (late December 2020) Indonesian Wikipedia articles.

The model was trained from scratch and achieved a perplexity of 48.27 on the validation dataset (20% of the articles). Many of the techniques used
are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), where Sylvain Gugger fine-tuned a [DistilGPT-2](https://huggingface.co/distilgpt2) on [Wikitext2](https://render.githubusercontent.com/view/ipynb?color_mode=dark&commit=43d63e390e8a82f7ae49aa1a877419343a213cb4&enc_url=68747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f68756767696e67666163652f6e6f7465626f6f6b732f343364363365333930653861383266376165343961613161383737343139333433613231336362342f6578616d706c65732f6c616e67756167655f6d6f64656c696e672e6970796e62&nwo=huggingface%2Fnotebooks&path=examples%2Flanguage_modeling.ipynb&repository_id=272452525&repository_type=Repository).

Hugging Face's [Transformers]((https://huggingface.co/transformers)) library was used to train the model -- utilizing the base RoBERTa model and their `Trainer` class. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.

## Model
| Model                | #params | Arch.    | Training/Validation data (text)      |
|----------------------|---------|----------|---------------------------------------|
| `indo-roberta-small` |   84M   | RoBERTa  | Indonesian Wikipedia (3.1 GB of text) |

## Evaluation Results
The model was trained for 3 epochs and the following is the final result once the training ended.

| train loss | valid loss | perplexity | total time |
|------------|------------|------------|------------|
|    4.071   |    3.876   |    48.27   |   3:40:55  |

## How to Use
### As Masked Language Model
```python
from transformers import pipeline

pretrained_name = "w11wo/indo-roberta-small"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Budi sedang <mask> di sekolah.")
```
### Feature Extraction in PyTorch
```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "w11wo/indo-roberta-small"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "Budi sedang berada di sekolah."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```

## Disclaimer
Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.

## Author
Indo RoBERTa Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: id
tags:
- indo-gpt2-small
license: mit
datasets:
- wikipedia
widget:
- text: "Nama saya Budi, dari Indonesia"
---

## Indo GPT-2 Small
Indo GPT-2 Small is a language model based on the [GPT-2 model](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). It was trained on the latest (late December 2020) Indonesian Wikipedia articles.

The model was originally HuggingFace's pretrained [English GPT-2 model](https://huggingface.co/transformers/model_doc/gpt2.html) and is later fine-tuned on the Indonesian dataset. Many of the techniques used
are based on a [notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb)/[blog](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787) shared by [Pierre Guillou](https://medium.com/@pierre_guillou), where Pierre Guillou fine-tuned the English GPT-2 model on a Portuguese dataset.

Frameworks used include HuggingFace's [Transformers](https://huggingface.co/transformers) and fast.ai's [Deep Learning library](https://docs.fast.ai/). PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.

## Model
| Model             | #params | Arch.       | Training /Validation data (text)      |
|-------------------|---------|-------------|---------------------------------------|
| `indo-gpt2-small` |  124M   | GPT-2 Small | Indonesian Wikipedia (3.1 GB of text) |

## Evaluation Results
The model was trained for only 1 epoch and the following is the final result once the training ended.

| epoch | train loss | valid loss | perplexity | total time |
|-------|------------|------------|------------|------------|
|   0   |    2.981   |    2.936   |    18.85   |   2:45:25  |

## How to Use (PyTorch)
### Load Model and Byte-level Tokenizer
```python
from transformers import GPT2TokenizerFast, GPT2LMHeadModel
pretrained_name = "w11wo/indo-gpt2-small"
tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_name)
tokenizer.model_max_length = 1024
model = GPT2LMHeadModel.from_pretrained(pretrained_name)
```

### Generate a Sequence
```python
# sample prompt
prompt = "Nama saya Budi, dari Indonesia"
input_ids = tokenizer.encode(prompt, return_tensors='pt')
model.eval()

# generate output using top-k sampling
sample_outputs = model.generate(input_ids,
                                pad_token_id=50256,
                                do_sample=True,
                                max_length=40,
                                min_length=40,
                                top_k=40,
                                num_return_sequences=1)

for i, sample_output in enumerate(sample_outputs):
    print(tokenizer.decode(sample_output.tolist()))
```

## Disclaimer
Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.

## Credits
Major thanks to Pierre Guillou for sharing his work, which did not only enable me to realize this project but also taught me tons of new, exciting stuff.

## Author
Indo GPT-2 Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
license: apache-2.0
tags:
- automatic-speech-recognition
- kresnik/zeroth_korean
- generated_from_trainer
datasets:
- zeroth_korean_asr
model-index:
- name: ''
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

#

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the KRESNIK/ZEROTH_KOREAN - CLEAN dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1313
- Wer: 0.2261
- Cer: 0.0899

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:------:|
| 5.0816        | 1.44  | 500   | 4.7941          | 1.0    | 1.0    |
| 4.2863        | 2.88  | 1000  | 4.2720          | 0.9961 | 0.9396 |
| 3.3962        | 4.32  | 1500  | 2.4010          | 1.0172 | 0.5761 |
| 2.4739        | 5.76  | 2000  | 1.3089          | 0.8718 | 0.4025 |
| 2.2215        | 7.2   | 2500  | 0.9522          | 0.6912 | 0.2600 |
| 2.0338        | 8.64  | 3000  | 0.8950          | 0.7262 | 0.2943 |
| 1.9666        | 10.09 | 3500  | 0.7557          | 0.6773 | 0.2694 |
| 1.8599        | 11.53 | 4000  | 0.6478          | 0.6161 | 0.2182 |
| 1.7943        | 12.97 | 4500  | 0.6073          | 0.6042 | 0.2170 |
| 1.7241        | 14.41 | 5000  | 0.5658          | 0.5344 | 0.1768 |
| 1.6469        | 15.85 | 5500  | 0.5329          | 0.5388 | 0.1854 |
| 1.5917        | 17.29 | 6000  | 0.5348          | 0.5577 | 0.2048 |
| 1.5364        | 18.73 | 6500  | 0.5058          | 0.5371 | 0.1946 |
| 1.4765        | 20.17 | 7000  | 0.4140          | 0.4248 | 0.1317 |
| 1.3973        | 21.61 | 7500  | 0.3850          | 0.4077 | 0.1183 |
| 1.3456        | 23.05 | 8000  | 0.3349          | 0.4035 | 0.1275 |
| 1.2933        | 24.49 | 8500  | 0.3296          | 0.3612 | 0.1013 |
| 1.2364        | 25.93 | 9000  | 0.3012          | 0.3248 | 0.0902 |
| 1.1763        | 27.38 | 9500  | 0.2968          | 0.3117 | 0.0868 |
| 1.1328        | 28.82 | 10000 | 0.2587          | 0.3096 | 0.0866 |
| 1.0703        | 30.26 | 10500 | 0.2327          | 0.3248 | 0.1120 |
| 1.021         | 31.7  | 11000 | 0.2210          | 0.3345 | 0.1215 |
| 0.9959        | 33.14 | 11500 | 0.2023          | 0.3201 | 0.1151 |
| 0.9495        | 34.58 | 12000 | 0.1985          | 0.2995 | 0.1192 |
| 0.921         | 36.02 | 12500 | 0.1924          | 0.3011 | 0.1218 |
| 0.8567        | 37.46 | 13000 | 0.1758          | 0.2679 | 0.0983 |
| 0.8448        | 38.9  | 13500 | 0.1602          | 0.2662 | 0.1027 |
| 0.8111        | 40.34 | 14000 | 0.1604          | 0.2596 | 0.1015 |
| 0.7823        | 41.78 | 14500 | 0.1475          | 0.2472 | 0.0980 |
| 0.765         | 43.23 | 15000 | 0.1453          | 0.2384 | 0.0939 |
| 0.7375        | 44.67 | 15500 | 0.1406          | 0.2327 | 0.0883 |
| 0.7233        | 46.11 | 16000 | 0.1377          | 0.2295 | 0.0892 |
| 0.6949        | 47.55 | 16500 | 0.1325          | 0.2304 | 0.0917 |
| 0.6838        | 48.99 | 17000 | 0.1321          | 0.2280 | 0.0906 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


---
language: lo
tags:
  - lao-roberta-base-pos-tagger
license: mit
widget:
  - text: "ຮ້ອງ ມ່ວນ ແທ້ ສຽງດີ ອິຫຼີ"
---

## Lao RoBERTa Base POS Tagger

Lao RoBERTa Base POS Tagger is a part-of-speech token-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Lao RoBERTa Base](https://huggingface.co/w11wo/lao-roberta-base) model, which is then fine-tuned on the [`Yunshan Cup 2020`](https://github.com/GKLMIP/Yunshan-Cup-2020) dataset consisting of tag-labelled Lao corpus.

After training, the model achieved an evaluation accuracy of 83.14%. On the benchmark test set, the model achieved an accuracy of 83.30%.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                         | #params | Arch.        | Training/Validation data (text) |
| ----------------------------- | ------- | ------------ | ------------------------------- |
| `lao-roberta-base-pos-tagger` | 124M    | RoBERTa Base | `Yunshan Cup 2020`              |

## Evaluation Results

The model was trained for 15 epochs, with a batch size of 8, a learning rate of 5e-5, with cosine annealing to 0. The best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Accuracy |
| ----- | ------------- | --------------- | -------- |
| 1     | 1.026100      | 0.733780        | 0.746021 |
| 2     | 0.646900      | 0.659625        | 0.775688 |
| 3     | 0.500400      | 0.576214        | 0.798523 |
| 4     | 0.385400      | 0.606503        | 0.805269 |
| 5     | 0.288000      | 0.652493        | 0.809092 |
| 6     | 0.204600      | 0.671678        | 0.815216 |
| 7     | 0.145200      | 0.704693        | 0.818209 |
| 8     | 0.098700      | 0.830561        | 0.816998 |
| 9     | 0.066100      | 0.883329        | 0.825232 |
| 10    | 0.043900      | 0.933347        | 0.825664 |
| 11    | 0.027200      | 0.992055        | 0.828449 |
| 12    | 0.017300      | 1.054874        | 0.830819 |
| 13    | 0.011500      | 1.081638        | 0.830940 |
| 14    | 0.008500      | 1.094252        | 0.831304 |
| 15    | 0.007400      | 1.097428        | 0.831442 |

## How to Use

### As Token Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/lao-roberta-base-pos-tagger"

nlp = pipeline(
    "token-classification",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("ຮ້ອງ ມ່ວນ ແທ້ ສຽງດີ ອິຫຼີ")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and the `Yunshan Cup 2020` dataset that may be carried over into the results of this model.

## Author

Lao RoBERTa Base POS Tagger was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: zh-HK
license: apache-2.0
tags:
  - automatic-speech-recognition
  - generated_from_trainer
  - robust-speech-event
datasets:
  - common_voice
model-index:
  - name: Wav2Vec2 XLS-R 300M Cantonese (zh-HK)
    results:
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice
          type: common_voice
          args: zh-HK
        metrics:
          - name: Test CER
            type: cer
            value: 31.73
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 7
          type: mozilla-foundation/common_voice_7_0
          args: zh-HK
        metrics:
          - name: Test CER
            type: cer
            value: 23.11
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 8
          type: mozilla-foundation/common_voice_8_0
          args: zh-HK
        metrics:
          - name: Test CER
            type: cer
            value: 23.02
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Robust Speech Event - Dev Data
          type: speech-recognition-community-v2/dev_data
          args: zh-HK
        metrics:
          - name: Test CER
            type: cer
            value: 56.60
---

# Wav2Vec2 XLS-R 300M Cantonese (zh-HK)

Wav2Vec2 XLS-R 300M Cantonese (zh-HK) is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the `zh-HK` subset of the [Common Voice](https://huggingface.co/datasets/common_voice) dataset.

This model was trained using HuggingFace's PyTorch framework and is part of the [Robust Speech Challenge Event](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.

All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-v2/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-v2/tensorboard) logged via Tensorboard.

## Model

| Model                          | #params | Arch. | Training/Validation data (text) |
| ------------------------------ | ------- | ----- | ------------------------------- |
| `wav2vec2-xls-r-300m-zh-HK-v2` | 300M    | XLS-R | `Common Voice zh-HK` Dataset    |

## Evaluation Results

The model achieves the following results on evaluation:

| Dataset                          | Loss   | CER    |
| -------------------------------- | ------ | ------ |
| `Common Voice`                   | 0.8089 | 31.73% |
| `Common Voice 7`                 | N/A    | 23.11% |
| `Common Voice 8`                 | N/A    | 23.02% |
| `Robust Speech Event - Dev Data` | N/A    | 56.60% |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:

- `learning_rate`: 0.0001
- `train_batch_size`: 8
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 4
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 2000
- `num_epochs`: 100.0
- `mixed_precision_training`: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    69.8341    | 1.34  |  500  |     80.0722     |  1.0   |  1.0   |
|    6.6418     | 2.68  | 1000  |     6.6346      |  1.0   |  1.0   |
|    6.2419     | 4.02  | 1500  |     6.2909      |  1.0   |  1.0   |
|    6.0813     | 5.36  | 2000  |     6.1150      |  1.0   |  1.0   |
|    5.9677     |  6.7  | 2500  |     6.0301      | 1.1386 | 1.0028 |
|    5.9296     | 8.04  | 3000  |     5.8975      | 1.2113 | 1.0058 |
|    5.6434     | 9.38  | 3500  |     5.5404      | 2.1624 | 1.0171 |
|    5.1974     | 10.72 | 4000  |     4.5440      | 2.1702 | 0.9366 |
|    4.3601     | 12.06 | 4500  |     3.3839      | 2.2464 | 0.8998 |
|    3.9321     | 13.4  | 5000  |     2.8785      | 2.3097 | 0.8400 |
|    3.6462     | 14.74 | 5500  |     2.5108      | 1.9623 | 0.6663 |
|    3.5156     | 16.09 | 6000  |     2.2790      | 1.6479 | 0.5706 |
|     3.32      | 17.43 | 6500  |     2.1450      | 1.8337 | 0.6244 |
|    3.1918     | 18.77 | 7000  |     1.8536      | 1.9394 | 0.6017 |
|    3.1139     | 20.11 | 7500  |     1.7205      | 1.9112 | 0.5638 |
|    2.8995     | 21.45 | 8000  |     1.5478      | 1.0624 | 0.3250 |
|    2.7572     | 22.79 | 8500  |     1.4068      | 1.1412 | 0.3367 |
|    2.6881     | 24.13 | 9000  |     1.3312      | 2.0100 | 0.5683 |
|    2.5993     | 25.47 | 9500  |     1.2553      | 2.0039 | 0.6450 |
|    2.5304     | 26.81 | 10000 |     1.2422      | 2.0394 | 0.5789 |
|    2.4352     | 28.15 | 10500 |     1.1582      | 1.9970 | 0.5507 |
|    2.3795     | 29.49 | 11000 |     1.1160      | 1.8255 | 0.4844 |
|    2.3287     | 30.83 | 11500 |     1.0775      | 1.4123 | 0.3780 |
|    2.2622     | 32.17 | 12000 |     1.0704      | 1.7445 | 0.4894 |
|    2.2225     | 33.51 | 12500 |     1.0272      | 1.7237 | 0.5058 |
|    2.1843     | 34.85 | 13000 |     0.9756      | 1.8042 | 0.5028 |
|      2.1      | 36.19 | 13500 |     0.9527      | 1.8909 | 0.6055 |
|    2.0741     | 37.53 | 14000 |     0.9418      | 1.9026 | 0.5880 |
|    2.0179     | 38.87 | 14500 |     0.9363      | 1.7977 | 0.5246 |
|    2.0615     | 40.21 | 15000 |     0.9635      | 1.8112 | 0.5599 |
|    1.9448     | 41.55 | 15500 |     0.9249      | 1.7250 | 0.4914 |
|    1.8966     | 42.89 | 16000 |     0.9023      | 1.5829 | 0.4319 |
|    1.8662     | 44.24 | 16500 |     0.9002      | 1.4833 | 0.4230 |
|    1.8136     | 45.58 | 17000 |     0.9076      | 1.1828 | 0.2987 |
|    1.7908     | 46.92 | 17500 |     0.8774      | 1.5773 | 0.4258 |
|    1.7354     | 48.26 | 18000 |     0.8727      | 1.5037 | 0.4024 |
|    1.6739     | 49.6  | 18500 |     0.8636      | 1.1239 | 0.2789 |
|    1.6457     | 50.94 | 19000 |     0.8516      | 1.2269 | 0.3104 |
|    1.5847     | 52.28 | 19500 |     0.8399      | 1.3309 | 0.3360 |
|    1.5971     | 53.62 | 20000 |     0.8441      | 1.3153 | 0.3335 |
|     1.602     | 54.96 | 20500 |     0.8590      | 1.2932 | 0.3433 |
|    1.5063     | 56.3  | 21000 |     0.8334      | 1.1312 | 0.2875 |
|    1.4631     | 57.64 | 21500 |     0.8474      | 1.1698 | 0.2999 |
|    1.4997     | 58.98 | 22000 |     0.8638      | 1.4279 | 0.3854 |
|    1.4301     | 60.32 | 22500 |     0.8550      | 1.2737 | 0.3300 |
|    1.3798     | 61.66 | 23000 |     0.8266      | 1.1802 | 0.2934 |
|    1.3454     | 63.0  | 23500 |     0.8235      | 1.3816 | 0.3711 |
|    1.3678     | 64.34 | 24000 |     0.8550      | 1.6427 | 0.5035 |
|    1.3761     | 65.68 | 24500 |     0.8510      | 1.6709 | 0.4907 |
|    1.2668     | 67.02 | 25000 |     0.8515      | 1.5842 | 0.4505 |
|    1.2835     | 68.36 | 25500 |     0.8283      | 1.5353 | 0.4221 |
|    1.2961     | 69.7  | 26000 |     0.8339      | 1.5743 | 0.4369 |
|    1.2656     | 71.05 | 26500 |     0.8331      | 1.5331 | 0.4217 |
|    1.2556     | 72.39 | 27000 |     0.8242      | 1.4708 | 0.4109 |
|    1.2043     | 73.73 | 27500 |     0.8245      | 1.4469 | 0.4031 |
|    1.2722     | 75.07 | 28000 |     0.8202      | 1.4924 | 0.4096 |
|     1.202     | 76.41 | 28500 |     0.8290      | 1.3807 | 0.3719 |
|    1.1679     | 77.75 | 29000 |     0.8195      | 1.4097 | 0.3749 |
|    1.1967     | 79.09 | 29500 |     0.8059      | 1.2074 | 0.3077 |
|    1.1241     | 80.43 | 30000 |     0.8137      | 1.2451 | 0.3270 |
|    1.1414     | 81.77 | 30500 |     0.8117      | 1.2031 | 0.3121 |
|     1.132     | 83.11 | 31000 |     0.8234      | 1.4266 | 0.3901 |
|    1.0982     | 84.45 | 31500 |     0.8064      | 1.3712 | 0.3607 |
|    1.0797     | 85.79 | 32000 |     0.8167      | 1.3356 | 0.3562 |
|    1.0119     | 87.13 | 32500 |     0.8215      | 1.2754 | 0.3268 |
|    1.0216     | 88.47 | 33000 |     0.8163      | 1.2512 | 0.3184 |
|    1.0375     | 89.81 | 33500 |     0.8137      | 1.2685 | 0.3290 |
|    0.9794     | 91.15 | 34000 |     0.8220      | 1.2724 | 0.3255 |
|    1.0207     | 92.49 | 34500 |     0.8165      | 1.2906 | 0.3361 |
|    1.0169     | 93.83 | 35000 |     0.8153      | 1.2819 | 0.3305 |
|    1.0127     | 95.17 | 35500 |     0.8187      | 1.2832 | 0.3252 |
|    0.9978     | 96.51 | 36000 |     0.8111      | 1.2612 | 0.3210 |
|    0.9923     | 97.85 | 36500 |     0.8076      | 1.2278 | 0.3122 |
|    1.0451     | 99.2  | 37000 |     0.8086      | 1.2451 | 0.3156 |

## Disclaimer

Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.

## Authors

Wav2Vec2 XLS-R 300M Cantonese (zh-HK) was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.

## Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


---
language: ko
license: apache-2.0
tags:
  - automatic-speech-recognition
  - generated_from_trainer
  - robust-speech-event
datasets:
  - kresnik/zeroth_korean
model-index:
  - name: Wav2Vec2 XLS-R 300M Korean LM
    results:
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Zeroth Korean
          type: kresnik/zeroth_korean
          args: clean
        metrics:
          - name: Test WER
            type: wer
            value: 30.94
          - name: Test CER
            type: cer
            value: 7.97
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Robust Speech Event - Dev Data
          type: speech-recognition-community-v2/dev_data
          args: ko
        metrics:
          - name: Test WER
            type: wer
            value: 68.34
          - name: Test CER
            type: cer
            value: 37.08
---

# Wav2Vec2 XLS-R 300M Korean LM

Wav2Vec2 XLS-R 300M Korean LM is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the [Zeroth Korean](https://huggingface.co/datasets/kresnik/zeroth_korean) dataset. A 5-gram Language model, trained on the Korean subset of [Open Subtitles](https://huggingface.co/datasets/open_subtitles), was then subsequently added to this model.

This model was trained using HuggingFace's PyTorch framework and is part of the [Robust Speech Challenge Event](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.

All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean-lm/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean-lm/tensorboard) logged via Tensorboard.

As for the N-gram language model training, we followed the [blog post tutorial](https://huggingface.co/blog/wav2vec2-with-ngram) provided by HuggingFace.

## Model

| Model                           | #params | Arch. | Training/Validation data (text) |
| ------------------------------- | ------- | ----- | ------------------------------- |
| `wav2vec2-xls-r-300m-korean-lm` | 300M    | XLS-R | `Zeroth Korean` Dataset         |

## Evaluation Results

The model achieves the following results on evaluation without a language model:

| Dataset                          | WER    | CER    |
| -------------------------------- | ------ | ------ |
| `Zeroth Korean`                  | 29.54% | 9.53%  |
| `Robust Speech Event - Dev Data` | 76.26% | 38.67% |

With the addition of the language model, it achieves the following results:

| Dataset                          | WER    | CER    |
| -------------------------------- | ------ | ------ |
| `Zeroth Korean`                  | 30.94% | 7.97%  |
| `Robust Speech Event - Dev Data` | 68.34% | 37.08% |

## Training procedure

The training process did not involve the addition of a language model. The following results were simply lifted from the original automatic speech recognition [model training](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean).

### Training hyperparameters

The following hyperparameters were used during training:

- `learning_rate`: 7.5e-05
- `train_batch_size`: 8
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 4
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 2000
- `num_epochs`: 50.0
- `mixed_precision_training`: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    19.7138    | 0.72  |  500  |     19.6427     |  1.0   |  1.0   |
|    4.8039     | 1.44  | 1000  |     4.7842      |  1.0   |  1.0   |
|    4.5619     | 2.16  | 1500  |     4.5608      | 0.9992 | 0.9598 |
|     4.254     | 2.88  | 2000  |     4.2729      | 0.9955 | 0.9063 |
|    4.1905     |  3.6  | 2500  |     4.2257      | 0.9903 | 0.8758 |
|    4.0683     | 4.32  | 3000  |     3.9294      | 0.9937 | 0.7911 |
|     3.486     | 5.04  | 3500  |     2.7045      | 1.0012 | 0.5934 |
|     2.946     | 5.75  | 4000  |     1.9691      | 0.9425 | 0.4634 |
|     2.634     | 6.47  | 4500  |     1.5212      | 0.8807 | 0.3850 |
|    2.4066     | 7.19  | 5000  |     1.2551      | 0.8177 | 0.3601 |
|    2.2651     | 7.91  | 5500  |     1.0423      | 0.7650 | 0.3039 |
|    2.1828     | 8.63  | 6000  |     0.9599      | 0.7273 | 0.3106 |
|    2.1023     | 9.35  | 6500  |     0.9482      | 0.7161 | 0.3063 |
|    2.0536     | 10.07 | 7000  |     0.8242      | 0.6767 | 0.2860 |
|    1.9803     | 10.79 | 7500  |     0.7643      | 0.6563 | 0.2637 |
|    1.9468     | 11.51 | 8000  |     0.7319      | 0.6441 | 0.2505 |
|    1.9178     | 12.23 | 8500  |     0.6937      | 0.6320 | 0.2489 |
|    1.8515     | 12.95 | 9000  |     0.6443      | 0.6053 | 0.2196 |
|    1.8083     | 13.67 | 9500  |     0.6286      | 0.6122 | 0.2148 |
|     1.819     | 14.39 | 10000 |     0.6015      | 0.5986 | 0.2074 |
|    1.7684     | 15.11 | 10500 |     0.5682      | 0.5741 | 0.1982 |
|    1.7195     | 15.83 | 11000 |     0.5385      | 0.5592 | 0.2007 |
|    1.7044     | 16.55 | 11500 |     0.5362      | 0.5524 | 0.2097 |
|    1.6879     | 17.27 | 12000 |     0.5119      | 0.5489 | 0.2083 |
|     1.656     | 17.98 | 12500 |     0.4990      | 0.5362 | 0.1968 |
|    1.6122     | 18.7  | 13000 |     0.4561      | 0.5092 | 0.1900 |
|    1.5919     | 19.42 | 13500 |     0.4778      | 0.5225 | 0.1975 |
|    1.5896     | 20.14 | 14000 |     0.4563      | 0.5098 | 0.1859 |
|    1.5589     | 20.86 | 14500 |     0.4362      | 0.4940 | 0.1725 |
|    1.5353     | 21.58 | 15000 |     0.4140      | 0.4826 | 0.1580 |
|    1.5441     | 22.3  | 15500 |     0.4031      | 0.4742 | 0.1550 |
|    1.5116     | 23.02 | 16000 |     0.3916      | 0.4748 | 0.1545 |
|    1.4731     | 23.74 | 16500 |     0.3841      | 0.4810 | 0.1542 |
|    1.4647     | 24.46 | 17000 |     0.3752      | 0.4524 | 0.1475 |
|    1.4328     | 25.18 | 17500 |     0.3587      | 0.4476 | 0.1461 |
|    1.4129     | 25.9  | 18000 |     0.3429      | 0.4242 | 0.1366 |
|    1.4062     | 26.62 | 18500 |     0.3450      | 0.4251 | 0.1355 |
|    1.3928     | 27.34 | 19000 |     0.3297      | 0.4145 | 0.1322 |
|    1.3906     | 28.06 | 19500 |     0.3210      | 0.4185 | 0.1336 |
|     1.358     | 28.78 | 20000 |     0.3131      | 0.3970 | 0.1275 |
|    1.3445     | 29.5  | 20500 |     0.3069      | 0.3920 | 0.1276 |
|    1.3159     | 30.22 | 21000 |     0.3035      | 0.3961 | 0.1255 |
|    1.3044     | 30.93 | 21500 |     0.2952      | 0.3854 | 0.1242 |
|    1.3034     | 31.65 | 22000 |     0.2966      | 0.3772 | 0.1227 |
|    1.2963     | 32.37 | 22500 |     0.2844      | 0.3706 | 0.1208 |
|    1.2765     | 33.09 | 23000 |     0.2841      | 0.3567 | 0.1173 |
|    1.2438     | 33.81 | 23500 |     0.2734      | 0.3552 | 0.1137 |
|    1.2487     | 34.53 | 24000 |     0.2703      | 0.3502 | 0.1118 |
|    1.2249     | 35.25 | 24500 |     0.2650      | 0.3484 | 0.1142 |
|    1.2229     | 35.97 | 25000 |     0.2584      | 0.3374 | 0.1097 |
|    1.2374     | 36.69 | 25500 |     0.2568      | 0.3337 | 0.1095 |
|    1.2153     | 37.41 | 26000 |     0.2494      | 0.3327 | 0.1071 |
|    1.1925     | 38.13 | 26500 |     0.2518      | 0.3366 | 0.1077 |
|    1.1908     | 38.85 | 27000 |     0.2437      | 0.3272 | 0.1057 |
|    1.1858     | 39.57 | 27500 |     0.2396      | 0.3265 | 0.1044 |
|    1.1808     | 40.29 | 28000 |     0.2373      | 0.3156 | 0.1028 |
|    1.1842     | 41.01 | 28500 |     0.2356      | 0.3152 | 0.1026 |
|    1.1668     | 41.73 | 29000 |     0.2319      | 0.3188 | 0.1025 |
|    1.1448     | 42.45 | 29500 |     0.2293      | 0.3099 | 0.0995 |
|    1.1327     | 43.17 | 30000 |     0.2265      | 0.3047 | 0.0979 |
|    1.1307     | 43.88 | 30500 |     0.2222      | 0.3078 | 0.0989 |
|    1.1419     | 44.6  | 31000 |     0.2215      | 0.3038 | 0.0981 |
|    1.1231     | 45.32 | 31500 |     0.2193      | 0.3013 | 0.0972 |
|     1.139     | 46.04 | 32000 |     0.2162      | 0.3007 | 0.0968 |
|    1.1114     | 46.76 | 32500 |     0.2122      | 0.2982 | 0.0960 |
|     1.111     | 47.48 | 33000 |     0.2125      | 0.2946 | 0.0948 |
|    1.0982     | 48.2  | 33500 |     0.2099      | 0.2957 | 0.0953 |
|     1.109     | 48.92 | 34000 |     0.2092      | 0.2955 | 0.0955 |
|    1.0905     | 49.64 | 34500 |     0.2088      | 0.2954 | 0.0953 |

## Disclaimer

Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.

## Authors

Wav2Vec2 XLS-R 300M Korean LM was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.

## Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


---
language:
- zh-HK
license: apache-2.0
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: ''
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

#

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the COMMON_VOICE - ZH-HK dataset.
It achieves the following results on the evaluation set:
- Loss: 1.0450
- Wer: 1.8991
- Cer: 0.4717

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 7.5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:------:|
| 77.5305       | 1.34  | 500   | 92.6669         | 1.0    | 1.0    |
| 9.3568        | 2.68  | 1000  | 7.6860          | 1.0    | 1.0    |
| 6.2735        | 4.02  | 1500  | 6.3664          | 1.0    | 1.0    |
| 6.1322        | 5.36  | 2000  | 6.1999          | 1.0    | 1.0    |
| 6.0244        | 6.7   | 2500  | 6.0719          | 1.0009 | 1.0000 |
| 5.9934        | 8.04  | 3000  | 5.9246          | 1.4023 | 1.0108 |
| 5.807         | 9.38  | 3500  | 5.7572          | 1.8696 | 1.0156 |
| 5.6889        | 10.72 | 4000  | 5.5060          | 1.9285 | 1.0035 |
| 5.4225        | 12.06 | 4500  | 4.9907          | 2.1646 | 1.0005 |
| 4.6644        | 13.4  | 5000  | 3.9605          | 2.0888 | 0.8890 |
| 4.2808        | 14.74 | 5500  | 3.4318          | 2.1360 | 0.8296 |
| 4.1329        | 16.09 | 6000  | 3.1680          | 2.2161 | 0.8428 |
| 3.8779        | 17.43 | 6500  | 2.9402          | 2.2343 | 0.8112 |
| 3.7395        | 18.77 | 7000  | 2.6735          | 2.1611 | 0.7561 |
| 3.5842        | 20.11 | 7500  | 2.3968          | 2.1364 | 0.6983 |
| 3.3861        | 21.45 | 8000  | 2.2187          | 2.1823 | 0.6786 |
| 3.2381        | 22.79 | 8500  | 2.0439          | 2.0273 | 0.6105 |
| 3.1511        | 24.13 | 9000  | 1.9231          | 2.0186 | 0.6136 |
| 3.0449        | 25.47 | 9500  | 1.8028          | 2.1044 | 0.6111 |
| 2.9652        | 26.81 | 10000 | 1.6802          | 2.0883 | 0.5933 |
| 2.8411        | 28.15 | 10500 | 1.5863          | 2.0476 | 0.5600 |
| 2.7381        | 29.49 | 11000 | 1.4835          | 2.0390 | 0.5425 |
| 2.6989        | 30.83 | 11500 | 1.4134          | 2.0879 | 0.5553 |
| 2.6437        | 32.17 | 12000 | 1.3482          | 1.9792 | 0.5212 |
| 2.5781        | 33.51 | 12500 | 1.3063          | 1.9926 | 0.5234 |
| 2.5456        | 34.85 | 13000 | 1.2719          | 1.9736 | 0.5055 |
| 2.489         | 36.19 | 13500 | 1.2196          | 1.9749 | 0.5008 |
| 2.442         | 37.53 | 14000 | 1.1836          | 1.9770 | 0.4999 |
| 2.4021        | 38.87 | 14500 | 1.1609          | 1.9861 | 0.5082 |
| 2.4228        | 40.21 | 15000 | 1.1361          | 1.9359 | 0.4882 |
| 2.3187        | 41.55 | 15500 | 1.1121          | 1.9493 | 0.4926 |
| 2.268         | 42.89 | 16000 | 1.0901          | 1.9346 | 0.4874 |
| 2.27          | 44.24 | 16500 | 1.0758          | 1.9277 | 0.4790 |
| 2.2367        | 45.58 | 17000 | 1.0699          | 1.9298 | 0.4804 |
| 2.2227        | 46.92 | 17500 | 1.0571          | 1.9268 | 0.4792 |
| 2.242         | 48.26 | 18000 | 1.0503          | 1.9078 | 0.4732 |
| 2.1974        | 49.6  | 18500 | 1.0460          | 1.9004 | 0.4724 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


---
language: id
tags:
  - indonesian-roberta-base-indolem-sentiment-classifier-fold-0
license: mit
datasets:
  - indolem
widget:
  - text: "Pelayanan hotel ini sangat baik."
---

## Indonesian RoBERTa Base IndoLEM Sentiment Classifier

Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).

A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.

On **fold 0**, the model achieved an F1 of 86.42% on dev/validation and 83.12% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-0` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |

## Evaluation Results

The model was trained for 10 epochs and the best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.563500      | 0.420457        | 0.796992 | 0.626728 | 0.680000  | 0.581197 |
| 2     | 0.293600      | 0.281360        | 0.884712 | 0.811475 | 0.779528  | 0.846154 |
| 3     | 0.163000      | 0.267922        | 0.904762 | 0.844262 | 0.811024  | 0.880342 |
| 4     | 0.090200      | 0.335411        | 0.899749 | 0.838710 | 0.793893  | 0.888889 |
| 5     | 0.065200      | 0.462526        | 0.897243 | 0.835341 | 0.787879  | 0.888889 |
| 6     | 0.039200      | 0.423001        | 0.912281 | 0.859438 | 0.810606  | 0.914530 |
| 7     | 0.025300      | 0.452100        | 0.912281 | 0.859438 | 0.810606  | 0.914530 |
| 8     | 0.010400      | 0.525200        | 0.914787 | 0.855932 | 0.848739  | 0.863248 |
| 9     | 0.007100      | 0.513585        | 0.909774 | 0.850000 | 0.829268  | 0.871795 |
| 10    | 0.007200      | 0.537254        | 0.917293 | 0.864198 | 0.833333  | 0.897436 |

## How to Use

### As Text Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.

## Author

Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: jv
tags:
- javanese-roberta-small
license: mit
datasets:
- wikipedia
widget:
- text: "Ing mangsa rendheng awakedhewe kudu pinter njaga <mask>."
---

## Javanese RoBERTa Small
Javanese RoBERTa Small is a masked language model based on the [RoBERTa model](https://arxiv.org/abs/1907.11692). It was trained on the latest (late December 2020) Javanese Wikipedia articles.

The model was originally HuggingFace's pretrained [English RoBERTa model](https://huggingface.co/roberta-base) and is later fine-tuned on the Javanese dataset. It achieved a perplexity of 33.30 on the validation dataset (20% of the articles). Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), and [fine-tuning tutorial notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb) written by [Pierre Guillou](https://huggingface.co/pierreguillou).

Hugging Face's [Transformers](https://huggingface.co/transformers) library was used to train the model -- utilizing the base RoBERTa model and their `Trainer` class. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.

## Model
| Model                    | #params | Arch.    | Training/Validation data (text)     |
|--------------------------|---------|----------|-------------------------------------|
| `javanese-roberta-small` |  124M   | RoBERTa  | Javanese Wikipedia (319 MB of text) |

## Evaluation Results
The model was trained for 5 epochs and the following is the final result once the training ended.

| train loss | valid loss | perplexity | total time |
|------------|------------|------------|------------|
|    3.481   |    3.506   |   33.30    |   1:11:43  |

## How to Use
### As Masked Language Model
```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-roberta-small"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Meja lan kursine lagi <mask>.")
```
### Feature Extraction in PyTorch
```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "w11wo/javanese-roberta-small"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```

## Disclaimer
Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.

## Author
Javanese RoBERTa Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: id
tags:
  - indonesian-roberta-base-indolem-sentiment-classifier-fold-1
license: mit
datasets:
  - indolem
widget:
  - text: "Pelayanan hotel ini sangat baik."
---

## Indonesian RoBERTa Base IndoLEM Sentiment Classifier

Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).

A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.

On **fold 1**, the model achieved an F1 of 84.43% on dev/validation and 86.59% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-1` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |

## Evaluation Results

The model was trained for 10 epochs and the best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.563400      | 0.438638        | 0.771930 | 0.528497 | 0.671053  | 0.435897 |
| 2     | 0.313700      | 0.286461        | 0.857143 | 0.724638 | 0.833333  | 0.641026 |
| 3     | 0.179800      | 0.255090        | 0.892231 | 0.812227 | 0.830357  | 0.794872 |
| 4     | 0.092300      | 0.360222        | 0.897243 | 0.828452 | 0.811475  | 0.846154 |
| 5     | 0.054400      | 0.345958        | 0.904762 | 0.844262 | 0.811024  | 0.880342 |
| 6     | 0.026300      | 0.466914        | 0.902256 | 0.836820 | 0.819672  | 0.854701 |
| 7     | 0.018500      | 0.517265        | 0.904762 | 0.836207 | 0.843478  | 0.829060 |
| 8     | 0.009900      | 0.544317        | 0.902256 | 0.835443 | 0.825000  | 0.846154 |
| 9     | 0.008800      | 0.570152        | 0.904762 | 0.833333 | 0.855856  | 0.811966 |
| 10    | 0.007500      | 0.569749        | 0.902256 | 0.829694 | 0.848214  | 0.811966 |

## How to Use

### As Text Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.

## Author

Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


---
language: id
tags:
  - indonesian-roberta-base-indonli
license: mit
datasets:
  - indonli
widget:
  - text: "Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih."
---

## Indonesian RoBERTa Base IndoNLI

Indonesian RoBERTa Base IndoNLI is a natural language inference (NLI) model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`IndoNLI`](https://github.com/ir-nlp-csui/indonli)'s dataset consisting of Indonesian Wikipedia, news, and Web articles [1].

After training, the model achieved an evaluation/dev accuracy of 77.06%. On the benchmark `test_lay` subset, the model achieved an accuracy of 74.24% and on the benchmark `test_expert` subset, the model achieved an accuracy of 61.66%.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                             | #params | Arch.        | Training/Validation data (text) |
| --------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indonli` | 124M    | RoBERTa Base | `IndoNLI`                       |

## Evaluation Results

The model was trained for 5 epochs, with a batch size of 16, a learning rate of 2e-5, a weight decay of 0.1, and a warmup ratio of 0.2, with linear annealing to 0. The best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Accuracy |
| ----- | ------------- | --------------- | -------- |
| 1     | 0.989200      | 0.691663        | 0.731452 |
| 2     | 0.673000      | 0.621913        | 0.766045 |
| 3     | 0.449900      | 0.662543        | 0.770596 |
| 4     | 0.293600      | 0.777059        | 0.768320 |
| 5     | 0.194200      | 0.948068        | 0.764224 |

## How to Use

### As NLI Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indonli"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and the `IndoNLI` dataset that may be carried over into the results of this model.

## References

[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). [IndoNLI: A Natural Language Inference Dataset for Indonesian](https://arxiv.org/abs/2110.14566). _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.

## Author

Indonesian RoBERTa Base IndoNLI was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.