| Classes      	| Total  	    | Train  	|         	| Valid  	|         	| Test   	|         	|
|-------------	|:------:	    |:------:	|:-------:	|:------:	|:-------:	|:------:	|:-------:	|
|            	| #     	    | #     	| %     	| # 	    | % 	    | #     	| %     	|
| PERSON      	|  **26130** 	| 19167  	|  73.35  	|  2733  	|  10.46  	|  4230  	|  16.19  	|
| GPE         	|  **11103** 	|  8193  	|  73.79  	|  1182  	|  10.65  	|  1728  	|   15.56 	|
| LOC         	|  **2467**  	|  1824  	|  73.94  	|  270   	|  10.94  	|  373   	|  15.12  	|
| ORG         	|  **7880**  	|  5688  	|  72.18  	|   880  	|  11.17  	|  1312  	|  16.65  	|
| LANGUAGE    	|   **467**  	|   342  	|  73.23  	|   52   	|  11.13  	|   73   	|  15.63  	|
| NAT_REL_POL 	|  **4970**  	|  3673  	|  73.90  	|   516  	|  10.38  	|   781  	|  15.71  	|
| DATETIME    	|  **9614**  	|  6960  	|  72.39  	|  1029  	|   10.7  	|  1625  	|   16.9  	|
| PERIOD      	|  **1188**  	|   862  	|  72.56  	|   129  	|  10.86  	|   197  	|  16.58  	|
| QUANTITY    	|  **1588**  	|  1161  	|  73.11  	|   181  	|   11.4  	|   246  	|  15.49  	|
| MONEY       	|  **1424**  	|  1041  	|  73.10  	|   159  	|  11.17  	|   224  	|  15.73  	|
| NUMERIC     	|  **7735**  	|  5734  	|  74.13  	|   814  	|  10.52  	|  1187  	|  15.35  	|
| ORDINAL     	|  **1893**  	|  1377  	|   72.74 	|   212  	|   11.2  	|   304  	|  16.06  	|
| FACILITY    	|  **1126**  	|   840  	|   74.6  	|   113  	|  10.04  	|   173  	|  15.36  	|
| WORK_OF_ART 	|  **1596**  	|  1157  	|  72.49  	|   176  	|  11.03  	|   263  	|  16.48  	|
| EVENT       	|  **1102**  	|   826  	|  74.95  	|   107  	|   9.71  	|   169  	|  15.34  	|


### BibTeX entry and citation info


Please consider citing the following [paper](https://arxiv.org/abs/1909.01247) as a thank you to the authors of the RONEC, even if it describes v1 of the corpus and you are using a model trained on v2:


```
Dumitrescu, Stefan Daniel, and Andrei-Marius Avram. "Introducing RONEC--the Romanian Named Entity Corpus." arXiv preprint arXiv:1909.01247 (2019).
```


or in .bibtex format:


```
@article{dumitrescu2019introducing,
  title={Introducing RONEC--the Romanian Named Entity Corpus},
  author={Dumitrescu, Stefan Daniel and Avram, Andrei-Marius},
  journal={arXiv preprint arXiv:1909.01247},
  year={2019}
}
```


model name: bert-base-romanian-ner


model author: dumitrescustefan


task: token-classification


tags: pytorch, bert, token-classification, ro, dataset:ronec, arxiv:1909.01247, transformers, license:mit, autonlp_compatible, infinity_compatible


## language: ro


# bert-base-romanian-uncased-v1


The BERT **base**, **uncased** model for Romanian, trained on a 15GB corpus, version ![v1.0](https://img.shields.io/badge/v1.0-21%20Apr%202020-ff6666)


### How to use


```python
from transformers import AutoTokenizer, AutoModel
import torch

# load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("dumitrescustefan/bert-base-romanian-uncased-v1", do_lower_case=True)
model = AutoModel.from_pretrained("dumitrescustefan/bert-base-romanian-uncased-v1")

# tokenize a sentence and run through the model
input_ids = torch.tensor(tokenizer.encode("Acesta este un test.", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)

# get encoding
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```


Remember to always sanitize your text! Replace `s` and `t` cedilla-letters to comma-letters with :


```
text = text.replace("ţ", "ț").replace("ş", "ș").replace("Ţ", "Ț").replace("Ş", "Ș")
```


because the model was **NOT** trained on cedilla `s` and `t`s. If you don't, you will have decreased performance due to <UNK>s and increased number of tokens per word.


### Evaluation


Evaluation is performed on Universal Dependencies [Romanian RRT](https://universaldependencies.org/treebanks/ro_rrt/index.html) UPOS, XPOS and LAS, and on a NER task based on [RONEC](https://github.com/dumitrescustefan/ronec). Details, as well as more in-depth tests not shown here, are given in the dedicated [evaluation page](https://github.com/dumitrescustefan/Romanian-Transformers/tree/master/evaluation/README.md).


The baseline is the [Multilingual BERT](https://github.com/google-research/bert/blob/master/multilingual.md) model `bert-base-multilingual-(un)cased`, as at the time of writing it was the only available BERT model that works on Romanian.


| Model                          |  UPOS |  XPOS  |  NER  |  LAS  |
|--------------------------------|:-----:|:------:|:-----:|:-----:|
| bert-base-multilingual-uncased | 97.65 |  95.72 | 83.91 | 87.65 |
| bert-base-romanian-uncased-v1  | **98.18** |  **96.84** | **85.26** | **89.61** |


### Corpus


The model is trained on the following corpora (stats in the table below are after cleaning):


| Corpus    	| Lines(M) 	| Words(M) 	| Chars(B) 	| Size(GB) 	|
|-----------	|:--------:	|:--------:	|:--------:	|:--------:	|
| OPUS      	|   55.05  	|  635.04  	|   4.045  	|    3.8   	|
| OSCAR     	|   33.56  	|  1725.82 	|  11.411  	|    11    	|
| Wikipedia 	|   1.54   	|   60.47  	|   0.411  	|    0.4   	|
| **Total**     	|   **90.15**  	|  **2421.33** 	|  **15.867**  	|   **15.2**   	|


#### Acknowledgements


- We'd like to thank [Sampo Pyysalo](https://github.com/spyysalo) from TurkuNLP for helping us out with the compute needed to pretrain the v1.0 BERT models. He's awesome!


model name: bert-base-romanian-uncased-v1


model author: dumitrescustefan


tags: pytorch, jax, bert, ro, transformers, infinity_compatible


## language: ro


# bert-base-romanian-cased-v1


The BERT **base**, **cased** model for Romanian, trained on a 15GB corpus, version ![v1.0](https://img.shields.io/badge/v1.0-21%20Apr%202020-ff6666)


### How to use


```python
from transformers import AutoTokenizer, AutoModel
import torch
# load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("dumitrescustefan/bert-base-romanian-cased-v1")
model = AutoModel.from_pretrained("dumitrescustefan/bert-base-romanian-cased-v1")
# tokenize a sentence and run through the model
input_ids = torch.tensor(tokenizer.encode("Acesta este un test.", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
# get encoding
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```


### Evaluation


| Model                          |  UPOS |  XPOS  |  NER  |  LAS  |
|--------------------------------|:-----:|:------:|:-----:|:-----:|
| bert-base-multilingual-cased   | 97.87 |  96.16 | 84.13 | 88.04 |
| bert-base-romanian-cased-v1    | **98.00** |  **96.46** | **85.88** | **89.69** |


model name: bert-base-romanian-cased-v1


pipeline_tag: sentence-similarity
tags:


- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers


## Usage


```
from sentence_transformers import SentenceTransformer, models
embedding_model = models.Transformer("yobi/klue-roberta-base-sts")
pooling_model = models.Pooling(
    embedding_model.get_word_embedding_dimension(),
    pooling_mode_mean_tokens=True,
)
model = SentenceTransformer(modules=[embedding_model,  pooling_model])
model.encode("안녕하세요.", convert_to_tensor=True)
```


model name: klue-roberta-base-sts


model author: yobi


task: sentence-similarity


tags: pytorch, roberta, feature-extraction, sentence-transformers, sentence-similarity, transformers, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- wikiann
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: electra-small-discriminator-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: wikiann
type: wikiann
args: en
metrics:
    - name: Precision
type: precision
value: 0.7330965535385425
    - name: Recall
type: recall
value: 0.7542632861138681
    - name: F1
type: f1
value: 0.7435293071244329
    - name: Accuracy
type: accuracy
value: 0.8883011190233978


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# electra-small-discriminator-finetuned-ner


This model is a fine-tuned version of [google/electra-small-discriminator](https://huggingface.co/google/electra-small-discriminator) on the wikiann dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3685
- Precision: 0.7331
- Recall: 0.7543
- F1: 0.7435
- Accuracy: 0.8883


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.5465        | 1.0   | 1250 | 0.4158          | 0.6932    | 0.7201 | 0.7064 | 0.8735   |
| 0.4037        | 2.0   | 2500 | 0.3817          | 0.7191    | 0.7470 | 0.7328 | 0.8828   |
| 0.3606        | 3.0   | 3750 | 0.3685          | 0.7331    | 0.7543 | 0.7435 | 0.8883   |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: electra-small-discriminator-finetuned-ner


model author: dbsamu


task: token-classification


tags: pytorch, tensorboard, electra, token-classification, dataset:wikiann, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- wikiann
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: distilbert-base-uncased-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: wikiann
type: wikiann
args: en
metrics:
    - name: Precision
type: precision
value: 0.8120642485217545
    - name: Recall
type: recall
value: 0.830235495804385
    - name: F1
type: f1
value: 0.8210493441599
    - name: Accuracy
type: accuracy
value: 0.9203828724683252


# distilbert-base-uncased-finetuned-ner


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the wikiann dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2781
- Precision: 0.8121
- Recall: 0.8302
- F1: 0.8210
- Accuracy: 0.9204


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.3504        | 1.0   | 1250 | 0.2922          | 0.7930    | 0.8075 | 0.8002 | 0.9115   |
| 0.2353        | 2.0   | 2500 | 0.2711          | 0.8127    | 0.8264 | 0.8195 | 0.9196   |
| 0.1745        | 3.0   | 3750 | 0.2781          | 0.8121    | 0.8302 | 0.8210 | 0.9204   |


### Framework versions


model name: distilbert-base-uncased-finetuned-ner


model author: dbsamu


task: token-classification


tags: pytorch, tensorboard, distilbert, token-classification, dataset:wikiann, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


## license: bsd-2-clause


model name: Unon


model author: Kneecapsnatcher


tags: license:bsd-2-clause


tags:


- conversational


# Aang DialoGPT Model


model name: DialoGPT-small-aang


model author: EEE


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


# Yoda DialoGPT Model


model name: DialoGPT-small-yoda


# Brooke DialoGPT Model


model name: DialoGPT-medium-brooke


# Named Entity Recognition using Transformers


This is a Fine-tuned version of BERT using HuggingFace transformers to perform Named Entity Recognition on Text data. BERT is a state-of-the-art model with attention mechanism as underlying architecture trained with masked-language-modeling and next-sentence-prediction objectives, used for various tasks including Question answering systems, Text Summarization, etc... which can also perform token classification tasks such as NER with great performance.


# Dataset


**CoNLL-2003** :
The shared task of CoNLL-2003 concerns language-independent named entity recognition. We will concentrate on four types of named entities: persons, locations, organizations, and names of miscellaneous entities that do not belong to the previous three groups.

**Link** : https://huggingface.co/datasets/conll2003


# Using this fine-tuned version


From python, download the whole pipeline and use it instantly using the following code :


```
from transformers import pipeline

# Loading the pipeline from hub
# Pipeline handles the preprocessing and post processing steps
model_checkpoint = "balamurugan1603/bert-finetuned-ner"
namedEntityRecogniser = pipeline(
    "token-classification", model=model_checkpoint, aggregation_strategy="simple"
)
```


Reference for using this pipeline to find NER tags can be found in this notebook.


model name: bert-finetuned-ner


model author: balamurugan1603


task: token-classification


tags: pytorch, tf, bert, token-classification, transformers, autonlp_compatible, infinity_compatible


language: jv
tags:


- javanese-distilbert-small
license: mit
datasets:
- wikipedia
widget:
- text: "Joko [MASK] wis kelas siji SMA."


## Javanese DistilBERT Small


Javanese DistilBERT Small is a masked language model based on the [DistilBERT model](https://arxiv.org/abs/1910.01108). It was trained on the latest (late December 2020) Javanese Wikipedia articles.


The model was originally HuggingFace's pretrained [English DistilBERT model](https://huggingface.co/distilbert-base-uncased) and is later fine-tuned on the Javanese dataset. It achieved a perplexity of 23.54 on the validation dataset (20% of the articles). Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), and [fine-tuning tutorial notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb) written by [Pierre Guillou](https://huggingface.co/pierreguillou).


Hugging Face's [Transformers](https://huggingface.co/transformers) library was used to train the model -- utilizing the base DistilBERT model and their `Trainer` class. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                       | #params | Arch.            | Training/Validation data (text)     |
|-----------------------------|---------|------------------|-------------------------------------|
| `javanese-distilbert-small` |   66M   | DistilBERT Small | Javanese Wikipedia (319 MB of text) |


## Evaluation Results


The model was trained for 5 epochs and the following is the final result once the training ended.


| train loss | valid loss | perplexity | total time |
|------------|------------|------------|------------|
|    3.088   |    3.153   |   23.54    |   1:46:37  |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-distilbert-small"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Aku mangan sate ing [MASK] bareng konco-konco")
```


### Feature Extraction in PyTorch


```python
from transformers import DistilBertModel, DistilBertTokenizerFast

pretrained_name = "w11wo/javanese-distilbert-small"
model = DistilBertModel.from_pretrained(pretrained_name)
tokenizer = DistilBertTokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.


## Author


Javanese DistilBERT Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-distilbert-small


model author: w11wo


task: fill-mask


tags: pytorch, tf, distilbert, fill-mask, jv, dataset:wikipedia, arxiv:1910.01108, transformers, javanese-distilbert-small, license:mit, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- automatic-speech-recognition
- kresnik/zeroth_korean
- generated_from_trainer
datasets:
- zeroth_korean_asr
model-index:
- name: ''
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the KRESNIK/ZEROTH_KOREAN - CLEAN dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0996
- Wer: 0.1022
- Cer: 0.0345


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:------:|
| 5.1239        | 0.72  | 500   | 4.8180          | 1.0    | 1.0    |
| 4.4011        | 1.44  | 1000  | 4.3980          | 0.9967 | 0.9385 |
| 4.1539        | 2.16  | 1500  | 4.1499          | 1.0171 | 0.7633 |
| 2.9662        | 2.88  | 2000  | 1.8843          | 0.9233 | 0.4671 |
| 2.4398        | 3.6   | 2500  | 1.1572          | 0.8259 | 0.3531 |
| 2.2207        | 4.32  | 3000  | 1.0066          | 0.7262 | 0.2824 |
| 2.1795        | 5.04  | 3500  | 0.8836          | 0.7523 | 0.3046 |
| 2.1027        | 5.75  | 4000  | 0.8822          | 0.6886 | 0.2540 |
| 2.0327        | 6.47  | 4500  | 0.8347          | 0.6906 | 0.2662 |
| 1.9621        | 7.19  | 5000  | 0.7796          | 0.6506 | 0.2385 |
| 1.9346        | 7.91  | 5500  | 0.7973          | 0.6873 | 0.2731 |
| 1.8845        | 8.63  | 6000  | 0.7259          | 0.6216 | 0.2169 |
| 1.8333        | 9.35  | 6500  | 0.6754          | 0.5926 | 0.2023 |
| 1.7733        | 10.07 | 7000  | 0.6137          | 0.5754 | 0.1977 |
| 1.721         | 10.79 | 7500  | 0.5678          | 0.5513 | 0.1827 |
| 1.6651        | 11.51 | 8000  | 0.5269          | 0.5148 | 0.1652 |
| 1.6084        | 12.23 | 8500  | 0.4882          | 0.4912 | 0.1602 |
| 1.5919        | 12.95 | 9000  | 0.4909          | 0.4928 | 0.1577 |
| 1.5654        | 13.67 | 9500  | 0.5053          | 0.5556 | 0.2023 |
| 1.5249        | 14.39 | 10000 | 0.4497          | 0.4645 | 0.1465 |
| 1.4697        | 15.11 | 10500 | 0.4128          | 0.5210 | 0.1800 |
| 1.4           | 15.83 | 11000 | 0.3737          | 0.4780 | 0.1609 |
| 1.3423        | 16.55 | 11500 | 0.3608          | 0.4059 | 0.1269 |
| 1.3118        | 17.27 | 12000 | 0.3304          | 0.4227 | 0.1382 |
| 1.2893        | 17.98 | 12500 | 0.3249          | 0.3644 | 0.1030 |
| 1.2752        | 18.7  | 13000 | 0.3277          | 0.3577 | 0.1053 |
| 1.2229        | 19.42 | 13500 | 0.3165          | 0.3778 | 0.1140 |
| 1.1846        | 20.14 | 14000 | 0.2839          | 0.3834 | 0.1320 |
| 1.165         | 20.86 | 14500 | 0.2809          | 0.4138 | 0.1662 |
| 1.119         | 21.58 | 15000 | 0.2561          | 0.3615 | 0.1289 |
| 1.1039        | 22.3  | 15500 | 0.2508          | 0.3173 | 0.1018 |
| 1.1019        | 23.02 | 16000 | 0.2687          | 0.3490 | 0.1204 |
| 1.0304        | 23.74 | 16500 | 0.2409          | 0.3303 | 0.1461 |
| 1.0399        | 24.46 | 17000 | 0.2501          | 0.3425 | 0.1229 |
| 0.9987        | 25.18 | 17500 | 0.2136          | 0.3334 | 0.1268 |
| 0.965         | 25.9  | 18000 | 0.2071          | 0.3247 | 0.1153 |
| 0.9424        | 26.62 | 18500 | 0.2079          | 0.3280 | 0.1238 |
| 0.9386        | 27.34 | 19000 | 0.2279          | 0.3286 | 0.1194 |
| 0.9087        | 28.06 | 19500 | 0.2040          | 0.3449 | 0.1399 |
| 0.8975        | 28.78 | 20000 | 0.1919          | 0.3399 | 0.1425 |
| 0.8631        | 29.5  | 20500 | 0.1790          | 0.3064 | 0.1238 |
| 0.8247        | 30.22 | 21000 | 0.1846          | 0.2307 | 0.0665 |
| 0.8254        | 30.93 | 21500 | 0.1870          | 0.2019 | 0.0534 |
| 0.8118        | 31.65 | 22000 | 0.1654          | 0.1915 | 0.0538 |
| 0.7761        | 32.37 | 22500 | 0.1626          | 0.2107 | 0.0705 |
| 0.7669        | 33.09 | 23000 | 0.1590          | 0.2342 | 0.0857 |
| 0.7305        | 33.81 | 23500 | 0.1490          | 0.1616 | 0.0434 |
| 0.7219        | 34.53 | 24000 | 0.1458          | 0.1578 | 0.0439 |
| 0.7024        | 35.25 | 24500 | 0.1457          | 0.1518 | 0.0421 |
| 0.6868        | 35.97 | 25000 | 0.1336          | 0.1527 | 0.0423 |
| 0.6761        | 36.69 | 25500 | 0.1290          | 0.1400 | 0.0403 |
| 0.6637        | 37.41 | 26000 | 0.1314          | 0.1439 | 0.0405 |
| 0.6312        | 38.13 | 26500 | 0.1303          | 0.1500 | 0.0473 |
| 0.6217        | 38.85 | 27000 | 0.1279          | 0.1439 | 0.0463 |
| 0.6213        | 39.57 | 27500 | 0.1254          | 0.1326 | 0.0387 |
| 0.5993        | 40.29 | 28000 | 0.1235          | 0.1729 | 0.0695 |
| 0.6075        | 41.01 | 28500 | 0.1230          | 0.1222 | 0.0356 |
| 0.5816        | 41.73 | 29000 | 0.1211          | 0.1646 | 0.0667 |
| 0.5619        | 42.45 | 29500 | 0.1175          | 0.1720 | 0.0752 |
| 0.541         | 43.17 | 30000 | 0.1114          | 0.1229 | 0.0395 |
| 0.5436        | 43.88 | 30500 | 0.1098          | 0.1216 | 0.0395 |
| 0.5441        | 44.6  | 31000 | 0.1116          | 0.1121 | 0.0340 |
| 0.5261        | 45.32 | 31500 | 0.1083          | 0.1093 | 0.0328 |
| 0.5184        | 46.04 | 32000 | 0.1053          | 0.1062 | 0.0336 |
| 0.506         | 46.76 | 32500 | 0.1046          | 0.1054 | 0.0353 |
| 0.5029        | 47.48 | 33000 | 0.1008          | 0.1025 | 0.0351 |
| 0.4856        | 48.2  | 33500 | 0.1011          | 0.1045 | 0.0353 |
| 0.496         | 48.92 | 34000 | 0.0995          | 0.1010 | 0.0345 |
| 0.4813        | 49.64 | 34500 | 0.0991          | 0.1021 | 0.0350 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xls-r-300m-korean-lm-v3


model author: w11wo


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:zeroth_korean_asr, transformers, license:apache-2.0, kresnik/zeroth_korean, generated_from_trainer, model-index


language: jv
tags:


- javanese-bert-small-imdb-classifier
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Dhuh Gusti, film iki elek banget. Aku getun ndelok !!!"


## Javanese BERT Small IMDB Classifier


Javanese BERT Small IMDB Classifier is a movie-classification model based on the [BERT model](https://arxiv.org/abs/1810.04805). It was trained on  Javanese IMDB movie reviews.


The model was originally [`w11wo/javanese-bert-small-imdb`](https://huggingface.co/w11wo/javanese-bert-small-imdb) which is then fine-tuned on the [`w11wo/imdb-javanese`](https://huggingface.co/datasets/w11wo/imdb-javanese) dataset consisting of Javanese IMDB movie reviews. It achieved an accuracy of 76.37% on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                                 | #params  | Arch.          | Training/Validation data (text) |
|---------------------------------------|----------|----------------|---------------------------------|
| `javanese-bert-small-imdb-classifier` |   110M   |   BERT Small   | Javanese IMDB (47.5 MB of text) |


## Evaluation Results


The model was trained for 5 epochs and the following is the final result once the training ended.


| train loss | valid loss |  accuracy  | total time |
|------------|------------|------------|------------|
|    0.131   |    1.113   |   0.763    |    59:16   |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-bert-small-imdb-classifier"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Film sing apik banget!")
```


## Disclaimer


Do consider the biases which came from the IMDB review that may be carried over into the results of this model.


## Author


Javanese BERT Small IMDB Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-bert-small-imdb-classifier


model author: w11wo


task: text-classification


tags: pytorch, tf, jax, bert, text-classification, jv, dataset:w11wo/imdb-javanese, arxiv:1810.04805, transformers, javanese-bert-small-imdb-classifier, license:mit, infinity_compatible


language: id
tags:


- indonesian-roberta-base-sentiment-classifier
license: mit
datasets:
- indonlu
widget:
- text: "Jangan sampai saya telpon bos saya ya!"


## Indonesian RoBERTa Base Sentiment Classifier


Indonesian RoBERTa Base Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indonlu`](https://hf.co/datasets/indonlu)'s `SmSA` dataset consisting of Indonesian comments and reviews.


After training, the model achieved an evaluation accuracy of 94.36% and F1-macro of 92.42%. On the benchmark test set, the model achieved an accuracy of 93.2% and F1-macro of 91.02%.


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.


## Model


| Model                                          | #params | Arch.        | Training/Validation data (text) |
| ---------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-sentiment-classifier` | 124M    | RoBERTa Base | `SmSA`                          |


## Evaluation Results


The model was trained for 5 epochs and the best model was loaded at the end.


| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.342600      | 0.213551        | 0.928571 | 0.898539 | 0.909803  | 0.890694 |
| 2     | 0.190700      | 0.213466        | 0.934127 | 0.901135 | 0.925297  | 0.882757 |
| 3     | 0.125500      | 0.219539        | 0.942857 | 0.920901 | 0.927511  | 0.915193 |
| 4     | 0.083600      | 0.235232        | 0.943651 | 0.924227 | 0.926494  | 0.922048 |
| 5     | 0.059200      | 0.262473        | 0.942063 | 0.920583 | 0.924084  | 0.917351 |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-sentiment-classifier"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Jangan sampai saya telpon bos saya ya!")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and the `SmSA` dataset that may be carried over into the results of this model.


## Author


Indonesian RoBERTa Base Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indonesian-roberta-base-sentiment-classifier


model author: w11wo


task: text-classification


tags: pytorch, tf, roberta, text-classification, id, dataset:indonlu, arxiv:1907.11692, transformers, indonesian-roberta-base-sentiment-classifier, license:mit, infinity_compatible


language: id
tags:


- indonesian-roberta-base-indolem-sentiment-classifier-fold-2
license: mit
datasets:
- indolem
widget:
- text: "Pelayanan hotel ini sangat baik."


## Indonesian RoBERTa Base IndoLEM Sentiment Classifier


Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).


A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.


On **fold 2**, the model achieved an F1 of 83.05% on dev/validation and 82.89% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.


## Model


| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-2` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |


## Evaluation Results


The model was trained for 10 epochs and the best model was loaded at the end.


| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.563300      | 0.436885        | 0.786967 | 0.568528 | 0.700000  | 0.478632 |
| 2     | 0.307800      | 0.301497        | 0.864662 | 0.763158 | 0.783784  | 0.743590 |
| 3     | 0.178900      | 0.287950        | 0.899749 | 0.818182 | 0.873786  | 0.769231 |
| 4     | 0.093100      | 0.345517        | 0.894737 | 0.820513 | 0.820513  | 0.820513 |
| 5     | 0.054500      | 0.434557        | 0.899749 | 0.826087 | 0.840708  | 0.811966 |
| 6     | 0.030900      | 0.487250        | 0.904762 | 0.827273 | 0.883495  | 0.777778 |
| 7     | 0.021700      | 0.494885        | 0.899749 | 0.830508 | 0.823529  | 0.837607 |
| 8     | 0.011500      | 0.580564        | 0.897243 | 0.814480 | 0.865385  | 0.769231 |
| 9     | 0.006000      | 0.604532        | 0.894737 | 0.820513 | 0.820513  | 0.820513 |
| 10    | 0.005000      | 0.616260        | 0.897243 | 0.822511 | 0.833333  | 0.811966 |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.


## Author


Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indonesian-roberta-base-indolem-sentiment-classifier-fold-2


model author: w11wo


task: text-classification


tags: pytorch, roberta, text-classification, id, dataset:indolem, arxiv:1907.11692, transformers, indonesian-roberta-base-indolem-sentiment-classifier-fold-2, license:mit, infinity_compatible


language: jv
tags:


- javanese-gpt2-small
license: mit
datasets:
- wikipedia
widget:
- text: "Jenengku Budi, saka Indonesia"


## Javanese GPT-2 Small


Javanese GPT-2 Small is a language model based on the [GPT-2 model](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). It was trained on the latest (late December 2020) Javanese Wikipedia articles.


The model was originally HuggingFace's pretrained [English GPT-2 model](https://huggingface.co/transformers/model_doc/gpt2.html) and is later fine-tuned on the Javanese dataset. Many of the techniques used
are based on a [notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb)/[blog](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787) shared by [Pierre Guillou](https://medium.com/@pierre_guillou), where Pierre Guillou fine-tuned the English GPT-2 model on a Portuguese dataset.


Frameworks used include HuggingFace's [Transformers](https://huggingface.co/transformers) and fast.ai's [Deep Learning library](https://docs.fast.ai/). PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                 | #params | Arch.       | Training /Validation data (text)    |
|-----------------------|---------|-------------|-------------------------------------|
| `javanese-gpt2-small` |  124M   | GPT-2 Small | Javanese Wikipedia (319 MB of text) |


## Evaluation Results


Before fine-tuning, the English GPT-2 model went through a validation step just to see how the model fairs prior to training.


| valid loss | perplexity |
|------------|------------|
|   10.845   |  51313.62  |


The model was then trained afterwards for 5 epochs and the following are the results.


| epoch | train loss | valid loss | perplexity | total time |
|-------|------------|------------|------------|------------|
|   0   |    4.336   |    4.110   |    60.94   |    22:28   |
|   1   |    3.598   |    3.543   |    34.58   |    23:27   |
|   2   |    3.161   |    3.331   |    27.98   |    24:17   |
|   3   |    2.974   |    3.265   |    26.18   |    25:03   |
|   4   |    2.932   |    3.234   |    25.39   |    25:06   |


## How to Use (PyTorch)


### Load Model and Byte-level Tokenizer


```python
from transformers import GPT2TokenizerFast, GPT2LMHeadModel
pretrained_name = "w11wo/javanese-gpt2-small"
tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_name)
tokenizer.model_max_length = 1024
model = GPT2LMHeadModel.from_pretrained(pretrained_name)
```


### Generate a Sequence


```python
# sample prompt
prompt = "Jenengku Budi, saka Indonesia"
input_ids = tokenizer.encode(prompt, return_tensors='pt')
model.eval()

# generate output using top-k sampling
sample_outputs = model.generate(input_ids,
                                pad_token_id=50256,
                                do_sample=True,
                                max_length=40,
                                min_length=40,
                                top_k=40,
                                num_return_sequences=1)

for i, sample_output in enumerate(sample_outputs):
    print(tokenizer.decode(sample_output.tolist()))
```


## Disclaimer


Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.


## Credits


Major thanks to Pierre Guillou for sharing his work, which did not only enable me to realize this project but also taught me tons of new, exciting stuff.


## Author


Javanese GPT-2 Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-gpt2-small


model author: w11wo


task: text-generation


tags: pytorch, tf, jax, gpt2, text-generation, jv, dataset:wikipedia, transformers, javanese-gpt2-small, license:mit


license: apache-2.0
tags:


- automatic-speech-recognition
- kresnik/zeroth_korean
- generated_from_trainer
datasets:
- zeroth_korean_asr
model-index:
- name: ''
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the KRESNIK/ZEROTH_KOREAN - CLEAN dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0996
- Wer: 0.1022
- Cer: 0.0345


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:------:|
| 5.1239        | 0.72  | 500   | 4.8180          | 1.0    | 1.0    |
| 4.4011        | 1.44  | 1000  | 4.3980          | 0.9967 | 0.9385 |
| 4.1539        | 2.16  | 1500  | 4.1499          | 1.0171 | 0.7633 |
| 2.9662        | 2.88  | 2000  | 1.8843          | 0.9233 | 0.4671 |
| 2.4398        | 3.6   | 2500  | 1.1572          | 0.8259 | 0.3531 |
| 2.2207        | 4.32  | 3000  | 1.0066          | 0.7262 | 0.2824 |
| 2.1795        | 5.04  | 3500  | 0.8836          | 0.7523 | 0.3046 |
| 2.1027        | 5.75  | 4000  | 0.8822          | 0.6886 | 0.2540 |
| 2.0327        | 6.47  | 4500  | 0.8347          | 0.6906 | 0.2662 |
| 1.9621        | 7.19  | 5000  | 0.7796          | 0.6506 | 0.2385 |
| 1.9346        | 7.91  | 5500  | 0.7973          | 0.6873 | 0.2731 |
| 1.8845        | 8.63  | 6000  | 0.7259          | 0.6216 | 0.2169 |
| 1.8333        | 9.35  | 6500  | 0.6754          | 0.5926 | 0.2023 |
| 1.7733        | 10.07 | 7000  | 0.6137          | 0.5754 | 0.1977 |
| 1.721         | 10.79 | 7500  | 0.5678          | 0.5513 | 0.1827 |
| 1.6651        | 11.51 | 8000  | 0.5269          | 0.5148 | 0.1652 |
| 1.6084        | 12.23 | 8500  | 0.4882          | 0.4912 | 0.1602 |
| 1.5919        | 12.95 | 9000  | 0.4909          | 0.4928 | 0.1577 |
| 1.5654        | 13.67 | 9500  | 0.5053          | 0.5556 | 0.2023 |
| 1.5249        | 14.39 | 10000 | 0.4497          | 0.4645 | 0.1465 |
| 1.4697        | 15.11 | 10500 | 0.4128          | 0.5210 | 0.1800 |
| 1.4           | 15.83 | 11000 | 0.3737          | 0.4780 | 0.1609 |
| 1.3423        | 16.55 | 11500 | 0.3608          | 0.4059 | 0.1269 |
| 1.3118        | 17.27 | 12000 | 0.3304          | 0.4227 | 0.1382 |
| 1.2893        | 17.98 | 12500 | 0.3249          | 0.3644 | 0.1030 |
| 1.2752        | 18.7  | 13000 | 0.3277          | 0.3577 | 0.1053 |
| 1.2229        | 19.42 | 13500 | 0.3165          | 0.3778 | 0.1140 |
| 1.1846        | 20.14 | 14000 | 0.2839          | 0.3834 | 0.1320 |
| 1.165         | 20.86 | 14500 | 0.2809          | 0.4138 | 0.1662 |
| 1.119         | 21.58 | 15000 | 0.2561          | 0.3615 | 0.1289 |
| 1.1039        | 22.3  | 15500 | 0.2508          | 0.3173 | 0.1018 |
| 1.1019        | 23.02 | 16000 | 0.2687          | 0.3490 | 0.1204 |
| 1.0304        | 23.74 | 16500 | 0.2409          | 0.3303 | 0.1461 |
| 1.0399        | 24.46 | 17000 | 0.2501          | 0.3425 | 0.1229 |
| 0.9987        | 25.18 | 17500 | 0.2136          | 0.3334 | 0.1268 |
| 0.965         | 25.9  | 18000 | 0.2071          | 0.3247 | 0.1153 |
| 0.9424        | 26.62 | 18500 | 0.2079          | 0.3280 | 0.1238 |
| 0.9386        | 27.34 | 19000 | 0.2279          | 0.3286 | 0.1194 |
| 0.9087        | 28.06 | 19500 | 0.2040          | 0.3449 | 0.1399 |
| 0.8975        | 28.78 | 20000 | 0.1919          | 0.3399 | 0.1425 |
| 0.8631        | 29.5  | 20500 | 0.1790          | 0.3064 | 0.1238 |
| 0.8247        | 30.22 | 21000 | 0.1846          | 0.2307 | 0.0665 |
| 0.8254        | 30.93 | 21500 | 0.1870          | 0.2019 | 0.0534 |
| 0.8118        | 31.65 | 22000 | 0.1654          | 0.1915 | 0.0538 |
| 0.7761        | 32.37 | 22500 | 0.1626          | 0.2107 | 0.0705 |
| 0.7669        | 33.09 | 23000 | 0.1590          | 0.2342 | 0.0857 |
| 0.7305        | 33.81 | 23500 | 0.1490          | 0.1616 | 0.0434 |
| 0.7219        | 34.53 | 24000 | 0.1458          | 0.1578 | 0.0439 |
| 0.7024        | 35.25 | 24500 | 0.1457          | 0.1518 | 0.0421 |
| 0.6868        | 35.97 | 25000 | 0.1336          | 0.1527 | 0.0423 |
| 0.6761        | 36.69 | 25500 | 0.1290          | 0.1400 | 0.0403 |
| 0.6637        | 37.41 | 26000 | 0.1314          | 0.1439 | 0.0405 |
| 0.6312        | 38.13 | 26500 | 0.1303          | 0.1500 | 0.0473 |
| 0.6217        | 38.85 | 27000 | 0.1279          | 0.1439 | 0.0463 |
| 0.6213        | 39.57 | 27500 | 0.1254          | 0.1326 | 0.0387 |
| 0.5993        | 40.29 | 28000 | 0.1235          | 0.1729 | 0.0695 |
| 0.6075        | 41.01 | 28500 | 0.1230          | 0.1222 | 0.0356 |
| 0.5816        | 41.73 | 29000 | 0.1211          | 0.1646 | 0.0667 |
| 0.5619        | 42.45 | 29500 | 0.1175          | 0.1720 | 0.0752 |
| 0.541         | 43.17 | 30000 | 0.1114          | 0.1229 | 0.0395 |
| 0.5436        | 43.88 | 30500 | 0.1098          | 0.1216 | 0.0395 |
| 0.5441        | 44.6  | 31000 | 0.1116          | 0.1121 | 0.0340 |
| 0.5261        | 45.32 | 31500 | 0.1083          | 0.1093 | 0.0328 |
| 0.5184        | 46.04 | 32000 | 0.1053          | 0.1062 | 0.0336 |
| 0.506         | 46.76 | 32500 | 0.1046          | 0.1054 | 0.0353 |
| 0.5029        | 47.48 | 33000 | 0.1008          | 0.1025 | 0.0351 |
| 0.4856        | 48.2  | 33500 | 0.1011          | 0.1045 | 0.0353 |
| 0.496         | 48.92 | 34000 | 0.0995          | 0.1010 | 0.0345 |
| 0.4813        | 49.64 | 34500 | 0.0991          | 0.1021 | 0.0350 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xls-r-300m-korean-v3


model author: w11wo


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:zeroth_korean_asr, transformers, license:apache-2.0, kresnik/zeroth_korean, generated_from_trainer, model-index


language: id
tags:


- indonesian-roberta-base-indolem-sentiment-classifier-fold-4
license: mit
datasets:
- indolem
widget:
- text: "Pelayanan hotel ini sangat baik."


## Indonesian RoBERTa Base IndoLEM Sentiment Classifier


Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).


A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.


On **fold 4**, the model achieved an F1 of 84.21% on dev/validation and 85.36% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.


## Model


| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-4` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |


## Evaluation Results


The model was trained for 10 epochs and the best model was loaded at the end.


| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.565600      | 0.441932        | 0.771930 | 0.542714 | 0.658537  | 0.461538 |
| 2     | 0.303300      | 0.302146        | 0.869674 | 0.785124 | 0.760000  | 0.811966 |
| 3     | 0.175300      | 0.337158        | 0.879699 | 0.769231 | 0.879121  | 0.683761 |
| 4     | 0.084000      | 0.343521        | 0.894737 | 0.815789 | 0.837838  | 0.794872 |
| 5     | 0.036700      | 0.511235        | 0.889724 | 0.821138 | 0.782946  | 0.863248 |
| 6     | 0.020300      | 0.628374        | 0.897243 | 0.812785 | 0.872549  | 0.760684 |
| 7     | 0.019400      | 0.609241        | 0.909774 | 0.842105 | 0.864865  | 0.820513 |
| 8     | 0.005400      | 0.623904        | 0.897243 | 0.825532 | 0.822034  | 0.829060 |
| 9     | 0.005300      | 0.641695        | 0.904762 | 0.836207 | 0.843478  | 0.829060 |
| 10    | 0.003900      | 0.645317        | 0.904762 | 0.836207 | 0.843478  | 0.829060 |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.


## Author


Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indonesian-roberta-base-indolem-sentiment-classifier-fold-4


model author: w11wo


task: text-classification


tags: pytorch, roberta, text-classification, id, dataset:indolem, arxiv:1907.11692, transformers, indonesian-roberta-base-indolem-sentiment-classifier-fold-4, license:mit, infinity_compatible


language: id
tags:


- indonesian-roberta-base-posp-tagger
license: mit
datasets:
- indonlu
widget:
- text: "Budi sedang pergi ke pasar."


## Indonesian RoBERTa Base POSP Tagger


Indonesian RoBERTa Base POSP Tagger is a part-of-speech token-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indonlu`](https://hf.co/datasets/indonlu)'s `POSP` dataset consisting of tag-labelled news.


After training, the model achieved an evaluation F1-macro of 95.34%. On the benchmark test set, the model achieved an accuracy of 93.99% and F1-macro of 88.93%.


## Model


| Model                                 | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-posp-tagger` | 124M    | RoBERTa Base | `POSP`                          |


## Evaluation Results


| Epoch | Training Loss | Validation Loss | Precision | Recall   | F1       | Accuracy |
| ----- | ------------- | --------------- | --------- | -------- | -------- | -------- |
| 1     | 0.898400      | 0.343731        | 0.894324  | 0.894324 | 0.894324 | 0.894324 |
| 2     | 0.294700      | 0.236619        | 0.929620  | 0.929620 | 0.929620 | 0.929620 |
| 3     | 0.214100      | 0.202723        | 0.938349  | 0.938349 | 0.938349 | 0.938349 |
| 4     | 0.171100      | 0.183630        | 0.945264  | 0.945264 | 0.945264 | 0.945264 |
| 5     | 0.143300      | 0.169744        | 0.948469  | 0.948469 | 0.948469 | 0.948469 |
| 6     | 0.124700      | 0.174946        | 0.947963  | 0.947963 | 0.947963 | 0.947963 |
| 7     | 0.109800      | 0.167450        | 0.951590  | 0.951590 | 0.951590 | 0.951590 |
| 8     | 0.101300      | 0.163191        | 0.952475  | 0.952475 | 0.952475 | 0.952475 |
| 9     | 0.093500      | 0.163255        | 0.953361  | 0.953361 | 0.953361 | 0.953361 |
| 10    | 0.089000      | 0.164673        | 0.953445  | 0.953445 | 0.953445 | 0.953445 |


## How to Use


### As Token Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-posp-tagger"

nlp = pipeline(
    "token-classification",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Budi sedang pergi ke pasar.")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and the `POSP` dataset that may be carried over into the results of this model.


## Author


Indonesian RoBERTa Base POSP Tagger was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indonesian-roberta-base-posp-tagger


model author: w11wo


task: token-classification


tags: pytorch, tf, roberta, token-classification, id, dataset:indonlu, arxiv:1907.11692, transformers, indonesian-roberta-base-posp-tagger, license:mit, autonlp_compatible, infinity_compatible


language: ko
license: apache-2.0
tags:


- automatic-speech-recognition
- generated_from_trainer
- robust-speech-event
datasets:
- kresnik/zeroth_korean
model-index:
- name: Wav2Vec2 XLS-R 300M Korean
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Zeroth Korean
type: kresnik/zeroth_korean
args: clean
metrics:
    - name: Test WER
type: wer
value: 29.54
    - name: Test CER
type: cer
value: 9.53
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: ko
metrics:
    - name: Test WER
type: wer
value: 76.26
    - name: Test CER
type: cer
value: 38.67


# Wav2Vec2 XLS-R 300M Korean


Wav2Vec2 XLS-R 300M Korean is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the [Zeroth Korean](https://huggingface.co/datasets/kresnik/zeroth_korean) dataset.


This model was trained using HuggingFace's PyTorch framework and is part of the [Robust Speech Challenge Event](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.


All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean/tensorboard) logged via Tensorboard.


## Model


| Model                        | #params | Arch. | Training/Validation data (text) |
| ---------------------------- | ------- | ----- | ------------------------------- |
| `wav2vec2-xls-r-300m-korean` | 300M    | XLS-R | `Zeroth Korean` Dataset         |


## Evaluation Results


The model achieves the following results on evaluation:


| Dataset                          | Loss   | WER    | CER    |
| -------------------------------- | ------ | ------ | ------ |
| `Zeroth Korean`                  | 0.2089 | 29.54% | 9.53%  |
| `Robust Speech Event - Dev Data` | N/A    | 76.26% | 38.67% |


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- `learning_rate`: 7.5e-05
- `train_batch_size`: 8
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 4
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 2000
- `num_epochs`: 50.0
- `mixed_precision_training`: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    19.7138    | 0.72  |  500  |     19.6427     |  1.0   |  1.0   |
|    4.8039     | 1.44  | 1000  |     4.7842      |  1.0   |  1.0   |
|    4.5619     | 2.16  | 1500  |     4.5608      | 0.9992 | 0.9598 |
|     4.254     | 2.88  | 2000  |     4.2729      | 0.9955 | 0.9063 |
|    4.1905     |  3.6  | 2500  |     4.2257      | 0.9903 | 0.8758 |
|    4.0683     | 4.32  | 3000  |     3.9294      | 0.9937 | 0.7911 |
|     3.486     | 5.04  | 3500  |     2.7045      | 1.0012 | 0.5934 |
|     2.946     | 5.75  | 4000  |     1.9691      | 0.9425 | 0.4634 |
|     2.634     | 6.47  | 4500  |     1.5212      | 0.8807 | 0.3850 |
|    2.4066     | 7.19  | 5000  |     1.2551      | 0.8177 | 0.3601 |
|    2.2651     | 7.91  | 5500  |     1.0423      | 0.7650 | 0.3039 |
|    2.1828     | 8.63  | 6000  |     0.9599      | 0.7273 | 0.3106 |
|    2.1023     | 9.35  | 6500  |     0.9482      | 0.7161 | 0.3063 |
|    2.0536     | 10.07 | 7000  |     0.8242      | 0.6767 | 0.2860 |
|    1.9803     | 10.79 | 7500  |     0.7643      | 0.6563 | 0.2637 |
|    1.9468     | 11.51 | 8000  |     0.7319      | 0.6441 | 0.2505 |
|    1.9178     | 12.23 | 8500  |     0.6937      | 0.6320 | 0.2489 |
|    1.8515     | 12.95 | 9000  |     0.6443      | 0.6053 | 0.2196 |
|    1.8083     | 13.67 | 9500  |     0.6286      | 0.6122 | 0.2148 |
|     1.819     | 14.39 | 10000 |     0.6015      | 0.5986 | 0.2074 |
|    1.7684     | 15.11 | 10500 |     0.5682      | 0.5741 | 0.1982 |
|    1.7195     | 15.83 | 11000 |     0.5385      | 0.5592 | 0.2007 |
|    1.7044     | 16.55 | 11500 |     0.5362      | 0.5524 | 0.2097 |
|    1.6879     | 17.27 | 12000 |     0.5119      | 0.5489 | 0.2083 |
|     1.656     | 17.98 | 12500 |     0.4990      | 0.5362 | 0.1968 |
|    1.6122     | 18.7  | 13000 |     0.4561      | 0.5092 | 0.1900 |
|    1.5919     | 19.42 | 13500 |     0.4778      | 0.5225 | 0.1975 |
|    1.5896     | 20.14 | 14000 |     0.4563      | 0.5098 | 0.1859 |
|    1.5589     | 20.86 | 14500 |     0.4362      | 0.4940 | 0.1725 |
|    1.5353     | 21.58 | 15000 |     0.4140      | 0.4826 | 0.1580 |
|    1.5441     | 22.3  | 15500 |     0.4031      | 0.4742 | 0.1550 |
|    1.5116     | 23.02 | 16000 |     0.3916      | 0.4748 | 0.1545 |
|    1.4731     | 23.74 | 16500 |     0.3841      | 0.4810 | 0.1542 |
|    1.4647     | 24.46 | 17000 |     0.3752      | 0.4524 | 0.1475 |
|    1.4328     | 25.18 | 17500 |     0.3587      | 0.4476 | 0.1461 |
|    1.4129     | 25.9  | 18000 |     0.3429      | 0.4242 | 0.1366 |
|    1.4062     | 26.62 | 18500 |     0.3450      | 0.4251 | 0.1355 |
|    1.3928     | 27.34 | 19000 |     0.3297      | 0.4145 | 0.1322 |
|    1.3906     | 28.06 | 19500 |     0.3210      | 0.4185 | 0.1336 |
|     1.358     | 28.78 | 20000 |     0.3131      | 0.3970 | 0.1275 |
|    1.3445     | 29.5  | 20500 |     0.3069      | 0.3920 | 0.1276 |
|    1.3159     | 30.22 | 21000 |     0.3035      | 0.3961 | 0.1255 |
|    1.3044     | 30.93 | 21500 |     0.2952      | 0.3854 | 0.1242 |
|    1.3034     | 31.65 | 22000 |     0.2966      | 0.3772 | 0.1227 |
|    1.2963     | 32.37 | 22500 |     0.2844      | 0.3706 | 0.1208 |
|    1.2765     | 33.09 | 23000 |     0.2841      | 0.3567 | 0.1173 |
|    1.2438     | 33.81 | 23500 |     0.2734      | 0.3552 | 0.1137 |
|    1.2487     | 34.53 | 24000 |     0.2703      | 0.3502 | 0.1118 |
|    1.2249     | 35.25 | 24500 |     0.2650      | 0.3484 | 0.1142 |
|    1.2229     | 35.97 | 25000 |     0.2584      | 0.3374 | 0.1097 |
|    1.2374     | 36.69 | 25500 |     0.2568      | 0.3337 | 0.1095 |
|    1.2153     | 37.41 | 26000 |     0.2494      | 0.3327 | 0.1071 |
|    1.1925     | 38.13 | 26500 |     0.2518      | 0.3366 | 0.1077 |
|    1.1908     | 38.85 | 27000 |     0.2437      | 0.3272 | 0.1057 |
|    1.1858     | 39.57 | 27500 |     0.2396      | 0.3265 | 0.1044 |
|    1.1808     | 40.29 | 28000 |     0.2373      | 0.3156 | 0.1028 |
|    1.1842     | 41.01 | 28500 |     0.2356      | 0.3152 | 0.1026 |
|    1.1668     | 41.73 | 29000 |     0.2319      | 0.3188 | 0.1025 |
|    1.1448     | 42.45 | 29500 |     0.2293      | 0.3099 | 0.0995 |
|    1.1327     | 43.17 | 30000 |     0.2265      | 0.3047 | 0.0979 |
|    1.1307     | 43.88 | 30500 |     0.2222      | 0.3078 | 0.0989 |
|    1.1419     | 44.6  | 31000 |     0.2215      | 0.3038 | 0.0981 |
|    1.1231     | 45.32 | 31500 |     0.2193      | 0.3013 | 0.0972 |
|     1.139     | 46.04 | 32000 |     0.2162      | 0.3007 | 0.0968 |
|    1.1114     | 46.76 | 32500 |     0.2122      | 0.2982 | 0.0960 |
|     1.111     | 47.48 | 33000 |     0.2125      | 0.2946 | 0.0948 |
|    1.0982     | 48.2  | 33500 |     0.2099      | 0.2957 | 0.0953 |
|     1.109     | 48.92 | 34000 |     0.2092      | 0.2955 | 0.0955 |
|    1.0905     | 49.64 | 34500 |     0.2088      | 0.2954 | 0.0953 |


## Disclaimer


Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.


## Authors


Wav2Vec2 XLS-R 300M Korean was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.


## Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xls-r-300m-korean


model author: w11wo


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, ko, dataset:kresnik/zeroth_korean, arxiv:2111.09296, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language: jv
tags:


- javanese-roberta-small-imdb
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Aku bakal menehi rating film iki 5 <mask>."


## Javanese RoBERTa Small IMDB


Javanese RoBERTa Small IMDB is a masked language model based on the [RoBERTa model](https://arxiv.org/abs/1907.11692). It was trained on Javanese IMDB movie reviews.


The model was originally the pretrained [Javanese RoBERTa Small model](https://huggingface.co/w11wo/javanese-roberta-small) and is later fine-tuned on the Javanese IMDB movie review dataset. It achieved a perplexity of 20.83 on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                         | #params  | Arch.             | Training/Validation data (text) |
|-------------------------------|----------|-------------------|---------------------------------|
| `javanese-roberta-small-imdb` |   124M   |   RoBERTa Small   | Javanese IMDB (47.5 MB of text) |


## Evaluation Results


The model was trained for 5 epochs and the following is the final result once the training ended.


| train loss | valid loss | perplexity | total time  |
|------------|------------|------------|-------------|
|    3.140   |    3.036   |   20.83    |   2:59:28   |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-roberta-small-imdb"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Aku mangan sate ing <mask> bareng konco-konco")
```


### Feature Extraction in PyTorch


```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "w11wo/javanese-roberta-small-imdb"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do consider the biases which came from the IMDB review that may be carried over into the results of this model.


## Author


Javanese RoBERTa Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-roberta-small-imdb


model author: w11wo


task: fill-mask


tags: pytorch, tf, jax, roberta, fill-mask, jv, dataset:w11wo/imdb-javanese, arxiv:1907.11692, transformers, javanese-roberta-small-imdb, license:mit, autonlp_compatible, infinity_compatible


language: zh-HK
license: apache-2.0
tags:


- automatic-speech-recognition
- generated_from_trainer
- robust-speech-event
datasets:
- common_voice
model-index:
- name: Wav2Vec2 XLS-R 300M Cantonese (zh-HK) LM
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice
type: common_voice
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 24.09
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 7
type: mozilla-foundation/common_voice_7_0
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 23.10
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 23.02
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 56.86


# Wav2Vec2 XLS-R 300M Cantonese (zh-HK) LM


Wav2Vec2 XLS-R 300M Cantonese (zh-HK) LM is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the `zh-HK` subset of the [Common Voice](https://huggingface.co/datasets/common_voice) dataset. A 5-gram Language model, trained on multiple [PyCantonese](https://pycantonese.org/data.html) corpora, was then subsequently added to this model.


This model was trained using HuggingFace's PyTorch framework and is part of the [Robust Speech Challenge Event](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.


All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-lm-v2/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-lm-v2/tensorboard) logged via Tensorboard.


As for the N-gram language model training, we followed the [blog post tutorial](https://huggingface.co/blog/wav2vec2-with-ngram) provided by HuggingFace.


## Model


| Model                             | #params | Arch. | Training/Validation data (text) |
| --------------------------------- | ------- | ----- | ------------------------------- |
| `wav2vec2-xls-r-300m-zh-HK-lm-v2` | 300M    | XLS-R | `Common Voice zh-HK` Dataset    |


## Evaluation Results


The model achieves the following results on evaluation without a language model:


| Dataset                          | CER    |
| -------------------------------- | ------ |
| `Common Voice`                   | 31.73% |
| `Common Voice 7`                 | 23.11% |
| `Common Voice 8`                 | 23.02% |
| `Robust Speech Event - Dev Data` | 56.60% |


With the addition of the language model, it achieves the following results:


| Dataset                          | CER    |
| -------------------------------- | ------ |
| `Common Voice`                   | 24.09% |
| `Common Voice 7`                 | 23.10% |
| `Common Voice 8`                 | 23.02% |
| `Robust Speech Event - Dev Data` | 56.86% |


## Training procedure


The training process did not involve the addition of a language model. The following results were simply lifted from the original automatic speech recognition [model training](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-v2).


### Training hyperparameters


The following hyperparameters were used during training:


- `learning_rate`: 0.0001
- `train_batch_size`: 8
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 4
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 2000
- `num_epochs`: 100.0
- `mixed_precision_training`: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    69.8341    | 1.34  |  500  |     80.0722     |  1.0   |  1.0   |
|    6.6418     | 2.68  | 1000  |     6.6346      |  1.0   |  1.0   |
|    6.2419     | 4.02  | 1500  |     6.2909      |  1.0   |  1.0   |
|    6.0813     | 5.36  | 2000  |     6.1150      |  1.0   |  1.0   |
|    5.9677     |  6.7  | 2500  |     6.0301      | 1.1386 | 1.0028 |
|    5.9296     | 8.04  | 3000  |     5.8975      | 1.2113 | 1.0058 |
|    5.6434     | 9.38  | 3500  |     5.5404      | 2.1624 | 1.0171 |
|    5.1974     | 10.72 | 4000  |     4.5440      | 2.1702 | 0.9366 |
|    4.3601     | 12.06 | 4500  |     3.3839      | 2.2464 | 0.8998 |
|    3.9321     | 13.4  | 5000  |     2.8785      | 2.3097 | 0.8400 |
|    3.6462     | 14.74 | 5500  |     2.5108      | 1.9623 | 0.6663 |
|    3.5156     | 16.09 | 6000  |     2.2790      | 1.6479 | 0.5706 |
|     3.32      | 17.43 | 6500  |     2.1450      | 1.8337 | 0.6244 |
|    3.1918     | 18.77 | 7000  |     1.8536      | 1.9394 | 0.6017 |
|    3.1139     | 20.11 | 7500  |     1.7205      | 1.9112 | 0.5638 |
|    2.8995     | 21.45 | 8000  |     1.5478      | 1.0624 | 0.3250 |
|    2.7572     | 22.79 | 8500  |     1.4068      | 1.1412 | 0.3367 |
|    2.6881     | 24.13 | 9000  |     1.3312      | 2.0100 | 0.5683 |
|    2.5993     | 25.47 | 9500  |     1.2553      | 2.0039 | 0.6450 |
|    2.5304     | 26.81 | 10000 |     1.2422      | 2.0394 | 0.5789 |
|    2.4352     | 28.15 | 10500 |     1.1582      | 1.9970 | 0.5507 |
|    2.3795     | 29.49 | 11000 |     1.1160      | 1.8255 | 0.4844 |
|    2.3287     | 30.83 | 11500 |     1.0775      | 1.4123 | 0.3780 |
|    2.2622     | 32.17 | 12000 |     1.0704      | 1.7445 | 0.4894 |
|    2.2225     | 33.51 | 12500 |     1.0272      | 1.7237 | 0.5058 |
|    2.1843     | 34.85 | 13000 |     0.9756      | 1.8042 | 0.5028 |
|      2.1      | 36.19 | 13500 |     0.9527      | 1.8909 | 0.6055 |
|    2.0741     | 37.53 | 14000 |     0.9418      | 1.9026 | 0.5880 |
|    2.0179     | 38.87 | 14500 |     0.9363      | 1.7977 | 0.5246 |
|    2.0615     | 40.21 | 15000 |     0.9635      | 1.8112 | 0.5599 |
|    1.9448     | 41.55 | 15500 |     0.9249      | 1.7250 | 0.4914 |
|    1.8966     | 42.89 | 16000 |     0.9023      | 1.5829 | 0.4319 |
|    1.8662     | 44.24 | 16500 |     0.9002      | 1.4833 | 0.4230 |
|    1.8136     | 45.58 | 17000 |     0.9076      | 1.1828 | 0.2987 |
|    1.7908     | 46.92 | 17500 |     0.8774      | 1.5773 | 0.4258 |
|    1.7354     | 48.26 | 18000 |     0.8727      | 1.5037 | 0.4024 |
|    1.6739     | 49.6  | 18500 |     0.8636      | 1.1239 | 0.2789 |
|    1.6457     | 50.94 | 19000 |     0.8516      | 1.2269 | 0.3104 |
|    1.5847     | 52.28 | 19500 |     0.8399      | 1.3309 | 0.3360 |
|    1.5971     | 53.62 | 20000 |     0.8441      | 1.3153 | 0.3335 |
|     1.602     | 54.96 | 20500 |     0.8590      | 1.2932 | 0.3433 |
|    1.5063     | 56.3  | 21000 |     0.8334      | 1.1312 | 0.2875 |
|    1.4631     | 57.64 | 21500 |     0.8474      | 1.1698 | 0.2999 |
|    1.4997     | 58.98 | 22000 |     0.8638      | 1.4279 | 0.3854 |
|    1.4301     | 60.32 | 22500 |     0.8550      | 1.2737 | 0.3300 |
|    1.3798     | 61.66 | 23000 |     0.8266      | 1.1802 | 0.2934 |
|    1.3454     | 63.0  | 23500 |     0.8235      | 1.3816 | 0.3711 |
|    1.3678     | 64.34 | 24000 |     0.8550      | 1.6427 | 0.5035 |
|    1.3761     | 65.68 | 24500 |     0.8510      | 1.6709 | 0.4907 |
|    1.2668     | 67.02 | 25000 |     0.8515      | 1.5842 | 0.4505 |
|    1.2835     | 68.36 | 25500 |     0.8283      | 1.5353 | 0.4221 |
|    1.2961     | 69.7  | 26000 |     0.8339      | 1.5743 | 0.4369 |
|    1.2656     | 71.05 | 26500 |     0.8331      | 1.5331 | 0.4217 |
|    1.2556     | 72.39 | 27000 |     0.8242      | 1.4708 | 0.4109 |
|    1.2043     | 73.73 | 27500 |     0.8245      | 1.4469 | 0.4031 |
|    1.2722     | 75.07 | 28000 |     0.8202      | 1.4924 | 0.4096 |
|     1.202     | 76.41 | 28500 |     0.8290      | 1.3807 | 0.3719 |
|    1.1679     | 77.75 | 29000 |     0.8195      | 1.4097 | 0.3749 |
|    1.1967     | 79.09 | 29500 |     0.8059      | 1.2074 | 0.3077 |
|    1.1241     | 80.43 | 30000 |     0.8137      | 1.2451 | 0.3270 |
|    1.1414     | 81.77 | 30500 |     0.8117      | 1.2031 | 0.3121 |
|     1.132     | 83.11 | 31000 |     0.8234      | 1.4266 | 0.3901 |
|    1.0982     | 84.45 | 31500 |     0.8064      | 1.3712 | 0.3607 |
|    1.0797     | 85.79 | 32000 |     0.8167      | 1.3356 | 0.3562 |
|    1.0119     | 87.13 | 32500 |     0.8215      | 1.2754 | 0.3268 |
|    1.0216     | 88.47 | 33000 |     0.8163      | 1.2512 | 0.3184 |
|    1.0375     | 89.81 | 33500 |     0.8137      | 1.2685 | 0.3290 |
|    0.9794     | 91.15 | 34000 |     0.8220      | 1.2724 | 0.3255 |
|    1.0207     | 92.49 | 34500 |     0.8165      | 1.2906 | 0.3361 |
|    1.0169     | 93.83 | 35000 |     0.8153      | 1.2819 | 0.3305 |
|    1.0127     | 95.17 | 35500 |     0.8187      | 1.2832 | 0.3252 |
|    0.9978     | 96.51 | 36000 |     0.8111      | 1.2612 | 0.3210 |
|    0.9923     | 97.85 | 36500 |     0.8076      | 1.2278 | 0.3122 |
|    1.0451     | 99.2  | 37000 |     0.8086      | 1.2451 | 0.3156 |


## Disclaimer


Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.


## Authors


Wav2Vec2 XLS-R 300M Cantonese (zh-HK) LM was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.


## Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-300m-zh-HK-lm-v2


model author: w11wo


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, zh-HK, dataset:common_voice, arxiv:2111.09296, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language: id
tags:


- indo-roberta-small
license: mit
datasets:
- wikipedia
widget:
- text: "Karena pandemi ini, kita harus <mask> di rumah saja."


## Indo RoBERTa Small


Indo RoBERTa Small is a masked language model based on the [RoBERTa model](https://arxiv.org/abs/1907.11692). It was trained on the latest (late December 2020) Indonesian Wikipedia articles.


The model was trained from scratch and achieved a perplexity of 48.27 on the validation dataset (20% of the articles). Many of the techniques used
are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), where Sylvain Gugger fine-tuned a [DistilGPT-2](https://huggingface.co/distilgpt2) on [Wikitext2](https://render.githubusercontent.com/view/ipynb?color_mode=dark&commit=43d63e390e8a82f7ae49aa1a877419343a213cb4&enc_url=68747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f68756767696e67666163652f6e6f7465626f6f6b732f343364363365333930653861383266376165343961613161383737343139333433613231336362342f6578616d706c65732f6c616e67756167655f6d6f64656c696e672e6970796e62&nwo=huggingface%2Fnotebooks&path=examples%2Flanguage_modeling.ipynb&repository_id=272452525&repository_type=Repository).


Hugging Face's [Transformers]((https://huggingface.co/transformers)) library was used to train the model -- utilizing the base RoBERTa model and their `Trainer` class. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                | #params | Arch.    | Training/Validation data (text)      |
|----------------------|---------|----------|---------------------------------------|
| `indo-roberta-small` |   84M   | RoBERTa  | Indonesian Wikipedia (3.1 GB of text) |


## Evaluation Results


The model was trained for 3 epochs and the following is the final result once the training ended.


| train loss | valid loss | perplexity | total time |
|------------|------------|------------|------------|
|    4.071   |    3.876   |    48.27   |   3:40:55  |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/indo-roberta-small"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Budi sedang <mask> di sekolah.")
```


### Feature Extraction in PyTorch


```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "w11wo/indo-roberta-small"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "Budi sedang berada di sekolah."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.


## Author


Indo RoBERTa Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indo-roberta-small


model author: w11wo


task: fill-mask


tags: pytorch, tf, jax, roberta, fill-mask, id, dataset:wikipedia, arxiv:1907.11692, transformers, indo-roberta-small, license:mit, autonlp_compatible, infinity_compatible


language: id
tags:


- indo-gpt2-small
license: mit
datasets:
- wikipedia
widget:
- text: "Nama saya Budi, dari Indonesia"


## Indo GPT-2 Small


Indo GPT-2 Small is a language model based on the [GPT-2 model](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). It was trained on the latest (late December 2020) Indonesian Wikipedia articles.


The model was originally HuggingFace's pretrained [English GPT-2 model](https://huggingface.co/transformers/model_doc/gpt2.html) and is later fine-tuned on the Indonesian dataset. Many of the techniques used
are based on a [notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb)/[blog](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787) shared by [Pierre Guillou](https://medium.com/@pierre_guillou), where Pierre Guillou fine-tuned the English GPT-2 model on a Portuguese dataset.


Frameworks used include HuggingFace's [Transformers](https://huggingface.co/transformers) and fast.ai's [Deep Learning library](https://docs.fast.ai/). PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model             | #params | Arch.       | Training /Validation data (text)      |
|-------------------|---------|-------------|---------------------------------------|
| `indo-gpt2-small` |  124M   | GPT-2 Small | Indonesian Wikipedia (3.1 GB of text) |


## Evaluation Results


The model was trained for only 1 epoch and the following is the final result once the training ended.


| epoch | train loss | valid loss | perplexity | total time |
|-------|------------|------------|------------|------------|
|   0   |    2.981   |    2.936   |    18.85   |   2:45:25  |


## How to Use (PyTorch)


### Load Model and Byte-level Tokenizer


```python
from transformers import GPT2TokenizerFast, GPT2LMHeadModel
pretrained_name = "w11wo/indo-gpt2-small"
tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_name)
tokenizer.model_max_length = 1024
model = GPT2LMHeadModel.from_pretrained(pretrained_name)
```


### Generate a Sequence


```python
# sample prompt
prompt = "Nama saya Budi, dari Indonesia"
input_ids = tokenizer.encode(prompt, return_tensors='pt')
model.eval()

# generate output using top-k sampling
sample_outputs = model.generate(input_ids,
                                pad_token_id=50256,
                                do_sample=True,
                                max_length=40,
                                min_length=40,
                                top_k=40,
                                num_return_sequences=1)

for i, sample_output in enumerate(sample_outputs):
    print(tokenizer.decode(sample_output.tolist()))
```


## Disclaimer


## Credits


Major thanks to Pierre Guillou for sharing his work, which did not only enable me to realize this project but also taught me tons of new, exciting stuff.


## Author


Indo GPT-2 Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indo-gpt2-small


model author: w11wo


task: text-generation


tags: pytorch, tf, jax, gpt2, text-generation, id, dataset:wikipedia, transformers, indo-gpt2-small, license:mit


license: apache-2.0
tags:


- automatic-speech-recognition
- kresnik/zeroth_korean
- generated_from_trainer
datasets:
- zeroth_korean_asr
model-index:
- name: ''
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the KRESNIK/ZEROTH_KOREAN - CLEAN dataset.
It achieves the following results on the evaluation set:


- Loss: 0.1313
- Wer: 0.2261
- Cer: 0.0899


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:------:|
| 5.0816        | 1.44  | 500   | 4.7941          | 1.0    | 1.0    |
| 4.2863        | 2.88  | 1000  | 4.2720          | 0.9961 | 0.9396 |
| 3.3962        | 4.32  | 1500  | 2.4010          | 1.0172 | 0.5761 |
| 2.4739        | 5.76  | 2000  | 1.3089          | 0.8718 | 0.4025 |
| 2.2215        | 7.2   | 2500  | 0.9522          | 0.6912 | 0.2600 |
| 2.0338        | 8.64  | 3000  | 0.8950          | 0.7262 | 0.2943 |
| 1.9666        | 10.09 | 3500  | 0.7557          | 0.6773 | 0.2694 |
| 1.8599        | 11.53 | 4000  | 0.6478          | 0.6161 | 0.2182 |
| 1.7943        | 12.97 | 4500  | 0.6073          | 0.6042 | 0.2170 |
| 1.7241        | 14.41 | 5000  | 0.5658          | 0.5344 | 0.1768 |
| 1.6469        | 15.85 | 5500  | 0.5329          | 0.5388 | 0.1854 |
| 1.5917        | 17.29 | 6000  | 0.5348          | 0.5577 | 0.2048 |
| 1.5364        | 18.73 | 6500  | 0.5058          | 0.5371 | 0.1946 |
| 1.4765        | 20.17 | 7000  | 0.4140          | 0.4248 | 0.1317 |
| 1.3973        | 21.61 | 7500  | 0.3850          | 0.4077 | 0.1183 |
| 1.3456        | 23.05 | 8000  | 0.3349          | 0.4035 | 0.1275 |
| 1.2933        | 24.49 | 8500  | 0.3296          | 0.3612 | 0.1013 |
| 1.2364        | 25.93 | 9000  | 0.3012          | 0.3248 | 0.0902 |
| 1.1763        | 27.38 | 9500  | 0.2968          | 0.3117 | 0.0868 |
| 1.1328        | 28.82 | 10000 | 0.2587          | 0.3096 | 0.0866 |
| 1.0703        | 30.26 | 10500 | 0.2327          | 0.3248 | 0.1120 |
| 1.021         | 31.7  | 11000 | 0.2210          | 0.3345 | 0.1215 |
| 0.9959        | 33.14 | 11500 | 0.2023          | 0.3201 | 0.1151 |
| 0.9495        | 34.58 | 12000 | 0.1985          | 0.2995 | 0.1192 |
| 0.921         | 36.02 | 12500 | 0.1924          | 0.3011 | 0.1218 |
| 0.8567        | 37.46 | 13000 | 0.1758          | 0.2679 | 0.0983 |
| 0.8448        | 38.9  | 13500 | 0.1602          | 0.2662 | 0.1027 |
| 0.8111        | 40.34 | 14000 | 0.1604          | 0.2596 | 0.1015 |
| 0.7823        | 41.78 | 14500 | 0.1475          | 0.2472 | 0.0980 |
| 0.765         | 43.23 | 15000 | 0.1453          | 0.2384 | 0.0939 |
| 0.7375        | 44.67 | 15500 | 0.1406          | 0.2327 | 0.0883 |
| 0.7233        | 46.11 | 16000 | 0.1377          | 0.2295 | 0.0892 |
| 0.6949        | 47.55 | 16500 | 0.1325          | 0.2304 | 0.0917 |
| 0.6838        | 48.99 | 17000 | 0.1321          | 0.2280 | 0.0906 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xls-r-300m-korean-v2


model author: w11wo


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:zeroth_korean_asr, transformers, license:apache-2.0, kresnik/zeroth_korean, generated_from_trainer, model-index


language: lo
tags:


- lao-roberta-base-pos-tagger
license: mit
widget:
- text: "ຮ້ອງ ມ່ວນ ແທ້ ສຽງດີ ອິຫຼີ"


## Lao RoBERTa Base POS Tagger


Lao RoBERTa Base POS Tagger is a part-of-speech token-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Lao RoBERTa Base](https://huggingface.co/w11wo/lao-roberta-base) model, which is then fine-tuned on the [`Yunshan Cup 2020`](https://github.com/GKLMIP/Yunshan-Cup-2020) dataset consisting of tag-labelled Lao corpus.


After training, the model achieved an evaluation accuracy of 83.14%. On the benchmark test set, the model achieved an accuracy of 83.30%.


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.


## Model


| Model                         | #params | Arch.        | Training/Validation data (text) |
| ----------------------------- | ------- | ------------ | ------------------------------- |
| `lao-roberta-base-pos-tagger` | 124M    | RoBERTa Base | `Yunshan Cup 2020`              |


## Evaluation Results


The model was trained for 15 epochs, with a batch size of 8, a learning rate of 5e-5, with cosine annealing to 0. The best model was loaded at the end.


| Epoch | Training Loss | Validation Loss | Accuracy |
| ----- | ------------- | --------------- | -------- |
| 1     | 1.026100      | 0.733780        | 0.746021 |
| 2     | 0.646900      | 0.659625        | 0.775688 |
| 3     | 0.500400      | 0.576214        | 0.798523 |
| 4     | 0.385400      | 0.606503        | 0.805269 |
| 5     | 0.288000      | 0.652493        | 0.809092 |
| 6     | 0.204600      | 0.671678        | 0.815216 |
| 7     | 0.145200      | 0.704693        | 0.818209 |
| 8     | 0.098700      | 0.830561        | 0.816998 |
| 9     | 0.066100      | 0.883329        | 0.825232 |
| 10    | 0.043900      | 0.933347        | 0.825664 |
| 11    | 0.027200      | 0.992055        | 0.828449 |
| 12    | 0.017300      | 1.054874        | 0.830819 |
| 13    | 0.011500      | 1.081638        | 0.830940 |
| 14    | 0.008500      | 1.094252        | 0.831304 |
| 15    | 0.007400      | 1.097428        | 0.831442 |


## How to Use


### As Token Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/lao-roberta-base-pos-tagger"

nlp = pipeline(
    "token-classification",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("ຮ້ອງ ມ່ວນ ແທ້ ສຽງດີ ອິຫຼີ")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and the `Yunshan Cup 2020` dataset that may be carried over into the results of this model.


## Author


Lao RoBERTa Base POS Tagger was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: lao-roberta-base-pos-tagger


model author: w11wo


task: token-classification


tags: pytorch, roberta, token-classification, lo, arxiv:1907.11692, transformers, lao-roberta-base-pos-tagger, license:mit, autonlp_compatible, infinity_compatible


language: zh-HK
license: apache-2.0
tags:


- automatic-speech-recognition
- generated_from_trainer
- robust-speech-event
datasets:
- common_voice
model-index:
- name: Wav2Vec2 XLS-R 300M Cantonese (zh-HK)
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice
type: common_voice
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 31.73
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 7
type: mozilla-foundation/common_voice_7_0
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 23.11
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 23.02
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 56.60


# Wav2Vec2 XLS-R 300M Cantonese (zh-HK)


Wav2Vec2 XLS-R 300M Cantonese (zh-HK) is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the `zh-HK` subset of the [Common Voice](https://huggingface.co/datasets/common_voice) dataset.


This model was trained using HuggingFace's PyTorch framework and is part of the [Robust Speech Challenge Event](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.


All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-v2/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-zh-HK-v2/tensorboard) logged via Tensorboard.


## Model


| Model                          | #params | Arch. | Training/Validation data (text) |
| ------------------------------ | ------- | ----- | ------------------------------- |
| `wav2vec2-xls-r-300m-zh-HK-v2` | 300M    | XLS-R | `Common Voice zh-HK` Dataset    |


## Evaluation Results


The model achieves the following results on evaluation:


| Dataset                          | Loss   | CER    |
| -------------------------------- | ------ | ------ |
| `Common Voice`                   | 0.8089 | 31.73% |
| `Common Voice 7`                 | N/A    | 23.11% |
| `Common Voice 8`                 | N/A    | 23.02% |
| `Robust Speech Event - Dev Data` | N/A    | 56.60% |


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- `learning_rate`: 0.0001
- `train_batch_size`: 8
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 4
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 2000
- `num_epochs`: 100.0
- `mixed_precision_training`: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    69.8341    | 1.34  |  500  |     80.0722     |  1.0   |  1.0   |
|    6.6418     | 2.68  | 1000  |     6.6346      |  1.0   |  1.0   |
|    6.2419     | 4.02  | 1500  |     6.2909      |  1.0   |  1.0   |
|    6.0813     | 5.36  | 2000  |     6.1150      |  1.0   |  1.0   |
|    5.9677     |  6.7  | 2500  |     6.0301      | 1.1386 | 1.0028 |
|    5.9296     | 8.04  | 3000  |     5.8975      | 1.2113 | 1.0058 |
|    5.6434     | 9.38  | 3500  |     5.5404      | 2.1624 | 1.0171 |
|    5.1974     | 10.72 | 4000  |     4.5440      | 2.1702 | 0.9366 |
|    4.3601     | 12.06 | 4500  |     3.3839      | 2.2464 | 0.8998 |
|    3.9321     | 13.4  | 5000  |     2.8785      | 2.3097 | 0.8400 |
|    3.6462     | 14.74 | 5500  |     2.5108      | 1.9623 | 0.6663 |
|    3.5156     | 16.09 | 6000  |     2.2790      | 1.6479 | 0.5706 |
|     3.32      | 17.43 | 6500  |     2.1450      | 1.8337 | 0.6244 |
|    3.1918     | 18.77 | 7000  |     1.8536      | 1.9394 | 0.6017 |
|    3.1139     | 20.11 | 7500  |     1.7205      | 1.9112 | 0.5638 |
|    2.8995     | 21.45 | 8000  |     1.5478      | 1.0624 | 0.3250 |
|    2.7572     | 22.79 | 8500  |     1.4068      | 1.1412 | 0.3367 |
|    2.6881     | 24.13 | 9000  |     1.3312      | 2.0100 | 0.5683 |
|    2.5993     | 25.47 | 9500  |     1.2553      | 2.0039 | 0.6450 |
|    2.5304     | 26.81 | 10000 |     1.2422      | 2.0394 | 0.5789 |
|    2.4352     | 28.15 | 10500 |     1.1582      | 1.9970 | 0.5507 |
|    2.3795     | 29.49 | 11000 |     1.1160      | 1.8255 | 0.4844 |
|    2.3287     | 30.83 | 11500 |     1.0775      | 1.4123 | 0.3780 |
|    2.2622     | 32.17 | 12000 |     1.0704      | 1.7445 | 0.4894 |
|    2.2225     | 33.51 | 12500 |     1.0272      | 1.7237 | 0.5058 |
|    2.1843     | 34.85 | 13000 |     0.9756      | 1.8042 | 0.5028 |
|      2.1      | 36.19 | 13500 |     0.9527      | 1.8909 | 0.6055 |
|    2.0741     | 37.53 | 14000 |     0.9418      | 1.9026 | 0.5880 |
|    2.0179     | 38.87 | 14500 |     0.9363      | 1.7977 | 0.5246 |
|    2.0615     | 40.21 | 15000 |     0.9635      | 1.8112 | 0.5599 |
|    1.9448     | 41.55 | 15500 |     0.9249      | 1.7250 | 0.4914 |
|    1.8966     | 42.89 | 16000 |     0.9023      | 1.5829 | 0.4319 |
|    1.8662     | 44.24 | 16500 |     0.9002      | 1.4833 | 0.4230 |
|    1.8136     | 45.58 | 17000 |     0.9076      | 1.1828 | 0.2987 |
|    1.7908     | 46.92 | 17500 |     0.8774      | 1.5773 | 0.4258 |
|    1.7354     | 48.26 | 18000 |     0.8727      | 1.5037 | 0.4024 |
|    1.6739     | 49.6  | 18500 |     0.8636      | 1.1239 | 0.2789 |
|    1.6457     | 50.94 | 19000 |     0.8516      | 1.2269 | 0.3104 |
|    1.5847     | 52.28 | 19500 |     0.8399      | 1.3309 | 0.3360 |
|    1.5971     | 53.62 | 20000 |     0.8441      | 1.3153 | 0.3335 |
|     1.602     | 54.96 | 20500 |     0.8590      | 1.2932 | 0.3433 |
|    1.5063     | 56.3  | 21000 |     0.8334      | 1.1312 | 0.2875 |
|    1.4631     | 57.64 | 21500 |     0.8474      | 1.1698 | 0.2999 |
|    1.4997     | 58.98 | 22000 |     0.8638      | 1.4279 | 0.3854 |
|    1.4301     | 60.32 | 22500 |     0.8550      | 1.2737 | 0.3300 |
|    1.3798     | 61.66 | 23000 |     0.8266      | 1.1802 | 0.2934 |
|    1.3454     | 63.0  | 23500 |     0.8235      | 1.3816 | 0.3711 |
|    1.3678     | 64.34 | 24000 |     0.8550      | 1.6427 | 0.5035 |
|    1.3761     | 65.68 | 24500 |     0.8510      | 1.6709 | 0.4907 |
|    1.2668     | 67.02 | 25000 |     0.8515      | 1.5842 | 0.4505 |
|    1.2835     | 68.36 | 25500 |     0.8283      | 1.5353 | 0.4221 |
|    1.2961     | 69.7  | 26000 |     0.8339      | 1.5743 | 0.4369 |
|    1.2656     | 71.05 | 26500 |     0.8331      | 1.5331 | 0.4217 |
|    1.2556     | 72.39 | 27000 |     0.8242      | 1.4708 | 0.4109 |
|    1.2043     | 73.73 | 27500 |     0.8245      | 1.4469 | 0.4031 |
|    1.2722     | 75.07 | 28000 |     0.8202      | 1.4924 | 0.4096 |
|     1.202     | 76.41 | 28500 |     0.8290      | 1.3807 | 0.3719 |
|    1.1679     | 77.75 | 29000 |     0.8195      | 1.4097 | 0.3749 |
|    1.1967     | 79.09 | 29500 |     0.8059      | 1.2074 | 0.3077 |
|    1.1241     | 80.43 | 30000 |     0.8137      | 1.2451 | 0.3270 |
|    1.1414     | 81.77 | 30500 |     0.8117      | 1.2031 | 0.3121 |
|     1.132     | 83.11 | 31000 |     0.8234      | 1.4266 | 0.3901 |
|    1.0982     | 84.45 | 31500 |     0.8064      | 1.3712 | 0.3607 |
|    1.0797     | 85.79 | 32000 |     0.8167      | 1.3356 | 0.3562 |
|    1.0119     | 87.13 | 32500 |     0.8215      | 1.2754 | 0.3268 |
|    1.0216     | 88.47 | 33000 |     0.8163      | 1.2512 | 0.3184 |
|    1.0375     | 89.81 | 33500 |     0.8137      | 1.2685 | 0.3290 |
|    0.9794     | 91.15 | 34000 |     0.8220      | 1.2724 | 0.3255 |
|    1.0207     | 92.49 | 34500 |     0.8165      | 1.2906 | 0.3361 |
|    1.0169     | 93.83 | 35000 |     0.8153      | 1.2819 | 0.3305 |
|    1.0127     | 95.17 | 35500 |     0.8187      | 1.2832 | 0.3252 |
|    0.9978     | 96.51 | 36000 |     0.8111      | 1.2612 | 0.3210 |
|    0.9923     | 97.85 | 36500 |     0.8076      | 1.2278 | 0.3122 |
|    1.0451     | 99.2  | 37000 |     0.8086      | 1.2451 | 0.3156 |


## Disclaimer


Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.


## Authors


Wav2Vec2 XLS-R 300M Cantonese (zh-HK) was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.


## Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-300m-zh-HK-v2


model author: w11wo


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, zh-HK, dataset:common_voice, arxiv:2111.09296, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language: ko
license: apache-2.0
tags:


- automatic-speech-recognition
- generated_from_trainer
- robust-speech-event
datasets:
- kresnik/zeroth_korean
model-index:
- name: Wav2Vec2 XLS-R 300M Korean LM
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Zeroth Korean
type: kresnik/zeroth_korean
args: clean
metrics:
    - name: Test WER
type: wer
value: 30.94
    - name: Test CER
type: cer
value: 7.97
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: ko
metrics:
    - name: Test WER
type: wer
value: 68.34
    - name: Test CER
type: cer
value: 37.08


# Wav2Vec2 XLS-R 300M Korean LM


Wav2Vec2 XLS-R 300M Korean LM is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the [Zeroth Korean](https://huggingface.co/datasets/kresnik/zeroth_korean) dataset. A 5-gram Language model, trained on the Korean subset of [Open Subtitles](https://huggingface.co/datasets/open_subtitles), was then subsequently added to this model.


All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean-lm/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean-lm/tensorboard) logged via Tensorboard.


As for the N-gram language model training, we followed the [blog post tutorial](https://huggingface.co/blog/wav2vec2-with-ngram) provided by HuggingFace.


## Model


| Model                           | #params | Arch. | Training/Validation data (text) |
| ------------------------------- | ------- | ----- | ------------------------------- |
| `wav2vec2-xls-r-300m-korean-lm` | 300M    | XLS-R | `Zeroth Korean` Dataset         |


## Evaluation Results


The model achieves the following results on evaluation without a language model: