| Dataset                          | WER    | CER    |
| -------------------------------- | ------ | ------ |
| `Zeroth Korean`                  | 29.54% | 9.53%  |
| `Robust Speech Event - Dev Data` | 76.26% | 38.67% |


With the addition of the language model, it achieves the following results:


| Dataset                          | WER    | CER    |
| -------------------------------- | ------ | ------ |
| `Zeroth Korean`                  | 30.94% | 7.97%  |
| `Robust Speech Event - Dev Data` | 68.34% | 37.08% |


## Training procedure


The training process did not involve the addition of a language model. The following results were simply lifted from the original automatic speech recognition [model training](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean).


### Training hyperparameters


The following hyperparameters were used during training:


- `learning_rate`: 7.5e-05
- `train_batch_size`: 8
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 4
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 2000
- `num_epochs`: 50.0
- `mixed_precision_training`: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    19.7138    | 0.72  |  500  |     19.6427     |  1.0   |  1.0   |
|    4.8039     | 1.44  | 1000  |     4.7842      |  1.0   |  1.0   |
|    4.5619     | 2.16  | 1500  |     4.5608      | 0.9992 | 0.9598 |
|     4.254     | 2.88  | 2000  |     4.2729      | 0.9955 | 0.9063 |
|    4.1905     |  3.6  | 2500  |     4.2257      | 0.9903 | 0.8758 |
|    4.0683     | 4.32  | 3000  |     3.9294      | 0.9937 | 0.7911 |
|     3.486     | 5.04  | 3500  |     2.7045      | 1.0012 | 0.5934 |
|     2.946     | 5.75  | 4000  |     1.9691      | 0.9425 | 0.4634 |
|     2.634     | 6.47  | 4500  |     1.5212      | 0.8807 | 0.3850 |
|    2.4066     | 7.19  | 5000  |     1.2551      | 0.8177 | 0.3601 |
|    2.2651     | 7.91  | 5500  |     1.0423      | 0.7650 | 0.3039 |
|    2.1828     | 8.63  | 6000  |     0.9599      | 0.7273 | 0.3106 |
|    2.1023     | 9.35  | 6500  |     0.9482      | 0.7161 | 0.3063 |
|    2.0536     | 10.07 | 7000  |     0.8242      | 0.6767 | 0.2860 |
|    1.9803     | 10.79 | 7500  |     0.7643      | 0.6563 | 0.2637 |
|    1.9468     | 11.51 | 8000  |     0.7319      | 0.6441 | 0.2505 |
|    1.9178     | 12.23 | 8500  |     0.6937      | 0.6320 | 0.2489 |
|    1.8515     | 12.95 | 9000  |     0.6443      | 0.6053 | 0.2196 |
|    1.8083     | 13.67 | 9500  |     0.6286      | 0.6122 | 0.2148 |
|     1.819     | 14.39 | 10000 |     0.6015      | 0.5986 | 0.2074 |
|    1.7684     | 15.11 | 10500 |     0.5682      | 0.5741 | 0.1982 |
|    1.7195     | 15.83 | 11000 |     0.5385      | 0.5592 | 0.2007 |
|    1.7044     | 16.55 | 11500 |     0.5362      | 0.5524 | 0.2097 |
|    1.6879     | 17.27 | 12000 |     0.5119      | 0.5489 | 0.2083 |
|     1.656     | 17.98 | 12500 |     0.4990      | 0.5362 | 0.1968 |
|    1.6122     | 18.7  | 13000 |     0.4561      | 0.5092 | 0.1900 |
|    1.5919     | 19.42 | 13500 |     0.4778      | 0.5225 | 0.1975 |
|    1.5896     | 20.14 | 14000 |     0.4563      | 0.5098 | 0.1859 |
|    1.5589     | 20.86 | 14500 |     0.4362      | 0.4940 | 0.1725 |
|    1.5353     | 21.58 | 15000 |     0.4140      | 0.4826 | 0.1580 |
|    1.5441     | 22.3  | 15500 |     0.4031      | 0.4742 | 0.1550 |
|    1.5116     | 23.02 | 16000 |     0.3916      | 0.4748 | 0.1545 |
|    1.4731     | 23.74 | 16500 |     0.3841      | 0.4810 | 0.1542 |
|    1.4647     | 24.46 | 17000 |     0.3752      | 0.4524 | 0.1475 |
|    1.4328     | 25.18 | 17500 |     0.3587      | 0.4476 | 0.1461 |
|    1.4129     | 25.9  | 18000 |     0.3429      | 0.4242 | 0.1366 |
|    1.4062     | 26.62 | 18500 |     0.3450      | 0.4251 | 0.1355 |
|    1.3928     | 27.34 | 19000 |     0.3297      | 0.4145 | 0.1322 |
|    1.3906     | 28.06 | 19500 |     0.3210      | 0.4185 | 0.1336 |
|     1.358     | 28.78 | 20000 |     0.3131      | 0.3970 | 0.1275 |
|    1.3445     | 29.5  | 20500 |     0.3069      | 0.3920 | 0.1276 |
|    1.3159     | 30.22 | 21000 |     0.3035      | 0.3961 | 0.1255 |
|    1.3044     | 30.93 | 21500 |     0.2952      | 0.3854 | 0.1242 |
|    1.3034     | 31.65 | 22000 |     0.2966      | 0.3772 | 0.1227 |
|    1.2963     | 32.37 | 22500 |     0.2844      | 0.3706 | 0.1208 |
|    1.2765     | 33.09 | 23000 |     0.2841      | 0.3567 | 0.1173 |
|    1.2438     | 33.81 | 23500 |     0.2734      | 0.3552 | 0.1137 |
|    1.2487     | 34.53 | 24000 |     0.2703      | 0.3502 | 0.1118 |
|    1.2249     | 35.25 | 24500 |     0.2650      | 0.3484 | 0.1142 |
|    1.2229     | 35.97 | 25000 |     0.2584      | 0.3374 | 0.1097 |
|    1.2374     | 36.69 | 25500 |     0.2568      | 0.3337 | 0.1095 |
|    1.2153     | 37.41 | 26000 |     0.2494      | 0.3327 | 0.1071 |
|    1.1925     | 38.13 | 26500 |     0.2518      | 0.3366 | 0.1077 |
|    1.1908     | 38.85 | 27000 |     0.2437      | 0.3272 | 0.1057 |
|    1.1858     | 39.57 | 27500 |     0.2396      | 0.3265 | 0.1044 |
|    1.1808     | 40.29 | 28000 |     0.2373      | 0.3156 | 0.1028 |
|    1.1842     | 41.01 | 28500 |     0.2356      | 0.3152 | 0.1026 |
|    1.1668     | 41.73 | 29000 |     0.2319      | 0.3188 | 0.1025 |
|    1.1448     | 42.45 | 29500 |     0.2293      | 0.3099 | 0.0995 |
|    1.1327     | 43.17 | 30000 |     0.2265      | 0.3047 | 0.0979 |
|    1.1307     | 43.88 | 30500 |     0.2222      | 0.3078 | 0.0989 |
|    1.1419     | 44.6  | 31000 |     0.2215      | 0.3038 | 0.0981 |
|    1.1231     | 45.32 | 31500 |     0.2193      | 0.3013 | 0.0972 |
|     1.139     | 46.04 | 32000 |     0.2162      | 0.3007 | 0.0968 |
|    1.1114     | 46.76 | 32500 |     0.2122      | 0.2982 | 0.0960 |
|     1.111     | 47.48 | 33000 |     0.2125      | 0.2946 | 0.0948 |
|    1.0982     | 48.2  | 33500 |     0.2099      | 0.2957 | 0.0953 |
|     1.109     | 48.92 | 34000 |     0.2092      | 0.2955 | 0.0955 |
|    1.0905     | 49.64 | 34500 |     0.2088      | 0.2954 | 0.0953 |


## Disclaimer


Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.


## Authors


Wav2Vec2 XLS-R 300M Korean LM was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.


## Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xls-r-300m-korean-lm


model author: w11wo


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, ko, dataset:kresnik/zeroth_korean, arxiv:2111.09296, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language:


- zh-HK
license: apache-2.0
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: ''
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the COMMON_VOICE - ZH-HK dataset.
It achieves the following results on the evaluation set:


- Loss: 1.0450
- Wer: 1.8991
- Cer: 0.4717


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 7.5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:------:|
| 77.5305       | 1.34  | 500   | 92.6669         | 1.0    | 1.0    |
| 9.3568        | 2.68  | 1000  | 7.6860          | 1.0    | 1.0    |
| 6.2735        | 4.02  | 1500  | 6.3664          | 1.0    | 1.0    |
| 6.1322        | 5.36  | 2000  | 6.1999          | 1.0    | 1.0    |
| 6.0244        | 6.7   | 2500  | 6.0719          | 1.0009 | 1.0000 |
| 5.9934        | 8.04  | 3000  | 5.9246          | 1.4023 | 1.0108 |
| 5.807         | 9.38  | 3500  | 5.7572          | 1.8696 | 1.0156 |
| 5.6889        | 10.72 | 4000  | 5.5060          | 1.9285 | 1.0035 |
| 5.4225        | 12.06 | 4500  | 4.9907          | 2.1646 | 1.0005 |
| 4.6644        | 13.4  | 5000  | 3.9605          | 2.0888 | 0.8890 |
| 4.2808        | 14.74 | 5500  | 3.4318          | 2.1360 | 0.8296 |
| 4.1329        | 16.09 | 6000  | 3.1680          | 2.2161 | 0.8428 |
| 3.8779        | 17.43 | 6500  | 2.9402          | 2.2343 | 0.8112 |
| 3.7395        | 18.77 | 7000  | 2.6735          | 2.1611 | 0.7561 |
| 3.5842        | 20.11 | 7500  | 2.3968          | 2.1364 | 0.6983 |
| 3.3861        | 21.45 | 8000  | 2.2187          | 2.1823 | 0.6786 |
| 3.2381        | 22.79 | 8500  | 2.0439          | 2.0273 | 0.6105 |
| 3.1511        | 24.13 | 9000  | 1.9231          | 2.0186 | 0.6136 |
| 3.0449        | 25.47 | 9500  | 1.8028          | 2.1044 | 0.6111 |
| 2.9652        | 26.81 | 10000 | 1.6802          | 2.0883 | 0.5933 |
| 2.8411        | 28.15 | 10500 | 1.5863          | 2.0476 | 0.5600 |
| 2.7381        | 29.49 | 11000 | 1.4835          | 2.0390 | 0.5425 |
| 2.6989        | 30.83 | 11500 | 1.4134          | 2.0879 | 0.5553 |
| 2.6437        | 32.17 | 12000 | 1.3482          | 1.9792 | 0.5212 |
| 2.5781        | 33.51 | 12500 | 1.3063          | 1.9926 | 0.5234 |
| 2.5456        | 34.85 | 13000 | 1.2719          | 1.9736 | 0.5055 |
| 2.489         | 36.19 | 13500 | 1.2196          | 1.9749 | 0.5008 |
| 2.442         | 37.53 | 14000 | 1.1836          | 1.9770 | 0.4999 |
| 2.4021        | 38.87 | 14500 | 1.1609          | 1.9861 | 0.5082 |
| 2.4228        | 40.21 | 15000 | 1.1361          | 1.9359 | 0.4882 |
| 2.3187        | 41.55 | 15500 | 1.1121          | 1.9493 | 0.4926 |
| 2.268         | 42.89 | 16000 | 1.0901          | 1.9346 | 0.4874 |
| 2.27          | 44.24 | 16500 | 1.0758          | 1.9277 | 0.4790 |
| 2.2367        | 45.58 | 17000 | 1.0699          | 1.9298 | 0.4804 |
| 2.2227        | 46.92 | 17500 | 1.0571          | 1.9268 | 0.4792 |
| 2.242         | 48.26 | 18000 | 1.0503          | 1.9078 | 0.4732 |
| 2.1974        | 49.6  | 18500 | 1.0460          | 1.9004 | 0.4724 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-300m-zh-HK


model author: w11wo


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, zh-HK, dataset:common_voice, transformers, license:apache-2.0, common_voice, generated_from_trainer, model-index


language: id
tags:


- indonesian-roberta-base-indolem-sentiment-classifier-fold-0
license: mit
datasets:
- indolem
widget:
- text: "Pelayanan hotel ini sangat baik."


## Indonesian RoBERTa Base IndoLEM Sentiment Classifier


Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).


A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.


On **fold 0**, the model achieved an F1 of 86.42% on dev/validation and 83.12% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.


## Model


| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-0` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |


## Evaluation Results


The model was trained for 10 epochs and the best model was loaded at the end.


| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.563500      | 0.420457        | 0.796992 | 0.626728 | 0.680000  | 0.581197 |
| 2     | 0.293600      | 0.281360        | 0.884712 | 0.811475 | 0.779528  | 0.846154 |
| 3     | 0.163000      | 0.267922        | 0.904762 | 0.844262 | 0.811024  | 0.880342 |
| 4     | 0.090200      | 0.335411        | 0.899749 | 0.838710 | 0.793893  | 0.888889 |
| 5     | 0.065200      | 0.462526        | 0.897243 | 0.835341 | 0.787879  | 0.888889 |
| 6     | 0.039200      | 0.423001        | 0.912281 | 0.859438 | 0.810606  | 0.914530 |
| 7     | 0.025300      | 0.452100        | 0.912281 | 0.859438 | 0.810606  | 0.914530 |
| 8     | 0.010400      | 0.525200        | 0.914787 | 0.855932 | 0.848739  | 0.863248 |
| 9     | 0.007100      | 0.513585        | 0.909774 | 0.850000 | 0.829268  | 0.871795 |
| 10    | 0.007200      | 0.537254        | 0.917293 | 0.864198 | 0.833333  | 0.897436 |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.


## Author


Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indonesian-roberta-base-indolem-sentiment-classifier-fold-0


model author: w11wo


task: text-classification


tags: pytorch, roberta, text-classification, id, dataset:indolem, arxiv:1907.11692, transformers, indonesian-roberta-base-indolem-sentiment-classifier-fold-0, license:mit, infinity_compatible


language: jv
tags:


- javanese-roberta-small
license: mit
datasets:
- wikipedia
widget:
- text: "Ing mangsa rendheng awakedhewe kudu pinter njaga <mask>."


## Javanese RoBERTa Small


Javanese RoBERTa Small is a masked language model based on the [RoBERTa model](https://arxiv.org/abs/1907.11692). It was trained on the latest (late December 2020) Javanese Wikipedia articles.


The model was originally HuggingFace's pretrained [English RoBERTa model](https://huggingface.co/roberta-base) and is later fine-tuned on the Javanese dataset. It achieved a perplexity of 33.30 on the validation dataset (20% of the articles). Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), and [fine-tuning tutorial notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb) written by [Pierre Guillou](https://huggingface.co/pierreguillou).


Hugging Face's [Transformers](https://huggingface.co/transformers) library was used to train the model -- utilizing the base RoBERTa model and their `Trainer` class. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                    | #params | Arch.    | Training/Validation data (text)     |
|--------------------------|---------|----------|-------------------------------------|
| `javanese-roberta-small` |  124M   | RoBERTa  | Javanese Wikipedia (319 MB of text) |


## Evaluation Results


The model was trained for 5 epochs and the following is the final result once the training ended.


| train loss | valid loss | perplexity | total time |
|------------|------------|------------|------------|
|    3.481   |    3.506   |   33.30    |   1:11:43  |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-roberta-small"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Meja lan kursine lagi <mask>.")
```


### Feature Extraction in PyTorch


```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "w11wo/javanese-roberta-small"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.


## Author


Javanese RoBERTa Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-roberta-small


model author: w11wo


task: fill-mask


tags: pytorch, tf, jax, roberta, fill-mask, jv, dataset:wikipedia, arxiv:1907.11692, transformers, javanese-roberta-small, license:mit, autonlp_compatible, infinity_compatible


language: id
tags:


- indonesian-roberta-base-indolem-sentiment-classifier-fold-1
license: mit
datasets:
- indolem
widget:
- text: "Pelayanan hotel ini sangat baik."


## Indonesian RoBERTa Base IndoLEM Sentiment Classifier


Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).


A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.


On **fold 1**, the model achieved an F1 of 84.43% on dev/validation and 86.59% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.


## Model


| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-1` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |


## Evaluation Results


The model was trained for 10 epochs and the best model was loaded at the end.


| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.563400      | 0.438638        | 0.771930 | 0.528497 | 0.671053  | 0.435897 |
| 2     | 0.313700      | 0.286461        | 0.857143 | 0.724638 | 0.833333  | 0.641026 |
| 3     | 0.179800      | 0.255090        | 0.892231 | 0.812227 | 0.830357  | 0.794872 |
| 4     | 0.092300      | 0.360222        | 0.897243 | 0.828452 | 0.811475  | 0.846154 |
| 5     | 0.054400      | 0.345958        | 0.904762 | 0.844262 | 0.811024  | 0.880342 |
| 6     | 0.026300      | 0.466914        | 0.902256 | 0.836820 | 0.819672  | 0.854701 |
| 7     | 0.018500      | 0.517265        | 0.904762 | 0.836207 | 0.843478  | 0.829060 |
| 8     | 0.009900      | 0.544317        | 0.902256 | 0.835443 | 0.825000  | 0.846154 |
| 9     | 0.008800      | 0.570152        | 0.904762 | 0.833333 | 0.855856  | 0.811966 |
| 10    | 0.007500      | 0.569749        | 0.902256 | 0.829694 | 0.848214  | 0.811966 |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.


## Author


Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indonesian-roberta-base-indolem-sentiment-classifier-fold-1


model author: w11wo


task: text-classification


tags: pytorch, roberta, text-classification, id, dataset:indolem, arxiv:1907.11692, transformers, indonesian-roberta-base-indolem-sentiment-classifier-fold-1, license:mit, infinity_compatible


language: id
tags:


- indonesian-roberta-base-indonli
license: mit
datasets:
- indonli
widget:
- text: "Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih."


## Indonesian RoBERTa Base IndoNLI


Indonesian RoBERTa Base IndoNLI is a natural language inference (NLI) model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`IndoNLI`](https://github.com/ir-nlp-csui/indonli)'s dataset consisting of Indonesian Wikipedia, news, and Web articles [1].


After training, the model achieved an evaluation/dev accuracy of 77.06%. On the benchmark `test_lay` subset, the model achieved an accuracy of 74.24% and on the benchmark `test_expert` subset, the model achieved an accuracy of 61.66%.


## Model


| Model                             | #params | Arch.        | Training/Validation data (text) |
| --------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indonli` | 124M    | RoBERTa Base | `IndoNLI`                       |


## Evaluation Results


The model was trained for 5 epochs, with a batch size of 16, a learning rate of 2e-5, a weight decay of 0.1, and a warmup ratio of 0.2, with linear annealing to 0. The best model was loaded at the end.


| Epoch | Training Loss | Validation Loss | Accuracy |
| ----- | ------------- | --------------- | -------- |
| 1     | 0.989200      | 0.691663        | 0.731452 |
| 2     | 0.673000      | 0.621913        | 0.766045 |
| 3     | 0.449900      | 0.662543        | 0.770596 |
| 4     | 0.293600      | 0.777059        | 0.768320 |
| 5     | 0.194200      | 0.948068        | 0.764224 |


## How to Use


### As NLI Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indonli"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and the `IndoNLI` dataset that may be carried over into the results of this model.


## References


[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). [IndoNLI: A Natural Language Inference Dataset for Indonesian](https://arxiv.org/abs/2110.14566). *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics.


## Author


Indonesian RoBERTa Base IndoNLI was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indonesian-roberta-base-indonli


model author: w11wo


task: text-classification


tags: pytorch, tf, roberta, text-classification, id, dataset:indonli, arxiv:1907.11692, arxiv:2110.14566, transformers, indonesian-roberta-base-indonli, license:mit, infinity_compatible


language: jv
tags:


- javanese-gpt2-small-imdb-classifier
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Film sing apik banget!"


## Javanese GPT-2 Small IMDB Classifier


Javanese GPT-2 Small IMDB Classifier is a movie-classification model based on the [GPT-2 model](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). It was trained on Javanese IMDB movie reviews.


The model was originally [`w11wo/javanese-gpt2-small-imdb`](https://huggingface.co/w11wo/javanese-gpt2-small-imdb) which is then fine-tuned on the [`w11wo/imdb-javanese`](https://huggingface.co/datasets/w11wo/imdb-javanese) dataset consisting of Javanese IMDB movie reviews. It achieved an accuracy of 76.70% on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                                 | #params  | Arch.           | Training/Validation data (text) |
|---------------------------------------|----------|-----------------|---------------------------------|
| `javanese-gpt2-small-imdb-classifier` |   124M   |   GPT-2 Small   | Javanese IMDB (47.5 MB of text) |


## Evaluation Results


The model was trained for 5 epochs and the following is the final result once the training ended.


| train loss | valid loss |  accuracy  | total time  |
|------------|------------|------------|-------------|
|    0.324   |    0.574   |   0.767    |   2:0:14    |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-gpt2-small-imdb-classifier"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Film sing apik banget!")
```


## Disclaimer


Do consider the biases which came from the IMDB review that may be carried over into the results of this model.


## Author


Javanese GPT-2 Small IMDB Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-gpt2-small-imdb-classifier


model author: w11wo


task: text-classification


tags: pytorch, tf, gpt2, text-classification, jv, dataset:w11wo/imdb-javanese, transformers, javanese-gpt2-small-imdb-classifier, license:mit


language: id
tags:


- indonesian-roberta-base-indolem-sentiment-classifier-fold-3
license: mit
datasets:
- indolem
widget:
- text: "Pelayanan hotel ini sangat baik."


## Indonesian RoBERTa Base IndoLEM Sentiment Classifier


Indonesian RoBERTa Base IndoLEM Sentiment Classifier is a sentiment-text-classification model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`indolem`](https://indolem.github.io/)'s [Sentiment Analysis](https://github.com/indolem/indolem/tree/main/sentiment) dataset consisting of Indonesian tweets and hotel reviews (Koto et al., 2020).


A 5-fold cross-validation experiment was performed, with splits provided by the original dataset authors. This model was trained on fold 0. You can find models trained on [fold 0](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-0), [fold 1](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-1), [fold 2](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-2), [fold 3](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3), and [fold 4](https://huggingface.co/w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-4), in their respective links.


On **fold 3**, the model achieved an F1 of 82.61% on dev/validation and 85.27% on test. On all **5 folds**, the models achieved an average F1 of 84.14% on dev/validation and 84.64% on test.


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.


## Model


| Model                                                         | #params | Arch.        | Training/Validation data (text) |
| ------------------------------------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indolem-sentiment-classifier-fold-3` | 124M    | RoBERTa Base | `IndoLEM`'s Sentiment Analysis  |


## Evaluation Results


The model was trained for 10 epochs and the best model was loaded at the end.


| Epoch | Training Loss | Validation Loss | Accuracy | F1       | Precision | Recall   |
| ----- | ------------- | --------------- | -------- | -------- | --------- | -------- |
| 1     | 0.566700      | 0.446229        | 0.786967 | 0.593301 | 0.673913  | 0.529915 |
| 2     | 0.317200      | 0.288581        | 0.877193 | 0.810811 | 0.739437  | 0.897436 |
| 3     | 0.171500      | 0.342168        | 0.894737 | 0.800000 | 0.903226  | 0.717949 |
| 4     | 0.103200      | 0.349750        | 0.892231 | 0.817021 | 0.813559  | 0.820513 |
| 5     | 0.059600      | 0.486529        | 0.899749 | 0.826087 | 0.840708  | 0.811966 |
| 6     | 0.031800      | 0.543524        | 0.882206 | 0.808163 | 0.773438  | 0.846154 |
| 7     | 0.016200      | 0.564764        | 0.899749 | 0.819820 | 0.866667  | 0.777778 |
| 8     | 0.008300      | 0.593180        | 0.897243 | 0.819383 | 0.845455  | 0.794872 |
| 9     | 0.010000      | 0.631985        | 0.889724 | 0.810345 | 0.817391  | 0.803419 |
| 10    | 0.005500      | 0.646718        | 0.892231 | 0.813853 | 0.824561  | 0.803419 |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indolem-sentiment-classifier-fold-3"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Pelayanan hotel ini sangat baik.")
```


## Disclaimer


Do consider the biases which come from both the pre-trained RoBERTa model and `IndoLEM`'s Sentiment Analysis dataset that may be carried over into the results of this model.


## Author


Indonesian RoBERTa Base IndoLEM Sentiment Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: indonesian-roberta-base-indolem-sentiment-classifier-fold-3


model author: w11wo


task: text-classification


tags: pytorch, roberta, text-classification, id, dataset:indolem, arxiv:1907.11692, transformers, indonesian-roberta-base-indolem-sentiment-classifier-fold-3, license:mit, infinity_compatible


language: jv
tags:


- javanese-roberta-small-imdb-classifier
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Aku bakal menehi rating film iki 1 bintang."


## Javanese RoBERTa Small IMDB Classifier


Javanese RoBERTa Small IMDB Classifier is a movie-classification model based on the [RoBERTa model](https://arxiv.org/abs/1907.11692). It was trained on Javanese IMDB movie reviews.


The model was originally [`w11wo/javanese-roberta-small-imdb`](https://huggingface.co/w11wo/javanese-roberta-small-imdb) which is then fine-tuned on the [`w11wo/imdb-javanese`](https://huggingface.co/datasets/w11wo/imdb-javanese) dataset consisting of Javanese IMDB movie reviews. It achieved an accuracy of 77.70% on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                                    | #params | Arch.            | Training/Validation data (text) |
|------------------------------------------|---------|------------------|---------------------------------|
| `javanese-roberta-small-imdb-classifier` |  124M   |  RoBERTa Small   | Javanese IMDB (47.5 MB of text) |


## Evaluation Results


The model was trained for 5 epochs and the following is the final result once the training ended.


| train loss | valid loss |  accuracy  | total time  |
|------------|------------|------------|-------------|
|    0.281   |    0.593   |   0.777    |   1:48:31   |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-roberta-small-imdb-classifier"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Film sing apik banget!")
```


## Disclaimer


Do consider the biases which came from the IMDB review that may be carried over into the results of this model.


## Author


Javanese RoBERTa Small IMDB Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-roberta-small-imdb-classifier


model author: w11wo


task: text-classification


tags: pytorch, tf, jax, roberta, text-classification, jv, dataset:w11wo/imdb-javanese, arxiv:1907.11692, transformers, javanese-roberta-small-imdb-classifier, license:mit, infinity_compatible


language: jv
tags:


- javanese-bert-small-imdb
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Fast and Furious iku film sing [MASK]."


## Javanese BERT Small IMDB


Javanese BERT Small IMDB is a masked language model based on the [BERT model](https://arxiv.org/abs/1810.04805). It was trained on Javanese IMDB movie reviews.


The model was originally the pretrained [Javanese BERT Small model](https://huggingface.co/w11wo/javanese-bert-small) and is later fine-tuned on the Javanese IMDB movie review dataset. It achieved a perplexity of 19.87 on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).


## Model


| Model                      | #params  | Arch.          | Training/Validation data (text) |
|----------------------------|----------|----------------|---------------------------------|
| `javanese-bert-small-imdb` |   110M   |   BERT Small   | Javanese IMDB (47.5 MB of text) |


## Evaluation Results


| train loss | valid loss | perplexity | total time  |
|------------|------------|------------|-------------|
|    3.070   |    2.989   |   19.87    |   3:12:33   |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-bert-small-imdb"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Aku mangan sate ing [MASK] bareng konco-konco")
```


### Feature Extraction in PyTorch


```python
from transformers import BertModel, BertTokenizerFast

pretrained_name = "w11wo/javanese-bert-small-imdb"
model = BertModel.from_pretrained(pretrained_name)
tokenizer = BertTokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


## Author


Javanese BERT Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-bert-small-imdb


model author: w11wo


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, jv, dataset:w11wo/imdb-javanese, arxiv:1810.04805, transformers, javanese-bert-small-imdb, license:mit, autonlp_compatible, infinity_compatible


language: jv
tags:


- javanese-distilbert-small-imdb
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Film favoritku yaiku Interstellar [MASK] Christopher Nolan."


## Javanese DistilBERT Small IMDB


Javanese DistilBERT Small IMDB is a masked language model based on the [DistilBERT model](https://arxiv.org/abs/1910.01108). It was trained on Javanese IMDB movie reviews.


The model was originally the pretrained [Javanese DistilBERT Small model](https://huggingface.co/w11wo/javanese-distilbert-small) and is later fine-tuned on the Javanese IMDB movie review dataset. It achieved a perplexity of 21.01 on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                            | #params  | Arch.                | Training/Validation data (text) |
|----------------------------------|----------|----------------------|---------------------------------|
| `javanese-distilbert-small-imdb` |   66M    |   DistilBERT Small   | Javanese IMDB (47.5 MB of text) |


## Evaluation Results


The model was trained for 5 epochs and the following is the final result once the training ended.


| train loss | valid loss | perplexity | total time  |
|------------|------------|------------|-------------|
|    3.126   |    3.039   |   21.01    |    5:6:4    |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-distilbert-small-imdb"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Aku mangan sate ing [MASK] bareng konco-konco")
```


### Feature Extraction in PyTorch


```python
from transformers import DistilBertModel, DistilBertTokenizerFast

pretrained_name = "w11wo/javanese-distilbert-small-imdb"
model = DistilBertModel.from_pretrained(pretrained_name)
tokenizer = DistilBertTokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do consider the biases which came from the IMDB review that may be carried over into the results of this model.


## Author


Javanese DistilBERT Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-distilbert-small-imdb


model author: w11wo


task: fill-mask


tags: pytorch, tf, distilbert, fill-mask, jv, dataset:w11wo/imdb-javanese, arxiv:1910.01108, transformers, javanese-distilbert-small-imdb, license:mit, autonlp_compatible, infinity_compatible


language: jv
tags:


- javanese-gpt2-small-imdb
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Train to Busan yaiku film sing digawe ing Korea Selatan"


## Javanese GPT-2 Small IMDB


Javanese GPT-2 Small IMDB is a causal language model based on the [GPT-2 model](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). It was trained on Javanese IMDB movie reviews.


The model was originally the pretrained [Javanese GPT-2 Small model](https://huggingface.co/w11wo/javanese-gpt2-small) and is later fine-tuned on the Javanese IMDB movie review dataset. It achieved a perplexity of 60.54 on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).


Hugging Face's `Trainer` class from the [Transformers]((https://huggingface.co/transformers)) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                      | #params  | Arch.           | Training/Validation data (text) |
|----------------------------|----------|-----------------|---------------------------------|
| `javanese-gpt2-small-imdb` |   124M   |   GPT-2 Small   | Javanese IMDB (47.5 MB of text) |


## Evaluation Results


| train loss | valid loss | perplexity | total time |
|------------|------------|------------|------------|
|    4.135   |    4.103   |   60.54    |   6:22:40  |


## How to Use (PyTorch)


### As Causal Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-gpt2-small-imdb"

nlp = pipeline(
    "text-generation",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Jenengku Budi, saka Indonesia")
```


### Feature Extraction in PyTorch


```python
from transformers import GPT2LMHeadModel, GPT2TokenizerFast

pretrained_name = "w11wo/javanese-gpt2-small-imdb"
model = GPT2LMHeadModel.from_pretrained(pretrained_name)
tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


## Author


Javanese GPT-2 Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-gpt2-small-imdb


model author: w11wo


task: text-generation


tags: pytorch, tf, jax, gpt2, text-generation, jv, dataset:w11wo/imdb-javanese, transformers, javanese-gpt2-small-imdb, license:mit


language: jv
tags:


- javanese-distilbert-small-imdb-classifier
license: mit
datasets:
- w11wo/imdb-javanese
widget:
- text: "Aku babar pisan ora nikmati film iki."


## Javanese DistilBERT Small IMDB Classifier


Javanese DistilBERT Small IMDB Classifier is a movie-classification model based on the [DistilBERT model](https://arxiv.org/abs/1910.01108). It was trained on Javanese IMDB movie reviews.


The model was originally [`w11wo/javanese-distilbert-small-imdb`](https://huggingface.co/w11wo/javanese-distilbert-small-imdb) which is then fine-tuned on the [`w11wo/imdb-javanese`](https://huggingface.co/datasets/w11wo/imdb-javanese) dataset consisting of Javanese IMDB movie reviews. It achieved an accuracy of 76.04% on the validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb) written by [Sylvain Gugger](https://github.com/sgugger).


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                                       | #params | Arch.            | Training/Validation data (text) |
|---------------------------------------------|---------|------------------|---------------------------------|
| `javanese-distilbert-small-imdb-classifier` |   66M   | DistilBERT Small | Javanese IMDB (47.5 MB of text) |


## Evaluation Results


The model was trained for 5 epochs and the following is the final result once the training ended.


| train loss | valid loss |  accuracy  | total time |
|------------|------------|------------|------------|
|    0.131   |    1.113   |   0.760    |   1:26:4   |


## How to Use


### As Text Classifier


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-distilbert-small-imdb-classifier"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Film sing apik banget!")
```


## Disclaimer


Do consider the biases which came from the IMDB review that may be carried over into the results of this model.


## Author


Javanese DistilBERT Small IMDB Classifier was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-distilbert-small-imdb-classifier


model author: w11wo


task: text-classification


tags: pytorch, tf, distilbert, text-classification, jv, dataset:w11wo/imdb-javanese, arxiv:1910.01108, transformers, javanese-distilbert-small-imdb-classifier, license:mit, infinity_compatible


language: jv
tags:


- javanese-bert-small
license: mit
datasets:
- wikipedia
widget:
- text: "Aku mangan sate ing [MASK] bareng konco-konco"


## Javanese BERT Small


Javanese BERT Small is a masked language model based on the [BERT model](https://arxiv.org/abs/1810.04805). It was trained on the latest (late December 2020) Javanese Wikipedia articles.


The model was originally HuggingFace's pretrained [English BERT model](https://huggingface.co/bert-base-uncased) and is later fine-tuned on the Javanese dataset. It achieved a perplexity of 22.00 on the validation dataset (20% of the articles). Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), and [fine-tuning tutorial notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb) written by [Pierre Guillou](https://huggingface.co/pierreguillou).


Hugging Face's [Transformers](https://huggingface.co/transformers) library was used to train the model -- utilizing the base BERT model and their `Trainer` class. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                 | #params  | Arch.          | Training/Validation data (text)     |
|-----------------------|----------|----------------|-------------------------------------|
| `javanese-bert-small` |   110M   |   BERT Small   | Javanese Wikipedia (319 MB of text) |


## Evaluation Results


| train loss | valid loss | perplexity | total time |
|------------|------------|------------|------------|
|    3.116   |    3.091   |   22.00    |   2:7:42   |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/javanese-bert-small"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Aku mangan sate ing [MASK] bareng konco-konco")
```


### Feature Extraction in PyTorch


```python
from transformers import BertModel, BertTokenizerFast

pretrained_name = "w11wo/javanese-bert-small"
model = BertModel.from_pretrained(pretrained_name)
tokenizer = BertTokenizerFast.from_pretrained(pretrained_name)

prompt = "Indonesia minangka negara gedhe."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do remember that although the dataset originated from Wikipedia, the model may not always generate factual texts. Additionally, the biases which came from the Wikipedia articles may be carried over into the results of this model.


## Author


Javanese BERT Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: javanese-bert-small


model author: w11wo


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, jv, dataset:wikipedia, arxiv:1810.04805, transformers, javanese-bert-small, license:mit, autonlp_compatible, infinity_compatible


language: lo
tags:


- lao-roberta-base
license: mit
datasets:
- oscar-corpus/OSCAR-2109


## Lao RoBERTa Base


Lao RoBERTa Base is a masked language model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. It was trained on the [OSCAR-2109](https://huggingface.co/datasets/oscar-corpus/OSCAR-2109) dataset, specifically the `deduplicated_lo` subset. The model was trained from scratch and achieved an evaluation loss of 1.4556 and an evaluation perplexity of 4.287.


This model was trained using HuggingFace's PyTorch framework and the training script found [here](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py). All training was done on a TPUv3-8, provided by the [TPU Research Cloud](https://sites.research.google/trc/about/) program. You can view the detailed training results in the [Training metrics](https://huggingface.co/w11wo/lao-roberta-base/tensorboard) tab, logged via Tensorboard.


## Model


| Model              | #params | Arch.   | Training/Validation data (text)      |
| ------------------ | ------- | ------- | ------------------------------------ |
| `lao-roberta-base` | 124M    | RoBERTa | OSCAR-2109 `deduplicated_lo` Dataset |


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0002
- train_batch_size: 128
- eval_batch_size: 128
- seed: 42
- distributed_type: tpu
- num_devices: 8
- total_train_batch_size: 1024
- total_eval_batch_size: 1024
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30.0


### Training results


| Training Loss | Epoch | Step | Validation Loss |
| :-----------: | :---: | :--: | :-------------: |
|    No log     |  1.0  | 216  |     5.8586      |
|    No log     |  2.0  | 432  |     5.5095      |
|     6.688     |  3.0  | 648  |     5.3976      |
|     6.688     |  4.0  | 864  |     5.3562      |
|    5.3629     |  5.0  | 1080 |     5.2912      |
|    5.3629     |  6.0  | 1296 |     5.2385      |
|     5.22      |  7.0  | 1512 |     5.1955      |
|     5.22      |  8.0  | 1728 |     5.1785      |
|     5.22      |  9.0  | 1944 |     5.1327      |
|    5.1248     | 10.0  | 2160 |     5.1243      |
|    5.1248     | 11.0  | 2376 |     5.0889      |
|    5.0591     | 12.0  | 2592 |     5.0732      |
|    5.0591     | 13.0  | 2808 |     5.0417      |
|    5.0094     | 14.0  | 3024 |     5.0388      |
|    5.0094     | 15.0  | 3240 |     4.9299      |
|    5.0094     | 16.0  | 3456 |     4.2991      |
|    4.7527     | 17.0  | 3672 |     3.6541      |
|    4.7527     | 18.0  | 3888 |     2.7826      |
|    3.4431     | 19.0  | 4104 |     2.2796      |
|    3.4431     | 20.0  | 4320 |     2.0213      |
|    2.2803     | 21.0  | 4536 |     1.8809      |
|    2.2803     | 22.0  | 4752 |     1.7615      |
|    2.2803     | 23.0  | 4968 |     1.6925      |
|    1.8601     | 24.0  | 5184 |     1.6205      |
|    1.8601     | 25.0  | 5400 |     1.5751      |
|    1.6697     | 26.0  | 5616 |     1.5391      |
|    1.6697     | 27.0  | 5832 |     1.5200      |
|    1.5655     | 28.0  | 6048 |     1.4866      |
|    1.5655     | 29.0  | 6264 |     1.4656      |
|    1.5655     | 30.0  | 6480 |     1.4627      |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/lao-roberta-base"
prompt = "REPLACE WITH MASKED PROMPT"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask(prompt)
```


### Feature Extraction in PyTorch


```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "w11wo/lao-roberta-base"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "ສະບາຍດີຊາວໂລກ."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.


## Author


Lao RoBERTa Base was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google's TPU-RC.


## Framework versions


- Transformers 4.13.0.dev0
- Pytorch 1.9.0+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: lao-roberta-base


model author: w11wo


task: fill-mask


tags: pytorch, tensorboard, roberta, fill-mask, lo, dataset:oscar-corpus/OSCAR-2109, arxiv:1907.11692, transformers, lao-roberta-base, license:mit, autonlp_compatible, infinity_compatible


language: ms
tags:


- malaysian-distilbert-small
license: mit
datasets:
- oscar
widget:
- text: "Hari ini adalah hari yang [MASK]!"


## Malaysian DistilBERT Small


Malaysian DistilBERT Small is a masked language model based on the [DistilBERT model](https://arxiv.org/abs/1910.01108). It was trained on the [OSCAR](https://huggingface.co/datasets/oscar) dataset, specifically the `unshuffled_original_ms` subset.


The model was originally HuggingFace's pretrained [English DistilBERT model](https://huggingface.co/distilbert-base-uncased) and is later fine-tuned on the Malaysian dataset. It achieved a perplexity of 10.33 on the validation dataset (20% of the dataset). Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), and [fine-tuning tutorial notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb) written by [Pierre Guillou](https://huggingface.co/pierreguillou).


Hugging Face's [Transformers](https://huggingface.co/transformers) library was used to train the model -- utilizing the base DistilBERT model and their `Trainer` class. PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow nonetheless.


## Model


| Model                        | #params | Arch.            | Training/Validation data (text)        |
|------------------------------|---------|------------------|----------------------------------------|
| `malaysian-distilbert-small` |   66M   | DistilBERT Small | OSCAR `unshuffled_original_ms` Dataset |


## Evaluation Results


The model was trained for 1 epoch and the following is the final result once the training ended.


| train loss | valid loss | perplexity | total time |
|------------|------------|------------|------------|
|    2.476   |    2.336   |   10.33    |   0:40:05  |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "w11wo/malaysian-distilbert-small"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Henry adalah seorang lelaki yang tinggal di [MASK].")
```


### Feature Extraction in PyTorch


```python
from transformers import DistilBertModel, DistilBertTokenizerFast

pretrained_name = "w11wo/malaysian-distilbert-small"
model = DistilBertModel.from_pretrained(pretrained_name)
tokenizer = DistilBertTokenizerFast.from_pretrained(pretrained_name)

prompt = "Bolehkah anda [MASK] Bahasa Melayu?"
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do consider the biases which came from the OSCAR dataset that may be carried over into the results of this model.


## Author


Malaysian DistilBERT Small was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: malaysian-distilbert-small


model author: w11wo


task: fill-mask


tags: pytorch, tf, distilbert, fill-mask, ms, dataset:oscar, arxiv:1910.01108, transformers, malaysian-distilbert-small, license:mit, autonlp_compatible, infinity_compatible


language: "fi"
thumbnail:
tags:


- automatic-speech-recognition
- Attention
- pytorch
- speechbrain


metrics:


- wer
- cer


# CRDNN with Attention trained on LP


This is a an initial model, partly wrong configuration, just to show an initial example.


model name: lp-initial-aed-short


model author: Gastron


task: automatic-speech-recognition


tags: fi, speechbrain, automatic-speech-recognition, Attention, pytorch


language: "en"
thumbnail:
tags:


- ASR
- CTC
- Attention
- pytorch
license: "apache-2.0"
datasets:
- librispeech
metrics:
- wer
- cer


# CRDNN with CTC/Attention and RNNLM trained on LibriSpeech


This repository provides all the necessary tools to perform automatic speech
recognition from an end-to-end system pretrained on LibriSpeech (EN) within
SpeechBrain. For a better experience we encourage you to learn more about
[SpeechBrain](https://speechbrain.github.io). The given ASR model performance are:


| Release | hyperparams file | Test WER | Model link | GPUs |
|:-------------:|:---------------------------:| -----:| -----:| --------:|
| 20-05-22 | BPE_1000.yaml | 3.08 | Not Available | 1xV100 32GB |
| 20-05-22 | BPE_5000.yaml | 2.89 | Not Available | 1xV100 32GB |


## Pipeline description


This ASR system is composed with 3 different but linked blocks:


1. Tokenizer (unigram) that transforms words into subword units and trained with
the train transcriptions of LibriSpeech.
1. Neural language model (RNNLM) trained on the full 10M words dataset.
1. Acoustic model (CRDNN + CTC/Attention). The CRDNN architecture is made of
N blocks of convolutional neural networks with normalisation and pooling on the
frequency domain. Then, a bidirectional LSTM is connected to a final DNN to obtain
the final acoustic representation that is given to the CTC and attention decoders.


## Intended uses & limitations


This model has been primilarly developed to be run within SpeechBrain as a pretrained ASR model
for the english language. Thanks to the flexibility of SpeechBrain, any of the 3 blocks
detailed above can be extracted and connected to you custom pipeline as long as SpeechBrain is
installed.


## Install SpeechBrain


First of all, please install SpeechBrain with the following command:


```
pip install \\we hide ! SpeechBrain is still private :p
```


Also, for this model, you need SentencePiece. Install with


```
pip install sentencepiece
```


Please notice that we encourage you to read our tutorials and learn more about
[SpeechBrain](https://speechbrain.github.io).


### Transcribing your own audio files


```python
from speechbrain.pretrained import EncoderDecoderASR

asr_model = EncoderDecoderASR.from_hparams(source="Gastron/asr-crdnn-librispeech")
asr_model.transcribe_file("path_to_your_file.wav")

```


### Obtaining encoded features


The SpeechBrain EncoderDecoderASR() class also provides an easy way to encode
the speech signal without running the decoding phase by calling
`EncoderDecoderASR.encode_batch()`


#### Referencing SpeechBrain


```
@misc{SB2021,
    author = {Ravanelli, Mirco and Parcollet, Titouan and Rouhe, Aku and Plantinga, Peter and Rastorgueva, Elena and Lugosch, Loren and Dawalatabad, Nauman and Ju-Chieh, Chou and Heba, Abdel and Grondin, Francois and Aris, William and Liao, Chien-Feng and Cornell, Samuele and Yeh, Sung-Lin and Na, Hwidong and Gao, Yan and Fu, Szu-Wei and Subakan, Cem and De Mori, Renato and Bengio, Yoshua },
    title = {SpeechBrain},
    year = {2021},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/speechbrain/speechbrain}},
  }
```


model name: asr-crdnn-librispeech


model author: Gastron


tags: en, dataset:librispeech, ASR, CTC, Attention, pytorch, license:apache-2.0


language:


- en
- es
- eu
datasets:
- squad
widget:
- text: "When was Florence Nightingale born?"
context: "Florence Nightingale, known for being the founder of modern nursing, was born in Florence, Italy, in 1820."
example_title: "English"
- text: "¿Por qué provincias pasa el Tajo?"
context: "El Tajo es el río más largo de la península ibérica, a la que atraviesa en su parte central, siguiendo un rumbo este-oeste, con una leve inclinación hacia el suroeste, que se acentúa cuando llega a Portugal, donde recibe el nombre de Tejo.


Nace en los montes Universales, en la sierra de Albarracín, sobre la rama occidental del sistema Ibérico y, después de recorrer 1007 km, llega al océano Atlántico en la ciudad de Lisboa. En su desembocadura forma el estuario del mar de la Paja, en el que vierte un caudal medio de 456 m³/s. En sus primeros 816 km atraviesa España, donde discurre por cuatro comunidades autónomas (Aragón, Castilla-La Mancha, Madrid y Extremadura) y un total de seis provincias (Teruel, Guadalajara, Cuenca, Madrid, Toledo y Cáceres)."
example_title: "Español"


- text: "Zer beste izenak ditu Tartalo?"
context: "Tartalo euskal mitologiako izaki begibakar artzain erraldoia da. Tartalo izena zenbait euskal hizkeratan herskari-bustidurarekin ahoskatu ohi denez, horrelaxe ere idazten da batzuetan: Ttarttalo. Euskal Herriko zenbait tokitan, Torto edo Anxo ere esaten diote."
example_title: "Euskara"


# ixambert-base-cased finetuned for QA


This is a basic implementation of the multilingual model ["ixambert-base-cased"](https://huggingface.co/ixa-ehu/ixambert-base-cased), fine-tuned on SQuAD v1.1, that is able to answer basic factual questions in English, Spanish and Basque. This model reaches a F1 score of 89.1 on the SQuAD 1.1 dev set.


## Overview


* **Language model:** ixambert-base-cased
* **Languages:** English, Spanish and Basque
* **Downstream task:** Extractive QA
* **Training data:** SQuAD v1.1
* **Eval data:** SQuAD v1.1
* **Infrastructure:** 1x GeForce RTX 2080


## Outputs


The model outputs the answer to the question, the start and end positions of the answer in the original context, and a score for the probability for that span of text to be the correct answer. For example:


```python
{'score': 0.9667195081710815, 'start': 101, 'end': 105, 'answer': '1820'}
```


## How to use


```python
from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline

model_name = "MarcBrun/ixambert-finetuned-squad"

# To get predictions
context = "Florence Nightingale, known for being the founder of modern nursing, was born in Florence, Italy, in 1820"
question = "When was Florence Nightingale born?"
qa = pipeline("question-answering", model=model_name, tokenizer=model_name)
pred = qa(question=question,context=context)

# To load the model and tokenizer
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
```


## Hyperparameters


```
batch_size = 8
n_epochs = 3
learning_rate = 2e-5
optimizer = AdamW
lr_schedule = linear
max_seq_len = 384
doc_stride = 128
```


model name: ixambert-finetuned-squad


model author: MarcBrun


task: question-answering


tags: pytorch, bert, question-answering, en, es, eu, dataset:squad, transformers, autonlp_compatible, infinity_compatible


language: es
tags:


- zero-shot-classification
- nli
- pytorch
datasets:
- xnli
license: mit
pipeline_tag: zero-shot-classification
widget:
- text: "El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo"
candidate_labels: "cultura, sociedad, economia, salud, deportes"


# bert-base-spanish-wwm-cased-xnli


**UPDATE, 15.10.2021: Check out our new zero-shot classifiers, much more lightweight and even outperforming this one: [zero-shot SELECTRA small](https://huggingface.co/Recognai/zeroshot_selectra_small) and [zero-shot SELECTRA medium](https://huggingface.co/Recognai/zeroshot_selectra_medium).**


## Model description


This model is a fine-tuned version of the [spanish BERT model](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased) with the Spanish portion of the XNLI dataset. You can have a look at the [training script](https://huggingface.co/Recognai/bert-base-spanish-wwm-cased-xnli/blob/main/zeroshot_training_script.py) for details of the training.


### How to use


You can use this model with Hugging Face's [zero-shot-classification pipeline](https://discuss.huggingface.co/t/new-pipeline-for-zero-shot-text-classification/681):


```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification",
                       model="Recognai/bert-base-spanish-wwm-cased-xnli")

classifier(
    "El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo",
    candidate_labels=["cultura", "sociedad", "economia", "salud", "deportes"],
    hypothesis_template="Este ejemplo es {}."
)
"""output
{'sequence': 'El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo',
 'labels': ['cultura', 'sociedad', 'economia', 'salud', 'deportes'],
 'scores': [0.38897448778152466,
  0.22997373342514038,
  0.1658431738615036,
  0.1205764189362526,
  0.09463217109441757]}
"""
```


## Eval results


Accuracy for the test set:


|                             | XNLI-es |
|-----------------------------|---------|
|bert-base-spanish-wwm-cased-xnli | 79.9% |


model name: bert-base-spanish-wwm-cased-xnli


model author: Recognai


task: zero-shot-classification


tags: pytorch, jax, bert, text-classification, es, dataset:xnli, transformers, zero-shot-classification, nli, license:mit, infinity_compatible


- zero-shot-classification
- nli
- pytorch
datasets:
- xnli
pipeline_tag: zero-shot-classification
license: apache-2
widget:
- text: "El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo"
candidate_labels: "cultura, sociedad, economia, salud, deportes"


# Zero-shot SELECTRA: A zero-shot classifier based on SELECTRA


*Zero-shot SELECTRA* is a [SELECTRA model](https://huggingface.co/Recognai/selectra_small) fine-tuned on the Spanish portion of the [XNLI dataset](https://huggingface.co/datasets/xnli). You can use it with Hugging Face's [Zero-shot pipeline](https://huggingface.co/transformers/master/main_classes/pipelines.html#transformers.ZeroShotClassificationPipeline) to make [zero-shot classifications](https://joeddav.github.io/blog/2020/05/29/ZSL.html).


In comparison to our previous zero-shot classifier [based on BETO](https://huggingface.co/Recognai/bert-base-spanish-wwm-cased-xnli), zero-shot SELECTRA is **much more lightweight**. As shown in the *Metrics* section, the *small* version (5 times fewer parameters) performs slightly worse, while the *medium* version (3 times fewer parameters) **outperforms** the BETO based zero-shot classifier.


## Usage


```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification",
                       model="Recognai/zeroshot_selectra_medium")

classifier(
    "El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo",
    candidate_labels=["cultura", "sociedad", "economia", "salud", "deportes"],
    hypothesis_template="Este ejemplo es {}."
)
"""Output
{'sequence': 'El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo',
 'labels': ['sociedad', 'cultura', 'salud', 'economia', 'deportes'],
 'scores': [0.3711881935596466,
  0.25650349259376526,
  0.17355826497077942,
  0.1641489565372467,
  0.03460107371211052]}
"""
```


The `hypothesis_template` parameter is important and should be in Spanish. **In the widget on the right, this parameter is set to its default value: "This example is {}.", so different results are expected.**


## Metrics


| Model | Params | XNLI (acc) | \*MLSUM (acc) |
| --- | --- | --- | --- |
| [zs BETO](https://huggingface.co/Recognai/bert-base-spanish-wwm-cased-xnli) | 110M | 0.799 | 0.530 |
| [zs SELECTRA medium](https://huggingface.co/Recognai/zeroshot_selectra_medium) | 41M | **0.807** | **0.589** |
| zs SELECTRA small | **22M** | 0.795 | 0.446 |


\*evaluated with zero-shot learning (ZSL)


- **XNLI**: The stated accuracy refers to the test portion of the [XNLI dataset](https://huggingface.co/datasets/xnli), after finetuning the model on the training portion.
- **MLSUM**: For this accuracy we take the test set of the [MLSUM dataset](https://huggingface.co/datasets/mlsum) and classify the summaries of 5 selected labels. For details, check out our [evaluation notebook](https://github.com/recognai/selectra/blob/main/zero-shot_classifier/evaluation.ipynb)


## Training


Check out our [training notebook](https://github.com/recognai/selectra/blob/main/zero-shot_classifier/training.ipynb) for all the details.


## Authors


- David Fidalgo ([GitHub](https://github.com/dcfidalgo))
- Daniel Vila ([GitHub](https://github.com/dvsrepo))
- Francisco Aranda ([GitHub](https://github.com/frascuchon))
- Javier Lopez ([GitHub](https://github.com/javispp))


model name: zeroshot_selectra_small


model author: Recognai


task: zero-shot-classification


tags: pytorch, electra, text-classification, es, dataset:xnli, transformers, zero-shot-classification, nli, license:apache-2, infinity_compatible


language:


- es
thumbnail: "url to a thumbnail used in social sharing"
license: apache-2.0
datasets:
- oscar


# SELECTRA: A Spanish ELECTRA


SELECTRA is a Spanish pre-trained language model based on [ELECTRA](https://github.com/google-research/electra).
We release a `small` and `medium` version with the following configuration:


| Model | Layers | Embedding/Hidden Size | Params | Vocab Size | Max Sequence Length | Cased |
| --- | --- | --- | --- | ---  | --- | --- |
| **SELECTRA small** | **12** | **256** | **22M** | **50k** | **512** | **True** |
| [SELECTRA medium](https://huggingface.co/Recognai/selectra_medium) | 12 | 384 | 41M | 50k | 512 | True |


**SELECTRA small (medium) is about 5 (3) times smaller than BETO but achieves comparable results** (see Metrics section below).


## Usage


From the original [ELECTRA model card](https://huggingface.co/google/electra-small-discriminator): "ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN."
The discriminator should therefore activate the logit corresponding to the fake input token, as the following example demonstrates:


```python
from transformers import ElectraForPreTraining, ElectraTokenizerFast

discriminator = ElectraForPreTraining.from_pretrained("Recognai/selectra_small")
tokenizer = ElectraTokenizerFast.from_pretrained("Recognai/selectra_small")

sentence_with_fake_token = "Estamos desayunando pan rosa con tomate y aceite de oliva."

inputs = tokenizer.encode(sentence_with_fake_token, return_tensors="pt")
logits = discriminator(inputs).logits.tolist()[0]

print("\t".join(tokenizer.tokenize(sentence_with_fake_token)))
print("\t".join(map(lambda x: str(x)[:4], logits[1:-1])))
"""Output:
Estamos desayun ##ando  pan     rosa    con     tomate  y       aceite  de      oliva   .
-3.1    -3.6    -6.9    -3.0    0.19    -4.5    -3.3    -5.1    -5.7    -7.7    -4.4    -4.2
"""
```


However, you probably want to use this model to fine-tune it on a downstream task.
We provide models fine-tuned on the [XNLI dataset](https://huggingface.co/datasets/xnli), which can be used together with the zero-shot classification pipeline:


- [Zero-shot SELECTRA small](https://huggingface.co/Recognai/zeroshot_selectra_small)
- [Zero-shot SELECTRA medium](https://huggingface.co/Recognai/zeroshot_selectra_medium)


## Metrics


We fine-tune our models on 3 different down-stream tasks:


- [XNLI](https://huggingface.co/datasets/xnli)
- [PAWS-X](https://huggingface.co/datasets/paws-x)
- [CoNLL2002 - NER](https://huggingface.co/datasets/conll2002)


For each task, we conduct 5 trials and state the mean and standard deviation of the metrics in the table below.
To compare our results to other Spanish language models, we provide the same metrics taken from the [evaluation table](https://github.com/PlanTL-SANIDAD/lm-spanish#evaluation-) of the [Spanish Language Model](https://github.com/PlanTL-SANIDAD/lm-spanish) repo.


| Model | CoNLL2002 - NER (f1) | PAWS-X (acc) | XNLI (acc) | Params |
| --- | --- | --- | --- | --- |
| SELECTRA small | 0.865 +- 0.004 | 0.896 +- 0.002 | 0.784 +- 0.002 | 22M |
| SELECTRA medium | 0.873 +- 0.003 | 0.896 +- 0.002 | 0.804 +- 0.002 | 41M |
|  |  |  |  |  |
| [mBERT](https://huggingface.co/bert-base-multilingual-cased) | 0.8691 | 0.8955 | 0.7876 | 178M |
| [BETO](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased) | 0.8759 | 0.9000 | 0.8130 | 110M |
| [RoBERTa-b](https://huggingface.co/BSC-TeMU/roberta-base-bne) | 0.8851 | 0.9000 | 0.8016 | 125M |
| [RoBERTa-l](https://huggingface.co/BSC-TeMU/roberta-large-bne) | 0.8772 | 0.9060 | 0.7958 | 355M |
| [Bertin](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1-512) | 0.8835 | 0.8990 | 0.7890 | 125M |
| [ELECTRICIDAD](https://huggingface.co/mrm8488/electricidad-base-discriminator) | 0.7954 | 0.9025 | 0.7878 | 109M |


Some details of our fine-tuning runs:


- epochs: 5
- batch-size: 32
- learning rate: 1e-4
- warmup proportion: 0.1
- linear learning rate decay
- layerwise learning rate decay


For all the details, check out our [selectra repo](https://github.com/recognai/selectra).


## Training


We pre-trained our SELECTRA models on the Spanish portion of the [Oscar](https://huggingface.co/datasets/oscar) dataset, which is about 150GB in size.
Each model version is trained for 300k steps, with a warm restart of the learning rate after the first 150k steps.
Some details of the training:


- steps: 300k
- batch-size: 128
- learning rate: 5e-4
- warmup steps: 10k
- linear learning rate decay
- TPU cores: 8 (v2-8)


For all details, check out our [selectra repo](https://github.com/recognai/selectra).


**Note:** Due to a misconfiguration in the pre-training scripts the embeddings of the vocabulary containing an accent were not optimized. If you fine-tune this model on a down-stream task, you might consider using a tokenizer that does not strip the accents:


```python
tokenizer = ElectraTokenizerFast.from_pretrained("Recognai/selectra_small", strip_accents=False)
```


## Motivation


Despite the abundance of excellent Spanish language models (BETO, BSC-BNE, Bertin, ELECTRICIDAD, etc.), we felt there was still a lack of distilled or compact Spanish language models and a lack of comparing those to their bigger siblings.


## Acknowledgment


This research was supported by the Google TPU Research Cloud (TRC) program.


## Authors


- David Fidalgo ([GitHub](https://github.com/dcfidalgo))
- Javier Lopez ([GitHub](https://github.com/javispp))
- Daniel Vila ([GitHub](https://github.com/dvsrepo))
- Francisco Aranda ([GitHub](https://github.com/frascuchon))


model name: selectra_small


model author: Recognai


tags: pytorch, electra, pretraining, es, dataset:oscar, transformers, license:apache-2.0, infinity_compatible


language: es
license: apache-2.0
datasets:


- wikipedia
widget:
- text: "Mi nombre es Juan y vivo en [MASK]."


# DistilBERT base multilingual model Spanish subset (cased)


This model is the Spanish extract of `distilbert-base-multilingual-cased` (https://huggingface.co/distilbert-base-multilingual-cased), a distilled version of the [BERT base multilingual model](bert-base-multilingual-cased). This model is cased: it does make a difference between english and English.


It uses the extraction method proposed by Geotrend described in https://github.com/Geotrend-research/smaller-transformers.


The resulting model has the same architecture as DistilmBERT: 6 layers, 768 dimension and 12 heads, with a total of **63M parameters** (compared to 134M parameters for DistilmBERT).


The goal of this model is to reduce even further the size of the `distilbert-base-multilingual` multilingual model by selecting only most frequent tokens for Spanish, reducing the size of the embedding layer. For more details visit the paper from the Geotrend team: Load What You Need: Smaller Versions of Multilingual BERT.


model name: distilbert-base-es-multilingual-cased


task: fill-mask


tags: pytorch, distilbert, fill-mask, es, dataset:wikipedia, transformers, license:apache-2.0, autonlp_compatible, infinity_compatible


language: es
tags:


- zero-shot-classification
- nli
- pytorch
datasets:
- xnli
pipeline_tag: zero-shot-classification
license: apache-2
widget:
- text: "El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo"
candidate_labels: "cultura, sociedad, economia, salud, deportes"


# Zero-shot SELECTRA: A zero-shot classifier based on SELECTRA


*Zero-shot SELECTRA* is a [SELECTRA model](https://huggingface.co/Recognai/selectra_small) fine-tuned on the Spanish portion of the [XNLI dataset](https://huggingface.co/datasets/xnli). You can use it with Hugging Face's [Zero-shot pipeline](https://huggingface.co/transformers/master/main_classes/pipelines.html#transformers.ZeroShotClassificationPipeline) to make [zero-shot classifications](https://joeddav.github.io/blog/2020/05/29/ZSL.html).


In comparison to our previous zero-shot classifier [based on BETO](https://huggingface.co/Recognai/bert-base-spanish-wwm-cased-xnli), zero-shot SELECTRA is **much more lightweight**. As shown in the *Metrics* section, the *small* version (5 times fewer parameters) performs slightly worse, while the *medium* version (3 times fewer parameters) **outperforms** the BETO based zero-shot classifier.


## Usage


```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification",
                       model="Recognai/zeroshot_selectra_medium")

classifier(
    "El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo",
    candidate_labels=["cultura", "sociedad", "economia", "salud", "deportes"],
    hypothesis_template="Este ejemplo es {}."
)
"""Output
{'sequence': 'El autor se perfila, a los 50 años de su muerte, como uno de los grandes de su siglo',
 'labels': ['sociedad', 'cultura', 'economia', 'salud', 'deportes'],
 'scores': [0.6450043320655823,
  0.16710571944713593,
  0.08507631719112396,
  0.0759836807847023,
  0.026829993352293968]}
"""
```


The `hypothesis_template` parameter is important and should be in Spanish. **In the widget on the right, this parameter is set to its default value: "This example is {}.", so different results are expected.**


## Demo and tutorial


If you want to see this model in action, we have created a basic tutorial using [Rubrix](https://www.rubrix.ml/), a free and open-source tool to *explore, annotate, and monitor data for NLP*.


The tutorial shows you how to evaluate this classifier for news categorization in Spanish, and how it could be used to build a training set for training a supervised classifier (which might be useful if you want obtain more precise results or improve the model over time).


You can [find the tutorial here](https://rubrix.readthedocs.io/en/master/tutorials/zeroshot_data_annotation.html).


See the video below showing the predictions within the annotation process (see that the predictions are almost correct for every example).


<video width="100%" controls><source src="https://github.com/recognai/rubrix-materials/raw/main/tutorials/videos/zeroshot_selectra_news_data_annotation.mp4" type="video/mp4"></video>


## Metrics


| Model | Params | XNLI (acc) | \*MLSUM (acc) |
| --- | --- | --- | --- |
| [zs BETO](https://huggingface.co/Recognai/bert-base-spanish-wwm-cased-xnli) | 110M | 0.799 | 0.530 |
| zs SELECTRA medium | 41M | **0.807** | **0.589** |
| [zs SELECTRA small](https://huggingface.co/Recognai/zeroshot_selectra_small) | **22M** | 0.795 | 0.446 |


\*evaluated with zero-shot learning (ZSL)


- **XNLI**: The stated accuracy refers to the test portion of the [XNLI dataset](https://huggingface.co/datasets/xnli), after finetuning the model on the training portion.
- **MLSUM**: For this accuracy we take the test set of the [MLSUM dataset](https://huggingface.co/datasets/mlsum) and classify the summaries of 5 selected labels. For details, check out our [evaluation notebook](https://github.com/recognai/selectra/blob/main/zero-shot_classifier/evaluation.ipynb)


## Training


Check out our [training notebook](https://github.com/recognai/selectra/blob/main/zero-shot_classifier/training.ipynb) for all the details.


## Authors


- David Fidalgo ([GitHub](https://github.com/dcfidalgo))
- Daniel Vila ([GitHub](https://github.com/dvsrepo))
- Francisco Aranda ([GitHub](https://github.com/frascuchon))
- Javier Lopez ([GitHub](https://github.com/javispp))


model name: zeroshot_selectra_medium


model author: Recognai


task: zero-shot-classification


tags: pytorch, electra, text-classification, es, dataset:xnli, transformers, zero-shot-classification, nli, license:apache-2, infinity_compatible


language:


- es
thumbnail: "url to a thumbnail used in social sharing"
license: apache-2.0
datasets:
- oscar


# SELECTRA: A Spanish ELECTRA


SELECTRA is a Spanish pre-trained language model based on [ELECTRA](https://github.com/google-research/electra).
We release a `small` and `medium` version with the following configuration:


| Model | Layers | Embedding/Hidden Size | Params | Vocab Size | Max Sequence Length | Cased |
| --- | --- | --- | --- | ---  | --- | --- |
| [SELECTRA small](https://huggingface.co/Recognai/selectra_small) | 12 | 256 | 22M | 50k | 512 | True |
| **SELECTRA medium** | **12** | **384** | **41M** | **50k** | **512** | **True** |


**SELECTRA small (medium) is about 5 (3) times smaller than BETO but achieves comparable results** (see Metrics section below).


## Usage


From the original [ELECTRA model card](https://huggingface.co/google/electra-small-discriminator): "ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN."
The discriminator should therefore activate the logit corresponding to the fake input token, as the following example demonstrates:


```python
from transformers import ElectraForPreTraining, ElectraTokenizerFast

discriminator = ElectraForPreTraining.from_pretrained("Recognai/selectra_small")
tokenizer = ElectraTokenizerFast.from_pretrained("Recognai/selectra_small")

sentence_with_fake_token = "Estamos desayunando pan rosa con tomate y aceite de oliva."

inputs = tokenizer.encode(sentence_with_fake_token, return_tensors="pt")
logits = discriminator(inputs).logits.tolist()[0]

print("\t".join(tokenizer.tokenize(sentence_with_fake_token)))
print("\t".join(map(lambda x: str(x)[:4], logits[1:-1])))
"""Output:
Estamos desayun ##ando  pan     rosa    con     tomate  y       aceite  de      oliva   .
-3.1    -3.6    -6.9    -3.0    0.19    -4.5    -3.3    -5.1    -5.7    -7.7    -4.4    -4.2
"""
```


However, you probably want to use this model to fine-tune it on a downstream task.
We provide models fine-tuned on the [XNLI dataset](https://huggingface.co/datasets/xnli), which can be used together with the zero-shot classification pipeline:


- [Zero-shot SELECTRA small](https://huggingface.co/Recognai/zeroshot_selectra_small)
- [Zero-shot SELECTRA medium](https://huggingface.co/Recognai/zeroshot_selectra_medium)


## Metrics


We fine-tune our models on 3 different down-stream tasks:


- [XNLI](https://huggingface.co/datasets/xnli)
- [PAWS-X](https://huggingface.co/datasets/paws-x)
- [CoNLL2002 - NER](https://huggingface.co/datasets/conll2002)


For each task, we conduct 5 trials and state the mean and standard deviation of the metrics in the table below.
To compare our results to other Spanish language models, we provide the same metrics taken from the [evaluation table](https://github.com/PlanTL-SANIDAD/lm-spanish#evaluation-) of the [Spanish Language Model](https://github.com/PlanTL-SANIDAD/lm-spanish) repo.


| Model | CoNLL2002 - NER (f1) | PAWS-X (acc) | XNLI (acc) | Params |
| --- | --- | --- | --- | --- |
| SELECTRA small | 0.865 +- 0.004 | 0.896 +- 0.002 | 0.784 +- 0.002 | 22M |
| SELECTRA medium | 0.873 +- 0.003 | 0.896 +- 0.002 | 0.804 +- 0.002 | 41M |
|  |  |  |  |  |
| [mBERT](https://huggingface.co/bert-base-multilingual-cased) | 0.8691 | 0.8955 | 0.7876 | 178M |
| [BETO](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased) | 0.8759 | 0.9000 | 0.8130 | 110M |
| [RoBERTa-b](https://huggingface.co/BSC-TeMU/roberta-base-bne) | 0.8851 | 0.9000 | 0.8016 | 125M |
| [RoBERTa-l](https://huggingface.co/BSC-TeMU/roberta-large-bne) | 0.8772 | 0.9060 | 0.7958 | 355M |
| [Bertin](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1-512) | 0.8835 | 0.8990 | 0.7890 | 125M |
| [ELECTRICIDAD](https://huggingface.co/mrm8488/electricidad-base-discriminator) | 0.7954 | 0.9025 | 0.7878 | 109M |


Some details of our fine-tuning runs:


- epochs: 5
- batch-size: 32
- learning rate: 1e-4
- warmup proportion: 0.1
- linear learning rate decay
- layerwise learning rate decay


For all the details, check out our [selectra repo](https://github.com/recognai/selectra).


## Training


We pre-trained our SELECTRA models on the Spanish portion of the [Oscar](https://huggingface.co/datasets/oscar) dataset, which is about 150GB in size.
Each model version is trained for 300k steps, with a warm restart of the learning rate after the first 150k steps.
Some details of the training:


- steps: 300k
- batch-size: 128
- learning rate: 5e-4
- warmup steps: 10k
- linear learning rate decay
- TPU cores: 8 (v2-8)


For all details, check out our [selectra repo](https://github.com/recognai/selectra).


**Note:** Due to a misconfiguration in the pre-training scripts the embeddings of the vocabulary containing an accent were not optimized. If you fine-tune this model on a down-stream task, you might consider using a tokenizer that does not strip the accents:


```python
tokenizer = ElectraTokenizerFast.from_pretrained("Recognai/selectra_small", strip_accents=False)
```


## Motivation


Despite the abundance of excellent Spanish language models (BETO, BSC-BNE, Bertin, ELECTRICIDAD, etc.), we felt there was still a lack of distilled or compact Spanish language models and a lack of comparing those to their bigger siblings.


## Acknowledgment


This research was supported by the Google TPU Research Cloud (TRC) program.


## Authors


- David Fidalgo ([GitHub](https://github.com/dcfidalgo))
- Javier Lopez ([GitHub](https://github.com/javispp))
- Daniel Vila ([GitHub](https://github.com/dvsrepo))
- Francisco Aranda ([GitHub](https://github.com/frascuchon))


model name: selectra_medium


model author: Recognai


tags: pytorch, electra, pretraining, es, dataset:oscar, transformers, license:apache-2.0, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-timit_asr-oogway
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-timit_asr-oogway


This model is a fine-tuned version of [OthmaneJ/distil-wav2vec2](https://huggingface.co/OthmaneJ/distil-wav2vec2) on the None dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30
- mixed_precision_training: Native AMP


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-timit_asr-oogway


model author: baby-oogway


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- glue
model-index:
- name: distilbert-sst2-mahtab
results: []


# distilbert-sst2-mahtab


This model is a fine-tuned version of [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english) on the glue dataset.
It achieves the following results on the evaluation set:


- eval_loss: 0.4982
- eval_accuracy: 0.8830
- eval_runtime: 2.3447
- eval_samples_per_second: 371.91
- eval_steps_per_second: 46.489
- epoch: 1.0
- step: 8419


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: distilbert-sst2-mahtab


model author: Motahar


task: text-classification


tags: pytorch, tensorboard, distilbert, text-classification, dataset:glue, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


language: en
license: apache-2.0
datasets:


- sst-2


# DistilBERT base uncased finetuned SST-2


This model is a fine-tune checkpoint of [DistilBERT-base-uncased](https://huggingface.co/distilbert-base-uncased), fine-tuned on SST-2.
This model reaches an accuracy of 91.3 on the dev set (for comparison, Bert bert-base-uncased version reaches an accuracy of 92.7).


# Fine-tuning hyper-parameters


- learning_rate = 1e-5
- batch_size = 32
- warmup = 600
- max_seq_length = 128
- num_train_epochs = 3.0


model name: distilbert-base-uncased-finetuned-sst-2-english


tags: pytorch, tf, rust, distilbert, text-classification, en, dataset:sst-2, transformers, license:apache-2.0, infinity_compatible


## license: apache-2.0
language: en


# BART (base-sized model)


BART model pre-trained on English language. It was introduced in the paper [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/abs/1910.13461) by Lewis et al. and first released in [this repository](https://github.com/pytorch/fairseq/tree/master/examples/bart).


Disclaimer: The team releasing BART did not write a model card for this model so this model card has been written by the Hugging Face team.


## Model description


BART is a transformer encoder-encoder (seq2seq) model with a bidirectional (BERT-like) encoder and an autoregressive (GPT-like) decoder. BART is pre-trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text.


BART is particularly effective when fine-tuned for text generation (e.g. summarization, translation) but also works well for comprehension tasks (e.g. text classification, question answering).


## Intended uses & limitations


You can use the raw model for text infilling. However, the model is mostly meant to be fine-tuned on a supervised dataset. See the [model hub](https://huggingface.co/models?search=bart) to look for fine-tuned versions on a task that interests you.


### How to use


Here is how to use this model in tf_transformers:


```python
from tf_transformers.models import BartModel
from transformers import BartTokenizer

tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartModel.from_pretrained('facebook/bart-base')

inputs_tf = {}
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

inputs_tf["encoder_input_ids"] = inputs["input_ids"]
inputs_tf["encoder_input_mask"] = inputs["attention_mask"]
inputs_tf["decoder_input_ids"] = decoder_input_ids
outputs_tf = model(inputs_tf)

```


### BibTeX entry and citation info


```bibtex
@article{DBLP:journals/corr/abs-1910-13461,
  author    = {Mike Lewis and
               Yinhan Liu and
               Naman Goyal and
               Marjan Ghazvininejad and
               Abdelrahman Mohamed and
               Omer Levy and
               Veselin Stoyanov and
               Luke Zettlemoyer},
  title     = {{BART:} Denoising Sequence-to-Sequence Pre-training for Natural Language
               Generation, Translation, and Comprehension},
  journal   = {CoRR},
  volume    = {abs/1910.13461},
  year      = {2019},
  url       = {http://arxiv.org/abs/1910.13461},
  eprinttype = {arXiv},
  eprint    = {1910.13461},
  timestamp = {Thu, 31 Oct 2019 14:02:26 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1910-13461.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```


model name: bart-base


model author: tftransformers


tags: en, arxiv:1910.13461, license:apache-2.0


language: en
tags:


- exbert
license: apache-2.0
datasets:
- bookcorpus
- wikipedia


# BERT base model (cased)


Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
[this paper](https://arxiv.org/abs/1810.04805) and first released in
[this repository](https://github.com/google-research/bert). This model is case-sensitive: it makes a difference between
english and English.


Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by
the Hugging Face team.


## Model description


BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
was pretrained with two objectives:


- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
the entire masked sentence through the model and has to predict the masked words. This is different from traditional
recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
sentence.
- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
predict if the two sentences were following each other or not.


This way, the model learns an inner representation of the English language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the BERT model as inputs.


## Intended uses & limitations


You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for
fine-tuned versions on a task that interests you.


Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.


### How to use


You can use this model directly with a pipeline for masked language modeling:
In tf_transformers


```python
from tf_transformers.models import BertModel
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = BertModel.from_pretrained("bert-base-cased")

text = "Replace me by any text you'd like."
inputs_tf = {}
inputs = tokenizer(text, return_tensors='tf')

inputs_tf["input_ids"] = inputs["input_ids"]
inputs_tf["input_type_ids"] = inputs["token_type_ids"]
inputs_tf["input_mask"] = inputs["attention_mask"]
outputs_tf = model(inputs_tf)
```


## Training data


The BERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
headers).


## Training procedure


### Preprocessing


The texts are tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are then of the form:


```
[CLS] Sentence A [SEP] Sentence B [SEP]
```