Model trained with SemEval 2019 Task 5: HatEval (SubTask B) corpus for Hate Speech detection in English. Base model is [BERTweet](https://huggingface.co/vinai/bertweet-base), a RoBERTa model trained in English tweets.


It is a multi-classifier model, with the following classes:


- **HS**: is it hate speech?
- **TR**: is it targeted to a specific individual?
- **AG**: is it aggressive?


## License


`pysentimiento` is an open-source library for non-commercial use and scientific research purposes only. Please be aware that models are trained with third-party datasets and are subject to their respective licenses.


1. [TASS Dataset license](http://tass.sepln.org/tass_data/download.php)
1. [SEMEval 2017 Dataset license]()


## Citation


If you use `pysentimiento` in your work, please cite [this paper](https://arxiv.org/abs/2106.09462)


```
@misc{perez2021pysentimiento,
      title={pysentimiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks},
      author={Juan Manuel Pérez and Juan Carlos Giudici and Franco Luque},
      year={2021},
      eprint={2106.09462},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


Enjoy! 🤗


model name: bertweet-hate-speech


model author: pysentimiento


task: text-classification


tags: pytorch, roberta, text-classification, en, arxiv:2106.09462, transformers, twitter, hate-speech, infinity_compatible


# robertuito-base-cased


**WORK IN PROGRESS**


# RoBERTuito


## A pre-trained language model for social media text in Spanish


[**READ THE FULL PAPER**](https://arxiv.org/abs/2111.09453)
[Github Repository](https://github.com/pysentimiento/robertuito)


*RoBERTuito* is a pre-trained language model for user-generated content in Spanish, trained following RoBERTa guidelines on 500 million tweets. *RoBERTuito* comes in 3 flavors: cased, uncased, and uncased+deaccented.


We tested *RoBERTuito* on a benchmark of tasks involving user-generated text in Spanish. It outperforms other pre-trained language models for this language such as *BETO*, *BERTin* and *RoBERTa-BNE*. The 4 tasks selected for evaluation were: Hate Speech Detection (using SemEval 2019 Task 5, HatEval dataset), Sentiment and Emotion Analysis (using TASS 2020 datasets), and Irony detection (using IrosVa 2019 dataset).


| model              | hate speech     | sentiment analysis   | emotion analysis   | irony detection  |   score |
|:-------------------|:----------------|:---------------------|:-------------------|:-----------------|---------:|
| robertuito-uncased | 0.801 ± 0.010  | 0.707 ± 0.004       | 0.551 ± 0.011     | 0.736 ± 0.008   | 0.6987  |
| robertuito-deacc   | 0.798 ± 0.008  | 0.702 ± 0.004       | 0.543 ± 0.015     | 0.740 ± 0.006   | 0.6958  |
| robertuito-cased   | 0.790 ± 0.012  | 0.701 ± 0.012       | 0.519 ± 0.032     | 0.719 ± 0.023   | 0.6822  |
| roberta-bne        | 0.766 ± 0.015  | 0.669 ± 0.006       | 0.533 ± 0.011     | 0.723 ± 0.017   | 0.6726  |
| bertin             | 0.767 ± 0.005  | 0.665 ± 0.003       | 0.518 ± 0.012     | 0.716 ± 0.008   | 0.6666  |
| beto-cased         | 0.768 ± 0.012  | 0.665 ± 0.004       | 0.521 ± 0.012     | 0.706 ± 0.007   | 0.6651  |
| beto-uncased       | 0.757 ± 0.012  | 0.649 ± 0.005       | 0.521 ± 0.006     | 0.702 ± 0.008   | 0.6571  |


We release the pre-trained models on huggingface model hub:


- [RoBERTuito uncased](https://huggingface.co/pysentimiento/robertuito-base-uncased)
- [RoBERTuito cased](https://huggingface.co/pysentimiento/robertuito-base-cased)
- [RoBERTuito deacc](https://huggingface.co/pysentimiento/robertuito-base-deacc)


## Masked LM


To test the masked LM, take into account that space is encoded inside SentencePiece's tokens. So, if you want to test


```
Este es un día<mask>
```


don't put a space between `día` and `<mask>`


## Usage


**IMPORTANT -- READ THIS FIRST**


*RoBERTuito* is not yet fully-integrated into `huggingface/transformers`. To use it, first install `pysentimiento`


```bash
pip install pysentimiento
```


and preprocess text using `pysentimiento.preprocessing.preprocess_tweet` before feeding it into the tokenizer


```python
from transformers import AutoTokenizer
from pysentimiento.preprocessing import preprocess_tweet

tokenizer = AutoTokenizer.from_pretrained('pysentimiento/robertuito-base-cased')

text = "Esto es un tweet estoy usando #Robertuito @pysentimiento 🤣"
preprocessed_text = preprocess_tweet(text, ha)

tokenizer.tokenize(preprocessed_text)
# ['<s>','▁Esto','▁es','▁un','▁tweet','▁estoy','▁usando','▁','▁hashtag','▁','▁ro','bert','uito','▁@usuario','▁','▁emoji','▁cara','▁revolviéndose','▁de','▁la','▁risa','▁emoji','</s>']
```


We are working on integrating this preprocessing step into a Tokenizer within `transformers` library


## Citation


If you use *RoBERTuito*, please cite our paper:


```bibtex
@misc{perez2021robertuito,
      title={RoBERTuito: a pre-trained language model for social media text in Spanish},
      author={Juan Manuel Pérez and Damián A. Furman and Laura Alonso Alemany and Franco Luque},
      year={2021},
      eprint={2111.09453},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: robertuito-base-cased


model author: pysentimiento


task: fill-mask


tags: pytorch, roberta, fill-mask, arxiv:2111.09453, transformers, autonlp_compatible, infinity_compatible


language:


- es


tags:


- twitter
- masked-lm


# robertuito-base-uncased


# RoBERTuito


[Github Repository](https://github.com/pysentimiento/robertuito)


## Masked LM


```
Este es un día<mask>
```


## Usage


**IMPORTANT -- READ THIS FIRST**


```bash
pip install pysentimiento
```


## Citation


model name: robertuito-base-uncased


model author: pysentimiento


task: fill-mask


tags: pytorch, roberta, fill-mask, es, arxiv:2111.09453, transformers, twitter, masked-lm, autonlp_compatible, infinity_compatible


language:


- es


tags:


- twitter
- hate-speech


# Hate Speech detection in Spanish


## robertuito-hate-speech


Repository: [https://github.com/pysentimiento/pysentimiento/](https://github.com/finiteautomata/pysentimiento/)


Model trained with SemEval 2019 Task 5: HatEval (SubTask B) corpus for Hate Speech detection in Spanish. Base model is [RoBERTuito](https://github.com/pysentimiento/robertuito), a RoBERTa model trained in Spanish tweets.


It is a multi-classifier model, with the following classes:


- **HS**: is it hate speech?
- **TR**: is it targeted to a specific individual?
- **AG**: is it aggressive?


## Results


Results for the four tasks evaluated in `pysentimiento`. Results are expressed as Macro F1 scores


| model         | emotion       | hate_speech   | irony         | sentiment     |
|:--------------|:--------------|:--------------|:--------------|:--------------|
| robertuito    | 0.560 ± 0.010 | 0.759 ± 0.007 | 0.739 ± 0.005 | 0.705 ± 0.003 |
| roberta       | 0.527 ± 0.015 | 0.741 ± 0.012 | 0.721 ± 0.008 | 0.670 ± 0.006 |
| bertin        | 0.524 ± 0.007 | 0.738 ± 0.007 | 0.713 ± 0.012 | 0.666 ± 0.005 |
| beto_uncased  | 0.532 ± 0.012 | 0.727 ± 0.016 | 0.701 ± 0.007 | 0.651 ± 0.006 |
| beto_cased    | 0.516 ± 0.012 | 0.724 ± 0.012 | 0.705 ± 0.009 | 0.662 ± 0.005 |
| mbert_uncased | 0.493 ± 0.010 | 0.718 ± 0.011 | 0.681 ± 0.010 | 0.617 ± 0.003 |
| biGRU         | 0.264 ± 0.007 | 0.592 ± 0.018 | 0.631 ± 0.011 | 0.585 ± 0.011 |


Note that for Hate Speech, these are the results for Semeval 2019, Task 5 Subtask B (HS+TR+AG detection)


## Citation


If you use this model in your research, please cite pysentimiento, RoBERTuito and HatEval papers:


```
% Pysentimiento
@misc{perez2021pysentimiento,
      title={pysentimiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks},
      author={Juan Manuel Pérez and Juan Carlos Giudici and Franco Luque},
      year={2021},
      eprint={2106.09462},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
% RoBERTuito paper
@misc{perez2021robertuito,
      title={RoBERTuito: a pre-trained language model for social media text in Spanish},
      author={Juan Manuel Pérez and Damián A. Furman and Laura Alonso Alemany and Franco Luque},
      year={2021},
      eprint={2111.09453},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
% HatEval paper

@inproceedings{hateval2019semeval,
  title={SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter},
  author={Basile, Valerio and Bosco, Cristina and Fersini, Elisabetta and Nozza, Debora and Patti, Viviana and Rangel, Francisco and Rosso, Paolo and Sanguinetti, Manuela},
  booktitle={Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019)},
  year={2019},
  publisher= {Association for Computational Linguistics}
}
```


model name: robertuito-hate-speech


model author: pysentimiento


task: text-classification


tags: pytorch, roberta, text-classification, es, arxiv:2106.09462, arxiv:2111.09453, transformers, twitter, hate-speech, infinity_compatible


# robertuito-base-deacc


# RoBERTuito


## A pre-trained language model for social media text in Spanish


[**READ THE FULL PAPER**](https://arxiv.org/abs/2111.09453)
[Github Repository](https://github.com/pysentimiento/robertuito)


*RoBERTuito* is a pre-trained language model for user-generated content in Spanish, trained following RoBERTa guidelines on 500 million tweets. *RoBERTuito* comes in 3 flavors: cased, uncased, and uncased+deaccented.


We tested *RoBERTuito* on a benchmark of tasks involving user-generated text in Spanish. It outperforms other pre-trained language models for this language such as *BETO*, *BERTin* and *RoBERTa-BNE*. The 4 tasks selected for evaluation were: Hate Speech Detection (using SemEval 2019 Task 5, HatEval dataset), Sentiment and Emotion Analysis (using TASS 2020 datasets), and Irony detection (using IrosVa 2019 dataset).


| model              | hate speech     | sentiment analysis   | emotion analysis   | irony detection  |   score |
|:-------------------|:----------------|:---------------------|:-------------------|:-----------------|---------:|
| robertuito-uncased | 0.801 ± 0.010  | 0.707 ± 0.004       | 0.551 ± 0.011     | 0.736 ± 0.008   | 0.6987  |
| robertuito-deacc   | 0.798 ± 0.008  | 0.702 ± 0.004       | 0.543 ± 0.015     | 0.740 ± 0.006   | 0.6958  |
| robertuito-cased   | 0.790 ± 0.012  | 0.701 ± 0.012       | 0.519 ± 0.032     | 0.719 ± 0.023   | 0.6822  |
| roberta-bne        | 0.766 ± 0.015  | 0.669 ± 0.006       | 0.533 ± 0.011     | 0.723 ± 0.017   | 0.6726  |
| bertin             | 0.767 ± 0.005  | 0.665 ± 0.003       | 0.518 ± 0.012     | 0.716 ± 0.008   | 0.6666  |
| beto-cased         | 0.768 ± 0.012  | 0.665 ± 0.004       | 0.521 ± 0.012     | 0.706 ± 0.007   | 0.6651  |
| beto-uncased       | 0.757 ± 0.012  | 0.649 ± 0.005       | 0.521 ± 0.006     | 0.702 ± 0.008   | 0.6571  |


We release the pre-trained models on huggingface model hub:


- [RoBERTuito uncased](https://huggingface.co/pysentimiento/robertuito-base-uncased)
- [RoBERTuito cased](https://huggingface.co/pysentimiento/robertuito-base-cased)
- [RoBERTuito deacc](https://huggingface.co/pysentimiento/robertuito-base-deacc)


## Masked LM


To test the masked LM, take into account that space is encoded inside SentencePiece's tokens. So, if you want to test


```
Este es un día<mask>
```


don't put a space between `día` and `<mask>`


## Usage


**IMPORTANT -- READ THIS FIRST**


*RoBERTuito* is not yet fully-integrated into `huggingface/transformers`. To use it, first install `pysentimiento`


```bash
pip install pysentimiento
```


and preprocess text using `pysentimiento.preprocessing.preprocess_tweet` before feeding it into the tokenizer


```python
from transformers import AutoTokenizer
from pysentimiento.preprocessing import preprocess_tweet

tokenizer = AutoTokenizer.from_pretrained('pysentimiento/robertuito-base-cased')

text = "Esto es un tweet estoy usando #Robertuito @pysentimiento 🤣"
preprocessed_text = preprocess_tweet(text, ha)

tokenizer.tokenize(preprocessed_text)
# ['<s>','▁Esto','▁es','▁un','▁tweet','▁estoy','▁usando','▁','▁hashtag','▁','▁ro','bert','uito','▁@usuario','▁','▁emoji','▁cara','▁revolviéndose','▁de','▁la','▁risa','▁emoji','</s>']
```


We are working on integrating this preprocessing step into a Tokenizer within `transformers` library


## Citation


If you use *RoBERTuito*, please cite our paper:


```bibtex
@misc{perez2021robertuito,
      title={RoBERTuito: a pre-trained language model for social media text in Spanish},
      author={Juan Manuel Pérez and Damián A. Furman and Laura Alonso Alemany and Franco Luque},
      year={2021},
      eprint={2111.09453},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: robertuito-base-deacc


model author: pysentimiento


task: fill-mask


tags: pytorch, roberta, fill-mask, arxiv:2111.09453, transformers, autonlp_compatible, infinity_compatible


# Irony detection in Spanish


## robertuito-irony


Repository: [https://github.com/pysentimiento/pysentimiento/](https://github.com/finiteautomata/pysentimiento/)


Model trained with IRosVA 2019 dataset for irony detection. Base model is [RoBERTuito](https://github.com/pysentimiento/robertuito), a RoBERTa model trained in Spanish tweets.


The positive class marks irony, the negative class marks not irony.


## Results


Results for the four tasks evaluated in `pysentimiento`. Results are expressed as Macro F1 scores


| model         | emotion       | hate_speech   | irony         | sentiment     |
|:--------------|:--------------|:--------------|:--------------|:--------------|
| robertuito    | 0.560 ± 0.010 | 0.759 ± 0.007 | 0.739 ± 0.005 | 0.705 ± 0.003 |
| roberta       | 0.527 ± 0.015 | 0.741 ± 0.012 | 0.721 ± 0.008 | 0.670 ± 0.006 |
| bertin        | 0.524 ± 0.007 | 0.738 ± 0.007 | 0.713 ± 0.012 | 0.666 ± 0.005 |
| beto_uncased  | 0.532 ± 0.012 | 0.727 ± 0.016 | 0.701 ± 0.007 | 0.651 ± 0.006 |
| beto_cased    | 0.516 ± 0.012 | 0.724 ± 0.012 | 0.705 ± 0.009 | 0.662 ± 0.005 |
| mbert_uncased | 0.493 ± 0.010 | 0.718 ± 0.011 | 0.681 ± 0.010 | 0.617 ± 0.003 |
| biGRU         | 0.264 ± 0.007 | 0.592 ± 0.018 | 0.631 ± 0.011 | 0.585 ± 0.011 |


Note that for Hate Speech, these are the results for Semeval 2019, Task 5 Subtask B (HS+TR+AG detection)


If you use this model in your research, please cite pysentimiento and RoBERTuito papers:


```
@misc{perez2021pysentimiento,
      title={pysentimiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks},
      author={Juan Manuel Pérez and Juan Carlos Giudici and Franco Luque},
      year={2021},
      eprint={2106.09462},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
@misc{perez2021robertuito,
      title={RoBERTuito: a pre-trained language model for social media text in Spanish},
      author={Juan Manuel Pérez and Damián A. Furman and Laura Alonso Alemany and Franco Luque},
      year={2021},
      eprint={2111.09453},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: robertuito-irony


task: text-classification


tags: pytorch, roberta, text-classification, arxiv:2106.09462, arxiv:2111.09453, transformers, infinity_compatible


language:


- es


tags:


- twitter
- sentiment-analysis


# Sentiment Analysis in Spanish


## robertuito-sentiment-analysis


Model trained with TASS 2020 corpus (around ~5k tweets) of several dialects of Spanish. Base model is [RoBERTuito](https://github.com/pysentimiento/robertuito), a RoBERTa model trained in Spanish tweets.


Uses `POS`, `NEG`, `NEU` labels.


## Results


Note that for Hate Speech, these are the results for Semeval 2019, Task 5 Subtask B


## Citation


model name: robertuito-sentiment-analysis


model author: pysentimiento


tags: pytorch, roberta, text-classification, es, arxiv:2106.09462, arxiv:2111.09453, transformers, twitter, sentiment-analysis, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-hindi-colab
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-hindi-colab


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-large-xls-r-300m-hindi-colab


model author: nimrah


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- conll2003
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: bert-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: conll2003
type: conll2003
args: conll2003
metrics:
    - name: Precision
type: precision
value: 0.9389679126695336
    - name: Recall
type: recall
value: 0.9554022214742511
    - name: F1
type: f1
value: 0.9471137804471137
    - name: Accuracy
type: accuracy
value: 0.9873138282215812


# bert-finetuned-ner


This model is a fine-tuned version of [bert-base-cased](https://huggingface.co/bert-base-cased) on the conll2003 dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0586
- Precision: 0.9390
- Recall: 0.9554
- F1: 0.9471
- Accuracy: 0.9873


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.0877        | 1.0   | 1756 | 0.0662          | 0.9081    | 0.9344 | 0.9210 | 0.9827   |
| 0.0376        | 2.0   | 3512 | 0.0599          | 0.9362    | 0.9502 | 0.9431 | 0.9862   |
| 0.0209        | 3.0   | 5268 | 0.0586          | 0.9390    | 0.9554 | 0.9471 | 0.9873   |


### Framework versions


- Transformers 4.14.1
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: bert-finetuned-ner


model author: butchland


task: token-classification


tags: pytorch, tensorboard, bert, token-classification, dataset:conll2003, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-kika5_my-colab
results: []


# wav2vec2-large-xls-r-300m-kika5_my-colab


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3860
- Wer: 0.3505


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 100
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 4.0007        | 4.82  | 400  | 0.6696          | 0.8283 |
| 0.2774        | 9.64  | 800  | 0.4231          | 0.5476 |
| 0.1182        | 14.46 | 1200 | 0.4253          | 0.5102 |
| 0.0859        | 19.28 | 1600 | 0.4600          | 0.4866 |
| 0.0693        | 24.1  | 2000 | 0.4030          | 0.4533 |
| 0.0611        | 28.92 | 2400 | 0.4189          | 0.4412 |
| 0.0541        | 33.73 | 2800 | 0.4272          | 0.4380 |
| 0.0478        | 38.55 | 3200 | 0.4537          | 0.4505 |
| 0.0428        | 43.37 | 3600 | 0.4349          | 0.4181 |
| 0.038         | 48.19 | 4000 | 0.4562          | 0.4199 |
| 0.0345        | 53.01 | 4400 | 0.4209          | 0.4310 |
| 0.0316        | 57.83 | 4800 | 0.4336          | 0.4058 |
| 0.0288        | 62.65 | 5200 | 0.4004          | 0.3920 |
| 0.025         | 67.47 | 5600 | 0.4115          | 0.3857 |
| 0.0225        | 72.29 | 6000 | 0.4296          | 0.3948 |
| 0.0182        | 77.11 | 6400 | 0.3963          | 0.3772 |
| 0.0165        | 81.93 | 6800 | 0.3921          | 0.3687 |
| 0.0152        | 86.75 | 7200 | 0.3969          | 0.3592 |
| 0.0133        | 91.57 | 7600 | 0.3803          | 0.3527 |
| 0.0118        | 96.39 | 8000 | 0.3860          | 0.3505 |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: wav2vec2-large-xls-r-300m-kika5_my-colab


model author: kika2000


task: automatic-speech-recognition


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-kika_my-colab
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-kika_my-colab


- Loss: 1.3300
- Wer: 0.5804


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.8067        | 4.82  | 400  | 1.2892          | 0.8886 |
| 0.3048        | 9.64  | 800  | 1.2285          | 0.6797 |
| 0.1413        | 14.46 | 1200 | 1.1970          | 0.6509 |
| 0.1047        | 19.28 | 1600 | 1.3628          | 0.6166 |
| 0.0799        | 24.1  | 2000 | 1.3345          | 0.6014 |
| 0.0638        | 28.92 | 2400 | 1.3300          | 0.5804 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: wav2vec2-large-xls-r-300m-kika_my-colab


model author: kika2000


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-kika4_my-colab
results: []


# wav2vec2-large-xls-r-300m-kika4_my-colab


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 70
- mixed_precision_training: Native AMP


### Framework versions


model name: wav2vec2-large-xls-r-300m-kika4_my-colab


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-georgian2-colab
results: []


# wav2vec2-large-xls-r-300m-georgian2-colab


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4317
- Wer: 0.4280


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.7071        | 4.76  | 400  | 0.6897          | 0.7844 |
| 0.2908        | 9.52  | 800  | 0.4630          | 0.5582 |
| 0.1392        | 14.29 | 1200 | 0.4501          | 0.5006 |
| 0.0977        | 19.05 | 1600 | 0.4593          | 0.4755 |
| 0.075         | 23.81 | 2000 | 0.4340          | 0.4401 |
| 0.0614        | 28.57 | 2400 | 0.4317          | 0.4280 |


model name: wav2vec2-large-xls-r-300m-kika10


model author: kika2000


task: automatic-speech-recognition


license: mit
tags:


- generated_from_trainer
metrics:
- accuracy
- f1
model-index:
- name: xlm-roberta-base-language-detection
results: []


# xlm-roberta-base-language-detection


This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the [Language Identification](https://huggingface.co/datasets/papluca/language-identification#additional-information) dataset.


## Model description


This model is an XLM-RoBERTa transformer model with a classification head on top (i.e. a linear layer on top of the pooled output).
For additional information please refer to the [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) model card or to the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Conneau et al.


## Intended uses & limitations


You can directly use this model as a language detector, i.e. for sequence classification tasks. Currently, it supports the following 20 languages:


`arabic (ar), bulgarian (bg), german (de), modern greek (el), english (en), spanish (es), french (fr), hindi (hi), italian (it), japanese (ja), dutch (nl), polish (pl), portuguese (pt), russian (ru), swahili (sw), thai (th), turkish (tr), urdu (ur), vietnamese (vi), and chinese (zh)`


## Training and evaluation data


The model was fine-tuned on the [Language Identification](https://huggingface.co/datasets/papluca/language-identification#additional-information) dataset, which consists of text sequences in 20 languages. The training set contains 70k samples, while the validation and test sets 10k each. The average accuracy on the test set is **99.6%** (this matches the average macro/weighted F1-score being the test set perfectly balanced). A more detailed evaluation is provided by the following table.


| Language | Precision | Recall | F1-score | support |
|:--------:|:---------:|:------:|:--------:|:-------:|
|ar        |0.998      |0.996   |0.997     |500      |
|bg        |0.998      |0.964   |0.981     |500      |
|de        |0.998      |0.996   |0.997     |500      |
|el        |0.996      |1.000   |0.998     |500      |
|en        |1.000      |1.000   |1.000     |500      |
|es        |0.967      |1.000   |0.983     |500      |
|fr        |1.000      |1.000   |1.000     |500      |
|hi        |0.994      |0.992   |0.993     |500      |
|it        |1.000      |0.992   |0.996     |500      |
|ja        |0.996      |0.996   |0.996     |500      |
|nl        |1.000      |1.000   |1.000     |500      |
|pl        |1.000      |1.000   |1.000     |500      |
|pt        |0.988      |1.000   |0.994     |500      |
|ru        |1.000      |0.994   |0.997     |500      |
|sw        |1.000      |1.000   |1.000     |500      |
|th        |1.000      |0.998   |0.999     |500      |
|tr        |0.994      |0.992   |0.993     |500      |
|ur        |1.000      |1.000   |1.000     |500      |
|vi        |0.992      |1.000   |0.996     |500      |
|zh        |1.000      |1.000   |1.000     |500      |


### Benchmarks


As a baseline to compare `xlm-roberta-base-language-detection` against, we have used the Python [langid](https://github.com/saffsd/langid.py) library. Since it comes pre-trained on 97 languages, we have used its `.set_languages()` method to constrain the language set to our 20 languages. The average accuracy of langid on the test set is **98.5%**. More details are provided by the table below.


| Language | Precision | Recall | F1-score | support |
|:--------:|:---------:|:------:|:--------:|:-------:|
|ar        |0.990      |0.970   |0.980     |500      |
|bg        |0.998      |0.964   |0.981     |500      |
|de        |0.992      |0.944   |0.967     |500      |
|el        |1.000      |0.998   |0.999     |500      |
|en        |1.000      |1.000   |1.000     |500      |
|es        |1.000      |0.968   |0.984     |500      |
|fr        |0.996      |1.000   |0.998     |500      |
|hi        |0.949      |0.976   |0.963     |500      |
|it        |0.990      |0.980   |0.985     |500      |
|ja        |0.927      |0.988   |0.956     |500      |
|nl        |0.980      |1.000   |0.990     |500      |
|pl        |0.986      |0.996   |0.991     |500      |
|pt        |0.950      |0.996   |0.973     |500      |
|ru        |0.996      |0.974   |0.985     |500      |
|sw        |1.000      |1.000   |1.000     |500      |
|th        |1.000      |0.996   |0.998     |500      |
|tr        |0.990      |0.968   |0.979     |500      |
|ur        |0.998      |0.996   |0.997     |500      |
|vi        |0.971      |0.990   |0.980     |500      |
|zh        |1.000      |1.000   |1.000     |500      |


## Training procedure


Fine-tuning was done via the `Trainer` API.


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 128
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2
- mixed_precision_training: Native AMP


### Training results


The validation results on the `valid` split of the Language Identification dataset are summarised here below.


| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|
| 0.2492        | 1.0   | 1094 | 0.0149          | 0.9969   | 0.9969 |
| 0.0101        | 2.0   | 2188 | 0.0103          | 0.9977   | 0.9977 |


In short, it achieves the following results on the validation set:


- Loss: 0.0101
- Accuracy: 0.9977
- F1: 0.9977


### Framework versions


- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


model name: xlm-roberta-base-language-detection


model author: papluca


task: text-classification


tags: pytorch, tf, xlm-roberta, text-classification, arxiv:1911.02116, transformers, license:mit, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: t5-small-finetuned-xsum-finetuned-billsum
results:
  - task:
name: Sequence-to-sequence Language Modeling
type: text2text-generation


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# t5-small-finetuned-xsum-finetuned-billsum


This model is a fine-tuned version of [Frederick0291/t5-small-finetuned-xsum](https://huggingface.co/Frederick0291/t5-small-finetuned-xsum) on an unknown dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| No log        | 1.0   | 330  | 1.8540          | 32.9258 | 14.9104 | 27.1067 | 27.208    | 18.8437 |


### Framework versions


- Transformers 4.10.2
- Pytorch 1.9.0+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: t5-small-finetuned-xsum


model author: Frederick0291


task: text2text-generation


tags: pytorch, t5, text2text-generation, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- billsum
metrics:
- rouge
model-index:
- name: t5-small-finetuned-billsum
results:
  - task:
name: Sequence-to-sequence Language Modeling
type: text2text-generation
dataset:
name: billsum
type: billsum
args: default
metrics:
    - name: Rouge1
type: rouge
value: 16.6044


# t5-small-finetuned-billsum


This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the billsum dataset.
It achieves the following results on the evaluation set:


- Loss: 2.0972
- Rouge1: 16.6044
- Rouge2: 12.8656
- Rougel: 15.7876
- Rougelsum: 15.9784
- Gen Len: 18.9948


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| 2.3854        | 1.0   | 2369 | 2.0972          | 16.6044 | 12.8656 | 15.7876 | 15.9784   | 18.9948 |


### Framework versions


model name: t5-small-finetuned-billsum


model author: Frederick0291


task: text2text-generation


tags: pytorch, tensorboard, t5, text2text-generation, dataset:billsum, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


pipeline_tag: sentence-similarity
tags:


- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers


# jhemmingsson/lab2


This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.


<!--- Describe your model here -->


## Usage (Sentence-Transformers)


Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:


```
pip install -U sentence-transformers
```


Then you can use the model like this:


```python
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('jhemmingsson/lab2')
embeddings = model.encode(sentences)
print(embeddings)
```


## Usage (HuggingFace Transformers)


Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.


```python
from transformers import AutoTokenizer, AutoModel
import torch

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('jhemmingsson/lab2')
model = AutoModel.from_pretrained('jhemmingsson/lab2')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)
```


## Evaluation Results


<!--- Describe how your model was evaluated -->


For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=jhemmingsson/lab2)


## Training


The model was trained with the parameters:


**DataLoader**:


`torch.utils.data.dataloader.DataLoader` of length 357 with parameters:


```
{'batch_size': 16, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
```


**Loss**:


`sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss`


Parameters of the fit()-Method:


```
{
    "epochs": 1,
    "evaluation_steps": 0,
    "evaluator": "NoneType",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 2e-05
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 100,
    "weight_decay": 0.01
}
```


## Full Model Architecture


```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DistilBertModel
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)
```


## Citing & Authors


<!--- Describe where people can find more information -->


model name: lab2


model author: jhemmingsson


task: sentence-similarity


tags: pytorch, distilbert, feature-extraction, sentence-transformers, sentence-similarity, transformers, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
- robust-speech-event
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-guarani-small
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-guarani-small


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4964
- Wer: 0.5957


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0002
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| No log        | 6.65  | 100  | 1.1326          | 1.0    |
| 1.6569        | 13.32 | 200  | 0.5264          | 0.6478 |
| 1.6569        | 19.97 | 300  | 0.5370          | 0.6261 |
| 0.2293        | 26.65 | 400  | 0.4964          | 0.5957 |


### Framework versions


- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


model name: wav2vec2-large-xls-r-300m-guarani-small


model author: jhonparra18


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


license: apache-2.0
tags:


- generated_from_trainer
- "es"
- "robust-speech-event"
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-spanish-large
results: []


# wav2vec2-large-xls-r-300m-spanish-large


This model is a fine-tuned version of [tomascufaro/xls-r-es-test](https://huggingface.co/tomascufaro/xls-r-es-test) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.1431
- Wer: 0.1197


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0002
- train_batch_size: 10
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 20
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 300
- num_epochs: 5
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.1769        | 0.15  | 400   | 0.1795          | 0.1698 |
| 0.217         | 0.3   | 800   | 0.2000          | 0.1945 |
| 0.2372        | 0.45  | 1200  | 0.1985          | 0.1859 |
| 0.2351        | 0.6   | 1600  | 0.1901          | 0.1772 |
| 0.2269        | 0.75  | 2000  | 0.1968          | 0.1783 |
| 0.2284        | 0.9   | 2400  | 0.1873          | 0.1771 |
| 0.2014        | 1.06  | 2800  | 0.1840          | 0.1696 |
| 0.1988        | 1.21  | 3200  | 0.1904          | 0.1730 |
| 0.1919        | 1.36  | 3600  | 0.1827          | 0.1630 |
| 0.1919        | 1.51  | 4000  | 0.1788          | 0.1629 |
| 0.1817        | 1.66  | 4400  | 0.1755          | 0.1558 |
| 0.1812        | 1.81  | 4800  | 0.1795          | 0.1638 |
| 0.1808        | 1.96  | 5200  | 0.1762          | 0.1603 |
| 0.1625        | 2.11  | 5600  | 0.1721          | 0.1557 |
| 0.1477        | 2.26  | 6000  | 0.1735          | 0.1504 |
| 0.1508        | 2.41  | 6400  | 0.1708          | 0.1478 |
| 0.157         | 2.56  | 6800  | 0.1644          | 0.1466 |
| 0.1491        | 2.71  | 7200  | 0.1638          | 0.1445 |
| 0.1458        | 2.86  | 7600  | 0.1582          | 0.1426 |
| 0.1387        | 3.02  | 8000  | 0.1607          | 0.1376 |
| 0.1269        | 3.17  | 8400  | 0.1559          | 0.1364 |
| 0.1172        | 3.32  | 8800  | 0.1521          | 0.1335 |
| 0.1203        | 3.47  | 9200  | 0.1534          | 0.1330 |
| 0.1177        | 3.62  | 9600  | 0.1485          | 0.1304 |
| 0.1167        | 3.77  | 10000 | 0.1498          | 0.1302 |
| 0.1194        | 3.92  | 10400 | 0.1463          | 0.1287 |
| 0.1053        | 4.07  | 10800 | 0.1483          | 0.1282 |
| 0.098         | 4.22  | 11200 | 0.1498          | 0.1267 |
| 0.0958        | 4.37  | 11600 | 0.1461          | 0.1233 |
| 0.0946        | 4.52  | 12000 | 0.1444          | 0.1218 |
| 0.094         | 4.67  | 12400 | 0.1434          | 0.1206 |
| 0.0932        | 4.82  | 12800 | 0.1424          | 0.1206 |
| 0.0912        | 4.98  | 13200 | 0.1431          | 0.1197 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-300m-spanish-large-noLM


model author: jhonparra18


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, es, robust-speech-event, model-index


tags:


- generated_from_trainer
- robust-speech-event
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-spanish-custom
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-spanish-custom


This model was trained from scratch on the common_voice dataset.
It achieves the following results on the evaluation set:


- eval_loss: 0.2245
- eval_wer: 0.2082
- eval_runtime: 801.6784
- eval_samples_per_second: 18.822
- eval_steps_per_second: 2.354
- epoch: 0.76
- step: 8400


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0002
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 200
- num_epochs: 10
- mixed_precision_training: Native AMP


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0


model name: wav2vec2-large-xls-r-300m-spanish-custom


model author: jhonparra18


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, generated_from_trainer, robust-speech-event, model-index


tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: ft_electra-small-turkish-uncased-discriminator_lr-2e-1_epochs-1
results: []


This model is a fine-tuned version of [loodos/electra-small-turkish-uncased-discriminator](https://huggingface.co/loodos/electra-small-turkish-uncased-discriminator) on the squad dataset.
It achieves the following results on the evaluation set:


- Loss: 5.9506


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.2
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 5.951         | 1.0   | 5818 | 5.9506          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: electra-small-turkish-uncased-discriminator


model author: husnu


task: question-answering


tags: pytorch, tensorboard, electra, question-answering, dataset:squad, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: xtremedistil-l6-h256-uncased-finetuned_lr-2e-05_epochs-6
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xtremedistil-l6-h256-uncased-finetuned_lr-2e-05_epochs-6


This model is a fine-tuned version of [microsoft/xtremedistil-l6-h256-uncased](https://huggingface.co/microsoft/xtremedistil-l6-h256-uncased) on the squad dataset.
It achieves the following results on the evaluation set:


- Loss: 1.2578


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 48
- eval_batch_size: 48
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 6


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.3828        | 1.0   | 1845  | 1.7946          |
| 1.5827        | 2.0   | 3690  | 1.4123          |
| 1.404         | 3.0   | 5535  | 1.3142          |
| 1.346         | 4.0   | 7380  | 1.2819          |
| 1.2871        | 5.0   | 9225  | 1.2630          |
| 1.2538        | 6.0   | 11070 | 1.2578          |


### Framework versions


model name: xtremedistil-l6-h256-uncased-finetuned_lr-2e-05_epochs-6


model author: husnu


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, dataset:squad, transformers, license:mit, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-6
results: []


# xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-6


This model is a fine-tuned version of [microsoft/xtremedistil-l6-h256-uncased](https://huggingface.co/microsoft/xtremedistil-l6-h256-uncased) on the Turkish squad dataset.
It achieves the following results on the evaluation set:


- Loss: 2.8135


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 350  | 3.8389          |
| 4.4474        | 2.0   | 700  | 3.3748          |
| 3.512         | 3.0   | 1050 | 3.0657          |
| 3.512         | 4.0   | 1400 | 2.9219          |
| 3.1526        | 5.0   | 1750 | 2.8517          |
| 2.9972        | 6.0   | 2100 | 2.8135          |


### Framework versions


model name: xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-6


model author: husnu


task: question-answering


license: mit
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-9
results: []


# xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-9


- Loss: 2.2340


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 9


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.5236        | 1.0   | 1050 | 3.0042          |
| 2.8489        | 2.0   | 2100 | 2.5866          |
| 2.5485        | 3.0   | 3150 | 2.3526          |
| 2.4067        | 4.0   | 4200 | 2.3535          |
| 2.3091        | 5.0   | 5250 | 2.2862          |
| 2.2401        | 6.0   | 6300 | 2.3989          |
| 2.1715        | 7.0   | 7350 | 2.2284          |
| 2.1414        | 8.0   | 8400 | 2.2298          |
| 2.1221        | 9.0   | 9450 | 2.2340          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-9


model author: husnu


task: question-answering


tags:


- generated_from_trainer
datasets:
- tsquad
model-index:
- name: electra-small-turkish-uncased-discriminator-finetuned_lr-2e-05_epochs-6
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# electra-small-turkish-uncased-discriminator-finetuned_lr-2e-05_epochs-6


This model is a fine-tuned version of [loodos/electra-small-turkish-uncased-discriminator](https://huggingface.co/loodos/electra-small-turkish-uncased-discriminator) on the turkish squad dataset.
It achieves the following results on the evaluation set:


- Loss: 2.0379


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 6


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 4.128         | 1.0   | 722  | 2.7187          |
| 3.0376        | 2.0   | 1444 | 2.4486          |
| 2.5304        | 3.0   | 2166 | 2.3485          |
| 2.4214        | 4.0   | 2888 | 2.0450          |
| 2.1568        | 5.0   | 3610 | 2.0576          |
| 2.0752        | 6.0   | 4332 | 2.0379          |


### Framework versions


model name: electra-small-turkish-uncased-discriminator-finetuned_lr-2e-05_epochs-6


model author: husnu


task: question-answering


tags: pytorch, tensorboard, electra, question-answering, dataset:tsquad, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: xtremedistil-l6-h256-uncased-finetuned_lr-2e-05_epochs-3
results: []


# xtremedistil-l6-h256-uncased-finetuned_lr-2e-05_epochs-3


This model is a fine-tuned version of [microsoft/xtremedistil-l6-h256-uncased](https://huggingface.co/microsoft/xtremedistil-l6-h256-uncased) on the squad dataset.
It achieves the following results on the evaluation set:


- Loss: 1.2864


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.6088        | 1.0   | 5533  | 1.4429          |
| 1.3928        | 2.0   | 11066 | 1.3183          |
| 1.3059        | 3.0   | 16599 | 1.2864          |


### Framework versions


model name: xtremedistil-l6-h256-uncased-finetuned_lr-2e-05_epochs-3


model author: husnu


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, dataset:squad, transformers, license:mit, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: electra-small-turkish-uncased-discriminator-finetuned_lr-2e-05_epochs-3
results: []


# electra-small-turkish-uncased-discriminator-finetuned_lr-2e-05_epochs-3


This model is a fine-tuned version of [loodos/electra-small-turkish-uncased-discriminator](https://huggingface.co/loodos/electra-small-turkish-uncased-discriminator) on the Turkish squad dataset.
It achieves the following results on the evaluation set:


- Loss: 2.1669


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.5305        | 1.0   | 5818  | 2.4024          |
| 2.3264        | 2.0   | 11636 | 2.2298          |
| 2.1762        | 3.0   | 17454 | 2.1669          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: electra-small-turkish-uncased-discriminator-finetuned_lr-2e-05_epochs-3


model author: husnu


task: question-answering


tags: pytorch, tensorboard, electra, question-answering, dataset:squad, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-3
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-3


This model is a fine-tuned version of [microsoft/xtremedistil-l6-h256-uncased](https://huggingface.co/microsoft/xtremedistil-l6-h256-uncased) on the turkish squad dataset.
It achieves the following results on the evaluation set:


- Loss: 2.6510


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.0113        | 1.0   | 1050 | 2.7529          |
| 2.838         | 2.0   | 2100 | 2.6510          |
| 2.8168        | 3.0   | 3150 | 2.6510          |


### Framework versions


model name: xtremedistil-l6-h256-uncased-TQUAD-finetuned_lr-2e-05_epochs-3


model author: husnu


task: question-answering


tags:


- generated_from_trainer
datasets:
- turkish squad v2
model-index:
- name: bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3TQUAD2-finetuned_lr-2e-05_epochs-3
results: []


# bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3TQUAD2-finetuned_lr-2e-05_epochs-3


This model is a fine-tuned version of [husnu/bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3](https://huggingface.co/husnu/bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3) on the turkish squad2 dataset.
It achieves the following results on the evaluation set:


- Loss: 1.9011


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.6404        | 1.0   | 2245 | 1.4524          |
| 0.403         | 2.0   | 4490 | 1.5638          |
| 0.2355        | 3.0   | 6735 | 1.9011          |


### Framework versions


model name: bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3TQUAD2-finetuned_lr-2e-05_epochs-3


model author: husnu


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, dataset:turkish squad v2, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3TQUAD2-finetuned_lr-2e-05_epochs-1
results: []


# bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3TQUAD2-finetuned_lr-2e-05_epochs-1


This model is a fine-tuned version of [husnu/bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3](https://huggingface.co/husnu/bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3) on the squad dataset.
It achieves the following results on the evaluation set:


- Loss: 1.4196


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.5885        | 1.0   | 2245 | 1.4196          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3TQUAD2-finetuned_lr-2e-05_epochs-1


model author: husnu


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, dataset:squad, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3


This model is a fine-tuned version of [dbmdz/bert-base-turkish-128k-cased](https://huggingface.co/dbmdz/bert-base-turkish-128k-cased) on the turkish squad dataset.
It achieves the following results on the evaluation set:


- Loss: 1.4724


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.3911        | 1.0   | 1281 | 1.4900          |
| 0.9058        | 2.0   | 2562 | 1.3471          |
| 0.6747        | 3.0   | 3843 | 1.4724          |


### Framework versions


model name: bert-base-turkish-128k-cased-finetuned_lr-2e-05_epochs-3


model author: husnu


task: question-answering


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: temp
results: []


# temp


This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4645
- Wer: 0.3527


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 2
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.4324        | 0.4   | 50   | 0.5800          | 0.4458 |
| 0.4027        | 0.8   | 100  | 0.5374          | 0.4109 |
| 0.3163        | 1.2   | 150  | 0.5285          | 0.3881 |
| 0.3064        | 1.6   | 200  | 0.5161          | 0.3815 |
| 0.3235        | 2.0   | 250  | 0.4645          | 0.3527 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: temp


model author: ying-tina


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab-32-epochs50-earlystop
results: []


# wav2vec2-base-timit-demo-colab-32-epochs50-earlystop


- Loss: 0.5208
- Wer: 0.3561


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 50
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.4294        | 4.0   | 500  | 1.3397          | 0.8966 |
| 0.5848        | 8.0   | 1000 | 0.4931          | 0.4585 |
| 0.2323        | 12.0  | 1500 | 0.4781          | 0.4008 |
| 0.14          | 16.0  | 2000 | 0.4294          | 0.3806 |
| 0.1026        | 20.0  | 2500 | 0.5098          | 0.3663 |
| 0.0725        | 24.0  | 3000 | 0.4527          | 0.3568 |
| 0.058         | 28.0  | 3500 | 0.5208          | 0.3561 |


### Framework versions


model name: wav2vec2-base-timit-demo-colab-32-epochs50-earlystop


model author: ying-tina


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab-32-epochs30
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-base-timit-demo-colab-32-epochs30


- Loss: 0.4615
- Wer: 0.3434


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.5243        | 4.0   | 500  | 1.4532          | 0.9540 |
| 0.6178        | 8.0   | 1000 | 0.5490          | 0.4627 |
| 0.223         | 12.0  | 1500 | 0.4513          | 0.3881 |
| 0.1299        | 16.0  | 2000 | 0.4573          | 0.3698 |
| 0.0875        | 20.0  | 2500 | 0.4950          | 0.3637 |
| 0.0613        | 24.0  | 3000 | 0.4327          | 0.3479 |
| 0.0478        | 28.0  | 3500 | 0.4615          | 0.3434 |


### Framework versions


model name: wav2vec2-base-timit-demo-colab-32-epochs30


model author: ying-tina


task: automatic-speech-recognition


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab
results: []


# wav2vec2-base-timit-demo-colab


This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.5127
- Wer: 0.3082


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.7645        | 2.01  | 500  | 2.5179          | 0.9999 |
| 1.1873        | 4.02  | 1000 | 0.5464          | 0.4798 |
| 0.46          | 6.02  | 1500 | 0.4625          | 0.4025 |
| 0.2869        | 8.03  | 2000 | 0.4252          | 0.3650 |
| 0.2213        | 10.04 | 2500 | 0.4340          | 0.3585 |
| 0.1905        | 12.05 | 3000 | 0.4310          | 0.3404 |
| 0.1545        | 14.06 | 3500 | 0.4547          | 0.3381 |
| 0.1206        | 16.06 | 4000 | 0.4902          | 0.3384 |
| 0.1116        | 18.07 | 4500 | 0.4767          | 0.3253 |
| 0.0925        | 20.08 | 5000 | 0.5248          | 0.3160 |
| 0.0897        | 22.09 | 5500 | 0.4960          | 0.3126 |
| 0.0687        | 24.1  | 6000 | 0.4876          | 0.3086 |
| 0.063         | 26.1  | 6500 | 0.4895          | 0.3065 |
| 0.0558        | 28.11 | 7000 | 0.5127          | 0.3082 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: wav2vec2-base-timit-demo-colab


model author: ying-tina


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab-test
results: []


# wav2vec2-base-timit-demo-colab-test


- Loss: 0.4283
- Wer: 0.3356


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.7386        | 4.0   | 500  | 2.2419          | 1.0    |
| 0.9366        | 8.0   | 1000 | 0.4789          | 0.4807 |
| 0.3118        | 12.0  | 1500 | 0.4197          | 0.3973 |
| 0.1784        | 16.0  | 2000 | 0.4216          | 0.3614 |
| 0.1297        | 20.0  | 2500 | 0.4298          | 0.3507 |
| 0.1091        | 24.0  | 3000 | 0.4365          | 0.3437 |
| 0.0819        | 28.0  | 3500 | 0.4283          | 0.3356 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-base-timit-demo-colab-test


model author: ying-tina


task: automatic-speech-recognition


license: apache-2.0
tags:


- generated_from_trainer
model-index:
name: wav2vec2-base-timit-demo-colab-32


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-base-timit-demo-colab-32


This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4488
- Wer: 0.3149


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.6155        | 4.0   | 500  | 2.2647          | 0.9992 |
| 0.9037        | 8.0   | 1000 | 0.4701          | 0.4336 |
| 0.3159        | 12.0  | 1500 | 0.4247          | 0.3575 |
| 0.1877        | 16.0  | 2000 | 0.4477          | 0.3442 |
| 0.1368        | 20.0  | 2500 | 0.4932          | 0.3384 |
| 0.1062        | 24.0  | 3000 | 0.4758          | 0.3202 |
| 0.0928        | 28.0  | 3500 | 0.4488          | 0.3149 |


### Framework versions


- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Tokenizers 0.10.3


model name: wav2vec2-base-timit-demo-colab-32


model author: ying-tina


task: automatic-speech-recognition


language:


- zh-HK
tags:
- automatic-speech-recognition
license: cc-by-sa-4.0
datasets:
- common_voice
metrics:
- cer


# Wav2vec2-large-xlsr-cantonese


This model was based on [wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53), finetuned using Common Voice/zh-HK/6.1.0.


The training code is similar to [user ctl](https://huggingface.co/ctl/wav2vec2-large-xlsr-cantonese), except that the number of training epochs was 80 (doubled) and fp16_backend is apex. The model was trained using a single RTX 3090 and docker image is nvidia/cuda:11.1-cudnn8-devel.


CER is 15.11% when evaluate against common voice zh-HK test set.


# Result (CER)


# Source Code


See this GitHub Repo [cantonese-selfish-project](https://github.com/scottykwok/cantonese-selfish-project/) and [demo video](https://youtu.be/k_9RQ-ilGEc).


# Usage


```python
import soundfile as sf
import torch
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

# load pretrained model
processor = Wav2Vec2Processor.from_pretrained("scottykwok/wav2vec2-large-xlsr-cantonese")
model = Wav2Vec2ForCTC.from_pretrained("scottykwok/wav2vec2-large-xlsr-cantonese")

# load audio - must be 16kHz mono
audio_input, sample_rate = sf.read('audio.wav')

# pad input values and return pt tensor
input_values = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt").input_values

# INFERENCE
# retrieve logits & take argmax
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)

# transcribe
transcription = processor.decode(predicted_ids[0])
print("-" *20)
print("Transcription:\n", transcription.lower())
print("-" *20)

```


model name: wav2vec2-large-xlsr-cantonese


model author: scottykwok


tags: pytorch, wav2vec2, automatic-speech-recognition, zh-HK, dataset:common_voice, transformers, license:cc-by-sa-4.0


tags:


- generated_from_trainer
datasets:
- null
model_index:
- name: gpt2-wikitext2
results:
  - task:
name: Causal Language Modeling
type: text-generation


# gpt2-wikitext2


This model is a fine-tuned version of [](https://huggingface.co/) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 6.3227


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 6.7523        | 1.0   | 2249 | 6.6652          |
| 6.4134        | 2.0   | 4498 | 6.3987          |
| 6.2507        | 3.0   | 6747 | 6.3227          |


### Framework versions


- Transformers 4.9.1
- Pytorch 1.9.0+cu102
- Datasets 1.10.2
- Tokenizers 0.10.3


model name: gpt2-wikitext2


model author: MYX4567


task: text-generation


tags: pytorch, tensorboard, gpt2, text-generation, transformers, generated_from_trainer


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad
model_index:
- name: distilbert-base-uncased-finetuned-squad
results:
  - task:
name: Question Answering
type: question-answering
dataset:
name: squad
type: squad
args: plain_text


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad dataset.
It achieves the following results on the evaluation set:


- Loss: 1.1520


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.2177        | 1.0   | 5533  | 1.1565          |
| 0.9472        | 2.0   | 11066 | 1.1174          |
| 0.7634        | 3.0   | 16599 | 1.1520          |


### Framework versions


model name: distilbert-base-uncased-finetuned-squad


model author: MYX4567


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, dataset:squad, transformers, license:apache-2.0, generated_from_trainer, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- null
model_index:
- name: distilgpt2-finetuned-wikitext2
results:
  - task:
name: Causal Language Modeling
type: text-generation


# distilgpt2-finetuned-wikitext2


This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 3.6428


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.76          | 1.0   | 2334 | 3.6658          |
| 3.6325        | 2.0   | 4668 | 3.6454          |
| 3.6068        | 3.0   | 7002 | 3.6428          |


### Framework versions


model name: distilgpt2-finetuned-wikitext2


model author: MYX4567


task: text-generation


tags: pytorch, tensorboard, gpt2, text-generation, transformers, license:apache-2.0, generated_from_trainer


tags:


- conversational


# Cartman Southpark DialoGPT2 small 18 epochs


model name: DialoGPT-real-cartman-small


model author: Bee-Garbs


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


# Harry Potter DialoGPT Model


model name: DialoGPT-small-harrypotter


model author: ppn


language: en
thumbnail: https://github.com/jackaduma
tags:


- exbert
- security
- cybersecurity
- cyber security
- threat hunting
- threat intelligence
license: apache-2.0
datasets:
- APTnotes
- Stucco-Data
- CASIE


# SecBERT


This is the pretrained model presented in [SecBERT: A Pretrained Language Model for Cyber Security Text](https://github.com/jackaduma/SecBERT/), which is a BERT model trained on cyber security text.


The training corpus was papers taken from


* [APTnotes](https://github.com/kbandla/APTnotes)
* [Stucco-Data: Cyber security data sources](https://stucco.github.io/data/)
* [CASIE: Extracting Cybersecurity Event Information from Text](https://ebiquity.umbc.edu/_file_directory_/papers/943.pdf)
* [SemEval-2018 Task 8: Semantic Extraction from CybersecUrity REports using Natural Language Processing (SecureNLP)](https://competitions.codalab.org/competitions/17262).


SecBERT has its own wordpiece vocabulary (secvocab) that's built to best match the training corpus.


We trained [SecBERT](https://huggingface.co/jackaduma/SecBERT) and [SecRoBERTa](https://huggingface.co/jackaduma/SecRoBERTa) versions.


Available models include:


* [`SecBERT`](https://huggingface.co/jackaduma/SecBERT)
* [`SecRoBERTa`](https://huggingface.co/jackaduma/SecRoBERTa)


## **Fill Mask**


We proposed to build language model which work on cyber security text, as result, it can improve downstream tasks (NER, Text Classification, Semantic Understand, Q&A) in Cyber Security Domain.


First, as below shows Fill-Mask pipeline in [Google Bert](), [AllenAI SciBert](https://github.com/allenai/scibert) and our [SecBERT](https://github.com/jackaduma/SecBERT) .


<!-- <img src="./fill-mask-result.png" width="150%" height="150%"> -->


The original repo can be found [here](https://github.com/jackaduma/SecBERT).


model name: SecBERT


model author: jackaduma


task: fill-mask


tags: pytorch, bert, fill-mask, en, dataset:APTnotes, dataset:Stucco-Data, dataset:CASIE, transformers, exbert, security, cybersecurity, cyber security, threat hunting, threat intelligence, license:apache-2.0, autonlp_compatible, infinity_compatible


# SecRoBERTa


This is the pretrained model presented in [SecBERT: A Pretrained Language Model for Cyber Security Text](https://github.com/jackaduma/SecBERT/), which is a SecRoBERTa model trained on cyber security text.


SecRoBERTa has its own wordpiece vocabulary (secvocab) that's built to best match the training corpus.


## ![fill-mask-result](https://github.com/jackaduma/SecBERT/blob/main/fill-mask-result.png?raw=true)


model name: SecRoBERTa


tags: pytorch, roberta, fill-mask, en, dataset:APTnotes, dataset:Stucco-Data, dataset:CASIE, transformers, exbert, security, cybersecurity, cyber security, threat hunting, threat intelligence, license:apache-2.0, autonlp_compatible, infinity_compatible


# BibTeX entry and citation info


```
@misc{pandya2021cascading,
      title={Cascading Adaptors to Leverage English Data to Improve Performance of Question Answering for Low-Resource Languages},
      author={Hariom A. Pandya and Bhavik Ardeshna and Dr. Brijesh S. Bhatt},
      year={2021},
      eprint={2112.09866},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: xlm-roberta-base-arabic


model author: bhavikardeshna


task: question-answering


tags: pytorch, xlm-roberta, question-answering, arxiv:2112.09866, transformers, autonlp_compatible


model name: xlm-roberta-base-vietnamese


model name: xlm-roberta-base-hindi


model name: xlm-roberta-base-chinese


model name: multilingual-bert-base-cased-spanish


tags: pytorch, bert, question-answering, arxiv:2112.09866, transformers, autonlp_compatible, infinity_compatible


model name: multilingual-bert-base-cased-english


model name: xlm-roberta-base-german


model name: multilingual-bert-base-cased-chinese


model name: multilingual-bert-base-cased-hindi


model name: xlm-roberta-base-spanish


model name: multilingual-bert-base-cased-arabic


# BibTeX entry and citation info


model name: multilingual-bert-base-cased-vietnamese


model name: multilingual-bert-base-cased-german


model author: bhavikardeshna


task: question-answering


language:


- en
tags:
- summarization
license: apache-2.0
datasets:
- arxiv_dataset
metrics:
- rouge
widget:
- text: "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data."


# Paper Title Generator


Generates titles for computer science papers given an abstract.


The model is a BERT2BERT Encoder-Decoder using the official `bert-base-uncased` checkpoint as initialization for the encoder and decoder.
It was fine-tuned on 318,500 computer science papers posted on arXiv.org between 2007 and 2020 and achieved a 26.3% Rouge2 F1-Score on held-out validation data.


**Live Demo:** [https://paper-titles.ey.r.appspot.com/](https://paper-titles.ey.r.appspot.com/)


model name: bert2bert-base-arxiv-titlegen


model author: Callidior


task: summarization


tags: pytorch, encoder-decoder, text2text-generation, en, dataset:arxiv_dataset, transformers, summarization, license:apache-2.0, autonlp_compatible


# shrugging-grace/tweetclassifier


## Model description


This model classifies tweets as either relating to the Covid-19 pandemic or not.


## Intended uses & limitations


It is intended to be used on tweets commenting on UK politics, in particular those trending with the #PMQs hashtag, as this refers to weekly Prime Ministers' Questions.


#### How to use


`LABEL_0` means that the tweet relates to Covid-19


`LABEL_1` means that the tweet does not relate to Covid-19


## Training data


The model was trained on 1000 tweets (with the "#PMQs'), which were manually labeled by the author. The tweets were collected between May-July 2020.


### BibTeX entry and citation info


This was based on a pretrained version of BERT.


@article{devlin2018bert,
title={Bert: Pre-training of deep bidirectional transformers for language understanding},
author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
journal={arXiv preprint arXiv:1810.04805},
year={2018}
}


model name: tweetclassifier


model author: shrugging-grace


task: text-classification


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


tags:


- conversational


# Harry Potter DialoGPT Model


model name: DialoGPT-small-harrypotter


model author: Kush


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


model name: argentina


model author: rafio


# Pangu-Alpha 2.6B


## Model Description


PanGu-α is proposed by a joint technical team headed by PCNL. It is the first large-scale Chinese pre-trained language model with 200 billion parameters trained on 2048 Ascend processors using an automatic hybrid parallel training strategy. The whole training process is done on the “Peng Cheng Cloud Brain II” computing platform with the domestic deep learning framework called MindSpore. The PengCheng·PanGu-α pre-training model can support rich applications, has strong few-shot learning capabilities, and has outstanding performance in text generation tasks such as knowledge question and answer, knowledge retrieval, knowledge reasoning, and reading comprehension.


This repository contains PyTorch implementation of PanGu model, with
2.6 billion parameters pretrained weights (FP32 precision), converted from original MindSpore checkpoint.


## Usage (Text Generation)


Currently PanGu model is not supported by transformers,
so `trust_remote_code=True` is required to load model implementation in this repo.


```python
from transformers import TextGenerationPipeline, AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("imone/pangu_2_6B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("imone/pangu_2_6B", trust_remote_code=True)

text_generator = TextGenerationPipeline(model, tokenizer)
# greedy search
print(text_generator("中国和美国和日本和法国和加拿大和澳大利亚的首都分别是哪里？", max_length=50))
```


Expected output:


```python
[{'generated_text': '中国和美国和日本和法国和加拿大和澳大利亚的首都分别是哪里？\n中国北京,美国华盛顿,日本东京,法国巴黎,加拿大多伦多,澳大利亚悉尼,新西兰奥克兰,澳大利亚墨尔本,新西兰奥克兰,'}]
```


model name: pangu_2_6B


model author: imone


task: text-generation


tags: pytorch, gpt_pangu, text-generation, transformers


# measurement_time


language: en
datasets:


- measurement_time


This is a [t5-small](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) fine-tuned version on the [math_dataset/measurement_time](https://www.tensorflow.org/datasets/catalog/math_dataset#mathdatasetmeasurement_time) for solving **measurement time equations** mission.


To load the model:
(necessary packages: !pip install transformers sentencepiece)


```python
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("dbernsohn/t5_measurement_time")
model = AutoModelWithLMHead.from_pretrained("dbernsohn/t5_measurement_time")
```


You can then use this model to solve algebra 1d equations into numbers.


```python
query = "How many minutes are there between 2:09 PM and 2:27 PM?"
input_text = f"{query} </s>"
features = tokenizer([input_text], return_tensors='pt')
model.to('cuda')
output = model.generate(input_ids=features['input_ids'].cuda(),
                        attention_mask=features['attention_mask'].cuda())

tokenizer.decode(output[0])
# <pad> 18</s>
```


Another examples:


+ How many minutes are there between 2:09 PM and 2:27 PM?
+ Answer:  18 Pred:  18


+ What is 116 minutes after 10:06 AM?
+ Answer:  12:02 PM Pred:  12:02 PM


+ What is 608 minutes after 3:14 PM?
+ Answer:  1:22 AM Pred:  1:22 AM


+ What is 64 minutes before 9:16 AM?
+ Answer:  8:12 AM Pred:  8:12 AM


+ What is 427 minutes before 4:27 AM?
+ Answer:  9:20 PM Pred:  9:20 PM


+ How many minutes are there between 6:36 PM and 12:15 AM?
+ Answer:  339 Pred:  339


+ What is 554 minutes before 5:24 PM?
+ Answer:  8:10 AM Pred:  8:10 AM


+ What is 307 minutes after 5:15 AM?
+ Answer:  10:22 AM Pred:  10:22 AM


The whole training process and hyperparameters are in my [GitHub repo](https://github.com/DorBernsohn/CodeLM/tree/main/MathLM)


> Created by [Dor Bernsohn](https://www.linkedin.com/in/dor-bernsohn-70b2b1146/)


model name: t5_measurement_time


model author: dbernsohn


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:measurement_time, transformers, autonlp_compatible


# roberta-java


language: Java
datasets:


- code_search_net


This is a [roberta](https://arxiv.org/pdf/1907.11692.pdf) pre-trained version on the [CodeSearchNet dataset](https://github.com/github/CodeSearchNet) for **Java** Mask Language Model mission.


```python
from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline
tokenizer = AutoTokenizer.from_pretrained("dbernsohn/roberta-java")
model = AutoModelWithLMHead.from_pretrained("dbernsohn/roberta-java")

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
)
```


You can then use this model to fill masked words in a Java code.


```python
code = """
String[] cars = {"Volvo", "BMW", "Ford", "Mazda"};
for (String i : cars) {
System.out.<mask>(i);
}
""".lstrip()

pred = {x["token_str"].replace("Ġ", ""): x["score"] for x in fill_mask(code)}
sorted(pred.items(), key=lambda kv: kv[1], reverse=True)
# [('println', 0.32571351528167725),
# ('get', 0.2897663116455078),
# ('remove', 0.0637081190943718),
# ('exit', 0.058875661343336105),
# ('print', 0.034190207719802856)]
```


The whole training process and hyperparameters are in my [GitHub repo](https://github.com/DorBernsohn/CodeLM/tree/main/CodeMLM)


model name: roberta-java


task: fill-mask


tags: pytorch, jax, roberta, fill-mask, Java, dataset:code_search_net, arxiv:1907.11692, transformers, autonlp_compatible, infinity_compatible


# roberta-go


language: Go
datasets:


This is a [roberta](https://arxiv.org/pdf/1907.11692.pdf) pre-trained version on the [CodeSearchNet dataset](https://github.com/github/CodeSearchNet) for **Golang** Mask Language Model mission.


To load the model:
(necessary packages: !pip install transformers sentencepiece)


```python
from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline
tokenizer = AutoTokenizer.from_pretrained("dbernsohn/roberta-go")
model = AutoModelWithLMHead.from_pretrained("dbernsohn/roberta-go")

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
)
```


```python
code = """
package main

import (
	"fmt"
	"runtime"
)

func main() {
	fmt.Print("Go runs on ")
	switch os := runtime.<mask>; os {
	case "darwin":
		fmt.Println("OS X.")
	case "linux":
		fmt.Println("Linux.")
	default:
		// freebsd, openbsd,
		// plan9, windows...
		fmt.Printf("%s.\n", os)
	}
}
""".lstrip()

pred = {x["token_str"].replace("Ġ", ""): x["score"] for x in fill_mask(code)}
sorted(pred.items(), key=lambda kv: kv[1], reverse=True)
[('GOOS', 0.11810332536697388),
 ('FileInfo', 0.04276798665523529),
 ('Stdout', 0.03572738170623779),
 ('Getenv', 0.025064032524824142),
 ('FileMode', 0.01462600938975811)]
```


model name: roberta-go


model author: dbernsohn


tags: pytorch, jax, roberta, fill-mask, Go, dataset:code_search_net, arxiv:1907.11692, transformers, autonlp_compatible, infinity_compatible


# t5_wikisql_SQL2en


language: en
datasets:


- wikisql


This is a [t5-small](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) fine-tuned version on the [wikisql dataset](https://huggingface.co/datasets/wikisql) for **SQL** to **English** **translation** text2text mission.


```python
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("dbernsohn/t5_wikisql_SQL2en")
model = AutoModelWithLMHead.from_pretrained("dbernsohn/t5_wikisql_SQL2en")
```


You can then use this model to translate SQL queries into plain english.


```python
query = "SELECT people FROM peoples where age > 10"
input_text = f"translate SQL to English: {query} </s>"
features = tokenizer([input_text], return_tensors='pt')

output = model.generate(input_ids=features['input_ids'].cuda(),
                        attention_mask=features['attention_mask'].cuda())

tokenizer.decode(output[0])
# Output: "What people are older than 10?"
```


The whole training process and hyperparameters are in my [GitHub repo](https://github.com/DorBernsohn/CodeLM/tree/main/SQLM)


> Created by [Dor Bernsohn](https://www.linkedin.com/in/dor-bernsohn-70b2b1146/)


model name: t5_wikisql_SQL2en


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:wikisql, transformers, autonlp_compatible


# roberta-python


language: python
datasets:


- code_search_net


This is a [roberta](https://arxiv.org/pdf/1907.11692.pdf) pre-trained version on the [CodeSearchNet dataset](https://github.com/github/CodeSearchNet) for **Python** Mask Language Model mission.


```python
from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline
tokenizer = AutoTokenizer.from_pretrained("dbernsohn/roberta-python")
model = AutoModelWithLMHead.from_pretrained("dbernsohn/roberta-python")

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
)
```


You can then use this model to fill masked words in a Python code.


```python
code = """
new_dict = {}
for k, v in my_dict.<mask>():
    new_dict[k] = v**2
""".lstrip()

pred = {x["token_str"].replace("Ġ", ""): x["score"] for x in fill_mask(code)}
sorted(pred.items(), key=lambda kv: kv[1], reverse=True)
# [('items', 0.7376779913902283),
# ('keys', 0.16238391399383545),
# ('values', 0.03965481370687485),
# ('iteritems', 0.03346433863043785),
# ('splitlines', 0.0032723243348300457)]
```


The whole training process and hyperparameters are in my [GitHub repo](https://github.com/DorBernsohn/CodeLM/tree/main/CodeMLM)


model name: roberta-python


model author: dbernsohn


task: fill-mask


tags: pytorch, jax, roberta, fill-mask, python, dataset:code_search_net, arxiv:1907.11692, transformers, autonlp_compatible, infinity_compatible


# roberta-php


language: php
datasets:


This is a [roberta](https://arxiv.org/pdf/1907.11692.pdf) pre-trained version on the [CodeSearchNet dataset](https://github.com/github/CodeSearchNet) for **php** Mask Language Model mission.


To load the model:
(necessary packages: !pip install transformers sentencepiece)


```python
from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline
tokenizer = AutoTokenizer.from_pretrained("dbernsohn/roberta-php")
model = AutoModelWithLMHead.from_pretrained("dbernsohn/roberta-php")

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
)
```


You can then use this model to fill masked words in a Java code.


```python
code = """
$people = array(
    array('name' => 'Kalle', 'salt' => 856412),
    array('name' => 'Pierre', 'salt' => 215863)
);

for($i = 0; $i < count($<mask>); ++$i) {
    $people[$i]['salt'] = mt_rand(000000, 999999);
}
""".lstrip()

pred = {x["token_str"].replace("Ġ", ""): x["score"] for x in fill_mask(code)}
sorted(pred.items(), key=lambda kv: kv[1], reverse=True)
# [('people', 0.785636842250824),
#  ('parts', 0.006270722020417452),
#  ('id', 0.0035842324141412973),
#  ('data', 0.0025512021966278553),
#  ('config', 0.002258970635011792)]
```


model name: roberta-php


tags: pytorch, jax, roberta, fill-mask, php, dataset:code_search_net, arxiv:1907.11692, transformers, autonlp_compatible, infinity_compatible


# algebra_linear_1d_composed


language: en
datasets:


- algebra_linear_1d_composed


This is a [t5-small](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) fine-tuned version on the [math_dataset/algebra_linear_1d_composed](https://www.tensorflow.org/datasets/catalog/math_dataset#mathdatasetalgebra_linear_1d_composed) for solving **algebra linear 1d composed equations** mission.


```python
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("dbernsohn/algebra_linear_1d_composed")
model = AutoModelWithLMHead.from_pretrained("dbernsohn/algebra_linear_1d_composed")
```


You can then use this model to solve algebra 1d equations into numbers.


```python
query = "Suppose -d = 5 - 16. Let b = -579 + 584. Solve -b*c + 36 = d for c."
input_text = f"{query} </s>"
features = tokenizer([input_text], return_tensors='pt')
model.to('cuda')
output = model.generate(input_ids=features['input_ids'].cuda(),
                        attention_mask=features['attention_mask'].cuda())

tokenizer.decode(output[0])
# <pad> 5</s>
```


Another examples:


+ Suppose -d = 5 - 16. Let b = -579 + 584. Solve -b*c + 36 = d for c.
+ Answer:  5 Pred:  5


+ Suppose 3*v - l + 9 = 4*v, 0 = -5*v + 5*l - 5. Let f(s) = 3*s**2 + 1. Let g be f(-1). Suppose 63 = g*x - x. Solve -5*i + v + x = 0 for i.
+ Answer:  5 Pred:  5


+ Let w be 2 - (0 - 0)/(-2). Let f = -110 - -110. Suppose f*m - 4*m + 3*m = 0. Solve m*v = -w*v for v.
+ Answer:  0 Pred:  0


+ Let a(h) = -34*h**3 - 15 + 3*h + 36*h**3 + 8*h**2 + 5*h**2. Let r be a(-6). Solve 2*z = r*z for z.
+ Answer:  0 Pred:  0


+ Suppose -3*p + 24 = -3*c, 0*c + 6 = -2*c. Suppose -67 = 4*i + 289. Let t = i + 94. Solve t = 2*y - p for y.
+ Answer:  5 Pred:  5


+ Let b = -36 + 53. Suppose -7*u - b = -73. Solve j + 3*j = -u for j.
+ Answer:  -2 Pred:  -2


+ Let h be 8*((-2)/2 + 14)*1. Let y = -101 + h. Solve y*p = -p for p.
+ Answer:  0 Pred:  0


+ Let b = 178 - 79. Let s be 9/(-1 - 2 - b/(-22)). Solve s = -k - k for k.
+ Answer:  -3 Pred:  -3


+ Suppose 31 = -4*z + 11, -3*k - 5*z - 22 = 0. Solve 23 = -11*p + k for p.
+ Answer:  -2 Pred:  -2


The whole training process and hyperparameters are in my [GitHub repo](https://github.com/DorBernsohn/CodeLM/tree/main/MathLM)


> Created by [Dor Bernsohn](https://www.linkedin.com/in/dor-bernsohn-70b2b1146/)


model name: algebra_linear_1d_composed


model author: dbernsohn


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:algebra_linear_1d_composed, transformers, autonlp_compatible


# t5_wikisql_en2SQL


language: en
datasets:


- wikisql


This is a [t5-small](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) fine-tuned version on the [wikisql dataset](https://huggingface.co/datasets/wikisql) for **English** to **SQL** **translation** text2text mission.