```python
import torch
from transformers import BertForMaskedLM as WoBertForMaskedLM
from wobert import WoBertTokenizer

pretrained_model_or_path_list = [
    "junnyu/wobert_chinese_plus_base", "junnyu/wobert_chinese_base"
]
for path in pretrained_model_or_path_list:
    text = "今天[MASK]很好，我[MASK]去公园玩。"
    tokenizer = WoBertTokenizer.from_pretrained(path)
    model = WoBertForMaskedLM.from_pretrained(path)
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs).logits[0]
    outputs_sentence = ""
    for i, id in enumerate(tokenizer.encode(text)):
        if id == tokenizer.mask_token_id:
            tokens = tokenizer.convert_ids_to_tokens(outputs[i].topk(k=5)[1])
            outputs_sentence += "[" + "||".join(tokens) + "]"
        else:
            outputs_sentence += "".join(
                tokenizer.convert_ids_to_tokens([id],
                                                skip_special_tokens=True))
    print(outputs_sentence)
# RoFormer    今天[天气||天||心情||阳光||空气]很好，我[想||要||打算||准备||喜欢]去公园玩。
# PLUS WoBERT 今天[天气||阳光||天||心情||空气]很好，我[想||要||打算||准备||就]去公园玩。
# WoBERT      今天[天气||阳光||天||心情||空气]很好，我[想||要||就||准备||也]去公园玩。
```


## 引用


Bibtex：


```tex
@techreport{zhuiyiwobert,
  title={WoBERT: Word-based Chinese BERT model - ZhuiyiAI},
  author={Jianlin Su},
  year={2020},
  url="https://github.com/ZhuiyiTechnology/WoBERT",
}
```


model name: wobert_chinese_plus_base


model author: junnyu


task: fill-mask


tags: pytorch, jax, bert, fill-mask, zh, transformers, wobert, autonlp_compatible, infinity_compatible


language: en
thumbnail: https://github.com/junnyu
tags:


- pytorch
- electra
- roformer
- rotary position embedding
license: mit
datasets:
- openwebtext


# 一、 个人在openwebtext数据集上添加rotary-position-embedding，训练得到的electra-small模型


# 二、 复现结果(dev dataset)


# 三、 训练细节


- 数据集 openwebtext
- 训练batch_size 256
- 学习率lr  5e-4
- 最大句子长度max_seqlen  128
- 训练total step  50W
- GPU RTX3090
- 训练时间总共耗费55h


# 四、wandb日志


- [**预训练日志**](https://wandb.ai/junyu/electra_rotary_small_pretrain?workspace=user-junyu)
- [**GLUE微调日志**](https://wandb.ai/junyu/electra_rotary_glue_100?workspace=user-junyu)


# 五、 使用


```python
import torch
from transformers import ElectraTokenizer,RoFormerModel
tokenizer = ElectraTokenizer.from_pretrained("junnyu/roformer_small_discriminator")
model = RoFormerModel.from_pretrained("junnyu/roformer_small_discriminator")
inputs = tokenizer("Beijing is the capital of China.", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
    print(outputs[0].shape)
```


model name: roformer_small_discriminator


task: feature-extraction


tags: pytorch, roformer, feature-extraction, en, dataset:openwebtext, transformers, electra, rotary position embedding, license:mit


language: zh
tags:


- roformer
- pytorch
- tf2.0
widget:
- text: "今天[MASK]很好，我想去公园玩！"


## 介绍


### tf版本


### pytorch版本+tf2.0版本


## pytorch使用


```python
import torch
from transformers import RoFormerForMaskedLM, RoFormerTokenizer

text = "今天[MASK]很好，我[MASK]去公园玩。"
tokenizer = RoFormerTokenizer.from_pretrained("junnyu/roformer_chinese_small")
pt_model = RoFormerForMaskedLM.from_pretrained("junnyu/roformer_chinese_small")
pt_inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    pt_outputs = pt_model(**pt_inputs).logits[0]
pt_outputs_sentence = "pytorch: "
for i, id in enumerate(tokenizer.encode(text)):
    if id == tokenizer.mask_token_id:
        tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k=5)[1])
        pt_outputs_sentence += "[" + "||".join(tokens) + "]"
    else:
        pt_outputs_sentence += "".join(
            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True))
print(pt_outputs_sentence)
# pytorch: 今天[天气||心情||感觉||环境||下午]很好，我[要||想||就||可以||去]去公园玩。
```


## tensorflow2.0使用


```python
import tensorflow as tf
from transformers import RoFormerTokenizer, TFRoFormerForMaskedLM
text = "今天[MASK]很好，我[MASK]去公园玩。"
tokenizer = RoFormerTokenizer.from_pretrained("junnyu/roformer_chinese_small")
tf_model = TFRoFormerForMaskedLM.from_pretrained("junnyu/roformer_chinese_small")
tf_inputs = tokenizer(text, return_tensors="tf")
tf_outputs = tf_model(**tf_inputs, training=False).logits[0]
tf_outputs_sentence = "tf2.0: "
for i, id in enumerate(tokenizer.encode(text)):
    if id == tokenizer.mask_token_id:
        tokens = tokenizer.convert_ids_to_tokens(
            tf.math.top_k(tf_outputs[i], k=5)[1])
        tf_outputs_sentence += "[" + "||".join(tokens) + "]"
    else:
        tf_outputs_sentence += "".join(
            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True))
print(tf_outputs_sentence)
# tf2.0 今天[天气||心情||感觉||环境||下午]很好，我[要||想||就||可以||去]去公园玩。
```


## 引用


Bibtex：


```tex
@misc{su2021roformer,
      title={RoFormer: Enhanced Transformer with Rotary Position Embedding},
      author={Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu},
      year={2021},
      eprint={2104.09864},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: roformer_chinese_small


model author: junnyu


task: fill-mask


tags: pytorch, tf, jax, roformer, fill-mask, zh, arxiv:2104.09864, transformers, tf2.0, autonlp_compatible


```python
import torch
from transformers import RoFormerForMaskedLM, RoFormerTokenizer

text = "今天[MASK]很好，我[MASK]去公园玩。"
tokenizer = RoFormerTokenizer.from_pretrained("junnyu/roformer_chinese_char_small")
pt_model = RoFormerForMaskedLM.from_pretrained("junnyu/roformer_chinese_char_small")
pt_inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    pt_outputs = pt_model(**pt_inputs).logits[0]
pt_outputs_sentence = "pytorch: "
for i, id in enumerate(tokenizer.encode(text)):
    if id == tokenizer.mask_token_id:
        tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k=5)[1])
        pt_outputs_sentence += "[" + "||".join(tokens) + "]"
    else:
        pt_outputs_sentence += "".join(
            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True))
print(pt_outputs_sentence)
# pytorch: 今天[也||都||又||还||我]很好，我[就||想||去||也||又]去公园玩。
```


```python
import tensorflow as tf
from transformers import RoFormerTokenizer, TFRoFormerForMaskedLM
text = "今天[MASK]很好，我[MASK]去公园玩。"
tokenizer = RoFormerTokenizer.from_pretrained("junnyu/roformer_chinese_char_small")
tf_model = TFRoFormerForMaskedLM.from_pretrained("junnyu/roformer_chinese_char_small")
tf_inputs = tokenizer(text, return_tensors="tf")
tf_outputs = tf_model(**tf_inputs, training=False).logits[0]
tf_outputs_sentence = "tf2.0: "
for i, id in enumerate(tokenizer.encode(text)):
    if id == tokenizer.mask_token_id:
        tokens = tokenizer.convert_ids_to_tokens(
            tf.math.top_k(tf_outputs[i], k=5)[1])
        tf_outputs_sentence += "[" + "||".join(tokens) + "]"
    else:
        tf_outputs_sentence += "".join(
            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True))
print(tf_outputs_sentence)
# tf2.0: 今天[也||都||又||还||我]很好，我[就||想||去||也||又]去公园玩。
```


model name: roformer_chinese_char_small


language: zh
tags:


- glycebert
inference: False


# https://github.com/JunnYu/ChineseBert_pytorch


# ChineseBert_pytorch


本项目主要自定义了tokenization_chinesebert_fast.py文件中的ChineseBertTokenizerFast代码。从而可以从huggingface.co调用。


```python
pretrained_tokenizer_name = "junnyu/ChineseBERT-base"
tokenizer = ChineseBertTokenizerFast.from_pretrained(pretrained_tokenizer_name)
```


# Paper


**[ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://arxiv.org/pdf/2106.16038.pdf)**\
*Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu and Jiwei Li*


# Install


```bash
pip install chinesebert
or
pip install git+https://github.com/JunnYu/ChineseBert_pytorch.git
```


# Usage


```python
import torch
from chinesebert import ChineseBertForMaskedLM, ChineseBertTokenizerFast, ChineseBertConfig

pretrained_tokenizer_name = "junnyu/ChineseBERT-base"
pretrained_model_name = "ShannonAI/ChineseBERT-base"

tokenizer = ChineseBertTokenizerFast.from_pretrained(pretrained_tokenizer_name)
config = ChineseBertConfig.from_pretrained(pretrained_tokenizer_name)
chinese_bert = ChineseBertForMaskedLM.from_pretrained(pretrained_model_name, config=config)

text = "北京是[MASK]国的首都。"
inputs = tokenizer(text, return_tensors="pt")
print(inputs)
maskpos = 4

with torch.no_grad():
    o = chinese_bert(**inputs)
    value, index = o.logits.softmax(-1)[0, maskpos].topk(10)

pred_tokens = tokenizer.convert_ids_to_tokens(index.tolist())
pred_values = value.tolist()

outputs = []
for t, p in zip(pred_tokens, pred_values):
    outputs.append(f"{t}|{round(p,4)}")
print(outputs)

# base  ['中|0.711', '我|0.2488', '祖|0.016', '法|0.0057', '美|0.0048', '全|0.0042', '韩|0.0015', '英|0.0011', '两|0.0008', '王|0.0006']
# large ['中|0.8341', '我|0.1479', '祖|0.0157', '全|0.0007', '国|0.0005', '帝|0.0001', '该|0.0001', '法|0.0001', '一|0.0001', '咱|0.0001']
```


# Reference


model name: ChineseBERT-base


model author: junnyu


task: fill-mask


tags: bert, fill-mask, zh, arxiv:2106.16038, transformers, glycebert, autonlp_compatible, infinity_compatible


language: zh
tags:


- glycebert
inference: False


```python
import torch
from chinesebert import ChineseBertForMaskedLM, ChineseBertTokenizerFast, ChineseBertConfig

pretrained_tokenizer_name = "junnyu/ChineseBERT-large"
pretrained_model_name = "ShannonAI/ChineseBERT-large"

tokenizer = ChineseBertTokenizerFast.from_pretrained(pretrained_tokenizer_name)
config = ChineseBertConfig.from_pretrained(pretrained_tokenizer_name)
chinese_bert = ChineseBertForMaskedLM.from_pretrained(pretrained_model_name, config=config)

text = "北京是[MASK]国的首都。"
inputs = tokenizer(text, return_tensors="pt")
print(inputs)
maskpos = 4

with torch.no_grad():
    o = chinese_bert(**inputs)
    value, index = o.logits.softmax(-1)[0, maskpos].topk(10)

pred_tokens = tokenizer.convert_ids_to_tokens(index.tolist())
pred_values = value.tolist()

outputs = []
for t, p in zip(pred_tokens, pred_values):
    outputs.append(f"{t}|{round(p,4)}")
print(outputs)

# base  ['中|0.711', '我|0.2488', '祖|0.016', '法|0.0057', '美|0.0048', '全|0.0042', '韩|0.0015', '英|0.0011', '两|0.0008', '王|0.0006']
# large ['中|0.8341', '我|0.1479', '祖|0.0157', '全|0.0007', '国|0.0005', '帝|0.0001', '该|0.0001', '法|0.0001', '一|0.0001', '咱|0.0001']
```


model name: ChineseBERT-large


- roformer
- pytorch
- tf2.0
- paddlepaddle
widget:
- text: "今天[MASK]很好，我想去公园玩！"


## 介绍


在13g的cluecorpussmall数据集上进行的预训练，使用了`Whole Mask LM` 和 `SOP` 任务


训练逻辑参考了这里。https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/ernie-1.0


## 训练细节：


- paddlepaddle+paddlenlp
- V100 x 4
- batch size 256
- max_seq_len 512
- max_lr 0.0001
- min_lr 0.00001
- weight_decay 0.01
- grad_clip 1.0
- 总共训练的句子`128*30w + 256*15w + 256*14.5w + 256*46.5w + 256*17w = 27648w`
- 约等于512 batch size, 100w步条件下的54%


最终loss：


```python
[2022-02-05 16:05:59,067] [    INFO] - global step 170100, loss: 2.651634932, lm_loss: 2.603405, sop_loss: 0.048229, speed: 1.06 steps/s, ips: 271.68 seqs/s, learning rate: 6.66465e-05, loss_scaling: 137438.96875, num_good_steps: 356, num_bad_steps: 0
[2022-02-05 16:07:28,227] [    INFO] - global step 170200, loss: 2.822231531, lm_loss: 2.662831, sop_loss: 0.159401, speed: 1.12 steps/s, ips: 287.13 seqs/s, learning rate: 6.66263e-05, loss_scaling: 137438.96875, num_good_steps: 59, num_bad_steps: 0
[2022-02-05 16:08:57,346] [    INFO] - global step 170300, loss: 2.710968971, lm_loss: 2.673646, sop_loss: 0.037323, speed: 1.12 steps/s, ips: 287.26 seqs/s, learning rate: 6.66061e-05, loss_scaling: 137438.96875, num_good_steps: 159, num_bad_steps: 0
[2022-02-05 16:10:26,698] [    INFO] - global step 170400, loss: 2.867662907, lm_loss: 2.619032, sop_loss: 0.248631, speed: 1.12 steps/s, ips: 286.51 seqs/s, learning rate: 6.65859e-05, loss_scaling: 137438.96875, num_good_steps: 259, num_bad_steps: 0
[2022-02-05 16:11:55,714] [    INFO] - global step 170500, loss: 3.158756495, lm_loss: 2.953678, sop_loss: 0.205079, speed: 1.12 steps/s, ips: 287.59 seqs/s, learning rate: 6.65657e-05, loss_scaling: 137438.96875, num_good_steps: 359, num_bad_steps: 0
[2022-02-05 16:13:24,869] [    INFO] - global step 170600, loss: 2.860815048, lm_loss: 2.754750, sop_loss: 0.106064, speed: 1.12 steps/s, ips: 287.14 seqs/s, learning rate: 6.65455e-05, loss_scaling: 137438.96875, num_good_steps: 33, num_bad_steps: 0
```


### tf版本


### pytorch版本+tf2.0版本


## pytorch使用


```python
import torch
from transformers import RoFormerForMaskedLM, BertTokenizer

text = "今天[MASK]很好，我[MASK]去公园玩。"
tokenizer = BertTokenizer.from_pretrained("junnyu/roformer_base_wwm_cluecorpussmall")
pt_model = RoFormerForMaskedLM.from_pretrained("junnyu/roformer_base_wwm_cluecorpussmall")
pt_inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    pt_outputs = pt_model(**pt_inputs).logits[0]
pt_outputs_sentence = "pytorch: "
for i, id in enumerate(tokenizer.encode(text)):
    if id == tokenizer.mask_token_id:
        tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k=5)[1])
        pt_outputs_sentence += "[" + "||".join(tokens) + "]"
    else:
        pt_outputs_sentence += "".join(
            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True))
print(pt_outputs_sentence)
# pytorch: 今天[天||人||气||阳||雨]很好，我[想||就||要||也||还]去公园玩。

```


## 引用


Bibtex：


```tex
@misc{su2021roformer,
      title={RoFormer: Enhanced Transformer with Rotary Position Embedding},
      author={Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu},
      year={2021},
      eprint={2104.09864},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: roformer_base_wwm_cluecorpussmall


model author: junnyu


task: fill-mask


tags: pytorch, roformer, fill-mask, zh, arxiv:2104.09864, transformers, tf2.0, paddlepaddle, autonlp_compatible


language: zh
tags:


- roformer
- pytorch
- tf2.0
widget:
- text: "今天[MASK]很好，我想去公园玩！"


## 介绍


```python
import torch
from transformers import RoFormerForMaskedLM, RoFormerTokenizer

text = "今天[MASK]很好，我想去公园玩！"
tokenizer = RoFormerTokenizer.from_pretrained("junnyu/roformer_chinese_base")
pt_model = RoFormerForMaskedLM.from_pretrained("junnyu/roformer_chinese_base")
pt_inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    pt_outputs = pt_model(**pt_inputs).logits[0]
pt_outputs_sentence = "pytorch: "
for i, id in enumerate(tokenizer.encode(text)):
    if id == tokenizer.mask_token_id:
        tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k=5)[1])
        pt_outputs_sentence += "[" + "||".join(tokens) + "]"
    else:
        pt_outputs_sentence += "".join(
            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True))
print(pt_outputs_sentence)
# pytorch: 今天[天气||天||阳光||太阳||空气]很好，我想去公园玩！
```


## tensorflow2.0使用


```python
import tensorflow as tf
from transformers import RoFormerTokenizer, TFRoFormerForMaskedLM
text = "今天[MASK]很好，我想去公园玩！"
tokenizer = RoFormerTokenizer.from_pretrained("junnyu/roformer_chinese_base")
tf_model = TFRoFormerForMaskedLM.from_pretrained("junnyu/roformer_chinese_base")
tf_inputs = tokenizer(text, return_tensors="tf")
tf_outputs = tf_model(**tf_inputs, training=False).logits[0]
tf_outputs_sentence = "tf2.0: "
for i, id in enumerate(tokenizer.encode(text)):
    if id == tokenizer.mask_token_id:
        tokens = tokenizer.convert_ids_to_tokens(
            tf.math.top_k(tf_outputs[i], k=5)[1])
        tf_outputs_sentence += "[" + "||".join(tokens) + "]"
    else:
        tf_outputs_sentence += "".join(
            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True))
print(tf_outputs_sentence)
# tf2.0: 今天[天气||天||阳光||太阳||空气]很好，我想去公园玩！
```


model name: roformer_chinese_base


tags: pytorch, tf, jax, roformer, fill-mask, zh, arxiv:2104.09864, transformers, tf2.0, autonlp_compatible


### tf版本


### pytorch版本+tf2.0版本


model name: roformer_chinese_sim_char_base


tags: pytorch, roformer, fill-mask, zh, transformers, tf2.0, autonlp_compatible


# paddle paddle版本的RoFormer


# 需要安装最新的paddlenlp


`pip install git+https://github.com/PaddlePaddle/PaddleNLP.git`


## 预训练模型转换


预训练模型可以从 huggingface/transformers 转换而来，方法如下（适用于roformer模型，其他模型按情况调整）：


1. 从huggingface.co获取roformer模型权重
1. 设置参数运行convert.py代码
1. 例子：
假设我想转换https://huggingface.co/junnyu/roformer_chinese_base 权重
   - (1)首先下载 https://huggingface.co/junnyu/roformer_chinese_base/tree/main 中的pytorch_model.bin文件,假设我们存入了`./roformer_chinese_base/pytorch_model.bin`
   - (2)运行convert.py
     ```bash
     python convert.py \
         --pytorch_checkpoint_path ./roformer_chinese_base/pytorch_model.bin \
         --paddle_dump_path ./roformer_chinese_base/model_state.pdparams
     ```
   - (3)最终我们得到了转化好的权重`./roformer_chinese_base/model_state.pdparams`


## 预训练MLM测试


### test_mlm.py


```python
import paddle
import argparse
from paddlenlp.transformers import RoFormerForPretraining, RoFormerTokenizer

def test_mlm(text, model_name):
    model = RoFormerForPretraining.from_pretrained(model_name)
    model.eval()
    tokenizer = RoFormerTokenizer.from_pretrained(model_name)
    tokens = ["[CLS]"]
    text_list = text.split("[MASK]")
    for i,t in enumerate(text_list):
        tokens.extend(tokenizer.tokenize(t))
        if i==len(text_list)-1:
            tokens.extend(["[SEP]"])
        else:
            tokens.extend(["[MASK]"])

    input_ids_list = tokenizer.convert_tokens_to_ids(tokens)
    input_ids = paddle.to_tensor([input_ids_list])

    with paddle.no_grad():
        pd_outputs = model(input_ids)[0][0]
    pd_outputs_sentence = "paddle: "
    for i, id in enumerate(input_ids_list):
        if id == tokenizer.convert_tokens_to_ids(["[MASK]"])[0]:
            tokens = tokenizer.convert_ids_to_tokens(pd_outputs[i].topk(5)[1].tolist())
            pd_outputs_sentence += "[" + "||".join(tokens) + "]"
        else:
            pd_outputs_sentence += "".join(
                tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)
            )
    print(pd_outputs_sentence)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_name", default="roformer-chinese-base", type=str, help="Pretrained roformer name or path."
    )
    parser.add_argument(
        "--text", default="今天[MASK]很好，我想去公园玩！", type=str, help="MLM text."
    )
    args = parser.parse_args()
    test_mlm(text=args.text, model_name=args.model_name)

```


### 输出


```bash
python test_mlm.py --model_name roformer-chinese-base --text 今天[MASK]很好，我想去公园玩！
# paddle: 今天[天气||天||阳光||太阳||空气]很好，我想去公园玩！
python test_mlm.py --model_name roformer-chinese-base --text 北京是[MASK]的首都！
# paddle: 北京是[中国||谁||中华人民共和国||我们||中华民族]的首都！
python test_mlm.py --model_name roformer-chinese-char-base --text 今天[MASK]很好，我想去公园玩！
# paddle: 今天[天||气||都||风||人]很好，我想去公园玩！
python test_mlm.py --model_name roformer-chinese-char-base --text 北京是[MASK]的首都！
# paddle: 北京是[谁||我||你||他||国]的首都！
```


model name: roformer_paddle


model author: junnyu


tags:


language: zh
tags:


## 介绍


### tf版本


### pytorch版本+tf2.0版本


model name: roformer_chinese_sim_char_small


task: fill-mask


language: el
pipeline_tag: fill-mask
thumbnail: https://github.com/nlpaueb/GreekBERT/raw/master/greek-bert-logo.png
widget:


- text: "Σήμερα είναι μια [MASK] μέρα."


# GreekBERT


A Greek version of BERT pre-trained language model.


<img src="https://github.com/nlpaueb/GreekBERT/raw/master/greek-bert-logo.png" width="600"/>


## Pre-training corpora


The pre-training corpora of `bert-base-greek-uncased-v1` include:


* The Greek part of [Wikipedia](https://el.wikipedia.org/wiki/Βικιπαίδεια:Αντίγραφα_της_βάσης_δεδομένων),
* The Greek part of [European Parliament Proceedings Parallel Corpus](https://www.statmt.org/europarl/), and
* The Greek part of [OSCAR](https://traces1.inria.fr/oscar/), a cleansed version of [Common Crawl](https://commoncrawl.org).


Future release will also include:


* The entire corpus of Greek legislation, as published by the [National Publication Office](http://www.et.gr),
* The entire corpus of EU legislation (Greek translation), as published in [Eur-Lex](https://eur-lex.europa.eu/homepage.html?locale=en).


## Pre-training details


* We trained BERT using the official code provided in Google BERT's GitHub repository (https://github.com/google-research/bert).* We then used [Hugging Face](https://huggingface.co)'s [Transformers](https://github.com/huggingface/transformers) conversion script to convert the TF checkpoint and vocabulary in the desired format in order to be able to load the model in two lines of code for both PyTorch and TF2 users.
* We released a model similar to the English `bert-base-uncased` model (12-layer, 768-hidden, 12-heads, 110M parameters).
* We chose to follow the same training set-up: 1 million training steps with batches of 256 sequences of length 512 with an initial learning rate 1e-4.
* We were able to use a single Google Cloud TPU v3-8 provided for free from [TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc), while also utilizing [GCP research credits](https://edu.google.com/programs/credits/research). Huge thanks to both Google programs for supporting us!


\* You can still have access to the original TensorFlow checkpoints from this [Google Drive folder](https://drive.google.com/drive/folders/1ZjlaE4nvdtgqXiVBTVHCF5I9Ff8ZmztE?usp=sharing).


## Requirements


We published `bert-base-greek-uncased-v1` as part of [Hugging Face](https://huggingface.co)'s [Transformers](https://github.com/huggingface/transformers) repository. So, you need to install the transformers library through pip along with PyTorch or Tensorflow 2.


```
pip install transformers
pip install (torch|tensorflow)
```


## Pre-process text (Deaccent - Lower)


**NOTICE:** Preprocessing is now natively supported by the default tokenizer. No need to include the following code.


In order to use `bert-base-greek-uncased-v1`, you have to pre-process texts to lowercase letters and remove all Greek diacritics.


```python

import unicodedata

def strip_accents_and_lowercase(s):
   return ''.join(c for c in unicodedata.normalize('NFD', s)
                  if unicodedata.category(c) != 'Mn').lower()

accented_string = "Αυτή είναι η Ελληνική έκδοση του BERT."
unaccented_string = strip_accents_and_lowercase(accented_string)

print(unaccented_string) # αυτη ειναι η ελληνικη εκδοση του bert.

```


## Load Pretrained Model


```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")
model = AutoModel.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")
```


## Use Pretrained Model as a Language Model


```python
import torch
from transformers import *

# Load model and tokenizer
tokenizer_greek = AutoTokenizer.from_pretrained('nlpaueb/bert-base-greek-uncased-v1')
lm_model_greek = AutoModelWithLMHead.from_pretrained('nlpaueb/bert-base-greek-uncased-v1')

# ================ EXAMPLE 1 ================
text_1 = 'O ποιητής έγραψε ένα [MASK] .'
# EN: 'The poet wrote a [MASK].'
input_ids = tokenizer_greek.encode(text_1)
print(tokenizer_greek.convert_ids_to_tokens(input_ids))
# ['[CLS]', 'o', 'ποιητης', 'εγραψε', 'ενα', '[MASK]', '.', '[SEP]']
outputs = lm_model_greek(torch.tensor([input_ids]))[0]
print(tokenizer_greek.convert_ids_to_tokens(outputs[0, 5].max(0)[1].item()))
# the most plausible prediction for [MASK] is "song"

# ================ EXAMPLE 2 ================
text_2 = 'Είναι ένας [MASK] άνθρωπος.'
# EN: 'He is a [MASK] person.'
input_ids = tokenizer_greek.encode(text_2)
print(tokenizer_greek.convert_ids_to_tokens(input_ids))
# ['[CLS]', 'ειναι', 'ενας', '[MASK]', 'ανθρωπος', '.', '[SEP]']
outputs = lm_model_greek(torch.tensor([input_ids]))[0]
print(tokenizer_greek.convert_ids_to_tokens(outputs[0, 3].max(0)[1].item()))
# the most plausible prediction for [MASK] is "good"

# ================ EXAMPLE 3 ================
text_3 = 'Είναι ένας [MASK] άνθρωπος και κάνει συχνά [MASK].'
# EN: 'He is a [MASK] person he does frequently [MASK].'
input_ids = tokenizer_greek.encode(text_3)
print(tokenizer_greek.convert_ids_to_tokens(input_ids))
# ['[CLS]', 'ειναι', 'ενας', '[MASK]', 'ανθρωπος', 'και', 'κανει', 'συχνα', '[MASK]', '.', '[SEP]']
outputs = lm_model_greek(torch.tensor([input_ids]))[0]
print(tokenizer_greek.convert_ids_to_tokens(outputs[0, 8].max(0)[1].item()))
# the most plausible prediction for the second [MASK] is "trips"
```


## Evaluation on downstream tasks


For detailed results read the article:


GREEK-BERT: The Greeks visiting Sesame Street. John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis and Ion Androutsopoulos. In the Proceedings of the 11th Hellenic Conference on Artificial Intelligence (SETN 2020). Held Online. 2020. (https://arxiv.org/abs/2008.12014)


### Named Entity Recognition with Greek NER dataset


| Model name          | Micro F1                              |
| ------------------- | ------------------------------------  |
BILSTM-CNN-CRF (Ma and Hovy, 2016)  | 76.4 ± 2.07
M-BERT-UNCASED (Devlin et al., 2019) | 81.5 ± 1.77
M-BERT-CASED (Devlin et al., 2019)| 82.1 ± 1.35
XLM-R  (Conneau et al., 2020)| 84.8 ± 1.50
GREEK-BERT (ours)   | **85.7 ± 1.00**


### Natural Language Inference with XNLI


| Model name          | Accuracy                              |
| ------------------- | ------------------------------------  |
DAM (Parikh et al., 2016) | 68.5 ± 1.71
M-BERT-UNCASED (Devlin et al., 2019) | 73.9 ± 0.64
M-BERT-CASED (Devlin et al., 2019) | 73.5 ± 0.49
XLM-R (Conneau et al., 2020) | 77.3 ± 0.41
GREEK-BERT (ours)   | **78.6 ± 0.62**


## Author


The model has been officially released with the article "GREEK-BERT: The Greeks visiting Sesame Street. John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis and Ion Androutsopoulos. In the Proceedings of the 11th Hellenic Conference on Artificial Intelligence (SETN 2020). Held Online. 2020" (https://arxiv.org/abs/2008.12014).


If you use the model, please cite the following:


```
@inproceedings{greek-bert,
author = {Koutsikakis, John and Chalkidis, Ilias and Malakasiotis, Prodromos and Androutsopoulos, Ion},
title = {GREEK-BERT: The Greeks Visiting Sesame Street},
year = {2020},
isbn = {9781450388788},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3411408.3411440},
booktitle = {11th Hellenic Conference on Artificial Intelligence},
pages = {110–117},
numpages = {8},
location = {Athens, Greece},
series = {SETN 2020}
}
```


Ilias Chalkidis on behalf of [AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr)


| Github: [@ilias.chalkidis](https://github.com/iliaschalkidis) | Twitter: [@KiddoThe2B](https://twitter.com/KiddoThe2B) |


## About Us


[AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr) develops algorithms, models, and systems that allow computers to process and generate natural language texts.


The group's current research interests include:


* question answering systems for databases, ontologies, document collections, and the Web, especially biomedical question answering,
* natural language generation from databases and ontologies, especially Semantic Web ontologies,
text classification, including filtering spam and abusive content,
* information extraction and opinion mining, including legal text analytics and sentiment analysis,
* natural language processing tools for Greek, for example parsers and named-entity recognizers,
machine learning in natural language processing, especially deep learning.


The group is part of the Information Processing Laboratory of the Department of Informatics of the Athens University of Economics and Business.


model name: bert-base-greek-uncased-v1


model author: nlpaueb


task: fill-mask


tags: pytorch, tf, jax, bert, pretraining, el, arxiv:2008.12014, transformers, fill-mask, autonlp_compatible, infinity_compatible


language: en
pipeline_tag: fill-mask
license: cc-by-sa-4.0
thumbnail: https://i.ibb.co/p3kQ7Rw/Screenshot-2020-10-06-at-12-16-36-PM.png
tags:


- legal
widget:
- text: "The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of police."


# LEGAL-BERT: The Muppets straight out of Law School


<img align="left" src="https://i.ibb.co/p3kQ7Rw/Screenshot-2020-10-06-at-12-16-36-PM.png" width="100"/>


LEGAL-BERT is a family of BERT models for the legal domain, intended to assist legal NLP research, computational law, and legal technology applications.  To pre-train the different variations of LEGAL-BERT, we collected 12 GB of diverse English legal text from several fields (e.g., legislation, court cases,  contracts) scraped from publicly available resources. Sub-domains variants (CONTRACTS-, EURLEX-, ECHR-) and/or general LEGAL-BERT perform better than using BERT out of the box for domain-specific tasks. A light-weight model (33% the size of BERT-BASE) pre-trained from scratch on legal data with competitive perfomance is also available.


I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras and I. Androutsopoulos. "LEGAL-BERT: The Muppets straight out of Law School". In Findings of Empirical Methods in Natural Language Processing (EMNLP 2020) (Short Papers), to be held online, 2020. (https://aclanthology.org/2020.findings-emnlp.261)


## Pre-training corpora


The pre-training corpora of LEGAL-BERT include:


* 116,062 documents of EU legislation, publicly available from EURLEX (http://eur-lex.europa.eu), the repository of EU Law running under the EU Publication Office.
* 61,826 documents of UK legislation, publicly available from the UK legislation portal (http://www.legislation.gov.uk).
* 19,867 cases from the European Court of Justice (ECJ), also available from EURLEX.
* 12,554 cases from HUDOC, the repository of the European Court of Human Rights (ECHR) (http://hudoc.echr.coe.int/eng).
* 164,141 cases from various courts across the USA, hosted in the Case Law Access Project portal (https://case.law).
* 76,366 US contracts from EDGAR, the database of US Securities and Exchange Commission (SECOM) (https://www.sec.gov/edgar.shtml).


## Pre-training details


* We trained BERT using the official code provided in Google BERT's github repository (https://github.com/google-research/bert).
* We released a model similar to the English BERT-BASE model (12-layer, 768-hidden, 12-heads, 110M parameters).
* We chose to follow the same training set-up: 1 million training steps with batches of 256 sequences of length 512 with an initial learning rate 1e-4.
* We were able to use a single Google Cloud TPU v3-8 provided for free from [TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc), while also utilizing [GCP research credits](https://edu.google.com/programs/credits/research). Huge thanks to both Google programs for supporting us!
* Part of LEGAL-BERT is a light-weight model pre-trained from scratch on legal data, which achieves comparable performance to larger models, while being much more efficient (approximately 4 times faster) with a smaller environmental footprint.


## Models list


| Model name          | Model Path                            | Training corpora    |
| ------------------- | ------------------------------------  | ------------------- |
| CONTRACTS-BERT-BASE | `nlpaueb/bert-base-uncased-contracts` | US contracts        |
| EURLEX-BERT-BASE    | `nlpaueb/bert-base-uncased-eurlex`    | EU legislation      |
| ECHR-BERT-BASE      | `nlpaueb/bert-base-uncased-echr`      | ECHR cases          |
| LEGAL-BERT-BASE *     | `nlpaueb/legal-bert-base-uncased`     | All                 |
| LEGAL-BERT-SMALL    | `nlpaueb/legal-bert-small-uncased`    | All                 |


\* LEGAL-BERT-BASE is the model referred to as LEGAL-BERT-SC in Chalkidis et al. (2020); a model trained from scratch in the legal corpora mentioned below using a newly created vocabulary by a sentence-piece tokenizer trained on the very same corpora.


\*\* As many of you expressed interest in the LEGAL-BERT-FP models (those relying on the original BERT-BASE checkpoint), they have been released in Archive.org (https://archive.org/details/legal_bert_fp), as these models are secondary and possibly only interesting for those who aim to dig deeper in the open questions of Chalkidis et al. (2020).


## Load Pretrained Model


```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("nlpaueb/legal-bert-base-uncased")
model = AutoModel.from_pretrained("nlpaueb/legal-bert-base-uncased")
```


## Use LEBAL-BERT variants as Language Models


| Corpus                             | Model                               | Masked token | Predictions  |
| --------------------------------- | ---------------------------------- | ------------ | ------------ |
|  | **BERT-BASE-UNCASED**                 |
| (Contracts) | This [MASK] Agreement is between General Motors and John Murray . | employment | ('new', '0.09'), ('current', '0.04'), ('proposed', '0.03'), ('marketing', '0.03'), ('joint', '0.02')
| (ECHR) | The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate | torture | ('torture', '0.32'), ('rape', '0.22'), ('abuse', '0.14'), ('death', '0.04'), ('violence', '0.03')
| (EURLEX) | Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . | bovine | ('farm', '0.25'), ('livestock', '0.08'), ('draft', '0.06'), ('domestic', '0.05'), ('wild', '0.05')
|  | **CONTRACTS-BERT-BASE**                 |
| (Contracts) | This [MASK] Agreement is between General Motors and John Murray . | employment | ('letter', '0.38'), ('dealer', '0.04'), ('employment', '0.03'), ('award', '0.03'), ('contribution', '0.02')
| (ECHR) | The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate | torture | ('death', '0.39'), ('imprisonment', '0.07'), ('contempt', '0.05'), ('being', '0.03'), ('crime', '0.02')
| (EURLEX) | Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . | bovine | (('domestic', '0.18'), ('laboratory', '0.07'), ('household', '0.06'), ('personal', '0.06'), ('the', '0.04')
|  | **EURLEX-BERT-BASE**                 |
| (Contracts) | This [MASK] Agreement is between General Motors and John Murray . | employment | ('supply', '0.11'), ('cooperation', '0.08'), ('service', '0.07'), ('licence', '0.07'), ('distribution', '0.05')
| (ECHR) | The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate | torture | ('torture', '0.66'), ('death', '0.07'), ('imprisonment', '0.07'), ('murder', '0.04'), ('rape', '0.02')
| (EURLEX) | Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . | bovine | ('live', '0.43'), ('pet', '0.28'), ('certain', '0.05'), ('fur', '0.03'), ('the', '0.02')
|  | **ECHR-BERT-BASE**                 |
| (Contracts) | This [MASK] Agreement is between General Motors and John Murray . | employment | ('second', '0.24'), ('latter', '0.10'), ('draft', '0.05'), ('bilateral', '0.05'), ('arbitration', '0.04')
| (ECHR) | The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate | torture | ('torture', '0.99'), ('death', '0.01'), ('inhuman', '0.00'), ('beating', '0.00'), ('rape', '0.00')
| (EURLEX) | Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . | bovine | ('pet', '0.17'), ('all', '0.12'), ('slaughtered', '0.10'), ('domestic', '0.07'), ('individual', '0.05')
|  | **LEGAL-BERT-BASE**                |
| (Contracts) | This [MASK] Agreement is between General Motors and John Murray . | employment | ('settlement', '0.26'), ('letter', '0.23'), ('dealer', '0.04'), ('master', '0.02'), ('supplemental', '0.02')
| (ECHR) | The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate | torture | ('torture', '1.00'), ('detention', '0.00'), ('arrest', '0.00'), ('rape', '0.00'), ('death', '0.00')
| (EURLEX) | Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . | bovine | ('live', '0.67'), ('beef', '0.17'), ('farm', '0.03'), ('pet', '0.02'), ('dairy', '0.01')
|  | **LEGAL-BERT-SMALL**                |
| (Contracts) | This [MASK] Agreement is between General Motors and John Murray . | employment | ('license', '0.09'), ('transition', '0.08'), ('settlement', '0.04'), ('consent', '0.03'), ('letter', '0.03')
| (ECHR) | The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate | torture | ('torture', '0.59'), ('pain', '0.05'), ('ptsd', '0.05'), ('death', '0.02'), ('tuberculosis', '0.02')
| (EURLEX) | Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . | bovine | ('all', '0.08'), ('live', '0.07'), ('certain', '0.07'), ('the', '0.07'), ('farm', '0.05')


## Evaluation on downstream tasks


Consider the experiments in the article "LEGAL-BERT: The Muppets straight out of Law School". Chalkidis et al., 2020, (https://aclanthology.org/2020.findings-emnlp.261)


## Author - Publication


```
@inproceedings{chalkidis-etal-2020-legal,
    title = "{LEGAL}-{BERT}: The Muppets straight out of Law School",
    author = "Chalkidis, Ilias  and
      Fergadiotis, Manos  and
      Malakasiotis, Prodromos  and
      Aletras, Nikolaos  and
      Androutsopoulos, Ion",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    doi = "10.18653/v1/2020.findings-emnlp.261",
    pages = "2898--2904"
}
```


Ilias Chalkidis on behalf of [AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr)


| Github: [@ilias.chalkidis](https://github.com/seolhokim) | Twitter: [@KiddoThe2B](https://twitter.com/KiddoThe2B) |


model name: legal-bert-base-uncased


model author: nlpaueb


task: fill-mask


tags: pytorch, tf, jax, bert, pretraining, en, transformers, license:cc-by-sa-4.0, legal, fill-mask, autonlp_compatible, infinity_compatible


tags:


- exbert


language: en
license: apache-2.0
datasets:


- bookcorpus
- wikipedia


# ALBERT Base v1


Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
[this paper](https://arxiv.org/abs/1909.11942) and first released in
[this repository](https://github.com/google-research/albert). This model, as all ALBERT models, is uncased: it does not make a difference
between english and English.


Disclaimer: The team releasing ALBERT did not write a model card for this model so this model card has been written by
the Hugging Face team.


## Model description


ALBERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
was pretrained with two objectives:


- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
the entire masked sentence through the model and has to predict the masked words. This is different from traditional
recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
sentence.
- Sentence Ordering Prediction (SOP): ALBERT uses a pretraining loss based on predicting the ordering of two consecutive segments of text.


This way, the model learns an inner representation of the English language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the ALBERT model as inputs.


ALBERT is particular in that it shares its layers across its Transformer. Therefore, all layers have the same weights. Using repeating layers results in a small memory footprint, however, the computational cost remains similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same number of (repeating) layers.


This is the first version of the base model. Version 2 is different from version 1 due to different dropout rates, additional training data, and longer training. It has better results in nearly all downstream tasks.


This model has the following configuration:


- 12 repeating layers
- 128 embedding dimension
- 768 hidden dimension
- 12 attention heads
- 11M parameters


## Intended uses & limitations


You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=albert) to look for
fine-tuned versions on a task that interests you.


Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.


### How to use


You can use this model directly with a pipeline for masked language modeling:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='albert-base-v1')
>>> unmasker("Hello I'm a [MASK] model.")
[
   {
      "sequence":"[CLS] hello i'm a modeling model.[SEP]",
      "score":0.05816134437918663,
      "token":12807,
      "token_str":"▁modeling"
   },
   {
      "sequence":"[CLS] hello i'm a modelling model.[SEP]",
      "score":0.03748830780386925,
      "token":23089,
      "token_str":"▁modelling"
   },
   {
      "sequence":"[CLS] hello i'm a model model.[SEP]",
      "score":0.033725276589393616,
      "token":1061,
      "token_str":"▁model"
   },
   {
      "sequence":"[CLS] hello i'm a runway model.[SEP]",
      "score":0.017313428223133087,
      "token":8014,
      "token_str":"▁runway"
   },
   {
      "sequence":"[CLS] hello i'm a lingerie model.[SEP]",
      "score":0.014405295252799988,
      "token":29104,
      "token_str":"▁lingerie"
   }
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import AlbertTokenizer, AlbertModel
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v1')
model = AlbertModel.from_pretrained("albert-base-v1")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import AlbertTokenizer, TFAlbertModel
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v1')
model = TFAlbertModel.from_pretrained("albert-base-v1")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


### Limitations and bias


Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
predictions:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='albert-base-v1')
>>> unmasker("The man worked as a [MASK].")

[
   {
      "sequence":"[CLS] the man worked as a chauffeur.[SEP]",
      "score":0.029577180743217468,
      "token":28744,
      "token_str":"▁chauffeur"
   },
   {
      "sequence":"[CLS] the man worked as a janitor.[SEP]",
      "score":0.028865724802017212,
      "token":29477,
      "token_str":"▁janitor"
   },
   {
      "sequence":"[CLS] the man worked as a shoemaker.[SEP]",
      "score":0.02581118606030941,
      "token":29024,
      "token_str":"▁shoemaker"
   },
   {
      "sequence":"[CLS] the man worked as a blacksmith.[SEP]",
      "score":0.01849772222340107,
      "token":21238,
      "token_str":"▁blacksmith"
   },
   {
      "sequence":"[CLS] the man worked as a lawyer.[SEP]",
      "score":0.01820771023631096,
      "token":3672,
      "token_str":"▁lawyer"
   }
]

>>> unmasker("The woman worked as a [MASK].")

[
   {
      "sequence":"[CLS] the woman worked as a receptionist.[SEP]",
      "score":0.04604868218302727,
      "token":25331,
      "token_str":"▁receptionist"
   },
   {
      "sequence":"[CLS] the woman worked as a janitor.[SEP]",
      "score":0.028220869600772858,
      "token":29477,
      "token_str":"▁janitor"
   },
   {
      "sequence":"[CLS] the woman worked as a paramedic.[SEP]",
      "score":0.0261906236410141,
      "token":23386,
      "token_str":"▁paramedic"
   },
   {
      "sequence":"[CLS] the woman worked as a chauffeur.[SEP]",
      "score":0.024797942489385605,
      "token":28744,
      "token_str":"▁chauffeur"
   },
   {
      "sequence":"[CLS] the woman worked as a waitress.[SEP]",
      "score":0.024124596267938614,
      "token":13678,
      "token_str":"▁waitress"
   }
]
```


This bias will also affect all fine-tuned versions of this model.


## Training data


The ALBERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
headers).


## Training procedure


### Preprocessing


The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 30,000. The inputs of the model are
then of the form:


```
[CLS] Sentence A [SEP] Sentence B [SEP]
```


### Training


The ALBERT procedure follows the BERT setup.


The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.


## Evaluation results


When fine-tuned on downstream tasks, the ALBERT models achieve the following results:


|                | Average  | SQuAD1.1 | SQuAD2.0 | MNLI     | SST-2    | RACE     |
|----------------|----------|----------|----------|----------|----------|----------|
|V2              |
|ALBERT-base     |82.3      |90.2/83.2 |82.1/79.3 |84.6      |92.9      |66.8      |
|ALBERT-large    |85.7      |91.8/85.2 |84.9/81.8 |86.5      |94.9      |75.2      |
|ALBERT-xlarge   |87.9      |92.9/86.4 |87.9/84.1 |87.9      |95.4      |80.7      |
|ALBERT-xxlarge  |90.9      |94.6/89.1 |89.8/86.9 |90.6      |96.8      |86.8      |
|V1              |
|ALBERT-base     |80.1      |89.3/82.3 | 80.0/77.1|81.6      |90.3      | 64.0     |
|ALBERT-large    |82.4      |90.6/83.9 | 82.3/79.4|83.5      |91.7      | 68.5     |
|ALBERT-xlarge   |85.5      |92.5/86.1 | 86.1/83.1|86.4      |92.4      | 74.8     |
|ALBERT-xxlarge  |91.0      |94.8/89.3 | 90.2/87.4|90.8      |96.9      | 86.5     |


### BibTeX entry and citation info


```bibtex
@article{DBLP:journals/corr/abs-1909-11942,
  author    = {Zhenzhong Lan and
               Mingda Chen and
               Sebastian Goodman and
               Kevin Gimpel and
               Piyush Sharma and
               Radu Soricut},
  title     = {{ALBERT:} {A} Lite {BERT} for Self-supervised Learning of Language
               Representations},
  journal   = {CoRR},
  volume    = {abs/1909.11942},
  year      = {2019},
  url       = {http://arxiv.org/abs/1909.11942},
  archivePrefix = {arXiv},
  eprint    = {1909.11942},
  timestamp = {Fri, 27 Sep 2019 13:04:21 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1909-11942.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```


model name: albert-base-v1


task: fill-mask


tags: pytorch, tf, albert, fill-mask, en, dataset:bookcorpus, dataset:wikipedia, arxiv:1909.11942, transformers, exbert, license:apache-2.0, autonlp_compatible, infinity_compatible


tags:


- conversational


# Harry Potter DialoGPT Model


model name: DialoGPT-small-harrypotter


model author: Akjder


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


license: mit
tags:


- generated_from_trainer
model-index:
- name: xlnet-base-cased-IUChatbot-ontologyDts-BertPretrainedTokenizerFast
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xlnet-base-cased-IUChatbot-ontologyDts-BertPretrainedTokenizerFast


This model is a fine-tuned version of [xlnet-base-cased](https://huggingface.co/xlnet-base-cased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3489


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 382  | 0.4695          |
| 0.5633        | 2.0   | 764  | 0.3361          |
| 0.3533        | 3.0   | 1146 | 0.3489          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: xlnet-base-cased-IUChatbot-ontologyDts-BertPretrainedTokenizerFast


model author: nntadotzip


task: question-answering


tags: pytorch, tensorboard, xlnet, question-answering, transformers, license:mit, generated_from_trainer, model-index, autonlp_compatible


license: mit
tags:


- generated_from_trainer
model-index:
- name: xlnet-base-cased-IUChatbot-ontologyDts-localParams
results: []


# xlnet-base-cased-IUChatbot-ontologyDts-localParams


- Loss: 0.0238


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.1172        | 1.0   | 1119 | 0.0657          |
| 0.0564        | 2.0   | 2238 | 0.0237          |
| 0.033         | 3.0   | 3357 | 0.0238          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.18.0
- Tokenizers 0.10.3


model name: xlnet-base-cased-IUChatbot-ontologyDts-localParams


model author: nntadotzip


task: question-answering


license: mit
tags:


- generated_from_trainer
model-index:
- name: xlnet-base-cased-IUChatbot-ontologyDts
results: []


# xlnet-base-cased-IUChatbot-ontologyDts


- Loss: 0.4965


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 318  | 0.5005          |
| 0.8222        | 2.0   | 636  | 0.4488          |
| 0.8222        | 3.0   | 954  | 0.4965          |


### Framework versions


model name: xlnet-base-cased-IUChatbot-ontologyDts


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-base-cased-IUChatbot-ontologyDts
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# bert-base-cased-IUChatbot-ontologyDts


This model is a fine-tuned version of [bert-base-cased](https://huggingface.co/bert-base-cased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2446


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 382  | 0.2686          |
| 0.3946        | 2.0   | 764  | 0.2535          |
| 0.2577        | 3.0   | 1146 | 0.2446          |


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: bert-base-cased-IUChatbot-ontologyDts


model author: nntadotzip


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


This is just a test


model name: mymodel


model author: dreji18


task: text-classification


tags: tf, distilbert, text-classification, transformers, infinity_compatible


tags:


- generated_from_trainer
datasets:
- wmt16_en_ro_pre_processed
model-index:
- name: t5-tiny-random-length-128-learning_rate-2e-05-weight_decay-0.01-finetuned-en-to-ro-TRAIN_EPOCHS-1
results: []


# t5-tiny-random-length-128-learning_rate-2e-05-weight_decay-0.01-finetuned-en-to-ro-TRAIN_EPOCHS-1


This model is a fine-tuned version of [patrickvonplaten/t5-tiny-random](https://huggingface.co/patrickvonplaten/t5-tiny-random) on the wmt16_en_ro_pre_processed dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP


### Framework versions


- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: t5-tiny-random-length-128-learning_rate-2e-05-weight_decay-0.01-finetuned-en-to-ro-TRAIN_EPOCHS-1


model author: diegor2


task: text2text-generation


tags: pytorch, tensorboard, t5, text2text-generation, dataset:wmt16_en_ro_pre_processed, transformers, generated_from_trainer, model-index, autonlp_compatible


tags:


- generated_from_trainer
datasets:
- wmt16_en_ro_pre_processed
model-index:
- name: t5-tiny-random-length-96-learning_rate-2e-05-weight_decay-0.01-finetuned-en-to-ro-TRAIN_EPOCHS-1
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# t5-tiny-random-length-96-learning_rate-2e-05-weight_decay-0.01-finetuned-en-to-ro-TRAIN_EPOCHS-1


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


### Framework versions


model name: t5-tiny-random-length-96-learning_rate-2e-05-weight_decay-0.01-finetuned-en-to-ro-TRAIN_EPOCHS-1


- generated_from_trainer
datasets:
- wmt16_en_ro_pre_processed
metrics:
- bleu
model-index:
- name: t5-tiny-random-length-96-learning_rate-2e-05-weight_decay-0.005-finetuned-en-to-ro-TRAIN_EPOCHS-1
results:
  - task:
name: Sequence-to-sequence Language Modeling
type: text2text-generation
dataset:
name: wmt16_en_ro_pre_processed
type: wmt16_en_ro_pre_processed
args: enro
metrics:
    - name: Bleu
type: bleu
value: 0.0002


# t5-tiny-random-length-96-learning_rate-2e-05-weight_decay-0.005-finetuned-en-to-ro-TRAIN_EPOCHS-1


This model is a fine-tuned version of [patrickvonplaten/t5-tiny-random](https://huggingface.co/patrickvonplaten/t5-tiny-random) on the wmt16_en_ro_pre_processed dataset.
It achieves the following results on the evaluation set:


- Loss: 6.4897
- Bleu: 0.0002
- Gen Len: 9.0


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Bleu   | Gen Len |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:-------:|
| 6.2585        | 1.0   | 76290 | 6.4897          | 0.0002 | 9.0     |


model name: t5-tiny-random-length-96-learning_rate-2e-05-weight_decay-0.005-finetuned-en-to-ro-TRAIN_EPOCHS-1


model author: diegor2


task: text2text-generation


FinBert Pretrained model to be used with downstream tasks


model name: pretrained-FinBERT


model author: sagar


tags: pytorch, transformers


language:


- uk


# ukr-roberta-base


## Pre-training corpora


Below is the list of corpora used along with the output of wc command (counting lines, words and characters). These corpora were concatenated and tokenized with HuggingFace Roberta Tokenizer.


| Tables        | Lines           | Words  | Characters  |
| ------------- |--------------:| -----:| -----:|
| [Ukrainian Wikipedia - May 2020](https://dumps.wikimedia.org/ukwiki/latest/ukwiki-latest-pages-articles.xml.bz2)      | 18 001 466| 201 207 739 | 2 647 891 947 |
| [Ukrainian OSCAR deduplicated dataset](https://oscar-public.huma-num.fr/shuffled/uk_dedup.txt.gz) | 56 560 011      |    2 250 210 650 | 29 705 050 592 |
| Sampled mentions from social networks | 11 245 710      |    128 461 796 | 1 632 567 763 |
| Total | 85 807 187      |    2 579 880 185 | 33 985 510 302 |


## Pre-training details


* Ukrainian Roberta was trained with code provided in [HuggingFace tutorial](https://huggingface.co/blog/how-to-train)
* Currently released model follows roberta-base-cased model architecture (12-layer, 768-hidden, 12-heads, 125M parameters)
* The model was trained on 4xV100 (85 hours)
* Training configuration you can find in the [original repository](https://github.com/youscan/language-models)


## Author


Vitalii Radchenko - contact me on Twitter [@vitaliradchenko](https://twitter.com/vitaliradchenko)


model name: ukr-roberta-base


model author: youscan


task: fill-mask


tags: pytorch, jax, roberta, fill-mask, uk, transformers, autonlp_compatible, infinity_compatible


tags:


- conversational


# Iroh DialoGPT Model


model name: DialoGPT-small-Iroh-Bot


model author: TVLG


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


language:


- id


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-large-xls-r-300m-Indonesian
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_7_0  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice id  # Required. Example: Common Voice zh-CN
args: id         # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 25.06 # Required. Example: 20.90
name: Test WER With LM   # Optional. Example: Test WER
    - type: cer    # Required. Example: wer
value: 6.50  # Required. Example: 20.90
name: Test CER With LM   # Optional. Example: Test WER


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-Indonesian


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4087
- Wer: 0.2461
- Cer: 0.0666


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 64
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 400
- num_epochs: 50
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|
| 5.0788        | 4.26  | 200  | 2.9389          | 1.0    | 1.0    |
| 2.8288        | 8.51  | 400  | 2.2535          | 1.0    | 0.8004 |
| 0.907         | 12.77 | 600  | 0.4558          | 0.4243 | 0.1095 |
| 0.4071        | 17.02 | 800  | 0.4013          | 0.3468 | 0.0913 |
| 0.3           | 21.28 | 1000 | 0.4167          | 0.3075 | 0.0816 |
| 0.2544        | 25.53 | 1200 | 0.4132          | 0.2835 | 0.0762 |
| 0.2145        | 29.79 | 1400 | 0.3878          | 0.2693 | 0.0729 |
| 0.1923        | 34.04 | 1600 | 0.4023          | 0.2623 | 0.0702 |
| 0.1681        | 38.3  | 1800 | 0.3984          | 0.2581 | 0.0686 |
| 0.1598        | 42.55 | 2000 | 0.3982          | 0.2493 | 0.0663 |
| 0.1464        | 46.81 | 2200 | 0.4087          | 0.2461 | 0.0666 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


model name: wav2vec2-large-xls-r-300m-Indonesian


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, id, dataset:mozilla-foundation/common_voice_7_0, transformers, license:apache-2.0, robust-speech-event, model-index


language:


- pa-IN


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-punjabi-V8-Abid
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice pa-IN  # Required. Example: Common Voice zh-CN
args: pa-IN         # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 36.02 # Required. Example: 20.90
name: Test WER With LM    # Optional. Example: Test WER
    - type: cer    # Required. Example: wer
value: 12.81   # Required. Example: 20.90
name: Test CER With LM  # Optional. Example: Test WER


# wav2vec2-large-xlsr-53-punjabi


This model is a fine-tuned version of [Harveenchadha/vakyansh-wav2vec2-punjabi-pam-10](https://huggingface.co/Harveenchadha/vakyansh-wav2vec2-punjabi-pam-10) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 1.2101
- Wer: 0.4939
- Cer: 0.2238


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id kingabzpro/wav2vec2-large-xlsr-53-punjabi --dataset mozilla-foundation/common_voice_8_0 --config pa-IN --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "kingabzpro/wav2vec2-large-xlsr-53-punjabi"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "pa-IN", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text

```


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 200
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|
| 11.0563       | 3.7   | 100  | 1.9492          | 0.7123 | 0.3872 |
| 1.6715        | 7.41  | 200  | 1.3142          | 0.6433 | 0.3086 |
| 0.9117        | 11.11 | 300  | 1.2733          | 0.5657 | 0.2627 |
| 0.666         | 14.81 | 400  | 1.2730          | 0.5598 | 0.2534 |
| 0.4225        | 18.52 | 500  | 1.2548          | 0.5300 | 0.2399 |
| 0.3209        | 22.22 | 600  | 1.2166          | 0.5229 | 0.2372 |
| 0.2678        | 25.93 | 700  | 1.1795          | 0.5041 | 0.2276 |
| 0.2088        | 29.63 | 800  | 1.2101          | 0.4939 | 0.2238 |


### Framework versions


model name: wav2vec2-large-xlsr-53-punjabi


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, pa-IN, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, robust-speech-event, model-index


language:


- ur


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-60-urdu
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_7_0 # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice ur  # Required. Example: Common Voice zh-CN
args: ur         # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 59.1  # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
args:
      - learning_rate: 0.0003
      - train_batch_size: 16
      - eval_batch_size: 8
      - seed: 42
      - gradient_accumulation_steps: 2
      - total_train_batch_size: 32
      - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
      - lr_scheduler_type: linear
      - lr_scheduler_warmup_steps: 200
      - num_epochs: 50
      - mixed_precision_training: Native AMP         # Optional. Example for BLEU: max_order
    - type: cer    # Required. Example: wer
value: 33.1  # Required. Example: 20.90
name: Test CER    # Optional. Example: Test WER
args:
      - learning_rate: 0.0003
      - train_batch_size: 16
      - eval_batch_size: 8
      - seed: 42
      - gradient_accumulation_steps: 2
      - total_train_batch_size: 32
      - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
      - lr_scheduler_type: linear
      - lr_scheduler_warmup_steps: 200
      - num_epochs: 50
      - mixed_precision_training: Native AMP         # Optional. Example for BLEU: max_order


# wav2vec2-large-xlsr-53-urdu


This model is a fine-tuned version of [Harveenchadha/vakyansh-wav2vec2-urdu-urm-60](https://huggingface.co/Harveenchadha/vakyansh-wav2vec2-urdu-urm-60) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Wer: 0.5913
- Cer: 0.3310


## Model description


The training and valid dataset is 0.58 hours. It was hard to train any model on lower number of so I decided to take vakyansh-wav2vec2-urdu-urm-60 checkpoint and finetune the wav2vec2 model.


## Training procedure


Trained on Harveenchadha/vakyansh-wav2vec2-urdu-urm-60 due to lesser number of samples.


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 200
- num_epochs: 50
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|
| 12.6045       | 8.33  | 100  | 8.4997          | 0.6978 | 0.3923 |
| 1.3367        | 16.67 | 200  | 5.0015          | 0.6515 | 0.3556 |
| 0.5344        | 25.0  | 300  | 9.3687          | 0.6393 | 0.3625 |
| 0.2922        | 33.33 | 400  | 9.2381          | 0.6236 | 0.3432 |
| 0.1867        | 41.67 | 500  | 6.2150          | 0.6035 | 0.3448 |
| 0.1166        | 50.0  | 600  | 6.4496          | 0.5913 | 0.3310 |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: wav2vec2-60-urdu


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, ur, dataset:mozilla-foundation/common_voice_7_0, transformers, license:apache-2.0, robust-speech-event, model-index


language: English
datasets:


- Andrada Olteanu Rickmorty-Scripts
tags:
- conversational
- Transformers
- gpt2
- Chatbot
- Rick&Morty
license: apache-2.0
metrics:
- Perplexity


# Source Code


[<img src="https://api.flatworld.co/wp-content/uploads/2020/10/DAGsHub-Logo.png" alt="dagshub" width="150"/>](https://dagshub.com/kingabzpro/DailoGPT-RickBot)


# Testing


```python
tokenizer = AutoTokenizer.from_pretrained('kingabzpro/DialoGPT-small-Rick-Bot')
model = AutoModelWithLMHead.from_pretrained('kingabzpro/DialoGPT-small-Rick-Bot')
# Let's chat for 4 lines
for step in range(4):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input(">> User:") + tokenizer.eos_token, return_tensors='pt')
    # print(new_user_input_ids)

    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) if step > 0 else new_user_input_ids

    # generated a response while limiting the total chat history to 1000 tokens,
    chat_history_ids = model.generate(
        bot_input_ids, max_length=200,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )

    # pretty print last ouput tokens from bot
    print("RickBot: {}".format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))
```


**Result**


perplexity : 8.53


model name: DialoGPT-small-Rick-Bot


task: conversational


tags: pytorch, gpt2, text-generation, English, dataset:Andrada Olteanu Rickmorty-Scripts, transformers, conversational, Transformers, Chatbot, Rick&Morty, license:apache-2.0


language:


- id


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-large-xls-r-1b-Indonesian
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice id  # Required. Example: Common Voice zh-CN
args: id         # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 45.51 # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
    - type: cer    # Required. Example: wer
value: 16.43  # Required. Example: 20.90
name: Test CER    # Optional. Example: Test WER


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-1b-Indonesian


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.9550
- Wer: 0.4551
- Cer: 0.1643


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 64
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 400
- num_epochs: 50
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|
| 3.663         | 7.69  | 200  | 0.7898          | 0.6039 | 0.1848 |
| 0.7424        | 15.38 | 400  | 1.0215          | 0.5615 | 0.1924 |
| 0.4494        | 23.08 | 600  | 1.0901          | 0.5249 | 0.1932 |
| 0.5075        | 30.77 | 800  | 1.1013          | 0.5079 | 0.1935 |
| 0.4671        | 38.46 | 1000 | 1.1034          | 0.4916 | 0.1827 |
| 0.1928        | 46.15 | 1200 | 0.9550          | 0.4551 | 0.1643 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


model name: wav2vec2-large-xls-r-1b-Indonesian


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, id, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, robust-speech-event, model-index


language:


- ur
license: apache-2.0
tags:
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
model-index:
- name: wav2vec2-large-xls-r-300m-Urdu
results:
  - task:
type: automatic-speech-recognition
name: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0
name: Common Voice 8
args: ur
metrics:
    - type: wer    # Required. Example: wer
value: 39.89  # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
    - name: Test CER
type: cer
value: 16.70


# wav2vec2-large-xls-r-300m-Urdu


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.9889
- Wer: 0.5607
- Cer: 0.2370


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id kingabzpro/wav2vec2-large-xls-r-300m-Urdu --dataset mozilla-foundation/common_voice_8_0 --config ur --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "kingabzpro/wav2vec2-large-xls-r-300m-Urdu"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "ur", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "اب نے ٹپیدسون دیتے ہیں"
```


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 200


### Training results


| Training Loss | Epoch  | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:------:|:----:|:---------------:|:------:|:------:|
| 3.6398        | 30.77  | 400  | 3.3517          | 1.0    | 1.0    |
| 2.9225        | 61.54  | 800  | 2.5123          | 1.0    | 0.8310 |
| 1.2568        | 92.31  | 1200 | 0.9699          | 0.6273 | 0.2575 |
| 0.8974        | 123.08 | 1600 | 0.9715          | 0.5888 | 0.2457 |
| 0.7151        | 153.85 | 2000 | 0.9984          | 0.5588 | 0.2353 |
| 0.6416        | 184.62 | 2400 | 0.9889          | 0.5607 | 0.2370 |


### Framework versions


### Eval results on Common Voice 8 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 52.03 | 39.89 |


model name: wav2vec2-large-xls-r-300m-Urdu


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, ur, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language:


- ur


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-urdu
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_7_0  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice ur  # Required. Example: Common Voice zh-CN
args: ur         # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 52.40 # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
args:
      - learning_rate: 0.0003
      - train_batch_size: 64
      - eval_batch_size: 8
      - seed: 42
      - gradient_accumulation_steps: 2
      - total_train_batch_size: 128
      - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
      - lr_scheduler_type: linear
      - lr_scheduler_warmup_steps: 100
      - num_epochs: 100
      - mixed_precision_training: Native AMP        # Optional. Example for BLEU: max_order
    - type: cer    # Required. Example: wer
value: 26.46  # Required. Example: 20.90
name: Test CER  # Optional. Example: Test WER
args:
      - learning_rate: 0.0003
      - train_batch_size: 64
      - eval_batch_size: 8
      - seed: 42
      - gradient_accumulation_steps: 2
      - total_train_batch_size: 128
      - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
      - lr_scheduler_type: linear
      - lr_scheduler_warmup_steps: 100
      - num_epochs: 100
      - mixed_precision_training: Native AMP
    - type: wer   # Required. Example: wer
value: 45.63  # Required. Example: 20.90
name: Test WER LM CV8     # Optional. Example: Test WER
    - type: cer # Required. Example: wer
value: 20.45  # Required. Example: 20.90
name: Test CER LM CV8    # Optional. Example: Test WER


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-Urdu


This model is a fine-tuned version of [Harveenchadha/vakyansh-wav2vec2-urdu-urm-60](https://huggingface.co/Harveenchadha/vakyansh-wav2vec2-urdu-urm-60) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Wer: 0.5747
- Cer: 0.3268


## Model description


The training and valid dataset is 0.58 hours. It was hard to train any model on lower number of so I decided to take vakyansh-wav2vec2-urdu-urm-60 checkpoint and finetune the wav2vec2 model.


## Training procedure


Trained on Harveenchadha/vakyansh-wav2vec2-urdu-urm-60 due to lesser number of samples.


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 64
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 100
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|
| 4.3054        | 16.67 | 50   | 9.0055          | 0.8306 | 0.4869 |
| 2.0629        | 33.33 | 100  | 9.5849          | 0.6061 | 0.3414 |
| 0.8966        | 50.0  | 150  | 4.8686          | 0.6052 | 0.3426 |
| 0.4197        | 66.67 | 200  | 12.3261         | 0.5817 | 0.3370 |
| 0.294         | 83.33 | 250  | 11.9653         | 0.5712 | 0.3328 |
| 0.2329        | 100.0 | 300  | 7.6846          | 0.5747 | 0.3268 |


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0


model name: wav2vec2-urdu


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, ur, dataset:mozilla-foundation/common_voice_7_0, transformers, license:apache-2.0, robust-speech-event, model-index


language:


- ar


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-xls-r-300m-arabic
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_7_0  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice ar # Required. Example: Common Voice zh-CN
args: ar        # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 38.83  # Required. Example: 20.90
name: Test WER With LM    # Optional. Example: Test WER
    - type: cer    # Required. Example: wer
value: 15.33  # Required. Example: 20.90
name: Test CER With LM   # Optional. Example: Test WER


# wav2vec2-large-xlsr-300-arabic


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4514
- Wer: 0.4256
- Cer: 0.1528


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_7_0` with split `test`


```bash
python eval.py --model_id kingabzpro/wav2vec2-large-xlsr-300-arabic --dataset mozilla-foundation/common_voice_7_0 --config ur --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "kingabzpro/wav2vec2-large-xlsr-300-arabic"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "ar", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
```


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 64
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 10
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|
| 5.4375        | 1.8   | 500  | 3.3330          | 1.0    | 1.0    |
| 2.2187        | 3.6   | 1000 | 0.7790          | 0.6501 | 0.2338 |
| 0.9471        | 5.4   | 1500 | 0.5353          | 0.5015 | 0.1822 |
| 0.7416        | 7.19  | 2000 | 0.4889          | 0.4490 | 0.1640 |
| 0.6358        | 8.99  | 2500 | 0.4514          | 0.4256 | 0.1528 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


model name: wav2vec2-large-xlsr-300-arabic


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, ar, dataset:mozilla-foundation/common_voice_7_0, transformers, license:apache-2.0, robust-speech-event, model-index


language:


- ur


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-urdu-V8-Abid
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice ur  # Required. Example: Common Voice zh-CN
args: ur         # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 44.63 # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
args:
      - learning_rate: 7.5e-05
      - train_batch_size: 16
      - eval_batch_size: 8
      - seed: 42
      - gradient_accumulation_steps: 2
      - total_train_batch_size: 32
      - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
      - lr_scheduler_type: linear
      - lr_scheduler_warmup_steps: 200
      - num_epochs: 50
      - mixed_precision_training: Native AMPP       # Optional. Example for BLEU: max_order
    - type: cer    # Required. Example: wer
value: 18.82  # Required. Example: 20.90
name: Test CER    # Optional. Example: Test WER
args:
      - learning_rate: 7.5e-05
      - train_batch_size: 16
      - eval_batch_size: 8
      - seed: 42
      - gradient_accumulation_steps: 2
      - total_train_batch_size: 32
      - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
      - lr_scheduler_type: linear
      - lr_scheduler_warmup_steps: 200
      - num_epochs: 50
      - mixed_precision_training: Native AMPP


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-60-Urdu-V8


This model is a fine-tuned version of [Harveenchadha/vakyansh-wav2vec2-urdu-urm-60](https://huggingface.co/Harveenchadha/vakyansh-wav2vec2-urdu-urm-60) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 11.4832
- Wer: 0.5729
- Cer: 0.3170


### Training hyperparameters


- learning_rate: 7.5e-05
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 200
- num_epochs: 50
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|
| 19.671        | 8.33  | 100  | 7.7671          | 0.8795 | 0.4492 |
| 2.085         | 16.67 | 200  | 9.2759          | 0.6201 | 0.3320 |
| 0.6633        | 25.0  | 300  | 8.7025          | 0.5738 | 0.3104 |
| 0.388         | 33.33 | 400  | 10.2286         | 0.5852 | 0.3128 |
| 0.2822        | 41.67 | 500  | 11.1953         | 0.5738 | 0.3174 |
| 0.2293        | 50.0  | 600  | 11.4832         | 0.5729 | 0.3170 |


### Framework versions


- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


model name: wav2vec2-60-Urdu-V8


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, ur, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, robust-speech-event, model-index


language: WOLOF
datasets:


- AI4D Baamtu Datamation - Automatic Speech Recognition in WOLOF
tags:
- speech
- audio
- automatic-speech-recognition
license: apache-2.0
metrics:
- WER


## Evaluation on WOLOF Test


```python
import pandas as pd
from datasets import load_dataset, load_metric,Dataset
from tqdm import tqdm
import torch
import soundfile as sf
import torchaudio
from transformers import Wav2Vec2ForCTC
from transformers import Wav2Vec2Processor
from transformers import Wav2Vec2FeatureExtractor
from transformers import Wav2Vec2CTCTokenizer

model_name = "kingabzpro/wav2vec2-large-xlsr-53-wolof"
device = "cuda"

model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(model_name)

val =pd.read_csv("../input/automatic-speech-recognition-in-wolof/Test.csv")
val["path"] = "../input/automatic-speech-recognition-in-wolof/Noise Removed/tmp/WOLOF_ASR_dataset/noise_remove/"+val["ID"]+".wav"
val.rename(columns = {'transcription':'sentence'}, inplace = True)
common_voice_val = Dataset.from_pandas(val)

def speech_file_to_array_fn_test(batch):
    speech_array, sampling_rate = sf.read(batch["path"])#(.wav) 16000 sample rate
    batch["speech"] = speech_array
    batch["sampling_rate"] = sampling_rate
    return batch

def prepare_dataset_test(batch):
    # check that all files have the correct sampling rate
    assert (
        len(set(batch["sampling_rate"])) == 1
    ), f"Make sure all inputs have the same sampling rate of {processor.feature_extractor.sampling_rate}."

    batch["input_values"] = processor(batch["speech"], padding=True,sampling_rate=batch["sampling_rate"][0]).input_values
    return batch

common_voice_val = common_voice_val.remove_columns([ "ID","age",  "down_votes", "gender",  "up_votes"]) # Remove columns
common_voice_val = common_voice_val.map(speech_file_to_array_fn_test, remove_columns=common_voice_val.column_names)# Applying speech_file_to_array function
common_voice_val = common_voice_val.map(prepare_dataset_test, remove_columns=common_voice_val.column_names, batch_size=8, num_proc=4, batched=True)# Applying prepare_dataset_test function

final_pred = []
for i in tqdm(range(common_voice_val.shape[0])):# Testing model on Wolof Dataset
    input_dict = processor(common_voice_val[i]["input_values"], return_tensors="pt", padding=True)

    logits = model(input_dict.input_values.to("cuda")).logits

    pred_ids = torch.argmax(logits, dim=-1)[0]
    prediction = processor.decode(pred_ids)
    final_pred.append(prediction)

```


You can check my result on [Zindi](https://zindi.africa/competitions/ai4d-baamtu-datamation-automatic-speech-recognition-in-wolof/leaderboard), I got 8th rank in AI4D Baamtu Datamation - Automatic Speech Recognition in WOLOF


**Result**: 7.88 %


model name: wav2vec2-large-xlsr-53-wolof


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, WOLOF, dataset:AI4D Baamtu Datamation - Automatic Speech Recognition in WOLOF, transformers, speech, audio, license:apache-2.0


language:


- sv-SE


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-large-xls-r-1b-Swedish
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice sv-SE # Required. Example: Common Voice zh-CN
args: sv-SE       # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 14.04  # Required. Example: 20.90
name: Test WER With LM   # Optional. Example: Test WER
args:
      - learning_rate: 7.5e-05
      - train_batch_size: 32
      - eval_batch_size: 8
      - seed: 42
      - gradient_accumulation_steps: 4
      - total_train_batch_size: 128
      - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
      - lr_scheduler_type: linear
      - lr_scheduler_warmup_steps: 1000
      - num_epochs: 50
      - mixed_precision_training: Native AMP    # Optional. Example for BLEU: max_order
    - type: cer    # Required. Example: wer
value: 4.86  # Required. Example: 20.90
name: Test CER  With LM   # Optional. Example: Test WER
args:
      - learning_rate: 7.5e-05
      - train_batch_size: 32
      - eval_batch_size: 8
      - seed: 42
      - gradient_accumulation_steps: 4
      - total_train_batch_size: 128
      - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
      - lr_scheduler_type: linear
      - lr_scheduler_warmup_steps: 1000
      - num_epochs: 50
      - mixed_precision_training: Native AMP
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: sv
metrics:
- name: Test WER
type: wer
value: 29.69
- name: Test CER
type: cer
value: 12.59


# wav2vec2-large-xls-r-1b-Swedish


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the common_voice dataset.
It achieves the following results on the evaluation set:


**Without LM**


- Loss: 0.3370
- Wer: 18.44
- Cer: 5.75


**With LM**


- Loss: 0.3370
- Wer: 14.04
- Cer: 4.86


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id kingabzpro/wav2vec2-large-xls-r-1b-Swedish --dataset mozilla-foundation/common_voice_8_0 --config sv-SE --split test
```


2. To evaluate on `speech-recognition-community-v2/dev_data`


```bash
python eval.py --model_id kingabzpro/wav2vec2-large-xls-r-1b-Swedish --dataset speech-recognition-community-v2/dev_data --config sv --split validation --chunk_length_s 5.0 --stride_length_s 1.0
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "kingabzpro/wav2vec2-large-xls-r-1b-Swedish"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "sv-SE", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
```


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 7.5e-05
- train_batch_size: 64
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 50
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|
| 3.1562        | 11.11 | 500  | 0.4830          | 0.3729 | 0.1169 |
| 0.5655        | 22.22 | 1000 | 0.3553          | 0.2381 | 0.0743 |
| 0.3376        | 33.33 | 1500 | 0.3359          | 0.2179 | 0.0696 |
| 0.2419        | 44.44 | 2000 | 0.3232          | 0.1844 | 0.0575 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


model name: wav2vec2-large-xls-r-1b-Swedish


model author: kingabzpro


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, sv-SE, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, robust-speech-event, model-index


language: Yorùbá
datasets:


- AI4D-Africa - Yorùbá Machine Translation Challenge
tags:
- text
- machine-translation
- language-translation
- seq2seq
- helsinki-nlp
license: apache-2.0
metrics:
- ROUGE


## Predicting English Translation


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Loading tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("kingabzpro/Helsinki-NLP-opus-yor-mul-en")
model = AutoModelForSeq2SeqLM.from_pretrained("kingabzpro/Helsinki-NLP-opus-yor-mul-en").to('cuda')

# Prediction
a = model.generate(**tokenizer.prepare_seq2seq_batch('Nínú ìpè kan lẹ́yìn ìgbà náà, wọ́n sọ fún aṣojú iléeṣẹ́ BlaBlaCar pé ètò náà ti yí padà, pé',return_tensors='pt').to('cuda'))
text = tokenizer.batch_decode(a)

# Cleaning text
text = str(text)
text = re.sub("<pad> ","",text)
text = re.sub("'","",text)
text = text.replace("[", "")
text = text.replace("]", "")
text
```


## Result


```
'In a statement after that hearing, the BualaCard’s representative was told that the event had changed, that he had turned up.'
```


## ROGUE Score


model name: Helsinki-NLP-opus-yor-mul-en


task: text2text-generation


tags: pytorch, marian, text2text-generation, Yorùbá, dataset:AI4D-Africa - Yorùbá Machine Translation Challenge, transformers, text, machine-translation, language-translation, seq2seq, helsinki-nlp, license:apache-2.0, autonlp_compatible


language:


- tt


license: apache-2.0
tags:


- automatic-speech-recognition
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
- cer
model-index:
- name: wav2vec2-large-xls-r-300m-Tatar
results:
  - task:
type: automatic-speech-recognition  # Required. Example: automatic-speech-recognition
name: Speech Recognition  # Optional. Example: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: Common Voice tt # Required. Example: Common Voice zh-CN
args: tt        # Optional. Example: zh-CN
metrics:
    - type: wer    # Required. Example: wer
value: 42.71 # Required. Example: 20.90
name: Test WER With LM   # Optional. Example: Test WER
    - type: cer    # Required. Example: wer
value: 11.18  # Required. Example: 20.90
name: Test CER With LM   # Optional. Example: Test WER


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-Tatar


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.5068
- Wer: 0.4263
- Cer: 0.1117


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id kingabzpro/wav2vec2-large-xls-r-300m-Tatar --dataset mozilla-foundation/common_voice_8_0 --config tt --split test
```


### Inference With LM