```
@inproceedings{budur-etal-2020-data,
    title = "Data and Representation for Turkish Natural Language Inference",
    author = "Budur, Emrah and
      \"{O}z\c{c}elik, R{\i}za and
      G\"{u}ng\"{o}r, Tunga",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics"
}
```


model name: bert-base-turkish-cased-nli-mean


model author: mys


task: feature-extraction


tags: tf, bert, feature-extraction, arxiv:2004.14963, transformers, infinity_compatible


## README


model name: test_repo


model author: Longines


tags:


thumbnail: https://raw.githubusercontent.com/RuolinZheng08/twewy-discord-chatbot/main/gif-demo/icon.png


- conversational


license: mit


# DialoGPT Trained on the Speech of a Game Character


This is an instance of [microsoft/DialoGPT-medium](https://huggingface.co/microsoft/DialoGPT-medium) trained on a game character, Joshua from [The World Ends With You](https://en.wikipedia.org/wiki/The_World_Ends_with_You). The data comes from [a Kaggle game script dataset](https://www.kaggle.com/ruolinzheng/twewy-game-script).


I built a Discord AI chatbot based on this model. [Check out my GitHub repo.](https://github.com/T3879/Joshua-Bot_Model/tree/main/twewy-discord-chatbot-main)


Chat with the model:


```python
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("r3cdhummingbird/DialoGPT-medium-joshua")

model = AutoModelWithLMHead.from_pretrained("r3cdhummingbird/DialoGPT-medium-joshua")

# Let's chat for 4 lines
for step in range(4):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input(">> User:") + tokenizer.eos_token, return_tensors='pt')
    # print(new_user_input_ids)

    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) if step > 0 else new_user_input_ids

    # generated a response while limiting the total chat history to 1000 tokens,
    chat_history_ids = model.generate(
        bot_input_ids, max_length=200,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )

    # pretty print last ouput tokens from bot
    print("JoshuaBot: {}".format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))
```


model name: DialoGPT-medium-joshua


model author: r3cdhummingbird


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational, license:mit


#HarryBoy


model name: HarryBoy


model author: AllwynJ


tags: pytorch, gpt2, text-generation, transformers, conversational


## license: artistic-2.0


model name: Atiqah


model author: Atiqah


tags: license:artistic-2.0


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: distilbert-base-uncased-finetuned-squad
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-squad


model author: hiiii23


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, dataset:squad, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- conversational


# awazimuruk DialoGPT Model


model name: DialoGPT-small-awazimuruk


model author: inywer


task: conversational


# leirbag DialoGPT Model


model name: DialoGPT-medium-leirbag


# ettengiv DialoGPT Model


model name: DialoGPT-medium-ettengiv


tags: conversational


# RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation


Model accompanying our INLG 2020 paper: [RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation](https://www.aclweb.org/anthology/2020.inlg-1.4.pdf)


## Where is the dataset?


Please visit the website of our project: [recipenlg.cs.put.poznan.pl](https://recipenlg.cs.put.poznan.pl/) to download it.


## How to use the model? Could you explain X andy Y?


Yes, sure! If you feel some information is missing in our paper, please check first in our [thesis](https://www.researchgate.net/publication/345308878_Cooking_recipes_generator_utilizing_a_deep_learning-based_language_model), which is much more detailed. In case of further questions, you're invited to send us a github issue, we will respond as fast as we can!


model name: recipenlg


model author: mbien


task: text-generation


tags: pytorch, jax, gpt2, text-generation, transformers


language: pl
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: mbien/wav2vec2-large-xlsr-polish
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice pl
type: common_voice
args: pl
metrics:
    - name: Test WER
type: wer
value: 23.01


# Wav2Vec2-Large-XLSR-53-Polish


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Polish using the [Common Voice](https://huggingface.co/datasets/common_voice) dataset.
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("common_voice", "pl", split="test[:2%]")

processor = Wav2Vec2Processor.from_pretrained("mbien/wav2vec2-large-xlsr-polish")
model = Wav2Vec2ForCTC.from_pretrained("mbien/wav2vec2-large-xlsr-polish")

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
	speech_array, sampling_rate = torchaudio.load(batch["path"])
	batch["speech"] = resampler(speech_array).squeeze().numpy()
	return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
	logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Polish test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "pl", split="test")
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("mbien/wav2vec2-large-xlsr-polish")
model = Wav2Vec2ForCTC.from_pretrained("mbien/wav2vec2-large-xlsr-polish")
model.to("cuda")

chars_to_ignore_regex = '[\—\…\,\?\.\!\-\;\:\"\“\„\%\‘\”\�\«\»\'\’]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
	batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
	speech_array, sampling_rate = torchaudio.load(batch["path"])
	batch["speech"] = resampler(speech_array).squeeze().numpy()
	return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
	inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

	with torch.no_grad():
		logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

	pred_ids = torch.argmax(logits, dim=-1)
	batch["pred_strings"] = processor.batch_decode(pred_ids)
	return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 23.01 %


## Training


The Common Voice `train`, `validation` datasets were used for training.


The script used for training can be found [here](https://colab.research.google.com/drive/1DvrFMoKp9h3zk_eXrJF2s4_TGDHh0tMc?usp=sharing)


model name: wav2vec2-large-xlsr-polish


model author: mbien


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, pl, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


# fdh-wikibio


Model used to prepare Biography Generator for EPFL Foundations of Digital Humanities course.


## Project description


Please read our report on FDH page: http://fdh.epfl.ch/index.php/WikiBio


## Project result


You're invited to read through our generated biographies!
https://wikibio.mbien.pl/


model name: fdh-wikibio


task: text-generation


# Predicting music popularity using DNNs


This is a model fine-tuned for music popularity classification, created as part of DH-401: Digital Musicology class on EPFL


## Team


* Elisa (elisa.michelet@epfl.ch)
* Michał (michal.bien@epfl.ch)
* Noé (noe.durandard@epfl.ch)


## Milestone 3


Main notebook presenting out results is available [here](https://nbviewer.jupyter.org/github/Glorf/DH-401/blob/main/milestone3.ipynb)


Notebook describing the details of Wav2Vec2.0 pre-training and fine-tuning for the task is available [here](https://nbviewer.jupyter.org/github/Glorf/DH-401/blob/main/milestone3-wav2vec2.ipynb)


## Milestone 2


Exploratory data analysis notebook is available [here](https://nbviewer.jupyter.org/github/Glorf/DH-401/blob/main/milestone2.ipynb)


## Milestone 1


Refined project proposal is available [here](https://github.com/Glorf/DH-401/blob/main/milestone0.md)


## Milestone 0


Original project proposal is available in git history [here](https://github.com/Glorf/DH-401/blob/bb14813ff2bbbd9cdc6b6eecf34c9e3c160598eb/milestone0.md)


model name: fma2vec2popularity


model author: mbien


tags: pytorch, wav2vec2, transformers


This is a pre-trained wav2vec2.0 model, trained on a fill Free Music Archive repository, created as part of DH-401: Digital Musicology class on EPFL


model name: fma2vec


task: feature-extraction


tags: pytorch, wav2vec2, feature-extraction, transformers


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- cosmos_qa
metrics:
- accuracy
model-index:
- name: t5_small_cosmos_qa
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# t5_small_cosmos_qa


This model is a fine-tuned version of [mamlong34/t5_small_race_mutlirc](https://huggingface.co/mamlong34/t5_small_race_mutlirc) on the cosmos_qa dataset.
It achieves the following results on the evaluation set:


- Loss: 0.5614
- Accuracy: 0.6067


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 3.0


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 0.4811        | 1.0   | 3158 | 0.5445          | 0.5548   |
| 0.4428        | 2.0   | 6316 | 0.5302          | 0.5836   |
| 0.3805        | 3.0   | 9474 | 0.5614          | 0.6067   |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.1
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: t5_small_cosmos_qa


model author: mamlong34


task: text2text-generation


tags: pytorch, t5, text2text-generation, dataset:cosmos_qa, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- race
metrics:
- accuracy
model-index:
- name: t5_base_race_cosmos_qa
results: []


# t5_base_race_cosmos_qa


This model is a fine-tuned version of [t5-base](https://huggingface.co/t5-base) on the race dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4414
- Accuracy: 0.7424


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 3.0


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Accuracy |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|
| 0.4355        | 1.0   | 10984 | 0.3910          | 0.7072   |
| 0.3233        | 2.0   | 21968 | 0.3833          | 0.7321   |
| 0.229         | 3.0   | 32952 | 0.4414          | 0.7424   |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: t5_base_race_cosmos_qa


model author: mamlong34


task: text2text-generation


tags: pytorch, t5, text2text-generation, dataset:race, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
metrics:
- accuracy
model-index:
- name: t5_small_race_mutlirc
results: []


# t5_small_race_mutlirc


This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.5760
- Accuracy: 0.5259


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step  | Accuracy | Validation Loss |
|:-------------:|:-----:|:-----:|:--------:|:---------------:|
| 0.6043        | 1.0   | 14141 | 0.4832   | 0.5925          |
| 0.5647        | 2.0   | 28282 | 0.5152   | 0.5659          |
| 0.5237        | 3.0   | 42423 | 0.5760   | 0.5259          |


### Framework versions


model name: t5_small_race_mutlirc


model author: mamlong34


task: text2text-generation


tags: pytorch, t5, text2text-generation, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- race
metrics:
- accuracy
model-index:
- name: t5_large_race_cosmos_qa
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# t5_large_race_cosmos_qa


This model is a fine-tuned version of [t5-large](https://huggingface.co/t5-large) on the race dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4382
- Accuracy: 0.8023


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 1e-05
- train_batch_size: 4
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 3.0


### Training results


| Training Loss | Epoch | Step  | Accuracy | Validation Loss |
|:-------------:|:-----:|:-----:|:--------:|:---------------:|
| 0.3513        | 1.0   | 10983 | 0.7714   | 0.3165          |
| 0.2109        | 2.0   | 21966 | 0.7986   | 0.3329          |
| 0.0929        | 3.0   | 32949 | 0.4382   | 0.8023          |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0
- Datasets 1.14.0
- Tokenizers 0.10.3


model name: t5_large_race_cosmos_qa


model author: mamlong34


task: text2text-generation


language: multilingual
datasets:


- common_voice
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
- robust-speech-event
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 for 56 language by Voidful
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice
type: common_voice
metrics:
    - name: Test CER
type: cer
value: 23.21


# wav2vec2-xlsr-multilingual-56


*56 language, 1 model Multilingual ASR*


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on 56 language using the [Common Voice](https://huggingface.co/datasets/common_voice).\
When using this model, make sure that your speech input is sampled at 16kHz.


For more detail: [https://github.com/voidful/wav2vec2-xlsr-multilingual-56](https://github.com/voidful/wav2vec2-xlsr-multilingual-56)


## Env setup:


```
!pip install torchaudio
!pip install datasets transformers
!pip install asrp
!wget -O lang_ids.pk https://huggingface.co/voidful/wav2vec2-xlsr-multilingual-56/raw/main/lang_ids.pk
```


## Usage


```
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
    AutoTokenizer,
    AutoModelWithLMHead
)
import torch
import re
import sys
import soundfile as sf
model_name = "voidful/wav2vec2-xlsr-multilingual-56"
device = "cuda"
processor_name = "voidful/wav2vec2-xlsr-multilingual-56"

import pickle
with open("lang_ids.pk", 'rb') as output:
    lang_ids = pickle.load(output)

model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

model.eval()

def load_file_to_data(file,sampling_rate=16_000):
    batch = {}
    speech, _ = torchaudio.load(file)
    if sampling_rate != '16_000' or sampling_rate != '16000':
        resampler = torchaudio.transforms.Resample(orig_freq=sampling_rate, new_freq=16_000)
        batch["speech"] = resampler.forward(speech.squeeze(0)).numpy()
        batch["sampling_rate"] = resampler.new_freq
    else:
        batch["speech"] = speech.squeeze(0).numpy()
        batch["sampling_rate"] = '16000'
    return batch

def predict(data):
    features = processor(data["speech"], sampling_rate=data["sampling_rate"], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits
        decoded_results = []
        for logit in logits:
            pred_ids = torch.argmax(logit, dim=-1)
            mask = pred_ids.ge(1).unsqueeze(-1).expand(logit.size())
            vocab_size = logit.size()[-1]
            voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
            comb_pred_ids = torch.argmax(voice_prob, dim=-1)
            decoded_results.append(processor.decode(comb_pred_ids))

    return decoded_results

def predict_lang_specific(data,lang_code):
    features = processor(data["speech"], sampling_rate=data["sampling_rate"], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits
        decoded_results = []
        for logit in logits:
            pred_ids = torch.argmax(logit, dim=-1)
            mask = ~pred_ids.eq(processor.tokenizer.pad_token_id).unsqueeze(-1).expand(logit.size())
            vocab_size = logit.size()[-1]
            voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
            filtered_input = pred_ids[pred_ids!=processor.tokenizer.pad_token_id].view(1,-1).to(device)
            if len(filtered_input[0]) == 0:
                decoded_results.append("")
            else:
                lang_mask = torch.empty(voice_prob.shape[-1]).fill_(0)
                lang_index = torch.tensor(sorted(lang_ids[lang_code]))
                lang_mask.index_fill_(0, lang_index, 1)
                lang_mask = lang_mask.to(device)
                comb_pred_ids = torch.argmax(lang_mask*voice_prob, dim=-1)
                decoded_results.append(processor.decode(comb_pred_ids))

    return decoded_results

predict(load_file_to_data('audio file path',sampling_rate=16_000)) # beware of the audio file sampling rate

predict_lang_specific(load_file_to_data('audio file path',sampling_rate=16_000),'en') # beware of the audio file sampling rate

```


## Result


| Common Voice Languages | Num. of data | Hour   | WER    | CER   |
|------------------------|--------------|--------|--------|-------|
| ar                     | 21744        | 81.5   | 75.29  | 31.23 |
| as                     | 394          | 1.1    | 95.37  | 46.05 |
| br                     | 4777         | 7.4    | 93.79  | 41.16 |
| ca                     | 301308       | 692.8  | 24.80  | 10.39 |
| cnh                    | 1563         | 2.4    | 68.11  | 23.10 |
| cs                     | 9773         | 39.5   | 67.86  | 12.57 |
| cv                     | 1749         | 5.9    | 95.43  | 34.03 |
| cy                     | 11615        | 106.7  | 67.03  | 23.97 |
| de                     | 262113       | 822.8  | 27.03  | 6.50  |
| dv                     | 4757         | 18.6   | 92.16  | 30.15 |
| el                     | 3717         | 11.1   | 94.48  | 58.67 |
| en                     | 580501       | 1763.6 | 34.87  | 14.84 |
| eo                     | 28574        | 162.3  | 37.77  | 6.23  |
| es                     | 176902       | 337.7  | 19.63  | 5.41  |
| et                     | 5473         | 35.9   | 86.87  | 20.79 |
| eu                     | 12677        | 90.2   | 44.80  | 7.32  |
| fa                     | 12806        | 290.6  | 53.81  | 15.09 |
| fi                     | 875          | 2.6    | 93.78  | 27.57 |
| fr                     | 314745       | 664.1  | 33.16  | 13.94 |
| fy-NL                  | 6717         | 27.2   | 72.54  | 26.58 |
| ga-IE                  | 1038         | 3.5    | 92.57  | 51.02 |
| hi                     | 292          | 2.0    | 90.95  | 57.43 |
| hsb                    | 980          | 2.3    | 89.44  | 27.19 |
| hu                     | 4782         | 9.3    | 97.15  | 36.75 |
| ia                     | 5078         | 10.4   | 52.00  | 11.35 |
| id                     | 3965         | 9.9    | 82.50  | 22.82 |
| it                     | 70943        | 178.0  | 39.09  | 8.72  |
| ja                     | 1308         | 8.2    | 99.21  | 62.06 |
| ka                     | 1585         | 4.0    | 90.53  | 18.57 |
| ky                     | 3466         | 12.2   | 76.53  | 19.80 |
| lg                     | 1634         | 17.1   | 98.95  | 43.84 |
| lt                     | 1175         | 3.9    | 92.61  | 26.81 |
| lv                     | 4554         | 6.3    | 90.34  | 30.81 |
| mn                     | 4020         | 11.6   | 82.68  | 30.14 |
| mt                     | 3552         | 7.8    | 84.18  | 22.96 |
| nl                     | 14398        | 71.8   | 57.18  | 19.01 |
| or                     | 517          | 0.9    | 90.93  | 27.34 |
| pa-IN                  | 255          | 0.8    | 87.95  | 42.03 |
| pl                     | 12621        | 112.0  | 56.14  | 12.06 |
| pt                     | 11106        | 61.3   | 53.24  | 16.32 |
| rm-sursilv             | 2589         | 5.9    | 78.17  | 23.31 |
| rm-vallader            | 931          | 2.3    | 73.67  | 21.76 |
| ro                     | 4257         | 8.7    | 83.84  | 21.95 |
| ru                     | 23444        | 119.1  | 61.83  | 15.18 |
| sah                    | 1847         | 4.4    | 94.38  | 38.46 |
| sl                     | 2594         | 6.7    | 84.21  | 20.54 |
| sv-SE                  | 4350         | 20.8   | 83.68  | 30.79 |
| ta                     | 3788         | 18.4   | 84.19  | 21.60 |
| th                     | 4839         | 11.7   | 141.87 | 37.16 |
| tr                     | 3478         | 22.3   | 66.77  | 15.55 |
| tt                     | 13338        | 26.7   | 86.80  | 33.57 |
| uk                     | 7271         | 39.4   | 70.23  | 14.34 |
| vi                     | 421          | 1.7    | 96.06  | 66.25 |
| zh-CN                  | 27284        | 58.7   | 89.67  | 23.96 |
| zh-HK                  | 12678        | 92.1   | 81.77  | 18.82 |
| zh-TW                  | 6402         | 56.6   | 85.08  | 29.07 |


model name: wav2vec2-xlsr-multilingual-56


model author: voidful


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, multilingual, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, robust-speech-event, license:apache-2.0, model-index


## license: afl-3.0


model name: part-1100000


task: text2text-generation


tags: pytorch, bart, text2text-generation, transformers, license:afl-3.0, autonlp_compatible


language: en
tags:


- bart
- distractor
- generation
- seq2seq
datasets:
- race
metrics:
- bleu
- rouge
pipeline_tag: text2text-generation
widget:
- text: "When you ' re having a holiday , one of the main questions to ask is which hotel or apartment to choose . However , when it comes to France , you have another special choice : treehouses . In France , treehouses are offered to travelers as a new choice in many places . The price may be a little higher , but you do have a chance to _ your childhood memories . Alain Laurens , one of France ' s top treehouse designers , said , ' Most of the people might have the experience of building a den when they were young . And they like that feeling of freedom when they are children . ' Its fairy - tale style gives travelers a special feeling . It seems as if they are living as a forest king and enjoying the fresh air in the morning . Another kind of treehouse is the ' star cube ' . It gives travelers the chance of looking at the stars shining in the sky when they are going to sleep . Each ' star cube ' not only offers all the comfortable things that a hotel provides for travelers , but also gives them a chance to look for stars by using a telescope . The glass roof allows you to look at the stars from your bed . </s> The passage mainly tells us </s> treehouses in france."


# bart-distractor-generation-pm


## Model description


This model is a sequence-to-sequence distractor generator which takes an answer, question and context as an input, and generates a distractor as an output. It is based on a pretrained `bart-base` model.\
This model trained with Parallel MLM refer to the [Paper](https://www.aclweb.org/anthology/2020.findings-emnlp.393/).\
For details, please see https://github.com/voidful/BDG.


## Intended uses & limitations


The model is trained to generate examinations-style multiple choice distractor. The model performs best with full sentence answers.


#### How to use


The model takes concatenated context, question and answers as an input sequence, and will generate a full distractor sentence as an output sequence. The max sequence length is 1024 tokens. Inputs should be organised into the following format:


```
context </s> question </s> answer
```


The input sequence can then be encoded and passed as the `input_ids` argument in the model's `generate()` method.


#### Limitations and bias


The model is limited to generating distractor in the same style as those found in [RACE](https://www.aclweb.org/anthology/D17-1082/). The generated distractors can potentially be leading or reflect biases that are present in the context. If the context is too short or completely absent, or if the context, question and answer do not match, the generated distractor is likely to be incoherent.


model name: bart-distractor-generation-pm


model author: voidful


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, dataset:race, transformers, distractor, generation, seq2seq, autonlp_compatible


language: en
tags:


- bart
- question
- generation
- seq2seq
datasets:
- unifiedQA
metrics:
- bleu
- rouge
pipeline_tag: text2text-generation
widget:
- text: "treehouses in france. \n When you ' re having a holiday , one of the main questions to ask is which hotel or apartment to choose . However , when it comes to France , you have another special choice : treehouses . In France , treehouses are offered to travelers as a new choice in many places . The price may be a little higher , but you do have a chance to _ your childhood memories . Alain Laurens , one of France ' s top treehouse designers , said , ' Most of the people might have the experience of building a den when they were young . And they like that feeling of freedom when they are children . ' Its fairy - tale style gives travelers a special feeling . It seems as if they are living as a forest king and enjoying the fresh air in the morning . Another kind of treehouse is the ' star cube ' . It gives travelers the chance of looking at the stars shining in the sky when they are going to sleep . Each ' star cube ' not only offers all the comfortable things that a hotel provides for travelers , but also gives them a chance to look for stars by using a telescope . The glass roof allows you to look at the stars from your bed ."


# unifiedqg-bart-base


## Model description


This model is a sequence-to-sequence question generator which takes an answer and context as an input, and generates a question as an output.\
It is based on a pretrained `bart-base` model.


The model takes concatenated context and answers as an input sequence, and will generate a full distractor sentence as an output sequence. The max sequence length is 1024 tokens. Inputs should be organised into the following format:


```
answer \n context
```


model name: unifiedqg-bart-base


tags: pytorch, bart, text2text-generation, en, dataset:unifiedQA, transformers, question, generation, seq2seq, autonlp_compatible


language: multilingual
datasets:


- NQ
- Trivia
- SQuAD
- MLQA
- DRCD


# dpr-ctx_encoder-bert-base-multilingual


## Description


Multilingual DPR Model base on bert-base-multilingual-cased.
[DPR model](https://arxiv.org/abs/2004.04906)
[DPR repo](https://github.com/facebookresearch/DPR)


## Data


1. [NQ](https://github.com/facebookresearch/DPR/blob/master/data/download_data.py)
1. [Trivia](https://github.com/facebookresearch/DPR/blob/master/data/download_data.py)
1. [SQuAD](https://github.com/facebookresearch/DPR/blob/master/data/download_data.py)
1. [DRCD*](https://github.com/DRCKnowledgeTeam/DRCD)
1. [MLQA*](https://github.com/facebookresearch/MLQA)


`question pairs for train`： 644,217\
`question pairs for dev`： 73,710


*DRCD and MLQA are converted using script from haystack [squad_to_dpr.py](https://github.com/deepset-ai/haystack/blob/master/haystack/retriever/squad_to_dpr.py)


## Training Script


I use the script from [haystack](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial9_DPR_training.ipynb)


## Usage


```python
from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer
tokenizer = DPRQuestionEncoderTokenizer.from_pretrained('voidful/dpr-question_encoder-bert-base-multilingual')
model = DPRQuestionEncoder.from_pretrained('voidful/dpr-question_encoder-bert-base-multilingual')
input_ids = tokenizer("Hello, is my dog cute ?", return_tensors='pt')["input_ids"]
embeddings = model(input_ids).pooler_output
```


Follow the tutorial from `haystack`:
[Better Retrievers via "Dense Passage Retrieval"](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial6_Better_Retrieval_via_DPR.ipynb)


```
from haystack.retriever.dense import DensePassageRetriever
retriever = DensePassageRetriever(document_store=document_store,
                                  query_embedding_model="voidful/dpr-question_encoder-bert-base-multilingual",
                                  passage_embedding_model="voidful/dpr-ctx_encoder-bert-base-multilingual",
                                  max_seq_len_query=64,
                                  max_seq_len_passage=256,
                                  batch_size=16,
                                  use_gpu=True,
                                  embed_title=True,
                                  use_fast_tokenizers=True)
```


model name: dpr-question_encoder-bert-base-multilingual


model author: voidful


task: feature-extraction


tags: pytorch, dpr, feature-extraction, multilingual, dataset:NQ, dataset:Trivia, dataset:SQuAD, dataset:MLQA, dataset:DRCD, arxiv:2004.04906, transformers


language: en
tags:


- bart
- distractor
- generation
- seq2seq
datasets:
- race
metrics:
- bleu
- rouge
pipeline_tag: text2text-generation
widget:
- text: "When you ' re having a holiday , one of the main questions to ask is which hotel or apartment to choose . However , when it comes to France , you have another special choice : treehouses . In France , treehouses are offered to travelers as a new choice in many places . The price may be a little higher , but you do have a chance to _ your childhood memories . Alain Laurens , one of France ' s top treehouse designers , said , ' Most of the people might have the experience of building a den when they were young . And they like that feeling of freedom when they are children . ' Its fairy - tale style gives travelers a special feeling . It seems as if they are living as a forest king and enjoying the fresh air in the morning . Another kind of treehouse is the ' star cube ' . It gives travelers the chance of looking at the stars shining in the sky when they are going to sleep . Each ' star cube ' not only offers all the comfortable things that a hotel provides for travelers , but also gives them a chance to look for stars by using a telescope . The glass roof allows you to look at the stars from your bed . </s> The passage mainly tells us </s> treehouses in france."


# bart-distractor-generation


## Model description


This model is a sequence-to-sequence distractor generator which takes an answer, question and context as an input, and generates a distractor as an output. It is based on a pretrained `bart-base` model.\
For details, please see https://github.com/voidful/BDG.


## Intended uses & limitations


The model is trained to generate examinations-style multiple choice distractor. The model performs best with full sentence answers.


#### How to use


The model takes concatenated context, question and answers as an input sequence, and will generate a full distractor sentence as an output sequence. The max sequence length is 1024 tokens. Inputs should be organised into the following format:


```
context </s> question </s> answer
```


The input sequence can then be encoded and passed as the `input_ids` argument in the model's `generate()` method.


For details, please see https://github.com/voidful/BDG.


#### Limitations and bias


The model is limited to generating distractor in the same style as those found in [RACE](https://www.aclweb.org/anthology/D17-1082/). The generated distractors can potentially be leading or reflect biases that are present in the context. If the context is too short or completely absent, or if the context, question and answer do not match, the generated distractor is likely to be incoherent.


model name: bart-distractor-generation


model author: voidful


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, dataset:race, transformers, distractor, generation, seq2seq, autonlp_compatible


language: en
tags:


- bart
- question
- generation
- seq2seq
datasets:
- unifiedQA
metrics:
- bleu
- rouge
pipeline_tag: text2text-generation
widget:
- text: "Harry Potter is a series of seven fantasy novels written by British author J. K. Rowling. The novels chronicle the lives of a young wizard, Harry Potter, and his friends Hermione Granger and Ron Weasley, all of whom are students at Hogwarts School of Witchcraft and Wizardry. The main story arc concerns Harry's struggle against Lord Voldemort, a dark wizard who intends to become immortal, overthrow the wizard governing body known as the Ministry of Magic and subjugate all wizards and Muggles(non-magical people)."


# context-only-question-generator


## Model description


This model is a sequence-to-sequence question generator which takes context as an input, and generates a question as an output.\
It is based on a pretrained `bart-base` model.


Inputs should be organised into the following format:


```
context
```


model name: context-only-question-generator


tags: pytorch, bart, text2text-generation, en, dataset:unifiedQA, transformers, question, generation, seq2seq, autonlp_compatible


language: en
datasets:


- librispeech
tags:
- audio
- automatic-speech-recognition
- speech
- asr
- hubert
license: apache-2.0
metrics:
- wer
- cer


# voidful/tts_hubert_cluster_bart_base


## Usage


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("voidful/tts_hubert_cluster_bart_base")
model = AutoModelForSeq2SeqLM.from_pretrained("voidful/tts_hubert_cluster_bart_base")
```


generate output


```python
gen_output = model.generate(input_ids=tokenizer("going along slushy country roads and speaking to damp audience in drifty school rooms day after day for a fortnight he'll have to put in an appearance at some place of worship on sunday morning and he can come to ask immediately afterwards",return_tensors='pt').input_ids, max_length=1024)
print(tokenizer.decode(gen_output[0], skip_special_tokens=True))
```


## Result


`:vtok402::vtok329::vtok329::vtok75::vtok75::vtok75::vtok44::vtok150::vtok150::vtok222::vtok280::vtok280::vtok138::vtok409::vtok409::vtok409::vtok46::vtok441:`


model name: tts_hubert_cluster_bart_base


model author: voidful


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, dataset:librispeech, transformers, audio, automatic-speech-recognition, speech, asr, hubert, license:apache-2.0, autonlp_compatible


language: zh
pipeline_tag: fill-mask
widget:


- text: "今天[MASK]情很好"


# albert_chinese_xlarge


This a albert_chinese_xlarge model from [Google's github](https://github.com/google-research/ALBERT)\
converted by huggingface's [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py)


## Notice


*Support AutoTokenizer*


Since sentencepiece is not used in albert_chinese_base model\
you have to call BertTokenizer instead of AlbertTokenizer !!!\
we can eval it using an example on MaskedLM


由於 albert_chinese_base 模型沒有用 sentencepiece\
用AlbertTokenizer會載不進詞表，因此需要改用BertTokenizer !!!\
我們可以跑MaskedLM預測來驗證這個做法是否正確


## Justify (驗證有效性)


```python
from transformers import AutoTokenizer, AlbertForMaskedLM
import torch
from torch.nn.functional import softmax

pretrained = 'voidful/albert_chinese_xlarge'
tokenizer = AutoTokenizer.from_pretrained(pretrained)
model = AlbertForMaskedLM.from_pretrained(pretrained)

inputtext = "今天[MASK]情很好"

maskpos = tokenizer.encode(inputtext, add_special_tokens=True).index(103)

input_ids = torch.tensor(tokenizer.encode(inputtext, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=input_ids)
loss, prediction_scores = outputs[:2]
logit_prob = softmax(prediction_scores[0, maskpos],dim=-1).data.tolist()
predicted_index = torch.argmax(prediction_scores[0, maskpos]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token, logit_prob[predicted_index])
```


Result: `心 0.9942440390586853`


model name: albert_chinese_xlarge


task: fill-mask


tags: pytorch, albert, fill-mask, zh, transformers, autonlp_compatible, infinity_compatible


# albert_chinese_small


This a albert_chinese_small model from [brightmart/albert_zh project](https://github.com/brightmart/albert_zh), albert_small_google_zh model\
converted by huggingface's [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py)


```python
from transformers import AutoTokenizer, AlbertForMaskedLM
import torch
from torch.nn.functional import softmax

pretrained = 'voidful/albert_chinese_small'
tokenizer = AutoTokenizer.from_pretrained(pretrained)
model = AlbertForMaskedLM.from_pretrained(pretrained)

inputtext = "今天[MASK]情很好"

maskpos = tokenizer.encode(inputtext, add_special_tokens=True).index(103)

input_ids = torch.tensor(tokenizer.encode(inputtext, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=input_ids)
loss, prediction_scores = outputs[:2]
logit_prob = softmax(prediction_scores[0, maskpos],dim=-1).data.tolist()
predicted_index = torch.argmax(prediction_scores[0, maskpos]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token, logit_prob[predicted_index])
```


Result: `感 0.6390823125839233`


model name: albert_chinese_small


model author: voidful


- text: "今天[MASK]情很好"


# albert_chinese_base


This a albert_chinese_base model from [Google's github](https://github.com/google-research/ALBERT)\
converted by huggingface's [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py)


## Notice


*Support AutoTokenizer*


## Justify (驗證有效性)


```python
from transformers import AutoTokenizer, AlbertForMaskedLM
import torch
from torch.nn.functional import softmax

pretrained = 'voidful/albert_chinese_base'
tokenizer = AutoTokenizer.from_pretrained(pretrained)
model = AlbertForMaskedLM.from_pretrained(pretrained)

inputtext = "今天[MASK]情很好"

maskpos = tokenizer.encode(inputtext, add_special_tokens=True).index(103)

input_ids = torch.tensor(tokenizer.encode(inputtext, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=input_ids)
loss, prediction_scores = outputs[:2]
logit_prob = softmax(prediction_scores[0, maskpos],dim=-1).data.tolist()
predicted_index = torch.argmax(prediction_scores[0, maskpos]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token, logit_prob[predicted_index])
```


Result: `感 0.36333346366882324`


model name: albert_chinese_base


task: fill-mask


language: en
tags:


- bart
- question
- generation
- seq2seq
datasets:
- eqg-race
metrics:
- bleu
- rouge
pipeline_tag: text2text-generation
widget:
- text: "When you ' re having a holiday , one of the main questions to ask is which hotel or apartment to choose . However , when it comes to France , you have another special choice : treehouses . In France , treehouses are offered to travelers as a new choice in many places . The price may be a little higher , but you do have a chance to _ your childhood memories . Alain Laurens , one of France ' s top treehouse designers , said , ' Most of the people might have the experience of building a den when they were young . And they like that feeling of freedom when they are children . ' Its fairy - tale style gives travelers a special feeling . It seems as if they are living as a forest king and enjoying the fresh air in the morning . Another kind of treehouse is the ' star cube ' . It gives travelers the chance of looking at the stars shining in the sky when they are going to sleep . Each ' star cube ' not only offers all the comfortable things that a hotel provides for travelers , but also gives them a chance to look for stars by using a telescope . The glass roof allows you to look at the stars from your bed . "


# voidful/bart-eqg-question-generator


## Model description


This model is a sequence-to-sequence question generator with only the context as an input, and generates a  question as an output.\
It is based on a pretrained `bart-base` model, and trained on [EQG-RACE](https://github.com/jemmryx/EQG-RACE) corpus.


## Intended uses & limitations


The model is trained to generate examinations-style multiple choice question.


#### How to use


The model takes context as an input sequence, and will generate a question as an output sequence. The max sequence length is 1024 tokens. Inputs should be organised into the following format:


```
context
```


The input sequence can then be encoded and passed as the `input_ids` argument in the model's `generate()` method.


model name: bart-eqg-question-generator


model author: voidful


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, dataset:eqg-race, transformers, question, generation, seq2seq, autonlp_compatible


language: en
datasets:


- librispeech
tags:
- audio
- automatic-speech-recognition
- speech
- asr
- hubert
license: apache-2.0
metrics:
- wer
- cer


# voidful/asr_hubert_cluster_bart_base


## Usage


download file


```shell
wget https://raw.githubusercontent.com/voidful/hubert-cluster-code/main/km_feat_100_layer_20
wget https://cdn-media.huggingface.co/speech_samples/sample1.flac
```


Hubert kmeans code


```python
import joblib
import torch
from transformers import Wav2Vec2FeatureExtractor, HubertModel
import soundfile as sf

class HubertCode(object):
    def __init__(self, hubert_model, km_path, km_layer):
        self.processor = Wav2Vec2FeatureExtractor.from_pretrained(hubert_model)
        self.model = HubertModel.from_pretrained(hubert_model)
        self.km_model = joblib.load(km_path)
        self.km_layer = km_layer
        self.C_np = self.km_model.cluster_centers_.transpose()
        self.Cnorm_np = (self.C_np ** 2).sum(0, keepdims=True)

        self.C = torch.from_numpy(self.C_np)
        self.Cnorm = torch.from_numpy(self.Cnorm_np)
        if torch.cuda.is_available():
            self.C = self.C.cuda()
            self.Cnorm = self.Cnorm.cuda()
            self.model = self.model.cuda()

    def __call__(self, filepath, sampling_rate=None):
        speech, sr = sf.read(filepath)
        input_values = self.processor(speech, return_tensors="pt", sampling_rate=sr).input_values
        if torch.cuda.is_available():
            input_values = input_values.cuda()
        hidden_states = self.model(input_values, output_hidden_states=True).hidden_states
        x = hidden_states[self.km_layer].squeeze()
        dist = (
                x.pow(2).sum(1, keepdim=True)
                - 2 * torch.matmul(x, self.C)
                + self.Cnorm
        )
        return dist.argmin(dim=1).cpu().numpy()
```


input


```python
hc = HubertCode("facebook/hubert-large-ll60k", './km_feat_100_layer_20', 20)
voice_ids = hc('./sample1.flac')
```


bart model


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("voidful/asr_hubert_cluster_bart_base")
model = AutoModelForSeq2SeqLM.from_pretrained("voidful/asr_hubert_cluster_bart_base")
```


generate output


```python
gen_output = model.generate(input_ids=tokenizer("".join([f":vtok{i}:" for i in voice_ids]),return_tensors='pt').input_ids,max_length=1024)
print(tokenizer.decode(gen_output[0], skip_special_tokens=True))
```


## Result


`going along slushy country roads and speaking to damp audience in drifty school rooms day after day for a fortnight he'll have to put in an appearance at some place of worship on sunday morning and he can come to ask immediately afterwards`


model name: asr_hubert_cluster_bart_base


model author: voidful


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, dataset:librispeech, transformers, audio, automatic-speech-recognition, speech, asr, hubert, license:apache-2.0, autonlp_compatible


language: en
tags:


- bart
- distractor
- generation
- seq2seq
datasets:
- race
metrics:
- bleu
- rouge
pipeline_tag: text2text-generation
widget:
- text: "When you ' re having a holiday , one of the main questions to ask is which hotel or apartment to choose . However , when it comes to France , you have another special choice : treehouses . In France , treehouses are offered to travelers as a new choice in many places . The price may be a little higher , but you do have a chance to _ your childhood memories . Alain Laurens , one of France ' s top treehouse designers , said , ' Most of the people might have the experience of building a den when they were young . And they like that feeling of freedom when they are children . ' Its fairy - tale style gives travelers a special feeling . It seems as if they are living as a forest king and enjoying the fresh air in the morning . Another kind of treehouse is the ' star cube ' . It gives travelers the chance of looking at the stars shining in the sky when they are going to sleep . Each ' star cube ' not only offers all the comfortable things that a hotel provides for travelers , but also gives them a chance to look for stars by using a telescope . The glass roof allows you to look at the stars from your bed . </s> The passage mainly tells us </s> treehouses in france."


# bart-distractor-generation-both


## Model description


This model is a sequence-to-sequence distractor generator which takes an answer, question and context as an input, and generates a distractor as an output. It is based on a pretrained `bart-base` model.\
This model trained with Parallel MLM & Answer Negative Regularization refer to the [Paper](https://www.aclweb.org/anthology/2020.findings-emnlp.393/).\
For details, please see https://github.com/voidful/BDG.


## Intended uses & limitations


The model is trained to generate examinations-style multiple choice distractor. The model performs best with full sentence answers.


#### How to use


The model takes concatenated context, question and answers as an input sequence, and will generate a full distractor sentence as an output sequence. The max sequence length is 1024 tokens. Inputs should be organised into the following format:


```
context </s> question </s> answer
```


For details, please see https://github.com/voidful/BDG.


#### Limitations and bias


The model is limited to generating distractor in the same style as those found in [RACE](https://www.aclweb.org/anthology/D17-1082/). The generated distractors can potentially be leading or reflect biases that are present in the context. If the context is too short or completely absent, or if the context, question and answer do not match, the generated distractor is likely to be incoherent.


model name: bart-distractor-generation-both


model author: voidful


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, dataset:race, transformers, distractor, generation, seq2seq, autonlp_compatible


language: zh
datasets:


- common_voice
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
- robust-speech-event
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Taiwanese Mandarin(zh-tw) by Voidful
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice zh-TW
type: common_voice
args: zh-TW
metrics:
    - name: Test CER
type: cer
value: 18.36


# Wav2Vec2-Large-XLSR-53-tw-gpt


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on zh-tw using the [Common Voice](https://huggingface.co/datasets/common_voice).\
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


[Colab trial](https://colab.research.google.com/drive/1e_z5jQHYbO2YKEaUgzb1ww1WwiAyydAj?usp=sharing)


```
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
    AutoTokenizer,
    AutoModelWithLMHead
)
import torch
import re
import sys

model_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
device = "cuda"
processor_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"

chars_to_ignore_regex = r"[¥•＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､　、〃〈〉《》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏﹑﹔·'℃°•·．﹑︰〈〉─《﹖﹣﹂﹁﹔！？｡。＂＃＄％＆＇（）＊＋，﹐－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏.．!\"#$%&()*+,\-.\:;<=>?@\[\]\\\/^_`{|}~]"

model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

tokenizer = AutoTokenizer.from_pretrained("ckiplab/gpt2-base-chinese")
gpt_model = AutoModelWithLMHead.from_pretrained("ckiplab/gpt2-base-chinese").to(device)

resampler = torchaudio.transforms.Resample(orig_freq=48_000, new_freq=16_000)

def load_file_to_data(file):
    batch = {}
    speech, _ = torchaudio.load(file)
    batch["speech"] = resampler.forward(speech.squeeze(0)).numpy()
    batch["sampling_rate"] = resampler.new_freq
    return batch

def predict(data):
    features = processor(data["speech"], sampling_rate=data["sampling_rate"], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    decoded_results = []
    for logit in logits:
        pred_ids = torch.argmax(logit, dim=-1)
        mask = pred_ids.ge(1).unsqueeze(-1).expand(logit.size())
        vocab_size = logit.size()[-1]
        voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
        gpt_input = torch.cat((torch.tensor([tokenizer.cls_token_id]).to(device),pred_ids[pred_ids>0]), 0)
        gpt_prob = torch.nn.functional.softmax(gpt_model(gpt_input).logits, dim=-1)[:voice_prob.size()[0],:]
        comb_pred_ids = torch.argmax(gpt_prob*voice_prob, dim=-1)
        decoded_results.append(processor.decode(comb_pred_ids))

    return decoded_results
```


Predict


```python
predict(load_file_to_data('voice file path'))
```


## Evaluation


The model can be evaluated as follows on the zh-tw test data of Common Voice.\
CER calculation refer to https://huggingface.co/ctl/wav2vec2-large-xlsr-cantonese


env setup:


```
!pip install editdistance
!pip install torchaudio
!pip install datasets transformers
```


## Evaluation without LM:


```python
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
)
import torch
import re
import sys
from transformers import AutoTokenizer, AutoModelWithLMHead
from datasets import  Audio
from math import log

model_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
device = "cuda"
processor_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
chars_to_ignore_regex = r"[¥•＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､　、〃〈〉《》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏﹑﹔·'℃°•·．﹑︰〈〉─《﹖﹣﹂﹁﹔！？｡。＂＃＄％＆＇（）＊＋，﹐－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏.．!\"#$%&()*+,\-.\:;<=>?@\[\]\\\/^_`{|}~]"

tokenizer = AutoTokenizer.from_pretrained("ckiplab/gpt2-base-chinese")
lm_model = AutoModelWithLMHead.from_pretrained("ckiplab/gpt2-base-chinese").to(device)
model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

ds = load_dataset("common_voice", 'zh-TW', split="test")
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
def map_to_array(batch):
    audio = batch["audio"]
    batch["speech"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
    batch["sampling_rate"] = audio["sampling_rate"]
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower().replace("’", "'")
    return batch
ds = ds.map(map_to_array)

def map_to_pred(batch):
    features = processor(batch["speech"], sampling_rate=batch["sampling_rate"][0], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["predicted"] = processor.batch_decode(pred_ids)
    batch["target"] = batch["sentence"]
    return batch

result = ds.map(map_to_pred, batched=True, batch_size=3, remove_columns=list(ds.features.keys()))

def cer_cal(groundtruth, hypothesis):
    err = 0
    tot = 0
    for p, t in zip(hypothesis, groundtruth):
        err += float(ed.eval(p.lower(), t.lower()))
        tot += len(t)
    return err / tot
print("CER: {:2f}".format(100 * cer_cal(result["target"],result["predicted"])))
```


`CER: 28.70`.\
`TIME: 04:08 min`


## Evaluation with GPT:


```python
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
)
import torch
import re
import sys
from transformers import AutoTokenizer, AutoModelWithLMHead
from datasets import  Audio
from math import log

model_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
device = "cuda"
processor_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
chars_to_ignore_regex = r"[¥•＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､　、〃〈〉《》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏﹑﹔·'℃°•·．﹑︰〈〉─《﹖﹣﹂﹁﹔！？｡。＂＃＄％＆＇（）＊＋，﹐－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏.．!\"#$%&()*+,\-.\:;<=>?@\[\]\\\/^_`{|}~]"

tokenizer = AutoTokenizer.from_pretrained("ckiplab/gpt2-base-chinese")
lm_model = AutoModelWithLMHead.from_pretrained("ckiplab/gpt2-base-chinese").to(device)
model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

ds = load_dataset("common_voice", 'zh-TW', split="test")
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
def map_to_array(batch):
    audio = batch["audio"]
    batch["speech"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
    batch["sampling_rate"] = audio["sampling_rate"]
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower().replace("’", "'")
    return batch
ds = ds.map(map_to_array)

def map_to_pred(batch):
    features = processor(batch["speech"], sampling_rate=batch["sampling_rate"][0], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    decoded_results = []
    for logit in logits:
        pred_ids = torch.argmax(logit, dim=-1)
        mask = pred_ids.ge(1).unsqueeze(-1).expand(logit.size())
        vocab_size = logit.size()[-1]
        voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
        lm_input = torch.cat((torch.tensor([tokenizer.cls_token_id]).to(device),pred_ids[pred_ids>0]), 0)
        lm_prob = torch.nn.functional.softmax(lm_model(lm_input).logits, dim=-1)[:voice_prob.size()[0],:]
        comb_pred_ids = torch.argmax(lm_prob*voice_prob, dim=-1)
        decoded_results.append(processor.decode(comb_pred_ids))

    batch["predicted"] = decoded_results
    batch["target"] = batch["sentence"]
    return batch

result = ds.map(map_to_pred, batched=True, batch_size=3, remove_columns=list(ds.features.keys()))

def cer_cal(groundtruth, hypothesis):
    err = 0
    tot = 0
    for p, t in zip(hypothesis, groundtruth):
        err += float(ed.eval(p.lower(), t.lower()))
        tot += len(t)
    return err / tot
print("CER: {:2f}".format(100 * cer_cal(result["target"],result["predicted"])))
```


`CER 25.70`.\
`TIME: 06:04 min`


## Evaluation with GPT + beam search:


```python
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
)
import torch
import re
import sys
from transformers import AutoTokenizer, AutoModelWithLMHead
from datasets import  Audio
from math import log

model_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
device = "cuda"
processor_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
chars_to_ignore_regex = r"[¥•＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､　、〃〈〉《》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏﹑﹔·'℃°•·．﹑︰〈〉─《﹖﹣﹂﹁﹔！？｡。＂＃＄％＆＇（）＊＋，﹐－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏.．!\"#$%&()*+,\-.\:;<=>?@\[\]\\\/^_`{|}~]"

tokenizer = AutoTokenizer.from_pretrained("ckiplab/gpt2-base-chinese")
lm_model = AutoModelWithLMHead.from_pretrained("ckiplab/gpt2-base-chinese").to(device)
model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

ds = load_dataset("common_voice", 'zh-TW', split="test")
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
def map_to_array(batch):
    audio = batch["audio"]
    batch["speech"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
    batch["sampling_rate"] = audio["sampling_rate"]
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower().replace("’", "'")
    return batch
ds = ds.map(map_to_array)

def map_to_pred(batch):
    features = processor(batch["speech"], sampling_rate=batch["sampling_rate"][0], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    decoded_results = []
    for logit in logits:
        sequences = [[[], 1.0]]
        pred_ids = torch.argmax(logit, dim=-1)
        mask = pred_ids.ge(1).unsqueeze(-1).expand(logit.size())
        vocab_size = logit.size()[-1]
        voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
        while True:
            all_candidates = list()
            exceed = False
            for seq in sequences:
                tokens, score = seq
                gpt_input = torch.tensor([tokenizer.cls_token_id]+tokens).to(device)
                gpt_prob = torch.nn.functional.softmax(lm_model(gpt_input).logits, dim=-1)[:len(gpt_input),:]
                if len(gpt_input) >= len(voice_prob):
                    exceed = True
                comb_pred_ids = gpt_prob*voice_prob[:len(gpt_input)]
                v,i = torch.topk(comb_pred_ids,50,dim=-1)
                for tok_id,tok_prob in zip(i.tolist()[-1],v.tolist()[-1]):
                    candidate = [tokens + [tok_id], score + -log(tok_prob)]
                    all_candidates.append(candidate)
            ordered = sorted(all_candidates, key=lambda tup: tup[1])
            sequences = ordered[:10]
            if exceed:
                break
        decoded_results.append(processor.decode(sequences[0][0]))

    batch["predicted"] = decoded_results
    batch["target"] = batch["sentence"]
    return batch

result = ds.map(map_to_pred, batched=True, batch_size=3, remove_columns=list(ds.features.keys()))

def cer_cal(groundtruth, hypothesis):
    err = 0
    tot = 0
    for p, t in zip(hypothesis, groundtruth):
        err += float(ed.eval(p.lower(), t.lower()))
        tot += len(t)
    return err / tot
print("CER: {:2f}".format(100 * cer_cal(result["target"],result["predicted"])))
```


`CER 18.36`.


## Evaluation with BERT:


```python
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
)
import torch
import re
import sys
from transformers import AutoTokenizer, AutoModelForMaskedLM

model_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
device = "cuda"
processor_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
chars_to_ignore_regex = r"[¥•＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､　、〃〈〉《》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏﹑﹔·'℃°•·．﹑︰〈〉─《﹖﹣﹂﹁﹔！？｡。＂＃＄％＆＇（）＊＋，﹐－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏.．!\"#$%&()*+,\-.\:;<=>?@\[\]\\\/^_`{|}~]"

tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese")
lm_model = AutoModelForMaskedLM.from_pretrained("bert-base-chinese").to(device)
model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

ds = load_dataset("common_voice", 'zh-TW', data_dir="./cv-corpus-6.1-2020-12-11", split="test")

resampler = torchaudio.transforms.Resample(orig_freq=48_000, new_freq=16_000)

def map_to_array(batch):
    speech, _ = torchaudio.load(batch["path"])
    batch["speech"] = resampler.forward(speech.squeeze(0)).numpy()
    batch["sampling_rate"] = resampler.new_freq
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower().replace("’", "'")
    return batch

ds = ds.map(map_to_array)

def map_to_pred(batch):
    features = processor(batch["speech"], sampling_rate=batch["sampling_rate"][0], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    decoded_results = []
    for logit in logits:
        pred_ids = torch.argmax(logit, dim=-1)
        mask = ~pred_ids.eq(tokenizer.pad_token_id).unsqueeze(-1).expand(logit.size())
        vocab_size = logit.size()[-1]
        voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
        lm_input = torch.masked_select(pred_ids, ~pred_ids.eq(tokenizer.pad_token_id)).unsqueeze(0)
        mask_lm_prob = voice_prob.clone()
        for i in range(lm_input.shape[-1]):
            masked_lm_input = lm_input.clone()
            masked_lm_input[0][i] = torch.tensor(tokenizer.mask_token_id).to('cuda')
            lm_prob = torch.nn.functional.softmax(lm_model(masked_lm_input).logits, dim=-1).squeeze(0)
            mask_lm_prob[i] = lm_prob[i]
        comb_pred_ids = torch.argmax(mask_lm_prob*voice_prob, dim=-1)
        decoded_results.append(processor.decode(comb_pred_ids))

    batch["predicted"] = decoded_results
    batch["target"] = batch["sentence"]
    return batch

result = ds.map(map_to_pred, batched=True, batch_size=1, remove_columns=list(ds.features.keys()))

def cer_cal(groundtruth, hypothesis):
    err = 0
    tot = 0
    for p, t in zip(hypothesis, groundtruth):
        err += float(ed.eval(p.lower(), t.lower()))
        tot += len(t)
    return err / tot
print("CER: {:2f}".format(100 * cer_cal(result["target"],result["predicted"])))
```


`CER 25.57`.\
`TIME: 09:49 min`


## Evaluation with T-TA:


setup


```
!git clone https://github.com/voidful/pytorch-tta.git
!mv ./pytorch-tta/tta ./tta
!wget https://github.com/voidful/pytorch-tta/releases/download/wiki_zh/wiki_zh.pt
```


```python
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
)
import torch
import re
import sys
from tta.modeling_tta import TTALMModel
from transformers import AutoTokenizer
import torch

model_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
device = "cuda"
processor_name = "voidful/wav2vec2-large-xlsr-53-tw-gpt"
chars_to_ignore_regex = r"[¥•＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､　、〃〈〉《》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏﹑﹔·'℃°•·．﹑︰〈〉─《﹖﹣﹂﹁﹔！？｡。＂＃＄％＆＇（）＊＋，﹐－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏.．!\"#$%&()*+,\-.\:;<=>?@\[\]\\\/^_`{|}~]"

tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese")
lm_model = TTALMModel("bert-base-chinese")
tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese")
lm_model.load_state_dict(torch.load("./wiki_zh.pt",map_location=torch.device('cuda')))
lm_model.to('cuda')
lm_model.eval()
model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

ds = load_dataset("common_voice", 'zh-TW', data_dir="./cv-corpus-6.1-2020-12-11", split="test")

resampler = torchaudio.transforms.Resample(orig_freq=48_000, new_freq=16_000)

def map_to_array(batch):
    speech, _ = torchaudio.load(batch["path"])
    batch["speech"] = resampler.forward(speech.squeeze(0)).numpy()
    batch["sampling_rate"] = resampler.new_freq
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower().replace("’", "'")
    return batch

ds = ds.map(map_to_array)

def map_to_pred(batch):
    features = processor(batch["speech"], sampling_rate=batch["sampling_rate"][0], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    decoded_results = []
    for logit in logits:
        pred_ids = torch.argmax(logit, dim=-1)
        mask = ~pred_ids.eq(tokenizer.pad_token_id).unsqueeze(-1).expand(logit.size())
        vocab_size = logit.size()[-1]
        voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
        lm_input = torch.masked_select(pred_ids, ~pred_ids.eq(tokenizer.pad_token_id)).unsqueeze(0)
        lm_prob = torch.nn.functional.softmax(lm_model.forward(lm_input)[0], dim=-1).squeeze(0)
        comb_pred_ids = torch.argmax(lm_prob*voice_prob, dim=-1)
        decoded_results.append(processor.decode(comb_pred_ids))

    batch["predicted"] = decoded_results
    batch["target"] = batch["sentence"]
    return batch

result = ds.map(map_to_pred, batched=True, batch_size=16, remove_columns=list(ds.features.keys()))

def cer_cal(groundtruth, hypothesis):
    err = 0
    tot = 0
    for p, t in zip(hypothesis, groundtruth):
        err += float(ed.eval(p.lower(), t.lower()))
        tot += len(t)
    return err / tot
print("CER: {:2f}".format(100 * cer_cal(result["target"],result["predicted"])))
```


`CER: 25.77`.\
`TIME: 06:01 min`


model name: wav2vec2-large-xlsr-53-tw-gpt


model author: voidful


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, zh, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, robust-speech-event, license:apache-2.0, model-index


language: zh
pipeline_tag: fill-mask
widget:


- text: "今天[MASK]情很好"


# albert_chinese_tiny


This a albert_chinese_tiny model from [brightmart/albert_zh project](https://github.com/brightmart/albert_zh), albert_tiny_google_zh model\
converted by huggingface's [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py)


## Notice


*Support AutoTokenizer*


Since sentencepiece is not used in albert_chinese_base model\
you have to call BertTokenizer instead of AlbertTokenizer !!!\
we can eval it using an example on MaskedLM


由於 albert_chinese_base 模型沒有用 sentencepiece\
用AlbertTokenizer會載不進詞表，因此需要改用BertTokenizer !!!\
我們可以跑MaskedLM預測來驗證這個做法是否正確


## Justify (驗證有效性)


```python
from transformers import AutoTokenizer, AlbertForMaskedLM
import torch
from torch.nn.functional import softmax

pretrained = 'voidful/albert_chinese_tiny'
tokenizer = AutoTokenizer.from_pretrained(pretrained)
model = AlbertForMaskedLM.from_pretrained(pretrained)

inputtext = "今天[MASK]情很好"

maskpos = tokenizer.encode(inputtext, add_special_tokens=True).index(103)

input_ids = torch.tensor(tokenizer.encode(inputtext, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=input_ids)
loss, prediction_scores = outputs[:2]
logit_prob = softmax(prediction_scores[0, maskpos],dim=-1).data.tolist()
predicted_index = torch.argmax(prediction_scores[0, maskpos]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token, logit_prob[predicted_index])
```


Result: `感 0.40312355756759644`


model name: albert_chinese_tiny


task: fill-mask


tags: pytorch, albert, fill-mask, zh, transformers, autonlp_compatible, infinity_compatible


language: zh
datasets:


- common_voice
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
- robust-speech-event
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Cantonese (Hong Kong) by Voidful
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice zh-HK
type: common_voice
args: zh-HK
metrics:
    - name: Test CER
type: cer
value: 16.41


# Wav2Vec2-Large-XLSR-53-hk


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Cantonese using the [Common Voice](https://huggingface.co/datasets/common_voice).\
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


[Colab trial](https://colab.research.google.com/drive/1nBRLf4Pwiply_y5rXWoaIB8LxX41tfEI?usp=sharing)


```
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
)
import torch
import re
import sys

model_name = "voidful/wav2vec2-large-xlsr-53-hk"
device = "cuda"
processor_name = "voidful/wav2vec2-large-xlsr-53-hk"

chars_to_ignore_regex = r"[¥•＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､　、〃〈〉《》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏﹑﹔·'℃°•·．﹑︰〈〉─《﹖﹣﹂﹁﹔！？｡。＂＃＄％＆＇（）＊＋，﹐－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏.．!\\"#$%&()*+,\\-.\\:;<=>?@\\[\\]\\\\\\/^_`{|}~]"

model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

resampler = torchaudio.transforms.Resample(orig_freq=48_000, new_freq=16_000)

def load_file_to_data(file):
    batch = {}
    speech, _ = torchaudio.load(file)
    batch["speech"] = resampler.forward(speech.squeeze(0)).numpy()
    batch["sampling_rate"] = resampler.new_freq
    return batch

def predict(data):
    features = processor(data["speech"], sampling_rate=data["sampling_rate"], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits
    pred_ids = torch.argmax(logits, dim=-1)
    return processor.batch_decode(pred_ids)

```


Predict


```python
predict(load_file_to_data('voice file path'))
```


## Evaluation


The model can be evaluated as follows on the Cantonese (Hong Kong) test data of Common Voice.\
CER calculation refer to https://huggingface.co/ctl/wav2vec2-large-xlsr-cantonese


```python
!mkdir cer
!wget -O cer/cer.py https://huggingface.co/ctl/wav2vec2-large-xlsr-cantonese/raw/main/cer.py
!pip install jiwer

import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
)
import torch
import re
import sys

cer = load_metric("./cer")
model_name = "voidful/wav2vec2-large-xlsr-53-hk"
device = "cuda"
processor_name = "voidful/wav2vec2-large-xlsr-53-hk"

chars_to_ignore_regex = r"[¥•＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､　、〃〈〉《》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏﹑﹔·'℃°•·．﹑︰〈〉─《﹖﹣﹂﹁﹔！？｡。＂＃＄％＆＇（）＊＋，﹐－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏.．!\\"#$%&()*+,\\-.\\:;<=>?@\\[\\]\\\\\\/^_`{|}~]"

model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)

ds = load_dataset("common_voice", 'zh-HK', data_dir="./cv-corpus-6.1-2020-12-11", split="test")

resampler = torchaudio.transforms.Resample(orig_freq=48_000, new_freq=16_000)

def map_to_array(batch):
    speech, _ = torchaudio.load(batch["path"])
    batch["speech"] = resampler.forward(speech.squeeze(0)).numpy()
    batch["sampling_rate"] = resampler.new_freq
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower().replace("’", "'")
    return batch

ds = ds.map(map_to_array)

def map_to_pred(batch):
    features = processor(batch["speech"], sampling_rate=batch["sampling_rate"][0], padding=True, return_tensors="pt")
    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)
    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["predicted"] = processor.batch_decode(pred_ids)
    batch["target"] = batch["sentence"]
    return batch

result = ds.map(map_to_pred, batched=True, batch_size=16, remove_columns=list(ds.features.keys()))

print("CER: {:2f}".format(100 * cer.compute(predictions=result["predicted"], references=result["target"])))
```


`CER 16.41`


model name: wav2vec2-large-xlsr-53-hk


model author: voidful


task: automatic-speech-recognition


- text: "今天[MASK]情很好"


# albert_chinese_large


This a albert_chinese_large model from [Google's github](https://github.com/google-research/ALBERT)\
converted by huggingface's [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py)


## Notice


*Support AutoTokenizer*


## Justify (驗證有效性)


```python
from transformers import AutoTokenizer, AlbertForMaskedLM
import torch
from torch.nn.functional import softmax

pretrained = 'voidful/albert_chinese_large'
tokenizer = AutoTokenizer.from_pretrained(pretrained)
model = AlbertForMaskedLM.from_pretrained(pretrained)

inputtext = "今天[MASK]情很好"

maskpos = tokenizer.encode(inputtext, add_special_tokens=True).index(103)

input_ids = torch.tensor(tokenizer.encode(inputtext, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=input_ids)
loss, prediction_scores = outputs[:2]
logit_prob = softmax(prediction_scores[0, maskpos],dim=-1).data.tolist()
predicted_index = torch.argmax(prediction_scores[0, maskpos]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token, logit_prob[predicted_index])
```


Result: `心 0.9422469735145569`


model name: albert_chinese_large


task: fill-mask


# albert_chinese_xxlarge


This a albert_chinese_xxlarge model from [Google's github](https://github.com/google-research/ALBERT)\
converted by huggingface's [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py)


```python
from transformers import AutoTokenizer, AlbertForMaskedLM
import torch
from torch.nn.functional import softmax

pretrained = 'voidful/albert_chinese_xxlarge'
tokenizer = AutoTokenizer.from_pretrained(pretrained)
model = AlbertForMaskedLM.from_pretrained(pretrained)

inputtext = "今天[MASK]情很好"

maskpos = tokenizer.encode(inputtext, add_special_tokens=True).index(103)

input_ids = torch.tensor(tokenizer.encode(inputtext, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=input_ids)
loss, prediction_scores = outputs[:2]
logit_prob = softmax(prediction_scores[0, maskpos],dim=-1).data.tolist()
predicted_index = torch.argmax(prediction_scores[0, maskpos]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token, logit_prob[predicted_index])
```


Result: `心 0.995713472366333`


model name: albert_chinese_xxlarge


model author: voidful


language: multilingual
datasets:


- NQ
- Trivia
- SQuAD
- MLQA
- DRCD


# dpr-ctx_encoder-bert-base-multilingual


## Description


Multilingual DPR Model base on bert-base-multilingual-cased.
[DPR model](https://arxiv.org/abs/2004.04906)
[DPR repo](https://github.com/facebookresearch/DPR)


## Data


1. [NQ](https://github.com/facebookresearch/DPR/blob/master/data/download_data.py)
1. [Trivia](https://github.com/facebookresearch/DPR/blob/master/data/download_data.py)
1. [SQuAD](https://github.com/facebookresearch/DPR/blob/master/data/download_data.py)
1. [DRCD*](https://github.com/DRCKnowledgeTeam/DRCD)
1. [MLQA*](https://github.com/facebookresearch/MLQA)


`question pairs for train`： 644,217\
`question pairs for dev`： 73,710


*DRCD and MLQA are converted using script from haystack [squad_to_dpr.py](https://github.com/deepset-ai/haystack/blob/master/haystack/retriever/squad_to_dpr.py)


## Training Script


I use the script from [haystack](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial9_DPR_training.ipynb)


## Usage


```python
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
tokenizer = DPRContextEncoderTokenizer.from_pretrained('voidful/dpr-ctx_encoder-bert-base-multilingual')
model = DPRContextEncoder.from_pretrained('voidful/dpr-ctx_encoder-bert-base-multilingual')
input_ids = tokenizer("Hello, is my dog cute ?", return_tensors='pt')["input_ids"]
embeddings = model(input_ids).pooler_output
```


Follow the tutorial from `haystack`:
[Better Retrievers via "Dense Passage Retrieval"](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial6_Better_Retrieval_via_DPR.ipynb)


```
from haystack.retriever.dense import DensePassageRetriever
retriever = DensePassageRetriever(document_store=document_store,
                                  query_embedding_model="voidful/dpr-question_encoder-bert-base-multilingual",
                                  passage_embedding_model="voidful/dpr-ctx_encoder-bert-base-multilingual",
                                  max_seq_len_query=64,
                                  max_seq_len_passage=256,
                                  batch_size=16,
                                  use_gpu=True,
                                  embed_title=True,
                                  use_fast_tokenizers=True)
```


model name: dpr-ctx_encoder-bert-base-multilingual


model author: voidful


tags: pytorch, dpr, multilingual, dataset:NQ, dataset:Trivia, dataset:SQuAD, dataset:MLQA, dataset:DRCD, arxiv:2004.04906, transformers


metrics:


- accuracy
- f1
- precision
- recall


model-index:


- name: twitter-roberta-base-sentiment-mlm-skep-class


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# twitter-roberta-base-sentiment-mlm-skep-class


This model was trained from scratch on an unkown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3955
- Accuracy: 0.9521
- F1: 0.9470
- Precision: 0.9406
- Recall: 0.9547


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine_with_restarts
- num_epochs: 20
- mixed_precision_training: Native AMP
- label_smoothing_factor: 0.1


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     | Precision | Recall |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|:---------:|:------:|
| 0.4542        | 1.0   | 1055 | 0.3885          | 0.9536   | 0.9492 | 0.9439    | 0.9552 |
| 0.3639        | 2.0   | 2110 | 0.3778          | 0.9593   | 0.9556 | 0.9552    | 0.9561 |
| 0.3375        | 3.0   | 3165 | 0.3955          | 0.9521   | 0.9470 | 0.9406    | 0.9547 |


### Framework versions


- Transformers 4.6.1
- Pytorch 1.7.0
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: twitter-roberta-base-sentiment-mlm-skep-class


model author: neoyipeng


task: text-classification


tags: pytorch, roberta, text-classification, transformers, model-index, infinity_compatible


model-index:


- name: twitter-roberta-base-sentiment-mlm-skep


# twitter-roberta-base-sentiment-mlm-skep


- Loss: 0.0003


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine_with_restarts
- num_epochs: 20
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.0122        | 1.0   | 936  | 0.0026          |
| 0.0003        | 2.0   | 1872 | 0.0003          |
| 0.0002        | 3.0   | 2808 | 0.0003          |


### Framework versions


model name: twitter-roberta-base-sentiment-mlm-skep


model author: neoyipeng


task: token-classification


tags: pytorch, roberta, token-classification, transformers, model-index, autonlp_compatible, infinity_compatible


metrics:


- accuracy
- f1
- precision
- recall


model-index:


- name: twitter-roberta-base-sentiment-mlm-class


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# twitter-roberta-base-sentiment-mlm-class


This model was trained from scratch on an unkown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3783
- Accuracy: 0.9592
- F1: 0.9553
- Precision: 0.9496
- Recall: 0.9618


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine_with_restarts
- num_epochs: 20
- mixed_precision_training: Native AMP
- label_smoothing_factor: 0.1


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     | Precision | Recall |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|:---------:|:------:|
| 0.4411        | 1.0   | 1055 | 0.3862          | 0.9528   | 0.9477 | 0.9426    | 0.9536 |
| 0.3612        | 2.0   | 2110 | 0.3730          | 0.9592   | 0.9548 | 0.9546    | 0.9550 |
| 0.3365        | 3.0   | 3165 | 0.3783          | 0.9592   | 0.9553 | 0.9496    | 0.9618 |


### Framework versions


- Transformers 4.6.1
- Pytorch 1.7.0
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: twitter-roberta-base-sentiment-mlm-class


model author: neoyipeng


task: text-classification


tags: pytorch, roberta, text-classification, transformers, model-index, infinity_compatible


model-index:


- name: twitter-roberta-base-sentiment-mlm


# twitter-roberta-base-sentiment-mlm


- Loss: 2.5119


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 6.9349        | 1.0   | 610  | 4.9601          |
| 4.7855        | 2.0   | 1220 | 4.0446          |
| 4.0519        | 3.0   | 1830 | 3.5383          |
| 3.6449        | 4.0   | 2440 | 3.2304          |
| 3.3727        | 5.0   | 3050 | 3.0593          |
| 3.1631        | 6.0   | 3660 | 2.8956          |
| 3.024         | 7.0   | 4270 | 2.8550          |
| 2.9008        | 8.0   | 4880 | 2.7255          |
| 2.8076        | 9.0   | 5490 | 2.6705          |
| 2.7414        | 10.0  | 6100 | 2.6554          |
| 2.6757        | 11.0  | 6710 | 2.5601          |
| 2.6246        | 12.0  | 7320 | 2.4616          |
| 2.5754        | 13.0  | 7930 | 2.5119          |


### Framework versions


model name: twitter-roberta-base-sentiment-mlm


model author: neoyipeng


task: fill-mask


tags: pytorch, roberta, fill-mask, transformers, model-index, autonlp_compatible, infinity_compatible


language: ["ru", "en"]
tags:


- russian
- fill-mask
- pretraining
- embeddings
- masked-lm
- tiny
- feature-extraction
- sentence-similarity
license: mit
widget:
- text: "Миниатюрная модель для [MASK] разных задач."


This is a very small distilled version of the [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) model for Russian and English (45 MB, 12M parameters). There is also an **updated version of this model**, [rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2), with a larger vocabulary and better quality on practically all Russian NLU tasks.


This model is useful if you want to fine-tune it for a relatively simple Russian task (e.g. NER or sentiment classification), and you care more about speed and size than about accuracy. It is approximately x10 smaller and faster than a base-sized BERT. Its `[CLS]` embeddings can be used as a sentence representation aligned between Russian and English.


It was trained on the [Yandex Translate corpus](https://translate.yandex.ru/corpus), [OPUS-100](https://huggingface.co/datasets/opus100) and [Tatoeba](https://huggingface.co/datasets/tatoeba), using MLM loss (distilled from [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased)), translation ranking loss, and `[CLS]` embeddings distilled from [LaBSE](https://huggingface.co/sentence-transformers/LaBSE), [rubert-base-cased-sentence](https://huggingface.co/DeepPavlov/rubert-base-cased-sentence), Laser and USE.


There is a more detailed [description in Russian](https://habr.com/ru/post/562064/).


Sentence embeddings can be produced as follows:


```python
# pip install transformers sentencepiece
import torch
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("cointegrated/rubert-tiny")
model = AutoModel.from_pretrained("cointegrated/rubert-tiny")
# model.cuda()  # uncomment it if you have a GPU

def embed_bert_cls(text, model, tokenizer):
    t = tokenizer(text, padding=True, truncation=True, return_tensors='pt')
    with torch.no_grad():
        model_output = model(**{k: v.to(model.device) for k, v in t.items()})
    embeddings = model_output.last_hidden_state[:, 0, :]
    embeddings = torch.nn.functional.normalize(embeddings)
    return embeddings[0].cpu().numpy()

print(embed_bert_cls('привет мир', model, tokenizer).shape)
# (312,)
```


model name: rubert-tiny


model author: cointegrated


task: feature-extraction


tags: pytorch, bert, pretraining, ru, en, transformers, russian, fill-mask, embeddings, masked-lm, tiny, feature-extraction, sentence-similarity, license:mit, autonlp_compatible, infinity_compatible


language: ["ru"]
tags:


- russian
- summarization
datasets:
- IlyaGusev/gazeta
- csebuetnlp/xlsum
- mlsum
- wiki_lingua
license: mit
widget:
- text: "Высота башни составляет 324 метра (1063 фута), примерно такая же высота, как у 81-этажного здания, и самое высокое сооружение в Париже. Его основание квадратно, размером 125 метров (410 футов) с любой стороны. Во время строительства Эйфелева башня превзошла монумент Вашингтона, став самым высоким искусственным сооружением в мире, и этот титул она удерживала в течение 41 года до завершения строительство здания Крайслер в Нью-Йорке в 1930 году. Это первое сооружение которое достигло высоты 300 метров. Из-за добавления вещательной антенны на вершине башни в 1957 году она сейчас выше здания Крайслер на 5,2 метра (17 футов). За исключением передатчиков, Эйфелева башня является второй самой высокой отдельно стоящей структурой во Франции после виадука Мийо."


This is a model for abstractive Russian summarization, based on [cointegrated/rut5-base-multitask](https://huggingface.co/cointegrated/rut5-base-multitask) and fine-tuned on 4 datasets.


It can be used as follows:


```python
from transformers import T5ForConditionalGeneration, T5Tokenizer
MODEL_NAME = 'cointegrated/rut5-base-absum'
model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME)
tokenizer = T5Tokenizer.from_pretrained(MODEL_NAME)
model.cuda();
model.eval();

def summarize(
    text, n_words=None, compression=None,
    max_length=1000, num_beams=3, do_sample=False, repetition_penalty=10.0,
    **kwargs
):
    """
    Summarize the text
    The following parameters are mutually exclusive:
    - n_words (int) is an approximate number of words to generate.
    - compression (float) is an approximate length ratio of summary and original text.
    """
    if n_words:
        text = '[{}] '.format(n_words) + text
    elif compression:
        text = '[{0:.1g}] '.format(compression) + text
    x = tokenizer(text, return_tensors='pt', padding=True).to(model.device)
    with torch.inference_mode():
        out = model.generate(
            **x,
            max_length=max_length, num_beams=num_beams,
            do_sample=do_sample, repetition_penalty=repetition_penalty,
            **kwargs
        )
    return tokenizer.decode(out[0], skip_special_tokens=True)

text = """Высота башни составляет 324 метра (1063 фута), примерно такая же высота, как у 81-этажного здания, и самое высокое сооружение в Париже. Его основание квадратно, размером 125 метров (410 футов) с любой стороны. Во время строительства Эйфелева башня превзошла монумент Вашингтона, став самым высоким искусственным сооружением в мире, и этот титул она удерживала в течение 41 года до завершения строительство здания Крайслер в Нью-Йорке в 1930 году. Это первое сооружение которое достигло высоты 300 метров. Из-за добавления вещательной антенны на вершине башни в 1957 году она сейчас выше здания Крайслер на 5,2 метра (17 футов). За исключением передатчиков, Эйфелева башня является второй самой высокой отдельно стоящей структурой во Франции после виадука Мийо."""
print(summarize(text))
# Эйфелева башня достигла высоты 300 метров.
print(summarize(text, n_words=10))
# Французская Эйфелева башня достигла высоты 300 метров.
```


model name: rut5-base-absum


task: summarization


tags: pytorch, t5, text2text-generation, ru, dataset:IlyaGusev/gazeta, dataset:csebuetnlp/xlsum, dataset:mlsum, dataset:wiki_lingua, transformers, russian, summarization, license:mit, autonlp_compatible


- russian
- classification
- sentiment
- multiclass
widget:
- text: "Какая гадость эта ваша заливная рыба!"


This is the [cointegrated/rubert-tiny](https://huggingface.co/cointegrated/rubert-tiny) model fine-tuned for classification of sentiment for short Russian texts.


The problem is formulated as multiclass classification: `negative` vs `neutral` vs `positive`.


## Usage


The function below estimates the sentiment of the given text:


```python
# !pip install transformers sentencepiece --quiet
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_checkpoint = 'cointegrated/rubert-tiny-sentiment-balanced'
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint)
if torch.cuda.is_available():
    model.cuda()

def get_sentiment(text, return_type='label'):
    """ Calculate sentiment of a text. `return_type` can be 'label', 'score' or 'proba' """
    with torch.no_grad():
        inputs = tokenizer(text, return_tensors='pt', truncation=True, padding=True).to(model.device)
        proba = torch.sigmoid(model(**inputs).logits).cpu().numpy()[0]
    if return_type == 'label':
        return model.config.id2label[proba.argmax()]
    elif return_type == 'score':
        return proba.dot([-1, 0, 1])
    return proba

text = 'Какая гадость эта ваша заливная рыба!'
# classify the text
print(get_sentiment(text, 'label'))  # negative
# score the text on the scale from -1 (very negative) to +1 (very positive)
print(get_sentiment(text, 'score'))  # -0.5894946306943893
# calculate probabilities of all labels
print(get_sentiment(text, 'proba'))  # [0.7870447  0.4947824  0.19755007]
```


## Training


We trained the model on [the datasets collected by Smetanin](https://github.com/sismetanin/sentiment-analysis-in-russian). We have converted all training data into a 3-class format and have up- and downsampled the training data to balance both the sources and the classes. The training code is available as [a Colab notebook](https://gist.github.com/avidale/e678c5478086c1d1adc52a85cb2b93e6). The metrics on the balanced test set are the following:


| Source | Macro F1 |
| ----------- | ----------- |
| SentiRuEval2016_banks | 0.83 |
| SentiRuEval2016_tele  | 0.74 |
| kaggle_news | 0.66 |
| linis | 0.50 |
| mokoron | 0.98 |
| rureviews | 0.72 |
| rusentiment | 0.67 |


model name: rubert-tiny-sentiment-balanced


model author: cointegrated


task: text-classification


tags: pytorch, bert, text-classification, ru, transformers, russian, classification, sentiment, multiclass, infinity_compatible


language: "ru"
tags:


- paraphrasing
- russian
license: mit


This is a small Russian paraphraser based on the [google/mt5-small](https://huggingface.co/google/mt5-small) model.
It has rather poor paraphrasing performance, but can be fine tuned for this or other tasks.


This model was created by taking the [alenusch/mt5small-ruparaphraser](https://huggingface.co/alenusch/mt5small-ruparaphraser) model and stripping 96% of its vocabulary which is unrelated to the Russian language or infrequent.


* The original model has 300M parameters, with 256M of them being input and output embeddings.
* After shrinking the `sentencepiece` vocabulary from 250K to 20K the number of model parameters reduced to 65M parameters, and model size reduced from 1.1GB to 246MB.
  * The first 5K tokens in the new vocabulary are taken from the original `mt5-small`.
  * The next 15K tokens are the most frequent tokens obtained by tokenizing a Russian web corpus from the [Leipzig corpora collection](https://wortschatz.uni-leipzig.de/en/download/Russian).


The model can be used as follows:


```
# !pip install transformers sentencepiece
import torch
from transformers import T5ForConditionalGeneration, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("cointegrated/rut5-small")
model = T5ForConditionalGeneration.from_pretrained("cointegrated/rut5-small")

text = 'Ехал Грека через реку, видит Грека в реке рак. '
inputs = tokenizer(text, return_tensors='pt')
with torch.no_grad():
    hypotheses = model.generate(
        **inputs,
        do_sample=True, top_p=0.95, num_return_sequences=10,
        repetition_penalty=2.5,
        max_length=32,
    )
for h in hypotheses:
    print(tokenizer.decode(h, skip_special_tokens=True))
```


model name: rut5-small


task: text2text-generation


tags: pytorch, jax, mt5, text2text-generation, ru, transformers, paraphrasing, russian, license:mit, autonlp_compatible