tags: pytorch, wav2vec2, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, audio, license:apache-2.0


language: en
datasets:


- superb
tags:
- speech
- audio
- hubert
license: apache-2.0


# Hubert-Large for Intent Classification


## Model description


This is a ported version of [S3PRL's Hubert for the SUPERB Intent Classification task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/fluent_commands).


The base model is [hubert-large-ll60k](https://huggingface.co/facebook/hubert-large-ll60k), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)


## Task and dataset description


Intent Classification (IC) classifies utterances into predefined classes to determine the intent of
speakers. SUPERB uses the
[Fluent Speech Commands](https://fluent.ai/fluent-speech-commands-a-dataset-for-spoken-language-understanding-research/)
dataset, where each utterance is tagged with three intent labels: **action**, **object**, and **location**.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ic-intent-classification---fluent-speech-commands).


## Usage examples


You can use the model directly like so:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "ic", split="test")
dataset = dataset.map(map_to_array)

model = HubertForSequenceClassification.from_pretrained("superb/hubert-large-superb-ic")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-large-superb-ic")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits

action_ids = torch.argmax(logits[:, :6], dim=-1).tolist()
action_labels = [model.config.id2label[_id] for _id in action_ids]

object_ids = torch.argmax(logits[:, 6:20], dim=-1).tolist()
object_labels = [model.config.id2label[_id + 6] for _id in object_ids]

location_ids = torch.argmax(logits[:, 20:24], dim=-1).tolist()
location_labels = [model.config.id2label[_id + 20] for _id in location_ids]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.9876`  | `N/A`         |


### BibTeX entry and citation info


```bibtex
@article{yang2021superb,
  title={SUPERB: Speech processing Universal PERformance Benchmark},
  author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
  journal={arXiv preprint arXiv:2105.01051},
  year={2021}
}
```


model name: hubert-large-superb-ic


model author: superb


task: audio-classification


tags: pytorch, hubert, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, audio, license:apache-2.0


language: en
datasets:


- superb
tags:
- speech
- audio
- wav2vec2
- audio-classification
license: apache-2.0
widget:
- example_title: IEMOCAP clip "happy"
src: https://cdn-media.huggingface.co/speech_samples/IEMOCAP_Ses01F_impro03_F013.wav
- example_title: IEMOCAP clip "neutral"
src: https://cdn-media.huggingface.co/speech_samples/IEMOCAP_Ses01F_impro04_F000.wav


# Wav2Vec2-Large for Emotion Recognition


## Model description


This is a ported version of
[S3PRL's Wav2Vec2 for the SUPERB Emotion Recognition task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/emotion).


The base model is [wav2vec2-large-lv60](https://huggingface.co/facebook/wav2vec2-large-lv60), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


## Task and dataset description


Emotion Recognition (ER) predicts an emotion class for each utterance. The most widely used ER dataset
[IEMOCAP](https://sail.usc.edu/iemocap/) is adopted, and we follow the conventional evaluation protocol:
we drop the unbalanced emotion classes to leave the final four classes with a similar amount of data points and
cross-validate on five folds of the standard splits.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#er-emotion-recognition).


## Usage examples


You can use the model via the Audio Classification pipeline:


```python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "er", split="session1")

classifier = pipeline("audio-classification", model="superb/wav2vec2-large-superb-er")
labels = classifier(dataset[0]["file"], top_k=5)
```


Or use the model directly:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "er", split="session1")
dataset = dataset.map(map_to_array)

model = Wav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-large-superb-er")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/wav2vec2-large-superb-er")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**session1**| `0.6564`  | `N/A`         |


### BibTeX entry and citation info


model name: wav2vec2-large-superb-er


model author: superb


task: audio-classification


tags: pytorch, wav2vec2, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, audio, license:apache-2.0


language: en
datasets:


- superb
tags:
- speech
- audio
- hubert
- audio-classification
widget:
- example_title: VoxCeleb Speaker id10003
src: https://cdn-media.huggingface.co/speech_samples/VoxCeleb1_00003.wav
- example_title: VoxCeleb Speaker id10004
src: https://cdn-media.huggingface.co/speech_samples/VoxCeleb_00004.wav
license: apache-2.0


# Hubert-Base for Speaker Identification


## Model description


This is a ported version of
[S3PRL's Hubert for the SUPERB Speaker Identification task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/voxceleb1).


The base model is [hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)


## Task and dataset description


Speaker Identification (SI) classifies each utterance for its speaker identity as a multi-class
classification, where speakers are in the same predefined set for both training and testing. The widely
used [VoxCeleb1](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html) dataset is adopted


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sid-speaker-identification).


## Usage examples


```python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "si", split="test")

classifier = pipeline("audio-classification", model="superb/hubert-base-superb-sid")
labels = classifier(dataset[0]["file"], top_k=5)
```


Or use the model directly:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "si", split="test")
dataset = dataset.map(map_to_array)

model = HubertForSequenceClassification.from_pretrained("superb/hubert-base-superb-sid")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-base-superb-sid")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:2]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.8142`  | `0.8071`         |


### BibTeX entry and citation info


```bibtex
@article{yang2021superb,
  title={SUPERB: Speech processing Universal PERformance Benchmark},
  author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
  journal={arXiv preprint arXiv:2105.01051},
  year={2021}
}
```


model name: hubert-base-superb-sid


model author: superb


task: audio-classification


tags: pytorch, hubert, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, audio, license:apache-2.0


language: en
datasets:


- superb
tags:
- speech
- audio
- wav2vec2
license: apache-2.0


# Wav2Vec2-Large for Intent Classification


## Model description


This is a ported version of [S3PRL's Wav2Vec2 for the SUPERB Intent Classification task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/fluent_commands).


The base model is [wav2vec2-large-lv60](https://huggingface.co/facebook/wav2vec2-large-lv60), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


## Task and dataset description


Intent Classification (IC) classifies utterances into predefined classes to determine the intent of
speakers. SUPERB uses the
[Fluent Speech Commands](https://fluent.ai/fluent-speech-commands-a-dataset-for-spoken-language-understanding-research/)
dataset, where each utterance is tagged with three intent labels: **action**, **object**, and **location**.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ic-intent-classification---fluent-speech-commands).


## Usage examples


You can use the model directly like so:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "ic", split="test")
dataset = dataset.map(map_to_array)

model = Wav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-large-superb-ic")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/wav2vec2-large-superb-ic")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits

action_ids = torch.argmax(logits[:, :6], dim=-1).tolist()
action_labels = [model.config.id2label[_id] for _id in action_ids]

object_ids = torch.argmax(logits[:, 6:20], dim=-1).tolist()
object_labels = [model.config.id2label[_id + 6] for _id in object_ids]

location_ids = torch.argmax(logits[:, 20:24], dim=-1).tolist()
location_labels = [model.config.id2label[_id + 20] for _id in location_ids]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.9528`  | `N/A`         |


### BibTeX entry and citation info


model name: wav2vec2-large-superb-ic


model author: superb


task: audio-classification


tags: pytorch, wav2vec2, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, audio, license:apache-2.0


language: en
datasets:


- superb
tags:
- speech
- audio
- wav2vec2
- audio-classification
widget:
- example_title: Speech Commands "down"
src: https://cdn-media.huggingface.co/speech_samples/keyword_spotting_down.wav
- example_title: Speech Commands "go"
src: https://cdn-media.huggingface.co/speech_samples/keyword_spotting_go.wav
license: apache-2.0


# Wav2Vec2-Base for Keyword Spotting


## Model description


This is a ported version of
[S3PRL's Wav2Vec2 for the SUPERB Keyword Spotting task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/speech_commands).


The base model is [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)


## Task and dataset description


Keyword Spotting (KS) detects preregistered keywords by classifying utterances into a predefined set of
words. The task is usually performed on-device for the fast response time. Thus, accuracy, model size, and
inference time are all crucial. SUPERB uses the widely used
[Speech Commands dataset v1.0](https://www.tensorflow.org/datasets/catalog/speech_commands) for the task.
The dataset consists of ten classes of keywords, a class for silence, and an unknown class to include the
false positive.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ks-keyword-spotting).


## Usage examples


You can use the model via the Audio Classification pipeline:


```python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "ks", split="test")

classifier = pipeline("audio-classification", model="superb/wav2vec2-base-superb-ks")
labels = classifier(dataset[0]["file"], top_k=5)
```


Or use the model directly:


```python
import torch
from datasets import load_dataset
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor
from torchaudio.sox_effects import apply_effects_file

effects = [["channels", "1"], ["rate", "16000"], ["gain", "-3.0"]]
def map_to_array(example):
    speech, _ = apply_effects_file(example["file"], effects)
    example["speech"] = speech.squeeze(0).numpy()
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
dataset = dataset.map(map_to_array)

model = Wav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-base-superb-ks")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/wav2vec2-base-superb-ks")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.9623`  | `0.9643`         |


### BibTeX entry and citation info


```bibtex
@article{yang2021superb,
  title={SUPERB: Speech processing Universal PERformance Benchmark},
  author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
  journal={arXiv preprint arXiv:2105.01051},
  year={2021}
}
```


model name: wav2vec2-base-superb-ks


model author: superb


task: audio-classification


language: en
datasets:


- superb
tags:
- speech
- audio
- wav2vec2
license: apache-2.0


# Wav2Vec2-Base for Intent Classification


## Model description


This is a ported version of [S3PRL's Wav2Vec2 for the SUPERB Intent Classification task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/fluent_commands).


## Task and dataset description


Intent Classification (IC) classifies utterances into predefined classes to determine the intent of
speakers. SUPERB uses the
[Fluent Speech Commands](https://fluent.ai/fluent-speech-commands-a-dataset-for-spoken-language-understanding-research/)
dataset, where each utterance is tagged with three intent labels: **action**, **object**, and **location**.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ic-intent-classification---fluent-speech-commands).


## Usage examples


You can use the model directly like so:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "ic", split="test")
dataset = dataset.map(map_to_array)

model = Wav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-base-superb-ic")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/wav2vec2-base-superb-ic")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits

action_ids = torch.argmax(logits[:, :6], dim=-1).tolist()
action_labels = [model.config.id2label[_id] for _id in action_ids]

object_ids = torch.argmax(logits[:, 6:20], dim=-1).tolist()
object_labels = [model.config.id2label[_id + 6] for _id in object_ids]

location_ids = torch.argmax(logits[:, 20:24], dim=-1).tolist()
location_labels = [model.config.id2label[_id + 20] for _id in location_ids]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.9235`  | `N/A`         |


### BibTeX entry and citation info


model name: wav2vec2-base-superb-ic


model author: superb


task: audio-classification


tags: pytorch, wav2vec2, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, audio, license:apache-2.0


language: en
datasets:


- superb
tags:
- speech
- audio
- wav2vec2
- audio-classification
license: apache-2.0
widget:
- example_title: IEMOCAP clip "happy"
src: https://cdn-media.huggingface.co/speech_samples/IEMOCAP_Ses01F_impro03_F013.wav
- example_title: IEMOCAP clip "neutral"
src: https://cdn-media.huggingface.co/speech_samples/IEMOCAP_Ses01F_impro04_F000.wav


# Wav2Vec2-Base for Emotion Recognition


## Model description


This is a ported version of
[S3PRL's Wav2Vec2 for the SUPERB Emotion Recognition task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/emotion).


The base model is [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)


## Task and dataset description


Emotion Recognition (ER) predicts an emotion class for each utterance. The most widely used ER dataset
[IEMOCAP](https://sail.usc.edu/iemocap/) is adopted, and we follow the conventional evaluation protocol:
we drop the unbalanced emotion classes to leave the final four classes with a similar amount of data points and
cross-validate on five folds of the standard splits.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#er-emotion-recognition).


## Usage examples


You can use the model via the Audio Classification pipeline:


```python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "er", split="session1")

classifier = pipeline("audio-classification", model="superb/wav2vec2-base-superb-er")
labels = classifier(dataset[0]["file"], top_k=5)
```


Or use the model directly:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "er", split="session1")
dataset = dataset.map(map_to_array)

model = Wav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-base-superb-er")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/wav2vec2-base-superb-er")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**session1**| `0.6343`  | `0.6258`         |


### BibTeX entry and citation info


```bibtex
@article{yang2021superb,
  title={SUPERB: Speech processing Universal PERformance Benchmark},
  author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
  journal={arXiv preprint arXiv:2105.01051},
  year={2021}
}
```


model name: wav2vec2-base-superb-er


model author: superb


task: audio-classification


datasets:


- superb
tags:
- library:s3prl
- benchmark:superb
- type:model


# Fine-tuned s3prl model


Upstream Model: superb-test-org/test-submission-with-example-expert


## Model description


[More information needed]


## Intended uses & limitations


## How to use


## Limitations and bias


## Training data


## Training procedure


## Evaluation results


model name: superb-test-org__test-submission-with-example-expert__d609b3c32044e50e3d5e9067bd97af1b42f04b0e


tags: tensorboard, dataset:superb, library:s3prl, benchmark:superb, type:model


# SUPERB Submission Template


Welcome to the [SUPERB Challenge](https://superbbenchmark.org/challenge)! SUPERB is a collection of benchmarking resources to evaluate the capability of a universal shared representation for speech processing. It comes with a benchmark on the publicly available datasets and a challenge on a secret/not released hidden dataset. In SUPERB Challenge, a challenging hidden dataset is newly recorded to evaluate the ultimate generaliziblity across various tasks and data.


You can participate the challenge by simply submitting your self-supervised (SSL) pretrained models (model definition & pretrained weights), and we benchmark it with the hidden dataset. This repository constains useful tools to let you easliy [submit](https://superbbenchmark.org/submit) your models ***privately*** for evaluation to [the challenge hidden-set leaderboard](https://superbbenchmark.org/leaderboard?track=constrained&subset=Hidden+Dev+Set).


1. Generate a submission template
1. Validate the format/interface correctness of your model
1. Upload to Huggingface's Hub (privately)
1. Submit the upload information to [SUPERB website](https://superbbenchmark.org/submit)


#### Note 1.


We accept pre-trained models in PyTorch by default. If you wish to submit upstreams in non-PyTorch frameworks, please [fill this form](https://docs.google.com/forms/d/e/1FAIpQLSe52jYL2Yk9oYqXfg_Bg0Sjp01a6HSLUhY5VohsZOE5sNmgsw/viewform)!


#### Note 2.


If you are not feasible to submit the pre-trained model, please [fill this form](https://docs.google.com/forms/d/e/1FAIpQLSdA44nArlIDfGV63WwtwXer4WAPQO1aBwEpAjDSNjbMQN-GJQ/viewform) for us to see how to help!


## Quickstart


### 1. Create an account and organization on the Hugging Face Hub


First create an account on the Hugging Face Hub and you can sign up [here](https://huggingface.co/join) if you haven't already! Next, create a new organization and invite the SUPERB Hidden Set Committee to join. You will upload your model to a repository under this organization so that members inside it can access the model which is not publicly available.


* [superb-hidden-set](https://huggingface.co/superb-hidden-set)


### 2. Create a template repository on your machine


The next step is to create a template repository on your local machine that contains various files and a CLI to help you validate and submit your pretrained models. The Hugging Face Hub uses [Git Large File Storage (LFS)](https://git-lfs.github.com) to manage large files, so first install it if you don't have it already. For example, on macOS you can run:


```bash
brew install git-lfs
git lfs install
```


Next, run the following commands to create the repository. We recommend creating a Python virtual environment for the project, e.g. with Anaconda:


```bash
# Create and activate a virtual environment
conda create -n superb-submit python=3.8 && conda activate superb-submit
# Install the following libraries
pip install cookiecutter huggingface-hub==0.0.16
# Create the template repository
cookiecutter git+https://huggingface.co/superb/superb-submission
```


This will ask you to specify your Hugging Face Hub username, password, organisation, and the name of the repository:


```
hf_hub_username [<huggingface>]:
hf_hub_password [<password>]:
hf_hub_organisation [superb-submissions]:
repo_name [<my-superb-submissions>]:
```


This will trigger the following steps:


1. Create a private dataset repository on the Hugging Face Hub under `{hf_hub_organisation}/{repo_name}`
1. Clone the repository to your local machine
1. Add various template files, commit them locally to the repository, and push them to the Hub


The resulting repository should have the following structure:


```
my-superb-submission
├── LICENSE
├── README.md               <- The README with submission instructions
├── cli.py                  <- The CLI for validating predictions etc
└── requirements.txt        <- The requirements packages for the submissions
├── expert.py               <- Your model definition
└── model.pt                <- Your model weights
```


### 3. Install the dependencies


The final step is to install the project's dependencies:


```bash
# Navigate to the template repository
cd my-superb-submission
# Install dependencies
python -m pip install -r requirements.txt
```


That's it! You're now all set to start pretraining your speech models - see the instructions below on how to submit them to the Hub.


## Submitting to the leaderboard


To make a submission to the [leaderboard](https://superbbenchmark.org/leaderboard?subset=Hidden+Dev+Set), there are 4 main steps:


1. Modify `expert.py` and change `model.pt` so we can initialize an upstream model following the [challenge policy](https://superbbenchmark.org/challenge#Upstream-Specification) by:

   ```python
   upstream = UpstreamExpert(ckpt="./model.pt")
   ```

   ***Package Dependency:*** Note that we only install `torch` package so far by following the above steps. If your model needs more packages, you can modify the `requirement.txt` to meet your need and install them inside the current conda environment. We will install the packages you list in the `requirement.txt` before initializing the upstream model.
1. Validate the upstream model's interface meets the requirements in the [challenge policy](https://superbbenchmark.org/challenge#Upstream-Specification). If everything is correct, you should see the following message: "All submission files validated! Now you can make a submission."

   ```
   python cli.py validate
   ```
1. Push the model to the Hub! If there are no errors, you should see the following message: "Upload successful!"

   ```
   python cli.py upload "commit message: my best model"
   ```
1. [Make a submission at SUPERB website](https://superbbenchmark.org/submit) by uniquely indentifying this uploaded model with the following information, which can be shown by:

   ```
   python cli.py info
   ```

   - Organization Name
   - Repository Name
   - Commit Hash (full 40 characters)


After you finish the above 4 steps. You will see a new entry in your [SUPERB profile page](https://superbbenchmark.org/profile) (need login) which does not have any benchmark numbers yet. Please wait for us to finetuned it on the hidden dataset and get the benchmark results. The results will be revealed within one week. Please stay tuned!


model name: superb-submission


model author: superb


tags:


language: en
datasets:


- superb
tags:
- speech
- audio-classification
- hubert
license: apache-2.0


# Hubert-Base for Intent Classification


## Model description


This is a ported version of [S3PRL's Hubert for the SUPERB Intent Classification task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/fluent_commands).


The base model is [hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)


## Task and dataset description


Intent Classification (IC) classifies utterances into predefined classes to determine the intent of
speakers. SUPERB uses the
[Fluent Speech Commands](https://fluent.ai/fluent-speech-commands-a-dataset-for-spoken-language-understanding-research/)
dataset, where each utterance is tagged with three intent labels: **action**, **object**, and **location**.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ic-intent-classification---fluent-speech-commands).


## Usage examples


You can use the model directly like so:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "ic", split="test")
dataset = dataset.map(map_to_array)

model = HubertForSequenceClassification.from_pretrained("superb/hubert-base-superb-ic")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-base-superb-ic")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits

action_ids = torch.argmax(logits[:, :6], dim=-1).tolist()
action_labels = [model.config.id2label[_id] for _id in action_ids]

object_ids = torch.argmax(logits[:, 6:20], dim=-1).tolist()
object_labels = [model.config.id2label[_id + 6] for _id in object_ids]

location_ids = torch.argmax(logits[:, 20:24], dim=-1).tolist()
location_labels = [model.config.id2label[_id + 20] for _id in location_ids]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.9834`  | `N/A`         |


### BibTeX entry and citation info


```bibtex
@article{yang2021superb,
  title={SUPERB: Speech processing Universal PERformance Benchmark},
  author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
  journal={arXiv preprint arXiv:2105.01051},
  year={2021}
}
```


model name: hubert-base-superb-ic


model author: superb


task: audio-classification


tags: pytorch, hubert, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, license:apache-2.0


language: en
datasets:


- superb
tags:
- speech
- audio
- hubert
- audio-classification
license: apache-2.0
widget:
- example_title: VoxCeleb Speaker id10003
src: https://cdn-media.huggingface.co/speech_samples/VoxCeleb1_00003.wav
- example_title: VoxCeleb Speaker id10004
src: https://cdn-media.huggingface.co/speech_samples/VoxCeleb_00004.wav


# Hubert-Large for Speaker Identification


## Model description


This is a ported version of
[S3PRL's Hubert for the SUPERB Speaker Identification task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/voxceleb1).


The base model is [hubert-large-ll60k](https://huggingface.co/facebook/hubert-large-ll60k), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


## Task and dataset description


Speaker Identification (SI) classifies each utterance for its speaker identity as a multi-class
classification, where speakers are in the same predefined set for both training and testing. The widely
used [VoxCeleb1](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html) dataset is adopted


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sid-speaker-identification).


## Usage examples


You can use the model via the Audio Classification pipeline:


```python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "si", split="test")

classifier = pipeline("audio-classification", model="superb/hubert-large-superb-sid")
labels = classifier(dataset[0]["file"], top_k=5)
```


Or use the model directly:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "si", split="test")
dataset = dataset.map(map_to_array)

model = HubertForSequenceClassification.from_pretrained("superb/hubert-large-superb-sid")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-large-superb-sid")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:2]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.9033`  | `0.9035`         |


### BibTeX entry and citation info


model name: hubert-large-superb-sid


model author: superb


task: audio-classification


tags: pytorch, hubert, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, audio, license:apache-2.0


language: en
datasets:


- superb
tags:
- speech
- audio
- hubert
- audio-classification
license: apache-2.0
widget:
- example_title: Speech Commands "down"
src: https://cdn-media.huggingface.co/speech_samples/keyword_spotting_down.wav
- example_title: Speech Commands "go"
src: https://cdn-media.huggingface.co/speech_samples/keyword_spotting_go.wav


# Hubert-Base for Keyword Spotting


## Model description


This is a ported version of [S3PRL's Hubert for the SUPERB Keyword Spotting task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/speech_commands).


The base model is [hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)


## Task and dataset description


Keyword Spotting (KS) detects preregistered keywords by classifying utterances into a predefined set of
words. The task is usually performed on-device for the fast response time. Thus, accuracy, model size, and
inference time are all crucial. SUPERB uses the widely used
[Speech Commands dataset v1.0](https://www.tensorflow.org/datasets/catalog/speech_commands) for the task.
The dataset consists of ten classes of keywords, a class for silence, and an unknown class to include the
false positive.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ks-keyword-spotting).


## Usage examples


```python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "ks", split="test")

classifier = pipeline("audio-classification", model="superb/hubert-base-superb-ks")
labels = classifier(dataset[0]["file"], top_k=5)
```


Or use the model directly:


```python
import torch
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor
from torchaudio.sox_effects import apply_effects_file

effects = [["channels", "1"], ["rate", "16000"], ["gain", "-3.0"]]
def map_to_array(example):
    speech, _ = apply_effects_file(example["file"], effects)
    example["speech"] = speech.squeeze(0).numpy()
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
dataset = dataset.map(map_to_array)

model = HubertForSequenceClassification.from_pretrained("superb/hubert-base-superb-ks")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-base-superb-ks")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.9630`  | `0.9672`         |


### BibTeX entry and citation info


```bibtex
@article{yang2021superb,
  title={SUPERB: Speech processing Universal PERformance Benchmark},
  author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
  journal={arXiv preprint arXiv:2105.01051},
  year={2021}
}
```


model name: hubert-base-superb-ks


model author: superb


task: audio-classification


language: en
datasets:


- superb
tags:
- speech
- audio
- hubert
- audio-classification
license: apache-2.0
widget:
- example_title: IEMOCAP clip "happy"
src: https://cdn-media.huggingface.co/speech_samples/IEMOCAP_Ses01F_impro03_F013.wav
- example_title: IEMOCAP clip "neutral"
src: https://cdn-media.huggingface.co/speech_samples/IEMOCAP_Ses01F_impro04_F000.wav


# Hubert-Base for Emotion Recognition


## Model description


This is a ported version of
[S3PRL's Hubert for the SUPERB Emotion Recognition task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/emotion).


## Task and dataset description


Emotion Recognition (ER) predicts an emotion class for each utterance. The most widely used ER dataset
[IEMOCAP](https://sail.usc.edu/iemocap/) is adopted, and we follow the conventional evaluation protocol:
we drop the unbalanced emotion classes to leave the final four classes with a similar amount of data points and
cross-validate on five folds of the standard splits.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#er-emotion-recognition).


## Usage examples


You can use the model via the Audio Classification pipeline:


```python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "er", split="session1")

classifier = pipeline("audio-classification", model="superb/hubert-base-superb-er")
labels = classifier(dataset[0]["file"], top_k=5)
```


Or use the model directly:


```python
import torch
import librosa
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "er", split="session1")
dataset = dataset.map(map_to_array)

model = HubertForSequenceClassification.from_pretrained("superb/hubert-base-superb-er")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-base-superb-er")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**session1**| `0.6492`  | `0.6359`         |


### BibTeX entry and citation info


model name: hubert-base-superb-er


model author: superb


task: audio-classification


tags: pytorch, hubert, audio-classification, en, dataset:superb, arxiv:2105.01051, transformers, speech, audio, license:apache-2.0


datasets:


- superb
tags:
- library:s3prl
- benchmark:superb
- type:model


# Fine-tuned s3prl model


Upstream Model: hubert


## Model description


[More information needed]


## Intended uses & limitations


## How to use


## Limitations and bias


## Training data


## Training procedure


## Evaluation results


model name: hubert__508944ac


tags: tensorboard, dataset:superb, library:s3prl, benchmark:superb, type:model


language: en
datasets:


- superb
tags:
- speech
- audio
- hubert
- audio-classification
license: apache-2.0
widget:
- example_title: Speech Commands "down"
src: https://cdn-media.huggingface.co/speech_samples/keyword_spotting_down.wav
- example_title: Speech Commands "go"
src: https://cdn-media.huggingface.co/speech_samples/keyword_spotting_go.wav


# Hubert-Large for Keyword Spotting


This is a ported version of
[S3PRL's Hubert for the SUPERB Keyword Spotting task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/speech_commands).


The base model is [hubert-large-ll60k](https://huggingface.co/facebook/hubert-large-ll60k), which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.


For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)


## Task and dataset description


Keyword Spotting (KS) detects preregistered keywords by classifying utterances into a predefined set of
words. The task is usually performed on-device for the fast response time. Thus, accuracy, model size, and
inference time are all crucial. SUPERB uses the widely used
[Speech Commands dataset v1.0](https://www.tensorflow.org/datasets/catalog/speech_commands) for the task.
The dataset consists of ten classes of keywords, a class for silence, and an unknown class to include the
false positive.


For the original model's training and evaluation instructions refer to the
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ks-keyword-spotting).


## Usage examples


```python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "ks", split="test")

classifier = pipeline("audio-classification", model="superb/hubert-large-superb-ks")
labels = classifier(dataset[0]["file"], top_k=5)
```


Or use the model directly:


```python
import torch
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor
from torchaudio.sox_effects import apply_effects_file

effects = [["channels", "1"], ["rate", "16000"], ["gain", "-3.0"]]
def map_to_array(example):
    speech, _ = apply_effects_file(example["file"], effects)
    example["speech"] = speech.squeeze(0).numpy()
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
dataset = dataset.map(map_to_array)

model = HubertForSequenceClassification.from_pretrained("superb/hubert-large-superb-ks")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-large-superb-ks")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
```


## Eval results


The evaluation metric is accuracy.


|        | **s3prl** | **transformers** |
|--------|-----------|------------------|
|**test**| `0.9529`  | `0.9532`         |


### BibTeX entry and citation info


```bibtex
@article{yang2021superb,
  title={SUPERB: Speech processing Universal PERformance Benchmark},
  author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
  journal={arXiv preprint arXiv:2105.01051},
  year={2021}
}
```


model name: hubert-large-superb-ks


model author: superb


task: audio-classification


# Fine-tuned Model Submission Template


This is a template reprository for the SUPERB benchmark for the *fine-tuned model* category. In this category, participants are asked to fine-tuned a pretrained model in each of SUPERB's downstream tasks and then store the model weights and hyperparameters in this repo.


There are four steps involved in making a submission:


1. Fine-tune a pretrained model on a downstream task.
1. Implement the `PreTrainedModel` interface defined in each `model.py` module.
1. Store the weights and hyperparameters in the task directory
1. Push all the files to the Hugging Face Hub.


model name: finetuned-model-upload-template


tags:


language:


- ru


- toxic comments classification


licenses:


- cc-by-nc-sa


## General concept of the model


This model is trained on the dataset of sensitive topics of the Russian language. The concept of sensitive topics is described [in this article ](https://www.aclweb.org/anthology/2021.bsnlp-1.4/) presented at the workshop for Balto-Slavic NLP at the EACL-2021 conference. Please note that this article describes the first version of the dataset, while the model is trained on the extended version of the dataset open-sourced on our [GitHub](https://github.com/skoltech-nlp/inappropriate-sensitive-topics/blob/main/Version2/sensitive_topics/sensitive_topics.csv) or on [kaggle](https://www.kaggle.com/nigula/russian-sensitive-topics). The properties of the dataset is the same as the one described in the article, the only difference is the size.


## Instructions


The model predicts combinations of 18 sensitive topics described in the [article](https://arxiv.org/abs/2103.05345). You can find step-by-step instructions for using the model [here](https://github.com/skoltech-nlp/inappropriate-sensitive-topics/blob/main/Version2/sensitive_topics/Inference.ipynb)


## Metrics


The dataset partially manually labeled samples and partially semi-automatically labeled samples. Learn more in our article. We tested the performance of the classifier only on the part of manually labeled data that is why some topics are not well represented in the test set.


|                   | precision | recall | f1-score | support |
|-------------------|-----------|--------|----------|---------|
| offline_crime     |      0.65 |   0.55 |      0.6 |     132 |
| online_crime      |       0.5 |   0.46 |     0.48 |      37 |
| drugs             |      0.87 |    0.9 |     0.88 |      87 |
| gambling          |       0.5 |   0.67 |     0.57 |       6 |
| pornography       |      0.73 |   0.59 |     0.65 |     204 |
| prostitution      |      0.75 |   0.69 |     0.72 |      91 |
| slavery           |      0.72 |   0.72 |     0.73 |      40 |
| suicide           |      0.33 |   0.29 |     0.31 |       7 |
| terrorism         |      0.68 |   0.57 |     0.62 |      47 |
| weapons           |      0.89 |   0.83 |     0.86 |     138 |
| body_shaming      |       0.9 |   0.67 |     0.77 |     109 |
| health_shaming    |      0.84 |   0.55 |     0.66 |     108 |
| politics          |      0.68 |   0.54 |      0.6 |     241 |
| racism            |      0.81 |   0.59 |     0.68 |     204 |
| religion          |      0.94 |   0.72 |     0.81 |     102 |
| sexual_minorities |      0.69 |   0.46 |     0.55 |     102 |
| sexism            |      0.66 |   0.64 |     0.65 |     132 |
| social_injustice  |      0.56 |   0.37 |     0.45 |     181 |
| none              |      0.62 |   0.67 |     0.64 |     250 |
| micro avg         |      0.72 |   0.61 |     0.66 |    2218 |
| macro avg         |       0.7 |    0.6 |     0.64 |    2218 |
| weighted avg      |      0.73 |   0.61 |     0.66 |    2218 |
| samples avg       |      0.75 |   0.66 |     0.68 |    2218 |


## Licensing Information


[Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).


[![CC BY-NC-SA 4.0](https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png)](http://creativecommons.org/licenses/by-nc-sa/4.0/)


## Citation


If you find this repository helpful, feel free to cite our publication:


```
@inproceedings{babakov-etal-2021-detecting,
    title = "Detecting Inappropriate Messages on Sensitive Topics that Could Harm a Company{'}s Reputation",
    author = "Babakov, Nikolay  and
      Logacheva, Varvara  and
      Kozlova, Olga  and
      Semenov, Nikita  and
      Panchenko, Alexander",
    booktitle = "Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing",
    month = apr,
    year = "2021",
    address = "Kiyv, Ukraine",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.bsnlp-1.4",
    pages = "26--36",
    abstract = "Not all topics are equally {``}flammable{''} in terms of toxicity: a calm discussion of turtles or fishing less often fuels inappropriate toxic dialogues than a discussion of politics or sexual minorities. We define a set of sensitive topics that can yield inappropriate and toxic messages and describe the methodology of collecting and labelling a dataset for appropriateness. While toxicity in user-generated data is well-studied, we aim at defining a more fine-grained notion of inappropriateness. The core of inappropriateness is that it can harm the reputation of a speaker. This is different from toxicity in two respects: (i) inappropriateness is topic-related, and (ii) inappropriate message is not toxic but still unacceptable. We collect and release two datasets for Russian: a topic-labelled dataset and an appropriateness-labelled dataset. We also release pre-trained classification models trained on this data.",
}
```


model name: russian-sensitive-topics


model author: Skoltech


task: text-classification


tags: pytorch, tf, jax, bert, text-classification, ru, arxiv:2103.05345, transformers, toxic comments classification, infinity_compatible


language:


- ru


tags:


- toxic comments classification


licenses:


- cc-by-nc-sa


## General concept of the model


#### Proposed usage


The **'inappropriateness'** substance we tried to collect in the dataset and detect with the model **is NOT a substitution of toxicity**, it is rather a derivative of toxicity. So the model based on our dataset could serve as **an additional layer of inappropriateness filtering after toxicity and obscenity filtration**. You can detect the exact sensitive topic by using [another model](https://huggingface.co/Skoltech/russian-sensitive-topics). The proposed pipeline is shown in the scheme below.


You can also train one classifier for both toxicity and inappropriateness detection. The data to be mixed with toxic labelled samples could be found on our [GitHub](https://github.com/skoltech-nlp/inappropriate-sensitive-topics/blob/main/Version2/appropriateness/Appropriateness.csv) or on [kaggle](https://www.kaggle.com/nigula/russianinappropriatemessages)


#### Inappropraiteness intuition


This model is trained on the dataset of inappropriate messages of the Russian language. Generally, an inappropriate utterance is an utterance that has not obscene words or any kind of toxic intent, but can still harm the reputation of the speaker. Find some sample for more intuition in the table below. Learn more about the concept of inappropriateness [in this article ](https://www.aclweb.org/anthology/2021.bsnlp-1.4/) presented at the workshop for Balto-Slavic NLP at the EACL-2021 conference. Please note that this article describes the first version of the dataset, while the model is trained on the extended version of the dataset open-sourced on our [GitHub](https://github.com/skoltech-nlp/inappropriate-sensitive-topics/blob/main/Version2/appropriateness/Appropriateness.csv) or on [kaggle](https://www.kaggle.com/nigula/russianinappropriatemessages). The properties of the dataset are the same as the one described in the article, the only difference is the size.


| Text                                                                                | Inappropriate | Comment                                                        | Unsafe topic  |
|------------------------------------------------------------------------------------------------------|---------------|----------------------------------------------------------------|---------------|
| Ладно бы видного деятеля завалили а тут какого то ноунейм нигру преступника                          | yes           | Оправдание убийства                                            | offline crime |
| Это нарушение УКРФ!                                                                                | no            | Осуждение нарушения закона                                     | offline crime |
| Религия - это насилие высшего уровня. Верующего принуждают к вере еще в младенчестве.                | yes           | Оскорбление чувств верующих                                    | religion      |
| Христиане, как вы относитесь к иноверцам, в частности к Гамалиилу и к уверовавшим в Иисуса фарисеям? | no            | Обычные вопросы к верующим без намека на оскорбление их чувств | religion      |
| Хорошо что проституция запрещена                                                                     | no            | Осуждение простуитуции                                         | prostitution  |
| Возьму бабки и сниму шлюх на выходных                                                                | yes           | Популяризация проституции                                      | prostitution  |


#### Metrics


The model was trained, validated, and tested only on the samples with 100% confidence, which allowed to get the following metrics on test set:


|              | precision | recall | f1-score | support |
|--------------|----------|--------|----------|---------|
| 0            | 0.92     | 0.93   | 0.93     | 7839    |
| 1            | 0.80     | 0.76   | 0.78     | 2726    |
| accuracy     |          |        | 0.89     | 10565   |
| macro avg    | 0.86     | 0.85   | 0.85     | 10565   |
| weighted avg | 0.89     | 0.89   | 0.89     | 10565   |


## Licensing Information


## Citation


## Contacts


If you have any questions please contact [Nikolay](mailto:N.Babakov@skoltech.ru)


model name: russian-inappropriate-messages


model author: Skoltech


task: text-classification


tags: pytorch, tf, jax, bert, text-classification, ru, transformers, toxic comments classification, infinity_compatible


language:


- ga
license: apache-2.0
tags:
- irish
- electra
widget:
- text: "Ceoltóir [MASK] ab ea Johnny Cash."


# gaELECTRA


[gaELECTRA](https://arxiv.org/abs/2107.12930) is an ELECTRA model trained on 7.9M Irish sentences. For more details, including the hyperparameters and pretraining corpora used please refer to our paper. For fine-tuning this model on a token classification task, e.g. Named Entity Recognition, use the discriminator model.


### Limitations and bias


Some data used to pretrain gaBERT was scraped from the web which potentially contains ethically problematic text (bias, hate, adult content, etc.). Consequently, downstream tasks/applications using gaBERT should be thoroughly tested with respect to ethical considerations.


### BibTeX entry and citation info


If you use this model in your research, please consider citing our paper:


```
@article{DBLP:journals/corr/abs-2107-12930,
  author    = {James Barry and
               Joachim Wagner and
               Lauren Cassidy and
               Alan Cowap and
               Teresa Lynn and
               Abigail Walsh and
               M{\'{\i}}che{\'{a}}l J. {\'{O}} Meachair and
               Jennifer Foster},
  title     = {gaBERT - an Irish Language Model},
  journal   = {CoRR},
  volume    = {abs/2107.12930},
  year      = {2021},
  url       = {https://arxiv.org/abs/2107.12930},
  archivePrefix = {arXiv},
  eprint    = {2107.12930},
  timestamp = {Fri, 30 Jul 2021 13:03:06 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2107-12930.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```


model name: electra-base-irish-cased-generator-v1


model author: DCU-NLP


task: fill-mask


tags: pytorch, electra, fill-mask, ga, arxiv:2107.12930, transformers, license:apache-2.0, irish, autonlp_compatible, infinity_compatible


model name: electra-base-irish-cased-discriminator-v1


tags: pytorch, electra, pretraining, ga, arxiv:2107.12930, transformers, license:apache-2.0, irish, infinity_compatible


- ga
license: apache-2.0
tags:
- irish
- bert
widget:
- text: "Ceoltóir [MASK] ab ea Johnny Cash."


# gaBERT


[gaBERT](https://arxiv.org/abs/2107.12930) is a BERT-base model trained on 7.9M Irish sentences. For more details, including the hyperparameters and pretraining corpora used please refer to our paper.


### How to use gaBERT with HuggingFace


```
from transformers import AutoModelWithLMHead, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("DCU-NLP/bert-base-irish-cased-v1")
model = AutoModelWithLMHead.from_pretrained("DCU-NLP/bert-base-irish-cased-v1")

sequence = f"Ceoltóir {tokenizer.mask_token} ab ea Johnny Cash."

input = tokenizer.encode(sequence, return_tensors="pt")
mask_token_index = torch.where(input == tokenizer.mask_token_id)[1]

token_logits = model(input)[0]
mask_token_logits = token_logits[0, mask_token_index, :]

top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()

for token in top_5_tokens:
    print(sequence.replace(tokenizer.mask_token, tokenizer.decode([token])))
```


model name: bert-base-irish-cased-v1


tags: pytorch, bert, fill-mask, ga, arxiv:2107.12930, transformers, license:apache-2.0, irish, autonlp_compatible, infinity_compatible


widget :


- text: "I like you. </s></s> I love you."


## bart-large-mnli


Trained by Facebook, [original source](https://github.com/pytorch/fairseq/tree/master/examples/bart)


model name: bart-large-mnli-bewgle


model author: bewgle


task: text-classification


tags: pytorch, bart, text-classification, transformers


## title: GFPEN
emoji: 📚
colorFrom: green
colorTo: blue
sdk: gradio
app_file: app.py
pinned: false


# Configuration


`title`: *string*\
Display title for the Space
`emoji`: *string*\
Space emoji (emoji-only character allowed)
`colorFrom`: *string*\
Color for Thumbnail gradient (red, yellow, green, blue, indigo, purple, pink, gray)
`colorTo`: *string*\
Color for Thumbnail gradient (red, yellow, green, blue, indigo, purple, pink, gray)
`sdk`: *string*\
Can be either `gradio` or `streamlit`
`app_file`: *string*\
Path to your main application file (which contains either `gradio` or `streamlit` Python code).\
Path is relative to the root of the repository.
`pinned`: *boolean*\
Whether the Space stays on top of your list.


model name: gpen


model author: Mahnoud


tags:


This is the *best performing* model used in the paper: "End-to-end Training For Financial Report Summarization"


model name: SumTO_FNS2020


model author: morenolq


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


This model is a fine-tuned version of [sshleifer/distilbart-cnn-12-6](https://huggingface.co/sshleifer/distilbart-cnn-12-6) on the BBC News Summary dataset (https://www.kaggle.com/pariza/bbc-news-summary).


The model has been generated as part of the in-lab practice of **Deep NLP course** currently held at Politecnico di Torino.


Training parameters:


- `num_train_epochs=2`
- `fp16=True`
- `per_device_train_batch_size=1`
- `warmup_steps=10`
- `weight_decay=0.01`
- `max_seq_length=100`


model name: distilbart-bbc


task: text2text-generation


tags: pytorch, bart, text2text-generation, transformers, autonlp_compatible


## license: other


model name: bangla_albert


model author: KamrusSamad


task: fill-mask


tags: pytorch, albert, fill-mask, transformers, license:other, autonlp_compatible, infinity_compatible


model name: tiny_A-2_H-2


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- conll2003
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: distilbert-base-uncased-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: conll2003
type: conll2003
args: conll2003
metrics:
    - name: Precision
type: precision
value: 0.9244616234124793
    - name: Recall
type: recall
value: 0.9364582168027744
    - name: F1
type: f1
value: 0.9304212515282871
    - name: Accuracy
type: accuracy
value: 0.9833987322668276


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-ner


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the conll2003 dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0623
- Precision: 0.9245
- Recall: 0.9365
- F1: 0.9304
- Accuracy: 0.9834


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.2377        | 1.0   | 878  | 0.0711          | 0.9176    | 0.9254 | 0.9215 | 0.9813   |
| 0.0514        | 2.0   | 1756 | 0.0637          | 0.9213    | 0.9346 | 0.9279 | 0.9831   |
| 0.031         | 3.0   | 2634 | 0.0623          | 0.9245    | 0.9365 | 0.9304 | 0.9834   |


### Framework versions


- Transformers 4.12.0
- Pytorch 1.9.0+cu111
- Datasets 1.14.0
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-ner


model author: thomaszz


task: token-classification


tags: pytorch, tensorboard, distilbert, token-classification, dataset:conll2003, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- conversational


# Naruto DialoGPT Model1.1


model name: DialoGPT-small-naruto1.1


model author: Sora4762


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


# Naruto DialoGPT Model


model name: DialoGPT-small-naruto


## language: fr
license: mit


# roberta-base-wechsel-french


Model trained with WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.


See the code here: https://github.com/CPJKU/wechsel


And the paper here: https://arxiv.org/abs/2112.06598


## Performance


### RoBERTa


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-french` | **82.43** | **90.88** | **86.65** |
| `camembert-base`   | 80.88 | 90.26 | 85.57 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-german` | **81.79** | **89.72** | **85.76** |
| `deepset/gbert-base`   | 78.64 | 89.46 | 84.05 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-chinese` | **78.32** | 80.55 | **79.44** |
| `bert-base-chinese`   | 76.55 | **82.05** | 79.30 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-swahili` | **75.05** | **87.39** | **81.22** |
| `xlm-roberta-base`   | 69.18 | 87.37 | 78.28 |


### GPT2


| Model | PPL |
|---|---|
| `gpt2-wechsel-french` | **19.71** |
| `gpt2` (retrained from scratch)  | 20.47 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-german` | **26.8** |
| `gpt2` (retrained from scratch)  | 27.63 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-chinese` | **51.97** |
| `gpt2` (retrained from scratch)  | 52.98 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-swahili` | **10.14** |
| `gpt2` (retrained from scratch)  | 10.58 |


See our paper for details.


## Citation


Please cite WECHSEL as


```
@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models},
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: roberta-base-wechsel-french


model author: benjamin


task: fill-mask


tags: pytorch, roberta, fill-mask, fr, arxiv:2112.06598, transformers, license:mit, autonlp_compatible, infinity_compatible


## language: sw
license: mit


# roberta-base-wechsel-swahili


## Performance


### RoBERTa


### GPT2


model name: roberta-base-wechsel-swahili


tags: pytorch, roberta, fill-mask, sw, arxiv:2112.06598, transformers, license:mit, autonlp_compatible, infinity_compatible


## language: de
license: mit


# roberta-base-wechsel-german


See our paper for details.


## Citation


Please cite WECHSEL as


model name: roberta-base-wechsel-german


tags: pytorch, roberta, fill-mask, de, arxiv:2112.06598, transformers, license:mit, autonlp_compatible, infinity_compatible


# gpt2-wechsel-german


model name: gpt2-wechsel-german


model author: benjamin


task: text-generation


tags: pytorch, gpt2, text-generation, de, arxiv:2112.06598, transformers, license:mit


language: de


widget:


- text: "In einer schockierenden Entdeckung fanden Wissenschaftler eine Herde Einhörner, die in einem abgelegenen, zuvor unerforschten Tal in den Anden lebten."


## license: mit


# GerPT2


German large and small versions of GPT2:


- https://huggingface.co/benjamin/gerpt2
- https://huggingface.co/benjamin/gerpt2-large


See the [GPT2 model card](https://huggingface.co/gpt2) for considerations on limitations and bias. See the [GPT2 documentation](https://huggingface.co/transformers/model_doc/gpt2.html) for details on GPT2.


## Comparison to [dbmdz/german-gpt2](https://huggingface.co/dbmdz/german-gpt2)


I evaluated both GerPT2-large and the other German GPT2, [dbmdz/german-gpt2](https://huggingface.co/dbmdz/german-gpt2) on the [CC-100](http://data.statmt.org/cc-100/) dataset and on the German Wikipedia:


|                   | CC-100 (PPL) | Wikipedia (PPL) |
|-------------------|--------------|-----------------|
| dbmdz/german-gpt2 | 49.47        | 62.92           |
| GerPT2            | 24.78        | 35.33           |
| GerPT2-large      | **16.08**    | **23.26**       |
|                   |              |                 |


See the script `evaluate.py` in the [GerPT2 Github repository](https://github.com/bminixhofer/gerpt2) for the code.


## Usage


```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("benjamin/gerpt2-large")
model = AutoModelForCausalLM.from_pretrained("benjamin/gerpt2-large")

prompt = "<your prompt>"

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe(prompt)[0]["generated_text"])
```


Also, two tricks might improve the generated text:


```python
output = model.generate(
    # during training an EOS token was used to mark the beginning of each text
    # so it can help to insert it at the start
    torch.tensor(
        [tokenizer.eos_token_id] + tokenizer.encode(prompt)
    ).unsqueeze(0),
    do_sample=True,
    # try setting bad_words_ids=[[0]] to disallow generating an EOS token, without this the model is
    # prone to ending generation early because a significant number of texts from the training corpus
    # is quite short
    bad_words_ids=[[0]],
    max_length=max_length,
)[0]
print(tokenizer.decode(output))
```


## Training details


GerPT2-large is trained on the entire German data (67GB) from the [CC-100 Corpus](http://data.statmt.org/cc-100/) and weights were initialized from the [English GPT2 model](https://huggingface.co/gpt2-large).
GerPT2-large was trained with:


- a batch size of 256
- using OneCycle learning rate with a maximum of 5e-3
- with AdamW with a weight decay of 0.01
- for 2 epochs


Training took roughly 12 days on 8 TPUv3 cores.


To train GerPT2-large, follow these steps. Scripts are located in the [Github repository](https://github.com/bminixhofer/gerpt2):


0. Download and unzip training data from http://data.statmt.org/cc-100/.
0. Train a tokenizer using `prepare/train_tokenizer.py`. As training data for the tokenizer I used a random subset of 5% of the CC-100 data.
0. (optionally) generate a German input embedding matrix with `prepare/generate_aligned_wte.py`. This uses a neat trick to semantically map tokens from the English tokenizer to tokens from the German tokenizer using aligned word embeddings. E. g.:


```
ĠMinde -> Ġleast
Ġjed -> Ġwhatsoever
flughafen -> Air
vermittlung -> employment
teilung -> ignment
ĠInterpretation -> Ġinterpretation
Ġimport -> Ġimported
hansa -> irl
genehmigungen -> exempt
ĠAuflist -> Ġlists
Ġverschwunden -> Ġdisappeared
ĠFlyers -> ĠFlyers
Kanal -> Channel
Ġlehr -> Ġteachers
Ġnahelie -> Ġconvenient
gener -> Generally
mitarbeiter -> staff
```


This helps a lot on a trial run I did, although I wasn't able to do a full comparison due to budget and time constraints. To use this WTE matrix it can be passed via the `wte_path` to the training script. Credit to [this blogpost](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787) for the idea of initializing GPT2 from English weights.


3. Tokenize the corpus using `prepare/tokenize_text.py`. This generates files for train and validation tokens in JSON Lines format.
3. Run the training script `train.py`! `run.sh` shows how this was executed for the full run with config `configs/tpu_large.json`.


## License


GerPT2 is licensed under the MIT License.


## Citing


Please cite GerPT2 as follows:


```
@misc{Minixhofer_GerPT2_German_large_2020,
author = {Minixhofer, Benjamin},
doi = {10.5281/zenodo.5509984},
month = {12},
title = {{GerPT2: German large and small versions of GPT2}},
url = {https://github.com/bminixhofer/gerpt2},
year = {2020}
}
```


## Acknowledgements


Thanks to [Hugging Face](https://huggingface.co) for awesome tools and infrastructure.
Huge thanks to [Artus Krohn-Grimberghe](https://twitter.com/artuskg) at [LYTiQ](https://www.lytiq.de/) for making this possible by sponsoring the resources used for training.


model name: gerpt2


model author: benjamin


task: text-generation


tags: pytorch, tf, jax, gpt2, text-generation, de, transformers, license:mit


## language: sw
license: mit


# gpt2-wechsel-swahili


Model trained with WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.


See the code here: https://github.com/CPJKU/wechsel


And the paper here: https://arxiv.org/abs/2112.06598


## Performance


### RoBERTa


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-french` | **82.43** | **90.88** | **86.65** |
| `camembert-base`   | 80.88 | 90.26 | 85.57 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-german` | **81.79** | **89.72** | **85.76** |
| `deepset/gbert-base`   | 78.64 | 89.46 | 84.05 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-chinese` | **78.32** | 80.55 | **79.44** |
| `bert-base-chinese`   | 76.55 | **82.05** | 79.30 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-swahili` | **75.05** | **87.39** | **81.22** |
| `xlm-roberta-base`   | 69.18 | 87.37 | 78.28 |


### GPT2


| Model | PPL |
|---|---|
| `gpt2-wechsel-french` | **19.71** |
| `gpt2` (retrained from scratch)  | 20.47 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-german` | **26.8** |
| `gpt2` (retrained from scratch)  | 27.63 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-chinese` | **51.97** |
| `gpt2` (retrained from scratch)  | 52.98 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-swahili` | **10.14** |
| `gpt2` (retrained from scratch)  | 10.58 |


See our paper for details.


## Citation


Please cite WECHSEL as


```
@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models},
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: gpt2-wechsel-swahili


model author: benjamin


task: text-generation


tags: pytorch, gpt2, text-generation, sw, arxiv:2112.06598, transformers, license:mit


## language: zh
license: mit


# roberta-base-wechsel-chinese


## Performance


### RoBERTa


### GPT2


model name: roberta-base-wechsel-chinese


task: fill-mask


tags: pytorch, roberta, fill-mask, zh, arxiv:2112.06598, transformers, license:mit, autonlp_compatible, infinity_compatible


# gpt2-wechsel-chinese


See our paper for details.


## Citation


Please cite WECHSEL as


model name: gpt2-wechsel-chinese


tags: pytorch, gpt2, text-generation, zh, arxiv:2112.06598, transformers, license:mit


language: de


widget:


- text: "In einer schockierenden Entdeckung fanden Wissenschaftler eine Herde Einhörner, die in einem abgelegenen, zuvor unerforschten Tal in den Anden lebten."


## license: mit


# GerPT2


German large and small versions of GPT2:


- https://huggingface.co/benjamin/gerpt2
- https://huggingface.co/benjamin/gerpt2-large


See the [GPT2 model card](https://huggingface.co/gpt2) for considerations on limitations and bias. See the [GPT2 documentation](https://huggingface.co/transformers/model_doc/gpt2.html) for details on GPT2.


## Comparison to [dbmdz/german-gpt2](https://huggingface.co/dbmdz/german-gpt2)


I evaluated both GerPT2-large and the other German GPT2, [dbmdz/german-gpt2](https://huggingface.co/dbmdz/german-gpt2) on the [CC-100](http://data.statmt.org/cc-100/) dataset and on the German Wikipedia:


|                   | CC-100 (PPL) | Wikipedia (PPL) |
|-------------------|--------------|-----------------|
| dbmdz/german-gpt2 | 49.47        | 62.92           |
| GerPT2            | 24.78        | 35.33           |
| GerPT2-large      | **16.08**    | **23.26**       |
|                   |              |                 |


See the script `evaluate.py` in the [GerPT2 Github repository](https://github.com/bminixhofer/gerpt2) for the code.


## Usage


```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("benjamin/gerpt2-large")
model = AutoModelForCausalLM.from_pretrained("benjamin/gerpt2-large")

prompt = "<your prompt>"

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe(prompt)[0]["generated_text"])
```


Also, two tricks might improve the generated text:


```python
output = model.generate(
    # during training an EOS token was used to mark the beginning of each text
    # so it can help to insert it at the start
    torch.tensor(
        [tokenizer.eos_token_id] + tokenizer.encode(prompt)
    ).unsqueeze(0),
    do_sample=True,
    # try setting bad_words_ids=[[0]] to disallow generating an EOS token, without this the model is
    # prone to ending generation early because a significant number of texts from the training corpus
    # is quite short
    bad_words_ids=[[0]],
    max_length=max_length,
)[0]
print(tokenizer.decode(output))
```


## Training details


GerPT2-large is trained on the entire German data (67GB) from the [CC-100 Corpus](http://data.statmt.org/cc-100/) and weights were initialized from the [English GPT2 model](https://huggingface.co/gpt2-large).
GerPT2-large was trained with:


- a batch size of 256
- using OneCycle learning rate with a maximum of 5e-3
- with AdamW with a weight decay of 0.01
- for 2 epochs


Training took roughly 12 days on 8 TPUv3 cores.


To train GerPT2-large, follow these steps. Scripts are located in the [Github repository](https://github.com/bminixhofer/gerpt2):


0. Download and unzip training data from http://data.statmt.org/cc-100/.
0. Train a tokenizer using `prepare/train_tokenizer.py`. As training data for the tokenizer I used a random subset of 5% of the CC-100 data.
0. (optionally) generate a German input embedding matrix with `prepare/generate_aligned_wte.py`. This uses a neat trick to semantically map tokens from the English tokenizer to tokens from the German tokenizer using aligned word embeddings. E. g.:


```
ĠMinde -> Ġleast
Ġjed -> Ġwhatsoever
flughafen -> Air
vermittlung -> employment
teilung -> ignment
ĠInterpretation -> Ġinterpretation
Ġimport -> Ġimported
hansa -> irl
genehmigungen -> exempt
ĠAuflist -> Ġlists
Ġverschwunden -> Ġdisappeared
ĠFlyers -> ĠFlyers
Kanal -> Channel
Ġlehr -> Ġteachers
Ġnahelie -> Ġconvenient
gener -> Generally
mitarbeiter -> staff
```


This helps a lot on a trial run I did, although I wasn't able to do a full comparison due to budget and time constraints. To use this WTE matrix it can be passed via the `wte_path` to the training script. Credit to [this blogpost](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787) for the idea of initializing GPT2 from English weights.


3. Tokenize the corpus using `prepare/tokenize_text.py`. This generates files for train and validation tokens in JSON Lines format.
3. Run the training script `train.py`! `run.sh` shows how this was executed for the full run with config `configs/tpu_large.json`.


## License


GerPT2 is licensed under the MIT License.


## Citing


Please cite GerPT2 as follows:


```
@misc{Minixhofer_GerPT2_German_large_2020,
author = {Minixhofer, Benjamin},
doi = {10.5281/zenodo.5509984},
month = {12},
title = {{GerPT2: German large and small versions of GPT2}},
url = {https://github.com/bminixhofer/gerpt2},
year = {2020}
}
```


## Acknowledgements


Thanks to [Hugging Face](https://huggingface.co) for awesome tools and infrastructure.
Huge thanks to [Artus Krohn-Grimberghe](https://twitter.com/artuskg) at [LYTiQ](https://www.lytiq.de/) for making this possible by sponsoring the resources used for training.


model name: gerpt2-large


model author: benjamin


task: text-generation


tags: pytorch, jax, gpt2, text-generation, de, transformers, license:mit


## language: fr
license: mit


# gpt2-wechsel-french


Model trained with WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.


See the code here: https://github.com/CPJKU/wechsel


And the paper here: https://arxiv.org/abs/2112.06598


## Performance


### RoBERTa


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-french` | **82.43** | **90.88** | **86.65** |
| `camembert-base`   | 80.88 | 90.26 | 85.57 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-german` | **81.79** | **89.72** | **85.76** |
| `deepset/gbert-base`   | 78.64 | 89.46 | 84.05 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-chinese` | **78.32** | 80.55 | **79.44** |
| `bert-base-chinese`   | 76.55 | **82.05** | 79.30 |


| Model | NLI Score | NER Score | Avg Score |
|---|---|---|---|
| `roberta-base-wechsel-swahili` | **75.05** | **87.39** | **81.22** |
| `xlm-roberta-base`   | 69.18 | 87.37 | 78.28 |


### GPT2


| Model | PPL |
|---|---|
| `gpt2-wechsel-french` | **19.71** |
| `gpt2` (retrained from scratch)  | 20.47 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-german` | **26.8** |
| `gpt2` (retrained from scratch)  | 27.63 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-chinese` | **51.97** |
| `gpt2` (retrained from scratch)  | 52.98 |


| Model | PPL |
|---|---|
| `gpt2-wechsel-swahili` | **10.14** |
| `gpt2` (retrained from scratch)  | 10.58 |


See our paper for details.


## Citation


Please cite WECHSEL as


```
@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models},
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: gpt2-wechsel-french


model author: benjamin


task: text-generation


tags: pytorch, gpt2, text-generation, fr, arxiv:2112.06598, transformers, license:mit


### **GPT-Macbeth**


A custom finetune of GPT-2 trained on a custom dataset of victorian literature


## Information


The goal of this finetune is to output high-quality victorian literature, while being customizable with Author's Note and being light to run (aka not being a GPT-Neo or GPT-Jax finetune, for now at least).


## Authors Note


Author's Note was added manually, so please appreciate it. :)


The format of it is [ Author: George Eliot; Genre: Horror, fantasy, novel; Tags: scary, magical, victorian ]
Some words will work well, some won't. Please make sure to have spaces before each ][.


Most popular victorian authors should work, but keep in mind that some authors (e.g. Mark Twain) will result in a somewhat weird behavior due to a quirk in the dataset that will be addressed in the next version of the finetune.


When it comes to the genres, "novel", "fiction", "horror" and "romance" work best, but from playing around with it, I've noticed that most other not too specific genres work pretty well too.


The tags are a bit complicated. Adding "normal" will result in a story without anything special (like no magic or fantasy element) and tends to be pretty low-pace. Using "real-life" will push the AI towards a historical/biographical path. Almost all tags should work. Using "man" or "woman" is supposed to semi-determine what gender the main character is, but it heavily depends on the chosen author.


## History


Version 0 - This was the first test version of the finetune, trained on GPT-2-small and with a really small dataset. The name was GPT-Kelini before it was renamed to GPT-Macbeth in V1.


Version 1 - The current version of the finetune. Trained on GPT-2-medium with a much, much bigger dataset compared to V0. Supports Author's Note


### Notes


Please use a very low temperature/randomness when using it, if you want to get anything out of it. Pumping the repetition penalty up helps a lot too.


The model was specifically converted to PyTorch so that most front-end GUI's should run it. It has been only tested on KoboldAI, but should theoretically work on others too.


For some odd reason, my finetune is capable of writing victorian NSFW content, if used the right way. No NSFW was in the dataset and considering the size of the model, it's really odd to see it do so. Perhaps the countless romantic novels in the dataset had something naughty in them, but I highly doubt it.


You may sometimes get roman numerals on random occasions, this shouldn't happen often, but if it does, it's again something that will be (manually, unfortunately) addressed in the next version of the finetune.


If you are wondering why I renamed my finetune to Macbeth, there are a few reasons: First, it sounds much better and smoother than Kelini, second, it's a play by Shakespeare that closely matches the writing style of some of the authors in my dataset, and third, the most important reason, it's was mentioned in Hamilton, so yes, my love with Hamilton is bleeding everywhere and yes, the next version of the dataset will try to have a Hamilton easter egg featuring the Author's Note.


### Credits


I want to thank HuggingFace for their tokenizer and everything they've done to make everything easier. Then is OpenAI for making GPT-2. I also want to thank most active people on the AIM Discord server in the community-projects channel. Thanks to Bran for finding a way to convert checkpoints to a PyTorch model, thanks to Mr. Seeker and Aedial for helping me in cleaning the dataset and to *finetune* from the NovelAI team for perhaps making my finetune output much better quality by telling me about the magic of the <\|endoftext\|> token.


P.S. If you happen to use it in something commercial or in an online demo or in any other way that is not for personal use, a credit will be greatly appreciated (and if you do something exciting with it, make sure to let me know, I'd be more than happy to see it being used by someone!).


model name: GPT-Macbeth


model author: Philipuss


tags: pytorch, tensorboard, gpt2, transformers


# capreolus/electra-base-msmarco


## Model description


ELECTRA-Base model (`google/electra-base-discriminator`) fine-tuned on the MS MARCO passage classification task. It is intended to be used as a `ForSequenceClassification` model, but requires some modification since it contains a BERT classification head rather than the standard ELECTRA classification head. See the [TFElectraRelevanceHead](https://github.com/capreolus-ir/capreolus/blob/master/capreolus/reranker/TFBERTMaxP.py) in the Capreolus BERT-MaxP implementation for a usage example.


This corresponds to the ELECTRA-Base model used to initialize PARADE (ELECTRA) in [PARADE: Passage Representation Aggregation for Document Reranking](https://arxiv.org/abs/2008.09093) by Li et al. It was converted from the released [TFv1 checkpoint](https://zenodo.org/record/3974431/files/vanilla_electra_base_on_MSMARCO.tar.gz). Please cite the PARADE paper if you use these weights.


model name: electra-base-msmarco


model author: Capreolus


task: text-classification


tags: pytorch, tf, electra, text-classification, arxiv:2008.09093, transformers, infinity_compatible


# capreolus/bert-base-msmarco


BERT-Base model (`google/bert_uncased_L-12_H-768_A-12`) fine-tuned on the MS MARCO passage classification task. It is intended to be used as a `ForSequenceClassification` model; see the [Capreolus BERT-MaxP implementation](https://github.com/capreolus-ir/capreolus/blob/master/capreolus/reranker/TFBERTMaxP.py) for a usage example.


This corresponds to the BERT-Base model used to initialize BERT-MaxP and PARADE variants in [PARADE: Passage Representation Aggregation for Document Reranking](https://arxiv.org/abs/2008.09093) by Li et al. It was converted from the released [TFv1 checkpoint](https://zenodo.org/record/3974431/files/vanilla_bert_base_on_MSMARCO.tar.gz). Please cite the PARADE paper if you use these weights.


model name: bert-base-msmarco


tags: pytorch, tf, jax, bert, text-classification, arxiv:2008.09093, transformers, infinity_compatible


tags:


- generated_from_trainer
model-index:
- name: bert-base-uncased-pretrained-mlm-coqa-stories
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# bert-base-uncased-pretrained-mlm-coqa-stories


This model was trained from scratch on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 1.8310


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.0573        | 1.0   | 2479 | 1.8805          |
| 1.9517        | 2.0   | 4958 | 1.8377          |
| 1.9048        | 3.0   | 7437 | 1.8310          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: bert-base-uncased-pretrained-mlm-coqa-stories


model author: alistvt


task: fill-mask


tags: pytorch, tensorboard, bert, fill-mask, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-base-uncased-pretrained-clm-coqa-stories
results: []


# bert-base-uncased-pretrained-clm-coqa-stories


This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0002


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.0201        | 1.0   | 2479 | 0.0018          |
| 0.0033        | 2.0   | 4958 | 0.0003          |
| 0.0014        | 3.0   | 7437 | 0.0002          |


### Framework versions


model name: bert-base-uncased-pretrained-clm-coqa-stories


model author: alistvt


task: text-generation


tags: pytorch, tensorboard, bert, text-generation, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


tags:


- generated_from_trainer
model-index:
- name: bert-base-uncased-pretrain-finetuned-coqa-falt
results: []


# bert-base-uncased-pretrain-finetuned-coqa-falt


This model is a fine-tuned version of [alistvt/bert-base-uncased-pretrained-mlm-coqa-stories](https://huggingface.co/alistvt/bert-base-uncased-pretrained-mlm-coqa-stories) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 2.8125


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 1e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 4


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 3.4039        | 0.29  | 2000  | 3.0921          |
| 3.1438        | 0.59  | 4000  | 2.8826          |
| 3.0252        | 0.88  | 6000  | 2.7885          |
| 2.7112        | 1.18  | 8000  | 2.7720          |
| 2.6703        | 1.47  | 10000 | 2.7581          |
| 2.6432        | 1.77  | 12000 | 2.7316          |
| 2.385         | 2.06  | 14000 | 2.7798          |
| 2.3314        | 2.36  | 16000 | 2.7836          |
| 2.3433        | 2.65  | 18000 | 2.7650          |
| 2.3604        | 2.95  | 20000 | 2.7585          |
| 2.2232        | 3.24  | 22000 | 2.8120          |
| 2.2094        | 3.53  | 24000 | 2.7945          |
| 2.2306        | 3.83  | 26000 | 2.8125          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.18.0
- Tokenizers 0.10.3


model name: bert-base-uncased-pretrain-finetuned-coqa-falt


model author: alistvt


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- generated_from_trainer
model-index:
- name: bert-base-uncased-pretrain-finetuned-coqa-falttened
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# bert-base-uncased-pretrain-finetuned-coqa-falttened


- Loss: 2.8655


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 3.2886        | 0.29  | 2000  | 3.0142          |
| 3.0801        | 0.59  | 4000  | 2.8347          |
| 2.9744        | 0.88  | 6000  | 2.7643          |
| 2.494         | 1.18  | 8000  | 2.7605          |
| 2.4417        | 1.47  | 10000 | 2.7790          |
| 2.4042        | 1.77  | 12000 | 2.7382          |
| 2.1285        | 2.06  | 14000 | 2.8588          |
| 2.0569        | 2.36  | 16000 | 2.8937          |
| 2.0794        | 2.65  | 18000 | 2.8511          |
| 2.0679        | 2.95  | 20000 | 2.8655          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: bert-base-uncased-pretrain-finetuned-coqa-falttened


model author: alistvt


task: question-answering


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilgpt2-finetuned-wikitext2
results:
  - task:
name: Causal Language Modeling
type: text-generation


# distilgpt2-finetuned-wikitext2


This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 3.6424


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.7608        | 1.0   | 2334 | 3.6655          |
| 3.6335        | 2.0   | 4668 | 3.6455          |
| 3.6066        | 3.0   | 7002 | 3.6424          |


### Framework versions


- Transformers 4.10.2
- Pytorch 1.9.0+cu102
- Datasets 1.12.0
- Tokenizers 0.10.3


model name: distilgpt2-finetuned-wikitext2


model author: Haotian


task: text-generation


tags: pytorch, tensorboard, gpt2, text-generation, transformers, license:apache-2.0, generated_from_trainer, model-index


language: uz
tags:


- uzbert
- uzbek
- bert
- cyrillic
license: mit
datasets:
- webcrawl


# UzBERT base model (uncased)


Pretrained model on Uzbek language (Cyrillic script) using a masked
language modeling and next sentence prediction objectives.


## How to use


You can use this model directly with a pipeline for masked language modeling:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='coppercitylabs/uzbert-base-uncased')
>>> unmasker("Алишер Навоий – улуғ ўзбек ва бошқа туркий халқларнинг [MASK], мутафаккири ва давлат арбоби бўлган.")

[
    {
        'token_str': 'шоири',
        'token': 13587,
        'score': 0.7974384427070618,
        'sequence': 'алишер навоий – улуғ ўзбек ва бошқа туркий халқларнинг шоири, мутафаккир ##и ва давлат арбоби бўлган.'
    },
    {
        'token_str': 'олими',
        'token': 18500,
        'score': 0.09166576713323593,
        'sequence': 'алишер навоий – улуғ ўзбек ва бошқа туркий халқларнинг олими, мутафаккир ##и ва давлат арбоби бўлган.'
    },
    {
        'token_str': 'асосчиси',
        'token': 7469,
        'score': 0.02451123297214508,
        'sequence': 'алишер навоий – улуғ ўзбек ва бошқа туркий халқларнинг асосчиси, мутафаккир ##и ва давлат арбоби бўлган.'
    },
    {
        'token_str': 'ёзувчиси',
        'token': 22439,
        'score': 0.017601722851395607,
        'sequence': 'алишер навоий – улуғ ўзбек ва бошқа туркий халқларнинг ёзувчиси, мутафаккир ##и ва давлат арбоби бўлган.'
    },
    {
        'token_str': 'устози',
        'token': 11494,
        'score': 0.010115668177604675,
        'sequence': 'алишер навоий – улуғ ўзбек ва бошқа туркий халқларнинг устози, мутафаккир ##и ва давлат арбоби бўлган.'
    }
]
```