CroSloEngual BERT is a trilingual model, using bert-base architecture, trained on Croatian, Slovenian, and English corpora. Focusing on three languages, the model performs better than [multilingual BERT](https://huggingface.co/bert-base-multilingual-cased), while still offering an option for cross-lingual knowledge transfer, which a monolingual model wouldn't.


Evaluation is presented in our article:


```
@Inproceedings{ulcar-robnik2020finest,
author = "Ulčar, M. and Robnik-Šikonja, M.",
year = 2020,
title = "{FinEst BERT} and {CroSloEngual BERT}: less is more in multilingual models",
editor = "Sojka, P and Kopeček, I and Pala, K and Horák, A",
booktitle = "Text, Speech, and Dialogue {TSD 2020}",
series = "Lecture Notes in Computer Science",
volume = 12284,
publisher = "Springer",
url = "https://doi.org/10.1007/978-3-030-58323-1_11",
}
```


The preprint is available at [arxiv.org/abs/2006.07890](https://arxiv.org/abs/2006.07890).


model name: crosloengual-bert


model author: EMBEDDIA


task: fill-mask


tags: pytorch, jax, bert, fill-mask, hr, sl, en, multilingual, arxiv:2006.07890, transformers, license:cc-by-4.0, autonlp_compatible, infinity_compatible


language:


- et


## license: cc-by-sa-4.0


# Usage


Load in transformers library with:


```
from transformers import AutoTokenizer, AutoModelForMaskedLM

  tokenizer = AutoTokenizer.from_pretrained("EMBEDDIA/est-roberta")
  model = AutoModelForMaskedLM.from_pretrained("EMBEDDIA/est-roberta")
```


# Est-RoBERTa


Est-RoBERTa model is a monolingual Estonian BERT-like model. It is closely related to French Camembert model https://camembert-model.fr/. The Estonian corpora used for training the model have 2.51 billion tokens in total. The subword vocabulary contains 40,000 tokens.


Est-RoBERTa was trained for 40 epochs.


model name: est-roberta


tags: pytorch, camembert, fill-mask, et, transformers, license:cc-by-sa-4.0, autonlp_compatible


- fi
- et
- en
- multilingual


## license: cc-by-4.0


# FinEst BERT


FinEst BERT is a trilingual model, using bert-base architecture, trained on Finnish, Estonian, and English corpora. Focusing on three languages, the model performs better than [multilingual BERT](https://huggingface.co/bert-base-multilingual-cased), while still offering an option for cross-lingual knowledge transfer, which a monolingual model wouldn't.


Evaluation is presented in our article:


model name: finest-bert


model author: EMBEDDIA


task: fill-mask


tags: pytorch, jax, bert, fill-mask, fi, et, en, multilingual, arxiv:2006.07890, transformers, license:cc-by-4.0, autonlp_compatible, infinity_compatible


language:


- lt
- lv
- en
- multilingual


## license: cc-by-sa-4.0


# LitLat BERT


LitLat BERT is a trilingual model, using xlm-roberta-base architecture, trained on Lithuanian, Latvian, and English corpora. Focusing on three languages, the model performs better than [multilingual BERT](https://huggingface.co/bert-base-multilingual-cased), while still offering an option for cross-lingual knowledge transfer, which a monolingual model wouldn't.


### Named entity recognition evaluation


We compare LitLat BERT with multilingual BERT (mBERT), XLM-RoBERTa (XLM-R) and monolingual Latvian BERT (LVBERT) (Znotins and Barzdins, 2020). The report the results as a macro F1 score of 3 named entity classes shared in all three datasets: person, location, organization.


Language | mBERT | XLM-R | LVBERT | LitLat
---|---|---|---|---
Latvian | 0.830 | 0.865 | 0.797 | **0.881**
Lithuanian | 0.797 | 0.817 | / | **0.850**
English | 0.939 | 0.937 | / | **0.943**


model name: litlat-bert


tags: pytorch, xlm-roberta, fill-mask, lt, lv, en, multilingual, transformers, license:cc-by-sa-4.0, autonlp_compatible


- sl


# Usage


Load in transformers library with:


```
from transformers import AutoTokenizer, AutoModelForMaskedLM

  tokenizer = AutoTokenizer.from_pretrained("EMBEDDIA/sloberta")
  model = AutoModelForMaskedLM.from_pretrained("EMBEDDIA/sloberta")
```


# SloBERTa


SloBERTa model is a monolingual Slovene BERT-like model. It is closely related to French Camembert model https://camembert-model.fr/. The corpora used for training the model have 3.47 billion tokens in total. The subword vocabulary contains 32,000 tokens. The scripts and programs used for data preparation and training the model are available on https://github.com/clarinsi/Slovene-BERT-Tool


SloBERTa was trained for 200,000 iterations or about 98 epochs.


## Corpora


The following corpora were used for training the model:


* Gigafida 2.0
* Kas 1.0
* Janes 1.0 (only Janes-news, Janes-forum, Janes-blog, Janes-wiki subcorpora)
* Slovenian parliamentary corpus siParl 2.0
* slWaC


model name: sloberta


model author: EMBEDDIA


task: fill-mask


tags: pytorch, camembert, fill-mask, sl, transformers, license:cc-by-sa-4.0, autonlp_compatible


tags:


- conversational


#12 epochs, each batch size 2, gradient accumulation steps 2, tail 20000


model name: XTC20000


model author: rlagusrlagus123


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


#12 epochs, each batch size 4, gradient accumulation steps 1, tail 4096.


#THIS SEEMS TO BE THE OPTIMAL SETUP.


model name: XTC4096


language:


- ru
tags:
- PyTorch
- Text2Image
thumbnail: "https://github.com/sberbank-ai/ru-clip"


# Model Card: ruCLIP


Disclaimer: The code for using model you can found [here](https://github.com/sberbank-ai/ru-clip).


# Model Details


The ruCLIP model was developed by researchers at SberDevices and Sber AI based on origin OpenAI paper.


# Model Type


The model uses a ViT-B/32 Transformer architecture (initialized from OpenAI checkpoint and freezed while training) as an image encoder and uses [ruGPT3Small](https://github.com/sberbank-ai/ru-gpts) as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.


# Documents


Our habr [post](https://habr.com/ru/company/sberdevices/blog/564440/).


# Usage


Code for using model you can obtain in our [repo](https://github.com/sberbank-ai/ru-clip).


```
from clip.evaluate.utils import (
    get_text_batch, get_image_batch, get_tokenizer,
    show_test_images, load_weights_only
)
import torch

# Load model and tokenizer
model, args = load_weights_only("ViT-B/32-small")
model = model.cuda().float().eval()
tokenizer = get_tokenizer()
# Load test images and prepare for model
images, texts = show_test_images(args)
input_ids, attention_mask = get_text_batch(["Это " + desc for desc in texts], tokenizer, args)
img_input = get_image_batch(images, args.img_transform, args)
# Call model
with torch.no_grad():
    logits_per_image, logits_per_text = model(
        img_input={"x": img_input},
        text_input={"x": input_ids, "attention_mask": attention_mask}
    )
```


# Performance


We evaluate our model on CIFAR100 and CIFAR10 datasets.


zero-shot classification CIFAR100 top1 accuracy 0.4057; top5 accuracy 0.6975.


zero-shot classification CIFAR10 top1 accuracy 0.7803; top5 accuracy 0.9834.


model name: ru-clip


model author: sberbank-ai


tags: ru, PyTorch, Text2Image


# ruclip-vit-base-patch32-224


**RuCLIP** (**Ru**ssian **C**ontrastive **L**anguage–**I**mage **P**retraining) is a multimodal model
for obtaining images and text similarities and rearranging captions and pictures.
RuCLIP builds on a large body of work on zero-shot transfer, computer vision, natural language processing and
multimodal learning.


Model was trained by [Sber AI](https://github.com/sberbank-ai) and [SberDevices](https://sberdevices.ru/) teams.


* Task: `text ranking`; `image ranking`; `zero-shot image classification`;
* Type: `encoder`
* Num Parameters: `150M`
* Training Data Volume: `240 million text-image pairs`
* Language: `Russian`
* Context Length: `77`
* Transformer Layers: `12`
* Transformer Width: `512`
* Transformer Heads: `8`
* Image Size: `224`
* Vision Layers: `12`
* Vision Width: `768`
* Vision Patch Size: `32`


## Usage [Github](https://github.com/sberbank-ai/ru-clip)


```
pip install ruclip
```


```python
clip, processor = ruclip.load("ruclip-vit-base-patch32-224", device="cuda")
```


## Performance


We have evaluated the performance on the following datasets:


| Dataset       | Metric Name    | Metric Result       |
|:--------------|:---------------|:--------------------|
| Food101       | acc            | 0.505		      	   |
| CIFAR10       | acc            | 0.818               |
| CIFAR100      | acc            | 0.504               |
| Birdsnap      | acc            | 0.115               |
| SUN397        | acc            | 0.452               |
| Stanford Cars | acc            | 0.433               |
| DTD           | acc            | 0.380	             |
| MNIST         | acc            | 0.447	             |
| STL10         | acc            | 0.932	             |
| PCam          | acc            | 0.501               |
| CLEVR         | acc            | 0.148               |
| Rendered SST2 | acc            | 0.489               |
| ImageNet      | acc            | 0.375               |
| FGVC Aircraft | mean-per-class | 0.033               |
| Oxford Pets   | mean-per-class | 0.560               |
| Caltech101    | mean-per-class | 0.786               |
| Flowers102    | mean-per-class | 0.401               |
| HatefulMemes  | roc-auc        | 0.564               |


# Authors


+ Alex Shonenkov: [Github](https://github.com/shonenkov), [Kaggle GM](https://www.kaggle.com/shonenkov)
+ Daniil Chesakov: [Github](https://github.com/Danyache)
+ Denis Dimitrov: [Github](https://github.com/denndimitrov)
+ Igor Pavlov: [Github](https://github.com/boomb0om)


model name: ruclip-vit-base-patch32-224


tags: pytorch, transformers


language:


- ru
tags:
- PyTorch
- Transformers


# BERT large model multitask (cased) for Sentence Embeddings in Russian language.


The model is described [in this article](https://habr.com/ru/company/sberdevices/blog/560748/)\
Russian SuperGLUE [metrics](https://russiansuperglue.com/login/submit_info/944)


For better quality, use mean token embeddings.


## Usage (HuggingFace Models Repository)


You can use the model directly from the model repository to compute sentence embeddings:


```python
from transformers import AutoTokenizer, AutoModel
import torch
#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
    return sum_embeddings / sum_mask
#Sentences we want sentence embeddings for
sentences = ['Привет! Как твои дела?',
             'А правда, что 42 твое любимое число?']
#Load AutoModel from huggingface model repository
tokenizer = AutoTokenizer.from_pretrained("sberbank-ai/sbert_large_mt_nlu_ru")
model = AutoModel.from_pretrained("sberbank-ai/sbert_large_mt_nlu_ru")
#Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=24, return_tensors='pt')
#Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)
#Perform pooling. In this case, mean pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
```


model name: sbert_large_mt_nlu_ru


model author: sberbank-ai


task: feature-extraction


tags: pytorch, tf, jax, bert, feature-extraction, ru, transformers, PyTorch, Transformers, infinity_compatible


- ru
tags:
- PyTorch
- Transformers
thumbnail: "https://github.com/sberbank-ai/model-zoo"


# ruT5-large


Model was trained by [SberDevices](https://sberdevices.ru/) team.


* Task: `text2text generation`
* Type: `encoder-decoder`
* Tokenizer: `bpe`
* Dict size: `32 101 `
* Num Parameters: `737 M`
* Training Data Volume `300 GB`


model name: ruT5-large


task: text2text-generation


tags: pytorch, t5, text2text-generation, ru, transformers, PyTorch, Transformers, autonlp_compatible


# ruclip-vit-large-patch14-224


**RuCLIP** (**Ru**ssian **C**ontrastive **L**anguage–**I**mage **P**retraining) is a multimodal model
for obtaining images and text similarities and rearranging captions and pictures.
RuCLIP builds on a large body of work on zero-shot transfer, computer vision, natural language processing and
multimodal learning.


Model was trained by [Sber AI](https://github.com/sberbank-ai) and [SberDevices](https://sberdevices.ru/) teams.


* Task: `text ranking`; `image ranking`; `zero-shot image classification`;
* Type: `encoder`
* Num Parameters: `430M`
* Training Data Volume: `240 million text-image pairs`
* Language: `Russian`
* Context Length: `77`
* Transformer Layers: `12`
* Transformer Width: `768`
* Transformer Heads: `12`
* Image Size: `224`
* Vision Layers: `24`
* Vision Width: `1024`
* Vision Patch Size: `14`


## Usage [Github](https://github.com/sberbank-ai/ru-clip)


```
pip install ruclip
```


```python
clip, processor = ruclip.load("ruclip-vit-large-patch14-224", device="cuda")
```


## Performance


We have evaluated the performance on the following datasets:


| Dataset       | Metric Name    | Metric Result       |
|:--------------|:---------------|:--------------------|
| Food101       | acc            | 0.597		      	   |
| CIFAR10       | acc            | 0.878	             |
| CIFAR100      | acc            | 0.511               |
| Birdsnap      | acc            | 0.172               |
| SUN397        | acc            | 0.484               |
| Stanford Cars | acc            | 0.559               |
| DTD           | acc            | 0.370	             |
| MNIST         | acc            | 0.337	             |
| STL10         | acc            | 0.934	             |
| PCam          | acc            | 0.520               |
| CLEVR         | acc            | 0.152               |
| Rendered SST2 | acc            | 0.529               |
| ImageNet      | acc            | 0.426               |
| FGVC Aircraft | mean-per-class | 0.046               |
| Oxford Pets   | mean-per-class | 0.604               |
| Caltech101    | mean-per-class | 0.777               |
| Flowers102    | mean-per-class | 0.455               |
| HatefulMemes  | roc-auc        | 0.530               |


# Authors


+ Alex Shonenkov: [Github](https://github.com/shonenkov), [Kaggle GM](https://www.kaggle.com/shonenkov)
+ Daniil Chesakov: [Github](https://github.com/Danyache)
+ Denis Dimitrov: [Github](https://github.com/denndimitrov)
+ Igor Pavlov: [Github](https://github.com/boomb0om)


model name: ruclip-vit-large-patch14-224


model author: sberbank-ai


tags: pytorch, transformers


# RuDOLPH-350M (Medium)


RuDOLPH: One Hyper-Modal Transformer can be creative as DALL-E and smart as CLIP


<img src="https://raw.githubusercontent.com/sberbank-ai/ru-dolph/master/pics/rudolph-generated.png" height="60" border="2"/>


* Task: `text2image generation`; `self reranking`; `text ranking`; `image ranking`; `image2text generation`; `zero-shot image classification`, `text2text generation`;
* Language: `Russian`
* Type: `encoder-decoder`
* Num Parameters: `350M`
* Training Data Volume: `156 million text-image pairs`


# Model Description


**Ru**ssian **D**iffusion **O**n **L**anguage **P**icture **H**yper-modality (RuDOLPH) 350M is a fast and light text-image-text transformer (350M GPT-3) designed for a quick and easy fine-tuning setup for the solution of various tasks: from generating images by text description and image classification to visual question answering and more. This model demonstrates the power of Hyper-modality Transformers.


*(!!!) Hyper-modality means generalized multi-modal, e.g., model that consists of two multi-modal parts: text-2-image and image-2-text becomes text and image hyper-modality model*


# Sparse Attention Mask


The primary proposed method is to modify the sparse transformer's attention mask to better control multi-modalities and up to the next level with "hyper-modality". It allows us to calculate the transitions of modalities in both directions, unlike another similar work DALL-E Transformer, which used only one direction, "text to image". The proposed "image to right text" direction is achieved by extension sparse attention mask to the right for auto-repressively text generation with image condition without attention to left text.


<img src="https://raw.githubusercontent.com/sberbank-ai/ru-dolph/master/pics/attention_masks.png" height="40" border="2"/>


+ Alex Shonenkov: [Github](https://github.com/shonenkov), [Kaggle GM](https://www.kaggle.com/shonenkov)
+ Michael Konstantinov: [Mishin Learning](https://t.me/mishin_learning), [Transformer Community](https://transformer.community/)


model name: RuDOLPH-350M


tags: pytorch


# ruclip-vit-base-patch16-224


**RuCLIP** (**Ru**ssian **C**ontrastive **L**anguage–**I**mage **P**retraining) is a multimodal model
for obtaining images and text similarities and rearranging captions and pictures.
RuCLIP builds on a large body of work on zero-shot transfer, computer vision, natural language processing and
multimodal learning.


Model was trained by [Sber AI](https://github.com/sberbank-ai) and [SberDevices](https://sberdevices.ru/) teams.


* Task: `text ranking`; `image ranking`; `zero-shot image classification`;
* Type: `encoder`
* Num Parameters: `150M`
* Training Data Volume: `240 million text-image pairs`
* Language: `Russian`
* Context Length: `77`
* Transformer Layers: `12`
* Transformer Width: `512`
* Transformer Heads: `8`
* Image Size: `224`
* Vision Layers: `12`
* Vision Width: `768`
* Vision Patch Size: `16`


## Usage [Github](https://github.com/sberbank-ai/ru-clip)


```
pip install ruclip
```


```python
clip, processor = ruclip.load("ruclip-vit-base-patch16-224", device="cuda")
```


## Performance


We have evaluated the performance on the following datasets:


| Dataset       | Metric Name    | Metric Result       |
|:--------------|:---------------|:--------------------|
| Food101       | acc            | 0.552		      	   |
| CIFAR10       | acc            | 0.810	             |
| CIFAR100      | acc            | 0.496               |
| Birdsnap      | acc            | 0.117               |
| SUN397        | acc            | 0.462               |
| Stanford Cars | acc            | 0.487               |
| DTD           | acc            | 0.401	             |
| MNIST         | acc            | 0.464	             |
| STL10         | acc            | 0.932	             |
| PCam          | acc            | 0.505               |
| CLEVR         | acc            | 0.128               |
| Rendered SST2 | acc            | 0.527               |
| ImageNet      | acc            | 0.401               |
| FGVC Aircraft | mean-per-class | 0.043               |
| Oxford Pets   | mean-per-class | 0.595               |
| Caltech101    | mean-per-class | 0.775               |
| Flowers102    | mean-per-class | 0.388               |
| HatefulMemes  | roc-auc        | 0.516               |


# Authors


+ Alex Shonenkov: [Github](https://github.com/shonenkov), [Kaggle GM](https://www.kaggle.com/shonenkov)
+ Daniil Chesakov: [Github](https://github.com/Danyache)
+ Denis Dimitrov: [Github](https://github.com/denndimitrov)
+ Igor Pavlov: [Github](https://github.com/boomb0om)


model name: ruclip-vit-base-patch16-224


model author: sberbank-ai


tags: pytorch, transformers


language:


- en
inference: false
pipeline_tag: false
datasets:
- conll2003
- wnut_17
- jnlpba
- conll2012
- BTC
- dfki-nlp/few-nerd
tags:
- PyTorch
model-index:
- name: "bert-base-NER-reptile-5-datasets"
results:
  - task:
name: few-shot-ner
type: named-entity-recognition
dataset:
name: few-nerd-inter
type: named-entity-recognition
metrics:
    - name: 5 way 1~2 shot
type: f1
value: 56.12
    - name: 5-way 5~10-shot
type: f1
value: 62.7
    - name: 10-way 1~2-shot
type: f1
value: 50.3
    - name: 10-way 5~10-shot
type: f1
value: 58.82


# BERT base uncased model pre-trained on 5 NER datasets


Model was trained by *SberIDP*. The pretraining process and technical details are described [in this article](https://habr.com/ru/company/sberbank/blog/649609/).


* Task: Named Entity Recognition
* Base model: [bert-base-uncased](https://huggingface.co/bert-base-uncased)
* Training Data is 5 datasets: [CoNLL-2003](https://aclanthology.org/W03-0419.pdf), [WNUT17](http://noisy-text.github.io/2017/emerging-rare-entities.html), [JNLPBA](http://www.geniaproject.org/shared-tasks/bionlp-jnlpba-shared-task-2004), [CoNLL-2012 (OntoNotes)](https://aclanthology.org/W12-4501.pdf), [BTC](https://www.derczynski.com/papers/btc.pdf)
* Testing was made in Few-Shot scenario on [Few-NERD dataset](https://github.com/thunlp/Few-NERD) using the model as a backbone for [StructShot](https://arxiv.org/abs/2010.02405)


The model is pretrained for NER task using [Reptile](https://openai.com/blog/reptile/) and can be finetuned for new entities with only a small amount of samples.


model name: bert-base-NER-reptile-5-datasets


task: token-classification


tags: pytorch, bert, token-classification, en, dataset:conll2003, dataset:wnut_17, dataset:jnlpba, dataset:conll2012, dataset:BTC, dataset:dfki-nlp/few-nerd, arxiv:2010.02405, transformers, PyTorch, model-index, autonlp_compatible, infinity_compatible


- ru
- en
pipeline_tag: text-to-image
tags:
- PyTorch
- Transformers
thumbnail: "https://github.com/sberbank-ai/ru-dalle"


# ruDALL-E Malevich (XL)


## Generate images from text


<img style="text-align:center; display:block;" src="https://huggingface.co/sberbank-ai/rudalle-Malevich/resolve/main/dalle-malevich.jpg" width="200">
"Avocado painting in the style of Malevich"


* [Technical Report (Russian)](https://habr.com/ru/company/sberbank/blog/586926)
* [Demo](https://rudalle.ru)


Model was trained by [Sber AI](https://github.com/sberbank-ai) and [SberDevices](https://sberdevices.ru/) teams.


* Task: `text2image generation`
* Type: `encoder-decoder`
* Num Parameters: `1.3 B`
* Training Data Volume: `120 million text-image pairs`


### Model Description


This is a 1.3 billion parameter model for Russian, recreating OpenAI's [DALL·E](https://openai.com/blog/dall-e/), a model capable of generating arbitrary images from a text prompt that describes the desired result.


The generation pipeline includes ruDALL-E, ruCLIP for ranging results, and a superresolution model.
You can use automatic translation into Russian to create desired images with ruDALL-E.


### How to Use


The easiest way to get familiar with the code and the models is to follow the inference notebook we provide in our [github repo](https://github.com/sberbank-ai/ru-dalle).


## Motivation


One might say that “investigate, master, and train” is our engineering motto.  Well, we caught the scent, and today we can say that we created from scratch a complete pipeline for generating images from descriptive textual input written in Russian.


Teams at SberAI, SberDevices, Samara University, AIRI and SberCloud all actively contributed.


We trained two versions of the model, each a different size, and named them after Russia’s great abstractionists: Vasily Kandinsky and Kazimir Malevich.


* ruDALL-E Kandinsky (XXL), with 12 billion parameters
* ruDALL-E Malevich (XL), having 1.3 billion parameters


Some of our models are already freely available:


* ruDALL-E Malevich (XL) [[GitHub](https://github.com/sberbank-ai/ru-dalle), [HuggingFace](https://huggingface.co/sberbank-ai/rudalle-Malevich)]
* Sber VQ-GAN [[GitHub](https://github.com/sberbank-ai/sber-vq-gan), [HuggingFace](https://huggingface.co/sberbank-ai/Sber-VQGAN)]
* ruCLIP Small [[GitHub](https://github.com/sberbank-ai/ru-clip), [HuggingFace](https://huggingface.co/sberbank-ai/ru-clip)]
* Super Resolution (Real ESRGAN) [[GitHub](https://github.com/sberbank-ai/Real-ESRGAN), [HuggingFace](https://huggingface.co/sberbank-ai/Real-ESRGAN)]
The latter two models are included in the pipeline for generating images from text (as you’ll see later on).


The models ruDALL-E Malevich (XL), ruDALL-E Kandinsky (XXL), ruCLIP Small, ruCLIP Large, and Super Resolution (Real ESRGAN) will also soon be available on [DataHub](https://mlspace.aicloud.sbercloud.ru/mlspace/datahub).


Training the ruDALL-E neural networks on the Christofari cluster has become the largest calculation task in Russia:


* ruDALL-E Kandinsky (XXL) was trained for 37 days on the 512 GPU TESLA V100, and then also for 11 more days on the 128 GPU TESLA V100, for a total of 20,352 GPU-days;
* ruDALL-E Malevich (XL) was trained for 8 days on the 128 GPU TESLA V100, and then also for 15 more days on the 192 GPU TESLA V100, for a total of 3,904 GPU-days.


Accordingly, training for both models totalled 24,256 GPU-days.


## Model capabilities


The long term goal of this research is the creation of multimodal neural networks. They will be able to pull on concepts from a variety of mediums---from text and visuals at first---in order to better understand the world as a whole.


Image generation might seem like the wrong rabbit hole in our century of big data and search engines.  But it actually addresses two important requirements that search is currently unable to cope with:


1. Being able to describe in writing exactly what you’re looking for and getting a completely new image created personally for you.
1. Being able to create at any time as many license-free illustrations as you could possibly want


"Grand Canyon"
<img style="text-align:center; display:block;" src="https://habrastorage.org/webt/kb/sv/ih/kbsvihfsmz3fx5mvitii0seimi0.jpeg" width="800">


"Salvador Dali picture"
<img style="text-align:center; display:block;" src="https://habrastorage.org/webt/r8/nl/oi/r8nloiq-l8j2ckg6pzh2pufsklm.jpeg" width="800">


"An eagle sits in a tree, looking to the side"
<img style="text-align:center; display:block;" src="https://habrastorage.org/r/w1560/getpro/habr/upload_files/10a/19c/fa2/10a19cfa2cc84aa7c8b99820890e908d.png" width="800">


"Elegant living room with green stuffed chairs"
<img style="text-align:center; display:block;" src="https://habrastorage.org/r/w1560/getpro/habr/upload_files/6fe/e69/d7c/6fee69d7c392239d587725799e0e41e4.png" width="800">


“Raccoon with a gun”
<img style="text-align:center; display:block;" src="https://habrastorage.org/r/w1560/getpro/habr/upload_files/3bb/1b8/7c4/3bb1b87c45bf9305cd342ae9900ac245.png" width="800">


“Pretty lake at sunset”
<img style="text-align:center; display:block;" src="https://habrastorage.org/r/w1560/getpro/habr/upload_files/241/781/fe9/241781fe99da510d4d5fea03af635e88.png" width="800">


model name: rudalle-Malevich


model author: sberbank-ai


task: text-to-image


tags: pytorch, ru, en, PyTorch, Transformers, text-to-image


# Emojich


### generate emojis from text


Model was trained by [Sber AI](https://github.com/sberbank-ai)


* Task: `text2image generation`
* Num Parameters: `1.3 B`
* Training Data Volume: `120 million text-image pairs` & [`2749 text-emoji pairs`](https://www.kaggle.com/shonenkov/russian-emoji)


### Model Description


😋 Emojich is a 1.3 billion params model from the family GPT3-like, it generates emoji-style images with the brain of ◾ Malevich.


### Fine-tuning stage:


The main goal of fine-tuning is trying to keep the generalization of [ruDALL-E Malevich (XL)](https://huggingface.co/sberbank-ai/rudalle-Malevich)
model on text to emoji tasks. ruDALL-E Malevich is a multi-modality big pretrained transformer, that uses images and texts.
The idea with freezing feedforward and self-attention layers in pretrained transformer is demonstrated high performance in changing different modalities.
Also, the model has a good chance for over-fitting text modality and lost generalization.
To deal with this problem is increased coefficient 10^3 in weighted cross-entropy loss for image codebooks part.


Full version of training code is available on Kaggle: [![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://www.kaggle.com/shonenkov/emojich-rudall-e)


### Examples of generated emojis


All examples are generated automatically (without manual cherry-picking) with hyper-parameters:
seed 42, batch size 16, top-k 2048, top-p 0.995, temperature 1.0, GPU A100.
For making better generative emojis should use more attempts (~512) and select the best one manually.


*Remember, the great art makers became "great" after creating just only one masterpiece.*


![](./pics/examples.png)


model name: rudalle-Emojich


model author: sberbank-ai


tags: pytorch


language:


- ru
tags:
- PyTorch
- Transformers
thumbnail: "https://github.com/sberbank-ai/ru-gpts"


# rugpt3medium\_based\_on\_gpt2


Model was trained with sequence length 1024 using transformers lib by [SberDevices](https://sberdevices.ru/) team on 80B tokens for 3 epoch. After that model was finetuned on 2048 context.


Total training time was around 16 days on 64 GPUs.\
Final perplexity on test set is `17.4`.


model name: rugpt3medium_based_on_gpt2


tags: pytorch, ru, transformers, PyTorch, Transformers


# rugpt2large


Model was trained with sequence length 1024 using transformers by [SberDevices](https://sberdevices.ru/) team on 170Gb data on 64 GPUs 3 weeks.


model name: rugpt2large


- ru
tags:
- PyTorch
- Transformers


# BERT large model (uncased) for Sentence Embeddings in Russian language.


The model is described [in this article](https://habr.com/ru/company/sberdevices/blog/527576/)\
For better quality, use mean token embeddings.


## Usage (HuggingFace Models Repository)


You can use the model directly from the model repository to compute sentence embeddings:


```python
from transformers import AutoTokenizer, AutoModel
import torch

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
    return sum_embeddings / sum_mask

#Sentences we want sentence embeddings for
sentences = ['Привет! Как твои дела?',
             'А правда, что 42 твое любимое число?']

#Load AutoModel from huggingface model repository
tokenizer = AutoTokenizer.from_pretrained("sberbank-ai/sbert_large_nlu_ru")
model = AutoModel.from_pretrained("sberbank-ai/sbert_large_nlu_ru")

#Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=24, return_tensors='pt')

#Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

#Perform pooling. In this case, mean pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
```


model name: sbert_large_nlu_ru


model author: sberbank-ai


task: feature-extraction


tags: pytorch, jax, bert, feature-extraction, ru, transformers, PyTorch, Transformers, infinity_compatible


# ruclip-vit-base-patch32-384


**RuCLIP** (**Ru**ssian **C**ontrastive **L**anguage–**I**mage **P**retraining) is a multimodal model
for obtaining images and text similarities and rearranging captions and pictures.
RuCLIP builds on a large body of work on zero-shot transfer, computer vision, natural language processing and
multimodal learning.


Model was trained by [Sber AI](https://github.com/sberbank-ai) and [SberDevices](https://sberdevices.ru/) teams.


* Task: `text ranking`; `image ranking`; `zero-shot image classification`;
* Type: `encoder`
* Num Parameters: `150M`
* Training Data Volume: `240 million text-image pairs`
* Language: `Russian`
* Context Length: `77`
* Transformer Layers: `12`
* Transformer Width: `512`
* Transformer Heads: `8`
* Image Size: `384`
* Vision Layers: `12`
* Vision Width: `768`
* Vision Patch Size: `32`


## Usage [Github](https://github.com/sberbank-ai/ru-clip)


```
pip install ruclip
```


```python
clip, processor = ruclip.load("ruclip-vit-base-patch32-384", device="cuda")
```


## Performance


We have evaluated the performance on the following datasets:


| Dataset       | Metric Name    | Metric Result               |
|:--------------|:---------------|:----------------------------|
| Food101       | acc            | 0.642                       |
| CIFAR10       | acc            | 0.862                       |
| CIFAR100      | acc            | 0.529                       |
| Birdsnap      | acc            | 0.161                       |
| SUN397        | acc            | 0.510                       |
| Stanford Cars | acc            | 0.572                       |
| DTD           | acc            | 0.390	                     |
| MNIST         | acc            | 0.404	                     |
| STL10         | acc            | 0.946	                     |
| PCam          | acc            | 0.506                       |
| CLEVR         | acc            | 0.188                       |
| Rendered SST2 | acc            | 0.508                       |
| ImageNet      | acc            | 0.451                       |
| FGVC Aircraft | mean-per-class | 0.053                       |
| Oxford Pets   | mean-per-class | 0.587                       |
| Caltech101    | mean-per-class | 0.834	                     |
| Flowers102    | mean-per-class | 0.449                       |
| HatefulMemes  | roc-auc        | 0.537                       |


# Authors


+ Alex Shonenkov: [Github](https://github.com/shonenkov), [Kaggle GM](https://www.kaggle.com/shonenkov)
+ Daniil Chesakov: [Github](https://github.com/Danyache)
+ Denis Dimitrov: [Github](https://github.com/denndimitrov)
+ Igor Pavlov: [Github](https://github.com/boomb0om)


model name: ruclip-vit-base-patch32-384


tags: pytorch, transformers


language:


- ru
tags:
- PyTorch
- Transformers
thumbnail: "https://github.com/sberbank-ai/ru-gpts"


# rugpt3small\_based\_on\_gpt2


Model was trained with sequence length 1024 using transformers by [SberDevices](https://sberdevices.ru/) team on 80B tokens around 3 epoch. After that model was finetuned on 2048 context.


Total training time took around one week on 32 GPUs.


model name: rugpt3small_based_on_gpt2


model author: sberbank-ai


task: text-generation


tags: pytorch, jax, gpt2, text-generation, ru, transformers, PyTorch, Transformers


# rugpt3xl


Model was trained with 512 sequence length using [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron](https://github.com/NVIDIA/Megatron-LM) code by [SberDevices](https://sberdevices.ru/) team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.\
*Note! Model has sparse attention blocks.*


Total training time was around 10 days on 256 GPUs.\
Final perplexity on test set is `12.05`.
Model parameters: 1.3B.


model name: rugpt3xl


tags: ru, PyTorch, Transformers


- ru
- en
tags:
- PyTorch
thumbnail: "https://github.com/sberbank-ai/Real-ESRGAN"


# Real-ESRGAN


PyTorch implementation of a Real-ESRGAN model trained on custom dataset. This model shows better results on faces compared to the original version. It is also easier to integrate this model into your projects.


Real-ESRGAN is an upgraded ESRGAN trained with pure synthetic data is capable of enhancing details while removing annoying artifacts for common real-world images.


- [Paper](https://arxiv.org/abs/2107.10833)
- [Official github](https://github.com/xinntao/Real-ESRGAN)
- [Ours github](https://github.com/sberbank-ai/Real-ESRGAN)


## Usage


Code for using model you can obtain in our [repo](https://github.com/sberbank-ai/Real-ESRGAN).


```python
import torch
from PIL import Image
import numpy as np
from realesrgan import RealESRGAN

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = RealESRGAN(device, scale=4)
model.load_weights('weights/RealESRGAN_x4.pth')

path_to_image = 'inputs/lr_image.png'
image = Image.open(path_to_image).convert('RGB')

sr_image = model.predict(image)

sr_image.save('results/sr_image.png')
```


model name: Real-ESRGAN


model author: sberbank-ai


tags: ru, en, arxiv:2107.10833, PyTorch


language:


# rugpt3large\_based\_on\_gpt2


Model was trained with sequence length 1024 using transformers lib by [SberDevices](https://sberdevices.ru/) team on 80B tokens for 3 epochs. After that model was finetuned 1 epoch with sequence length 2048.


Total training time was around 14 days on 128 GPUs for 1024 context and few days on 16 GPUs for 2048 context.\
Final perplexity on test set is `13.6`.


model name: rugpt3large_based_on_gpt2


task: text-generation


- ru
tags:
- PyTorch
- Transformers
thumbnail: "https://github.com/sberbank-ai/model-zoo"


# ruRoberta-large


Model was trained by [SberDevices](https://sberdevices.ru/) team.


* Task: `mask filling`
* Type: `encoder`
* Tokenizer: `bbpe`
* Dict size: `50 257`
* Num Parameters: `355 M`
* Training Data Volume `250 GB`


model name: ruRoberta-large


task: fill-mask


tags: pytorch, roberta, fill-mask, ru, transformers, PyTorch, Transformers, autonlp_compatible, infinity_compatible


# Sber-VQGAN


## Part of the ruDALL-E Malevich (XL)


model name: Sber-VQGAN


model author: sberbank-ai


tags:


language:


# ruT5-base


* Task: `text2text generation`
* Type: `encoder-decoder`
* Tokenizer: `bpe`
* Dict size: `32 101`
* Num Parameters: `222 M`
* Training Data Volume `300 GB`


model name: ruT5-base


task: text2text-generation


tags: pytorch, t5, text2text-generation, ru, transformers, PyTorch, Transformers, autonlp_compatible


# ruBert-large


* Task: `mask filling`
* Type: `encoder`
* Tokenizer: `bpe`
* Dict size: `120 138`
* Num Parameters: `427 M`
* Training Data Volume `30 GB`


model name: ruBert-large


tags: pytorch, ru, transformers, PyTorch, Transformers


* Task: `mask filling`
* Type: `encoder`
* Tokenizer: `bpe`
* Dict size: `120 138`
* Num Parameters: `178 M`
* Training Data Volume `30 GB`


model name: ruBert-base


## license: afl-3.0


model name: nlpProject


model author: padalah


tags: license:afl-3.0


tags:


- conversational


#Chandler Bing DialoGPT Model


model name: DialogGPT-small-chandlerbingg


model author: wadeed


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


license: mit
tags:


- generated_from_trainer
datasets:
- amazon_reviews_multi
model-index:
- name: xlm-roberta-base-finetuned-marc
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xlm-roberta-base-finetuned-marc


This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the amazon_reviews_multi dataset.
It achieves the following results on the evaluation set:


- Loss: 1.0171
- Mae: 0.5310


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss | Mae    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 1.1404        | 1.0   | 308  | 1.0720          | 0.5398 |
| 0.9805        | 2.0   | 616  | 1.0171          | 0.5310 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: xlm-roberta-base-finetuned-marc


model author: ashish-chouhan


task: text-classification


tags: pytorch, tensorboard, xlm-roberta, text-classification, dataset:amazon_reviews_multi, transformers, license:mit, generated_from_trainer, model-index


language: "nl"
thumbnail: "https://github.com/iPieter/RobBERT/raw/master/res/robbert_logo.png"
tags:


- Dutch
- Flemish
- RoBERTa
- RobBERT
- RobBERTje
license: mit
datasets:
- oscar
- oscar (NL)
- dbrd
- lassy-ud
- europarl-mono
- conll2002
widget:
- text: "Hallo, ik ben RobBERTje, een gedistilleerd <mask> taalmodel van de KU Leuven."


# About RobBERTje


RobBERTje is a collection of distilled models based on [RobBERT](http://github.com/iPieter/robbert). There are multiple models with different sizes and different training settings, which you can choose for your use-case.


We are also continuously working on releasing better-performing models, so watch [the repository](http://github.com/iPieter/robbertje) for updates.


# News


- **July 2, 2021**: Publicly released 4 RobBERTje models.
- **May 12, 2021**: RobBERTje was accepted at [CLIN31](https://www.clin31.ugent.be) for an oral presentation!


# The models


| Model        | Description | Parameters | Training size | Huggingface id                                                                     |
|--------------|-------------|------------------|-------------------|------------------------------------------------------------------------------------|
| Non-shuffled | Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation.            | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-non-shuffled](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-non-shuffled)  |
| Shuffled     | Trained on the publicly available and shuffled OSCAR corpus.            | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-shuffled](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-shuffled)     |
| Merged (p=0.5)       | Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%.           | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-merged](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-merged)       |
| BORT         | A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT).            | 46 M             | 1 GB              | this model         |


# Results


## Intrinsic results


We calculated the *pseudo perplexity* (PPPL) from [cite](), which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.


| Model             | PPPL      |
|-------------------|-----------|
| RobBERT (teacher) | 7.76      |
| Non-shuffled      | 12.95     |
| Shuffled          | 18.74     |
| Merged (p=0.5)    | 17.10     |
| BORT              | 26.44     |


## Extrinsic results


We also evaluated our models on sereral downstream tasks, just like the teacher model RobBERT. Since that evaluation, a [Dutch NLI task named SICK-NL](https://arxiv.org/abs/2101.05716) was also released and we evaluated our models with it as well.


| Model            | DBRD      | DIE-DAT   | NER       | POS       |SICK-NL   |
|------------------|-----------|-----------|-----------|-----------|----------|
| RobBERT (teacher)|94.4       | 99.2      |89.1       |96.4       | 84.2     |
| Non-shuffled     |90.2       | 98.4      |82.9       |95.5       | 83.4     |
| Shuffled         |92.5       | 98.2      |82.7       |95.6       | 83.4     |
| Merged (p=0.5)   |92.9       | 96.5      |81.8       |95.2       | 82.8     |
| BORT             |89.6       | 92.2      |79.7       |94.3       | 81.0     |


model name: robbertje-1-gb-bort


model author: DTAI-KULeuven


task: fill-mask


tags: pytorch, roberta, fill-mask, nl, dataset:oscar, dataset:oscar (NL), dataset:dbrd, dataset:lassy-ud, dataset:europarl-mono, dataset:conll2002, arxiv:2101.05716, transformers, Dutch, Flemish, RoBERTa, RobBERT, RobBERTje, license:mit, autonlp_compatible, infinity_compatible


language: "multilingual"
tags:


- Dutch
- French
- English
- Tweets
- Sentiment analysis
widget:
- text: "I really wish I could leave my house after midnight, this makes no sense!"


# Measuring Shifts in Attitudes Towards COVID-19 Measures in Belgium Using Multilingual BERT


[Blog post »](https://people.cs.kuleuven.be/~pieter.delobelle/attitudes-towards-covid-19-measures/?utm_source=huggingface&utm_medium=social&utm_campaign=corona_tweets)   · [paper »](http://arxiv.org/abs/2104.09947)


This model can be used to determine if a tweet expresses support or not for a curfew. The model was trained on manually labeled tweets from Belgium in Dutch, French and English.


We categorized several months worth of these Tweets by topic (government COVID measure) and opinion expressed. Below is a timeline of the relative number of Tweets on the curfew topic (middle) and the fraction of those Tweets that find the curfew too strict, too loose, or a suitable measure (bottom), with the number of daily cases in Belgium to give context on the pandemic situation (top).


Models used in this paper are on HuggingFace:


- https://huggingface.co/DTAI-KULeuven/mbert-corona-tweets-belgium-curfew-support
- https://huggingface.co/DTAI-KULeuven/mbert-corona-tweets-belgium-topics


model name: mbert-corona-tweets-belgium-curfew-support


task: text-classification


tags: pytorch, jax, bert, text-classification, multilingual, arxiv:2104.09947, transformers, Dutch, French, English, Tweets, Sentiment analysis, infinity_compatible


language: "nl"
thumbnail: "https://github.com/iPieter/RobBERT/raw/master/res/robbert_logo.png"
tags:


- Dutch
- Flemish
- RoBERTa
- RobBERT
- RobBERTje
license: mit
datasets:
- oscar
- oscar (NL)
- dbrd
- lassy-ud
- europarl-mono
- conll2002
widget:
- text: "Hallo, ik ben RobBERTje, een gedistilleerd <mask> taalmodel van de KU Leuven."


# About RobBERTje


RobBERTje is a collection of distilled models based on [RobBERT](http://github.com/iPieter/robbert). There are multiple models with different sizes and different training settings, which you can choose for your use-case.


We are also continuously working on releasing better-performing models, so watch [the repository](http://github.com/iPieter/robbertje) for updates.


# News


- **July 2, 2021**: Publicly released 4 RobBERTje models.
- **May 12, 2021**: RobBERTje was accepted at [CLIN31](https://www.clin31.ugent.be) for an oral presentation!


# The models


| Model        | Description | Parameters | Training size | Huggingface id                                                                     |
|--------------|-------------|------------------|-------------------|------------------------------------------------------------------------------------|
| Non-shuffled | Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation.            | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-non-shuffled](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-non-shuffled) |
| Shuffled     | Trained on the publicly available and shuffled OSCAR corpus.            | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-shuffled](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-shuffled)     |
| Merged (p=0.5)       | Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%.           | 74 M             | 1 GB              | this model       |
| BORT         | A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT).            | 46 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-bort](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-bort)         |


# Results


## Intrinsic results


We calculated the *pseudo perplexity* (PPPL) from [cite](), which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.


| Model             | PPPL      |
|-------------------|-----------|
| RobBERT (teacher) | 7.76      |
| Non-shuffled      | 12.95     |
| Shuffled          | 18.74     |
| Merged (p=0.5)    | 17.10     |
| BORT              | 26.44     |


## Extrinsic results


We also evaluated our models on sereral downstream tasks, just like the teacher model RobBERT. Since that evaluation, a [Dutch NLI task named SICK-NL](https://arxiv.org/abs/2101.05716) was also released and we evaluated our models with it as well.


| Model            | DBRD      | DIE-DAT   | NER       | POS       |SICK-NL   |
|------------------|-----------|-----------|-----------|-----------|----------|
| RobBERT (teacher)|94.4       | 99.2      |89.1       |96.4       | 84.2     |
| Non-shuffled     |90.2       | 98.4      |82.9       |95.5       | 83.4     |
| Shuffled         |92.5       | 98.2      |82.7       |95.6       | 83.4     |
| Merged (p=0.5)   |92.9       | 96.5      |81.8       |95.2       | 82.8     |
| BORT             |89.6       | 92.2      |79.7       |94.3       | 81.0     |


model name: robbertje-1-gb-merged


model author: DTAI-KULeuven


task: fill-mask


tags: pytorch, roberta, fill-mask, nl, dataset:oscar, dataset:oscar (NL), dataset:dbrd, dataset:lassy-ud, dataset:europarl-mono, dataset:conll2002, arxiv:2101.05716, transformers, Dutch, Flemish, RoBERTa, RobBERT, RobBERTje, license:mit, autonlp_compatible, infinity_compatible


# About RobBERTje


| Model        | Description | Parameters | Training size | Huggingface id                                                                     |
|--------------|-------------|------------------|-------------------|------------------------------------------------------------------------------------|
| Non-shuffled | Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation.            | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-non-shuffled](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-non-shuffled) |
| Shuffled     | Trained on the publicly available and shuffled OSCAR corpus.            | 74 M             | 1 GB              | this model     |
| Merged (p=0.5)       | Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%.           | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-merged](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-merged)       |
| BORT         | A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT).            | 46 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-bort](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-bort)         |


model name: robbertje-1-gb-shuffled


language: "multilingual"
tags:


- Dutch
- French
- English
- Tweets
- Topic classification
widget:
- text: "I really can't wait for this lockdown to be over and go back to waking up early."


# Measuring Shifts in Attitudes Towards COVID-19 Measures in Belgium Using Multilingual BERT


[Blog post »](https://people.cs.kuleuven.be/~pieter.delobelle/attitudes-towards-covid-19-measures/?utm_source=huggingface&utm_medium=social&utm_campaign=corona_tweets)   · [paper »](http://arxiv.org/abs/2104.09947)


We categorized several months worth of these Tweets by topic (government COVID measure) and opinion expressed. Below is a timeline of the relative number of Tweets on the curfew topic (middle) and the fraction of those Tweets that find the curfew too strict, too loose, or a suitable measure (bottom), with the number of daily cases in Belgium to give context on the pandemic situation (top).


Models used in this paper are on HuggingFace:


- https://huggingface.co/DTAI-KULeuven/mbert-corona-tweets-belgium-curfew-support
- https://huggingface.co/DTAI-KULeuven/mbert-corona-tweets-belgium-topics


model name: mbert-corona-tweets-belgium-topics


task: text-classification


tags: pytorch, jax, bert, text-classification, multilingual, arxiv:2104.09947, transformers, Dutch, French, English, Tweets, Topic classification, infinity_compatible


language: "nl"
thumbnail: "https://github.com/iPieter/RobBERT/raw/master/res/robbert_logo.png"
tags:


- Dutch
- Flemish
- RoBERTa
- RobBERT
- RobBERTje
license: mit
datasets:
- oscar
- oscar (NL)
- dbrd
- lassy-ud
- europarl-mono
- conll2002
widget:
- text: "Hallo, ik ben RobBERTje, een gedistilleerd <mask> taalmodel van de KU Leuven."


RobBERTje is a collection of distilled models based on [RobBERT](http://github.com/iPieter/robbert). There are multiple models with different sizes and different training settings, which you can choose for your use-case.


We are also continuously working on releasing better-performing models, so watch [the repository](http://github.com/iPieter/robbertje) for updates.


# News


- **July 2, 2021**: Publicly released 4 RobBERTje models.
- **May 12, 2021**: RobBERTje was accepted at [CLIN31](https://www.clin31.ugent.be) for an oral presentation!


# The models


| Model        | Description | Parameters | Training size | Huggingface id                                                                     |
|--------------|-------------|------------------|-------------------|------------------------------------------------------------------------------------|
| Non-shuffled | Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation.            | 74 M             | 1 GB              | this model |
| Shuffled     | Trained on the publicly available and shuffled OSCAR corpus.            | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-shuffled](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-shuffled)     |
| Merged (p=0.5)       | Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%.           | 74 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-merged](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-merged)       |
| BORT         | A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT).            | 46 M             | 1 GB              | [DTAI-KULeuven/robbertje-1-gb-bort](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-bort)         |


# Results


## Intrinsic results


We calculated the *pseudo perplexity* (PPPL) from [cite](), which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.


| Model             | PPPL      |
|-------------------|-----------|
| RobBERT (teacher) | 7.76      |
| Non-shuffled      | 12.95     |
| Shuffled          | 18.74     |
| Merged (p=0.5)    | 17.10     |
| BORT              | 26.44     |


## Extrinsic results


We also evaluated our models on sereral downstream tasks, just like the teacher model RobBERT. Since that evaluation, a [Dutch NLI task named SICK-NL](https://arxiv.org/abs/2101.05716) was also released and we evaluated our models with it as well.


| Model            | DBRD      | DIE-DAT   | NER       | POS       |SICK-NL   |
|------------------|-----------|-----------|-----------|-----------|----------|
| RobBERT (teacher)|94.4       | 99.2      |89.1       |96.4       | 84.2     |
| Non-shuffled     |90.2       | 98.4      |82.9       |95.5       | 83.4     |
| Shuffled         |92.5       | 98.2      |82.7       |95.6       | 83.4     |
| Merged (p=0.5)   |92.9       | 96.5      |81.8       |95.2       | 82.8     |
| BORT             |89.6       | 92.2      |79.7       |94.3       | 81.0     |


model name: robbertje-1-gb-non-shuffled


model author: DTAI-KULeuven


task: fill-mask


tags: pytorch, roberta, fill-mask, nl, dataset:oscar, dataset:oscar (NL), dataset:dbrd, dataset:lassy-ud, dataset:europarl-mono, dataset:conll2002, arxiv:2101.05716, transformers, Dutch, Flemish, RoBERTa, RobBERT, RobBERTje, license:mit, autonlp_compatible, infinity_compatible


tags:


- conversational


# SHAY0 Dialo GPT Model


model name: DialoGPT-small-shayo


model author: ShayoGun


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


'epoch'=3
'eval_loss': 0.2554474174976349
'loss': 0.3272
'train_loss': 0.6488333813678379
'Perplexity': 1.29


model name: ctg-ovk


model author: ardauzunoglu


tags: pytorch, tensorboard, distilbert, fill-mask, transformers, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
metrics:
- accuracy
- f1
model-index:
- name: gp-classification
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# gp-classification


This model is a fine-tuned version of [dbmdz/bert-base-turkish-cased](https://huggingface.co/dbmdz/bert-base-turkish-cased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0013
- Accuracy: 0.9997
- F1: 0.9997


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|
| 0.0215        | 1.0   | 956  | 0.0051          | 0.9987   | 0.9987 |
| 0.0033        | 2.0   | 1912 | 0.0088          | 0.9984   | 0.9985 |
| 0.001         | 3.0   | 2868 | 0.0036          | 0.9995   | 0.9995 |
| 0.0005        | 4.0   | 3824 | 0.0012          | 0.9997   | 0.9997 |
| 0.0           | 5.0   | 4780 | 0.0013          | 0.9997   | 0.9997 |


### Framework versions


- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


model name: gp-classification


model author: ardauzunoglu


task: text-classification


tags: pytorch, tensorboard, bert, text-classification, transformers, license:mit, generated_from_trainer, model-index, infinity_compatible


license: mit
tags:


- generated_from_trainer
metrics:
- accuracy
- f1
model-index:
- name: c_ovk
results: []


# c_ovk


- Loss: 0.2516
- Accuracy: 0.9249
- F1: 0.9044


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|
| 0.4038        | 1.0   | 2462 | 0.2424          | 0.9117   | 0.8848 |
| 0.2041        | 2.0   | 4924 | 0.2323          | 0.9230   | 0.9028 |
| 0.1589        | 3.0   | 7386 | 0.2516          | 0.9249   | 0.9044 |


### Framework versions


model name: c_ovk


model author: ardauzunoglu


task: text-classification


Model Card coming soon


model name: manglish-offensive-language-identification


model author: seanbenhur


tags: pytorch, onnx, bert, text-classification, transformers, infinity_compatible


## license: agpl-3.0


model name: tamil_tokenizer


task: text-generation


tags: gpt2, text-generation, transformers, license:agpl-3.0


language:


- ta
- en
tags:
- Text Classification
license: apache-2.0
datasets:
- dravidiancodemixed
metrics:
- f1
- accuracy


Model card Coming soon


model name: tanglish-offensive-language-identification


tags: pytorch, onnx, bert, text-classification, ta, en, dataset:dravidiancodemixed, transformers, Text Classification, license:apache-2.0, infinity_compatible


Model Card Coming Soon


model name: kanglish-offensive-language-identification


tags: pytorch, onnx, roberta, text-classification, transformers, infinity_compatible


- "hn"
- "bn"
- "mn"
tags:
- Text Classification


license: 	wtfpl
datasets:


- ComMA
metrics:
- F1-Score
widget:
- text: "but who in the holy hell says to relate with it,or inspired by it😂😂,i'm a 23 yr old student,and i say it's wrong,watch for entertainment purpose,and those who get inspired by such movies,its their mental problem.and all the praise that shahid's getting is for dark charachter that he portrays.and those sittis she's talking abt,don't we hear those when a villian arrives on [screen.my](http://screen.my/) point is bash sexism,whether it's by a man or a group of woman.and as far as i remember,those girls were not shown as dark characters,as kabir singh is🙂"
- text: "सही है, बोलने के अधिकार पर गाली दो, parotest के अधिकार पर पुलिश का सर फोड़ो ,मादरचोदो अधिकारो का कब सही इस्तेमाल करोगें🐷🐷🐷😠😠😠🖕"


# Automatic Identification of Gender Bias in Hindi,Bengali,Meitei Codemixed Texts


This is a XLM-Align-Base model trained on CoMMA dataset of 12k samples


- This is an extension from our previous paper: [Hypers at ComMA@ICON: Modelling Aggressiveness, Gender Bias and Communal Bias Identification](https://arxiv.org/abs/2112.15417).


## Example Usage


```python
import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
from transformers import set_seed

set_seed(425)

text = "some gender biased text"
pipe = pipeline("text-classification", model="seanbenhur/MuLTiGENBiaS")

def predict_pipe(text):
    prediction = pipe(text, return_all_scores=True)[0]
    return prediction

if __name__ == "__main__":
  target = predict_pipe(text)
  print(target)

```


### Some concerns


- Note: The model is trained on relatively lower samples (i.e 12k) but with mix of four languages Hindi, Bengali, Meitei, and English. It contains both native on codemixed scripts, So the model might perform poorly on many text samples and might not generalize well. We are continuously improving the model.


model name: MuLTiGENBiaS


model author: seanbenhur


task: text-classification


tags: pytorch, tf, onnx, xlm-roberta, text-classification, hn, bn, mn, dataset:ComMA, arxiv:2112.15417, transformers, Text Classification, license:wtfpl


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-base-uncased-finetuned-addresso
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# bert-base-uncased-finetuned-addresso


This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on an unknown dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 5
- eval_batch_size: 5
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Framework versions


- Transformers 4.12.5
- Pytorch 1.8.1
- Datasets 1.15.1
- Tokenizers 0.10.3


model name: bert-base-uncased-finetuned-addresso


model author: annafavaro


task: text-classification


tags: pytorch, tensorboard, bert, text-classification, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


language: de
datasets:


- common_voice
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Large 53
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice de
type: common_voice
args: de
metrics:
    - name: Test WER
type: wer
value: 29.35


# Wav2Vec2-Large-XLSR-53-German


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on German using 3% of the [Common Voice](https://huggingface.co/datasets/common_voice) dataset.
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("common_voice", "de", split="test[:2%]")

processor = Wav2Vec2Processor.from_pretrained("marcel/wav2vec2-large-xlsr-german-demo")
model = Wav2Vec2ForCTC.from_pretrained("marcel/wav2vec2-large-xlsr-german-demo")

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
	speech_array, sampling_rate = torchaudio.load(batch["path"])
	batch["speech"] = resampler(speech_array).squeeze().numpy()
	return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
	logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the {language} test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "de", split="test[:10%]")
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("marcel/wav2vec2-large-xlsr-german-demo")
model = Wav2Vec2ForCTC.from_pretrained("marcel/wav2vec2-large-xlsr-german-demo")
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\”\�\カ\æ\無\ན\カ\臣\ѹ\…\«\»\ð\ı\„\幺\א\ב\比\ш\ע\)\ứ\в\œ\ч\+\—\ш\‚\נ\м\ń\乡\$\=\ש\ф\支\(\°\и\к\̇]'
substitutions = {
    'e' : '[\ə\é\ě\ę\ê\ế\ế\ë\ė\е]',
    'o' : '[\ō\ô\ô\ó\ò\ø\ọ\ŏ\õ\ő\о]',
    'a' : '[\á\ā\ā\ă\ã\å\â\à\ą\а]',
    'c' : '[\č\ć\ç\с]',
    'l' : '[\ł]',
    'u' : '[\ú\ū\ứ\ů]',
    'und' : '[\&]',
    'r' : '[\ř]',
    'y' : '[\ý]',
    's' : '[\ś\š\ș\ş]',
    'i' : '[\ī\ǐ\í\ï\î\ï]',
    'z' : '[\ź\ž\ź\ż]',
    'n' : '[\ñ\ń\ņ]',
    'g' : '[\ğ]',
    'ss' : '[\ß]',
    't' : '[\ț\ť]',
    'd' : '[\ď\đ]',
    "'": '[\ʿ\་\’\`\´\ʻ\`\‘]',
    'p': '\р'
}
resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    for x in substitutions:
        batch["sentence"] = re.sub(substitutions[x], x, batch["sentence"])
        speech_array, sampling_rate = torchaudio.load(batch["path"])
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 29.35 %


## Training


The first 3% of the Common Voice `train`, `validation` datasets were used for training.


The script used for training can be found TODO


model name: wav2vec2-large-xlsr-german-demo


model author: marcel


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, de, dataset:common_voice, dataset:wer, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language: de
datasets:


- common_voice
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Large 53
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice de
type: common_voice
args: de
metrics:
    - name: Test WER
type: wer
value: 15.80


# Wav2Vec2-Large-XLSR-53-German


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on German using the [Common Voice](https://huggingface.co/datasets/common_voice) dataset.
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("common_voice", "de", split="test[:2%]")

processor = Wav2Vec2Processor.from_pretrained("marcel/wav2vec2-large-xlsr-53-german")
model = Wav2Vec2ForCTC.from_pretrained("marcel/wav2vec2-large-xlsr-53-german")

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
	speech_array, sampling_rate = torchaudio.load(batch["path"])
	batch["speech"] = resampler(speech_array).squeeze().numpy()
	return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
	logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "de", split="test")
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("marcel/wav2vec2-large-xlsr-53-german")
model = Wav2Vec2ForCTC.from_pretrained("marcel/wav2vec2-large-xlsr-53-german")
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\”\�\カ\æ\無\ན\カ\臣\ѹ\…\«\»\ð\ı\„\幺\א\ב\比\ш\ע\)\ứ\в\œ\ч\+\—\ш\‚\נ\м\ń\乡\$\=\ש\ф\支\(\°\и\к\̇]'
substitutions = {
    'e' : '[\ə\é\ě\ę\ê\ế\ế\ë\ė\е]',
    'o' : '[\ō\ô\ô\ó\ò\ø\ọ\ŏ\õ\ő\о]',
    'a' : '[\á\ā\ā\ă\ã\å\â\à\ą\а]',
    'c' : '[\č\ć\ç\с]',
    'l' : '[\ł]',
    'u' : '[\ú\ū\ứ\ů]',
    'und' : '[\&]',
    'r' : '[\ř]',
    'y' : '[\ý]',
    's' : '[\ś\š\ș\ş]',
    'i' : '[\ī\ǐ\í\ï\î\ï]',
    'z' : '[\ź\ž\ź\ż]',
    'n' : '[\ñ\ń\ņ]',
    'g' : '[\ğ]',
    'ss' : '[\ß]',
    't' : '[\ț\ť]',
    'd' : '[\ď\đ]',
    "'": '[\ʿ\་\’\`\´\ʻ\`\‘]',
    'p': '\р'
}
resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    for x in substitutions:
        batch["sentence"] = re.sub(substitutions[x], x, batch["sentence"])
        speech_array, sampling_rate = torchaudio.load(batch["path"])
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


The model can also be evaluated with in 10% chunks which needs less ressources (to be tested).


```
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
import jiwer
lang_id = "de"

processor = Wav2Vec2Processor.from_pretrained("marcel/wav2vec2-large-xlsr-53-german")
model = Wav2Vec2ForCTC.from_pretrained("marcel/wav2vec2-large-xlsr-53-german")
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\”\�\カ\æ\無\ན\カ\臣\ѹ\…\«\»\ð\ı\„\幺\א\ב\比\ш\ע\)\ứ\в\œ\ч\+\—\ш\‚\נ\м\ń\乡\$\=\ש\ф\支\(\°\и\к\̇]'
substitutions = {
    'e' : '[\ə\é\ě\ę\ê\ế\ế\ë\ė\е]',
    'o' : '[\ō\ô\ô\ó\ò\ø\ọ\ŏ\õ\ő\о]',
    'a' : '[\á\ā\ā\ă\ã\å\â\à\ą\а]',
    'c' : '[\č\ć\ç\с]',
    'l' : '[\ł]',
    'u' : '[\ú\ū\ứ\ů]',
    'und' : '[\&]',
    'r' : '[\ř]',
    'y' : '[\ý]',
    's' : '[\ś\š\ș\ş]',
    'i' : '[\ī\ǐ\í\ï\î\ï]',
    'z' : '[\ź\ž\ź\ż]',
    'n' : '[\ñ\ń\ņ]',
    'g' : '[\ğ]',
    'ss' : '[\ß]',
    't' : '[\ț\ť]',
    'd' : '[\ď\đ]',
    "'": '[\ʿ\་\’\`\´\ʻ\`\‘]',
    'p': '\р'
}
resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    for x in substitutions:
        batch["sentence"] = re.sub(substitutions[x], x, batch["sentence"])
        speech_array, sampling_rate = torchaudio.load(batch["path"])
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch

H, S, D, I = 0, 0, 0, 0
for i in range(10):
    print("test["+str(10*i)+"%:"+str(10*(i+1))+"%]")
    test_dataset = load_dataset("common_voice", "de", split="test["+str(10*i)+"%:"+str(10*(i+1))+"%]")
    test_dataset = test_dataset.map(speech_file_to_array_fn)
    result = test_dataset.map(evaluate, batched=True, batch_size=8)
    predictions = result["pred_strings"]
    targets = result["sentence"]
    chunk_metrics = jiwer.compute_measures(targets, predictions)
    H = H + chunk_metrics["hits"]
    S = S + chunk_metrics["substitutions"]
    D = D + chunk_metrics["deletions"]
    I = I + chunk_metrics["insertions"]
WER = float(S + D + I) / float(H + S + D)
print("WER: {:2f}".format(WER*100))
```


**Test Result**: 15.80 %


## Training


The first 50% of the Common Voice `train`, and 12% of the `validation` datasets were used for training (30 epochs on first 12% and 3 epochs on the remainder).


model name: wav2vec2-large-xlsr-53-german


model author: marcel


task: automatic-speech-recognition


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-base-timit-demo-colab


This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4473
- Wer: 0.3380


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.1048        | 4.0   | 500  | 0.4370          | 0.3475 |
| 0.0871        | 8.0   | 1000 | 0.4489          | 0.3405 |
| 0.0651        | 12.0  | 1500 | 0.4473          | 0.3380 |
| 0.0703        | 16.0  | 2000 | 0.4473          | 0.3380 |
| 0.0676        | 20.0  | 2500 | 0.4473          | 0.3380 |
| 0.0714        | 24.0  | 3000 | 0.4473          | 0.3380 |
| 0.0742        | 28.0  | 3500 | 0.4473          | 0.3380 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-base-timit-demo-colab


model author: rwang97


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


language: si
tags:


- SinBERTo
- Sinhala
- roberta


### Overview


SinBerto is a small language model trained on a small news corpus. SinBerto is trained on Sinhala Language which is a low resource language compared to other languages.


### Model Specifications.


model : [Roberta](https://arxiv.org/abs/1907.11692)


vocab_size=52_000,
max_position_embeddings=514,
num_attention_heads=12,
num_hidden_layers=6,
type_vocab_size=1


### How to use from the Transformers Library


from transformers import AutoTokenizer, AutoModelForMaskedLM


tokenizer = AutoTokenizer.from_pretrained("Kalindu/SinBerto")


model = AutoModelForMaskedLM.from_pretrained("Kalindu/SinBerto")


### OR Clone the model repo


git lfs install


git clone https://huggingface.co/Kalindu/SinBerto


model name: SinBerto


model author: Kalindu


task: fill-mask


tags: pytorch, roberta, fill-mask, si, arxiv:1907.11692, transformers, SinBERTo, Sinhala, autonlp_compatible, infinity_compatible


language: en
datasets:


- libri_light
- common_voice
- switchboard
- fisher
tags:
- speech
- automatic-speech-recognition
- CTC
- Attention
- wav2vec2
license: apache-2.0


# Wav2Vec2-Large-Robust - Finetuned on Switchboard (300 hours)


## Note : Model has not been initialized. If you want to use it without further finetuning, do a forward pass first to recalculate the normalized weights of the positional convolutional layer :


```ipython
 with torch.no_grad():
    model(torch.randn((1,300_000)))
```


[Facebook's Wav2Vec2](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/)


The base model pretrained on 16kHz sampled speech audio.
Speech datasets from multiple domains were used to pretrain the model:


- [Libri-Light](https://github.com/facebookresearch/libri-light): open-source audio books from the LibriVox project; clean, read-out audio data
- [CommonVoice](https://huggingface.co/datasets/common_voice): crowd-source collected audio data; read-out text snippets
- [Switchboard](https://catalog.ldc.upenn.edu/LDC97S62): telephone speech corpus; noisy telephone data
- [Fisher](https://catalog.ldc.upenn.edu/LDC2004T19): conversational telephone speech; noisy telephone data


When using the model make sure that your speech input is also sampled at 16Khz.
Check out [this blog](https://huggingface.co/blog/fine-tune-wav2vec2-english) for more information.


[Paper Robust Wav2Vec2](https://arxiv.org/abs/2104.01027)


Authors: Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, Michael Auli


**Abstract**
Self-supervised learning of speech representations has been a very active research area but most work is focused on a single domain such as read audio books for which there exist large quantities of labeled and unlabeled data. In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain. Our experiments show that using target domain data during pre-training leads to large performance improvements across a variety of setups. On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%. This has obvious practical implications since it is much easier to obtain unlabeled target domain data than labeled data. Moreover, we find that pre-training on multiple domains improves generalization performance on domains not seen during training. Code and models will be made available at this https URL.
The original model can be found under https://github.com/pytorch/fairseq/tree/master/examples/wav2vec#wav2vec-20.


# Usage


See [this notebook](https://colab.research.google.com/drive/1FjTsqbYKphl9kL-eILgUc-bl4zVThL8F?usp=sharing) for more information on how to fine-tune the model.


model name: wav2vec2-large-robust-SB300


model author: leonardvorbeck


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, en, dataset:libri_light, dataset:common_voice, dataset:switchboard, dataset:fisher, arxiv:2104.01027, transformers, speech, CTC, Attention, license:apache-2.0


language: en
datasets:


# Wav2Vec2-Large-Robust - Finetuned on Librispeech (960 hours)


model name: wav2vec2-large-robust-LS960


language: ml
datasets:


- Indic TTS Malayalam Speech Corpus
- Openslr Malayalam Speech Corpus
- SMC Malayalam Speech Corpus
- IIIT-H Indic Speech Databases
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Malayalam XLSR Wav2Vec2 Large 53
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Test split of combined dataset using all datasets mentioned above
type: custom
args: ml
metrics:
    - name: Test WER
type: wer
value: 28.43


# Wav2Vec2-Large-XLSR-53-ml


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on ml (Malayalam) using the [Indic TTS Malayalam Speech Corpus (via Kaggle)](https://www.kaggle.com/kavyamanohar/indic-tts-malayalam-speech-corpus), [Openslr Malayalam Speech Corpus](http://openslr.org/63/), [SMC Malayalam Speech Corpus](https://blog.smc.org.in/malayalam-speech-corpus/) and [IIIT-H Indic Speech Databases](http://speech.iiit.ac.in/index.php/research-svl/69.html). The notebooks used to train model are available [here](https://github.com/gauthamsuresh09/wav2vec2-large-xlsr-53-malayalam/). When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = <load-test-split-of-combined-dataset> # Details on loading this dataset in the evaluation section

processor = Wav2Vec2Processor.from_pretrained("gvs/wav2vec2-large-xlsr-malayalam")
model = Wav2Vec2ForCTC.from_pretrained("gvs/wav2vec2-large-xlsr-malayalam")

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the audio files as arrays
def speech_file_to_array_fn(batch):
  speech_array, sampling_rate = torchaudio.load(batch["path"])
  batch["speech"] = resampler(speech_array).squeeze().numpy()
  return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
  logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"])
```


## Evaluation


The model can be evaluated as follows on the test data of combined custom dataset. For more details on dataset preparation, check the notebooks mentioned at the end of this file.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
from datasets import load_dataset, load_metric
from pathlib import Path

# The custom dataset needs to be created using notebook mentioned at the end of this file
data_dir = Path('<path-to-custom-dataset>')

dataset_folders = {
    'iiit': 'iiit_mal_abi',
    'openslr': 'openslr',
    'indic-tts': 'indic-tts-ml',
    'msc-reviewed': 'msc-reviewed-speech-v1.0+20200825',
}

# Set directories for datasets
openslr_male_dir = data_dir / dataset_folders['openslr'] / 'male'
openslr_female_dir = data_dir / dataset_folders['openslr'] / 'female'
iiit_dir = data_dir / dataset_folders['iiit']
indic_tts_male_dir = data_dir / dataset_folders['indic-tts'] / 'male'
indic_tts_female_dir = data_dir / dataset_folders['indic-tts'] / 'female'
msc_reviewed_dir = data_dir / dataset_folders['msc-reviewed']

# Load the datasets
openslr_male = load_dataset("json", data_files=[f"{str(openslr_male_dir.absolute())}/sample_{i}.json" for i in range(2023)], split="train")
openslr_female = load_dataset("json", data_files=[f"{str(openslr_female_dir.absolute())}/sample_{i}.json" for i in range(2103)], split="train")
iiit = load_dataset("json", data_files=[f"{str(iiit_dir.absolute())}/sample_{i}.json" for i in range(1000)], split="train")
indic_tts_male = load_dataset("json", data_files=[f"{str(indic_tts_male_dir.absolute())}/sample_{i}.json" for i in range(5649)], split="train")
indic_tts_female = load_dataset("json", data_files=[f"{str(indic_tts_female_dir.absolute())}/sample_{i}.json" for i in range(2950)], split="train")
msc_reviewed = load_dataset("json", data_files=[f"{str(msc_reviewed_dir.absolute())}/sample_{i}.json" for i in range(1541)], split="train")

# Create test split as 20%, set random seed as well.
test_size = 0.2
random_seed=1
openslr_male_splits = openslr_male.train_test_split(test_size=test_size, seed=random_seed)
openslr_female_splits = openslr_female.train_test_split(test_size=test_size, seed=random_seed)
iiit_splits = iiit.train_test_split(test_size=test_size, seed=random_seed)
indic_tts_male_splits = indic_tts_male.train_test_split(test_size=test_size, seed=random_seed)
indic_tts_female_splits = indic_tts_female.train_test_split(test_size=test_size, seed=random_seed)
msc_reviewed_splits = msc_reviewed.train_test_split(test_size=test_size, seed=random_seed)

# Get combined test dataset
split_list = [openslr_male_splits, openslr_female_splits, indic_tts_male_splits, indic_tts_female_splits, msc_reviewed_splits, iiit_splits]
test_dataset = datasets.concatenate_datasets([split['test'] for split in split_list)

wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("gvs/wav2vec2-large-xlsr-malayalam")
model = Wav2Vec2ForCTC.from_pretrained("gvs/wav2vec2-large-xlsr-malayalam")
model.to("cuda")

resamplers = {
    48000: torchaudio.transforms.Resample(48_000, 16_000),
}

chars_to_ignore_regex = '[\\\\,\\\\?\\\\.\\\\!\\\\-\\\\;\\\\:\\\\"\\\\“\\\\%\\\\‘\\\\”\\\\�Utrnle\\\\_]'
unicode_ignore_regex = r'[\\\\u200e]'

# Preprocessing the datasets.
# We need to read the audio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"])
    batch["sentence"] = re.sub(unicode_ignore_regex, '', batch["sentence"])
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    # Resample if its not in 16kHz
    if sampling_rate != 16000:
        batch["speech"] = resamplers[sampling_rate](speech_array).squeeze().numpy()
    else:
        batch["speech"] = speech_array.squeeze().numpy()
    # If more than one dimension is present, pick first one
    if batch["speech"].ndim > 1:
        batch["speech"] = batch["speech"][0]
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the audio files as arrays
def evaluate(batch):
  inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

  with torch.no_grad():
    logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

  pred_ids = torch.argmax(logits, dim=-1)
  batch["pred_strings"] = processor.batch_decode(pred_ids)
  return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result (WER)**: 28.43 %


## Training


A combined dataset was created using [Indic TTS Malayalam Speech Corpus (via Kaggle)](https://www.kaggle.com/kavyamanohar/indic-tts-malayalam-speech-corpus), [Openslr Malayalam Speech Corpus](http://openslr.org/63/), [SMC Malayalam Speech Corpus](https://blog.smc.org.in/malayalam-speech-corpus/) and [IIIT-H Indic Speech Databases](http://speech.iiit.ac.in/index.php/research-svl/69.html). The datasets were downloaded and was converted to HF Dataset format using [this notebook](https://github.com/gauthamsuresh09/wav2vec2-large-xlsr-53-malayalam/blob/main/make_hf_dataset.ipynb)


The notebook used for training and evaluation can be found [here](https://github.com/gauthamsuresh09/wav2vec2-large-xlsr-53-malayalam/blob/main/fine-tune-xlsr-wav2vec2-on-malayalam-asr-with-transformers_v2.ipynb)


model name: wav2vec2-large-xlsr-malayalam


model author: gvs


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, ml, dataset:Indic TTS Malayalam Speech Corpus, dataset:Openslr Malayalam Speech Corpus, dataset:SMC Malayalam Speech Corpus, dataset:IIIT-H Indic Speech Databases, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


This model is a downstream optimization of [`vuiseng9/bert-base-squadv1-block-pruning-hybrid-filled-lt`](https://huggingface.co/vuiseng9/bert-base-squadv1-block-pruning-hybrid-filled-lt) using [OpenVINO/NNCF](https://github.com/openvinotoolkit/nncf). Applied optimization includes:


1. magnitude sparsification at 57.92% upon initialization so that sparsity over all linear layers of bert-base is at 90%. Parameters are ranked globally via thier absolute norm. Only linear layers of self-attention and ffnn are targeted.
1. Custom distillation with large model `bert-large-uncased-whole-word-masking-finetuned-squad`


```
  eval_exact_match = 80.4447
  eval_f1          = 87.7678
  eval_samples     =   10784
```


# Setup


```bash
# OpenVINO/NNCF
git clone https://github.com/vuiseng9/nncf && cd nncf
git checkout tld-poc
git reset --hard 1dec7afe7a4b567c059fcf287ea2c234980fded2
python setup.py develop
pip install -r examples/torch/requirements.txt

# Huggingface nn_pruning
git clone https://github.com/vuiseng9/nn_pruning && cd nn_pruning
git checkout reproduce-evaluation
git reset --hard 2d4e196d694c465e43e5fbce6c3836d0a60e1446
pip install -e ".[dev]"

# Huggingface Transformers
git clone https://github.com/vuiseng9/transformers && cd transformers
git checkout tld-poc
git reset --hard 10a1e29d84484e48fd106f58957d9ffc89dc43c5
pip install -e .
head -n 1 examples/pytorch/question-answering/requirements.txt | xargs -i pip install {}

# Additional dependencies
pip install onnx
```


# Train


```bash
git clone https://huggingface.co/vuiseng9/bert-base-squadv1-block-pruning-hybrid-filled-lt
BASE_MODEL=/path/to/cloned_repo_above #to-revise

wget https://huggingface.co/vuiseng9/bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-57.92sparse-lt/raw/main/nncf_bert_squad_sparsity.json
NNCF_CFG=/path/to/downloaded_nncf_cfg_above #to-revise

OUTROOT=/path/to/train_output_root #to-revise
WORKDIR=transformers/examples/pytorch/question-answering #to-revise
RUNID=bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-57.92sparse-lt

cd $WORKDIR

OUTDIR=$OUTROOT/$RUNID
mkdir -p $OUTDIR

export CUDA_VISIBLE_DEVICES=0
NEPOCH=5

python run_qa.py \
    --model_name_or_path vuiseng9/bert-base-squadv1-block-pruning-hybrid \
    --optimize_model_before_eval \
    --optimized_checkpoint $BASE_MODEL \
    --dataset_name squad \
    --do_eval \
    --do_train \
    --evaluation_strategy steps \
    --eval_steps 250 \
    --learning_rate 3e-5 \
    --lr_scheduler_type cosine_with_restarts \
    --warmup_ratio 0.25 \
    --cosine_cycles 1 \
    --teacher bert-large-uncased-whole-word-masking-finetuned-squad \
    --teacher_ratio 0.9 \
    --num_train_epochs $NEPOCH \
    --per_device_eval_batch_size 128 \
    --per_device_train_batch_size 16 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --save_steps 250 \
    --nncf_config $NNCF_CFG \
    --logging_steps 1 \
    --overwrite_output_dir \
    --run_name $RUNID \
    --output_dir $OUTDIR
```


# Eval


This repo must be cloned locally.


```bash
git clone https://huggingface.co/vuiseng9/bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-57.92sparse-lt
MODELROOT=/path/to/cloned_repo_above #to-revise

export CUDA_VISIBLE_DEVICES=0

OUTDIR=eval-bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-57.92sparse-lt
WORKDIR=transformers/examples/pytorch/question-answering #to-revise
cd $WORKDIR
mkdir $OUTDIR

nohup python run_qa.py  \
    --model_name_or_path vuiseng9/bert-base-squadv1-block-pruning-hybrid  \
    --dataset_name squad  \
    --optimize_model_before_eval \
    --qat_checkpoint $MODELROOT/checkpoint-20000  \
    --nncf_config $MODELROOT/nncf_bert_squad_sparsity.json  \
    --to_onnx $OUTDIR/bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-57.92sparse-lt.onnx  \
    --do_eval  \
    --per_device_eval_batch_size 128  \
    --max_seq_length 384  \
    --doc_stride 128  \
    --overwrite_output_dir \
    --output_dir $OUTDIR 2>&1 | tee $OUTDIR/run.log &
```


model name: bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-57.92sparse-lt


model author: vuiseng9


tags: pytorch, onnx, bert, transformers, infinity_compatible


This model is developed with transformers v4.9.1.


```
  m             = 0.8444
  eval_samples  =   9815

  mm            = 0.8495
  eval_samples  =   9832
```


```bash
#!/usr/bin/env bash

export CUDA_VISIBLE_DEVICES=0

OUTDIR=bert-mnli
NEPOCH=3

WORKDIR=transformers/examples/pytorch/text-classification
cd $WORKDIR

python run_glue.py \
    --model_name_or_path bert-base-uncased \
    --task_name mnli \
    --max_seq_length 128 \
    --do_train \
    --per_device_train_batch_size 32 \
    --learning_rate 2e-5 \
    --num_train_epochs $NEPOCH \
    --logging_steps 1 \
    --evaluation_strategy steps \
    --save_steps 3000 \
    --do_eval \
    --per_device_eval_batch_size 128 \
    --eval_steps 250 \
    --output_dir $OUTDIR
    --overwrite_output_dir
```


```bash
export CUDA_VISIBLE_DEVICES=0

OUTDIR=eval-bert-mnli
WORKDIR=transformers/examples/pytorch/text-classification
cd $WORKDIR

nohup python run_glue.py \
    --model_name_or_path vuiseng9/bert-mnli  \
    --task_name mnli  \
    --do_eval  \
    --per_device_eval_batch_size 128  \
    --max_seq_length 128  \
    --overwrite_output_dir \
    --output_dir $OUTDIR 2>&1 | tee $OUTDIR/run.log &
```


model name: bert-mnli


task: text-classification


tags: pytorch, bert, text-classification, transformers, infinity_compatible


1. magnitude sparsification at 60% upon initialization. Parameters are ranked globally via thier absolute norm. Only linear layers of self-attention and ffnn are targeted.
1. NNCF Quantize-Aware Training - Symmetric 8-bit for both weight and activation on all learnable layers.
1. Custom distillation with large model `bert-large-uncased-whole-word-masking-finetuned-squad`


```
  eval_exact_match = 80.3122
  eval_f1          = 87.6162
  eval_samples     =   10784
```


# Setup


# Train


```bash
git clone https://huggingface.co/vuiseng9/bert-base-squadv1-block-pruning-hybrid-filled-lt
BASE_MODEL=/path/to/cloned_repo_above #to-revise

wget https://huggingface.co/vuiseng9/bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-60.0sparse-qat-lt/raw/main/nncf_bert_squad_sparsity.json
NNCF_CFG=/path/to/downloaded_nncf_cfg_above #to-revise

OUTROOT=/path/to/train_output_root #to-revise
WORKDIR=transformers/examples/pytorch/question-answering #to-revise
RUNID=bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-60.0sparse-qat-lt

cd $WORKDIR

OUTDIR=$OUTROOT/$RUNID
mkdir -p $OUTDIR

export CUDA_VISIBLE_DEVICES=0
NEPOCH=5

python run_qa.py \
    --model_name_or_path vuiseng9/bert-base-squadv1-block-pruning-hybrid \
    --optimize_model_before_eval \
    --optimized_checkpoint $BASE_MODEL \
    --dataset_name squad \
    --do_eval \
    --do_train \
    --evaluation_strategy steps \
    --eval_steps 250 \
    --learning_rate 3e-5 \
    --lr_scheduler_type cosine_with_restarts \
    --warmup_ratio 0.25 \
    --cosine_cycles 1 \
    --teacher bert-large-uncased-whole-word-masking-finetuned-squad \
    --teacher_ratio 0.9 \
    --num_train_epochs $NEPOCH \
    --per_device_eval_batch_size 128 \
    --per_device_train_batch_size 16 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --save_steps 250 \
    --nncf_config $NNCF_CFG \
    --logging_steps 1 \
    --overwrite_output_dir \
    --run_name $RUNID \
    --output_dir $OUTDIR
```


# Eval


This repo must be cloned locally.


```bash
git clone https://huggingface.co/vuiseng9/bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-60.0sparse-qat-lt
MODELROOT=/path/to/cloned_repo_above #to-revise

export CUDA_VISIBLE_DEVICES=0

OUTDIR=eval-bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-60.0sparse-qat-lt
WORKDIR=transformers/examples/pytorch/question-answering #to-revise
cd $WORKDIR
mkdir $OUTDIR

nohup python run_qa.py  \
    --model_name_or_path vuiseng9/bert-base-squadv1-block-pruning-hybrid  \
    --dataset_name squad  \
    --optimize_model_before_eval \
    --qat_checkpoint $MODELROOT/checkpoint-22000  \
    --nncf_config $MODELROOT/nncf_bert_squad_sparsity.json  \
    --to_onnx $OUTDIR/bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-60.0sparse-qat-lt.onnx  \
    --do_eval  \
    --per_device_eval_batch_size 128  \
    --max_seq_length 384  \
    --doc_stride 128  \
    --overwrite_output_dir \
    --output_dir $OUTDIR 2>&1 | tee $OUTDIR/run.log &
```


model name: bert-base-squadv1-block-pruning-hybrid-filled-lt-nncf-60.0sparse-qat-lt


model author: vuiseng9


language: en
datasets:


- librispeech_asr
tags:
- audio
- automatic-speech-recognition
license: apache-2.0


# Wav2Vec2-Base-100h


This is a fork of [`facebook/wav2vec2-base-100h`](https://huggingface.co/facebook/wav2vec2-base-100h)


### Changes & Notes


1. Document reproducible evaluation (below) to new transformer and datasets version.
1. Use batch size of 1 to reproduce results.
1. Validated with `transformers v4.15.0`, `datasets 1.18.0`
1. You may need to manually install pypkg `librosa`, `jiwer`


## Evaluation