---
pipeline_tag: sentence-similarity
license: apache-2.0
language:
- cs
- da
- de
- en
- es
- fi
- fr
- he
- hr
- hu
- id
- it
- nl
- 'no'
- pl
- pt
- ro
- ru
- sv
- tr
- vi
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
datasets:
- clips/mfaq
widget:
  source_sentence: "<Q>How many models can I host on HuggingFace?"
  sentences:
    - "All plans come with unlimited private models and datasets."
    - "AutoNLP is an automatic way to train and deploy state-of-the-art NLP models, seamlessly integrated with the Hugging Face ecosystem."
    - "Based on how much training data and model variants are created, we send you a compute cost and payment link - as low as $10 per job."

---

# MFAQ

We present a multilingual FAQ retrieval model trained on the [MFAQ dataset](https://huggingface.co/datasets/clips/mfaq), it ranks candidate answers according to a given question.

## Installation

```
pip install sentence-transformers transformers
```

## Usage
You can use MFAQ with sentence-transformers or directly with a HuggingFace model.
In both cases, questions need to be prepended with `<Q>`, and answers with `<A>`.

#### Sentence Transformers
```python
from sentence_transformers import SentenceTransformer

question = "<Q>How many models can I host on HuggingFace?"
answer_1 = "<A>All plans come with unlimited private models and datasets."
answer_2 = "<A>AutoNLP is an automatic way to train and deploy state-of-the-art NLP models, seamlessly integrated with the Hugging Face ecosystem."
answer_3 = "<A>Based on how much training data and model variants are created, we send you a compute cost and payment link - as low as $10 per job."

model = SentenceTransformer('clips/mfaq')
embeddings = model.encode([question, answer_1, answer_3, answer_3])
print(embeddings)
```

#### HuggingFace Transformers

```python
from transformers import AutoTokenizer, AutoModel
import torch

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

question = "<Q>How many models can I host on HuggingFace?"
answer_1 = "<A>All plans come with unlimited private models and datasets."
answer_2 = "<A>AutoNLP is an automatic way to train and deploy state-of-the-art NLP models, seamlessly integrated with the Hugging Face ecosystem."
answer_3 = "<A>Based on how much training data and model variants are created, we send you a compute cost and payment link - as low as $10 per job."

tokenizer = AutoTokenizer.from_pretrained('clips/mfaq')
model = AutoModel.from_pretrained('clips/mfaq')

# Tokenize sentences
encoded_input = tokenizer([question, answer_1, answer_3, answer_3], padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, max pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
```

## Training
You can find the training script for the model [here](https://github.com/clips/mfaq).

## People
This model was developed by [Maxime De Bruyn](https://www.linkedin.com/in/maximedebruyn/), Ehsan Lotfi, Jeska Buhmann and Walter Daelemans.

## Citation information
```
@misc{debruyn2021mfaq,
      title={MFAQ: a Multilingual FAQ Dataset},
      author={Maxime De Bruyn and Ehsan Lotfi and Jeska Buhmann and Walter Daelemans},
      year={2021},
      eprint={2109.12870},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


#Slovak RoBERTA Masked Language Model

###83Mil Parameters in small model

Medium and Large models coming soon!

RoBERTA pretrained tokenizer vocab and merges included.

---

##Training params:
- **Dataset**:
  8GB Slovak Monolingual dataset including ParaCrawl (monolingual), OSCAR, and several gigs of my own findings and cleaning.
- **Preprocessing**:
  Tokenized with a pretrained ByteLevelBPETokenizer trained on the same dataset. Uncased, with s, pad, /s, unk, and mask special tokens.
- **Evaluation results**:
  - Mnoho ľudí tu<mask>
    * žije.
    * žijú.
    * je.
    * trpí.
  - Ako sa<mask>
    * máte
    * máš
    * má
    * hovorí
  - Plážová sezóna pod Zoborom patrí medzi<mask> obdobia.
    * ročné
    * najkrajšie
    * najobľúbenejšie
    * najnáročnejšie

- **Limitations**:
  The current model is fairly small, although it works very well. This model is meant to be finetuned on downstream tasks e.g. Part-of-Speech tagging, Question Answering, anything in GLUE or SUPERGLUE.

- **Credit**:
  If you use this or any of my models in research or professional work, please credit me - Christopher Brousseau in said work.


---
tags:
- conversational
---

# Rick And Morty DialoGPT Model


---
tags:
- conversational
---

# Harry Potter DialoGPT Model


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: distilbert-base-uncased-finetuned-indosquad-v2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-finetuned-indosquad-v2

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.6650

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 4

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.9015        | 1.0   | 9676  | 1.5706          |
| 1.6438        | 2.0   | 19352 | 1.5926          |
| 1.4714        | 3.0   | 29028 | 1.5253          |
| 1.3486        | 4.0   | 38704 | 1.6650          |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: albert-base-v2-finetuned-squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# albert-base-v2-finetuned-squad

This model is a fine-tuned version of [albert-base-v2](https://huggingface.co/albert-base-v2) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3222

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.1893        | 1.0   | 3052 | 0.2808          |
| 0.1209        | 2.0   | 6104 | 0.2787          |
| 0.069         | 3.0   | 9156 | 0.3222          |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: distilbert-base-uncased-finetuned-squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-finetuned-squad

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4232

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.7604        | 1.0   | 6366  | 1.5329          |
| 1.4784        | 2.0   | 12732 | 1.3930          |
| 1.3082        | 3.0   | 19098 | 1.4232          |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


---
language:
  - af
  - am
  - ar
  - as
  - az
  - be
  - bg
  - bn
  - bo
  - bs
  - ca
  - ceb
  - co
  - cs
  - cy
  - da
  - de
  - el
  - en
  - eo
  - es
  - et
  - eu
  - fa
  - fi
  - fr
  - fy
  - ga
  - gd
  - gl
  - gu
  - ha
  - haw
  - he
  - hi
  - hmn
  - hr
  - ht
  - hu
  - hy
  - id
  - ig
  - is
  - it
  - ja
  - jv
  - ka
  - kk
  - km
  - kn
  - ko
  - ku
  - ky
  - la
  - lb
  - lo
  - lt
  - lv
  - mg
  - mi
  - mk
  - ml
  - mn
  - mr
  - ms
  - mt
  - my
  - ne
  - nl
  - no
  - ny
  - or
  - pa
  - pl
  - pt
  - ro
  - ru
  - rw
  - si
  - sk
  - sl
  - sm
  - sn
  - so
  - sq
  - sr
  - st
  - su
  - sv
  - sw
  - ta
  - te
  - tg
  - th
  - tk
  - tl
  - tr
  - tt
  - ug
  - uk
  - ur
  - uz
  - vi
  - wo
  - xh
  - yi
  - yo
  - zh
  - zu
tags:
  - bert
  - sentence_embedding
  - multilingual
  - google
  - sentence-similarity
license: apache-2.0
datasets:
  - CommonCrawl
  - Wikipedia
---

# LaBSE

## Model description

Language-agnostic BERT Sentence Encoder (LaBSE) is a BERT-based model trained for sentence embedding for 109 languages. The pre-training process combines masked language modeling with translation language modeling. The model is useful for getting multilingual sentence embeddings and for bi-text retrieval.

- Model: [HuggingFace's model hub](https://huggingface.co/setu4993/LaBSE).
- Paper: [arXiv](https://arxiv.org/abs/2007.01852).
- Original model: [TensorFlow Hub](https://tfhub.dev/google/LaBSE/2).
- Blog post: [Google AI Blog](https://ai.googleblog.com/2020/08/language-agnostic-bert-sentence.html).
- Conversion from TensorFlow to PyTorch: [GitHub](https://github.com/setu4993/convert-labse-tf-pt).

This is migrated from the v2 model on the TF Hub, which uses dict-based input. The embeddings produced by both the versions of the model are [equivalent](https://github.com/setu4993/convert-labse-tf-pt/blob/ec3a019159a54ed6493181a64486c2808c01f216/tests/test_conversion.py#L31).

## Usage

Using the model:

```python
import torch
from transformers import BertModel, BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained("setu4993/LaBSE")
model = BertModel.from_pretrained("setu4993/LaBSE")
model = model.eval()

english_sentences = [
    "dog",
    "Puppies are nice.",
    "I enjoy taking long walks along the beach with my dog.",
]
english_inputs = tokenizer(english_sentences, return_tensors="pt", padding=True)

with torch.no_grad():
    english_outputs = model(**english_inputs)
```

To get the sentence embeddings, use the pooler output:

```python
english_embeddings = english_outputs.pooler_output
```

Output for other languages:

```python
italian_sentences = [
    "cane",
    "I cuccioli sono carini.",
    "Mi piace fare lunghe passeggiate lungo la spiaggia con il mio cane.",
]
japanese_sentences = ["犬", "子犬はいいです", "私は犬と一緒にビーチを散歩するのが好きです"]
italian_inputs = tokenizer(italian_sentences, return_tensors="pt", padding=True)
japanese_inputs = tokenizer(japanese_sentences, return_tensors="pt", padding=True)

with torch.no_grad():
    italian_outputs = model(**italian_inputs)
    japanese_outputs = model(**japanese_inputs)

italian_embeddings = italian_outputs.pooler_output
japanese_embeddings = japanese_outputs.pooler_output
```

For similarity between sentences, an L2-norm is recommended before calculating the similarity:

```python
import torch.nn.functional as F

def similarity(embeddings_1, embeddings_2):
    normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
    normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
    return torch.matmul(
        normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
    )

print(similarity(english_embeddings, italian_embeddings))
print(similarity(english_embeddings, japanese_embeddings))
print(similarity(italian_embeddings, japanese_embeddings))
```

## Details

Details about data, training, evaluation and performance metrics are available in the [original paper](https://arxiv.org/abs/2007.01852).

### BibTeX entry and citation info

```bibtex
@misc{feng2020languageagnostic,
      title={Language-agnostic BERT Sentence Embedding},
      author={Fangxiaoyu Feng and Yinfei Yang and Daniel Cer and Naveen Arivazhagan and Wei Wang},
      year={2020},
      eprint={2007.01852},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


---
language:
  - ar
  - de
  - en
  - es
  - fr
  - it
  - ja
  - ko
  - nl
  - pl
  - pt
  - ru
  - th
  - tr
  - zh
tags:
  - bert
  - sentence_embedding
  - multilingual
  - google
  - sentence-similarity
  - labse
license: apache-2.0
datasets:
  - CommonCrawl
  - Wikipedia
---

# LaBSE

## Model description

Smaller Language-agnostic BERT Sentence Encoder (LaBSE) is a BERT-based model distilled from the [original LaBSE model](https://huggingface.co/setu4993/LaBSE) to 15 languages (from the original 109 languages) using the techniques described in the paper ['Load What You Need: Smaller Versions of Multilingual BERT'](https://arxiv.org/abs/2010.05609) by [Ukjae Jeong](https://github.com/jeongukjae/).

- Model: [HuggingFace's model hub](https://huggingface.co/setu4993/smaller-LaBSE).
- Original model: [TensorFlow Hub](https://tfhub.dev/jeongukjae/smaller_LaBSE_15lang/1).
- Distillation source: [GitHub](https://github.com/jeongukjae/smaller-labse).
- Conversion from TensorFlow to PyTorch: [GitHub](https://github.com/setu4993/convert-labse-tf-pt).

## Usage

Using the model:

```python
import torch
from transformers import BertModel, BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained("setu4993/smaller-LaBSE")
model = BertModel.from_pretrained("setu4993/smaller-LaBSE")
model = model.eval()

english_sentences = [
    "dog",
    "Puppies are nice.",
    "I enjoy taking long walks along the beach with my dog.",
]
english_inputs = tokenizer(english_sentences, return_tensors="pt", padding=True)

with torch.no_grad():
    english_outputs = model(**english_inputs)
```

To get the sentence embeddings, use the pooler output:

```python
english_embeddings = english_outputs.pooler_output
```

Output for other languages:

```python
italian_sentences = [
    "cane",
    "I cuccioli sono carini.",
    "Mi piace fare lunghe passeggiate lungo la spiaggia con il mio cane.",
]
japanese_sentences = ["犬", "子犬はいいです", "私は犬と一緒にビーチを散歩するのが好きです"]
italian_inputs = tokenizer(italian_sentences, return_tensors="pt", padding=True)
japanese_inputs = tokenizer(japanese_sentences, return_tensors="pt", padding=True)

with torch.no_grad():
    italian_outputs = model(**italian_inputs)
    japanese_outputs = model(**japanese_inputs)

italian_embeddings = italian_outputs.pooler_output
japanese_embeddings = japanese_outputs.pooler_output
```

For similarity between sentences, an L2-norm is recommended before calculating the similarity:

```python
import torch.nn.functional as F

def similarity(embeddings_1, embeddings_2):
    normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
    normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
    return torch.matmul(
        normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
    )

print(similarity(english_embeddings, italian_embeddings))
print(similarity(english_embeddings, japanese_embeddings))
print(similarity(italian_embeddings, japanese_embeddings))
```

## Details

Details about data, training, evaluation and performance metrics are available in the [original paper](https://arxiv.org/abs/2007.01852).

### BibTeX entry and citation info

```bibtex
@misc{feng2020languageagnostic,
      title={Language-agnostic BERT Sentence Embedding},
      author={Fangxiaoyu Feng and Yinfei Yang and Daniel Cer and Naveen Arivazhagan and Wei Wang},
      year={2020},
      eprint={2007.01852},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


The original link of these models is:
https://zenodo.org/record/4604066#.YKtNrqgzZPY
which is accessible by espnet utils
The are ported to this repo for users who don't have espnet dependencies.


Modified from: https://huggingface.co/pkufool/icefall_asr_aishell_conformer_ctc
1. remove unused parts by ctc greedy search for tutorial only.


---
tags:
- conversational
---

# The Office - Pam DialoGPT Model


---
language:
  - es
tags:
  - irony
  - sarcasm
  - spanish
widget:
- text: "¡Cómo disfruto peleándome con los Transformers!"
  example_title: "Ironic"
- text: "Madrid es la capital de España"
  example_title: "Non ironic"

---

# RoBERTa base finetuned for Spanish irony detection

## Model description

Model to perform irony detection in Spanish. This is a finetuned version of the [RoBERTa-base-bne model](https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne) on the [IroSvA](https://www.autoritas.net/IroSvA2019/) corpus. Only the Spanish from Spain variant was used in the training process. It comprises 2,400 tweets labeled as ironic/non-ironic.


---
language:
- ne
license: apache-2.0
tags:
- automatic-speech-recognition
- robust-speech-event
datasets:
- openslr
model-index:
- name: wav2vec2-large-xlsr-53-nepali-test
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: openslr
      type: openslr
      args: ne
    metrics:
       - name: Test WER
         type: wer
         value: 67.83

---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-large-xlsr-53-nepali-test

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4601
- Wer: 0.6783

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 50
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 7.7821        | 1.28  | 400   | 3.2052          | 1.0003 |
| 1.9443        | 2.56  | 800   | 0.7338          | 1.0301 |
| 0.7656        | 3.84  | 1200  | 0.5165          | 0.9149 |
| 0.5543        | 5.13  | 1600  | 0.4498          | 0.8551 |
| 0.4398        | 6.41  | 2000  | 0.4472          | 0.8139 |
| 0.3911        | 7.69  | 2400  | 0.4071          | 0.7938 |
| 0.3374        | 8.97  | 2800  | 0.4210          | 0.7897 |
| 0.2847        | 10.26 | 3200  | 0.4379          | 0.7693 |
| 0.2586        | 11.54 | 3600  | 0.3983          | 0.7523 |
| 0.239         | 12.82 | 4000  | 0.4140          | 0.7537 |
| 0.2141        | 14.1  | 4400  | 0.4165          | 0.7319 |
| 0.1979        | 15.38 | 4800  | 0.4247          | 0.7402 |
| 0.1901        | 16.67 | 5200  | 0.4212          | 0.7388 |
| 0.1802        | 17.95 | 5600  | 0.4359          | 0.7247 |
| 0.1619        | 19.23 | 6000  | 0.4499          | 0.7292 |
| 0.1622        | 20.51 | 6400  | 0.4226          | 0.7260 |
| 0.1591        | 21.79 | 6800  | 0.4339          | 0.7319 |
| 0.1391        | 23.08 | 7200  | 0.4344          | 0.7167 |
| 0.1373        | 24.36 | 7600  | 0.4451          | 0.7215 |
| 0.1293        | 25.64 | 8000  | 0.4216          | 0.7195 |
| 0.1197        | 26.92 | 8400  | 0.4471          | 0.7143 |
| 0.12          | 28.2  | 8800  | 0.4718          | 0.7119 |
| 0.1128        | 29.49 | 9200  | 0.4883          | 0.7032 |
| 0.1084        | 30.77 | 9600  | 0.5145          | 0.7129 |
| 0.1018        | 32.05 | 10000 | 0.4833          | 0.7115 |
| 0.0984        | 33.33 | 10400 | 0.5101          | 0.7029 |
| 0.0903        | 34.61 | 10800 | 0.4645          | 0.6942 |
| 0.0835        | 35.9  | 11200 | 0.4666          | 0.6966 |
| 0.0828        | 37.18 | 11600 | 0.4648          | 0.6939 |
| 0.0751        | 38.46 | 12000 | 0.4773          | 0.6870 |
| 0.0711        | 39.74 | 12400 | 0.4611          | 0.6949 |
| 0.068         | 41.03 | 12800 | 0.4525          | 0.6963 |
| 0.0615        | 42.31 | 13200 | 0.4702          | 0.6883 |
| 0.0574        | 43.59 | 13600 | 0.4822          | 0.6883 |
| 0.0573        | 44.87 | 14000 | 0.4722          | 0.6894 |
| 0.0563        | 46.15 | 14400 | 0.4578          | 0.6811 |
| 0.0497        | 47.44 | 14800 | 0.4637          | 0.6797 |
| 0.0487        | 48.72 | 15200 | 0.4656          | 0.6793 |
| 0.0468        | 50.0  | 15600 | 0.4601          | 0.6783 |

### Framework versions

- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0


---
language:
- ne
license: apache-2.0
tags:
- automatic-speech-recognition
- robust-speech-event
datasets:
- openslr
model-index:
- name: wav2vec2-large-xlsr-300m-nepali
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: openslr
      type: openslr
      args: ne
    metrics:
       - name: Test WER
         type: wer
         value: 25.02
---
# wav2vec2-large-xlsr-300m-nepali

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m)

The dataset used to train are :

1. [OpenSLR-54 Corpus](https://openslr.org/54/)
2. External Data

For evaluation on publicly available datasets, can use [OpenSLR-43 corpus - Model is not trained on this data](http://www.openslr.org/43) which is also available on [HuggingFace as train-set](https://huggingface.co/datasets/openslr/viewer/SLR43/train) where it achieves **27% WER and 8.3% CER** with 5 gram language model.

Script to Evaluate the Model on OpenSLR-43 train set :

	python3 eval.py --model_id prajin/wav2vec2-large-xlsr-300m-nepali --dataset openslr --config SLR43  --split train --log_outputs

Below evaluation result is the evaluation on separated 10000 samples from total training dataset.

It achieves the following results on the evaluation set Without using Language Model :
- Loss: 0.2625
- Wer: 0.3426

With Language model ( 5 gram )
- Wer: 0.2502

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 6e-05
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.3468        | 0.04  | 400   | 0.2624          | 0.3479 |
| 0.2792        | 0.08  | 800   | 0.2696          | 0.3490 |
| 0.245         | 0.12  | 1200  | 0.2751          | 0.3502 |
| 0.2453        | 0.16  | 1600  | 0.2754          | 0.3523 |
| 0.2332        | 0.2   | 2000  | 0.2779          | 0.3517 |
| 0.2321        | 0.24  | 2400  | 0.2775          | 0.3528 |
| 0.2708        | 0.28  | 2800  | 0.2764          | 0.3533 |
| 0.2709        | 0.32  | 3200  | 0.2723          | 0.3544 |
| 0.2715        | 0.36  | 3600  | 0.2739          | 0.3545 |
| 0.2732        | 0.4   | 4000  | 0.2707          | 0.3498 |
| 0.2643        | 0.44  | 4400  | 0.2696          | 0.3499 |
| 0.2682        | 0.47  | 4800  | 0.2672          | 0.3492 |
| 0.2687        | 0.51  | 5200  | 0.2644          | 0.3474 |
| 0.269         | 0.55  | 5600  | 0.2619          | 0.3502 |
| 0.2675        | 0.59  | 6000  | 0.2606          | 0.3477 |
| 0.2656        | 0.63  | 6400  | 0.2597          | 0.3463 |
| 0.2667        | 0.67  | 6800  | 0.2607          | 0.3458 |
| 0.2639        | 0.71  | 7200  | 0.2601          | 0.3480 |
| 0.2631        | 0.75  | 7600  | 0.2582          | 0.3447 |
| 0.2589        | 0.79  | 8000  | 0.2577          | 0.3438 |
| 0.2554        | 0.83  | 8400  | 0.2557          | 0.3439 |
| 0.2687        | 0.87  | 8800  | 0.2546          | 0.3438 |
| 0.2574        | 0.91  | 9200  | 0.2537          | 0.3434 |
| 0.2623        | 0.95  | 9600  | 0.2530          | 0.3433 |
| 0.2675        | 0.99  | 10000 | 0.2530          | 0.3426 |

### Framework versions

- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0


---
license: mit
tags:
- generated_from_trainer
model-index:
- name: gpt2-wikitext2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# gpt2-wikitext2

This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 6.1112

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 6.5571        | 1.0   | 2249 | 6.4684          |
| 6.1921        | 2.0   | 4498 | 6.1984          |
| 6.0016        | 3.0   | 6747 | 6.1112          |

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: bert-base-cased-wikitext2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert-base-cased-wikitext2

This model is a fine-tuned version of [bert-base-cased](https://huggingface.co/bert-base-cased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 6.8575

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 7.0964        | 1.0   | 2346 | 7.0532          |
| 6.9055        | 2.0   | 4692 | 6.8710          |
| 6.8574        | 3.0   | 7038 | 6.8917          |

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: distilbert-base-uncased-finetuned-cola
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: glue
      type: glue
      args: cola
    metrics:
    - name: Matthews Correlation
      type: matthews_correlation
      value: 0.5543972545286807
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-finetuned-cola

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:
- Loss: 0.8273
- Matthews Correlation: 0.5544

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.5256        | 1.0   | 535  | 0.5419          | 0.4248               |
| 0.3486        | 2.0   | 1070 | 0.5187          | 0.4999               |
| 0.2406        | 3.0   | 1605 | 0.6580          | 0.5054               |
| 0.1692        | 4.0   | 2140 | 0.7455          | 0.5403               |
| 0.1343        | 5.0   | 2675 | 0.8273          | 0.5544               |

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- matthews_correlation
model-index:
- name: chinese-bert-wwm-ext-finetuned-cola
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# chinese-bert-wwm-ext-finetuned-cola

This model is a fine-tuned version of [hfl/chinese-bert-wwm-ext](https://huggingface.co/hfl/chinese-bert-wwm-ext) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5747
- Matthews Correlation: 0.4085

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:-----:|:---------------:|:--------------------:|
| 0.5824        | 1.0   | 66375 | 0.5746          | 0.4083               |
| 0.5824        | 2.0   | 66376 | 0.5747          | 0.4085               |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.7.1
- Datasets 1.15.1
- Tokenizers 0.10.3


---
language: "en"
tags:
- fill-mask

---

# ClinicalBERT - Bio + Clinical BERT Model

The [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) paper contains four unique clinicalBERT models: initialized with BERT-Base (`cased_L-12_H-768_A-12`) or BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`) & trained on either all MIMIC notes or only discharge summaries.

This model card describes the Bio+Clinical BERT model, which was initialized from [BioBERT](https://arxiv.org/abs/1901.08746) & trained on all MIMIC notes.

## Pretraining Data
The `Bio_ClinicalBERT` model was trained on all notes from [MIMIC III](https://www.nature.com/articles/sdata201635), a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see [here](https://mimic.physionet.org/). All notes from the `NOTEEVENTS` table were included (~880M words).

## Model Pretraining

### Note Preprocessing
Each note in MIMIC was first split into sections using a rules-based section splitter (e.g. discharge summary notes were split into "History of Present Illness", "Family History", "Brief Hospital Course", etc. sections). Then each section was split into sentences using SciSpacy (`en core sci md` tokenizer).

### Pretraining Procedures
The model was trained using code from [Google's BERT repository](https://github.com/google-research/bert) on a GeForce GTX TITAN X 12 GB GPU. Model parameters were initialized with BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`).

### Pretraining Hyperparameters
We used a batch size of 32, a maximum sequence length of 128, and a learning rate of 5 · 10−5 for pre-training our models. The models trained on all MIMIC notes  were trained for 150,000 steps. The dup factor for duplicating input data with different masks was set to 5. All other default parameters were used (specifically, masked language model probability = 0.15
and max predictions per sequence = 20).

## How to use the model

Load the model via the transformers library:
```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
```

## More Information

Refer to the original paper, [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) (NAACL Clinical NLP Workshop 2019) for additional details and performance on NLI and NER tasks.

## Questions?

Post a Github issue on the [clinicalBERT repo](https://github.com/EmilyAlsentzer/clinicalBERT) or email emilya@mit.edu with any questions.


---
language: "en"
tags:
- fill-mask
---

# ClinicalBERT - Bio + Discharge Summary BERT Model

The [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) paper contains four unique clinicalBERT models: initialized with BERT-Base (`cased_L-12_H-768_A-12`) or BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`) & trained on either all MIMIC notes or only discharge summaries.

This model card describes the Bio+Discharge Summary BERT model, which was initialized from [BioBERT](https://arxiv.org/abs/1901.08746) & trained on only discharge summaries from MIMIC.

## Pretraining Data
The `Bio_Discharge_Summary_BERT` model was trained on all discharge summaries from [MIMIC III](https://www.nature.com/articles/sdata201635), a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see [here](https://mimic.physionet.org/). All notes from the `NOTEEVENTS` table were included (~880M words).

## Model Pretraining

### Note Preprocessing
Each note in MIMIC was first split into sections using a rules-based section splitter (e.g. discharge summary notes were split into "History of Present Illness", "Family History", "Brief Hospital Course", etc. sections). Then each section was split into sentences using SciSpacy (`en core sci md` tokenizer).

### Pretraining Procedures
The model was trained using code from [Google's BERT repository](https://github.com/google-research/bert) on a GeForce GTX TITAN X 12 GB GPU. Model parameters were initialized with BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`).

### Pretraining Hyperparameters
We used a batch size of 32, a maximum sequence length of 128, and a learning rate of 5 · 10−5 for pre-training our models. The models trained on all MIMIC notes  were trained for 150,000 steps. The dup factor for duplicating input data with different masks was set to 5. All other default parameters were used (specifically, masked language model probability = 0.15
and max predictions per sequence = 20).

## How to use the model

Load the model via the transformers library:
```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_Discharge_Summary_BERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_Discharge_Summary_BERT")
```

## More Information

Refer to the original paper, [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) (NAACL Clinical NLP Workshop 2019) for additional details and performance on NLI and NER tasks.

## Questions?

Post a Github issue on the [clinicalBERT repo](https://github.com/EmilyAlsentzer/clinicalBERT) or email emilya@mit.edu with any questions.


---
tags:
- image-classification
library_name: generic
---

# Dog vs Cat Image Classification with FastAI CNN

Training is based in FastAI [Quick Start](https://docs.fast.ai/quick_start.html). Example training

## Training

The model was trained as follows

```python
path = untar_data(URLs.PETS)/'images'

def is_cat(x): return x[0].isupper()
dls = ImageDataLoaders.from_name_func(
    path, get_image_files(path), valid_pct=0.2, seed=42,
    label_func=is_cat, item_tfms=Resize(224))

learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(1)
```


---
pipeline_tag: sentence-similarity
language: multilingual
license: apache-2.0
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
---

# sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.

## Usage (Sentence-Transformers)

Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:

```
pip install -U sentence-transformers
```

Then you can use the model like this:

```python
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
embeddings = model.encode(sentences)
print(embeddings)
```

## Usage (HuggingFace Transformers)
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.

```python
from transformers import AutoTokenizer, AutoModel
import torch

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
model = AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, max pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)
```

## Evaluation Results

For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2)

## Full Model Architecture
```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)
```

## Citing & Authors

This model was trained by [sentence-transformers](https://www.sbert.net/).

If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "http://arxiv.org/abs/1908.10084",
}
```


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model_index:
- name: distilbert-base-uncased-finetuned-cola
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: glue
      type: glue
      args: cola
    metric:
      name: Matthews Correlation
      type: matthews_correlation
      value: 0.42176824452830747
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-finetuned-cola

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4981
- Matthews Correlation: 0.4218

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.5248        | 1.0   | 535  | 0.4981          | 0.4218               |

### Framework versions

- Transformers 4.9.1
- Pytorch 1.9.0+cu102
- Datasets 1.10.2
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- glue
metrics:
- accuracy
model_index:
- name: distilbert-base-uncased-finetuned-sst2
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: glue
      type: glue
      args: sst2
    metric:
      name: Accuracy
      type: accuracy
      value: 0.9151376146788991
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-finetuned-sst2

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3651
- Accuracy: 0.9151

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Accuracy |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|
| 0.1902        | 1.0   | 4210  | 0.3102          | 0.9117   |
| 0.1293        | 2.0   | 8420  | 0.3672          | 0.9048   |
| 0.084         | 3.0   | 12630 | 0.3651          | 0.9151   |
| 0.0682        | 4.0   | 16840 | 0.3971          | 0.9037   |
| 0.0438        | 5.0   | 21050 | 0.4720          | 0.9117   |

### Framework versions

- Transformers 4.9.1
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


A checkpoint for training Persian T5 model. This repository can be cloned and pre-training can be resumed. This model uses flax and is for training.

For more information and getting the training code please refer to:

https://github.com/puraminy/parsT5


A monolingual T5 model for Persian trained on OSCAR 21.09 (https://oscar-corpus.com/) corpus with self-supervised method. 35 Gig deduplicated version of Persian data was used for pre-training the model.

It's similar to the English T5 model but just for Persian. You may need to fine-tune it on your specific task.

Example code:

```
from transformers import T5ForConditionalGeneration,AutoTokenizer

import torch

model_name = "Ahmad/parsT5-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)

input_ids = tokenizer.encode('دانش آموزان به <extra_id_0> میروند و <extra_id_1> میخوانند.', return_tensors='pt')
with torch.no_grad():
    hypotheses = model.generate(input_ids)
for h in hypotheses:
    print(tokenizer.decode(h))
```

Steps: 725000

Accuracy: 0.66

Training More?
========

To train the model further please refer to its github repository at:

https://github.com/puraminy/parsT5


Just for performing some experiments. Do not use.


---
license: apache-2.0
tags:
- generated_from_trainer
- automatic-speech-recognition
- NbAiLab/NPSC
- robust-speech-event
- no
- nb-NO
datasets:
- NbAiLab/NPSC
language:
- nb-NO
model-index:
- name: XLSR-300M-bokmaal
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
      args: 16K_mp3_bokmaal
    metrics:
       - name: Test (Bokmål) WER
         type: wer
         value: 0.07699635320946434
       - name: Test (Bokmål) CER
         type: cer
         value: 0.0284288464829
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# XLSR-300M-bokmaal

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the NBAILAB/NPSC - 16K_MP3_BOKMAAL dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1635
- Wer: 0.1005

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 15.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 3.0307        | 0.32  | 500   | 3.0026          | 1.0    |
| 2.7865        | 0.64  | 1000  | 2.4849          | 0.9926 |
| 0.7522        | 0.95  | 1500  | 0.4567          | 0.3594 |
| 0.5703        | 1.27  | 2000  | 0.3440          | 0.2586 |
| 0.4762        | 1.59  | 2500  | 0.2925          | 0.2178 |
| 0.4585        | 1.91  | 3000  | 0.2442          | 0.1981 |
| 0.4013        | 2.23  | 3500  | 0.2495          | 0.1818 |
| 0.449         | 2.54  | 4000  | 0.2152          | 0.1808 |
| 0.355         | 2.86  | 4500  | 0.2179          | 0.1670 |
| 0.3142        | 3.18  | 5000  | 0.1953          | 0.1542 |
| 0.3242        | 3.5   | 5500  | 0.2103          | 0.1526 |
| 0.3016        | 3.82  | 6000  | 0.1911          | 0.1477 |
| 0.2713        | 4.13  | 6500  | 0.1836          | 0.1422 |
| 0.2807        | 4.45  | 7000  | 0.1924          | 0.1447 |
| 0.2929        | 4.77  | 7500  | 0.1848          | 0.1402 |
| 0.2595        | 5.09  | 8000  | 0.1783          | 0.1330 |
| 0.2289        | 5.41  | 8500  | 0.1901          | 0.1313 |
| 0.2567        | 5.72  | 9000  | 0.1784          | 0.1298 |
| 0.2401        | 6.04  | 9500  | 0.1956          | 0.1298 |
| 0.2098        | 6.36  | 10000 | 0.1748          | 0.1277 |
| 0.2246        | 6.68  | 10500 | 0.1777          | 0.1254 |
| 0.2197        | 7.0   | 11000 | 0.1703          | 0.1222 |
| 0.2122        | 7.32  | 11500 | 0.1917          | 0.1221 |
| 0.2746        | 7.63  | 12000 | 0.1769          | 0.1215 |
| 0.2148        | 7.95  | 12500 | 0.1736          | 0.1193 |
| 0.1915        | 8.27  | 13000 | 0.1814          | 0.1161 |
| 0.2462        | 8.59  | 13500 | 0.1748          | 0.1166 |
| 0.1872        | 8.91  | 14000 | 0.1769          | 0.1133 |
| 0.1886        | 9.22  | 14500 | 0.1852          | 0.1143 |
| 0.1789        | 9.54  | 15000 | 0.1696          | 0.1126 |
| 0.1692        | 9.86  | 15500 | 0.1817          | 0.1122 |
| 0.1765        | 10.18 | 16000 | 0.1769          | 0.1093 |
| 0.1699        | 10.5  | 16500 | 0.1604          | 0.1084 |
| 0.1591        | 10.81 | 17000 | 0.1777          | 0.1080 |
| 0.1499        | 11.13 | 17500 | 0.1645          | 0.1074 |
| 0.163         | 11.45 | 18000 | 0.1704          | 0.1065 |
| 0.1597        | 11.77 | 18500 | 0.1576          | 0.1064 |
| 0.1484        | 12.09 | 19000 | 0.1637          | 0.1041 |
| 0.1464        | 12.4  | 19500 | 0.1631          | 0.1047 |
| 0.156         | 12.72 | 20000 | 0.1686          | 0.1029 |
| 0.1625        | 13.04 | 20500 | 0.1648          | 0.1023 |
| 0.1395        | 13.36 | 21000 | 0.1688          | 0.1027 |
| 0.1387        | 13.68 | 21500 | 0.1670          | 0.1013 |
| 0.1434        | 13.99 | 22000 | 0.1677          | 0.1017 |
| 0.1442        | 14.31 | 22500 | 0.1688          | 0.1008 |
| 0.1439        | 14.63 | 23000 | 0.1647          | 0.1004 |
| 0.137         | 14.95 | 23500 | 0.1636          | 0.1006 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


Just for performing some experiments. Do not use.

This needed to be restarted at 100k. I am getting memory errors at the end of the epoch. Not really sure why.

Step 2 is therefore on train_2__4. Static learning rate for a while. The first 100k ended at 0.59. This is decent so early. No point in running more epochs here though. Changing the corpus and continue training.


---
license: apache-2.0
tags:
- automatic-speech-recognition
- NbAiLab/NPSC
- generated_from_trainer
datasets:
- npsc
model-index:
- name: xls-npsc
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xls-npsc

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the NBAILAB/NPSC - 48K_MP3 dataset.
It achieves the following results on the evaluation set:
- Loss: 3.5006
- Wer: 1.0

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 7.5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 10.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.18.1.dev0
- Tokenizers 0.10.3


---
license: cc0-1.0
tags:
- automatic-speech-recognition
- NbAiLab/NPSC
- generated_from_trainer
- robust-speech-event
model-index:
- name: wav2vec2-large-voxrex-npsc
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-large-voxrex-npsc

This model is a fine-tuned version of [KBLab/wav2vec2-large-voxrex](https://huggingface.co/KBLab/wav2vec2-large-voxrex) on the NBAILAB/NPSC - 16K_MP3 dataset.
It achieves the following results on the evaluation set:
- Loss: nan
- Wer: 1.0

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 15.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 2.9728        | 0.32  | 500   | 2.9449          | 1.0    |
| 2.5099        | 0.64  | 1000  | 1.8492          | 0.9910 |
| 0.7872        | 0.97  | 1500  | 0.4467          | 0.3774 |
| 0.5993        | 1.29  | 2000  | 0.3181          | 0.2819 |
| 0.5134        | 1.61  | 2500  | 0.2638          | 0.2401 |
| 0.4544        | 1.93  | 3000  | 0.2287          | 0.2091 |
| 0.4085        | 2.26  | 3500  | 0.2153          | 0.1918 |
| 0.3921        | 2.58  | 4000  | 0.2004          | 0.1804 |
| 0.4613        | 2.9   | 4500  | 0.1905          | 0.1732 |
| 0.3402        | 3.22  | 5000  | 0.1778          | 0.1659 |
| 0.3258        | 3.55  | 5500  | 0.1732          | 0.1571 |
| 0.3044        | 3.87  | 6000  | 0.1677          | 0.1497 |
| 0.2914        | 4.19  | 6500  | 0.1597          | 0.1420 |
| 0.278         | 4.51  | 7000  | 0.1574          | 0.1386 |
| 0.2858        | 4.84  | 7500  | 0.1552          | 0.1300 |
| 0.2585        | 5.16  | 8000  | 0.1523          | 0.1276 |
| 0.2827        | 5.48  | 8500  | 0.1448          | 0.1265 |
| 0.3365        | 5.8   | 9000  | 0.1411          | 0.1232 |
| 0.2488        | 6.13  | 9500  | 0.1456          | 0.1195 |
| 0.2406        | 6.45  | 10000 | 0.1414          | 0.1194 |
| 0.2488        | 6.77  | 10500 | 0.1393          | 0.1173 |
| 0.3084        | 7.09  | 11000 | 0.1379          | 0.1164 |
| 0.2365        | 7.41  | 11500 | 0.1387          | 0.1165 |
| 0.2217        | 7.74  | 12000 | 0.1381          | 0.1132 |
| 0.2381        | 8.06  | 12500 | 0.1360          | 0.1126 |
| 0.2329        | 8.38  | 13000 | 0.1357          | 0.1124 |
| 0.2103        | 8.7   | 13500 | 0.1335          | 0.1087 |
| 0.2366        | 9.03  | 14000 | 0.1388          | 0.1105 |
| 0.2289        | 9.35  | 14500 | 0.1383          | 0.1098 |
| 0.2486        | 9.67  | 15000 | 0.1386          | 0.1087 |
| **0.2772**        | **9.99**  | **15500** | **0.1598**          | **0.1093** |
| 0.2728        | 10.32 | 16000 | 0.1814          | 0.1110 |
| 0.3437        | 10.64 | 16500 | 0.2505          | 0.1124 |
| 0.431         | 10.96 | 17000 | 0.2828          | 0.1143 |
| 0.3929        | 11.28 | 17500 | 0.2977          | 0.1149 |
| 0.4396        | 11.61 | 18000 | 0.3198          | 0.1170 |
| 0.59          | 11.93 | 18500 | 0.4158          | 0.1315 |
| 0.7813        | 12.25 | 19000 | 0.6123          | 0.2208 |
| 0.9345        | 12.57 | 19500 | 0.6815          | 0.2885 |
| 0.998         | 12.89 | 20000 | 0.7587          | 0.1991 |
| 1.0493        | 13.22 | 20500 | 0.7583          | 0.1996 |
| 1.438         | 13.54 | 21000 | nan             | 1.0    |
| 0.0           | 13.86 | 21500 | nan             | 1.0    |
| 0.0           | 14.18 | 22000 | nan             | 1.0    |
| 0.0           | 14.51 | 22500 | nan             | 1.0    |
| 0.0           | 14.83 | 23000 | nan             | 1.0    |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu113
- Datasets 1.18.3.dev0
- Tokenizers 0.11.0


---
language: no
license: cc-by-4.0
tags:
- norwegian
- roberta
pipeline_tag: fill-mask
widget:
- text: På biblioteket kan du <mask> en bok.
- text: Dette er et <mask> eksempel.
- text: Av og til kan en språkmodell gi et <mask> resultat.
- text: Som ansat får du <mask> for at bidrage til borgernes adgang til dansk kulturarv, til forskning og til samfundets demokratiske udvikling.
---

# This is just an internal test used for debugging.

Please do not use this model for anything. Visit the local repo instead.

# Nordic Roberta Wikipedia
## Description
Nord roberta model trainined on the swedish danish and norwegian wikipedia.

## Evaluation
Evaluation on Named Entity recognition in Danish.

I finetuned each model on 3 epochs on DaNE, repeated it 5 times for each model, and calculated 95% confidence intervals for the means. Here are the results:

xlm-roberta-base : 88.01 +- 0.43

flax-community/nordic-roberta-wiki: 85.75 +- 0.69 (this model)

Maltehb/danish-bert-botxo: 85.38 +- 0.55

flax-community/roberta-base-danish:  80.14 +- 1.47

flax-community/roberta-base-scandinavian : 78.03 +- 3.02

Maltehb/-l-ctra-danish-electra-small-cased: 57.87 +- 3.19

NbAiLab/nb-bert-base : 30.24 +- 1.21

Randomly initialised RoBERTa model: 19.79 +- 2.00

Evaluation on Sentiment analysis in Dansish
Here are the results on test set, where each model has been trained 5 times, and the “+-” refers to a 95% confidence interval of the mean score:

Maltehb/danish-bert-botxo: 65.19 +- 0.53

NbAiLab/nb-bert-base : 63.80 +- 0.77

xlm-roberta-base : 63.55 +- 1.59

flax-community/nordic-roberta-wiki : 56.46 +- 1.77

flax-community/roberta-base-danish : 54.73 +- 8.96

flax-community/roberta-base-scandinavian : 44.28 +- 9.21

Maltehb/-l-ctra-danish-electra-small-cased : 47.78 +- 12.65

Randomly initialised RoBERTa model: 36.96 +- 1.02

Maltehb/roberta-base-scandinavian : 33.65 +- 8.32

## Model series
This model is part of a series of models training on TPU with Flax Jax during Huggingface Flax/Jax challenge.

## Gpt models

## Swedish Gpt
https://huggingface.co/birgermoell/swedish-gpt/

## Swedish gpt wiki
https://huggingface.co/flax-community/swe-gpt-wiki

# Nordic gpt wiki
https://huggingface.co/flax-community/nordic-gpt-wiki

## Dansk gpt wiki
https://huggingface.co/flax-community/dansk-gpt-wiki

## Norsk gpt wiki
https://huggingface.co/flax-community/norsk-gpt-wiki

## Roberta models

## Nordic Roberta Wiki
https://huggingface.co/flax-community/nordic-roberta-wiki

## Swe Roberta Wiki Oscar
https://huggingface.co/flax-community/swe-roberta-wiki-oscar

## Roberta Swedish Scandi
https://huggingface.co/birgermoell/roberta-swedish-scandi

## Roberta Swedish
https://huggingface.co/birgermoell/roberta-swedish

## Swedish T5 model
https://huggingface.co/birgermoell/t5-base-swedish


---
language: no
license: cc-by-4.0
tags:
- seq2seq
datasets:
- Norwegian Nynorsk/Bokmål
---
# 🇳🇴 Norwegian T5 Base model Trained on the NCC🇳🇴

This is a Norwegian T5-base model trained on the Norwegian Colossal Corpus (NCC) on a TPU v3-8. It needs to be finetuned on a specific task before being used for anything.

Currently the model is training. It is expected that it should be finished by the end of August 2021.

 The following setting were used in training:
```bash
./run_t5_mlm_flax_streaming.py \
    --output_dir="./" \
    --model_type="t5" \
    --config_name="./" \
    --tokenizer_name="./" \
    --dataset_name="pere/norwegian_colossal_corpus_v2_short100k" \
    --max_seq_length="512" \
    --weight_decay="0.01" \
    --per_device_train_batch_size="32" \
    --per_device_eval_batch_size="32" \
    --learning_rate="8e-3" \
    --warmup_steps="5000" \
    --overwrite_output_dir \
    --cache_dir /mnt/disks/flaxdisk/cache/ \
    --num_train_epochs="5" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --logging_steps="500" \
    --num_train_steps="1000000" \
    --num_eval_samples="5000" \
    --save_steps="5000" \
    --eval_steps="5000" \
    --preprocessing_num_workers 96 \
    --adafactor \
    --push_to_hub
 ```


---
language: no
license: cc-by-4.0
tags:
- norwegian
- roberta
pipeline_tag: fill-mask
widget:
- text: På biblioteket kan du <mask> en bok.
- text: Dette er et <mask> eksempel.
- text: Av og til kan en språkmodell gi et <mask> resultat.
- text: Som ansat får du <mask> for at bidrage til borgernes adgang til dansk kulturarv, til forskning og til samfundets demokratiske udvikling.
---

norwegian-roberta-base but with higher learning rate and batch size


---
license: apache-2.0
tags:
- automatic-speech-recognition
- NbAiLab/NPSC
- robust-speech-event
- no
- nb-NO
datasets:
- NbAiLab/NPSC
language:
- nb-NO
model-index:
- name: wav2vec2-xls-r-1b-npsc-bokmaal-low-27k
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
      args: 16K_mp3_bokmaal
    metrics:
       - name: Test (Bokmål) WER
         type: wer
         value: 0.06659482136499659
       - name: Test (Bokmål) CER
         type: cer
         value: 0.025337273801039607
---


---
license: cc0-1.0
tags:
- automatic-speech-recognition
- NbAiLab/NPSC
- generated_from_trainer
datasets:
- npsc
model-index:
- name: xls-npsc-oh
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xls-npsc-oh

This model is a fine-tuned version of [KBLab/wav2vec2-large-voxrex](https://huggingface.co/KBLab/wav2vec2-large-voxrex) on the NBAILAB/NPSC - 48K_MP3 dataset.
It achieves the following results on the evaluation set:
- Loss: 0.2106
- Wer: 0.8586

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 5.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 2.1093        | 2.61  | 1000 | 0.2572          | 0.9293 |

### Framework versions

- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.18.1.dev0
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_trainer
- automatic-speech-recognition
- NbAiLab/NPSC
- robust-speech-event
- no
- nb-NO
datasets:
- NbAiLab/NPSC
language:
- nb-NO
model-index:
- name: wav2vec2-xls-r-300m-npsc-bokmaal
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
      args: 16K_mp3_bokmaal
    metrics:
       - name: Test (Bokmål) WER
         type: wer
         value: 0.07556265455560153
       - name: Test (Bokmål) CER
         type: cer
         value: 0.028191288775481386
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xls-r-300m-npsc-bokmaal

This model was trained from scratch on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1663
- Wer: 0.0932

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 15.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.0969        | 0.32  | 500   | 0.1773          | 0.1054 |
| 0.0929        | 0.64  | 1000  | 0.1672          | 0.1061 |
| 0.1018        | 0.97  | 1500  | 0.1770          | 0.1067 |
| 0.0871        | 1.29  | 2000  | 0.1832          | 0.1087 |
| 0.0908        | 1.61  | 2500  | 0.1830          | 0.1101 |
| 0.0975        | 1.93  | 3000  | 0.1848          | 0.1100 |
| 0.0936        | 2.26  | 3500  | 0.1853          | 0.1113 |
| 0.1025        | 2.58  | 4000  | 0.1958          | 0.1149 |
| 0.0989        | 2.9   | 4500  | 0.1776          | 0.1123 |
| 0.0946        | 3.22  | 5000  | 0.1825          | 0.1097 |
| 0.0859        | 3.55  | 5500  | 0.1864          | 0.1072 |
| 0.0867        | 3.87  | 6000  | 0.1886          | 0.1081 |
| 0.0783        | 4.19  | 6500  | 0.1883          | 0.1063 |
| 0.0804        | 4.51  | 7000  | 0.1831          | 0.1063 |
| 0.0797        | 4.84  | 7500  | 0.1884          | 0.1058 |
| 0.0705        | 5.16  | 8000  | 0.1802          | 0.1057 |
| 0.0795        | 5.48  | 8500  | 0.1854          | 0.1038 |
| 0.0711        | 5.8   | 9000  | 0.1766          | 0.1032 |
| 0.0973        | 6.13  | 9500  | 0.1663          | 0.1014 |
| 0.087         | 6.45  | 10000 | 0.1664          | 0.1014 |
| 0.0962        | 6.77  | 10500 | 0.1631          | 0.1009 |
| 0.0857        | 7.09  | 11000 | 0.1659          | 0.1002 |
| 0.0882        | 7.41  | 11500 | 0.1668          | 0.1007 |
| 0.0784        | 7.74  | 12000 | 0.1688          | 0.0996 |
| 0.0838        | 8.06  | 12500 | 0.1675          | 0.0984 |
| 0.0863        | 8.38  | 13000 | 0.1639          | 0.0979 |
| 0.0763        | 8.7   | 13500 | 0.1638          | 0.0980 |
| 0.0822        | 9.03  | 14000 | 0.1709          | 0.0972 |
| 0.0769        | 9.35  | 14500 | 0.1700          | 0.0965 |
| 0.0838        | 9.67  | 15000 | 0.1703          | 0.0974 |
| 0.0799        | 9.99  | 15500 | 0.1667          | 0.0957 |
| 0.0712        | 10.32 | 16000 | 0.1754          | 0.0960 |
| 0.0737        | 10.64 | 16500 | 0.1725          | 0.0968 |
| 0.0851        | 10.96 | 17000 | 0.1733          | 0.0958 |
| 0.076         | 11.28 | 17500 | 0.1682          | 0.0954 |
| 0.0712        | 11.61 | 18000 | 0.1713          | 0.0943 |
| 0.0745        | 11.93 | 18500 | 0.1662          | 0.0951 |
| 0.0864        | 12.25 | 19000 | 0.1692          | 0.0947 |
| 0.0937        | 12.57 | 19500 | 0.1624          | 0.0943 |
| 0.0915        | 12.89 | 20000 | 0.1678          | 0.0942 |
| 0.0926        | 13.22 | 20500 | 0.1641          | 0.0945 |
| 0.0912        | 13.54 | 21000 | 0.1665          | 0.0937 |
| 0.0917        | 13.86 | 21500 | 0.1648          | 0.0936 |
| 0.094         | 14.18 | 22000 | 0.1635          | 0.0935 |
| 0.0864        | 14.51 | 22500 | 0.1678          | 0.0934 |
| 0.0899        | 14.83 | 23000 | 0.1663          | 0.0932 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu113
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_trainer
- automatic-speech-recognition
- NbAiLab/NPSC
- robust-speech-event
- no
- nn-NO
datasets:
- NbAiLab/NPSC
language:
- nn-NO
model-index:
- name: wav2vec2-large-voxrex-npsc-nynorsk
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
      args: 16K_mp3_nynorsk
    metrics:
       - name: Test (Nynorsk) WER
         type: wer
         value: 0.12220762155059132
       - name: Test (Nynorsk) CER
         type: cer
         value: 0.04195612578778549
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-large-voxrex-npsc-nynorsk

This model is a fine-tuned version of [KBLab/wav2vec2-large-voxrex](https://huggingface.co/KBLab/wav2vec2-large-voxrex) on the NBAILAB/NPSC - 16K_MP3_NYNORSK dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4142
- Wer: 0.1576

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 7.5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 40.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.086         | 2.17  | 500  | 3.0773          | 1.0    |
| 2.8532        | 4.35  | 1000 | 2.8393          | 1.0    |
| 0.9738        | 6.52  | 1500 | 0.7283          | 0.4890 |
| 0.6763        | 8.7   | 2000 | 0.5340          | 0.3662 |
| 0.5303        | 10.87 | 2500 | 0.4521          | 0.3140 |
| 0.4765        | 13.04 | 3000 | 0.4181          | 0.2853 |
| 0.4219        | 15.22 | 3500 | 0.4156          | 0.2934 |
| 0.3564        | 17.39 | 4000 | 0.3925          | 0.2509 |
| 0.3282        | 19.57 | 4500 | 0.3824          | 0.2420 |
| 0.3118        | 21.74 | 5000 | 0.3636          | 0.2354 |
| 0.2919        | 23.91 | 5500 | 0.3615          | 0.2281 |
| 0.2961        | 26.09 | 6000 | 0.3548          | 0.2255 |
| 0.284         | 28.26 | 6500 | 0.3526          | 0.2209 |
| 0.2566        | 30.43 | 7000 | 0.3526          | 0.2205 |
| 0.2422        | 32.61 | 7500 | 0.3569          | 0.2173 |
| 0.2472        | 34.78 | 8000 | 0.3592          | 0.2166 |
| 0.2337        | 36.96 | 8500 | 0.3625          | 0.2172 |
| 0.2315        | 39.13 | 9000 | 0.3580          | 0.2155 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.0+cu113
- Datasets 1.18.3
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
- automatic-speech-recognition
- NbAiLab/NPSC
- robust-speech-event
- no
- nn-NO
datasets:
- NbAiLab/NPSC
language:
- nn-NO
model-index:
- name: XLSR-300M-nynorsk
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
      args: 16K_mp3_nynorsk
    metrics:
       - name: Test (Nynorsk) WER
         type: wer
         value: 0.12136286840623241
       - name: Test (Nynorsk) CER
         type: cer
         value: 0.041988362534453025
---


---
license: apache-2.0
tags:
- automatic-speech-recognition
- NbAiLab/NPSC
- generated_from_trainer
model-index:
- name: wav2vec2-xlsr-300M-NPSC-OH
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xlsr-300M-NPSC-OH

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the NBAILAB/NPSC - 16K_MP3 dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1692
- Wer: 0.1663

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 7.5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 13
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 15.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 3.1638        | 0.66  | 500   | 3.0686          | 1.0    |
| 2.9311        | 1.31  | 1000  | 2.9208          | 1.0    |
| 2.4175        | 1.97  | 1500  | 1.5009          | 0.9049 |
| 1.4442        | 2.63  | 2000  | 0.4426          | 0.3783 |
| 1.2624        | 3.28  | 2500  | 0.3193          | 0.2998 |
| 1.1889        | 3.94  | 3000  | 0.2867          | 0.2630 |
| 1.1315        | 4.6   | 3500  | 0.2566          | 0.2444 |
| 1.0864        | 5.26  | 4000  | 0.2368          | 0.2294 |
| 1.093         | 5.91  | 4500  | 0.2240          | 0.2151 |
| 1.0368        | 6.57  | 5000  | 0.2117          | 0.2056 |
| 1.0178        | 7.23  | 5500  | 0.2020          | 0.1954 |
| 1.0035        | 7.88  | 6000  | 0.2005          | 0.1924 |
| 0.9759        | 8.54  | 6500  | 0.1971          | 0.1863 |
| 0.9795        | 9.2   | 7000  | 0.1892          | 0.1812 |
| 0.9601        | 9.85  | 7500  | 0.1863          | 0.1795 |
| 0.9673        | 10.51 | 8000  | 0.1809          | 0.1761 |
| 0.9233        | 11.17 | 8500  | 0.1818          | 0.1755 |
| 0.9382        | 11.83 | 9000  | 0.1767          | 0.1741 |
| 0.9242        | 12.48 | 9500  | 0.1743          | 0.1703 |
| 0.9703        | 13.14 | 10000 | 0.1711          | 0.1711 |
| 0.9139        | 13.8  | 10500 | 0.1718          | 0.1672 |
| 0.9073        | 14.45 | 11000 | 0.1700          | 0.1665 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
language: no
license: cc-by-4.0
thumbnail: https://raw.githubusercontent.com/NBAiLab/notram/master/images/nblogo_2.png
pipeline_tag: zero-shot-classification
tags:
- nb-bert
- zero-shot-classification
- pytorch
- tensorflow
- norwegian
- bert
datasets:
- mnli
- multi_nli
- xnli
widget:
- example_title: Nyhetsartikkel om FHI
  text: Folkehelseinstituttets mest optimistiske anslag er at alle voksne er ferdigvaksinert innen midten av september.
  candidate_labels: helse, politikk, sport, religion
---

**Release 1.0** (March 11, 2021)

# NB-Bert base model finetuned on Norwegian machine translated MNLI

## Description
The most effective way of creating a good classifier is to finetune a pre-trained model for the specific task at hand. However, in many cases this is simply impossible.
[Yin et al.](https://arxiv.org/abs/1909.00161) proposed a very clever way of using pre-trained MNLI models as zero-shot sequence classifiers. The methods works by reformulating the question to an MNLI hypothesis. If we want to figure out if a text is about "sport", we simply state that "This text is about sport" ("Denne teksten handler om sport").

When the model is finetuned on the 400k large MNLI task, it is in many cases able to solve this classification tasks. There are no MNLI-set of this size in Norwegian but we have trained it on a machine translated version of the original MNLI-set.

## Testing the model
For testing the model, we recommend the [NbAiLab Colab Notebook](https://colab.research.google.com/gist/peregilk/769b5150a2f807219ab8f15dd11ea449/nbailab-mnli-norwegian-demo.ipynb)

## Hugging Face zero-shot-classification pipeline
The easiest way to try this out is by using the Hugging Face pipeline. Please, note that you will get better results when using Norwegian hypothesis template instead of the default English one.
```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification", model="NbAiLab/nb-bert-base-mnli")
```
You can then use this pipeline to classify sequences into any of the class names you specify.
```python
sequence_to_classify = 'Folkehelseinstituttets mest optimistiske anslag er at alle voksne er ferdigvaksinert innen midten av september.'
candidate_labels = ['politikk', 'helse', 'sport', 'religion']
hypothesis_template = 'Dette eksempelet er {}.'
classifier(sequence_to_classify, candidate_labels, hypothesis_template=hypothesis_template, multi_class=True)

# {'labels': ['helse', 'politikk', 'sport', 'religion'],
# 'scores': [0.4210019111633301, 0.0674605593085289, 0.000840459018945694, 0.0007541406666859984],
# 'sequence': 'Folkehelseinstituttets mest optimistiske anslag er at alle over 18 år er ferdigvaksinert innen midten av september.'}

```

## More information

For more information on the model, see

https://github.com/NBAiLab/notram

Here you will also find a Colab explaining more in details how to use the zero-shot-classification pipeline.


---
license: apache-2.0
tags:
- generated_from_trainer
- automatic-speech-recognition
- NbAiLab/NPSC
- robust-speech-event
- no
- nb-NO
datasets:
- NbAiLab/NPSC
language:
- nb-NO
- generated_from_trainer
model-index:
- name: wav2vec2-xls-r-1b-npsc-bokmaal
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
      args: 16K_mp3_bokmaal
    metrics:
       - name: Test (Bokmål) WER
         type: wer
         value: 0.07901700231893541
       - name: Test (Bokmål) CER
         type: cer
         value: 0.029734583252347752
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xls-r-1b-npsc

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the [NbAiLab/NPSC (16K_mp3_bokmaal)](https://huggingface.co/datasets/NbAiLab/NPSC/viewer/16K_mp3_bokmaal/train) dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1598
- WER: 0.0966

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 15.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.8361        | 0.32  | 500   | 0.6304          | 0.4970 |
| 0.5703        | 0.64  | 1000  | 0.3195          | 0.2775 |
| 0.5451        | 0.97  | 1500  | 0.2700          | 0.2246 |
| 0.47          | 1.29  | 2000  | 0.2564          | 0.2329 |
| 0.4063        | 1.61  | 2500  | 0.2459          | 0.2099 |
| 0.374         | 1.93  | 3000  | 0.2175          | 0.1894 |
| 0.3297        | 2.26  | 3500  | 0.2036          | 0.1755 |
| 0.3145        | 2.58  | 4000  | 0.1957          | 0.1757 |
| 0.3989        | 2.9   | 4500  | 0.1923          | 0.1723 |
| 0.271         | 3.22  | 5000  | 0.1889          | 0.1649 |
| 0.2758        | 3.55  | 5500  | 0.1768          | 0.1588 |
| 0.2683        | 3.87  | 6000  | 0.1720          | 0.1534 |
| 0.2341        | 4.19  | 6500  | 0.1689          | 0.1471 |
| 0.2316        | 4.51  | 7000  | 0.1706          | 0.1405 |
| 0.2383        | 4.84  | 7500  | 0.1637          | 0.1426 |
| 0.2148        | 5.16  | 8000  | 0.1584          | 0.1347 |
| 0.2085        | 5.48  | 8500  | 0.1601          | 0.1387 |
| 0.2944        | 5.8   | 9000  | 0.1566          | 0.1294 |
| 0.1944        | 6.13  | 9500  | 0.1494          | 0.1271 |
| 0.1853        | 6.45  | 10000 | 0.1561          | 0.1247 |
| 0.235         | 6.77  | 10500 | 0.1461          | 0.1215 |
| 0.2286        | 7.09  | 11000 | 0.1447          | 0.1167 |
| 0.1781        | 7.41  | 11500 | 0.1502          | 0.1199 |
| 0.1714        | 7.74  | 12000 | 0.1425          | 0.1179 |
| 0.1725        | 8.06  | 12500 | 0.1427          | 0.1173 |
| 0.143         | 8.38  | 13000 | 0.1448          | 0.1142 |
| 0.154         | 8.7   | 13500 | 0.1392          | 0.1104 |
| 0.1447        | 9.03  | 14000 | 0.1404          | 0.1094 |
| 0.1471        | 9.35  | 14500 | 0.1404          | 0.1088 |
| 0.1479        | 9.67  | 15000 | 0.1414          | 0.1133 |
| 0.1607        | 9.99  | 15500 | 0.1458          | 0.1171 |
| 0.166         | 10.32 | 16000 | 0.1652          | 0.1264 |
| 0.188         | 10.64 | 16500 | 0.1713          | 0.1322 |
| 0.1461        | 10.96 | 17000 | 0.1423          | 0.1111 |
| 0.1289        | 11.28 | 17500 | 0.1388          | 0.1097 |
| 0.1273        | 11.61 | 18000 | 0.1438          | 0.1074 |
| 0.1317        | 11.93 | 18500 | 0.1312          | 0.1066 |
| 0.1448        | 12.25 | 19000 | 0.1446          | 0.1042 |
| 0.1424        | 12.57 | 19500 | 0.1386          | 0.1015 |
| 0.1392        | 12.89 | 20000 | 0.1379          | 0.1005 |
| 0.1408        | 13.22 | 20500 | 0.1408          | 0.0992 |
| 0.1239        | 13.54 | 21000 | 0.1338          | 0.0968 |
| 0.1244        | 13.86 | 21500 | 0.1335          | 0.0957 |
| 0.1254        | 14.18 | 22000 | 0.1382          | 0.0950 |
| 0.1597        | 14.51 | 22500 | 0.1544          | 0.0970 |
| 0.1566        | 14.83 | 23000 | 0.1589          | 0.0963 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu113
- Datasets 1.18.3.dev0
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_trainer
- automatic-speech-recognition
- NbAiLab/NPSC
- robust-speech-event
- no
- nb-NO
datasets:
- NbAiLab/NPSC
language:
- nb-NO
model-index:
- name: wav2vec2-large-voxrex-npsc-bokmaal
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
      args: 16K_mp3_bokmaal
    metrics:
       - name: Test (Bokmål) WER
         type: wer
         value: 0.0711942036236974
       - name: Test (Bokmål) CER
         type: cer
         value: 0.026624736900685928
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-large-voxrex-npsc-bokmaal

This model was trained from scratch on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1311
- Wer: 0.1038

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 8.379967082059723e-06
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 0.1
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.2127        | 0.32  | 500  | 0.1335          | 0.1047 |
| 0.1976        | 0.64  | 1000 | 0.1309          | 0.1039 |
| 0.1887        | 0.97  | 1500 | 0.1306          | 0.1040 |
| 0.18          | 1.29  | 2000 | 0.1311          | 0.1038 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu113
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_trainer
- automatic-speech-recognition
- NbAiLab/NPSC
- robust-speech-event
- no
- nn-NO
datasets:
- NbAiLab/NPSC
language:
- nn-NO
model-index:
- name: wav2vec2-xlsr-1B-NPSC-NN
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
      args: 16K_mp3_nynorsk
    metrics:
       - name: Test (Nynorsk) WER
         type: wer
         value: 0.13347099680871036
       - name: Test (Nynorsk) CER
         type: cer
         value: 0.04537322093454329
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xlsr-1B-NPSC-NN

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the NBAILAB/NPSC - 16K_MP3 dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4562
- Wer: 0.1531

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 6e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 1.6894        | 1.08  | 500   | 1.2423          | 0.8619 |
| 0.7543        | 2.15  | 1000  | 0.5956          | 0.3817 |
| 0.5481        | 3.23  | 1500  | 0.5043          | 0.3246 |
| 0.4661        | 4.3   | 2000  | 0.4813          | 0.2793 |
| 0.3901        | 5.38  | 2500  | 0.4371          | 0.2592 |
| 0.3512        | 6.45  | 3000  | 0.4216          | 0.2458 |
| 0.3016        | 7.53  | 3500  | 0.3814          | 0.2257 |
| 0.278         | 8.6   | 4000  | 0.4151          | 0.2145 |
| 0.2435        | 9.68  | 4500  | 0.4816          | 0.2130 |
| 0.2122        | 10.75 | 5000  | 0.4489          | 0.2137 |
| 0.1949        | 11.83 | 5500  | 0.3978          | 0.2063 |
| 0.1929        | 12.9  | 6000  | 0.3823          | 0.2026 |
| 0.1757        | 13.98 | 6500  | 0.3409          | 0.1965 |
| 0.1771        | 15.05 | 7000  | 0.3844          | 0.1936 |
| 0.1452        | 16.13 | 7500  | 0.3749          | 0.1900 |
| 0.1341        | 17.2  | 8000  | 0.4407          | 0.2026 |
| 0.13          | 18.28 | 8500  | 0.4253          | 0.1883 |
| 0.1183        | 19.35 | 9000  | 0.4311          | 0.1880 |
| 0.118         | 20.43 | 9500  | 0.4431          | 0.1882 |
| 0.1123        | 21.51 | 10000 | 0.4753          | 0.1820 |
| 0.1037        | 22.58 | 10500 | 0.4087          | 0.1834 |
| 0.1066        | 23.66 | 11000 | 0.4151          | 0.1845 |
| 0.0977        | 24.73 | 11500 | 0.4367          | 0.1783 |
| 0.0968        | 25.81 | 12000 | 0.4237          | 0.1756 |
| 0.0835        | 26.88 | 12500 | 0.4729          | 0.1781 |
| 0.0919        | 27.96 | 13000 | 0.4153          | 0.1701 |
| 0.0677        | 29.03 | 13500 | 0.4317          | 0.1693 |
| 0.0726        | 30.11 | 14000 | 0.4380          | 0.1736 |
| 0.066         | 31.18 | 14500 | 0.4384          | 0.1681 |
| 0.0713        | 32.26 | 15000 | 0.4215          | 0.1629 |
| 0.0605        | 33.33 | 15500 | 0.4574          | 0.1714 |
| 0.0632        | 34.41 | 16000 | 0.4343          | 0.1642 |
| 0.0567        | 35.48 | 16500 | 0.4231          | 0.1601 |
| 0.0556        | 36.56 | 17000 | 0.4404          | 0.1667 |
| 0.0426        | 37.63 | 17500 | 0.4459          | 0.1625 |
| 0.0445        | 38.71 | 18000 | 0.4484          | 0.1629 |
| 0.0463        | 39.78 | 18500 | 0.4508          | 0.1596 |
| 0.0448        | 40.86 | 19000 | 0.4395          | 0.1605 |
| 0.0434        | 41.94 | 19500 | 0.4490          | 0.1607 |
| 0.0347        | 43.01 | 20000 | 0.4772          | 0.1582 |
| 0.0332        | 44.09 | 20500 | 0.4729          | 0.1582 |
| 0.037         | 45.16 | 21000 | 0.4559          | 0.1573 |
| 0.0328        | 46.24 | 21500 | 0.4664          | 0.1560 |
| 0.0366        | 47.31 | 22000 | 0.4543          | 0.1543 |
| 0.0377        | 48.39 | 22500 | 0.4507          | 0.1560 |
| 0.0331        | 49.46 | 23000 | 0.4567          | 0.1533 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
language: no
license: cc-by-4.0
tags:
- norwegian
- bert
thumbnail: nblogo_3.png
pipeline_tag: fill-mask
widget:
- text: På biblioteket kan du låne en [MASK].
---

- **Release 1.0beta** (April 29, 2021)

# NB-BERT-large (beta)

## Description

NB-BERT-large is a general BERT-large model built on the large digital collection at the National Library of Norway.

This model is trained from scratch on a wide variety of Norwegian text (both bokmål and nynorsk) from the last 200 years using a monolingual Norwegian vocabulary.

## Intended use & limitations

The 1.0 version of the model is general, and should be fine-tuned for any particular use. Some fine-tuning sets may be found on Github, see

* https://github.com/NBAiLab/notram

## Training data

The model is trained on a wide variety of text. The training set is described on

* https://github.com/NBAiLab/notram

## More information

For more information on the model, see

https://github.com/NBAiLab/notram


Just for performing some experiments. Do not use.


Just for performing some experiments. Do not use.

Since the loss seem to start going up, I did have to restore this from 9e945cb0636bde60bec30bd7df5db30f80401cc7 (2 step 600k/200). I am then restarting with warmup decaying from 1e-4.

That did failed. Checked out c94b5bb43b05fc798f9db013d940b05b3b47cd98 instead and restarted step 3 from here.


---
language:
- nb-NO
license: apache-2.0
tags:
- automatic-speech-recognition
- generated_from_trainer
- no
- nb-NO
- robust-speech-event
- model_for_talk
datasets:
- NbAiLab/NPSC
model-index:
- name: XLS-R-300M-LM - Norwegian
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: NPSC
      type: NbAiLab/NPSC
    metrics:
       - name: Eval WER
         type: wer
         value: 15.40
       - name: Eval CER
         type: cer
         value: 5.48

---

# XLS-R-300M-LM - Norwegian

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the Norwegian [NPSC](https://huggingface.co/datasets/NbAiLab/NPSC) dataset.

### Scores without Language Model
Without using a language model, it achieves the following scores on the NPSC Eval set
It achieves the following results on the evaluation set without a language model:
- WER: 0.2110
- CER: 0.0622

### Scores with Language Model
A 5-gram KenLM was added to boost the models performance. The language model was created on a corpus mainly consisting of online newspapers, public reports and Wikipedia data. After this we are getting these values.

- WER: 0.1540
- CER: 0.0548

## Team
The model is developed by Rolv-Arild Braaten, Per Egil Kummervold, Andre Kåsen, Javier de la Rosa, Per Erik Solberg, and Freddy Wetjen. Name in alphabetic order.

## Model description
This current version is based on checkpoint 8500 of [NbAiLab/wav2vec2-xlsr-300M-NPSC-OH](https://huggingface.co/NbAiLab/wav2vec2-xlsr-300M-NPSC-OH).

## Intended uses & limitations
Demo version only. The model will be updated later this week.

## Training and evaluation data
The model is trained and evaluated on [NPSC](https://huggingface.co/datasets/NbAiLab/NPSC). Unfortunately there is no Norwegian test data in Common Voice, and currently the model is only evaluated on the validation set of NPSC..

## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 7.5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 30.0 (But interrupted after 8500 steps, approx 6 epochs)
- mixed_precision_training: Native AMP


---
language: no
license: cc-by-4.0
tags:
- norwegian
- bert
pipeline_tag: fill-mask
widget:
- text: På biblioteket kan du [MASK] en bok.
- text: Dette er et [MASK] eksempel.
- text: Av og til kan en språkmodell gi et [MASK] resultat.
- text: Som ansat får du [MASK] for at bidrage til borgernes adgang til dansk kulturarv, til forskning og til samfundets demokratiske udvikling.
---
- **Release 1.1** (March 11, 2021)
- **Release 1.0** (January 13, 2021)

# NB-BERT-base

## Description

NB-BERT-base is a general BERT-base model built on the large digital collection at the National Library of Norway.

This model is based on the same structure as [BERT Cased multilingual model](https://github.com/google-research/bert/blob/master/multilingual.md), and is trained on a wide variety of Norwegian text (both bokmål and nynorsk) from the last 200 years.

## Intended use & limitations

The 1.1 version of the model is general, and should be fine-tuned for any particular use. Some fine-tuning sets may be found on GitHub, see

* https://github.com/NBAiLab/notram

## Training data

The model is trained on a wide variety of text. The training set is described on

* https://github.com/NBAiLab/notram

## More information

For more information on the model, see

https://github.com/NBAiLab/notram


---
language: no
license: cc-by-4.0
tags:
- seq2seq
datasets:
- Norwegian Nynorsk/Bokmål
---
# 🇳🇴 Norwegian T5 Base model Trained on the NCC🇳🇴

This is a Norwegian T5-base model trained on the Norwegian Colossal Corpus (NCC) on a TPU v3-8.

This model is currently training. It will finish in January 2022. Please do not use yet..
 ```


---
language: no
license: cc-by-4.0
tags:
- norwegian
- bert
- ner
thumbnail: nblogo_3.png
pipeline_tag: token-classification
widget:
- text: Trond Giske har bekreftet på spørsmål fra Adresseavisen at Hansen leide et rom i hans leilighet i Trondheim.
---

**Release 1.0** (November 17, 2021)

# nb-bert-base-ner

## Description
NB-Bert base model fine-tuned on the Named Entity Recognition task using the [NorNE dataset](https://huggingface.co/datasets/NbAiLab/norne).

## Usage

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained("NbAiLab/nb-bert-base-ner")
model = AutoModelForTokenClassification.from_pretrained("NbAiLab/nb-bert-base-ner")

nlp = pipeline("ner", model=model, tokenizer=tokenizer)
example = "Jeg heter Kjell og bor i Oslo."

ner_results = nlp(example)
print(ner_results)
```


---
language: no
license: cc-by-4.0
tags:
- norwegian
- roberta
pipeline_tag: fill-mask
widget:
- text: På biblioteket kan du <mask> en bok.
- text: Dette er et <mask> eksempel.
- text: Av og til kan en språkmodell gi et <mask> resultat.
- text: Som ansat får du <mask> for at bidrage til borgernes adgang til dansk kulturarv, til forskning og til samfundets demokratiske udvikling.
---

# This is just a Test Model. Do NOT use for anything!

Continued pretrained from the nb-roberta-base.

The domain specific pretraining is done on the 102GB (Scandinavian corpus)[https://huggingface.co/datasets/NbAiLab/scandinavian].

## Train for 180k steps for 128 sequences:
```bash
./run_mlm_flax_stream.py \
    --output_dir="./" \
    --model_type="roberta" \
    --config_name="./" \
    --tokenizer_name="./" \
    --model_name_or_path="./" \
    --dataset_name="NbAiLab/scandinavian" \
    --max_seq_length="128" \
    --weight_decay="0.01" \
    --per_device_train_batch_size="128" \
    --per_device_eval_batch_size="128" \
    --learning_rate="6e-5" \
    --warmup_steps="5000" \
    --overwrite_output_dir \
    --cache_dir /mnt/disks/flaxdisk/cache/ \
    --num_train_steps="180000" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --logging_steps="10000" \
    --save_steps="10000" \
    --eval_steps="10000" \
    --preprocessing_num_workers 96 \
    --auth_token True \
    --adafactor \
    --push_to_hub
```
## Train for 20k steps for 512 sequences:
```bash
./run_mlm_flax_stream.py \
    --output_dir="./" \
    --model_type="roberta" \
    --config_name="./" \
    --tokenizer_name="./" \
    --model_name_or_path="./" \
    --dataset_name="NbAiLab/scandinavian" \
    --max_seq_length="512" \
    --weight_decay="0.01" \
    --per_device_train_batch_size="48" \
    --per_device_eval_batch_size="48" \
    --learning_rate="3e-5" \
    --warmup_steps="5000" \
    --overwrite_output_dir \
    --cache_dir /mnt/disks/flaxdisk/cache/ \
    --num_train_steps="20000" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --logging_steps="20000" \
    --save_steps="10000" \
    --eval_steps="10000" \
    --preprocessing_num_workers 96 \
    --auth_token True \
    --adafactor \
    --push_to_hub
```

Approximate additional training time: 1 week.


---
language: no
license: cc-by-4.0
tags:
- norwegian
- bert
pipeline_tag: fill-mask
widget:
- text: På biblioteket kan du [MASK] en bok.
- text: Dette er et [MASK] eksempel.
- text: Av og til kan en språkmodell gi et [MASK] resultat.
- text: Som ansat får du [MASK] for at bidrage til borgernes adgang til dansk kulturarv, til forskning og til samfundets demokratiske udvikling.
---
Internal. Waiting to be evaluated.


---
tags:
- conversational
---

# Ryuji DialoGPT Model


---
tags:
- conversational
---

# Harry Potter DialoGPT Model