---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: t5-small-finetuned-en-to-ro-dataset_20-input_64
  results:
  - task:
      name: Sequence-to-sequence Language Modeling
      type: text2text-generation
    dataset:
      name: wmt16
      type: wmt16
      args: ro-en
    metrics:
    - name: Bleu
      type: bleu
      value: 8.6652
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-small-finetuned-en-to-ro-dataset_20-input_64

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the wmt16 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4335
- Bleu: 8.6652
- Gen Len: 18.2596

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu   | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:-------:|
| 0.6351        | 1.0   | 7629 | 1.4335          | 8.6652 | 18.2596 |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: t5-small-finetuned-en-to-ro-dataset_20
  results:
  - task:
      name: Sequence-to-sequence Language Modeling
      type: text2text-generation
    dataset:
      name: wmt16
      type: wmt16
      args: ro-en
    metrics:
    - name: Bleu
      type: bleu
      value: 7.3293
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-small-finetuned-en-to-ro-dataset_20

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the wmt16 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4052
- Bleu: 7.3293
- Gen Len: 18.2556

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu   | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:-------:|
| 0.6029        | 1.0   | 7629 | 1.4052          | 7.3293 | 18.2556 |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: t5-small-finetuned-en-to-ro-epoch.04375
  results:
  - task:
      name: Sequence-to-sequence Language Modeling
      type: text2text-generation
    dataset:
      name: wmt16
      type: wmt16
      args: ro-en
    metrics:
    - name: Bleu
      type: bleu
      value: 7.3292
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-small-finetuned-en-to-ro-epoch.04375

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the wmt16 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4137
- Bleu: 7.3292
- Gen Len: 18.2541

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 0.04375
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu   | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:-------:|
| 0.6211        | 0.04  | 1669 | 1.4137          | 7.3292 | 18.2541 |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
language: fi
metrics:
- wer
- cer
tags:
- generated_from_trainer
- automatic-speech-recognition
- fi
- finnish
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
model-index:
- name: wav2vec2-xlsr-1b-finnish
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 7
      type: mozilla-foundation/common_voice_7_0
      args: fi
    metrics:
       - name: Test WER
         type: wer
         value: 13.11
       - name: Test CER
         type: cer
         value: 2.23

---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xlsr-1b-finnish

**Note**: there is a better V2 version of this model which has been fine-tuned longer with 16 hours of more data: [aapot/wav2vec2-xlsr-1b-finnish-v2](https://huggingface.co/aapot/wav2vec2-xlsr-1b-finnish-v2)

**Note**: there is a language model version of this acoustic only model which achieves better test results thanks to the added language model: [aapot/wav2vec2-xlsr-1b-finnish-lm](https://huggingface.co/aapot/wav2vec2-xlsr-1b-finnish-lm)

This acoustic model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) for Finnish ASR. The model has been fine-tuned with 259.57 hours of Finnish transcribed speech data.
It achieves the following results on the Common Voice 7 test set together without language model:
- Wer: 13.11
- Cer: 2.23

## Model description

TODO

## Intended uses & limitations

TODO

## Training and evaluation data

This model was fine-tuned with 259.57 hours of Finnish transcribed speech data from following datasets:

| Dataset                                                                                | Hours    | % of total hours |
|:------------------------------------------------------------------------------------------------------------------------------|:--------:|:----------------:|
| [Common Voice 7.0 Finnish train+evaluation+other splits](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) | 9.70 h   | 3.74 %           |
| [Finnish parliament session 2](https://b2share.eudat.eu/records/4df422d631544ce682d6af1d4714b2d4)                             | 0.24 h   | 0.09 %           |
| [VoxPopuli Finnish](https://github.com/facebookresearch/voxpopuli)                                                            | 5.94 h   | 2.29 %           |
| [CSS10 Finnish](https://github.com/kyubyong/css10)                                                                            | 10.32 h  | 3.98 %           |
| [Aalto Finnish Parliament ASR Corpus](http://urn.fi/urn:nbn:fi:lb-2021051903)                                                 | 228.00 h | 87.84 %          |
| [Finnish Broadcast Corpus](http://urn.fi/urn:nbn:fi:lb-2016042502)                                                            | 5.37 h   | 2.07 %           |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: [8-bit Adam](https://github.com/facebookresearch/bitsandbytes) with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 5
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.968         | 0.18  | 500   | 0.4870          | 0.4720 |
| 0.6557        | 0.36  | 1000  | 0.2450          | 0.2931 |
| 0.647         | 0.54  | 1500  | 0.1818          | 0.2255 |
| 0.5297        | 0.72  | 2000  | 0.1698          | 0.2354 |
| 0.5802        | 0.9   | 2500  | 0.1581          | 0.2355 |
| 0.6351        | 1.07  | 3000  | 0.1689          | 0.2336 |
| 0.4626        | 1.25  | 3500  | 0.1719          | 0.3099 |
| 0.4526        | 1.43  | 4000  | 0.1434          | 0.2069 |
| 0.4692        | 1.61  | 4500  | 0.1645          | 0.2192 |
| 0.4584        | 1.79  | 5000  | 0.1483          | 0.1987 |
| 0.4234        | 1.97  | 5500  | 0.1499          | 0.2178 |
| 0.4243        | 2.15  | 6000  | 0.1345          | 0.2070 |
| 0.4108        | 2.33  | 6500  | 0.1383          | 0.1850 |
| 0.4048        | 2.51  | 7000  | 0.1338          | 0.1811 |
| 0.4085        | 2.69  | 7500  | 0.1290          | 0.1780 |
| 0.4026        | 2.87  | 8000  | 0.1239          | 0.1650 |
| 0.4033        | 3.04  | 8500  | 0.1346          | 0.1657 |
| 0.3986        | 3.22  | 9000  | 0.1310          | 0.1850 |
| 0.3867        | 3.4   | 9500  | 0.1273          | 0.1741 |
| 0.3658        | 3.58  | 10000 | 0.1219          | 0.1672 |
| 0.382         | 3.76  | 10500 | 0.1306          | 0.1698 |
| 0.3847        | 3.94  | 11000 | 0.1230          | 0.1577 |
| 0.3691        | 4.12  | 11500 | 0.1310          | 0.1615 |
| 0.3593        | 4.3   | 12000 | 0.1296          | 0.1622 |
| 0.3619        | 4.48  | 12500 | 0.1285          | 0.1601 |
| 0.3361        | 4.66  | 13000 | 0.1261          | 0.1569 |
| 0.3603        | 4.84  | 13500 | 0.1235          | 0.1533 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
license: apache-2.0
language: fi
metrics:
- wer
- cer
tags:
- generated_from_trainer
- automatic-speech-recognition
- fi
- finnish
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
model-index:
- name: wav2vec2-xlsr-300m-finnish
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 7
      type: mozilla-foundation/common_voice_7_0
      args: fi
    metrics:
       - name: Test WER
         type: wer
         value: 17.92
       - name: Test CER
         type: cer
         value: 3.36

---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xlsr-300m-finnish

This acoustic model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) for Finnish ASR. The model has been fine-tuned with 275.6 hours of Finnish transcribed speech data.
It achieves the following results on the Common Voice 7 test set without language model:
- Wer: 17.92
- Cer: 3.36

## Model description

TODO

## Intended uses & limitations

TODO

## Training and evaluation data

This model was fine-tuned with 275.6 hours of Finnish transcribed speech data from following datasets:

| Dataset                                                                                | Hours    | % of total hours |
|:------------------------------------------------------------------------------------------------------------------------------|:--------:|:----------------:|
| [Common Voice 7.0 Finnish train+evaluation+other splits](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) | 9.70 h   | 3.52 %           |
| [Finnish parliament session 2](https://b2share.eudat.eu/records/4df422d631544ce682d6af1d4714b2d4)                             | 0.24 h   | 0.09 %           |
| [VoxPopuli Finnish](https://github.com/facebookresearch/voxpopuli)                                                            | 21.97 h  | 7.97 %           |
| [CSS10 Finnish](https://github.com/kyubyong/css10)                                                                            | 10.32 h  | 3.74 %           |
| [Aalto Finnish Parliament ASR Corpus](http://urn.fi/urn:nbn:fi:lb-2021051903)                                                 | 228.00 h | 82.73 %          |
| [Finnish Broadcast Corpus](http://urn.fi/urn:nbn:fi:lb-2016042502)                                                            | 5.37 h   | 1.95 %           |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: [8-bit Adam](https://github.com/facebookresearch/bitsandbytes) with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 10
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.973         | 0.17  | 500   | 0.5750          | 0.6844 |
| 0.713         | 0.34  | 1000  | 0.3356          | 0.4518 |
| 0.6563        | 0.5   | 1500  | 0.3007          | 0.4039 |
| 0.642         | 0.67  | 2000  | 0.2619          | 0.3674 |
| 0.6203        | 0.84  | 2500  | 0.2488          | 0.3558 |
| 0.6016        | 1.01  | 3000  | 0.2795          | 0.3835 |
| 0.5423        | 1.17  | 3500  | 0.2652          | 0.3310 |
| 0.5639        | 1.34  | 4000  | 0.2479          | 0.3462 |
| 0.586         | 1.51  | 4500  | 0.2409          | 0.3295 |
| 0.5169        | 1.68  | 5000  | 0.2728          | 0.3352 |
| 0.5176        | 1.84  | 5500  | 0.2254          | 0.3149 |
| 0.4983        | 2.01  | 6000  | 0.2169          | 0.3009 |
| 0.4982        | 2.18  | 6500  | 0.2215          | 0.3079 |
| 0.4898        | 2.35  | 7000  | 0.2174          | 0.3023 |
| 0.4922        | 2.51  | 7500  | 0.2217          | 0.3081 |
| 0.5025        | 2.68  | 8000  | 0.2002          | 0.2710 |
| 0.4745        | 2.85  | 8500  | 0.1935          | 0.2783 |
| 0.4377        | 3.02  | 9000  | 0.1859          | 0.2742 |
| 0.4511        | 3.18  | 9500  | 0.2038          | 0.2786 |
| 0.4411        | 3.35  | 10000 | 0.1863          | 0.2651 |
| 0.4501        | 3.52  | 10500 | 0.1948          | 0.2605 |
| 0.4557        | 3.69  | 11000 | 0.1872          | 0.2695 |
| 0.4493        | 3.85  | 11500 | 0.1888          | 0.2632 |
| 0.4047        | 4.02  | 12000 | 0.1818          | 0.2559 |
| 0.4319        | 4.19  | 12500 | 0.1896          | 0.2648 |
| 0.4162        | 4.36  | 13000 | 0.1953          | 0.2595 |
| 0.4046        | 4.52  | 13500 | 0.1864          | 0.2606 |
| 0.4195        | 4.69  | 14000 | 0.1843          | 0.2467 |
| 0.4146        | 4.86  | 14500 | 0.1686          | 0.2450 |
| 0.378         | 5.03  | 15000 | 0.1731          | 0.2401 |
| 0.3792        | 5.19  | 15500 | 0.1676          | 0.2325 |
| 0.3855        | 5.36  | 16000 | 0.1740          | 0.2326 |
| 0.4029        | 5.53  | 16500 | 0.1674          | 0.2345 |
| 0.386         | 5.7   | 17000 | 0.1735          | 0.2280 |
| 0.3811        | 5.86  | 17500 | 0.1692          | 0.2258 |
| 0.3607        | 6.03  | 18000 | 0.1797          | 0.2279 |
| 0.3604        | 6.2   | 18500 | 0.1651          | 0.2206 |
| 0.3362        | 6.37  | 19000 | 0.1627          | 0.2199 |
| 0.3611        | 6.53  | 19500 | 0.1652          | 0.2172 |
| 0.3671        | 6.7   | 20000 | 0.1564          | 0.2140 |
| 0.3769        | 6.87  | 20500 | 0.1525          | 0.2101 |
| 0.3539        | 7.04  | 21000 | 0.1639          | 0.2096 |
| 0.3225        | 7.21  | 21500 | 0.1611          | 0.2087 |
| 0.3323        | 7.37  | 22000 | 0.1633          | 0.2008 |
| 0.3327        | 7.54  | 22500 | 0.1692          | 0.1975 |
| 0.3456        | 7.71  | 23000 | 0.1555          | 0.1991 |
| 0.3058        | 7.88  | 23500 | 0.1590          | 0.1959 |
| 0.3034        | 8.04  | 24000 | 0.1531          | 0.1973 |
| 0.2925        | 8.21  | 24500 | 0.1583          | 0.1978 |
| 0.2967        | 8.38  | 25000 | 0.1546          | 0.1906 |
| 0.2974        | 8.55  | 25500 | 0.1540          | 0.1869 |
| 0.3131        | 8.71  | 26000 | 0.1534          | 0.1850 |
| 0.3306        | 8.88  | 26500 | 0.1482          | 0.1844 |
| 0.2842        | 9.05  | 27000 | 0.1490          | 0.1854 |
| 0.2879        | 9.22  | 27500 | 0.1463          | 0.1799 |
| 0.27          | 9.38  | 28000 | 0.1454          | 0.1798 |
| 0.2874        | 9.55  | 28500 | 0.1504          | 0.1787 |
| 0.2757        | 9.72  | 29000 | 0.1512          | 0.1784 |
| 0.3017        | 9.89  | 29500 | 0.1484          | 0.1800 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.3
- Tokenizers 0.11.0


---
license: apache-2.0
language: fi
metrics:
- wer
- cer
tags:
- generated_from_trainer
- automatic-speech-recognition
- fi
- finnish
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
model-index:
- name: wav2vec2-xlsr-300m-finnish-lm
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 7
      type: mozilla-foundation/common_voice_7_0
      args: fi
    metrics:
       - name: Test WER
         type: wer
         value: 8.24
       - name: Test CER
         type: cer
         value: 1.98

---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xlsr-300m-finnish-lm

This acoustic model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) for Finnish ASR. The model has been fine-tuned with 275.6 hours of Finnish transcribed speech data.
It achieves the following results on the Common Voice 7 test set together with language model (Finnish KenLM):
- Wer: 8.24
- Cer: 1.98

## Model description

TODO

## Intended uses & limitations

TODO

## Training and evaluation data

This model was fine-tuned with 275.6 hours of Finnish transcribed speech data from following datasets:

| Dataset                                                                                | Hours    | % of total hours |
|:------------------------------------------------------------------------------------------------------------------------------|:--------:|:----------------:|
| [Common Voice 7.0 Finnish train+evaluation+other splits](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) | 9.70 h   | 3.52 %           |
| [Finnish parliament session 2](https://b2share.eudat.eu/records/4df422d631544ce682d6af1d4714b2d4)                             | 0.24 h   | 0.09 %           |
| [VoxPopuli Finnish](https://github.com/facebookresearch/voxpopuli)                                                            | 21.97 h  | 7.97 %           |
| [CSS10 Finnish](https://github.com/kyubyong/css10)                                                                            | 10.32 h  | 3.74 %           |
| [Aalto Finnish Parliament ASR Corpus](http://urn.fi/urn:nbn:fi:lb-2021051903)                                                 | 228.00 h | 82.73 %          |
| [Finnish Broadcast Corpus](http://urn.fi/urn:nbn:fi:lb-2016042502)                                                            | 5.37 h   | 1.95 %           |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: [8-bit Adam](https://github.com/facebookresearch/bitsandbytes) with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 10
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.973         | 0.17  | 500   | 0.5750          | 0.6844 |
| 0.713         | 0.34  | 1000  | 0.3356          | 0.4518 |
| 0.6563        | 0.5   | 1500  | 0.3007          | 0.4039 |
| 0.642         | 0.67  | 2000  | 0.2619          | 0.3674 |
| 0.6203        | 0.84  | 2500  | 0.2488          | 0.3558 |
| 0.6016        | 1.01  | 3000  | 0.2795          | 0.3835 |
| 0.5423        | 1.17  | 3500  | 0.2652          | 0.3310 |
| 0.5639        | 1.34  | 4000  | 0.2479          | 0.3462 |
| 0.586         | 1.51  | 4500  | 0.2409          | 0.3295 |
| 0.5169        | 1.68  | 5000  | 0.2728          | 0.3352 |
| 0.5176        | 1.84  | 5500  | 0.2254          | 0.3149 |
| 0.4983        | 2.01  | 6000  | 0.2169          | 0.3009 |
| 0.4982        | 2.18  | 6500  | 0.2215          | 0.3079 |
| 0.4898        | 2.35  | 7000  | 0.2174          | 0.3023 |
| 0.4922        | 2.51  | 7500  | 0.2217          | 0.3081 |
| 0.5025        | 2.68  | 8000  | 0.2002          | 0.2710 |
| 0.4745        | 2.85  | 8500  | 0.1935          | 0.2783 |
| 0.4377        | 3.02  | 9000  | 0.1859          | 0.2742 |
| 0.4511        | 3.18  | 9500  | 0.2038          | 0.2786 |
| 0.4411        | 3.35  | 10000 | 0.1863          | 0.2651 |
| 0.4501        | 3.52  | 10500 | 0.1948          | 0.2605 |
| 0.4557        | 3.69  | 11000 | 0.1872          | 0.2695 |
| 0.4493        | 3.85  | 11500 | 0.1888          | 0.2632 |
| 0.4047        | 4.02  | 12000 | 0.1818          | 0.2559 |
| 0.4319        | 4.19  | 12500 | 0.1896          | 0.2648 |
| 0.4162        | 4.36  | 13000 | 0.1953          | 0.2595 |
| 0.4046        | 4.52  | 13500 | 0.1864          | 0.2606 |
| 0.4195        | 4.69  | 14000 | 0.1843          | 0.2467 |
| 0.4146        | 4.86  | 14500 | 0.1686          | 0.2450 |
| 0.378         | 5.03  | 15000 | 0.1731          | 0.2401 |
| 0.3792        | 5.19  | 15500 | 0.1676          | 0.2325 |
| 0.3855        | 5.36  | 16000 | 0.1740          | 0.2326 |
| 0.4029        | 5.53  | 16500 | 0.1674          | 0.2345 |
| 0.386         | 5.7   | 17000 | 0.1735          | 0.2280 |
| 0.3811        | 5.86  | 17500 | 0.1692          | 0.2258 |
| 0.3607        | 6.03  | 18000 | 0.1797          | 0.2279 |
| 0.3604        | 6.2   | 18500 | 0.1651          | 0.2206 |
| 0.3362        | 6.37  | 19000 | 0.1627          | 0.2199 |
| 0.3611        | 6.53  | 19500 | 0.1652          | 0.2172 |
| 0.3671        | 6.7   | 20000 | 0.1564          | 0.2140 |
| 0.3769        | 6.87  | 20500 | 0.1525          | 0.2101 |
| 0.3539        | 7.04  | 21000 | 0.1639          | 0.2096 |
| 0.3225        | 7.21  | 21500 | 0.1611          | 0.2087 |
| 0.3323        | 7.37  | 22000 | 0.1633          | 0.2008 |
| 0.3327        | 7.54  | 22500 | 0.1692          | 0.1975 |
| 0.3456        | 7.71  | 23000 | 0.1555          | 0.1991 |
| 0.3058        | 7.88  | 23500 | 0.1590          | 0.1959 |
| 0.3034        | 8.04  | 24000 | 0.1531          | 0.1973 |
| 0.2925        | 8.21  | 24500 | 0.1583          | 0.1978 |
| 0.2967        | 8.38  | 25000 | 0.1546          | 0.1906 |
| 0.2974        | 8.55  | 25500 | 0.1540          | 0.1869 |
| 0.3131        | 8.71  | 26000 | 0.1534          | 0.1850 |
| 0.3306        | 8.88  | 26500 | 0.1482          | 0.1844 |
| 0.2842        | 9.05  | 27000 | 0.1490          | 0.1854 |
| 0.2879        | 9.22  | 27500 | 0.1463          | 0.1799 |
| 0.27          | 9.38  | 28000 | 0.1454          | 0.1798 |
| 0.2874        | 9.55  | 28500 | 0.1504          | 0.1787 |
| 0.2757        | 9.72  | 29000 | 0.1512          | 0.1784 |
| 0.3017        | 9.89  | 29500 | 0.1484          | 0.1800 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.3
- Tokenizers 0.11.0


---
license: apache-2.0
language: fi
metrics:
- wer
- cer
tags:
- generated_from_trainer
- automatic-speech-recognition
- fi
- finnish
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
model-index:
- name: wav2vec2-xlsr-1b-finnish-v2
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 7
      type: mozilla-foundation/common_voice_7_0
      args: fi
    metrics:
       - name: Test WER
         type: wer
         value: 9.73
       - name: Test CER
         type: cer
         value: 1.65

---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xlsr-1b-finnish-v2

**Note**: there is a language model version of this acoustic only model which achieves better test results thanks to the added language model: [aapot/wav2vec2-xlsr-1b-finnish-lm-v2](https://huggingface.co/aapot/wav2vec2-xlsr-1b-finnish-lm-v2)

This acoustic model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) for Finnish ASR. The model has been fine-tuned with 275.6 hours of Finnish transcribed speech data.
It achieves the following results on the Common Voice 7 test set without language model:
- Wer: 9.73
- Cer: 1.65

## Model description

TODO

## Intended uses & limitations

TODO

## Training and evaluation data

This model was fine-tuned with 275.6 hours of Finnish transcribed speech data from following datasets:

| Dataset                                                                                | Hours    | % of total hours |
|:------------------------------------------------------------------------------------------------------------------------------|:--------:|:----------------:|
| [Common Voice 7.0 Finnish train+evaluation+other splits](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) | 9.70 h   | 3.52 %           |
| [Finnish parliament session 2](https://b2share.eudat.eu/records/4df422d631544ce682d6af1d4714b2d4)                             | 0.24 h   | 0.09 %           |
| [VoxPopuli Finnish](https://github.com/facebookresearch/voxpopuli)                                                            | 21.97 h  | 7.97 %           |
| [CSS10 Finnish](https://github.com/kyubyong/css10)                                                                            | 10.32 h  | 3.74 %           |
| [Aalto Finnish Parliament ASR Corpus](http://urn.fi/urn:nbn:fi:lb-2021051903)                                                 | 228.00 h | 82.73 %          |
| [Finnish Broadcast Corpus](http://urn.fi/urn:nbn:fi:lb-2016042502)                                                            | 5.37 h   | 1.95 %           |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: [8-bit Adam](https://github.com/facebookresearch/bitsandbytes) with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 10
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.7778        | 0.17  | 500   | 0.2851          | 0.3572 |
| 0.5506        | 0.34  | 1000  | 0.1595          | 0.2130 |
| 0.6569        | 0.5   | 1500  | 0.1458          | 0.2046 |
| 0.5997        | 0.67  | 2000  | 0.1374          | 0.1975 |
| 0.542         | 0.84  | 2500  | 0.1390          | 0.1956 |
| 0.4815        | 1.01  | 3000  | 0.1266          | 0.1813 |
| 0.6982        | 1.17  | 3500  | 0.1441          | 0.1965 |
| 0.4522        | 1.34  | 4000  | 0.1232          | 0.1822 |
| 0.4655        | 1.51  | 4500  | 0.1209          | 0.1702 |
| 0.4069        | 1.68  | 5000  | 0.1149          | 0.1688 |
| 0.4226        | 1.84  | 5500  | 0.1121          | 0.1560 |
| 0.3993        | 2.01  | 6000  | 0.1091          | 0.1557 |
| 0.406         | 2.18  | 6500  | 0.1115          | 0.1553 |
| 0.4098        | 2.35  | 7000  | 0.1144          | 0.1560 |
| 0.3995        | 2.51  | 7500  | 0.1028          | 0.1476 |
| 0.4101        | 2.68  | 8000  | 0.1129          | 0.1511 |
| 0.3636        | 2.85  | 8500  | 0.1025          | 0.1517 |
| 0.3534        | 3.02  | 9000  | 0.1068          | 0.1480 |
| 0.3836        | 3.18  | 9500  | 0.1072          | 0.1459 |
| 0.3531        | 3.35  | 10000 | 0.0928          | 0.1367 |
| 0.3649        | 3.52  | 10500 | 0.1042          | 0.1426 |
| 0.3645        | 3.69  | 11000 | 0.0979          | 0.1433 |
| 0.3685        | 3.85  | 11500 | 0.0947          | 0.1346 |
| 0.3325        | 4.02  | 12000 | 0.0991          | 0.1352 |
| 0.3497        | 4.19  | 12500 | 0.0919          | 0.1358 |
| 0.3303        | 4.36  | 13000 | 0.0888          | 0.1272 |
| 0.3323        | 4.52  | 13500 | 0.0888          | 0.1277 |
| 0.3452        | 4.69  | 14000 | 0.0894          | 0.1279 |
| 0.337         | 4.86  | 14500 | 0.0917          | 0.1289 |
| 0.3114        | 5.03  | 15000 | 0.0942          | 0.1313 |
| 0.3099        | 5.19  | 15500 | 0.0902          | 0.1239 |
| 0.3079        | 5.36  | 16000 | 0.0871          | 0.1256 |
| 0.3293        | 5.53  | 16500 | 0.0861          | 0.1263 |
| 0.3123        | 5.7   | 17000 | 0.0876          | 0.1203 |
| 0.3093        | 5.86  | 17500 | 0.0848          | 0.1226 |
| 0.2903        | 6.03  | 18000 | 0.0914          | 0.1221 |
| 0.297         | 6.2   | 18500 | 0.0841          | 0.1185 |
| 0.2797        | 6.37  | 19000 | 0.0858          | 0.1165 |
| 0.2878        | 6.53  | 19500 | 0.0874          | 0.1161 |
| 0.2974        | 6.7   | 20000 | 0.0835          | 0.1173 |
| 0.3051        | 6.87  | 20500 | 0.0835          | 0.1178 |
| 0.2941        | 7.04  | 21000 | 0.0852          | 0.1155 |
| 0.258         | 7.21  | 21500 | 0.0832          | 0.1132 |
| 0.2778        | 7.37  | 22000 | 0.0829          | 0.1110 |
| 0.2751        | 7.54  | 22500 | 0.0822          | 0.1069 |
| 0.2887        | 7.71  | 23000 | 0.0819          | 0.1103 |
| 0.2509        | 7.88  | 23500 | 0.0787          | 0.1055 |
| 0.2501        | 8.04  | 24000 | 0.0807          | 0.1076 |
| 0.2399        | 8.21  | 24500 | 0.0784          | 0.1052 |
| 0.2539        | 8.38  | 25000 | 0.0772          | 0.1075 |
| 0.248         | 8.55  | 25500 | 0.0772          | 0.1055 |
| 0.2689        | 8.71  | 26000 | 0.0763          | 0.1027 |
| 0.2855        | 8.88  | 26500 | 0.0756          | 0.1035 |
| 0.2421        | 9.05  | 27000 | 0.0771          | 0.0998 |
| 0.2497        | 9.22  | 27500 | 0.0756          | 0.0971 |
| 0.2367        | 9.38  | 28000 | 0.0741          | 0.0974 |
| 0.2473        | 9.55  | 28500 | 0.0739          | 0.0982 |
| 0.2396        | 9.72  | 29000 | 0.0756          | 0.0991 |
| 0.2602        | 9.89  | 29500 | 0.0737          | 0.0975 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.3
- Tokenizers 0.11.0


---
license: apache-2.0
language: fi
metrics:
- wer
- cer
tags:
- generated_from_trainer
- automatic-speech-recognition
- fi
- finnish
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
model-index:
- name: wav2vec2-xlsr-1b-finnish-lm-v2
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 7
      type: mozilla-foundation/common_voice_7_0
      args: fi
    metrics:
       - name: Test WER
         type: wer
         value: 4.19
       - name: Test CER
         type: cer
         value: 0.90

---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xlsr-1b-finnish-lm-v2

This acoustic model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) for Finnish ASR. The model has been fine-tuned with 275.6 hours of Finnish transcribed speech data.
It achieves the following results on the Common Voice 7 test set together with language model (Finnish KenLM):
- Wer: 4.19
- Cer: 0.90

## Model description

TODO

## Intended uses & limitations

TODO

## Training and evaluation data

This model was fine-tuned with 275.6 hours of Finnish transcribed speech data from following datasets:

| Dataset                                                                                | Hours    | % of total hours |
|:------------------------------------------------------------------------------------------------------------------------------|:--------:|:----------------:|
| [Common Voice 7.0 Finnish train+evaluation+other splits](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) | 9.70 h   | 3.52 %           |
| [Finnish parliament session 2](https://b2share.eudat.eu/records/4df422d631544ce682d6af1d4714b2d4)                             | 0.24 h   | 0.09 %           |
| [VoxPopuli Finnish](https://github.com/facebookresearch/voxpopuli)                                                            | 21.97 h  | 7.97 %           |
| [CSS10 Finnish](https://github.com/kyubyong/css10)                                                                            | 10.32 h  | 3.74 %           |
| [Aalto Finnish Parliament ASR Corpus](http://urn.fi/urn:nbn:fi:lb-2021051903)                                                 | 228.00 h | 82.73 %          |
| [Finnish Broadcast Corpus](http://urn.fi/urn:nbn:fi:lb-2016042502)                                                            | 5.37 h   | 1.95 %           |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: [8-bit Adam](https://github.com/facebookresearch/bitsandbytes) with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 10
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.7778        | 0.17  | 500   | 0.2851          | 0.3572 |
| 0.5506        | 0.34  | 1000  | 0.1595          | 0.2130 |
| 0.6569        | 0.5   | 1500  | 0.1458          | 0.2046 |
| 0.5997        | 0.67  | 2000  | 0.1374          | 0.1975 |
| 0.542         | 0.84  | 2500  | 0.1390          | 0.1956 |
| 0.4815        | 1.01  | 3000  | 0.1266          | 0.1813 |
| 0.6982        | 1.17  | 3500  | 0.1441          | 0.1965 |
| 0.4522        | 1.34  | 4000  | 0.1232          | 0.1822 |
| 0.4655        | 1.51  | 4500  | 0.1209          | 0.1702 |
| 0.4069        | 1.68  | 5000  | 0.1149          | 0.1688 |
| 0.4226        | 1.84  | 5500  | 0.1121          | 0.1560 |
| 0.3993        | 2.01  | 6000  | 0.1091          | 0.1557 |
| 0.406         | 2.18  | 6500  | 0.1115          | 0.1553 |
| 0.4098        | 2.35  | 7000  | 0.1144          | 0.1560 |
| 0.3995        | 2.51  | 7500  | 0.1028          | 0.1476 |
| 0.4101        | 2.68  | 8000  | 0.1129          | 0.1511 |
| 0.3636        | 2.85  | 8500  | 0.1025          | 0.1517 |
| 0.3534        | 3.02  | 9000  | 0.1068          | 0.1480 |
| 0.3836        | 3.18  | 9500  | 0.1072          | 0.1459 |
| 0.3531        | 3.35  | 10000 | 0.0928          | 0.1367 |
| 0.3649        | 3.52  | 10500 | 0.1042          | 0.1426 |
| 0.3645        | 3.69  | 11000 | 0.0979          | 0.1433 |
| 0.3685        | 3.85  | 11500 | 0.0947          | 0.1346 |
| 0.3325        | 4.02  | 12000 | 0.0991          | 0.1352 |
| 0.3497        | 4.19  | 12500 | 0.0919          | 0.1358 |
| 0.3303        | 4.36  | 13000 | 0.0888          | 0.1272 |
| 0.3323        | 4.52  | 13500 | 0.0888          | 0.1277 |
| 0.3452        | 4.69  | 14000 | 0.0894          | 0.1279 |
| 0.337         | 4.86  | 14500 | 0.0917          | 0.1289 |
| 0.3114        | 5.03  | 15000 | 0.0942          | 0.1313 |
| 0.3099        | 5.19  | 15500 | 0.0902          | 0.1239 |
| 0.3079        | 5.36  | 16000 | 0.0871          | 0.1256 |
| 0.3293        | 5.53  | 16500 | 0.0861          | 0.1263 |
| 0.3123        | 5.7   | 17000 | 0.0876          | 0.1203 |
| 0.3093        | 5.86  | 17500 | 0.0848          | 0.1226 |
| 0.2903        | 6.03  | 18000 | 0.0914          | 0.1221 |
| 0.297         | 6.2   | 18500 | 0.0841          | 0.1185 |
| 0.2797        | 6.37  | 19000 | 0.0858          | 0.1165 |
| 0.2878        | 6.53  | 19500 | 0.0874          | 0.1161 |
| 0.2974        | 6.7   | 20000 | 0.0835          | 0.1173 |
| 0.3051        | 6.87  | 20500 | 0.0835          | 0.1178 |
| 0.2941        | 7.04  | 21000 | 0.0852          | 0.1155 |
| 0.258         | 7.21  | 21500 | 0.0832          | 0.1132 |
| 0.2778        | 7.37  | 22000 | 0.0829          | 0.1110 |
| 0.2751        | 7.54  | 22500 | 0.0822          | 0.1069 |
| 0.2887        | 7.71  | 23000 | 0.0819          | 0.1103 |
| 0.2509        | 7.88  | 23500 | 0.0787          | 0.1055 |
| 0.2501        | 8.04  | 24000 | 0.0807          | 0.1076 |
| 0.2399        | 8.21  | 24500 | 0.0784          | 0.1052 |
| 0.2539        | 8.38  | 25000 | 0.0772          | 0.1075 |
| 0.248         | 8.55  | 25500 | 0.0772          | 0.1055 |
| 0.2689        | 8.71  | 26000 | 0.0763          | 0.1027 |
| 0.2855        | 8.88  | 26500 | 0.0756          | 0.1035 |
| 0.2421        | 9.05  | 27000 | 0.0771          | 0.0998 |
| 0.2497        | 9.22  | 27500 | 0.0756          | 0.0971 |
| 0.2367        | 9.38  | 28000 | 0.0741          | 0.0974 |
| 0.2473        | 9.55  | 28500 | 0.0739          | 0.0982 |
| 0.2396        | 9.72  | 29000 | 0.0756          | 0.0991 |
| 0.2602        | 9.89  | 29500 | 0.0737          | 0.0975 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.3
- Tokenizers 0.11.0


---
license: apache-2.0
language: fi
metrics:
- wer
- cer
tags:
- generated_from_trainer
- automatic-speech-recognition
- fi
- finnish
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
model-index:
- name: wav2vec2-xlsr-1b-finnish-lm
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 7
      type: mozilla-foundation/common_voice_7_0
      args: fi
    metrics:
       - name: Test WER
         type: wer
         value: 5.65
       - name: Test CER
         type: cer
         value: 1.20

---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xlsr-1b-finnish-lm

**Note**: there is a better V2 version of this model which has been fine-tuned longer with 16 hours of more data: [aapot/wav2vec2-xlsr-1b-finnish-lm-v2](https://huggingface.co/aapot/wav2vec2-xlsr-1b-finnish-lm-v2)

This acoustic model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) for Finnish ASR. The model has been fine-tuned with 259.57 hours of Finnish transcribed speech data.
It achieves the following results on the Common Voice 7 test set together with language model (Finnish KenLM):
- Wer: 5.65
- Cer: 1.20

## Model description

TODO

## Intended uses & limitations

TODO

## Training and evaluation data

This model was fine-tuned with 259.57 hours of Finnish transcribed speech data from following datasets:

| Dataset                                                                                | Hours    | % of total hours |
|:------------------------------------------------------------------------------------------------------------------------------|:--------:|:----------------:|
| [Common Voice 7.0 Finnish train+evaluation+other splits](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) | 9.70 h   | 3.74 %           |
| [Finnish parliament session 2](https://b2share.eudat.eu/records/4df422d631544ce682d6af1d4714b2d4)                             | 0.24 h   | 0.09 %           |
| [VoxPopuli Finnish](https://github.com/facebookresearch/voxpopuli)                                                            | 5.94 h   | 2.29 %           |
| [CSS10 Finnish](https://github.com/kyubyong/css10)                                                                            | 10.32 h  | 3.98 %           |
| [Aalto Finnish Parliament ASR Corpus](http://urn.fi/urn:nbn:fi:lb-2021051903)                                                 | 228.00 h | 87.84 %          |
| [Finnish Broadcast Corpus](http://urn.fi/urn:nbn:fi:lb-2016042502)                                                            | 5.37 h   | 2.07 %           |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: [8-bit Adam](https://github.com/facebookresearch/bitsandbytes) with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 5
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.968         | 0.18  | 500   | 0.4870          | 0.4720 |
| 0.6557        | 0.36  | 1000  | 0.2450          | 0.2931 |
| 0.647         | 0.54  | 1500  | 0.1818          | 0.2255 |
| 0.5297        | 0.72  | 2000  | 0.1698          | 0.2354 |
| 0.5802        | 0.9   | 2500  | 0.1581          | 0.2355 |
| 0.6351        | 1.07  | 3000  | 0.1689          | 0.2336 |
| 0.4626        | 1.25  | 3500  | 0.1719          | 0.3099 |
| 0.4526        | 1.43  | 4000  | 0.1434          | 0.2069 |
| 0.4692        | 1.61  | 4500  | 0.1645          | 0.2192 |
| 0.4584        | 1.79  | 5000  | 0.1483          | 0.1987 |
| 0.4234        | 1.97  | 5500  | 0.1499          | 0.2178 |
| 0.4243        | 2.15  | 6000  | 0.1345          | 0.2070 |
| 0.4108        | 2.33  | 6500  | 0.1383          | 0.1850 |
| 0.4048        | 2.51  | 7000  | 0.1338          | 0.1811 |
| 0.4085        | 2.69  | 7500  | 0.1290          | 0.1780 |
| 0.4026        | 2.87  | 8000  | 0.1239          | 0.1650 |
| 0.4033        | 3.04  | 8500  | 0.1346          | 0.1657 |
| 0.3986        | 3.22  | 9000  | 0.1310          | 0.1850 |
| 0.3867        | 3.4   | 9500  | 0.1273          | 0.1741 |
| 0.3658        | 3.58  | 10000 | 0.1219          | 0.1672 |
| 0.382         | 3.76  | 10500 | 0.1306          | 0.1698 |
| 0.3847        | 3.94  | 11000 | 0.1230          | 0.1577 |
| 0.3691        | 4.12  | 11500 | 0.1310          | 0.1615 |
| 0.3593        | 4.3   | 12000 | 0.1296          | 0.1622 |
| 0.3619        | 4.48  | 12500 | 0.1285          | 0.1601 |
| 0.3361        | 4.66  | 13000 | 0.1261          | 0.1569 |
| 0.3603        | 4.84  | 13500 | 0.1235          | 0.1533 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
language: fi
datasets:
- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Finnish by Aapo Tanskanen
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice fi
      type: common_voice
      args: fi
    metrics:
       - name: Test WER
         type: wer
         value: 32.378771
---

# Wav2Vec2-Large-XLSR-53-Finnish

Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Finnish using the [Common Voice](https://huggingface.co/datasets/common_voice), [CSS10 Finnish](https://www.kaggle.com/bryanpark/finnish-single-speaker-speech-dataset) and [Finnish parliament session 2](https://b2share.eudat.eu/records/4df422d631544ce682d6af1d4714b2d4) datasets.
When using this model, make sure that your speech input is sampled at 16kHz.

## Usage

The model can be used directly (without a language model) as follows:

```python
import librosa
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("common_voice", "fi", split="test[:2%]")

processor = Wav2Vec2Processor.from_pretrained("aapot/wav2vec2-large-xlsr-53-finnish")
model = Wav2Vec2ForCTC.from_pretrained("aapot/wav2vec2-large-xlsr-53-finnish")

resampler = lambda sr, y: librosa.resample(y.numpy().squeeze(), sr, 16_000)

# Preprocessing the datasets.
# We need to read the audio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(sampling_rate, speech_array).squeeze()
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```

## Evaluation

The model can be evaluated as follows on the Finnish test data of Common Voice.

```python
import librosa
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "fi", split="test")
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("aapot/wav2vec2-large-xlsr-53-finnish")
model = Wav2Vec2ForCTC.from_pretrained("aapot/wav2vec2-large-xlsr-53-finnish")
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\‘\”\�\'\...\…\–\é]'
resampler = lambda sr, y: librosa.resample(y.numpy().squeeze(), sr, 16_000)

# Preprocessing the datasets.
# We need to read the audio files as arrays
def speech_file_to_array_fn(batch):
  batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
  speech_array, sampling_rate = torchaudio.load(batch["path"])
  batch["speech"] = resampler(sampling_rate, speech_array).squeeze()
  return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the audio files as arrays
def evaluate(batch):
  inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

  with torch.no_grad():
    logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

  pred_ids = torch.argmax(logits, dim=-1)
  batch["pred_strings"] = processor.batch_decode(pred_ids)
  return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```

**Test Result**: 32.378771 %

## Training

The Common Voice `train`, `validation` and `other` datasets were used for training as well as `CSS10 Finnish` and `Finnish parliament session 2` datasets.

The script used for training can be found from [Google Colab](https://colab.research.google.com/drive/1vnEGC9BnNRmVyIHj-0UsVulh_cUYSGWA?usp=sharing)


Please treat TILDE as a BertLMHeadModel model:
```
from transformers import BertLMHeadModel, BertTokenizerFast

model = BertLMHeadModel.from_pretrained("ielab/TILDE")
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
```

Github: https://github.com/ielab/TILDE


TILDEv2 trained with passages expand with TILDE (m=128)


uniCOIL trained with passages expand with TILDE (m=128)


TILDEv2 trained with passages expand with TILDE (m=200)


uniCOIL trained with passages expand with TILDE (m=200)


---
tags:
- conversational
---

# pocobot


---
thumbnail: https://huggingface.co/front/thumbnails/dialogpt.png
tags:
- conversational
license: mit
---
Chat with the model:
```python
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("r3dhummingbird/DialoGPT-medium-joshua")
model = AutoModelWithLMHead.from_pretrained("r3dhummingbird/DialoGPT-medium-joshua")
# Let's chat for 4 lines
for step in range(4):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input(">> User:") + tokenizer.eos_token, return_tensors='pt')
    # print(new_user_input_ids)
    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) if step > 0 else new_user_input_ids
    # generated a response while limiting the total chat history to 1000 tokens,
    chat_history_ids = model.generate(
        bot_input_ids, max_length=200,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )

    # pretty print last ouput tokens from bot
    print("JoshuaBot: {}".format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))
```


# Titlewave: t5-small

This is one of two models used in the Titlewave project. See https://github.com/tennessejoyce/TitleWave for more information.

This model was fine-tuned on a dataset of Stack Overflow posts, with a ConditionalGeneration head that summarizes the body of a question in order to suggest a title.


---
language: en
license: cc-by-4.0
pipeline_tag: summarization
widget:
- text: "Example question body."
---

# Titlewave: t5-base

## Model description

Titlewave is a Chrome extension that helps you choose better titles for your Stack Overflow questions. See https://github.com/tennessejoyce/TitleWave for more information.
This is one of two NLP models used in the Titlewave project, and its purpose is to suggests a new title based on on the body of the question. The companion model (https://huggingface.co/tennessejoyce/titlewave-bert-base-uncased) classifies whether question will be answered or not just based on the title

## Intended use

Try out different titles for your Stack Overflow post, and see which one gives you the best chance of recieving an answer.
This model can be used in your browser as a Chrome extension by following the installation instructions at https://github.com/tennessejoyce/TitleWave.
Or load it in Python like this (which will automatically download the model to your machine):

```python
>>> from transformers import pipeline
>>> classifier = pipeline('summarization', model='tennessejoyce/titlewave-t5-base')
>>> body = """"Example question body."""
>>> classifier(body)

[{'summary_text': 'Example title suggestion?'}]
```

## Training data

The weights were initialized from the BERT base model (https://huggingface.co/bert-base-uncased), which was trained on BookCorpus and English Wikipedia.
Then the model was fine-tuned on the dataset of previous Stack Overflow post titles (https://archive.org/details/stackexchange).
Specifically I used three years of posts from 2017-2019, filtered out posts which were closed, and selected 25% of the remaining posts at random to use in
the training set. In order to improve the quality of the titles generated, the model was trained only on questions with an accepted answer.

## Evaluation

See https://github.com/tennessejoyce/TitleWave/blob/master/model_training/test_summarizer.ipynb for the performance of the title generation model on the test set.


---
language: en
license: cc-by-4.0
widget:
- text: "[Gmail API] How can I extract plain text from an email sent to me?"
---

# Titlewave: bert-base-uncased

## Model description

Titlewave is a Chrome extension that helps you choose better titles for your Stack Overflow questions. See the [github repository](https://github.com/tennessejoyce/TitleWave) for more information.
This is one of two NLP models used in the Titlewave project, and its purpose is to classify whether question will be answered or not just based on the title. The [companion model](https://huggingface.co/tennessejoyce/titlewave-t5-small) suggests a new title based on on the body of the question.

## Intended use

Try out different titles for your Stack Overflow post, and see which one gives you the best chance of receiving an answer.
You can use the model through the API on this page (hosted by HuggingFace) or install the Chrome extension by following the instructions on the [github repository](https://github.com/tennessejoyce/TitleWave), which integrates the tool directly into the Stack Overflow website.

You can also run the model locally in Python like this (which automatically downloads the model to your machine):

```python
>>> from transformers import pipeline
>>> classifier = pipeline('sentiment-analysis', model='tennessejoyce/titlewave-bert-base-uncased')
>>> classifier('[Gmail API] How can I extract plain text from an email sent to me?')

[{'label': 'Answered', 'score': 0.8053370714187622}]
```
The 'score' in the output represents the probability of getting an answer with this title: 80.5%.

## Training data

The weights were initialized from the [BERT base model](https://huggingface.co/bert-base-uncased), which was trained on BookCorpus and English Wikipedia.
Then the model was fine-tuned on the dataset of previous Stack Overflow post titles, which is publicly available [here](https://archive.org/details/stackexchange).
Specifically I used three years of posts from 2017-2019, filtered out posts which were closed (e.g., duplicates, off-topic), and selected 5% of the remaining posts at random to use in the training set, and the same amount for validation and test sets (278,155 posts each).

## Training procedure

The model was fine-tuned for two epochs with a batch size of 32 (17,384 steps total) using 16-bit mixed precision.
After some hyperparameter tuning, I found that the following two-phase training procedure yields the best performance (ROC-AUC score) on the validation set:
* In the first epoch, all layers were frozen except for the last two (pooling layer and classification layer) and a learning rate of 3e-4 was used.
* In the second epoch all layers were unfrozen, and the learning rate was decreased by a factor of 10 to 3e-5.

Otherwise, all parameters we set to the defaults listed [here](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments),
including the AdamW optimizer and a linearly decreasing learning schedule (both of which were reset between the two epochs). See the [github repository](https://github.com/tennessejoyce/TitleWave) for the scripts that were used to train the model.

## Evaluation

See [this notebook](https://github.com/tennessejoyce/TitleWave/blob/master/model_training/test_classifier.ipynb) for the performance of the title classification model on the test set.


---
language: en
tags:
- tapas
license: apache-2.0
datasets:
- wikisql
---

# TAPAS base model fine-tuned on WikiSQL (in a supervised fashion)

his model has 2 versions which can be used. The default version corresponds to the `tapas_wikisql_sqa_inter_masklm_base_reset` checkpoint of the [original Github repository](https://github.com/google-research/tapas).
This model was pre-trained on MLM and an additional step which the authors call intermediate pre-training, and then fine-tuned in a chain on [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253), and [WikiSQL](https://github.com/salesforce/WikiSQL). It uses relative position embeddings (i.e. resetting the position index at every cell of the table).

The other (non-default) version which can be used is:
- `no_reset`, which corresponds to `tapas_wikisql_sqa_inter_masklm_base` (intermediate pre-training, absolute position embeddings).

Disclaimer: The team releasing TAPAS did not write a model card for this model so this model card has been written by
the Hugging Face team and contributors.

## Model description

TAPAS is a BERT-like transformers model pretrained on a large corpus of English data from Wikipedia in a self-supervised fashion.
This means it was pretrained on the raw tables and associated texts only, with no humans labelling them in any way (which is why it
can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
was pretrained with two objectives:

- Masked language modeling (MLM): taking a (flattened) table and associated context, the model randomly masks 15% of the words in
  the input, then runs the entire (partially masked) sequence through the model. The model then has to predict the masked words.
  This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other,
  or from autoregressive models like GPT which internally mask the future tokens. It allows the model to learn a bidirectional
  representation of a table and associated text.
- Intermediate pre-training: to encourage numerical reasoning on tables, the authors additionally pre-trained the model by creating
  a balanced dataset of millions of syntactically created training examples. Here, the model must predict (classify) whether a sentence
  is supported or refuted by the contents of a table. The training examples are created based on synthetic as well as counterfactual statements.

This way, the model learns an inner representation of the English language used in tables and associated texts, which can then be used
to extract features useful for downstream tasks such as answering questions about a table, or determining whether a sentence is entailed
or refuted by the contents of a table. Fine-tuning is done by adding a cell selection head and aggregation head on top of the pre-trained model, and then jointly train these randomly initialized classification heads with the base model on SQA and WikiSQL.

## Intended uses & limitations

You can use this model for answering questions related to a table.

For code examples, we refer to the documentation of TAPAS on the HuggingFace website.

## Training procedure

### Preprocessing

The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
then of the form:

```
[CLS] Question [SEP] Flattened table [SEP]
```

The authors did first convert the WikiSQL dataset into the format of SQA using automatic conversion scripts.

### Fine-tuning

The model was fine-tuned on 32 Cloud TPU v3 cores for 50,000 steps with maximum sequence length 512 and batch size of 512.
In this setup, fine-tuning takes around 10 hours. The optimizer used is Adam with a learning rate of 6.17164e-5, and a warmup
ratio of 0.1424. See the [paper](https://arxiv.org/abs/2004.02349) for more details (tables 11 and 12).

### BibTeX entry and citation info

```bibtex
@misc{herzig2020tapas,
      title={TAPAS: Weakly Supervised Table Parsing via Pre-training},
      author={Jonathan Herzig and Paweł Krzysztof Nowak and Thomas Müller and Francesco Piccinno and Julian Martin Eisenschlos},
      year={2020},
      eprint={2004.02349},
      archivePrefix={arXiv},
      primaryClass={cs.IR}
}
```

```bibtex
@misc{eisenschlos2020understanding,
      title={Understanding tables with intermediate pre-training},
      author={Julian Martin Eisenschlos and Syrine Krichene and Thomas Müller},
      year={2020},
      eprint={2010.00571},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

```bibtex
@article{DBLP:journals/corr/abs-1709-00103,
  author    = {Victor Zhong and
               Caiming Xiong and
               Richard Socher},
  title     = {Seq2SQL: Generating Structured Queries from Natural Language using
               Reinforcement Learning},
  journal   = {CoRR},
  volume    = {abs/1709.00103},
  year      = {2017},
  url       = {http://arxiv.org/abs/1709.00103},
  archivePrefix = {arXiv},
  eprint    = {1709.00103},
  timestamp = {Mon, 13 Aug 2018 16:48:41 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1709-00103.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```


---
language: en
tags:
-   multiberts
-   multiberts-seed_24
license: apache-2.0
---

# MultiBERTs - Seed 24

MultiBERTs is a collection of checkpoints and a statistical library to support
robust research on BERT. We provide 25 BERT-base models trained with
similar hyper-parameters as
[the original BERT model](https://github.com/google-research/bert) but
with different random seeds, which causes variations in the initial weights and order of
training instances. The aim is to distinguish findings that apply to a specific
artifact (i.e., a particular instance of the model) from those that apply to the
more general procedure.

We also provide 140 intermediate checkpoints captured
during the course of pre-training (we saved 28 checkpoints for the first 5 runs).

The models were originally released through
[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our
paper
[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).

This is model #24.

## Model Description

This model is a reproduction of
[BERT-base uncased](https://github.com/google-research/bert), for English: it
is a Transformers model pretrained on a large corpus of English data, using the
Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)
objectives.

The intended uses, limitations, training data and training procedure are similar
to [BERT-base uncased](https://github.com/google-research/bert). Two major
differences with the original model:

*   We pre-trained the MultiBERTs models for 2 million steps using sequence
    length 512 (instead of 1 million steps using sequence length 128 then 512).
*   We used an alternative version of Wikipedia and Books Corpus, initially
    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).

This is a best-effort reproduction, and so it is probable that differences with
the original model have gone unnoticed. The performance of MultiBERTs on GLUE is oftentimes comparable to that of original
BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).
See our [technical report](https://arxiv.org/abs/2106.16163) for more details.

### How to use

Using code from
[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on
Tensorflow:

```
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_24')
model = TFBertModel.from_pretrained("google/multiberts-seed_24")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

PyTorch version:

```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_24')
model = BertModel.from_pretrained("google/multiberts-seed_24")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

## Citation info

```bibtex
@article{sellam2021multiberts,
  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},
  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},
  journal={arXiv preprint arXiv:2106.16163},
  year={2021}
}
```


---
thumbnail: https://huggingface.co/front/thumbnails/google.png

license: apache-2.0
---

BERT Miniatures
===

This is the set of 24 BERT models referenced in [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://arxiv.org/abs/1908.08962) (English only, uncased, trained with WordPiece masking).

We have shown that the standard BERT recipe (including model architecture and training objective) is effective on a wide range of model sizes, beyond BERT-Base and BERT-Large. The smaller BERT models are intended for environments with restricted computational resources. They can be fine-tuned in the same manner as the original BERT models. However, they are most effective in the context of knowledge distillation, where the fine-tuning labels are produced by a larger and more accurate teacher.

Our goal is to enable research in institutions with fewer computational resources and encourage the community to seek directions of innovation alternative to increasing model capacity.

You can download the 24 BERT miniatures either from the [official BERT Github page](https://github.com/google-research/bert/), or via HuggingFace from the links below:

|   |H=128|H=256|H=512|H=768|
|---|:---:|:---:|:---:|:---:|
| **L=2**  |[**2/128 (BERT-Tiny)**][2_128]|[2/256][2_256]|[2/512][2_512]|[2/768][2_768]|
| **L=4**  |[4/128][4_128]|[**4/256 (BERT-Mini)**][4_256]|[**4/512 (BERT-Small)**][4_512]|[4/768][4_768]|
| **L=6**  |[6/128][6_128]|[6/256][6_256]|[6/512][6_512]|[6/768][6_768]|
| **L=8**  |[8/128][8_128]|[8/256][8_256]|[**8/512 (BERT-Medium)**][8_512]|[8/768][8_768]|
| **L=10** |[10/128][10_128]|[10/256][10_256]|[10/512][10_512]|[10/768][10_768]|
| **L=12** |[12/128][12_128]|[12/256][12_256]|[12/512][12_512]|[**12/768 (BERT-Base)**][12_768]|

Note that the BERT-Base model in this release is included for completeness only; it was re-trained under the same regime as the original model.

Here are the corresponding GLUE scores on the test set:

|Model|Score|CoLA|SST-2|MRPC|STS-B|QQP|MNLI-m|MNLI-mm|QNLI(v2)|RTE|WNLI|AX|
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|BERT-Tiny|64.2|0.0|83.2|81.1/71.1|74.3/73.6|62.2/83.4|70.2|70.3|81.5|57.2|62.3|21.0|
|BERT-Mini|65.8|0.0|85.9|81.1/71.8|75.4/73.3|66.4/86.2|74.8|74.3|84.1|57.9|62.3|26.1|
|BERT-Small|71.2|27.8|89.7|83.4/76.2|78.8/77.0|68.1/87.0|77.6|77.0|86.4|61.8|62.3|28.6|
|BERT-Medium|73.5|38.0|89.6|86.6/81.6|80.4/78.4|69.6/87.9|80.0|79.1|87.7|62.2|62.3|30.5|

For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained for 4 epochs:
- batch sizes: 8, 16, 32, 64, 128
- learning rates: 3e-4, 1e-4, 5e-5, 3e-5

If you use these models, please cite the following paper:

```
@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}
```

[2_128]: https://huggingface.co/google/bert_uncased_L-2_H-128_A-2
[2_256]: https://huggingface.co/google/bert_uncased_L-2_H-256_A-4
[2_512]: https://huggingface.co/google/bert_uncased_L-2_H-512_A-8
[2_768]: https://huggingface.co/google/bert_uncased_L-2_H-768_A-12
[4_128]: https://huggingface.co/google/bert_uncased_L-4_H-128_A-2
[4_256]: https://huggingface.co/google/bert_uncased_L-4_H-256_A-4
[4_512]: https://huggingface.co/google/bert_uncased_L-4_H-512_A-8
[4_768]: https://huggingface.co/google/bert_uncased_L-4_H-768_A-12
[6_128]: https://huggingface.co/google/bert_uncased_L-6_H-128_A-2
[6_256]: https://huggingface.co/google/bert_uncased_L-6_H-256_A-4
[6_512]: https://huggingface.co/google/bert_uncased_L-6_H-512_A-8
[6_768]: https://huggingface.co/google/bert_uncased_L-6_H-768_A-12
[8_128]: https://huggingface.co/google/bert_uncased_L-8_H-128_A-2
[8_256]: https://huggingface.co/google/bert_uncased_L-8_H-256_A-4
[8_512]: https://huggingface.co/google/bert_uncased_L-8_H-512_A-8
[8_768]: https://huggingface.co/google/bert_uncased_L-8_H-768_A-12
[10_128]: https://huggingface.co/google/bert_uncased_L-10_H-128_A-2
[10_256]: https://huggingface.co/google/bert_uncased_L-10_H-256_A-4
[10_512]: https://huggingface.co/google/bert_uncased_L-10_H-512_A-8
[10_768]: https://huggingface.co/google/bert_uncased_L-10_H-768_A-12
[12_128]: https://huggingface.co/google/bert_uncased_L-12_H-128_A-2
[12_256]: https://huggingface.co/google/bert_uncased_L-12_H-256_A-4
[12_512]: https://huggingface.co/google/bert_uncased_L-12_H-512_A-8
[12_768]: https://huggingface.co/google/bert_uncased_L-12_H-768_A-12


---
language: en
tags:
-   multiberts
-   multiberts-seed_6
license: apache-2.0
---

# MultiBERTs - Seed 6

MultiBERTs is a collection of checkpoints and a statistical library to support
robust research on BERT. We provide 25 BERT-base models trained with
similar hyper-parameters as
[the original BERT model](https://github.com/google-research/bert) but
with different random seeds, which causes variations in the initial weights and order of
training instances. The aim is to distinguish findings that apply to a specific
artifact (i.e., a particular instance of the model) from those that apply to the
more general procedure.

We also provide 140 intermediate checkpoints captured
during the course of pre-training (we saved 28 checkpoints for the first 5 runs).

The models were originally released through
[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our
paper
[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).

This is model #6.

## Model Description

This model is a reproduction of
[BERT-base uncased](https://github.com/google-research/bert), for English: it
is a Transformers model pretrained on a large corpus of English data, using the
Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)
objectives.

The intended uses, limitations, training data and training procedure are similar
to [BERT-base uncased](https://github.com/google-research/bert). Two major
differences with the original model:

*   We pre-trained the MultiBERTs models for 2 million steps using sequence
    length 512 (instead of 1 million steps using sequence length 128 then 512).
*   We used an alternative version of Wikipedia and Books Corpus, initially
    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).

This is a best-effort reproduction, and so it is probable that differences with
the original model have gone unnoticed. The performance of MultiBERTs on GLUE is oftentimes comparable to that of original
BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).
See our [technical report](https://arxiv.org/abs/2106.16163) for more details.

### How to use

Using code from
[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on
Tensorflow:

```
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_6')
model = TFBertModel.from_pretrained("google/multiberts-seed_6")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

PyTorch version:

```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_6')
model = BertModel.from_pretrained("google/multiberts-seed_6")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

## Citation info

```bibtex
@article{sellam2021multiberts,
  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},
  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},
  journal={arXiv preprint arXiv:2106.16163},
  year={2021}
}
```


---
language: en
tags:
-   multiberts
-   multiberts-seed_0
-   multiberts-seed_0-step_80k
license: apache-2.0
---

# MultiBERTs, Intermediate Checkpoint - Seed 0, Step 80k

MultiBERTs is a collection of checkpoints and a statistical library to support
robust research on BERT. We provide 25 BERT-base models trained with
similar hyper-parameters as
[the original BERT model](https://github.com/google-research/bert) but
with different random seeds, which causes variations in the initial weights and order of
training instances. The aim is to distinguish findings that apply to a specific
artifact (i.e., a particular instance of the model) from those that apply to the
more general procedure.

We also provide 140 intermediate checkpoints captured
during the course of pre-training (we saved 28 checkpoints for the first 5 runs).

The models were originally released through
[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our
paper
[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).

This is model #0, captured at step 80k (max: 2000k, i.e., 2M steps).

## Model Description

This model was captured during a reproduction of
[BERT-base uncased](https://github.com/google-research/bert), for English: it
is a Transformers model pretrained on a large corpus of English data, using the
Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)
objectives.

The intended uses, limitations, training data and training procedure for the fully trained model are similar
to [BERT-base uncased](https://github.com/google-research/bert). Two major
differences with the original model:

*   We pre-trained the MultiBERTs models for 2 million steps using sequence
    length 512 (instead of 1 million steps using sequence length 128 then 512).
*   We used an alternative version of Wikipedia and Books Corpus, initially
    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).

This is a best-effort reproduction, and so it is probable that differences with
the original model have gone unnoticed. The performance of MultiBERTs on GLUE after full training is oftentimes comparable to that of original
BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).
See our [technical report](https://arxiv.org/abs/2106.16163) for more details.

### How to use

Using code from
[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on
Tensorflow:

```
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_0-step_80k')
model = TFBertModel.from_pretrained("google/multiberts-seed_0-step_80k")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

PyTorch version:

```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_0-step_80k')
model = BertModel.from_pretrained("google/multiberts-seed_0-step_80k")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

## Citation info

```bibtex
@article{sellam2021multiberts,
  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},
  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},
  journal={arXiv preprint arXiv:2106.16163},
  year={2021}
}
```


---
language: en
tags:
-   multiberts
-   multiberts-seed_8
license: apache-2.0
---

# MultiBERTs - Seed 8

MultiBERTs is a collection of checkpoints and a statistical library to support
robust research on BERT. We provide 25 BERT-base models trained with
similar hyper-parameters as
[the original BERT model](https://github.com/google-research/bert) but
with different random seeds, which causes variations in the initial weights and order of
training instances. The aim is to distinguish findings that apply to a specific
artifact (i.e., a particular instance of the model) from those that apply to the
more general procedure.

We also provide 140 intermediate checkpoints captured
during the course of pre-training (we saved 28 checkpoints for the first 5 runs).

The models were originally released through
[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our
paper
[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).

This is model #8.

## Model Description

This model is a reproduction of
[BERT-base uncased](https://github.com/google-research/bert), for English: it
is a Transformers model pretrained on a large corpus of English data, using the
Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)
objectives.

The intended uses, limitations, training data and training procedure are similar
to [BERT-base uncased](https://github.com/google-research/bert). Two major
differences with the original model:

*   We pre-trained the MultiBERTs models for 2 million steps using sequence
    length 512 (instead of 1 million steps using sequence length 128 then 512).
*   We used an alternative version of Wikipedia and Books Corpus, initially
    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).

This is a best-effort reproduction, and so it is probable that differences with
the original model have gone unnoticed. The performance of MultiBERTs on GLUE is oftentimes comparable to that of original
BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).
See our [technical report](https://arxiv.org/abs/2106.16163) for more details.

### How to use

Using code from
[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on
Tensorflow:

```
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_8')
model = TFBertModel.from_pretrained("google/multiberts-seed_8")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

PyTorch version:

```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_8')
model = BertModel.from_pretrained("google/multiberts-seed_8")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

## Citation info

```bibtex
@article{sellam2021multiberts,
  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},
  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},
  journal={arXiv preprint arXiv:2106.16163},
  year={2021}
}
```


---
language: en
tags:
-   multiberts
-   multiberts-seed_0
license: apache-2.0
---

# MultiBERTs - Seed 0

MultiBERTs is a collection of checkpoints and a statistical library to support
robust research on BERT. We provide 25 BERT-base models trained with
similar hyper-parameters as
[the original BERT model](https://github.com/google-research/bert) but
with different random seeds, which causes variations in the initial weights and order of
training instances. The aim is to distinguish findings that apply to a specific
artifact (i.e., a particular instance of the model) from those that apply to the
more general procedure.

We also provide 140 intermediate checkpoints captured
during the course of pre-training (we saved 28 checkpoints for the first 5 runs).

The models were originally released through
[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our
paper
[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).

This is model #0.

## Model Description

This model is a reproduction of
[BERT-base uncased](https://github.com/google-research/bert), for English: it
is a Transformers model pretrained on a large corpus of English data, using the
Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)
objectives.

The intended uses, limitations, training data and training procedure are similar
to [BERT-base uncased](https://github.com/google-research/bert). Two major
differences with the original model:

*   We pre-trained the MultiBERTs models for 2 million steps using sequence
    length 512 (instead of 1 million steps using sequence length 128 then 512).
*   We used an alternative version of Wikipedia and Books Corpus, initially
    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).

This is a best-effort reproduction, and so it is probable that differences with
the original model have gone unnoticed. The performance of MultiBERTs on GLUE is oftentimes comparable to that of original
BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).
See our [technical report](https://arxiv.org/abs/2106.16163) for more details.

### How to use

Using code from
[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on
Tensorflow:

```
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_0')
model = TFBertModel.from_pretrained("google/multiberts-seed_0")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

PyTorch version:

```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_0')
model = BertModel.from_pretrained("google/multiberts-seed_0")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

## Citation info

```bibtex
@article{sellam2021multiberts,
  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},
  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},
  journal={arXiv preprint arXiv:2106.16163},
  year={2021}
}
```


---
language: en
tags:
-   multiberts
-   multiberts-seed_0
-   multiberts-seed_0-step_300k
license: apache-2.0
---

# MultiBERTs, Intermediate Checkpoint - Seed 0, Step 300k

MultiBERTs is a collection of checkpoints and a statistical library to support
robust research on BERT. We provide 25 BERT-base models trained with
similar hyper-parameters as
[the original BERT model](https://github.com/google-research/bert) but
with different random seeds, which causes variations in the initial weights and order of
training instances. The aim is to distinguish findings that apply to a specific
artifact (i.e., a particular instance of the model) from those that apply to the
more general procedure.

We also provide 140 intermediate checkpoints captured
during the course of pre-training (we saved 28 checkpoints for the first 5 runs).

The models were originally released through
[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our
paper
[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).

This is model #0, captured at step 300k (max: 2000k, i.e., 2M steps).

## Model Description

This model was captured during a reproduction of
[BERT-base uncased](https://github.com/google-research/bert), for English: it
is a Transformers model pretrained on a large corpus of English data, using the
Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)
objectives.

The intended uses, limitations, training data and training procedure for the fully trained model are similar
to [BERT-base uncased](https://github.com/google-research/bert). Two major
differences with the original model:

*   We pre-trained the MultiBERTs models for 2 million steps using sequence
    length 512 (instead of 1 million steps using sequence length 128 then 512).
*   We used an alternative version of Wikipedia and Books Corpus, initially
    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).

This is a best-effort reproduction, and so it is probable that differences with
the original model have gone unnoticed. The performance of MultiBERTs on GLUE after full training is oftentimes comparable to that of original
BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).
See our [technical report](https://arxiv.org/abs/2106.16163) for more details.

### How to use

Using code from
[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on
Tensorflow:

```
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_0-step_300k')
model = TFBertModel.from_pretrained("google/multiberts-seed_0-step_300k")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

PyTorch version:

```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_0-step_300k')
model = BertModel.from_pretrained("google/multiberts-seed_0-step_300k")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

## Citation info

```bibtex
@article{sellam2021multiberts,
  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},
  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},
  journal={arXiv preprint arXiv:2106.16163},
  year={2021}
}
```


---
language: en
tags:
-   multiberts
-   multiberts-seed_3
-   multiberts-seed_3-step_1700k
license: apache-2.0
---

# MultiBERTs, Intermediate Checkpoint - Seed 3, Step 1700k

MultiBERTs is a collection of checkpoints and a statistical library to support
robust research on BERT. We provide 25 BERT-base models trained with
similar hyper-parameters as
[the original BERT model](https://github.com/google-research/bert) but
with different random seeds, which causes variations in the initial weights and order of
training instances. The aim is to distinguish findings that apply to a specific
artifact (i.e., a particular instance of the model) from those that apply to the
more general procedure.

We also provide 140 intermediate checkpoints captured
during the course of pre-training (we saved 28 checkpoints for the first 5 runs).

The models were originally released through
[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our
paper
[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).

This is model #3, captured at step 1700k (max: 2000k, i.e., 2M steps).

## Model Description

This model was captured during a reproduction of
[BERT-base uncased](https://github.com/google-research/bert), for English: it
is a Transformers model pretrained on a large corpus of English data, using the
Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)
objectives.

The intended uses, limitations, training data and training procedure for the fully trained model are similar
to [BERT-base uncased](https://github.com/google-research/bert). Two major
differences with the original model:

*   We pre-trained the MultiBERTs models for 2 million steps using sequence
    length 512 (instead of 1 million steps using sequence length 128 then 512).
*   We used an alternative version of Wikipedia and Books Corpus, initially
    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).

This is a best-effort reproduction, and so it is probable that differences with
the original model have gone unnoticed. The performance of MultiBERTs on GLUE after full training is oftentimes comparable to that of original
BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).
See our [technical report](https://arxiv.org/abs/2106.16163) for more details.

### How to use

Using code from
[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on
Tensorflow:

```
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_3-step_1700k')
model = TFBertModel.from_pretrained("google/multiberts-seed_3-step_1700k")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

PyTorch version:

```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_3-step_1700k')
model = BertModel.from_pretrained("google/multiberts-seed_3-step_1700k")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

## Citation info

```bibtex
@article{sellam2021multiberts,
  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},
  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},
  journal={arXiv preprint arXiv:2106.16163},
  year={2021}
}
```


---
language: en
tags:
- fnet
license: apache-2.0
datasets:
- c4
---

# FNet large model

Pretrained model on English language using a masked language modeling (MLM) and next sentence prediction (NSP) objective. It was
introduced in [this paper](https://arxiv.org/abs/2105.03824) and first released in [this repository](https://github.com/google-research/google-research/tree/master/f_net).
This model is cased: it makes a difference between english and English. The model achieves 0.58 accuracy on MLM objective and 0.80 on NSP objective.

Disclaimer: This model card has been written by [gchhablani](https://huggingface.co/gchhablani).

## Model description

FNet is a transformers model with attention replaced with fourier transforms. Hence, the inputs do not contain an `attention_mask`. It is pretrained on a large corpus of
English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling
them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and
labels from those texts. More precisely, it was pretrained with two objectives:

- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
  recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
  GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
  sentence.
- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
  they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
  predict if the two sentences were following each other or not.

This way, the model learns an inner representation of the English language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the FNet model as inputs.

This model has the following configuration:
- 24-layer
- 1024 hidden dimension

## Intended uses & limitations

You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=fnet) to look for
fine-tuned versions on a task that interests you.

Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.

### How to use

You can use this model directly with a pipeline for masked language modeling:

**Note: The mask filling pipeline doesn't work exactly as the original model performs masking after converting to tokens. In masking pipeline an additional space is added after the [MASK].**

```python
>>> from transformers import FNetForMaskedLM, FNetTokenizer, pipeline
>>> tokenizer = FNetTokenizer.from_pretrained("google/fnet-large")
>>> model = FNetForMaskedLM.from_pretrained("google/fnet-large")
>>> unmasker = pipeline('fill-mask', model=model, tokenizer=tokenizer)
>>> unmasker("Hello I'm a [MASK] model.")

[
    {"sequence": "hello i'm a. model.", "score": 0.12840192019939423, "token": 16678, "token_str": "."},
    {"sequence": "hello i'm a a model.", "score": 0.07460460811853409, "token": 8, "token_str": "a"},
    {"sequence": "hello i'm a, model.", "score": 0.05011311173439026, "token": 16680, "token_str": ","},
    {"sequence": "hello i'm a and model.", "score": 0.047409165650606155, "token": 36, "token_str": "and"},
    {"sequence": "hello i'm a the model.", "score": 0.0269990973174572, "token": 13, "token_str": "the"},
]

```

Here is how to use this model to get the features of a given text in PyTorch:

**Note: You must specify the maximum sequence length to be 512 and truncate/pad to the same length because the original model has no attention mask and considers all the hidden states during forward pass.**

```python
from transformers import FNetTokenizer, FNetModel
tokenizer = FNetTokenizer.from_pretrained("google/fnet-large")
model = FNetModel.from_pretrained("google/fnet-large")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt', padding='max_length', truncation=True, max_length=512)
output = model(**encoded_input)
```

### Limitations and bias

Even if the training data used for this model could be characterized as fairly neutral, this model can have biased predictions. However, the model's MLM accuracy may also affect answers. Given below are some example where gender-bias could be expected:

```python
>>> from transformers import FNetForMaskedLM, FNetTokenizer, pipeline
>>> tokenizer = FNetTokenizer.from_pretrained("google/fnet-large")
>>> model = FNetForMaskedLM.from_pretrained("google/fnet-large")
>>> unmasker = pipeline('fill-mask', model=model, tokenizer=tokenizer)
>>> unmasker("The man worked as a [MASK].")

[
    {"sequence": "the man worked as a a.", "score": 0.39862048625946045, "token": 8, "token_str": "a"},
    {"sequence": "the man worked as a the.", "score": 0.20786496996879578, "token": 13, "token_str": "the"},
    {"sequence": "the man worked as a as.", "score": 0.012523212470114231, "token": 106, "token_str": "as"},
    {"sequence": "the man worked as a an.", "score": 0.010838045738637447, "token": 102, "token_str": "an"},
    {"sequence": "the man worked as a and.", "score": 0.006571347825229168, "token": 36, "token_str": "and"},
]

>>> unmasker("The woman worked as a [MASK].")

[
    {"sequence": "the woman worked as a the.", "score": 0.3320266902446747, "token": 13, "token_str": "the"},
    {"sequence": "the woman worked as a a.", "score": 0.2591220438480377, "token": 8, "token_str": "a"},
    {"sequence": "the woman worked as a as.", "score": 0.011250585317611694, "token": 106, "token_str": "as"},
    {"sequence": "the woman worked as a an.", "score": 0.010153685696423054, "token": 102, "token_str": "an"},
    {"sequence": "the woman worked as a and.", "score": 0.010126154869794846, "token": 36, "token_str": "and"},
]
```

This bias will also affect all fine-tuned versions of this model.

## Training data

The FNet model was pretrained on [C4](https://huggingface.co/datasets/c4), a cleaned version of the Common Crawl dataset.

## Training procedure

### Preprocessing

The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 32,000. The inputs of the model are
then of the form:

```
[CLS] Sentence A [SEP] Sentence B [SEP]
```

With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus and in
the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
"sentences" has a combined length of less than 512 tokens.

The details of the masking procedure for each sentence are the following:
- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.

### Pretraining

The model was trained on 4 cloud TPUs in Pod configuration (16 TPU chips total) for one million steps with a batch size
of 256. The sequence length was limited to 512 tokens. The optimizer
used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
learning rate warmup for 10,000 steps and linear decay of the learning rate after.

## Evaluation results

When fine-tuned on downstream tasks, this model achieves the following results:

Glue test results:

| Task | MNLI-(m/mm) | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  | Average |
|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
|      | 78/76   | 85 | 85 | 94  | 78 | 84  | 88 | 69| 81.9    |

### BibTeX entry and citation info

```bibtex
@article{DBLP:journals/corr/abs-2105-03824,
  author    = {James Lee{-}Thorp and
               Joshua Ainslie and
               Ilya Eckstein and
               Santiago Onta{\~{n}}{\'{o}}n},
  title     = {FNet: Mixing Tokens with Fourier Transforms},
  journal   = {CoRR},
  volume    = {abs/2105.03824},
  year      = {2021},
  url       = {https://arxiv.org/abs/2105.03824},
  archivePrefix = {arXiv},
  eprint    = {2105.03824},
  timestamp = {Fri, 14 May 2021 12:13:30 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2105-03824.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```

## Contributions
Thanks to [@gchhablani](https://huggingface.co/gchhablani) for adding this model.