- 🤗 [`GroNLP/bert-base-dutch-cased-upos-alpino`](https://huggingface.co/GroNLP/bert-base-dutch-cased-upos-alpino) (Dutch)
- 🤗 [`GroNLP/bert-base-dutch-cased-upos-alpino-gronings`](https://huggingface.co/GroNLP/bert-base-dutch-cased-upos-alpino-gronings) (Gronings)
- 🤗 [`GroNLP/bert-base-dutch-cased-upos-alpino-frisian`](https://huggingface.co/GroNLP/bert-base-dutch-cased-upos-alpino-frisian) (West Frisian)


model name: bert-base-dutch-cased-frisian


model author: GroNLP


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, fy, arxiv:2105.02855, transformers, BERTje, autonlp_compatible, infinity_compatible


language: nl
tags:


- BERTje
- pos


Wietse de Vries • Martijn Bartelds • Malvina Nissim • Martijn Wieling


# Adapting Monolingual Models: Data can be Scarce when Language Similarity is High


This model is part of this paper + code:


- 📝 [Paper](https://arxiv.org/abs/2105.02855)
- 💻 [Code](https://github.com/wietsedv/low-resource-adapt)


## Models


The best fine-tuned models for Gronings and West Frisian are available on the HuggingFace model hub:


### Lexical layers


These models are identical to [BERTje](https://github.com/wietsedv/bertje), but with different lexical layers (`bert.embeddings.word_embeddings`).


- 🤗 [`GroNLP/bert-base-dutch-cased`](https://huggingface.co/GroNLP/bert-base-dutch-cased) (Dutch; source language)
- 🤗 [`GroNLP/bert-base-dutch-cased-gronings`](https://huggingface.co/GroNLP/bert-base-dutch-cased-gronings) (Gronings)
- 🤗 [`GroNLP/bert-base-dutch-cased-frisian`](https://huggingface.co/GroNLP/bert-base-dutch-cased-frisian) (West Frisian)


### POS tagging


These models share the same fine-tuned Transformer layers + classification head, but with the retrained lexical layers from the models above.


model name: bert-base-dutch-cased-upos-alpino


model author: GroNLP


task: token-classification


tags: pytorch, tf, jax, bert, token-classification, nl, arxiv:2105.02855, transformers, BERTje, pos, autonlp_compatible, infinity_compatible


language: gos
tags:


- BERTje
- pos


model name: bert-base-dutch-cased-upos-alpino-gronings


tags: pytorch, tf, jax, bert, token-classification, gos, arxiv:2105.02855, transformers, BERTje, pos, autonlp_compatible, infinity_compatible


- BERTje


model name: bert-base-dutch-cased-gronings


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, gos, arxiv:2105.02855, transformers, BERTje, autonlp_compatible, infinity_compatible


language: fy
tags:


## Models


### Lexical layers


### POS tagging


- 🤗 [`GroNLP/bert-base-dutch-cased-upos-alpino`](https://huggingface.co/GroNLP/bert-base-dutch-cased-upos-alpino) (Dutch)
- 🤗 [`GroNLP/bert-base-dutch-cased-upos-alpino-gronings`](https://huggingface.co/GroNLP/bert-base-dutch-cased-upos-alpino-gronings) (Gronings)
- 🤗 [`GroNLP/bert-base-dutch-cased-upos-alpino-frisian`](https://huggingface.co/GroNLP/bert-base-dutch-cased-upos-alpino-frisian) (West Frisian)


model name: bert-base-dutch-cased-upos-alpino-frisian


model author: GroNLP


task: token-classification


tags: pytorch, tf, jax, bert, token-classification, fy, arxiv:2105.02855, transformers, BERTje, pos, autonlp_compatible, infinity_compatible


language: it
tags:


- adaption
- recycled
- gpt2-small
pipeline_tag: text-generation


# GPT-2 recycled for Italian (small, adapted lexical embeddings)


[Wietse de Vries](https://www.semanticscholar.org/author/Wietse-de-Vries/144611157) •
[Malvina Nissim](https://www.semanticscholar.org/author/M.-Nissim/2742475)


## Model description


This model is based on the small OpenAI GPT-2 ([`gpt2`](https://huggingface.co/gpt2)) model.


The Transformer layer weights in this model are identical to the original English, model but the lexical layer has been retrained for an Italian vocabulary.


For details, check out our paper on [arXiv](https://arxiv.org/abs/2012.05628) and the code on [Github](https://github.com/wietsedv/gpt2-recycle).


## Related models


### Dutch


- [`gpt2-small-dutch-embeddings`](https://huggingface.co/GroNLP/gpt2-small-dutch-embeddings): Small model size with only retrained lexical embeddings.
- [`gpt2-small-dutch`](https://huggingface.co/GroNLP/gpt2-small-dutch):  Small model size with retrained lexical embeddings and additional fine-tuning of the full model. (**Recommended**)
- [`gpt2-medium-dutch-embeddings`](https://huggingface.co/GroNLP/gpt2-medium-dutch-embeddings): Medium model size with only retrained lexical embeddings.


### Italian


- [`gpt2-small-italian-embeddings`](https://huggingface.co/GroNLP/gpt2-small-italian-embeddings): Small model size with only retrained lexical embeddings.
- [`gpt2-small-italian`](https://huggingface.co/GroNLP/gpt2-small-italian):  Small model size with retrained lexical embeddings and additional fine-tuning of the full model. (**Recommended**)
- [`gpt2-medium-italian-embeddings`](https://huggingface.co/GroNLP/gpt2-medium-italian-embeddings): Medium model size with only retrained lexical embeddings.


## How to use


```python
from transformers import pipeline

pipe = pipeline("text-generation", model="GroNLP/gpt2-small-italian-embeddings")
```


```python
from transformers import AutoTokenizer, AutoModel, TFAutoModel

tokenizer = AutoTokenizer.from_pretrained("GroNLP/gpt2-small-italian-embeddings")
model = AutoModel.from_pretrained("GroNLP/gpt2-small-italian-embeddings")  # PyTorch
model = TFAutoModel.from_pretrained("GroNLP/gpt2-small-italian-embeddings")  # Tensorflow
```


## BibTeX entry


```bibtex
@misc{devries2020good,
      title={As good as new. How to successfully recycle English GPT-2 to make models for other languages},
      author={Wietse de Vries and Malvina Nissim},
      year={2020},
      eprint={2012.05628},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: gpt2-small-italian-embeddings


model author: GroNLP


task: text-generation


tags: pytorch, tf, jax, gpt2, text-generation, it, arxiv:2012.05628, transformers, adaption, recycled, gpt2-small


language: nl
tags:


# GPT-2 recycled for Dutch (small, adapted lexical embeddings)


## Model description


The Transformer layer weights in this model are identical to the original English, model but the lexical layer has been retrained for a Dutch vocabulary.


## Related models


### Dutch


### Italian


```python
from transformers import pipeline

pipe = pipeline("text-generation", model="GroNLP/gpt2-small-dutch-embeddings")
```


```python
from transformers import AutoTokenizer, AutoModel, TFAutoModel

tokenizer = AutoTokenizer.from_pretrained("GroNLP/gpt2-small-dutch-embeddings")
model = AutoModel.from_pretrained("GroNLP/gpt2-small-dutch-embeddings")  # PyTorch
model = TFAutoModel.from_pretrained("GroNLP/gpt2-small-dutch-embeddings")  # Tensorflow
```


model name: gpt2-small-dutch-embeddings


tags: pytorch, tf, jax, gpt2, text-generation, nl, arxiv:2012.05628, transformers, adaption, recycled, gpt2-small


language: it
tags:


- adaption
- recycled
- gpt2-small
pipeline_tag: text-generation


# GPT-2 recycled for Italian (small)


[Wietse de Vries](https://www.semanticscholar.org/author/Wietse-de-Vries/144611157) •
[Malvina Nissim](https://www.semanticscholar.org/author/M.-Nissim/2742475)


This model is based on the small OpenAI GPT-2 ([`gpt2`](https://huggingface.co/gpt2)) model.


## How to use


```python
from transformers import pipeline

pipe = pipeline("text-generation", model="GroNLP/gpt2-small-italian")
```


```python
from transformers import AutoTokenizer, AutoModel, TFAutoModel

tokenizer = AutoTokenizer.from_pretrained("GroNLP/gpt2-small-italian")
model = AutoModel.from_pretrained("GroNLP/gpt2-small-italian")  # PyTorch
model = TFAutoModel.from_pretrained("GroNLP/gpt2-small-italian")  # Tensorflow
```


## BibTeX entry


model name: gpt2-small-italian


model author: GroNLP


task: text-generation


language:


- fi
license: apache-2.0
tags:
- finnish
- roberta
datasets:
- Finnish-NLP/mc4_fi_cleaned
- wikipedia
widget:
- text: "Moikka olen <mask> kielimalli."


# RoBERTa large model for Finnish


This **Finnish-NLP/roberta-large-finnish-v2** model is a new version of the previously trained [Finnish-NLP/roberta-large-finnish](https://huggingface.co/Finnish-NLP/roberta-large-finnish) model. Training hyperparameters were same but the training dataset was cleaned better with the goal to get better performing language model through the better cleaned data. Based on the model evaluations (check the table at the end), slightly better cleaned data didn't seem to produce better performing model.


Pretrained RoBERTa model on Finnish language using a masked language modeling (MLM) objective. RoBERTa was introduced in
[this paper](https://arxiv.org/abs/1907.11692) and first released in
[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta). This model is case-sensitive: it
makes a difference between finnish and Finnish.


## Model description


Finnish RoBERTa is a transformers model pretrained on a large corpus of Finnish data in a self-supervised fashion. This means
it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
publicly available data) with an automatic process to generate inputs and labels from those texts.


More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model
randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict
the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one
after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to
learn a bidirectional representation of the sentence.


This way, the model learns an inner representation of the Finnish language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the RoBERTa model as inputs.


## Intended uses & limitations


You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.


Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.


### How to use


You can use this model directly with a pipeline for masked language modeling:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='Finnish-NLP/roberta-large-finnish-v2')
>>> unmasker("Moikka olen <mask> kielimalli.")

[{'score': 0.04741518571972847,
  'token': 763,
  'token_str': ' hyvä',
  'sequence': 'Moikka olen hyvä kielimalli.'},
 {'score': 0.036977022886276245,
  'token': 505,
  'token_str': ' myös',
  'sequence': 'Moikka olen myös kielimalli.'},
 {'score': 0.025283709168434143,
  'token': 3089,
  'token_str': ' huono',
  'sequence': 'Moikka olen huono kielimalli.'},
 {'score': 0.022848006337881088,
  'token': 1852,
  'token_str': ' toinen',
  'sequence': 'Moikka olen toinen kielimalli.'},
 {'score': 0.019232941791415215,
  'token': 1029,
  'token_str': ' siis',
  'sequence': 'Moikka olen siis kielimalli.'}]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import RobertaTokenizer, RobertaModel
tokenizer = RobertaTokenizer.from_pretrained('Finnish-NLP/roberta-large-finnish-v2')
model = RobertaModel.from_pretrained('Finnish-NLP/roberta-large-finnish-v2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import RobertaTokenizer, TFRobertaModel
tokenizer = RobertaTokenizer.from_pretrained('Finnish-NLP/roberta-large-finnish-v2')
model = TFRobertaModel.from_pretrained('Finnish-NLP/roberta-large-finnish-v2', from_pt=True)
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


### Limitations and bias


The training data used for this model contains a lot of unfiltered content from the internet, which is far from
neutral. Therefore, the model can have biased predictions.


## Training data


This Finnish RoBERTa model was pretrained on the combination of five datasets:


- [mc4_fi_cleaned](https://huggingface.co/datasets/Finnish-NLP/mc4_fi_cleaned), the dataset mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 dataset and further cleaned it with our own text data cleaning codes (check the dataset repo).
- [wikipedia](https://huggingface.co/datasets/wikipedia) We used the Finnish subset of the wikipedia (August 2021) dataset
- [Yle Finnish News Archive](http://urn.fi/urn:nbn:fi:lb-2017070501)
- [Finnish News Agency Archive (STT)](http://urn.fi/urn:nbn:fi:lb-2018121001)
- [The Suomi24 Sentences Corpus](http://urn.fi/urn:nbn:fi:lb-2020021803)


Raw datasets were cleaned to filter out bad quality and non-Finnish examples. Together these cleaned datasets were around 84GB of text.


## Training procedure


### Preprocessing


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
with `<s>` and the end of one by `</s>`


The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `<mask>`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.


Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).


### Pretraining


The model was trained on TPUv3-8 VM, sponsored by the [Google TPU Research Cloud](https://sites.research.google/trc/about/), for 520k train steps (2 epochs, batch size 512) with a sequence length of 128 and continuing for 520k steps (1 epoch, batch size 64) with a sequence length of 512. The optimizer used for the 128 sequence training was AdamW, and for the 512 sequence training it was Adafactor (to save memory). Learning rate was 2e-4, \\(\beta_{1} = 0.9\\), \\(\beta_{2} = 0.98\\) and \\(\epsilon = 1e-6\\), learning rate warmup for 1500 steps and linear decay of the learning rate after.


## Evaluation results


Evaluation was done by fine-tuning the model on downstream text classification task with two different labeled datasets: [Yle News](https://github.com/spyysalo/yle-corpus) and [Eduskunta](https://github.com/aajanki/eduskunta-vkk). Yle News classification fine-tuning was done with two different sequence lengths: 128 and 512 but Eduskunta only with 128 sequence length.
When fine-tuned on those datasets, this model (the first row of the table) achieves the following accuracy results compared to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) model and to our previous [Finnish-NLP/roberta-large-finnish](https://huggingface.co/Finnish-NLP/roberta-large-finnish) model:


|                                        | Average  | Yle News 128 length | Yle News 512 length | Eduskunta 128 length |
|----------------------------------------|----------|---------------------|---------------------|----------------------|
|Finnish-NLP/roberta-large-finnish-v2    |88.17     |94.46                |95.22                |74.83                 |
|Finnish-NLP/roberta-large-finnish       |88.02     |94.53                |95.23                |74.30                 |
|TurkuNLP/bert-base-finnish-cased-v1     |**88.82** |**94.90**            |**95.49**            |**76.07**             |


To conclude, this model didn't significantly improve compared to our previous [Finnish-NLP/roberta-large-finnish](https://huggingface.co/Finnish-NLP/roberta-large-finnish) model. This model is also slightly (~ 1%) losing to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) model.


## Team Members


- Aapo Tanskanen, [Hugging Face profile](https://huggingface.co/aapot), [LinkedIn profile](https://www.linkedin.com/in/aapotanskanen/)
- Rasmus Toivanen [Hugging Face profile](https://huggingface.co/RASMUS), [LinkedIn profile](https://www.linkedin.com/in/rasmustoivanen/)


Feel free to contact us for more details 🤗


model name: roberta-large-finnish-v2


model author: Finnish-NLP


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, fi, dataset:Finnish-NLP/mc4_fi_cleaned, dataset:wikipedia, arxiv:1907.11692, transformers, license:apache-2.0, finnish, autonlp_compatible, infinity_compatible


language:


- fi
license: apache-2.0
tags:
- finnish
- gpt2
datasets:
- Finnish-NLP/mc4_fi_cleaned
- wikipedia
widget:
- text: "Olipa kerran tekoäly"


# GPT2 Finnish


Readme coming soon


model name: gpt2-finnish


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, fi, dataset:Finnish-NLP/mc4_fi_cleaned, dataset:wikipedia, transformers, license:apache-2.0, finnish


- fi
license: apache-2.0
tags:
- finnish
- roberta
datasets:
- Finnish-NLP/mc4_fi_cleaned
- wikipedia
widget:
- text: "Moikka olen <mask> kielimalli."


# RoBERTa large model trained with WECHSEL method for Finnish


Pretrained RoBERTa model on Finnish language using a masked language modeling (MLM) objective with WECHSEL method. RoBERTa was introduced in
[this paper](https://arxiv.org/abs/1907.11692) and first released in
[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta).


WECHSEL method (Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models)  was introduced in [this paper](https://arxiv.org/abs/2112.06598) and first released in [this repository](https://github.com/CPJKU/wechsel).


This model is case-sensitive: it makes a difference between finnish and Finnish.


## Model description


Finnish RoBERTa is a transformers model pretrained on a large corpus of Finnish data in a self-supervised fashion. This means
it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
publicly available data) with an automatic process to generate inputs and labels from those texts.


More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model
randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict
the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one
after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to
learn a bidirectional representation of the sentence.


This way, the model learns an inner representation of the Finnish language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the RoBERTa model as inputs.


## WECHSEL method


Using the WECHSEL method, we first took the pretrained English [roberta-large](https://huggingface.co/roberta-large) model, changed its tokenizer with our Finnish tokenizer and initialized model's token embeddings such that they are close to semantically similar English tokens by utilizing multilingual static word embeddings (by fastText) covering English and Finnish. We were able to confirm the WECHSEL paper's findings that using this method you can save pretraining time and thus computing resources. To get idea of the WECHSEL method's training time savings you can check the table below illustrating the MLM evaluation accuracies during the pretraining compared to the [Finnish-NLP/roberta-large-finnish-v2](https://huggingface.co/Finnish-NLP/roberta-large-finnish-v2) which was trained from scratch:


|                                          | 10k train steps  | 100k train steps | 200k train steps | 270k train steps |
|------------------------------------------|------------------|------------------|------------------|------------------|
|Finnish-NLP/roberta-large-wechsel-finnish |37.61 eval acc    |58.14 eval acc    |61.60 eval acc    |62.77 eval acc    |
|Finnish-NLP/roberta-large-finnish-v2      |13.83 eval acc    |55.87 eval acc    |58.58 eval acc    |59.47 eval acc    |


Downstream finetuning text classification tests can be found from the end but there this model trained with WECHSEL method didn't significantly improve the downstream performances. However, based on tens of qualitative fill-mask task example tests we noticed that for fill-mask task this WECHSEL model significantly outperforms our other models trained from scratch.


## Intended uses & limitations


You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.


Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.


### How to use


You can use this model directly with a pipeline for masked language modeling:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='Finnish-NLP/roberta-large-wechsel-finnish')
>>> unmasker("Moikka olen <mask> kielimalli.")

[{'sequence': 'Moikka olen hyvä kielimalli.',
  'score': 0.07757357507944107,
  'token': 763,
  'token_str': ' hyvä'},
 {'sequence': 'Moikka olen suomen kielimalli.',
  'score': 0.05297883599996567,
  'token': 3641,
  'token_str': ' suomen'},
 {'sequence': 'Moikka olen kuin kielimalli.',
  'score': 0.03747279942035675,
  'token': 523,
  'token_str': ' kuin'},
 {'sequence': 'Moikka olen suomalainen kielimalli.',
  'score': 0.031031042337417603,
  'token': 4966,
  'token_str': ' suomalainen'},
 {'sequence': 'Moikka olen myös kielimalli.',
  'score': 0.026489052921533585,
  'token': 505,
  'token_str': ' myös'}]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import RobertaTokenizer, RobertaModel
tokenizer = RobertaTokenizer.from_pretrained('Finnish-NLP/roberta-large-wechsel-finnish')
model = RobertaModel.from_pretrained('Finnish-NLP/roberta-large-wechsel-finnish')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import RobertaTokenizer, TFRobertaModel
tokenizer = RobertaTokenizer.from_pretrained('Finnish-NLP/roberta-large-wechsel-finnish')
model = TFRobertaModel.from_pretrained('Finnish-NLP/roberta-large-wechsel-finnish', from_pt=True)
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


### Limitations and bias


The training data used for this model contains a lot of unfiltered content from the internet, which is far from
neutral. Therefore, the model can have biased predictions.


## Training data


This Finnish RoBERTa model was pretrained on the combination of five datasets:


- [mc4_fi_cleaned](https://huggingface.co/datasets/Finnish-NLP/mc4_fi_cleaned), the dataset mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 dataset and further cleaned it with our own text data cleaning codes (check the dataset repo).
- [wikipedia](https://huggingface.co/datasets/wikipedia) We used the Finnish subset of the wikipedia (August 2021) dataset
- [Yle Finnish News Archive](http://urn.fi/urn:nbn:fi:lb-2017070501)
- [Finnish News Agency Archive (STT)](http://urn.fi/urn:nbn:fi:lb-2018121001)
- [The Suomi24 Sentences Corpus](http://urn.fi/urn:nbn:fi:lb-2020021803)


Raw datasets were cleaned to filter out bad quality and non-Finnish examples. Together these cleaned datasets were around 84GB of text.


## Training procedure


### Preprocessing


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
with `<s>` and the end of one by `</s>`


The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `<mask>`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.


Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).


### Pretraining


The model was trained on TPUv3-8 VM, sponsored by the [Google TPU Research Cloud](https://sites.research.google/trc/about/), for 270k steps (a bit over 1 epoch, 512 batch size) with a sequence length of 128 and continuing for 180k steps (batch size 64) with a sequence length of 512. The optimizer used was Adafactor (to save memory). Learning rate was 2e-4, \\(\beta_{1} = 0.9\\), \\(\beta_{2} = 0.98\\) and \\(\epsilon = 1e-6\\), learning rate warmup for 2500 steps and linear decay of the learning rate after.


## Evaluation results


Evaluation was done by fine-tuning the model on downstream text classification task with two different labeled datasets: [Yle News](https://github.com/spyysalo/yle-corpus) and [Eduskunta](https://github.com/aajanki/eduskunta-vkk). Yle News classification fine-tuning was done with two different sequence lengths: 128 and 512 but Eduskunta only with 128 sequence length.
When fine-tuned on those datasets, this model (the first row of the table) achieves the following accuracy results compared to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) model and to our previous [Finnish-NLP/roberta-large-finnish-v2](https://huggingface.co/Finnish-NLP/roberta-large-finnish-v2) and [Finnish-NLP/roberta-large-finnish](https://huggingface.co/Finnish-NLP/roberta-large-finnish) models:


|                                          | Average  | Yle News 128 length | Yle News 512 length | Eduskunta 128 length |
|------------------------------------------|----------|---------------------|---------------------|----------------------|
|Finnish-NLP/roberta-large-wechsel-finnish |88.19     |**94.91**            |95.18                |74.47                 |
|Finnish-NLP/roberta-large-finnish-v2      |88.17     |94.46                |95.22                |74.83                 |
|Finnish-NLP/roberta-large-finnish         |88.02     |94.53                |95.23                |74.30                 |
|TurkuNLP/bert-base-finnish-cased-v1       |**88.82** |94.90                |**95.49**            |**76.07**             |


To conclude, this model didn't significantly improve compared to our previous models which were trained from scratch instead of using the WECHSEL method as in this model. This model is also slightly (~ 1%) losing to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) model.


## Team Members


- Aapo Tanskanen, [Hugging Face profile](https://huggingface.co/aapot), [LinkedIn profile](https://www.linkedin.com/in/aapotanskanen/)
- Rasmus Toivanen [Hugging Face profile](https://huggingface.co/RASMUS), [LinkedIn profile](https://www.linkedin.com/in/rasmustoivanen/)


Feel free to contact us for more details 🤗


model name: roberta-large-wechsel-finnish


model author: Finnish-NLP


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, fi, dataset:Finnish-NLP/mc4_fi_cleaned, dataset:wikipedia, arxiv:1907.11692, arxiv:2112.06598, transformers, license:apache-2.0, finnish, autonlp_compatible, infinity_compatible


language:


- fi
license: apache-2.0
tags:
- finnish
- gpt2
datasets:
- Finnish-NLP/mc4_fi_cleaned
- wikipedia
widget:
- text: "Olipa kerran tekoäly"


# GPT2-medium Finnish


Readme coming soon


model name: gpt2-medium-finnish


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, fi, dataset:Finnish-NLP/mc4_fi_cleaned, dataset:wikipedia, transformers, license:apache-2.0, finnish


- fi
license: apache-2.0
tags:
- finnish
- roberta
datasets:
- Finnish-NLP/mc4_fi_cleaned
- wikipedia
widget:
- text: "Moikka olen <mask> kielimalli."


# RoBERTa large model for Finnish


Pretrained RoBERTa model on Finnish language using a masked language modeling (MLM) objective. RoBERTa was introduced in
[this paper](https://arxiv.org/abs/1907.11692) and first released in
[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta). This model is case-sensitive: it
makes a difference between finnish and Finnish.


## Model description


Finnish RoBERTa is a transformers model pretrained on a large corpus of Finnish data in a self-supervised fashion. This means
it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
publicly available data) with an automatic process to generate inputs and labels from those texts.


More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model
randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict
the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one
after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to
learn a bidirectional representation of the sentence.


This way, the model learns an inner representation of the Finnish language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the RoBERTa model as inputs.


## Intended uses & limitations


You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.


Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.


### How to use


You can use this model directly with a pipeline for masked language modeling:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='Finnish-NLP/roberta-large-finnish')
>>> unmasker("Moikka olen <mask> kielimalli.")

[{'sequence': 'Moikka olen hyvä kielimalli.',
  'score': 0.1535797119140625,
  'token': 767,
  'token_str': ' hyvä'},
 {'sequence': 'Moikka olen paras kielimalli.',
  'score': 0.04795042425394058,
  'token': 2888,
  'token_str': ' paras'},
 {'sequence': 'Moikka olen huono kielimalli.',
  'score': 0.04251479730010033,
  'token': 3217,
  'token_str': ' huono'},
 {'sequence': 'Moikka olen myös kielimalli.',
  'score': 0.027469098567962646,
  'token': 520,
  'token_str': ' myös'},
 {'sequence': 'Moikka olen se kielimalli.',
  'score': 0.013878575526177883,
  'token': 358,
  'token_str': ' se'}]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import RobertaTokenizer, RobertaModel
tokenizer = RobertaTokenizer.from_pretrained('Finnish-NLP/roberta-large-finnish')
model = RobertaModel.from_pretrained('Finnish-NLP/roberta-large-finnish')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import RobertaTokenizer, TFRobertaModel
tokenizer = RobertaTokenizer.from_pretrained('Finnish-NLP/roberta-large-finnish')
model = TFRobertaModel.from_pretrained('Finnish-NLP/roberta-large-finnish', from_pt=True)
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


### Limitations and bias


The training data used for this model contains a lot of unfiltered content from the internet, which is far from
neutral. Therefore, the model can have biased predictions.


## Training data


This Finnish RoBERTa model was pretrained on the combination of five datasets:


- [mc4](https://huggingface.co/datasets/mc4), the dataset mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 dataset
- [wikipedia](https://huggingface.co/datasets/wikipedia) We used the Finnish subset of the wikipedia (August 2021) dataset
- [Yle Finnish News Archive](http://urn.fi/urn:nbn:fi:lb-2017070501)
- [Finnish News Agency Archive (STT)](http://urn.fi/urn:nbn:fi:lb-2018121001)
- [The Suomi24 Sentences Corpus](http://urn.fi/urn:nbn:fi:lb-2020021803)


Raw datasets were cleaned to filter out bad quality and non-Finnish examples. Together these cleaned datasets were around 78GB of text.


## Training procedure


### Preprocessing


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
with `<s>` and the end of one by `</s>`


The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `<mask>`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.


Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).


### Pretraining


The model was trained on TPUv3-8 VM, sponsored by the [Google TPU Research Cloud](https://sites.research.google/trc/about/), for 2 epochs with a sequence length of 128 and continuing for one more epoch with a sequence length of 512. The optimizer used is Adafactor with a learning rate of 2e-4, \\(\beta_{1} = 0.9\\), \\(\beta_{2} = 0.98\\) and \\(\epsilon = 1e-6\\), learning rate warmup for 1500 steps and linear decay of the learning rate after.


## Evaluation results


Evaluation was done by fine-tuning the model on downstream text classification task with two different labeled datasets: [Yle News](https://github.com/spyysalo/yle-corpus) and [Eduskunta](https://github.com/aajanki/eduskunta-vkk). Yle News classification fine-tuning was done with two different sequence lengths: 128 and 512 but Eduskunta only with 128 sequence length.
When fine-tuned on those datasets, this model (the first row of the table) achieves the following accuracy results compared to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) and to our previous [Finnish RoBERTa-large](https://huggingface.co/flax-community/RoBERTa-large-finnish) trained during the Hugging Face JAX/Flax community week:


|                                        | Average  | Yle News 128 length | Yle News 512 length | Eduskunta 128 length |
|----------------------------------------|----------|---------------------|---------------------|----------------------|
|Finnish-NLP/roberta-large-finnish       |88.02     |94.53                |95.23                |74.30                 |
|TurkuNLP/bert-base-finnish-cased-v1     |**88.82** |**94.90**            |**95.49**            |**76.07**             |
|flax-community/RoBERTa-large-finnish    |87.72     |94.42                |95.06                |73.67                 |


To conclude, this model improves on our previous [Finnish RoBERTa-large](https://huggingface.co/flax-community/RoBERTa-large-finnish) model trained during the Hugging Face JAX/Flax community week but is still slightly (~ 1%) losing to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) model.


## Team Members


- Aapo Tanskanen, [Hugging Face profile](https://huggingface.co/aapot), [LinkedIn profile](https://www.linkedin.com/in/aapotanskanen/)
- Rasmus Toivanen [Hugging Face profile](https://huggingface.co/RASMUS), [LinkedIn profile](https://www.linkedin.com/in/rasmustoivanen/)
- Tommi Vehviläinen [Hugging Face profile](https://huggingface.co/Tommi)


Feel free to contact us for more details 🤗


model name: roberta-large-finnish


model author: Finnish-NLP


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, fi, dataset:Finnish-NLP/mc4_fi_cleaned, dataset:wikipedia, arxiv:1907.11692, transformers, license:apache-2.0, finnish, autonlp_compatible, infinity_compatible


# MS-BERT


## Introduction


This repository provides codes and models of MS-BERT.
MS-BERT was pre-trained on notes from neurological examination for Multiple Sclerosis (MS) patients at St. Michael's Hospital in Toronto, Canada.


## Data


The dataset contained approximately 75,000 clinical notes, for about 5000 patients, totaling to over 35.7 million words.
These notes were collected from patients who visited St. Michael's Hospital MS Clinic between 2015 to 2019.
The notes contained a variety of information pertaining to a neurological exam.
For example, a note can contain information on the patient's condition, their progress over time and diagnosis.
The gender split within the dataset was observed to be 72% female and 28% male ([which reflects the natural discrepancy seen in MS](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3707353/)).
Further sections will describe how MS-BERT was pre trained through the use of these clinically relevant and rich neurological notes.


## Data pre-processing


The data was pre-processed to remove any identifying information. This includes information on: patient names, doctor names, hospital names, patient identification numbers, phone numbers, addresses, and time. In order to de-identify the information, we used a curated database that contained patient and doctor information. This curated database was paired with regular expressions to find and remove any identifying pieces of information. Each of these identifiers were replaced with a specific token. These tokens were chosen based on three criteria: (1) they belong to the current BERT vocab, (2), they have relatively the same semantic meaning as the word they are replacing, and (3), the token is not found in the original unprocessed dataset. The replacements that met the criteria above were as follows:


Female first names -> Lucie


Male first names -> Ezekiel


Last/family names -> Salamanca.


Dates -> 2010s


Patient IDs -> 999


Phone numbers -> 1718


Addresses -> Silesia


Time -> 1610


Locations/Hospital/Clinic names -> Troy


## Pre-training


The starting point for our model is the already pre-trained and fine-tuned BLUE-BERT base. We further pre-train it using the masked language modelling task from the huggingface transformers [library](https://github.com/huggingface).


The hyperparameters can be found in the config file in this repository or [here](https://s3.amazonaws.com/models.huggingface.co/bert/NLP4H/ms_bert/config.json)


## Acknowledgements


We would like to thank the researchers and staff at the Data Science and Advanced Analytics (DSAA) department, St. Michael’s Hospital, for providing consistent support and guidance throughout this project.
We would also like to thank Dr. Marzyeh Ghassemi, Taylor Killan, Nathan Ng and Haoran Zhang for providing us the opportunity to work on this exciting project.


## Disclaimer


MS-BERT shows the results of research conducted at the Data Science and Advanced Analytics (DSAA) department, St. Michael’s Hospital. The results produced by MS-BERT are not intended for direct diagnostic use or medical decision-making without review and oversight by a clinical professional. Individuals should not make decisions about their health solely on the basis of the results produced by MS-BERT. St. Michael’s Hospital does not independently verify the validity or utility of the results produced by MS-BERT. If you have questions about the results produced by MS-BERT please consult a healthcare professional. If you would like more information about the research conducted at DSAA please contact [Zhen Yang](mailto:zhen.yang@unityhealth.to). If you would like more information on neurological examination notes please contact [Dr. Tony Antoniou](mailto:tony.antoniou@unityhealth.to) or [Dr. Jiwon Oh](mailto:jiwon.oh@unityhealth.to) from the MS clinic at St. Michael's Hospital.


model name: ms_bert


model author: NLP4H


task: fill-mask


tags: pytorch, jax, bert, fill-mask, transformers, autonlp_compatible, infinity_compatible


language: de
widget:


- text: My name is Karl and I live in Aachen.
tags:
- translation
datasets:
- wmt19
license: gpl


English to German Translation


Model Name: Tanhim/translation-En2De

language: German or Deutsch

thumbnail: https://huggingface.co/Tanhim/translation-En2De


### How to use


You can use this model directly with a pipeline for machine translation. Since the generation relies on some randomness, I
set a seed for reproducibility:


```python
>>> from transformers import pipeline, set_seed
>>> text_En2De= pipeline('translation', model='Tanhim/translation-En2De', tokenizer='Tanhim/translation-En2De')
>>> set_seed(42)
>>> text_En2De("My name is Karl and I live in Aachen")

```


### beta version


model name: translation-En2De


model author: Tanhim


task: translation


tags: pytorch, marian, text2text-generation, de, dataset:wmt19, transformers, translation, license:gpl, autonlp_compatible


- text: Hallo, ich bin ein Sprachmodell


## license: gpl


GPT2 Model for German Language


Model Name: Tanhim/gpt2-model-de

language: German or Deutsch

thumbnail: https://huggingface.co/Tanhim/gpt2-model-de

datasets: Ten Thousand German News Articles Dataset


You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, I
set a seed for reproducibility:


```python
>>> from transformers import pipeline, set_seed
>>> generation= pipeline('text-generation', model='Tanhim/gpt2-model-de', tokenizer='Tanhim/gpt2-model-de')
>>> set_seed(42)
>>> generation("Hallo, ich bin ein Sprachmodell,", max_length=30, num_return_sequences=5)

```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("Tanhim/gpt2-model-de")
model = AutoModelWithLMHead.from_pretrained("Tanhim/gpt2-model-de")
text = "Ersetzen Sie mich durch einen beliebigen Text, den Sie wünschen."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


Citation request:
If you use the model of this repository in your research, please consider citing the following way:


```python
@misc{GermanTransformer,
  author = {Tanhim Islam},
  title = {{PyTorch Based Transformer Machine Learning Model for German Text Generation Task}},
  howpublished = "\url{https://huggingface.co/Tanhim/gpt2-model-de}",
  year = {2021},
  note = "[Online; accessed 17-June-2021]"
}
```


model name: gpt2-model-de


task: text-generation


tags: pytorch, gpt2, text-generation, de, transformers, license:gpl


pipeline_tag: sentence-similarity
tags:


- sentence-transformers
- feature-extraction
- sentence-similarity
language: en
license: apache-2.0


# all-MiniLM-L6-v2


This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.


## Usage (Sentence-Transformers)


Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:


```
pip install -U sentence-transformers
```


Then you can use the model like this:


```python
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
embeddings = model.encode(sentences)
print(embeddings)
```


## Usage (HuggingFace Transformers)


Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.


```python
from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

# Normalize embeddings
sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)

print("Sentence embeddings:")
print(sentence_embeddings)
```


## Evaluation Results


For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/all-MiniLM-L6-v2)


## Background


The project aims to train sentence embedding models on very large sentence level datasets using a self-supervised
contrastive learning objective. We used the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model and fine-tuned in on a
1B sentence pairs dataset. We use a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset.


We developped this model during the
[Community week using JAX/Flax for NLP & CV](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104),
organized by Hugging Face. We developped this model as part of the project:
[Train the Best Sentence Embedding Model Ever with 1B Training Pairs](https://discuss.huggingface.co/t/train-the-best-sentence-embedding-model-ever-with-1b-training-pairs/7354). We benefited from efficient hardware infrastructure to run the project: 7 TPUs v3-8, as well as intervention from Googles Flax, JAX, and Cloud team member about efficient deep learning frameworks.


## Intended uses


Our model is intented to be used as a sentence and short paragraph encoder. Given an input text, it ouptuts a vector which captures
the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.


By default, input text longer than 256 word pieces is truncated.


## Training procedure


### Pre-training


We use the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model. Please refer to the model card for more detailed information about the pre-training procedure.


### Fine-tuning


We fine-tune the model using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch.
We then apply the cross entropy loss by comparing with true pairs.


#### Hyper parameters


We trained ou model on a TPU v3-8. We train the model during 100k steps using a batch size of 1024 (128 per TPU core).
We use a learning rate warm up of 500. The sequence length was limited to 128 tokens. We used the AdamW optimizer with
a 2e-5 learning rate. The full training script is accessible in this current repository: `train_script.py`.


#### Training data


We use the concatenation from multiple datasets to fine-tune our model. The total number of sentence pairs is above 1 billion sentences.
We sampled each dataset given a weighted probability which configuration is detailed in the `data_config.json` file.


| Dataset                                                  | Paper                                    | Number of training tuples  |
|--------------------------------------------------------|:----------------------------------------:|:--------------------------:|
| [Reddit comments (2015-2018)](https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit) | [paper](https://arxiv.org/abs/1904.06472) | 726,484,430 |
| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Abstracts) | [paper](https://aclanthology.org/2020.acl-main.447/) | 116,288,806 |
| [WikiAnswers](https://github.com/afader/oqa#wikianswers-corpus) Duplicate question pairs | [paper](https://doi.org/10.1145/2623330.2623677) | 77,427,422 |
| [PAQ](https://github.com/facebookresearch/PAQ) (Question, Answer) pairs | [paper](https://arxiv.org/abs/2102.07033) | 64,371,441 |
| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Titles) | [paper](https://aclanthology.org/2020.acl-main.447/) | 52,603,982 |
| [S2ORC](https://github.com/allenai/s2orc) (Title, Abstract) | [paper](https://aclanthology.org/2020.acl-main.447/) | 41,769,185 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Body) pairs  | - | 25,316,456 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title+Body, Answer) pairs  | - | 21,396,559 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Answer) pairs  | - | 21,396,559 |
| [MS MARCO](https://microsoft.github.io/msmarco/) triplets | [paper](https://doi.org/10.1145/3404835.3462804) | 9,144,553 |
| [GOOAQ: Open Question Answering with Diverse Answer Types](https://github.com/allenai/gooaq) | [paper](https://arxiv.org/pdf/2104.08727.pdf) | 3,012,496 |
| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 1,198,260 |
| [Code Search](https://huggingface.co/datasets/code_search_net) | - | 1,151,414 |
| [COCO](https://cocodataset.org/#home) Image captions | [paper](https://link.springer.com/chapter/10.1007%2F978-3-319-10602-1_48) | 828,395|
| [SPECTER](https://github.com/allenai/specter) citation triplets | [paper](https://doi.org/10.18653/v1/2020.acl-main.207) | 684,100 |
| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Question, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 681,164 |
| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Question) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 659,896 |
| [SearchQA](https://huggingface.co/datasets/search_qa) | [paper](https://arxiv.org/abs/1704.05179) | 582,261 |
| [Eli5](https://huggingface.co/datasets/eli5) | [paper](https://doi.org/10.18653/v1/p19-1346) | 325,475 |
| [Flickr 30k](https://shannon.cs.illinois.edu/DenotationGraph/) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/229/33) | 317,695 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles) | | 304,525 |
| AllNLI ([SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) | [paper SNLI](https://doi.org/10.18653/v1/d15-1075), [paper MultiNLI](https://doi.org/10.18653/v1/n18-1101) | 277,230 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (bodies) | | 250,519 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles+bodies) | | 250,460 |
| [Sentence Compression](https://github.com/google-research-datasets/sentence-compression) | [paper](https://www.aclweb.org/anthology/D13-1155/) | 180,000 |
| [Wikihow](https://github.com/pvl/wikihow_pairs_dataset) | [paper](https://arxiv.org/abs/1810.09305) | 128,542 |
| [Altlex](https://github.com/chridey/altlex/) | [paper](https://aclanthology.org/P16-1135.pdf) | 112,696 |
| [Quora Question Triplets](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs) | - | 103,663 |
| [Simple Wikipedia](https://cs.pomona.edu/~dkauchak/simplification/) | [paper](https://www.aclweb.org/anthology/P11-2117/) | 102,225 |
| [Natural Questions (NQ)](https://ai.google.com/research/NaturalQuestions) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/1455) | 100,231 |
| [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) | [paper](https://aclanthology.org/P18-2124.pdf) | 87,599 |
| [TriviaQA](https://huggingface.co/datasets/trivia_qa) | - | 73,346 |
| **Total** | | **1,170,060,424** |


model name: sentence-transformer


model author: coyotte508


task: sentence-similarity


tags: pytorch, bert, feature-extraction, en, arxiv:1904.06472, arxiv:2102.07033, arxiv:2104.08727, arxiv:1704.05179, arxiv:1810.09305, sentence-transformers, sentence-similarity, license:apache-2.0, infinity_compatible


language: tw
datasets:


- common_voice
tags:
- speech
- audio
- automatic-speech-recognition


model name: wav2vec2large-xlsr-akan


model author: azunre


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, tw, dataset:common_voice, transformers, speech, audio


language:


- en
tags:
- text2text-generation
license: mit
datasets:
- wikifactcheck
widget:
- text: "Little Miss Sunshine was filmed over 30 days."


# BART base negative claim generation model


This is a BART-based model fine-tuned for negative claim generation. This model is used in the data augmentation process described in the paper [CrossAug: A Contrastive Data Augmentation Method for Debiasing Fact Verification Models](https://arxiv.org/abs/2109.15107). The model has been fine-tuned using the parallel and opposing claims from WikiFactCheck-English dataset.


## Usage


```python
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_name = 'minwhoo/bart-base-negative-claim-generation'
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
model.to('cuda' if torch.cuda.is_available() else 'cpu')

examples = [
    "Little Miss Sunshine was filmed over 30 days.",
    "Magic Johnson did not play for the Lakers.",
    "Claire Danes is wedded to an actor from England."
]

batch = tokenizer(examples, max_length=1024, padding=True, truncation=True, return_tensors="pt")
out = model.generate(batch['input_ids'].to(model.device), num_beams=5)
negative_examples = tokenizer.batch_decode(out, skip_special_tokens=True)
print(negative_examples)
# ['Little Miss Sunshine was filmed less than 3 days.', 'Magic Johnson played for the Lakers.', 'Claire Danes is married to an actor from France.']
```


## Citation


```
@inproceedings{lee2021crossaug,
  title={CrossAug: A Contrastive Data Augmentation Method for Debiasing Fact Verification Models},
  author={Minwoo Lee and Seungpil Won and Juae Kim and Hwanhee Lee and Cheoneum Park and Kyomin Jung},
  booktitle={Proceedings of the 30th ACM International Conference on Information & Knowledge Management},
  publisher={Association for Computing Machinery},
  series={CIKM '21},
  year={2021}
}
```


model name: bart-base-negative-claim-generation


model author: minwhoo


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, dataset:wikifactcheck, arxiv:2109.15107, transformers, license:mit, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- imdb
model-index:
- name: distilbert-base-uncased-finetuned-imdb
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-imdb


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the imdb dataset.
It achieves the following results on the evaluation set:


- Loss: 2.4722


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.7117        | 1.0   | 157  | 2.4977          |
| 2.5783        | 2.0   | 314  | 2.4241          |
| 2.5375        | 3.0   | 471  | 2.4358          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-imdb


model author: longnhit07


task: fill-mask


tags: pytorch, tensorboard, distilbert, fill-mask, dataset:imdb, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


title: ArcaneGAN
emoji: 🚀
colorFrom: blue
colorTo: blue
sdk: gradio
app_file: app.py
pinned: false


model name: test


model author: Despin89


tags:


# finetuned-wav2vec2-960h


This model was trained as a part of my **GSoC'21 (Google Summer of Code)** project. It is fine-tuned on 960h of **LibriSpeech dataset** (`train-clean-100`, `train-clean-360`, `train-other-500`) and evaluated on `test-clean` data.


| WER (word error rate) | 5.67 |
|-----------------------|------|


You can find code for training here: https://github.com/vasudevgupta7/gsoc-wav2vec2.


model name: finetuned-wav2vec2-960h


model author: vasudevgupta


tags: tf, transformers


TensorFlow version of [facebook/wav2vec2-base-960h](https://huggingface.co/facebook/wav2vec2-base-960h). Obtained using script from https://github.com/vasudevgupta7/gsoc-wav2vec2.


model name: gsoc-wav2vec2-960h


DL research papers **Title -> abstract**


**Using this model**


```python
from transformers import pipeline, GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("vasudevgupta/dl-hack-distilgpt2")
model = GPT2LMHeadModel.from_pretrained("vasudevgupta/dl-hack-distilgpt2")
agent = pipeline("text-generation", model=model, tokenizer=tokenizer)

print(agent("An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", max_length=200))
```


model name: dl-hack-distilgpt2


task: text-generation


tags: pytorch, jax, gpt2, text-generation, transformers


Moved here: https://huggingface.co/google/bigbird-pegasus-large-pubmed


model name: bigbird-pegasus-large-pubmed


task: text2text-generation


tags: pytorch, bigbird_pegasus, text2text-generation, transformers, autonlp_compatible


TensorFlow equivalent of [`facebook/wav2vec2-large-xlsr-53`](https://huggingface.co/facebook/wav2vec2-large-xlsr-53)


model name: gsoc-wav2vec2-xlsr-53


Wav2Vec2 Model (initialized from [`facebook/wav2vec2-base`](https://huggingface.co/facebook/wav2vec2-base)) with **no** LM head.


Model weights are converted into TensorFlow using following script:


```shell
python3 convert_torch_to_tf.py --hf_model_id "facebook/wav2vec2-base"
```


**TF SavedModel** is obtained by running following commands:


```shell
git clone https://huggingface.co/vasudevgupta/gsoc-wav2vec2

python3 export2hub.py \
--hf_model_id facebook/wav2vec2-base \
--saved_model_dir gsoc-wav2vec2/saved-model \
--seqlen 246000

cd gsoc-wav2vec2 && tar -czf saved-model.tar.gz saved-model
```


Project Link: https://github.com/vasudevgupta7/gsoc-wav2vec2


model name: gsoc-wav2vec2


model author: vasudevgupta


tags: tf, transformers


Moved here: https://huggingface.co/google/bigbird-roberta-large


model name: bigbird-roberta-large


task: fill-mask


tags: pytorch, big_bird, fill-mask, transformers, autonlp_compatible


TensorFlow equivalent of [facebook/wav2vec2-large-robust](https://huggingface.co/facebook/wav2vec2-large-robust)


model name: gsoc-wav2vec2-robust


Moved here: https://huggingface.co/google/bigbird-pegasus-large-arxiv


model name: bigbird-pegasus-large-arxiv


task: text2text-generation


**Project GitHub:** https://github.com/vasudevgupta7/transformers-adapters


**Notes**


* base model can be downloaded from `facebook/mbart-large-cc25`
* `adapters-hin-eng.pt`: adapters hin-eng
* `adapters-guj-eng.pt`: adapters guj-eng


model name: offnote-mbart-adapters-bhasha


tags:


Moved here: https://huggingface.co/google/bigbird-pegasus-large-bigpatent


model name: bigbird-pegasus-large-bigpatent


model author: vasudevgupta


tags: pytorch, bigbird_pegasus, text2text-generation, transformers, autonlp_compatible


Moved here: https://huggingface.co/google/bigbird-base-trivia-itc


model name: bigbird-base-trivia-itc


task: question-answering


tags: pytorch, big_bird, question-answering, transformers, autonlp_compatible


TensorFlow version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base). Obtained using script from https://github.com/vasudevgupta7/gsoc-wav2vec2.


model name: tf-wav2vec2-base


tags: tf, transformers


Deep Learning research papers **Title -> abstract**


model name: dl-hack-pegasus-large


task: text2text-generation


tags: pytorch, pegasus, text2text-generation, transformers, autonlp_compatible


Moved here: https://huggingface.co/google/bigbird-roberta-base


model name: bigbird-roberta-base


task: fill-mask


language: en
license: apache-2.0
datasets: natural_questions
widget:


- text: "Who added BigBird to HuggingFace Transformers?"
context: "BigBird Pegasus just landed! Thanks to Vasudev Gupta, BigBird Pegasus from Google AI is merged into HuggingFace Transformers. Check it out today!!!"


This checkpoint is obtained after training `FlaxBigBirdForQuestionAnswering` (with extra pooler head) on [`natural_questions`](https://huggingface.co/datasets/natural_questions) dataset on TPU v3-8. This dataset takes around ~100 GB on disk. But thanks to Cloud TPUs and Jax, each epoch took just 4.5 hours. Script for training can be found here: https://github.com/vasudevgupta7/bigbird


**Use this model just like any other model from 🤗Transformers**


```python
from transformers import FlaxBigBirdForQuestionAnswering, BigBirdTokenizerFast

model_id = "vasudevgupta/flax-bigbird-natural-questions"
model = BigBirdForQuestionAnswering.from_pretrained(model_id)
tokenizer = BigBirdTokenizerFast.from_pretrained(model_id)
```


In case you are interested in predicting category (null, long, short, yes, no) as well, use `FlaxBigBirdForNaturalQuestions` (instead of `FlaxBigBirdForQuestionAnswering`) from my training script.


| Exact Match | 55.12 |
|-------------|-------|


Evaluation script: https://colab.research.google.com/github/vasudevgupta7/bigbird/blob/main/notebooks/evaluate-flax-natural-questions.ipynb


model name: flax-bigbird-natural-questions


model author: vasudevgupta


tags: jax, big_bird, en, dataset:natural_questions, transformers, license:apache-2.0


DL research papers **Title -> abstract**


**Using this model**


```python
from transformers import pipeline, GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("vasudevgupta/dl-hack-gpt2-large")
model = GPT2LMHeadModel.from_pretrained("vasudevgupta/dl-hack-gpt2-large")
agent = pipeline("text-generation", model=model, tokenizer=tokenizer)

print(agent("An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", max_length=200))
```


model name: dl-hack-gpt2-large


task: text-generation


tags: pytorch, jax, gpt2, text-generation, transformers


| Content | Link |
|---------|------|
| Training Notebook | https://colab.research.google.com/drive/15gRxjjH_X1kcBYSJPFCNkXR6_iqmrb5b?usp=sharing&authuser=1 |
| Qualitative Evaluation Notebook | https://colab.research.google.com/drive/1WvGYGsSyrBa2KM9kbac8VVNIr9T2bNvC?usp=sharing |
| Dataset Samples | https://colab.research.google.com/drive/1RKfZx-0V5OcvGDba3bu9Urv9OIttEPz7?usp=sharing |
| Logging (wandb) | https://wandb.ai/7vasudevgupta/iitm-tts?workspace=user-7vasudevgupta |


## Dataset statistics (`data/male`)


1. Total audio hours of training data: 5.16 hours
1. Total number of samples: 2318


| Data Type | min | max | std | mean |
|-----------|-----|-----|-----|------|
| audio (duration in sec) | 1.6 | 32.17 | 3.76 | 8.022 |
| phonemes | 17 | 369 | 45.261 | 97.139 |
| hindi text (tokenized using space) | 2 | 16 | 2.212 | 4.203 |


model name: tts


tags:


This checkpoint is obtained after training `BigBirdForQuestionAnswering` (with extra pooler head) on [`natural_questions`](https://huggingface.co/datasets/natural_questions) dataset for ~ 2 weeks on 2 K80 GPUs. Script for training can be found here: https://github.com/vasudevgupta7/bigbird


| Exact Match | 47.44 |
|-------------|-------|


```python
from transformers import BigBirdForQuestionAnswering

model_id = "vasudevgupta/bigbird-roberta-natural-questions"
model = BigBirdForQuestionAnswering.from_pretrained(model_id)
tokenizer = BigBirdTokenizer.from_pretrained(model_id)
```


In case you are interested in predicting category (null, long, short, yes, no) as well, use `BigBirdForNaturalQuestions` (instead of `BigBirdForQuestionAnswering`) from my training script.


model name: bigbird-roberta-natural-questions


model author: vasudevgupta


task: question-answering


tags: pytorch, big_bird, question-answering, en, dataset:natural_questions, transformers, license:apache-2.0, autonlp_compatible


datasets: pib
widget:


- text: "હેય! હું વાસુદેવ ગુપ્તા છું"


mBART (a pre-trained model by Facebook) is pre-trained to de-noise multiple languages simultaneously with BART objective.


Checkpoint available in this repository is obtained after fine-tuning `facebook/mbart-large-cc25` on all samples (~60K) from Bhasha (pib_v1.3) Gujarati-English parallel corpus. This checkpoint gives decent results for Gujarati-english translation.


model name: mbart-bhasha-guj-eng


task: text2text-generation


tags: pytorch, mbart, text2text-generation, dataset:pib, transformers, autonlp_compatible


- text: "नमस्ते! मैं वासुदेव गुप्ता हूं"


Checkpoint available in this repository is obtained after fine-tuning `facebook/mbart-large-cc25` on all samples (~260K) from Bhasha (pib_v1.3) Hindi-English parallel corpus. This checkpoint gives decent results for Hindi-english translation.


model name: mbart-bhasha-hin-eng


This model is trained as a part of **InterIIT'21 competition**, on the dataset provided by Bridgei2i. It is able to do multilingual (Hindi, English, Hinglish) summarization (many -> one) & is capable of generating summaries in English regardless of the input language.


| Rouge-L               | Sacrebleu | Headline Similarity (using sentence-transformers) |
|-----------------------|-----------|---------------------------------------------------|
| p=0.46 r=0.49 f1=0.52 | 23.46     | 0.75                                              |


mBART is initialized from **facebook/mbart-large-cc25** and is trained as per strategy mentioned in our [GitHub](https://github.com/vasudevgupta7/Bridgei2i-Winning-Solutions).


model name: mbart-summarizer-interiit


model author: vasudevgupta


tags: pytorch, mbart, text2text-generation, transformers, autonlp_compatible


Checkpoint available in this repository is obtained after fine-tuning `facebook/mbart-large-cc25` on 0.5 M samples from IIT-B Hindi-English parallel corpus. This checkpoint gives decent results for Hindi-english translation.


model name: mbart-iitb-hin-eng


labeled by "YES" : 1, "NO" : 0, "No Answer" : 2
fine tuned by klue/roberta-large


model name: ko-boolq-model


model author: quarter100


task: text-classification


tags: pytorch, roberta, text-classification, transformers, infinity_compatible


tags:


- conversational


# Chandler Bing DialoGPT Model


model name: DialoGPT-small-chandler


model author: theChanChanMan


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


This the repo for the final project


model name: bert-base-uncased-emotion-AD


model author: dhairya2303


tags: tf, distilbert, text-classification, transformers, infinity_compatible


{'sadness':0,'joy':1,'love':2,'anger':3,'fear':4,'surprise':5}


model name: bert-base-uncased-emotion_holler


## TextAttack Model Cardand the glue dataset loaded using the `nlp` library. The model was fine-tuned


for 5 epochs with a batch size of 64, a learning
rate of 3e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.8235858101629914, as measured by the
eval set accuracy, found after 2 epochs.


For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: distilbert-base-uncased-CoLA


model author: textattack


task: text-classification


tags: pytorch, distilbert, text-classification, transformers, infinity_compatible


## TextAttack Model Card


    This `bert-base-cased` model was fine-tuned for sequence classificationusing TextAttack
    and the glue dataset loaded using the `nlp` library. The model was fine-tuned
    for 3 epochs with a batch size of 128, a learning
    rate of 1e-05, and a maximum sequence length of 128.
    Since this was a regression task, the model was trained with a mean squared error loss function.
    The best score the model achieved on this task was 0.8244429996636282, as measured by the
    eval set pearson correlation, found after 2 epochs.

    For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: bert-base-cased-STS-B


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


This `xlnet-base-cased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.7111913357400722, as measured by the
eval set accuracy, found after 3 epochs.


model name: xlnet-base-cased-RTE


task: text-generation


tags: pytorch, xlnet, text-generation, transformers


for 5 epochs with a batch size of 32, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.850431447746884, as measured by the
eval set accuracy, found after 1 epoch.


model name: roberta-base-CoLA


tags: pytorch, jax, roberta, text-classification, transformers, infinity_compatible


    This `bert-base-uncased` model was fine-tuned for sequence classificationusing TextAttack
    and the rotten_tomatoes dataset loaded using the `nlp` library. The model was fine-tuned
    for 10 epochs with a batch size of 16, a learning
    rate of 2e-05, and a maximum sequence length of 128.
    Since this was a classification task, the model was trained with a cross-entropy loss function.
    The best score the model achieved on this task was 0.875234521575985, as measured by the
    eval set accuracy, found after 4 epochs.

    For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: bert-base-uncased-rotten-tomatoes


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the yelp_polarity dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 3e-05, and a maximum sequence length of 512.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.975078947368421, as measured by the
eval set accuracy, found after 3 epochs.


model name: albert-base-v2-yelp-polarity


model author: textattack


task: text-classification


tags: pytorch, albert, text-classification, transformers, infinity_compatible


## TextAttack Model Card


This `bert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 8, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.7256317689530686, as measured by the
eval set accuracy, found after 2 epochs.


model name: bert-base-uncased-RTE


This `xlnet-base-cased` model was fine-tuned for sequence classification using TextAttack
and the imdb dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 2e-05, and a maximum sequence length of 512.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.95352, as measured by the
eval set accuracy, found after 2 epochs.


model name: xlnet-base-cased-imdb


task: text-generation


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 64, a learning
rate of 3e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.776173285198556, as measured by the
eval set accuracy, found after 4 epochs.


For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: albert-base-v2-RTE


This `bert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 64, a learning
rate of 5e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.5633802816901409, as measured by the
eval set accuracy, found after 1 epoch.


model name: bert-base-uncased-WNLI


    This `distilbert-base-cased` model was fine-tuned for sequence classificationusing TextAttack
    and the snli dataset loaded using the `nlp` library. The model was fine-tuned
    for 3 epochs with a batch size of 256, a learning
    rate of 2e-05, and a maximum sequence length of 128.
    Since this was a classification task, the model was trained with a cross-entropy loss function.
    The best score the model achieved on this task was 0.8768542979069295, as measured by the
    eval set accuracy, found after 2 epochs.

    For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: distilbert-base-cased-snli


tags: pytorch, distilbert, transformers, infinity_compatible


## TextAttack Model CardThis `bert-base-uncased` model was fine-tuned for sequence classification using TextAttack


and the ag_news dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 3e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9514473684210526, as measured by the
eval set accuracy, found after 3 epochs.


model name: bert-base-uncased-ag-news


model author: textattack


task: text-classification


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


## TextAttack Model Cardfor 5 epochs with a batch size of 32, a learning


rate of 3e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.7976989453499521, as measured by the
eval set accuracy, found after 2 epochs.


model name: xlnet-base-cased-CoLA


task: text-generation


tags: pytorch, xlnet, text-generation, transformers


## TextAttack Model Card


This `bert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the yelp_polarity dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 5e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9699473684210527, as measured by the
eval set accuracy, found after 4 epochs.


model name: bert-base-uncased-yelp-polarity


## TextAttack Model CardSince this was a classification task, the model was trained with a cross-entropy loss function.


The best score the model achieved on this task was 0.7256317689530686, as measured by the
eval set accuracy, found after 4 epochs.


model name: facebook-bart-base-RTE


task: text2text-generation


tags: pytorch, bart, text2text-generation, transformers, autonlp_compatible


This `bert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 2e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.8774509803921569, as measured by the
eval set accuracy, found after 1 epoch.


model name: bert-base-uncased-MRPC


model author: textattack


task: text-classification


## TextAttack Model CardThis `roberta-base` model was fine-tuned for sequence classification using TextAttack


and the ag_news dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 5e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9469736842105263, as measured by the
eval set accuracy, found after 4 epochs.


For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: roberta-base-ag-news


tags: pytorch, jax, roberta, text-classification, transformers, infinity_compatible


## TextAttack Model Card


This `roberta-base` model was fine-tuned for sequence classification using TextAttack
and the imdb dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 64, a learning
rate of 3e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.91436, as measured by the
eval set accuracy, found after 2 epochs.


model name: roberta-base-imdb


## TextAttack Model CardThis `albert-base-v2` model was fine-tuned for sequence classification using TextAttack


and the ag_news dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9471052631578948, as measured by the
eval set accuracy, found after 3 epochs.


model name: albert-base-v2-ag-news


tags: pytorch, albert, text-classification, transformers, infinity_compatible


## TextAttack Model Cardand the glue dataset loaded using the `nlp` library. The model was fine-tuned


for 5 epochs with a batch size of 32, a learning
rate of 3e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.8245445829338447, as measured by the
eval set accuracy, found after 2 epochs.


model name: albert-base-v2-CoLA


model author: textattack


task: text-classification


## TextAttack Model CardThis `distilbert-base-uncased` model was fine-tuned for sequence classification using TextAttack


and the ag_news dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9478947368421052, as measured by the
eval set accuracy, found after 1 epoch.


model name: distilbert-base-uncased-ag-news


tags: pytorch, distilbert, text-classification, transformers, infinity_compatible


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the snli dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 64, a learning
rate of 2e-05, and a maximum sequence length of 64.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9060150375939849, as measured by the
eval set accuracy, found after 2 epochs.


model name: albert-base-v2-snli


## TextAttack Model Card


This `roberta-base` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.7942238267148014, as measured by the
eval set accuracy, found after 3 epochs.


model name: roberta-base-RTE


This `distilbert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.6570397111913358, as measured by the
eval set accuracy, found after 4 epochs.


model name: distilbert-base-uncased-RTE


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 64, a learning
rate of 2e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.5915492957746479, as measured by the
eval set accuracy, found after 0 epoch.


model name: albert-base-v2-WNLI


    This `roberta-base` model was fine-tuned for sequence classificationusing TextAttack
    and the rotten_tomatoes dataset loaded using the `nlp` library. The model was fine-tuned
    for 10 epochs with a batch size of 64, a learning
    rate of 2e-05, and a maximum sequence length of 128.
    Since this was a classification task, the model was trained with a cross-entropy loss function.
    The best score the model achieved on this task was 0.9033771106941839, as measured by the
    eval set accuracy, found after 2 epochs.

    For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: roberta-base-rotten-tomatoes


model author: textattack


task: text-classification


tags: pytorch, jax, roberta, text-classification, transformers, infinity_compatible


This `roberta-base` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 8, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a regression task, the model was trained with a mean squared error loss function.
The best score the model achieved on this task was 0.9108696741479216, as measured by the
eval set pearson correlation, found after 4 epochs.


For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: roberta-base-STS-B


This `xlnet-base-cased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 8, a learning
rate of 5e-05, and a maximum sequence length of 128.
Since this was a regression task, the model was trained with a mean squared error loss function.
The best score the model achieved on this task was 0.8892630070017784, as measured by the
eval set pearson correlation, found after 4 epochs.


model name: xlnet-base-cased-STS-B


task: text-generation


tags: pytorch, xlnet, text-generation, transformers


## TextAttack Model Card


This `distilbert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 2e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.8578431372549019, as measured by the
eval set accuracy, found after 1 epoch.


model name: distilbert-base-uncased-MRPC


## bert-base-uncased fine-tuned with TextAttack on the rotten_tomatoes dataset


    This `bert-base-uncased` model was fine-tuned for sequence classificationusing TextAttack
    and the rotten_tomatoes dataset loaded using the `nlp` library. The model was fine-tuned
    for 10 epochs with a batch size of 64, a learning
    rate of 5e-05, and a maximum sequence length of 128.
    Since this was a classification task, the model was trained with a cross-entropy loss function.
    The best score the model achieved on this task was 0.875234521575985, as measured by the
    eval set accuracy, found after 4 epochs.

    For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: bert-base-uncased-rotten_tomatoes


task: fill-mask


tags: pytorch, jax, tensorboard, bert, fill-mask, transformers, autonlp_compatible, infinity_compatible


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.8970588235294118, as measured by the
eval set accuracy, found after 4 epochs.


model name: albert-base-v2-MRPC


model author: textattack


task: text-classification


tags: pytorch, albert, text-classification, transformers, infinity_compatible


This `bert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the imdb dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.89088, as measured by the
eval set accuracy, found after 4 epochs.


model name: bert-base-uncased-imdb


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the rotten_tomatoes dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 64, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.8808630393996247, as measured by the
eval set accuracy, found after 1 epoch.


model name: albert-base-v2-rotten-tomatoes


## TextAttack Model Card


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 3e-05, and a maximum sequence length of 64.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9254587155963303, as measured by the
eval set accuracy, found after 2 epochs.


model name: albert-base-v2-SST-2


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the imdb dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.89236, as measured by the
eval set accuracy, found after 3 epochs.


For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: albert-base-v2-imdb


This `roberta-base` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 3e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9117647058823529, as measured by the
eval set accuracy, found after 2 epochs.


model name: roberta-base-MRPC


tags: pytorch, jax, roberta, text-classification, transformers, infinity_compatible


    This `distilbert-base-uncased` model was fine-tuned for sequence classificationusing TextAttack
    and the rotten_tomatoes dataset loaded using the `nlp` library. The model was fine-tuned
    for 3 epochs with a batch size of 128, a learning
    rate of 1e-05, and a maximum sequence length of 128.
    Since this was a classification task, the model was trained with a cross-entropy loss function.
    The best score the model achieved on this task was 0.8395872420262664, as measured by the
    eval set accuracy, found after 2 epochs.

    For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: distilbert-base-uncased-rotten-tomatoes


model author: textattack


task: text-classification


tags: pytorch, distilbert, text-classification, transformers, infinity_compatible


This `roberta-base` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 5e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.5633802816901409, as measured by the
eval set accuracy, found after 0 epoch.


model name: roberta-base-WNLI


This `distilbert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the imdb dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.88, as measured by the
eval set accuracy, found after 2 epochs.


model name: distilbert-base-uncased-imdb


## TextAttack Model Card


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 5e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9073707642839476, as measured by the
eval set accuracy, found after 3 epochs.


model name: albert-base-v2-QQP


tags: pytorch, albert, text-classification, transformers, infinity_compatible


This `xlnet-base-cased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 3e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.5774647887323944, as measured by the
eval set accuracy, found after 0 epoch.


model name: xlnet-base-cased-WNLI


task: text-generation


tags: pytorch, xlnet, text-generation, transformers


## TextAttack Model Cardrate of 2e-05, and a maximum sequence length of 128.


Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.7256317689530686, as measured by the
eval set accuracy, found after 4 epochs.


model name: facebook-bart-base-glue-RTE


model author: textattack


task: text2text-generation


tags: pytorch, bart, text2text-generation, transformers, autonlp_compatible


This `xlnet-base-cased` model was fine-tuned for sequence classification using TextAttack
and the rotten_tomatoes dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 16, a learning
rate of 2e-05, and a maximum sequence length of 128.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.9071294559099438, as measured by the
eval set accuracy, found after 2 epochs.


For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: xlnet-base-cased-rotten-tomatoes


## TextAttack Model Card


This `distilbert-base-uncased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 128, a learning
rate of 2e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.5633802816901409, as measured by the
eval set accuracy, found after 0 epoch.


model name: distilbert-base-uncased-WNLI


task: text-classification


## roberta-base fine-tuned with TextAttack on the rotten_tomatoes dataset


    This `roberta-base` model was fine-tuned for sequence classificationusing TextAttack
    and the rotten_tomatoes dataset loaded using the `nlp` library. The model was fine-tuned
    for 10 epochs with a batch size of 128, a learning
    rate of 5e-05, and a maximum sequence length of 128.
    Since this was a classification task, the model was trained with a cross-entropy loss function.
    The best score the model achieved on this task was 0.9033771106941839, as measured by the
    eval set accuracy, found after 9 epochs.

    For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: roberta-base-rotten_tomatoes


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


This `albert-base-v2` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 3e-05, and a maximum sequence length of 128.
Since this was a regression task, the model was trained with a mean squared error loss function.
The best score the model achieved on this task was 0.9064220351504577, as measured by the
eval set pearson correlation, found after 3 epochs.


model name: albert-base-v2-STS-B


model author: textattack


## albert-base-v2 fine-tuned with TextAttack on the rotten_tomatoes dataset


    This `albert-base-v2` model was fine-tuned for sequence classificationusing TextAttack
    and the rotten_tomatoes dataset loaded using the `nlp` library. The model was fine-tuned
    for 10 epochs with a batch size of 128, a learning
    rate of 2e-05, and a maximum sequence length of 128.
    Since this was a classification task, the model was trained with a cross-entropy loss function.
    The best score the model achieved on this task was 0.8855534709193246, as measured by the
    eval set accuracy, found after 1 epoch.

    For more information, check out [TextAttack on Github](https://github.com/QData/TextAttack).


model name: albert-base-v2-rotten_tomatoes


tags: pytorch, tensorboard, albert, fill-mask, transformers, autonlp_compatible, infinity_compatible


This `xlnet-base-cased` model was fine-tuned for sequence classification using TextAttack
and the glue dataset loaded using the `nlp` library. The model was fine-tuned
for 5 epochs with a batch size of 32, a learning
rate of 5e-05, and a maximum sequence length of 256.
Since this was a classification task, the model was trained with a cross-entropy loss function.
The best score the model achieved on this task was 0.8897058823529411, as measured by the
eval set accuracy, found after 2 epochs.


model name: xlnet-base-cased-MRPC


task: text-generation


tags: pytorch, xlnet, text-generation, transformers


language:


- hr
- sl
- en
- multilingual


## license: cc-by-4.0


# CroSloEngual BERT