```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-2_H-256


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "作为电子[MASK]的平台，京东绝对是领先者。如今的刘强[MASK]已经是身价过[MASK]的老板。"


# Chinese BART


## Model description


This model is pre-trained by [UER-py](https://arxiv.org/abs/1909.05658).


## How to use


You can use this model directly with a pipeline for text2text generation :


```python
>>> from transformers import BertTokenizer, BartForConditionalGeneration, Text2TextGenerationPipeline
>>> tokenizer = BertTokenizer.from_pretrained("uer/bart-chinese-4-768-cluecorpussmall")
>>> model = BartForConditionalGeneration.from_pretrained("uer/bart-chinese-4-768-cluecorpussmall")
>>> text2text_generator = Text2TextGenerationPipeline(model, tokenizer)
>>> text2text_generator("中国的首都是[MASK]京", max_length=50, do_sample=False)
    [{'generated_text': '中 国 的 首 都 是 北 京'}]
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) Common Crawl and some short messages are used as training data.


## Training procedure


The model is pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 512.


we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bart_from_uer_to_huggingface.py --input_model_path cluecorpussmall_bart_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 4
```


### BibTeX entry and citation info


```
@article{lewis2019bart,
  title={Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension},
  author={Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Ves and Zettlemoyer, Luke},
  journal={arXiv preprint arXiv:1910.13461},
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: bart-chinese-4-768-cluecorpussmall


model author: uer


task: text2text-generation


tags: pytorch, bart, text2text-generation, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, transformers, autonlp_compatible


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-6_H-512


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-2_H-128


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


# **Evaluation of the NER models in medical dataset**


The goal of the whole project is to  compare the NER models and feature evaluation in the medical dataset, and the program of model comparison needs to be executed in the GPU environment. Here are the instructions for the two project.


## 1. Model Comparison


### 1.1 Environment setting:


(1) Python 3 environment (Python 3.6 and above)


The user can click the link (https://www.python.org/) to select the appropriate python version and download.


(2) Some related package in python


The version of the package we used is as follows:


```shell
Transformers: 4.8.2
NERDA: 0.9.5
Pytorch: 1.8.1+cu101
Tensorflow: 2.3.0
```


The user can execute the following command in python environment.


```shell
pip install tensorflow-gpu==2.3.0 -i https://pypi.doubanio.com/simple
pip install transformers==4.8.2
pip install NERDA
pip install sentencepiece
pip install torch==1.8.1+cu101 torchvision==0.9.1+cu101 torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
```


### 1.2 The process of implementation


(1) Training and testing


Users can check the "training&testing.ipynb" file. The user can load the models to be trained and download them locally, or directly import it into the internal model of transformers website.


For example:


```python
# Model loading in the "training&testing.ipynb" file

transformer = '../../Model/bigbird-roberta-base/'
or
transformer = 'google/bigbird-roberta-base'
```


Address of model download:


```http
https://huggingface.co/dmis-lab/biobert-base-cased-v1.1
https://huggingface.co/roberta-base
https://huggingface.co/google/bigbird-roberta-base
https://huggingface.co/microsoft/deberta-base
```


The user can download models through the above websites and put them in the "model" folder.


(2) Prediction program


Users can load the trained models and input new text to make that the model recognize the entities in the text. We give five trained models with the best training effect for RoBERTa, BigBird, DeBERTa, and BioBERT NER models ( The suffix of the five models ends with ". bin" ). These models is saved in "Trained model" file.


```python
import torch
model = torch.load('../../Trained model/deberta_x.bin')
model.predict_text('Number of glucocorticoid receptors in lymphocytes and their sensitivity to hormone action.')

->> ([['Number', 'of', 'glucocorticoid', 'receptors', 'in', 'lymphocytes', 'and', 'their', 'sensitivity', 'to', 'hormone','action','.']],
[['O', 'O', 'B-protein','I-protein','o','B-cell_type','O','O','O','O','O','O','O']])
```


## 2. Feature Evaluation


### 2.1 Environment setting:


(1) Some related package in python


Packages we used is as follows, users can download the latest packages by ”pip install package name“ commend.


```shell
1. warnings
2. matplotlib
3. pandas
4. seaborn
5. statsmodels
6. sklearn
```


### 2.2 The process of implementation


Users can check the "feature_selection.ipynb" and "feature_evaluation.ipynb"file. Due to the privacy of the data, we did not upload the feature data, so users can view different methods of feature selection in this file.


### 3. Contact


If user have any questions, please contact us.


(1) Sizhu Wu - [wu.sizhu@imicams.ac.cn]


(2) Shengyu Liu - [liu.shengyu@imicams.ac.cn]


model name: Evaluation_of_NER_models


model author: Shengyu


tags:


language: multilingual


- Extract Names


## license: apache-2.0


## Extract names in any language.


model name: extract_names


model author: opensource


task: token-classification


tags: tf, xlm-roberta, token-classification, multilingual, transformers, Extract Names, license:apache-2.0, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilbert-base-uncased-finetuned-squad
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 1.7713


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.0325        | 1.0   | 585  | 1.7520          |
| 1.609         | 2.0   | 1170 | 1.7713          |


### Framework versions


- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


model name: distilbert-base-uncased-finetuned-squad


model author: LenaSchmidt


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- conversational


# My Awesome Model


model name: DialoGPT-small-Neku


model author: Emi2160


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


language:


- fr


- {fr}
- {gpt2}


This model aims at being a french conversational agent. This consists of a fine-tuning of Dialo-GPT for french language. The dataset used gathers 36k conversations extracted from books, movies, interviews and dialogues for learning french.


More details about the model can be found [there](https://github.com/emil2000dza/DialoGPT-fine-tuned-for-french-language)


model name: dialogpt-for-french-language


model author: emil2000


task: text-generation


tags: pytorch, gpt2, text-generation, fr, transformers


license: cc-by-4.0
tags:


- generated_from_trainer
model-index:
- name: roberta-base-squad2-finetuned-squad
results: []


# roberta-base-squad2-finetuned-squad


This model is a fine-tuned version of [deepset/roberta-base-squad2](https://huggingface.co/deepset/roberta-base-squad2) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 5.0220


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 30


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 74   | 1.7148          |
| No log        | 2.0   | 148  | 1.6994          |
| No log        | 3.0   | 222  | 1.7922          |
| No log        | 4.0   | 296  | 1.9947          |
| No log        | 5.0   | 370  | 2.0753          |
| No log        | 6.0   | 444  | 2.2096          |
| 0.9547        | 7.0   | 518  | 2.3070          |
| 0.9547        | 8.0   | 592  | 2.6947          |
| 0.9547        | 9.0   | 666  | 2.7169          |
| 0.9547        | 10.0  | 740  | 2.8503          |
| 0.9547        | 11.0  | 814  | 3.1990          |
| 0.9547        | 12.0  | 888  | 3.4931          |
| 0.9547        | 13.0  | 962  | 3.6575          |
| 0.3191        | 14.0  | 1036 | 3.1863          |
| 0.3191        | 15.0  | 1110 | 3.7922          |
| 0.3191        | 16.0  | 1184 | 3.6336          |
| 0.3191        | 17.0  | 1258 | 4.1156          |
| 0.3191        | 18.0  | 1332 | 4.1353          |
| 0.3191        | 19.0  | 1406 | 3.9888          |
| 0.3191        | 20.0  | 1480 | 4.4290          |
| 0.1904        | 21.0  | 1554 | 4.0473          |
| 0.1904        | 22.0  | 1628 | 4.5048          |
| 0.1904        | 23.0  | 1702 | 4.4026          |
| 0.1904        | 24.0  | 1776 | 4.2864          |
| 0.1904        | 25.0  | 1850 | 4.3941          |
| 0.1904        | 26.0  | 1924 | 4.4921          |
| 0.1904        | 27.0  | 1998 | 4.9139          |
| 0.1342        | 28.0  | 2072 | 4.8914          |
| 0.1342        | 29.0  | 2146 | 5.0148          |
| 0.1342        | 30.0  | 2220 | 5.0220          |


### Framework versions


- Transformers 4.11.0
- Pytorch 1.9.0+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: roberta-base-squad2-finetuned-squad


model author: avioo1


task: question-answering


tags: pytorch, tensorboard, roberta, question-answering, transformers, license:cc-by-4.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: distilbert-base-uncased-finetuned-squad
results:
  - task:
name: Question Answering
type: question-answering
dataset:
name: squad
type: squad
args: plain_text


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad dataset.
It achieves the following results on the evaluation set:


- Loss: 1.2125


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.2637        | 1.0   | 5533 | 1.2125          |


### Framework versions


- Transformers 4.10.2
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-squad


model author: avioo1


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, dataset:squad, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: distilbert-base-uncased-finetuned-cola
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: glue
type: glue
args: cola
metrics:
    - name: Matthews Correlation
type: matthews_correlation
value: 0.5232819075279987


# distilbert-base-uncased-finetuned-cola


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:


- Loss: 0.5327
- Matthews Correlation: 0.5233


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.5314        | 1.0   | 535  | 0.4955          | 0.4270               |
| 0.3545        | 2.0   | 1070 | 0.5327          | 0.5233               |
| 0.2418        | 3.0   | 1605 | 0.6180          | 0.5132               |
| 0.1722        | 4.0   | 2140 | 0.7344          | 0.5158               |
| 0.1243        | 5.0   | 2675 | 0.8581          | 0.5196               |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-cola


model author: Kien


task: text-classification


tags: pytorch, tensorboard, distilbert, text-classification, dataset:glue, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


tags:


- conversational


#rick DialoGPT Model


model name: DialoGPT-small-rick


model author: Cryptikdw


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


- text2text-generation
library_name: generic


random test repo


model name: 12412fsasf


model author: p-christ


task: text2text-generation


tags: pytorch, t5, text2text-generation, generic


# KoCLIP


This repository includes


## Installation


Create a virtual env and install `requirements.txt`.


```
pip install -r requirements.txt
```


For Google Cloud TPU VM please follow necessary installation steps here:


[Pytorch on TPU VM](https://cloud.google.com/tpu/docs/pytorch-xla-ug-tpu-vm)
[JAX/Flax on TPU VM](https://cloud.google.com/tpu/docs/jax-quickstart-tpu-vm)


model name: koclip


model author: flax-community


tags:


widget:


- text: "अध्यक्ष <mask> पवार आणि उपमुख्यमंत्री अजित पवार यांची भेट घेतली."
- text: "मोठी बातमी! उद्या दुपारी <mask> वाजता जाहीर होणार दहावीचा निकाल"


# RoBERTa base model for Marathi language (मराठी भाषा)


Pretrained model on Marathi language using a masked language modeling (MLM) objective. RoBERTa was introduced in
[this paper](https://arxiv.org/abs/1907.11692) and first released in
[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta). We trained RoBERTa model for Marathi Language during community week hosted by Huggingface 🤗 using JAX/Flax for NLP & CV jax.


<img src="https://user-images.githubusercontent.com/15062408/126040902-ea8808db-ec30-4a3f-bf95-5d3b10d674e9.png" alt="huggingface-marathi-roberta" width="350" height="350" style="text-align: center">


## Model description


Marathi RoBERTa is a transformers model pretrained on a large corpus of Marathi data in a self-supervised fashion.


## Intended uses & limitations❗


You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. We used this model to fine tune on text classification task for iNLTK and indicNLP news text classification problem statement. Since marathi mc4 dataset is made by scraping marathi newspapers text, it will involve some biases which will also affect all fine-tuned versions of this model.


### How to use❓


You can use this model directly with a pipeline for masked language modeling:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='flax-community/roberta-base-mr')
>>> unmasker("मोठी बातमी! उद्या दुपारी <mask> वाजता जाहीर होणार दहावीचा निकाल")
[{'score': 0.057209037244319916,'sequence': 'मोठी बातमी! उद्या दुपारी आठ वाजता जाहीर होणार दहावीचा निकाल',
  'token': 2226,
  'token_str': 'आठ'},
 {'score': 0.02796074189245701,
  'sequence': 'मोठी बातमी! उद्या दुपारी २० वाजता जाहीर होणार दहावीचा निकाल',
  'token': 987,
  'token_str': '२०'},
 {'score': 0.017235398292541504,
  'sequence': 'मोठी बातमी! उद्या दुपारी नऊ वाजता जाहीर होणार दहावीचा निकाल',
  'token': 4080,
  'token_str': 'नऊ'},
 {'score': 0.01691395975649357,
  'sequence': 'मोठी बातमी! उद्या दुपारी २१ वाजता जाहीर होणार दहावीचा निकाल',
  'token': 1944,
  'token_str': '२१'},
 {'score': 0.016252165660262108,
  'sequence': 'मोठी बातमी! उद्या दुपारी  ३ वाजता जाहीर होणार दहावीचा निकाल',
  'token': 549,
  'token_str': ' ३'}]
```


## Training data 🏋🏻♂


The RoBERTa Marathi model was pretrained on `mr` dataset of C4 multilingual dataset:

[C4 (Colossal Clean Crawled Corpus)](https://yknzhu.wixsite.com/mbweb), Introduced by Raffel et al. in [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://paperswithcode.com/paper/exploring-the-limits-of-transfer-learning).


The dataset can be downloaded in a pre-processed form from [allennlp](https://github.com/allenai/allennlp/discussions/5056) or huggingface's datsets - [mc4 dataset](https://huggingface.co/datasets/mc4).
Marathi (`mr`) dataset consists of 14 billion tokens, 7.8 million docs and with weight ~70 GB of text.


## Data Cleaning 🧹


Though initial `mc4` marathi corpus size ~70 GB, Through data exploration, it was observed it contains docs from different languages especially thai, chinese etc. So we had to clean the dataset before traning tokenizer and model. Surprisingly, results after cleaning Marathi mc4 corpus data:


#### **Train set:**


Clean docs count 1581396 out of 7774331. <br>
**~20.34%** of whole marathi train split is actually Marathi.


#### **Validation set**


Clean docs count 1700 out of 7928. <br>
**~19.90%** of whole marathi validation split is actually Marathi.


## Training procedure 👨🏻💻


### Preprocessing


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
with `<s>` and the end of one by `</s>`
The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `<mask>`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.
Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).


### Pretraining


The model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) **8 v3 TPU cores** for 42K steps with a batch size of 128 and a sequence length of 128. The
optimizer used is Adam with a learning rate of 3e-4, β1 = 0.9, β2 = 0.98 and
ε = 1e-8, a weight decay of 0.01, learning rate warmup for 1,000 steps and linear decay of the learning
rate after.


We tracked experiments and hyperparameter tunning on weights and biases platform. Here is link to main dashboard: <br>
[Link to Weights and Biases Dashboard for Marathi RoBERTa model](https://wandb.ai/nipunsadvilkar/roberta-base-mr/runs/19qtskbg?workspace=user-nipunsadvilkar)


#### **Pretraining Results 📊**


RoBERTa Model reached **eval accuracy of 85.28%** around ~35K step **with train loss at 0.6507 and eval loss at 0.6219**.


## Fine Tuning on downstream tasks


We performed fine-tuning on downstream tasks. We used following datasets for classification:


1. [IndicNLP Marathi news classification](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)
1. [iNLTK Marathi news headline classification](https://www.kaggle.com/disisbig/marathi-news-dataset)


### **Fine tuning on downstream task results (Segregated)**


#### 1. [IndicNLP Marathi news classification](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)


IndicNLP Marathi news dataset consists 3 classes - `['lifestyle', 'entertainment', 'sports']` - with following docs distribution as per classes:


| train | eval | test
| -- | -- | --
| 9672 | 477 | 478


💯 Our Marathi RoBERTa **`roberta-base-mr` model outperformed both classifier ** mentioned in [Arora, G. (2020). iNLTK](https://www.semanticscholar.org/paper/iNLTK%3A-Natural-Language-Toolkit-for-Indic-Languages-Arora/5039ed9e100d3a1cbbc25a02c82f6ee181609e83/figure/3) and [Kunchukuttan, Anoop et al. AI4Bharat-IndicNLP.](https://www.semanticscholar.org/paper/AI4Bharat-IndicNLP-Corpus%3A-Monolingual-Corpora-and-Kunchukuttan-Kakwani/7997d432925aff0ba05497d2893c09918298ca55/figure/4)


Dataset | FT-W | FT-WC | INLP | iNLTK | **roberta-base-mr 🏆**
-- | -- | -- | -- | -- | --
iNLTK Headlines | 83.06 | 81.65 | 89.92 | 92.4 | **97.48**


**🤗 Huggingface Model hub repo:**<br>
`roberta-base-mr` fine tuned on iNLTK Headlines classification dataset model:


🧪 Fine tuning experiment's weight and biases dashboard [link](https://wandb.ai/nipunsadvilkar/huggingface/runs/1242bike?workspace=user-nipunsadvilkar)


#### 2. [iNLTK Marathi news headline classification](https://www.kaggle.com/disisbig/marathi-news-dataset)


This dataset consists 3 classes - `['state', 'entertainment', 'sports']` -  with following docs distribution as per classes:


| train | eval | test
| -- | -- | --
| 9658 | 1210 | 1210


💯 Here as well **`roberta-base-mr` outperformed `iNLTK` marathi news text classifier**.


Dataset | iNLTK ULMFiT | **roberta-base-mr 🏆**
-- | -- | --
iNLTK news dataset (kaggle) | 92.4 | **94.21**


**🤗 Huggingface Model hub repo:**<br>
`roberta-base-mr` fine tuned on iNLTK news classification dataset model:


Fine tuning experiment's weight and biases dashboard [link](https://wandb.ai/nipunsadvilkar/huggingface/runs/2u5l9hon?workspace=user-nipunsadvilkar)


## **Want to check how above models generalise on real world Marathi data?**


Head to 🤗 Huggingface's spaces 🪐 to play with all three models:


1. Mask Language Modelling with Pretrained Marathi RoBERTa model:

[**`flax-community/roberta-base-mr`**](https://huggingface.co/flax-community/roberta-base-mr)
1. Marathi Headline classifier:

[**`flax-community/mr-indicnlp-classifier`**](https://huggingface.co/flax-community/mr-indicnlp-classifier)
1. Marathi news classifier:

[**`flax-community/mr-inltk-classifier`**](https://huggingface.co/flax-community/mr-inltk-classifier)


![alt text](https://huggingface.co/docs/assets/hub/icon-space.svg)
[Streamlit app of Pretrained Roberta Marathi model on Huggingface Spaces](https://huggingface.co/spaces/flax-community/roberta-base-mr)


## Team Members


- Nipun Sadvilkar [@nipunsadvilkar](https://github.com/nipunsadvilkar)
- Haswanth Aekula [@hassiahk](https://github.com/hassiahk)


## Credits


Huge thanks to Huggingface 🤗 & Google Jax/Flax team for such a wonderful community week. Especially for providing such massive computing resource. Big thanks to [@patil-suraj](https://github.com/patil-suraj) & [@patrickvonplaten](https://github.com/patrickvonplaten) for mentoring during whole week.


<img src=https://pbs.twimg.com/media/E443fPjX0AY1BsR.jpg:large>


model name: roberta-base-mr


model author: flax-community


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, arxiv:1907.11692, transformers, autonlp_compatible, infinity_compatible


# roberta-base-als-demo


**roberta-base-als-demo** is a model trained by Patrick von Platen to demonstrate how to train a roberta-base model from scratch on the Alemannic language.


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Useful links


- [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
- [Community Week README](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/README.md)
- [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
- [Model Repository](https://huggingface.co/flax-community/roberta-base-als-demo)


model name: roberta-base-als-demo


tags: jax, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


## language: en
tags: vae
license: apache-2.0


# T5-VAE-Wiki (flax)


A Transformer-VAE made using flax.


It has been trained to interpolate on sentences form wikipedia.


Done as part of Huggingface community training ([see forum post](https://discuss.huggingface.co/t/train-a-vae-to-interpolate-on-english-sentences/7548)).


Builds on T5, using an autoencoder to convert it into an MMD-VAE ([more info](http://fras.uk/ml/large%20prior-free%20models/transformer-vae/2020/08/13/Transformers-as-Variational-Autoencoders.html)).


## How to use from the 🤗/transformers library


Add model repo as a submodule:


```bash
git submodule add https://github.com/Fraser-Greenlee/t5-vae-flax.git t5_vae_flax
```


```python
from transformers import AutoTokenizer
from t5_vae_flax.src.t5_vae import FlaxT5VaeForAutoencoding

tokenizer = AutoTokenizer.from_pretrained("t5-base")

model = FlaxT5VaeForAutoencoding.from_pretrained("flax-community/t5-vae-wiki")
```


## Setup


Run `setup_tpu_vm_venv.sh` to setup a virtual enviroment on a TPU VM for training.


model name: t5-vae-wiki


model author: flax-community


tags: jax, transformer_vae, en, transformers, vae, license:apache-2.0


language: sw
widget:


- text: "Si kila mwenye makucha <mask> simba."
datasets:
- flax-community/swahili-safi


## RoBERTa in Swahili


This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by [HuggingFace](https://huggingface.co). All training was done on a TPUv3-8 VM sponsored by the Google Cloud team.


## How to use


```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("flax-community/roberta-swahili")

model = AutoModelForMaskedLM.from_pretrained("flax-community/roberta-swahili")

print(round((model.num_parameters())/(1000*1000)),"Million Parameters")

105 Million Parameters
```


#### **Training Data**:


This model was trained on [Swahili Safi](https://huggingface.co/datasets/flax-community/swahili-safi)


#### **Results**:


```
Eval metrics: {'f1': 86%}
```


This [model](https://huggingface.co/flax-community/roberta-swahili-news-classification) was fine-tuned based off this model for the
[Zindi News Classification Challenge](https://zindi.africa/hackathons/ai4d-swahili-news-classification-challenge)


#### **More Details**:


For more details and Demo please check [HF Swahili Space](https://huggingface.co/spaces/flax-community/Swahili)


model name: roberta-swahili


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, sw, dataset:flax-community/swahili-safi, transformers, autonlp_compatible, infinity_compatible


language: da
license: cc-by-4.0
tags:


- scandinavian
- roberta
pipeline_tag: fill-mask
widget:
- text: På biblioteket kan du låne en <mask>.


# Scandinavian Roberta Base - MC4


## Description


This is a sample reference model for Flax/Jax training using only on the MC4. It is trained for roughly three day on a TPU v3-8. Training procedure...


My description


model name: roberta-base-scandinavian


model author: flax-community


tags: pytorch, jax, tensorboard, roberta, fill-mask, da, transformers, license:cc-by-4.0, scandinavian, autonlp_compatible, infinity_compatible


# CLIP-Vision-mBART50 Seq2Seq Encoder-Decoder Model


Pretrained CLIP-Vision-mBART50 pre-trained on subset of translated Conceptual-12M image-text pairs using a seq2seq model training objective. 2.5M cleaned English image-text pairs are translated using Marian Model for respective languages to 2.5M examples each in English, French, German and Spanish. We trained CLIP-Vision-mBART50 model during community week hosted by Huggingface 🤗 using JAX/Flax.


## Model description


CLIP-Vision-mBART50 is a modified transformers model which takes in visual embeddings from CLIP-Vision transformer and feeds into the `encoder_hidden_states` of a mBART50 decoder. This is done for deep cross-modal interaction via `cross-attention` between the two modes. The decoder then predicts logits for the `input_ids` provided and can be used for generation.


## Intended uses & limitations❗


You can use the raw model for encoder decoder network where you want the encoder to encode images and decoder to decode text.


Note that this model is primarily aimed at being fine-tuned on tasks like multi-lingual/mono-lingual image captioning.


### How to use❓


You will need to clone the model from [here](https://github.com/gchhablani/multilingual-image-captioning). An example of usage is shown below:


```python
from torchvision.io import read_image
import numpy as  np
import os, wget
from transformers import CLIPProcessor, MBart50TokenizerFast
from model.flax_clip_vision_mbart.modeling_clip_vision_mbart import FlaxCLIPVisionMBartForConditionalGeneration
img = wget("http://images.cocodataset.org/val2017/000000397133.jpg")
img = read_image(img) # reading image
clip_processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')
clip_outputs = clip_processor(images=img)
clip_outputs['pixel_values'][0] = clip_outputs['pixel_values'][0].transpose(1,2,0) # Need to transpose images as model expected channel last images.
tokenizer = MBart50TokenizerFast.from_pretrained('facebook/mbart-large-50"')
model = FlaxCLIPVisionBertForMaskedLM.from_pretrained('flax-community/clip-vit-base-patch32_mbart-large-50')
output_ids = model.generate(batch["pixel_values"], forced_bos_token_id=tokenizer.lang_code_to_id["es_XX"], num_beams=4, max_length=64).sequences  # "es_XX is the language code in which you want the translation
# en_XX: English, fr_XX: French, es_XX: Spanish, de_DE: Deutsch
output_string = tokenizer.batch_decode(output_ids.reshape(-1, 64), skip_special_tokens=True, max_length=64)
output_string  # Un restaurante u otro lugar para comer en el Hotel
```


## Training data 🏋🏻♂


The Multi-lingual image captioning model was trained on a subset of Conceptual 12M dataset by Google:

[Conceptual 12M](https://github.com/google-research-datasets/conceptual-12m), Introduced by Changpinyo et al. in [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://arxiv.org/abs/2102.08981).


The translated dataset can be downloaded from [conceptual-12m-multilingual-marian](https://huggingface.co/datasets/flax-community/conceptual-12m-multilingual-marian). We do not provide images as we do not own any of them. One can download images from the `image_url` section of the original Conceptual 12M dataset.


## Data Cleaning 🧹


Though the original dataset contains 12M image-text pairs, a lot of the URLs are invalid now, and in some cases, images are corrupt or broken. We remove such examples from our data, which leaves us with approximately 10M image-text pairs.


#### **Train set:**


Total data: 10010625 captions, 2502656 images <br>


Language-wise captions distribution:

English: 2502656

Spanish: 2502656

Deutsch: 2502656

French: 2502656


#### **Validation set**


Total data: 110592 captions, 27648 images <br>


Language-wise captions distribution:

English: 27648

Spanish: 27648

Deutsch: 27648

French: 27648


## Training procedure 👨🏻💻


### Training


The model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) **8 v3 TPU cores** for 42K steps with a batch size of 128 and a sequence length of 128. The
optimizer used is Adam with a learning rate of 3e-4, β1 = 0.9, β2 = 0.98 and
ε = 1e-8, a weight decay of 0.01, learning rate warmup for 1,000 steps and linear decay of the learning
rate after.


We tracked experiments using Tensorboard which can be found in `Training Metrics` tab. BLEU scores for languages other than English might be wrongly tracked but the model gives good performance in other languages too as evident from the evaluation scores.


#### **Pretraining Results 📊**


Our model reached **eval loss of ~2.6** around ~60k steps. Here are the BLEU scores (out of 1) for different languages:


|Language                  |BLEU-1|BLEU-2|BLEU-3|BLEU-4|
|--------------------------|------|------|------|------|
|English                   | 0.13083| 0.08887| 0.06681 | 0.04899|
|Spanish                   | 0.15981| 0.09858| 0.06918| 0.04776|
|German                    | 0.14234| 0.09817| 0.07405| 0.0515|
|French                    | 0.13021| 0.08862| 0.06598| 0.04647|


Model used: ckpt-51999/


In order to reproduce the results, one can use the [evaluation script](https://github.com/gchhablani/multilingual-image-captioning/blob/main/evaluation.py) available in this project's repository.


## **App Demo**


You can try out our model on 🤗 Huggingface's spaces 🪐 :
[Streamlit app of Multi-lingual Image Captioning model on Huggingface Spaces](https://huggingface.co/spaces/flax-community/multilingual-image-captioning)


## Team Members


- Bhavitvya Malik [@bhavitvyamalik](https://github.com/bhavitvyamalik)
- Gunjan Chhablani [@gchhablani](https://github.com/gchhablani)


## Credits


Thanks to Huggingface 🤗 & Google JAX/FLAX team for such a wonderful community week. Big thanks to [@patrickvonplaten](https://github.com/patrickvonplaten) and [@patil-suraj](https://github.com/patil-suraj) for helping us with our solution during the community week.


<img src=https://pbs.twimg.com/media/E443fPjX0AY1BsR.jpg:large>


model name: clip-vit-base-patch32_mbart-large-50


model author: flax-community


task: text2text-generation


tags: jax, tensorboard, clip-vision-mbart, text2text-generation, arxiv:2102.08981, transformers, autonlp_compatible


# GPT-Code-Clippy-125M-Code-Search-All


> **Please refer to our new [GitHub Wiki](https://github.com/ncoop57/gpt-code-clippy/wiki) which documents our efforts in detail in creating the open source version of GitHub  Copilot**


## Model Description


GPT-CC-125M-Code-Search is a [GPT-Neo-125M model](https://huggingface.co/EleutherAI/gpt-neo-125M) finetuned using causal language modeling on all languages in the [CodeSearchNet Challenge dataset](https://huggingface.co/datasets/code_search_net). This model is specialized to autocomplete methods in multiple programming languages.


## Training data


[CodeSearchNet Challenge dataset](https://huggingface.co/datasets/code_search_net).


## Training procedure


The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_flax.py).


```bash
./run_clm_flax.py \
    --output_dir $HOME/gpt-neo-125M-code-search-all \
    --model_name_or_path="EleutherAI/gpt-neo-125M" \
    --dataset_name code_search_net \
    --dataset_config_name="all" \
    --do_train --do_eval \
    --block_size="512" \
    --per_device_train_batch_size="32" \
    --per_device_eval_batch_size="64" \
    --preprocessing_num_workers="8" \
    --learning_rate="1.2e-4" \
    --num_train_epochs 20 \
    --warmup_steps 3000 \
    --adam_beta1="0.9" \
    --adam_beta2="0.95" \
    --weight_decay="0.1" \
    --overwrite_output_dir \
    --logging_steps="25" \
    --eval_steps="500" \
    --push_to_hub="False" \
    --report_to="all" \
    --dtype="bfloat16" \
    --skip_memory_metrics="True" \
    --save_steps="500" \
    --save_total_limit 10 \
    --report_to="wandb" \
    --run_name="gpt-neo-125M-code-search-all"
```


## Intended Use and Limitations


The model is finetuned methods from several languages and is intended to autocomplete methods given some prompt (method signature and docstring).


### How to use


You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:


```py

from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-neo-125M-code-clippy-code-search-all")

tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-neo-125M-code-clippy-code-search-all")

prompt = """def greet(name):
  '''A function to greet user. Given a user name it should say hello'''
"""

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)

start = input_ids.size(1)

out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2,

                     early_stopping=True, eos_token_id=tokenizer.eos_token_id, )

print(tokenizer.decode(out[0][start:]))

```


### Limitations and Biases


The model is intended to be used for research purposes and comes with no guarantees of quality of generated code.


GPT-CC is finetuned from GPT-Neo and might have inherited biases and limitations from it. See [GPT-Neo model card](https://huggingface.co/EleutherAI/gpt-neo-125M#limitations-and-biases) for details.


## Eval results


Coming soon...


model name: gpt-neo-125M-code-search-all


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt_neo, text-generation, transformers


language: da
widget:


- text: "Jeg elsker livet"


# GPT2-svenska-wikipedia


A Danish GPT2 style model trained using Flax CLM pipeline on the Danish
part of the wiki40b dataset.


## Model series


This model is part of a series of models training on TPU with Flax Jax during Huggingface Flax/Jax challenge.


## Gpt models


## Swedish Gpt


## Swedish gpt wiki


# Nordic gpt wiki


## Dansk gpt wiki


## Norsk gpt wiki


## Roberta models


## Nordic Roberta Wiki


## Swe Roberta Wiki Oscar


## Roberta Swedish Scandi


## Roberta Swedish


## Swedish T5 model


## Data cleaning and preprocessing


The data was cleaned and preprocessed using the following script. Make sure to install depencies for beam_runner to make the dataset work.


```python
from datasets import load_dataset
def load_and_clean_wiki():
    dataset = load_dataset('wiki40b', 'da', beam_runner='DirectRunner', split="train")
    #dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner')
    dataset = dataset.remove_columns(['wikidata_id', 'version_id'])
    filtered_dataset = dataset.map(filter_wikipedia)
    # filtered_dataset[:3]
    # print(filtered_dataset[:3])
    return filtered_dataset

def filter_wikipedia(batch):
    batch["text"] = " ".join(batch["text"].split("\
_START_SECTION_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_PARAGRAPH_\
"))
    batch["text"] = " ".join(batch["text"].split("_NEWLINE_"))
    batch["text"] = " ".join(batch["text"].split("\xa0"))
    return batch
```


## Training script


The following training script was used to train the model.


```bash
./run_clm_flax.py     --output_dir="${MODEL_DIR}"     --model_type="gpt2"     --config_name="${MODEL_DIR}"     --tokenizer_name="${MODEL_DIR}"     --dataset_name="wiki40b"     --dataset_config_name="da"     --do_train --do_eval     --block_size="512"     --per_device_train_batch_size="64"     --per_device_eval_batch_size="64"     --learning_rate="5e-3" --warmup_steps="1000"     --adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01"     --overwrite_output_dir     --num_train_epochs="20"     --logging_steps="500"     --save_steps="1000"     --eval_steps="2500"     --push_to_hub
```


model name: dansk-gpt-wiki


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, da, transformers


# wav2vec2-base-turkish


model name: wav2vec2-base-turkish


tags: tensorboard


language: sv
widget:


- text: "Jag är en svensk språkmodell."


# GPT2-svenska-wikipedia


A swedish GPT2 style model trained using Flax CLM pipeline on the Swedish
part of the wiki40b dataset.


## Model series


## Gpt models


## Swedish Gpt


## Swedish gpt wiki


# Nordic gpt wiki


## Dansk gpt wiki


## Norsk gpt wiki


## Roberta models


## Nordic Roberta Wiki


## Swe Roberta Wiki Oscar


## Roberta Swedish Scandi


## Roberta Swedish


## Swedish T5 model


## Data cleaning and preprocessing


```python
from datasets import load_dataset
def load_and_clean_wiki():
    dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner', split="train")
    #dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner')
    dataset = dataset.remove_columns(['wikidata_id', 'version_id'])
    filtered_dataset = dataset.map(filter_wikipedia)
    # filtered_dataset[:3]
    # print(filtered_dataset[:3])
    return filtered_dataset

def filter_wikipedia(batch):
    batch["text"] = " ".join(batch["text"].split("\
_START_SECTION_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_PARAGRAPH_\
"))
    batch["text"] = " ".join(batch["text"].split("_NEWLINE_"))
    batch["text"] = " ".join(batch["text"].split("\xa0"))
    return batch
```


## Training script


```bash
./run_clm_flax.py     --output_dir="${MODEL_DIR}"     --model_type="gpt2"     --config_name="${MODEL_DIR}"     --tokenizer_name="${MODEL_DIR}"     --dataset_name="wiki40b"     --dataset_config_name="sv"     --do_train --do_eval     --block_size="512"     --per_device_train_batch_size="64"     --per_device_eval_batch_size="64"     --learning_rate="5e-3" --warmup_steps="1000"     --adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01"     --overwrite_output_dir     --num_train_epochs="20"     --logging_steps="500"     --save_steps="1000"     --eval_steps="2500"     --push_to_hub
```


model name: swe-gpt-wiki


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, sv, transformers


language: sv
license: cc-by-4.0
tags:


- swedish
- roberta
pipeline_tag: fill-mask
widget:
- text: Meninged med livet är <mask>.


# Nordic Roberta Wikipedia


## Description


Nord roberta model trainined on the swedish danish and norwegian wikipedia.


## Evaluation


Evaluation on Named Entity recognition in Danish.


I finetuned each model on 3 epochs on DaNE, repeated it 5 times for each model, and calculated 95% confidence intervals for the means. Here are the results:


xlm-roberta-base : 88.01 +- 0.43


flax-community/nordic-roberta-wiki: 85.75 +- 0.69 (this model)


Maltehb/danish-bert-botxo: 85.38 +- 0.55


flax-community/roberta-base-danish:  80.14 +- 1.47


flax-community/roberta-base-scandinavian : 78.03 +- 3.02


Maltehb/-l-ctra-danish-electra-small-cased: 57.87 +- 3.19


NbAiLab/nb-bert-base : 30.24 +- 1.21


Randomly initialised RoBERTa model: 19.79 +- 2.00


Evaluation on Sentiment analysis in Dansish
Here are the results on test set, where each model has been trained 5 times, and the “+-” refers to a 95% confidence interval of the mean score:


Maltehb/danish-bert-botxo: 65.19 +- 0.53


NbAiLab/nb-bert-base : 63.80 +- 0.77


xlm-roberta-base : 63.55 +- 1.59


flax-community/nordic-roberta-wiki : 56.46 +- 1.77


flax-community/roberta-base-danish : 54.73 +- 8.96


flax-community/roberta-base-scandinavian : 44.28 +- 9.21


Maltehb/-l-ctra-danish-electra-small-cased : 47.78 +- 12.65


Randomly initialised RoBERTa model: 36.96 +- 1.02


Maltehb/roberta-base-scandinavian : 33.65 +- 8.32


## Model series


This model is part of a series of models training on TPU with Flax Jax during Huggingface Flax/Jax challenge.


## Gpt models


## Swedish Gpt


## Swedish gpt wiki


# Nordic gpt wiki


## Dansk gpt wiki


## Norsk gpt wiki


## Roberta models


## Nordic Roberta Wiki


## Swe Roberta Wiki Oscar


## Roberta Swedish Scandi


## Roberta Swedish


## Swedish T5 model


model name: nordic-roberta-wiki


model author: flax-community


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, feature-extraction, sv, transformers, license:cc-by-4.0, swedish, fill-mask, autonlp_compatible, infinity_compatible


language: si
tags:


- Sinhala
- text-generation
- gpt2
datasets:
- mc4


# Sinhala GPT2 trained on MC4 (manually cleaned)


### Overview


This is a smaller GPT2 model trained on [MC4](https://github.com/allenai/allennlp/discussions/5056) Sinhala dataset. As Sinhala is one of those low resource languages, there are only a handful of models been trained. So, this would be a great place to start training for more downstream tasks.


This model uses a manually cleaned version of MC4 dataset which can be found [here](https://huggingface.co/datasets/keshan/clean-si-mc4). Although the dataset is relatively small ~3GB. The finetuned model on [news articles](https://huggingface.co/keshan/sinhala-gpt2-newswire) generates good and acceptable results.


## Model Specification


The model chosen for training is GPT2 with the following specifications:


1. vocab_size=50257
1. n_embd=768
1. n_head=12
1. n_layer=12
1. n_positions=1024


## How to Use


You can use this model directly with a pipeline for causal language modeling:


```py
from transformers import pipeline
generator = pipeline('text-generation', model='flax-community/Sinhala-gpt2')
generator("මම", max_length=50, num_return_sequences=5)
```


model name: Sinhala-gpt2


task: feature-extraction


tags: pytorch, tf, jax, tensorboard, gpt2, feature-extraction, si, dataset:mc4, transformers, Sinhala, text-generation


# Transformer-VAE (flax) (WIP)


A Transformer-VAE made using flax.


Done as part of Huggingface community training ([see forum post](https://discuss.huggingface.co/t/train-a-vae-to-interpolate-on-english-sentences/7548)).


Builds on T5, using an autoencoder to convert it into an MMD-VAE.


[See training logs.](https://wandb.ai/fraser/flax-vae)


## ToDo


- [ ] Basic training script working. (Fraser + Theo)
- [ ] Add MMD loss (Theo)
- [ ] Save a wikipedia sentences dataset to Huggingface (see original https://github.com/ChunyuanLI/Optimus/blob/master/data/download_datasets.md) (Mina)
- [ ] Make a tokenizer using the OPTIMUS tokenized dataset.
- [ ] Train on the OPTIMUS wikipedia sentences dataset.
- [ ] Make Huggingface widget interpolating sentences! (???) https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects#how-to-build-a-demo


Optional ToDos:


- [ ] Add Funnel transformer encoder to FLAX (don't need weights).
- [ ] Train a Funnel-encoder + T5-decoder transformer VAE.
- [ ] Additional datasets:
- [ ] Poetry (https://www.gwern.net/GPT-2#data-the-project-gutenberg-poetry-corpus)
- [ ] 8-bit music (https://github.com/chrisdonahue/LakhNES)


## Setup


Follow all steps to install dependencies from https://cloud.google.com/tpu/docs/jax-quickstart-tpu-vm


- [ ] Find dataset storage site.
- [ ] Ask JAX team for dataset storage.


model name: transformer-vae


model author: flax-community


tags:


# RobIt


**RobIt** is a RoBERTa-base model for Italian. It has been trained from scratch on the Italian portion of the OSCAR dataset using [Flax](https://github.com/google/flax), including training scripts.


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Team members


- Prateek Agrawal (prateekagrawal)
- Tanay Mehta (yotanay)
- Shreya Gupta (Sheyz-max)
- Ruchi Bhatia (ruchi798)


## Dataset :


- config : **unshuffled_deduplicated_it**
- Size of downloaded dataset files: **26637.62 MB**
- Size of the generated dataset: **70661.48 MB**
- Total amount of disk used: **97299.10 MB**


## Useful links


- [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
- [Community Week README](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/README.md)
- [Community Week thread](https://discuss.huggingface.co/t/robit-pretrain-roberta-base-from-scratch-in-italian/7564)
- [Community Week channel](https://discord.gg/NTyQNUNs)
- [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
- [Model Repository](https://huggingface.co/flax-community/robit-roberta-base-it/)


model name: robit-roberta-base-it


task: fill-mask


tags: jax, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


language: id
tags:


- indonesian-roberta-large
license: mit
datasets:
- oscar
widget:
- text: "Budi telat ke sekolah karena ia <mask>."


## Indonesian RoBERTa Large


Indonesian RoBERTa Large is a masked language model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. It was trained on the [OSCAR](https://huggingface.co/datasets/oscar) dataset, specifically the `unshuffled_deduplicated_id` subset. The model was trained from scratch and achieved an evaluation loss of 4.801 and an evaluation accuracy of 29.8%.


This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by HuggingFace. All training was done on a TPUv3-8 VM, sponsored by the Google Cloud team.


All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/flax-community/indonesian-roberta-large/tree/main) tab, as well as the [Training metrics](https://huggingface.co/flax-community/indonesian-roberta-large/tensorboard) logged via Tensorboard.


## Model


| Model                      | #params | Arch.   | Training/Validation data (text)            |
| -------------------------- | ------- | ------- | ------------------------------------------ |
| `indonesian-roberta-large` | 355M    | RoBERTa | OSCAR `unshuffled_deduplicated_id` Dataset |


## Evaluation Results


The model was trained for 10 epochs and the following is the final result once the training ended.


| train loss | valid loss | valid accuracy | total time |
| ---------- | ---------- | -------------- | ---------- |
| 5.19       | 4.801      | 0.298          | 2:8:32:28  |


## How to Use


### As Masked Language Model


```python
from transformers import pipeline

pretrained_name = "flax-community/indonesian-roberta-large"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Budi sedang <mask> di sekolah.")
```


### Feature Extraction in PyTorch


```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "flax-community/indonesian-roberta-large"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "Budi sedang berada di sekolah."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Team Members


- Wilson Wongso ([@w11wo](https://hf.co/w11wo))
- Steven Limcorn ([@stevenlimcorn](https://hf.co/stevenlimcorn))
- Samsul Rahmadani ([@munggok](https://hf.co/munggok))
- Chew Kok Wah ([@chewkokwah](https://hf.co/chewkokwah))


model name: indonesian-roberta-large


model author: flax-community


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, id, dataset:oscar, arxiv:1907.11692, transformers, indonesian-roberta-large, license:mit, autonlp_compatible, infinity_compatible


language:


- en
pipeline_tag: text-to-image
inference: false


## DALL·E mini - Generate images from text


<img style="text-align:center; display:block;" src="https://raw.githubusercontent.com/borisdayma/dalle-mini/main/img/logo.png" width="200">


* [Technical Report](https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-mini--Vmlldzo4NjIxODA)
* [Demo](https://huggingface.co/spaces/flax-community/dalle-mini)


### Model Description


This is an attempt to replicate OpenAI's [DALL·E](https://openai.com/blog/dall-e/), a model capable of generating arbitrary images from a text prompt that describes the desired result.


![DALL·E mini demo screenshot](img/demo_screenshot.png)


This model's architecture is a simplification of the original, and leverages previous open source efforts and available pre-trained models. Results have lower quality than OpenAI's, but the model can be trained and used on less demanding hardware. Our training was performed on a single TPU v3-8 for a few days.


### Components of the Architecture


The system relies on the Flax/JAX infrastructure, which are ideal for TPU training. TPUs are not required, both Flax and JAX run very efficiently on GPU backends.


The main components of the architecture include:


* An encoder, based on [BART](https://arxiv.org/abs/1910.13461). The encoder transforms a sequence of input text tokens to a sequence of image tokens. The input tokens are extracted from the text prompt by using the model's tokenizer. The image tokens are a fixed-length sequence, and they represent indices in a VQGAN-based pre-trained codebook.
* A decoder, which converts the image tokens to image pixels. As mentioned above, the decoder is based on a [VQGAN model](https://compvis.github.io/taming-transformers/).


The model definition we use for the encoder can be downloaded from our [Github repo](https://github.com/borisdayma/dalle-mini). The encoder is represented by the class `CustomFlaxBartForConditionalGeneration`.


To use the decoder, you need to follow the instructions in our accompanying VQGAN model in the hub, [flax-community/vqgan_f16_16384](https://huggingface.co/flax-community/vqgan_f16_16384).


### How to Use


The easiest way to get familiar with the code and the models is to follow the inference notebook we provide in our [github repo](https://github.com/borisdayma/dalle-mini/blob/main/dev/inference/inference_pipeline.ipynb). For your convenience, you can open it in Google Colaboratory: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/borisdayma/dalle-mini/blob/main/dev/inference/inference_pipeline.ipynb)


If you just want to test the trained model and see what it comes up with, please visit [our demo](https://huggingface.co/spaces/flax-community/dalle-mini), available in 🤗 Spaces.


### Additional Details


Our [report](https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-mini--Vmlldzo4NjIxODA) contains more details about how the model was trained and shows many examples that demonstrate its capabilities.


model name: dalle-mini


model author: flax-community


task: text-to-image


tags: jax, bart, text2text-generation, en, arxiv:1910.13461, transformers, text-to-image, autonlp_compatible


tags:


- vision


# Model Card: clip-rsicd


## Model Details


This model is a fine-tuned [CLIP by OpenAI](https://huggingface.co/openai/clip-vit-base-patch32). It is designed with an aim to improve zero-shot image classification, text-to-image and image-to-image retrieval specifically on remote sensing images.


### Model Date


July 2021


### Model Type


The base model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.


### Model Version


We release several checkpoints for `clip-rsicd` model. Refer to [our github repo](https://github.com/arampacha/CLIP-rsicd#evaluation-results) for performance metrics on zero-shot classification for each of those.


### Training


To reproduce the fine-tuning procedure one can use released [script](https://github.com/arampacha/CLIP-rsicd/blob/master/run_clip_flax_tv.py).
The model was trained using batch size 1024, adafactor optimizer with linear warmup and decay with peak learning rate 1e-4 on 1 TPU-v3-8.
Full log of the training run can be found on [WandB](https://wandb.ai/wandb/hf-flax-clip-rsicd/runs/2dj1exsw).


### Demo


Check out the model text-to-image and image-to-image capabilities using [this demo](https://huggingface.co/spaces/sujitpal/clip-rsicd-demo).


### Documents


- [Fine-tuning CLIP on RSICD with HuggingFace and flax/jax on colab using TPU](https://colab.research.google.com/github/arampacha/CLIP-rsicd/blob/master/nbs/Fine_tuning_CLIP_with_HF_on_TPU.ipynb)


### Use with Transformers


```python3
from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("flax-community/clip-rsicd-v2")
processor = CLIPProcessor.from_pretrained("flax-community/clip-rsicd-v2")

url = "https://raw.githubusercontent.com/arampacha/CLIP-rsicd/master/data/stadium_1.jpg"
image = Image.open(requests.get(url, stream=True).raw)

labels = ["residential area", "playground", "stadium", "forest", "airport"]
inputs = processor(text=[f"a photo of a {l}" for l in labels], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
for l, p in zip(labels, probs[0]):
    print(f"{l:<16} {p:.4f}")
```


[Try it on colab](https://colab.research.google.com/github/arampacha/CLIP-rsicd/blob/master/nbs/clip_rsicd_zero_shot.ipynb)


## Model Use


### Intended Use


The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification.


In addition, we can imagine applications in defense and law enforcement, climate change and global warming, and even some consumer applications. A partial list of applications can be found [here](https://github.com/arampacha/CLIP-rsicd#applications). In general we think such models can be useful as digital assistants for humans engaged in searching through large collections of images.


We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.


#### Primary intended uses


The primary intended users of these models are AI researchers.


We primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models.


## Data


The model was trained on publicly available remote sensing image captions datasets. Namely [RSICD](https://github.com/201528014227051/RSICD_optimal), [UCM](https://mega.nz/folder/wCpSzSoS#RXzIlrv--TDt3ENZdKN8JA) and [Sydney](https://mega.nz/folder/pG4yTYYA#4c4buNFLibryZnlujsrwEQ). More information on the datasets used can be found on [our project page](https://github.com/arampacha/CLIP-rsicd#dataset).


## Performance and Limitations


### Performance


| Model-name                       | k=1   | k=3   | k=5   | k=10  |
| -------------------------------- | ----- | ----- | ----- | ----- |
| original CLIP                    | 0.572 | 0.745 | 0.837 | 0.939 |
| clip-rsicd-v2 (this model)       | **0.883** | **0.968** | **0.982** | **0.998** |


## Limitations


The model is fine-tuned on RSI data but can contain some biases and limitations of the original CLIP model. Refer to [CLIP model card](https://huggingface.co/openai/clip-vit-base-patch32#limitations) for details on those.


model name: clip-rsicd-v2


model author: flax-community


task: feature-extraction


tags: pytorch, jax, clip, feature-extraction, transformers, vision


# Cosmos QA (gpt2)


> This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/train-a-gpt2-model-for-contextual-common-sense-reasoning-using-the-cosmos-qa-dataset/7463), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Team Members


-Rohan V Kashyap ([Rohan](https://huggingface.co/Rohan))
-Vivek V Kashyap ([Vivek](https://huggingface.co/Vivek))


## Dataset


[Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://huggingface.co/datasets/cosmos_qa).This dataset contains a set of 35,600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions.Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly.The questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people's everyday narratives.


### Example


```json
{"Context":["It's a very humbling experience when you need someone
to dress you every morning, tie your shoes, and put your hair
up. Every menial task takes an unprecedented amount of effort.
It made me appreciate Dan even more. But anyway I shan't
dwell on this (I'm not dying after all) and not let it detract from
my lovely 5 days with my friends visiting from Jersey."],

"Question":["What's a possible reason the writer needed someone to
dress him every morning?"],

"Multiple Choice":["A: The writer doesn't like putting effort into these tasks.",
"B: The writer has a physical disability.",
"C: The writer is bad at doing his own hair.",
"D: None of the above choices."]
"link":"https://arxiv.org/pdf/1909.00277.pdf"
}
```


## How to use


```bash
# Installing requirements
pip install transformers
pip install datasets
```


```python
from model_file import FlaxGPT2ForMultipleChoice
from datasets import Dataset
model_path="flax-community/gpt2-Cosmos"
model = FlaxGPT2ForMultipleChoice.from_pretrained(model_path,input_shape=(1,4,1))

dataset=Dataset.from_csv('./')

def preprocess(example):
    example['context&question']=example['context']+example['question']
    example['first_sentence']=[example['context&question'],example['context&question'],example['context&question'],example['context&question']]
    example['second_sentence']=example['answer0'],example['answer1'],example['answer2'],example['answer3']
    return example

dataset=dataset.map(preprocess)

def tokenize(examples):
    a=tokenizer(examples['first_sentence'],examples['second_sentence'],padding='max_length',truncation=True,max_length=256,return_tensors='jax')
    a['labels']=examples['label']
    return a

dataset=dataset.map(tokenize)

input_id=jnp.array(dataset['input_ids'])
att_mask=jnp.array(dataset['attention_mask'])

outputs=model(input_id,att_mask)

final_output=jnp.argmax(outputs,axis=-1)

print(f"the predction of the dataset : {final_output}")
```


```
The Correct answer:-Option 1
```


## Preprocessing


The texts are tokenized using the GPT2 tokenizer.To feed the inputs of multiple choice we concatenated context and question as first input and all the 4 possible choices as the second input to our tokenizer.


## Evaluation


The following tables summarize the scores obtained by the **GPT2-CosmosQA**.The ones  marked as (^) are the baseline models.


|      Model      |  Dev Acc | Test Acc  |
|:---------------:|:-----:|:-----:|
| BERT-FT Multiway^| 68.3.| 68.4  |
|  GPT-FT   ^    |  54.0 | 54.4. |
| GPT2-CosmosQA  | 60.3 | 59.7 |


## Inference


This project was mainly  to test the  common sense understanding of the  GPT2-model.We finetuned on a Dataset known as CosmosQ requires reasoning beyond the exact text spans in the context.The above results shows that GPT2 model is doing better than most of the base line models given that  it only used to predict the next word in the pre-training objective.


## Credits


Huge thanks to Huggingface 🤗 & Google Jax/Flax team for such a wonderful community week. Especially for providing such massive computing resource. Big thanks to [@patil-suraj](https://github.com/patil-suraj) & [@patrickvonplaten](https://github.com/patrickvonplaten) for mentoring during whole week.


model name: gpt2-Cosmos


model author: flax-community


tags: jax, tensorboard, gpt2, arxiv:1909.00277, transformers


widget:


- text: "शुभ प्रभात। आशा करता हूं कि आपका <mask> शुभ हो"


roberta-pretraining-hindi


model name: roberta-pretraining-hindi


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


language: ta
datasets:


- oscar
- IndicNLP
widget:
- text: 'ஒரு ஊரிலே ஒரு காக்கைக்கு'


# GPT2-Tamil


This repository is created as part of the Flax/Jax community week by Huggingface. The aim of this project is to pretrain a language model using GPT-2 specifically for Tamil language.


## Setup:


To setup the project, run the following command,


```python
pip install -r requirements.txt
```


## Model:


Pretrained model on Tamil language using a causal language modeling (CLM) objective.


## Dataset Used:


The GTP-2 model is trained on [oscar dataset - ta](https://huggingface.co/datasets/oscar)


## Intended uses & limitations:


You can use the raw model for next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=gpt) to look for fine-tuned versions on a task that interests you.


## How to pretrain the model:


To perform training, do the following steps,


- Export the model directory (where you want to store the model artifacts like config, tokenizer, etc.)


```python
>>> export MODEL_DIR=<model_dir>
```


- Create the config.json by running the following command,


```python
>>> python src/create_config.py
```


- Create the tokenizer by running the following command,


```python
>>> python src/train_tokenizer.py
```


- Once the config and tokenizer is created, run the following script to start training the flax model


```python
>>> python scripts/train_gpt2-oscar-tamil.sh
```


## How to use:


To perform language generation using the model, pipeline can be used directly.


- First convert the flax model to pytorch using the following command,


```python
python src/convert_flax_to_pytorch.py
```


- Use the following snippet to perform language generation,


```python
 >>> from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline
 >>> model_name = 'abinayam/gpt-2-tamil'
 >>> model = AutoModelWithLMHead.from_pretrained(model_name)
 >>> tokenizer = AutoTokenizer.from_pretrained(model_name)
 >>> set_seed(42)
 >>> input_text = "ஒரு ஊரிலே ஒரு காக்கைக்கு"
 >>> max_len = 300
 >>> no_seq = 5
 >>> generator = pipeline('text-generation', model=model, tokenizer=tokenizer)
 >>> sequence = generator(input_text, max_length=max_len, num_return_sequences=no_seq)
```


model name: gpt-2-tamil


model author: flax-community


task: text-generation


tags: pytorch, tensorboard, gpt2, text-generation, ta, dataset:oscar, dataset:IndicNLP, transformers


# GPT-Neo-125M-Code-Clippy-Dedup


> **Please refer to our new [GitHub Wiki](https://github.com/ncoop57/gpt-code-clippy/wiki) which documents our efforts in detail in creating the open source version of GitHub  Copilot**


## Model Description


PT-Neo-125M-Code-Clippy-Dedup is a [GPT-Neo-125M model](https://huggingface.co/EleutherAI/gpt-neo-125M) finetuned using causal language modeling on our deduplicated version of the Code Clippy Data dataset, which was scraped from public Github repositories (more information in the provided link). This model is specialized to autocomplete methods in multiple programming languages.


## Training data


[Code Clippy Data dataset](https://huggingface.co/datasets/code_search_net).


## Training procedure


In this model's training we tried to stabilize the training by limiting the types of files we were using to train to only those that contained file extensions for popular programming languages as our dataset contains other types of files as well such as `.txt` or project configuration files. We used the following extensions to filter by:


The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_streaming_filter_flax.py).


```bash
./run_clm_streaming_filter_flax.py \
    --output_dir $HOME/gpt-neo-125M-code-clippy-dedup \
    --model_name_or_path="EleutherAI/gpt-neo-125M" \
    --dataset_name $HOME/gpt-code-clippy/data_processing/code_clippy_filter.py \
    --data_dir $HOME/code_clippy_data/code_clippy_dedup_data \
    --text_column_name="text" \
    --do_train --do_eval \
    --block_size="2048" \
    --per_device_train_batch_size="8" \
    --per_device_eval_batch_size="16" \
    --preprocessing_num_workers="8" \
    --learning_rate="1e-4" \
    --max_steps 100000 \
    --warmup_steps 2000 \
    --decay_steps 30000 \
    --adam_beta1="0.9" \
    --adam_beta2="0.95" \
    --weight_decay="0.1" \
    --overwrite_output_dir \
    --logging_steps="25" \
    --eval_steps="500" \
    --push_to_hub="False" \
    --report_to="all" \
    --dtype="bfloat16" \
    --skip_memory_metrics="True" \
    --save_steps="500" \
    --save_total_limit 10 \
    --gradient_accumulation_steps 16 \
    --report_to="wandb" \
    --run_name="gpt-neo-125M-code-clippy-dedup-filtered-no-resize-2048bs" \
    --max_eval_samples 2000 \
    --save_optimizer true
```


## Intended Use and Limitations


The model is finetuned text file from github repositories (mostly programming languages but also markdown and other project related files).


### How to use


You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:


```py

from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-neo-125M-code-clippy-dedup")

tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-neo-125M-code-clippy-dedup")

prompt = """def greet(name):
  '''A function to greet user. Given a user name it should say hello'''
"""

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)

start = input_ids.size(1)

out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2,

                     early_stopping=True, eos_token_id=tokenizer.eos_token_id, )

print(tokenizer.decode(out[0][start:]))

```


### Limitations and Biases


The model is intended to be used for research purposes and comes with no guarantees of quality of generated code.


The paper ["Evaluating Large Language Models Trained on Code"](https://arxiv.org/abs/2107.03374) from OpenAI has a good discussion on what the impact of a large language model trained on code could be. Therefore, some parts of their discuss are highlighted here as it pertains to this dataset and models that may be trained from it. **As well as some differences in views from the paper, particularly around legal implications**.