# Twitter-roBERTa-base_SemEval18_Emodetection


This is a Twitter-roBERTa-base model trained on ~7000 tweets in English annotated for 11 emotion categories in [SemEval-2018 Task 1: Affect in Tweets: SubTask 5: Emotion Classification](https://competitions.codalab.org/competitions/17751).


Run the classifier on the test set of the competition:


```python
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModel
from torch.utils.data import DataLoader
import torch
import pandas as pd

# choose GPU when available
device = 'cuda' if torch.cuda.is_available() else 'cpu'

tokenizer = AutoTokenizer.from_pretrained("cardiffnlp/twitter-roberta-base",model_max_length=512)

# build custom model with classification layer on top and a dropout layer before
class RobertaClass(torch.nn.Module):

    def __init__(self):
        super(RobertaClass, self).__init__()
        self.l1 = AutoModel.from_pretrained("cardiffnlp/twitter-roberta-base",return_dict=False)
        self.l2 = torch.nn.Dropout(0.3)
        self.l3 = torch.nn.Linear(768, 11)

    def forward(self, input_ids, attention_mask):
        _, output_1= self.l1(input_ids=input_ids, attention_mask=attention_mask)
        output_2 = self.l2(output_1)
        output = self.l3(output_2)

        return output

model_name="twitter-roberta-base_semeval18_emodetection/pytorch_model.bin"

model=RobertaClass()

model.load_state_dict(torch.load(model_name,map_location=torch.device(device)))

model.eval()

# run on more than 1 GPU
model = torch.nn.DataParallel(model)

model.to(device)

twnames=['anger','anticipation','disgust','fear','joy','love','optimism','pessimism','sadness','surprise','trust']

# load from hugging face dataset hub
testset_raw = load_dataset('sem_eval_2018_task_1','subtask5.english',split='test')

# remove old columns
testset=testset_raw.remove_columns(twnames+["ID"])

# tokenize
testset_tokenized = testset.map(lambda e: tokenizer(e['Tweet'], truncation=True, padding='max_length'), batched=True)

testset_tokenized=testset_tokenized.remove_columns("Tweet")

testset_tokenized.set_format(type='torch', columns=['input_ids', 'attention_mask'])

outfile="predicted_2018-E-c-En-test-gold.txt"

MAX_LEN = 512
VALID_BATCH_SIZE = 8
# set batch size according to available RAM
# VALID_BATCH_SIZE = 1000

# set num_workers for parallel processing
inference_params = {'batch_size': VALID_BATCH_SIZE,
                'shuffle': False,
                # 'num_workers': 1
                }

inference_loader = DataLoader(testset_tokenized, **inference_params)

open(outfile,"w").close()
with torch.no_grad():
    # change lines for progress manager
    # for _, data in tqdm(enumerate(inference_loader, 0),total=len(inference_loader)):
    for _, data in enumerate(inference_loader, 0):
        outputs = model(input_ids=data['input_ids'],attention_mask=data['attention_mask'])
        fin_outputs=torch.sigmoid(outputs).cpu().detach().numpy().tolist()
        pd.DataFrame(fin_outputs).to_csv(outfile,index=False,header=False,sep="\t",mode='a')

# # dataset from file (one text per line)
# from datasets import Dataset

# with open(linesoftextfile,"rb") as textfile:
#     textdict={"text":[x.decode().rstrip("\n") for x in textfile.readlines()]}

# inference_dataset=Dataset.from_dict(textdict)
# del(textdict)
```


model name: twitter-roberta-base_semeval18_emodetection


model author: maxpe


task: text-classification


tags: pytorch, roberta, text-classification, transformers, infinity_compatible


# BERTIN-roBERTa-base-Spanish_SemEval18_Emodetection


This is a BERTIN-roBERTa-base-Spanish model trained on ~3500 tweets in Spanish annotated for 11 emotion categories in [SemEval-2018 Task 1: Affect in Tweets: SubTask 5: Emotion Classification](https://competitions.codalab.org/competitions/17751).


```python
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModel
from torch.utils.data import DataLoader
import torch
import pandas as pd

# choose GPU when available
device = 'cuda' if torch.cuda.is_available() else 'cpu'

tokenizer = AutoTokenizer.from_pretrained("bertin-project/bertin-roberta-base-spanish",model_max_length=512)

# build custom model with classification layer on top and a dropout layer before
class RobertaClass(torch.nn.Module):

    def __init__(self):
        super(RobertaClass, self).__init__()
        self.l1 = AutoModel.from_pretrained("bertin-project/bertin-roberta-base-spanish",return_dict=False)
        self.l2 = torch.nn.Dropout(0.3)
        self.l3 = torch.nn.Linear(768, 11)

    def forward(self, input_ids, attention_mask):
        _, output_1= self.l1(input_ids=input_ids, attention_mask=attention_mask)
        output_2 = self.l2(output_1)
        output = self.l3(output_2)

        return output

model_name="bertin-roberta-base-spanish_semeval18_emodetection/pytorch_model.bin"

model=RobertaClass()

model.load_state_dict(torch.load(model_name,map_location=torch.device(device)))

model.eval()

# run on more than 1 GPU
model = torch.nn.DataParallel(model)

model.to(device)

twnames=['anger','anticipation','disgust','fear','joy','love','optimism','pessimism','sadness','surprise','trust']

# load from hugging face dataset hub
testset_raw = load_dataset('sem_eval_2018_task_1','subtask5.spanish',split='test')

# remove old columns
testset=testset_raw.remove_columns(twnames+["ID"])

# tokenize
testset_tokenized = testset.map(lambda e: tokenizer(e['Tweet'], truncation=True, padding='max_length'), batched=True)

testset_tokenized=testset_tokenized.remove_columns("Tweet")

testset_tokenized.set_format(type='torch', columns=['input_ids', 'attention_mask'])

outfile="predicted_2018-E-c-Es-test-gold.txt"

MAX_LEN = 512
VALID_BATCH_SIZE = 8
# set batch size according to available RAM
# VALID_BATCH_SIZE = 1000

# set num_workers for parallel processing
inference_params = {'batch_size': VALID_BATCH_SIZE,
                'shuffle': False,
                # 'num_workers': 1
                }

inference_loader = DataLoader(testset_tokenized, **inference_params)

open(outfile,"w").close()
with torch.no_grad():
    # change lines for progress manager
    # for _, data in tqdm(enumerate(inference_loader, 0),total=len(inference_loader)):
    for _, data in enumerate(inference_loader, 0):
        outputs = model(input_ids=data['input_ids'],attention_mask=data['attention_mask'])
        fin_outputs=torch.sigmoid(outputs).cpu().detach().numpy().tolist()
        pd.DataFrame(fin_outputs).to_csv(outfile,index=False,header=False,sep="\t",mode='a')

# # dataset from file (one text per line)
# from datasets import Dataset

# with open(linesoftextfile,"rb") as textfile:
#     textdict={"text":[x.decode().rstrip("\n") for x in textfile.readlines()]}

# inference_dataset=Dataset.from_dict(textdict)
# del(textdict)
```


model name: bertin-roberta-base-spanish_semeval18_emodetection


tags:


- image-classification
- pytorch
- huggingpics
metrics:
- accuracy


model-index:


- name: architectural_styles
results:
  - task:
name: Image Classification
type: image-classification
metrics:
    - name: Accuracy
type: accuracy
value: 0.7796609997749329


### What style is that?


This model can help identify five architectural styles that were prominent in the early to mid 20th century. Check back for updates including more architectural styles and more accurate predictions as this model diversifies and improves its training.


Upload a photograph of a building to the File Uploader on the right. The Image Classifier will predict its architectural style using a database of over 700 images. Scroll down to read more about each style.


### Classical Revival (1895 - 1950)


The Classical Revival or Neoclassical style is one of the most commonly seen across the state and the country. This style was inspired by the World's Columbian Exposition in Chicago held in 1893 which promoted a renewed interest in the classical forms. This style encompasses many different styles, including Colonial Revival, Greek Revival, Neoclassical Revival and Mediterranean Revival. Colonial Revival is most commonly used in residential dwellings, while Greek and Neoclassical Revival styles are commonly used in commercial buildings like banks, post offices, and municipal buildings.


![classical revival architecture](images/ex_classical_revival_architecture.jpg)


#### Queen Anne (1880-1910)


The Queen Anne style was one of a number of popular  architectural styles that emerged in the United States during the Victorian Period. It ranges from high style, like the image pictured here, to more vernacular styles that exhibit the Queen Anne form without its high style architectural details.


![queen anne architecture](images/ex_queen_anne_architecture.jpg)


#### Craftsman Bungalow (1900-1930)


The terms “craftsman” and “bungalow” are often used interchangably, however, “craftsman” refers to the Arts and Crafts movement and is considered an architectural style, whereas “bungalow” is the form of house. Bungalows often exhibit a craftsman style.


![craftsman bungalow architecture](images/ex_craftsman_bungalow_architecture.jpg)


#### Tudor Cottage (1910-1950)


Tudor homes are inspired by the Medieval period and can range is size and style. In general, the Tudor style features steeply pitched roofs, often with a cat-slide roof line, predominately brick construction, sometimes accented with half-timber framing, front-facing, prominently placed brick or stone chimneys, and tall windows with rectangular or diamond-shaped panes. Front doors are typically off-center with a round arch at the top of the door or doorway.


![tudor cottage architecture](images/ex_tudor_cottage_architecture.jpg)


#### Mid-Century Modern Ranch (1930-1970)


The Ranch style originated in southern California in the mid-1930s. In the 1940s, the Ranch was one of the small house types financed by the Federal Housing Administration (FHA), along with Minimal Traditional and other small house styles. The Ranch house began to pick up popularity as the financial controls that encouraged small house building lifted following WWII; by the 1950s it was the most predominant residential style in the country.


![mid-century modern ranch](images/ex_mid-century_modern_ranch.jpg)


This model was created with HuggingPics🤗🖼 Image Classifier!
Make your own!: [the demo on Google Colab](https://colab.research.google.com/github/nateraw/huggingpics/blob/main/HuggingPics.ipynb).


model name: architectural_styles


model author: gatecitypreservation


task: image-classification


tags: pytorch, tensorboard, vit, image-classification, transformers, huggingpics, model-index


Valentino Rossi brought down the curtain on an illustrious 25-year MotoGP career on Sunday at the season-ending Valencia Grand Prix


model name: Rossi


model author: wifis


tags:


language:


- en
- fr
- ro
- de
datasets:
- c4
tags:
- summarization
- translation


## license: apache-2.0


[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html)


Pretraining Dataset: [C4](https://huggingface.co/datasets/c4)


Other Community Checkpoints: [here](https://huggingface.co/models?search=t5)


Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)


Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*


## Abstract


Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


model name: t5-3b


task: translation


tags: pytorch, tf, t5, text2text-generation, en, fr, ro, de, dataset:c4, arxiv:1910.10683, transformers, summarization, translation, license:apache-2.0, autonlp_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-8_H-128


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


pipeline_tag: sentence-similarity
tags:


- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
license: apache-2.0


模型正在测试中


model name: sbert-base-chinese-nli


task: sentence-similarity


tags: pytorch, bert, feature-extraction, sentence-transformers, sentence-similarity, transformers, license:apache-2.0, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-12_H-128


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-12_H-512


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-4_H-768


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: zh
widget:


- text: "北京上个月召开了两会"


# Chinese RoBERTa-Base Models for Text Classification


## Model description


This is the set of 5 Chinese RoBERTa-Base classification models fine-tuned by [UER-py](https://arxiv.org/abs/1909.05658). You can download the 5 Chinese RoBERTa-Base classification models either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo) (in UER-py format), or via HuggingFace from the links below:


|    Dataset     |                           Link                            |
| :-----------: | :-------------------------------------------------------: |
|  **JD full**  |   [**roberta-base-finetuned-jd-full-chinese**](https://huggingface.co/uer/roberta-base-finetuned-jd-full-chinese)   |
| **JD binary** | [**roberta-base-finetuned-jd-binary-chinese**](https://huggingface.co/uer/roberta-base-finetuned-jd-binary-chinese) |
| **Dianping**  |  [**roberta-base-finetuned-dianping-chinese**](https://huggingface.co/uer/roberta-base-finetuned-dianping-chinese)  |
|   **Ifeng**   |     [**roberta-base-finetuned-ifeng-chinese**](https://huggingface.co/uer/roberta-base-finetuned-ifeng-chinese)     |
| **Chinanews** | [**roberta-base-finetuned-chinanews-chinese**](https://huggingface.co/uer/roberta-base-finetuned-chinanews-chinese) |


## How to use


You can use this model directly with a pipeline for text classification (take the case of roberta-base-finetuned-chinanews-chinese):


```python
>>> from transformers import AutoModelForSequenceClassification,AutoTokenizer,pipeline
>>> model = AutoModelForSequenceClassification.from_pretrained('uer/roberta-base-finetuned-chinanews-chinese')
>>> tokenizer = AutoTokenizer.from_pretrained('uer/roberta-base-finetuned-chinanews-chinese')
>>> text_classification = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)
>>> text_classification("北京上个月召开了两会")
    [{'label': 'mainland China politics', 'score': 0.7211663722991943}]
```


## Training data


5 Chinese text classification datasets are used. JD full, JD binary, and Dianping datasets consist of user reviews of different sentiment polarities. Ifeng and Chinanews consist of first paragraphs of news articles of different topic classes. They are collected by [Glyph](https://github.com/zhangxiangxiao/glyph) project and more details are discussed in corresponding [paper](https://arxiv.org/abs/1708.02657).


## Training procedure


Models are fine-tuned by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We fine-tune three epochs with a sequence length of 512 on the basis of the pre-trained model [chinese_roberta_L-12_H-768](https://huggingface.co/uer/chinese_roberta_L-12_H-768). At the end of each epoch, the model is saved when the best performance on development set is achieved. We use the same hyper-parameters on different models.


Taking the case of roberta-base-finetuned-chinanews-chinese


```
python3 run_classifier.py --pretrained_model_path models/cluecorpussmall_roberta_base_seq512_model.bin-250000 \
                          --vocab_path models/google_zh_vocab.txt \
                          --train_path datasets/glyph/chinanews/train.tsv \
                          --dev_path datasets/glyph/chinanews/dev.tsv \
                          --output_model_path models/chinanews_classifier_model.bin \
                          --learning_rate 3e-5 --batch_size 32 --epochs_num 3 --seq_length 512 \
                          --embedding word_pos_seg --encoder transformer --mask fully_visible
```


```
python3 scripts/convert_bert_text_classification_from_uer_to_huggingface.py --input_model_path models/chinanews_classifier_model.bin \
                                                                            --output_model_path pytorch_model.bin \
                                                                            --layers_num 12
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{zhang2017encoding,
  title={Which encoding is the best for text classification in chinese, english, japanese and korean?},
  author={Zhang, Xiang and LeCun, Yann},
  journal={arXiv preprint arXiv:1708.02657},
  year={2017}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: roberta-base-finetuned-chinanews-chinese


model author: uer


task: text-classification


tags: pytorch, tf, jax, bert, text-classification, zh, arxiv:1909.05658, arxiv:1708.02657, transformers, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-10_H-512


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-2_H-512


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-2_H-768


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget:


- text: "北京是[MASK]国的首都。"


# Chinese RoBERTa Miniatures


## Model description


This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).


[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.


You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:


|          |           H=128           |           H=256           |            H=512            |            H=768            |
| -------- | :-----------------------: | :-----------------------: | :-------------------------: | :-------------------------: |
| **L=2**  | [**2/128 (Tiny)**](https://huggingface.co/uer/chinese_roberta_L-2_H-128) |      [2/256](https://huggingface.co/uer/chinese_roberta_L-2_H-256)       |       [2/512](https://huggingface.co/uer/chinese_roberta_L-2_H-512)        |       [2/768](https://huggingface.co/uer/chinese_roberta_L-2_H-768)        |
| **L=4**  |      [4/128](https://huggingface.co/uer/chinese_roberta_L-4_H-128)       | [**4/256 (Mini)**](https://huggingface.co/uer/chinese_roberta_L-4_H-256) | [**4/512 (Small)**](https://huggingface.co/uer/chinese_roberta_L-4_H-512)  |       [4/768](https://huggingface.co/uer/chinese_roberta_L-4_H-768)        |
| **L=6**  |      [6/128](https://huggingface.co/uer/chinese_roberta_L-6_H-128)       |      [6/256](https://huggingface.co/uer/chinese_roberta_L-6_H-256)       |       [6/512](https://huggingface.co/uer/chinese_roberta_L-6_H-512)        |       [6/768](https://huggingface.co/uer/chinese_roberta_L-6_H-768)        |
| **L=8**  |      [8/128](https://huggingface.co/uer/chinese_roberta_L-8_H-128)       |      [8/256](https://huggingface.co/uer/chinese_roberta_L-8_H-256)       | [**8/512 (Medium)**](https://huggingface.co/uer/chinese_roberta_L-8_H-512) |       [8/768](https://huggingface.co/uer/chinese_roberta_L-8_H-768)        |
| **L=10** |     [10/128](https://huggingface.co/uer/chinese_roberta_L-10_H-128)      |     [10/256](https://huggingface.co/uer/chinese_roberta_L-10_H-256)      |      [10/512](https://huggingface.co/uer/chinese_roberta_L-10_H-512)       |      [10/768](https://huggingface.co/uer/chinese_roberta_L-10_H-768)       |
| **L=12** |     [12/128](https://huggingface.co/uer/chinese_roberta_L-12_H-128)      |     [12/256](https://huggingface.co/uer/chinese_roberta_L-12_H-256)      |      [12/512](https://huggingface.co/uer/chinese_roberta_L-12_H-512)       | [**12/768 (Base)**](https://huggingface.co/uer/chinese_roberta_L-12_H-768) |


Here are scores on the devlopment set of six Chinese tasks:


| Model          | Score | douban | chnsenticorp | lcqmc | tnews(CLUE) | iflytek(CLUE) | ocnli(CLUE) |
| -------------- | :---: | :----: | :----------: | :---: | :---------: | :-----------: | :---------: |
| RoBERTa-Tiny   | 72.3  |  83.0  |     91.4     | 81.8  |    62.0     |     55.0      |    60.3     |
| RoBERTa-Mini   | 75.7  |  84.8  |     93.7     | 86.1  |    63.9     |     58.3      |    67.4     |
| RoBERTa-Small  | 76.8  |  86.5  |     93.4     | 86.5  |    65.1     |     59.4      |    69.7     |
| RoBERTa-Medium | 77.8  |  87.6  |     94.8     | 88.1  |    65.6     |     59.5      |    71.2     |
| RoBERTa-Base   | 79.5  |  89.1  |     95.2     | 89.2  |    67.0     |     60.9      |    75.5     |


For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained with the sequence length of 128:


- epochs: 3, 5, 8
- batch sizes: 32, 64
- learning rates: 3e-5, 1e-4, 3e-4


## How to use


You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa-Medium):


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/chinese_roberta_L-8_H-512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]',
     'score': 0.8701988458633423,
     'token': 1266,
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227,
     'token': 1298,
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]',
     'score': 0.0037803512532263994,
     'token': 691,
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183,
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307,
     'token_str': '望'}
]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = BertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L-8_H-512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L-8_H-512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Training data


[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.


## Training procedure


Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.


Taking the case of RoBERTa-Medium


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e-4 --batch_size 64 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --target mlm
```


```
python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin-1000000 \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e-5 --batch_size 16 \
                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
```


Finally, we convert the pre-trained model into Huggingface's format:


```
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin-250000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --target mlm
```


### BibTeX entry and citation info


```
@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```


model name: chinese_roberta_L-10_H-768


model author: uer


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, Chinese, dataset:CLUECorpusSmall, arxiv:1909.05658, arxiv:1908.08962, transformers, autonlp_compatible, infinity_compatible


language: Chinese
datasets: CLUECorpusSmall
widget: