Update README

README.md

@@ -0,0 +1,77 @@
+---
+language: Chinese
+widget: 
+- text: "最美的不是下雨天，是曾与你躲过雨的屋檐"
+
+
+---
+
+
+# Chinese GPT2 Model
+
+## Model description
+
+The model is used to generate Chinese lyrics. You can download the model  from the link [gpt2-chinese-lyric](https://huggingface.co/uer/gpt2-chinese-lyric)
+
+## How to use
+
+You can use the model directly with a pipeline for text generation:
+
+```python
+>>> from transformers import BertTokenizer, GPT2LMHeadModel, TextGenerationPipeline
+>>> tokenizer = BertTokenizer.from_pretrained("uer/gpt2-chinese-lyric")
+>>> model = GPT2LMHeadModel.from_pretrained("uer/gpt2-chinese-lyric")
+>>> text_generator = TextGenerationPipeline(model, tokenizer)   
+>>> text_generator("最美的不是下雨天，是曾与你躲过雨的屋檐", max_length=100, do_sample=True)
+    [{'generated_text': '最美的不是下雨天，是曾与你躲过雨的屋檐 ， 下 课 铃 声 响 起 的 瞬 间 ， 我 们 的 笑 脸 ， 有 太 多 回 忆 在 浮 现 ， 是 你 总 在 我 身 边 ， 不 知 道 会 不 会 再 见 ， 从 现 在 开 始 到 永 远 ， 想 说 的 语 言 凝 结 成 一 句 ， 不 管 我 们 是 否 能 够 兑 现 ， 想 说 的 语 言 凝 结'}]
+```
+
+
+
+## Training data
+
+Training data contains 150,000 Chinese lyrics which are collected by [Chinese-Lyric-Corpus](https://github.com/gaussic/Chinese-Lyric-Corpus) and [MusicLyricChatbot](https://github.com/liuhuanyong/MusicLyricChatbot) projects
+
+## Training procedure
+
+The model is pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud TI-ONE](https://cloud.tencent.com/product/tione/). We pre-train 100,000 steps with a sequence length of 512 on the basis of the model [gpt2-base-chinese-cluecorpussmall](https://huggingface.co/uer/gpt2-base-chinese-cluecorpussmall)
+
+```
+python3 preprocess.py --corpus_path corpora/lyric.txt \
+                      --vocab_path models/google_zh_vocab.txt \
+                      --dataset_path lyric_lm_seq512_dataset.pt \
+                      --seq_length 512 --processes_num 32 --target lm 
+```
+
+```
+python3 pretrain.py --dataset_path lyric_lm_seq512_dataset.pt \
+					--pretrained_model_path gpt2-base-chinese-cluecorpussmall/pytorch_model.bin\
+                    --vocab_path models/google_zh_vocab.txt \
+                    --output_model_path models/lyric_gpt2_seq512_model.bin \
+                    --config_path models/bert_base_config.json --learning_rate 5e-5 \
+                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 --tie_weight \
+                    --embedding word_pos --remove_embedding_layernorm \
+                    --encoder transformer --mask causal --layernorm_positioning pre \
+                    --target lm --batch_size 64 --total_steps 100000 \
+                    --save_checkpoint_steps 10000 --report_steps 5000
+```
+
+Finally, we convert the pre-trained model into Huggingface's format:
+
+```
+python3 scripts/convert_gpt2_from_uer_to_huggingface.py --input_model_path lyric_gpt2_seq512_model.bin-100000 \
+                                                        --output_model_path pytorch_model.bin \
+                                                        --layers_num 12
+```
+
+### BibTeX entry and citation info
+
+```
+@article{zhao2019uer,
+  title={UER: An Open-Source Toolkit for Pre-training Models},
+  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
+  journal={EMNLP-IJCNLP 2019},
+  pages={241},
+  year={2019}
+}
+```

config.json

@@ -20,7 +20,7 @@
   "task_specific_params": {
     "text-generation": {
       "do_sample": true,
-      "max_length": 512
+      "max_length": 128
     }
   },
   "tokenizer_class": "BertTokenizer",

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ef616ae6d77ded35cea7e4bdd3dbc78e2b52d31223eab9f435893a4ba0216b61
+oid sha256:98d71975ce622315019ba39f0815b00a7b2b00773e43194cd3fa923abf48ffa1
 size 419348431

tf_model.h5

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf24c2c912cc60f43082d2211c4bd0ea84d7d6c5cbdaa7501d15d00defba0520
+oid sha256:7db5c6e8ca0b5846f12c5bd30822c17de26781fa0f117ff0ea7673b6b321e875
 size 406876496