shibing624 commited on
Commit
c61f3ec
1 Parent(s): 6899b00

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +132 -1
README.md CHANGED
@@ -1,3 +1,134 @@
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - zh
4
+ tags:
5
+ - chatglm
6
+ - pytorch
7
+ - zh
8
+ - Text2Text-Generation
9
+ license: "apache-2.0"
10
+ widget:
11
+ - text: "对下面中文拼写纠错:\n少先队员因该为老人让坐。\n答:"
12
+
13
  ---
14
+
15
+ # Chinese Spelling Correction LoRA Model
16
+ ChatGLM3-6B中文纠错LoRA模型
17
+
18
+ `shibing624/chatglm3-6b-csc-chinese-lora` evaluate test data:
19
+
20
+ The overall performance of shibing624/chatglm3-6b-csc-chinese-lora on CSC **test**:
21
+
22
+ |prefix|input_text|target_text|pred|
23
+ |:-- |:--- |:--- |:-- |
24
+ |对下面文本纠错:|少先队员因该为老人让坐。|少先队员应该为老人让座。|少先队员应该为老人让座。|
25
+
26
+ 在CSC测试集上生成结果纠错准确率高,由于是基于ChatGLM3-6B模型,结果常常能带给人惊喜,不仅能纠错,还带有句子润色和改写功能。
27
+
28
+
29
+ ## Usage
30
+
31
+ 本项目开源在 pycorrector 项目:[textgen](https://github.com/shibing624/pycorrector),可支持ChatGLM原生模型和LoRA微调后的模型,通过如下命令调用:
32
+
33
+ Install package:
34
+ ```shell
35
+ pip install -U pycorrector
36
+ ```
37
+
38
+ ```python
39
+ from pycorrector.gpt.gpt_model import GptModel
40
+ model = GptModel("chatglm", "THUDM/chatglm3-6b", peft_name="shibing624/chatglm3-6b-csc-chinese-lora")
41
+ r = model.predict(["对下面文本纠错:\n少先队员因该为老人让坐。"])
42
+ print(r) # ['少先队员应该为老人让座。']
43
+ ```
44
+
45
+ ## Usage (HuggingFace Transformers)
46
+ Without [pycorrector](https://github.com/shibing624/pycorrector), you can use the model like this:
47
+
48
+ First, you pass your input through the transformer model, then you get the generated sentence.
49
+
50
+ Install package:
51
+ ```
52
+ pip install transformers
53
+ ```
54
+
55
+ ```python
56
+ import sys
57
+ from peft import PeftModel
58
+ from transformers import AutoModel, AutoTokenizer
59
+
60
+ sys.path.append('..')
61
+
62
+ model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True, device_map='auto')
63
+ model = PeftModel.from_pretrained(model, "shibing624/chatglm3-6b-csc-chinese-lora")
64
+ model = model.half().cuda() # fp16
65
+ tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True)
66
+
67
+ sents = ['对下面中文拼写纠错:\n少先队员因该为老人让坐。',
68
+ '对下面中文拼写纠错:\n下个星期,我跟我朋唷打算去法国玩儿。']
69
+ for s in sents:
70
+ response = model.chat(tokenizer, s, max_length=128, eos_token_id=tokenizer.eos_token_id)
71
+ print(response)
72
+ ```
73
+
74
+ output:
75
+ ```shell
76
+ 少先队员应该为老人让座。
77
+ 下个星期,我跟我朋友打算去法国玩儿。
78
+ ```
79
+
80
+
81
+ 模型文件组成:
82
+ ```
83
+ chatglm3-6b-csc-chinese-lora
84
+ ├── adapter_config.json
85
+ └── adapter_model.bin
86
+ ```
87
+
88
+ #### 训练参数:
89
+
90
+ ![loss](train_loss.png)
91
+
92
+ - num_epochs: 5
93
+ - per_device_train_batch_size: 6
94
+ - learning_rate: 2e-05
95
+ - best steps: 25100
96
+ - train_loss: 0.0834
97
+ - lr_scheduler_type: linear
98
+ - base model: THUDM/chatglm3-6b
99
+ - warmup_steps: 50
100
+ - "save_strategy": "steps"
101
+ - "save_steps": 500
102
+ - "save_total_limit": 10
103
+ - "bf16": false
104
+ - "fp16": true
105
+ - "optim": "adamw_torch"
106
+ - "ddp_find_unused_parameters": false
107
+ - "gradient_checkpointing": true
108
+ - max_seq_length: 512
109
+ - max_length: 512
110
+ - prompt_template_name: vicuna
111
+ - 6 * V100 32GB, training 48 hours
112
+
113
+ ### 训练数据集
114
+ 训练集包括以下数据:
115
+
116
+ - 中文拼写纠错数据集:https://huggingface.co/datasets/shibing624/CSC
117
+ - 中文语法纠错数据集:https://github.com/shibing624/pycorrector/tree/llm/examples/data/grammar
118
+ - 通用GPT4问答数据集:https://huggingface.co/datasets/shibing624/sharegpt_gpt4
119
+
120
+
121
+ 如果需要训练GPT模型,请参考[https://github.com/shibing624/pycorrector](https://github.com/shibing624/pycorrector)
122
+
123
+
124
+
125
+ ## Citation
126
+
127
+ ```latex
128
+ @software{pycorrector,
129
+ author = {Ming Xu},
130
+ title = {pycorrector: Text Error Correction Tool},
131
+ year = {2023},
132
+ url = {https://github.com/shibing624/pycorrector},
133
+ }
134
+ ```