Update README.md
Browse files
README.md
CHANGED
@@ -31,7 +31,11 @@ The vision of OpenCSG is to empower every industry, every company, and every ind
|
|
31 |
|
32 |
## Model Description
|
33 |
|
34 |
-
Phi-2 is a 2.7 billion-parameter Transformer model trained on augmented data sources, including synthetic NLP texts and filtered websites, alongside existing data used for Phi-1.5. It performs nearly state-of-the-art on benchmarks for common sense, language understanding, and logical reasoning, despite having fewer than 13 billion parameters.
|
|
|
|
|
|
|
|
|
35 |
<br>
|
36 |
|
37 |
This is the repository for the base 13B version finetuned based on [CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf).
|
@@ -39,6 +43,8 @@ This is the repository for the base 13B version finetuned based on [CodeLlama-13
|
|
39 |
| Model Size | Base Model |
|
40 |
| --- | ----------------------------------------------------------------------------- |
|
41 |
| phi-2 | [opencsg/Opencsg-phi-2-v0.1](https://huggingface.co/opencsg/opencsg-phi-2-v0.1) |
|
|
|
|
|
42 |
| 7B | [opencsg/Opencsg-CodeLlama-7b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-7b-v0.1) |
|
43 |
| 13B | [opencsg/Opencsg-CodeLlama-13b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-13b-v0.1) |
|
44 |
| 34B | [opencsg/Opencsg-CodeLlama-34b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-34b-v0.1) |
|
@@ -63,6 +69,11 @@ To simplify the comparison, we chosed the Pass@1 metric for the Python language,
|
|
63 |
|
64 |
| Model | HumanEval python pass@1 |
|
65 |
| --- |----------------------------------------------------------------------------- |
|
|
|
|
|
|
|
|
|
|
|
66 |
| CodeLlama-7b-hf | 30.5%|
|
67 |
| **opencsg-CodeLlama-7b-v0.1** | **43.9%** |
|
68 |
| **opencsg-CodeLlama-7b-v0.2** | **50.0%** |
|
@@ -79,6 +90,8 @@ To simplify the comparison, we chosed the Pass@1 metric for the Python language,
|
|
79 |
|
80 |
|
81 |
|
|
|
|
|
82 |
**TODO**
|
83 |
- We will provide more benchmark scores on fine-tuned models in the future.
|
84 |
- We will provide different practical problems to evaluate the performance of fine-tuned models in the field of software engineering.
|
@@ -87,34 +100,25 @@ To simplify the comparison, we chosed the Pass@1 metric for the Python language,
|
|
87 |
|
88 |
# Model Usage
|
89 |
|
90 |
-
```
|
91 |
-
from transformers import AutoTokenizer
|
92 |
-
import transformers
|
93 |
import torch
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
94 |
|
95 |
-
|
96 |
-
|
97 |
-
|
98 |
-
pipeline = transformers.pipeline(
|
99 |
-
"text-generation",
|
100 |
-
model=model,
|
101 |
-
torch_dtype=torch.float16,
|
102 |
-
device_map="auto",
|
103 |
-
)
|
104 |
-
input_text = "#write a quick sort algorithm."
|
105 |
-
sequences = pipeline(
|
106 |
-
input_text,
|
107 |
-
do_sample=False,
|
108 |
-
top_k=10,
|
109 |
-
temperature=0.1,
|
110 |
-
top_p=0.95,
|
111 |
-
num_return_sequences=1,
|
112 |
-
eos_token_id=tokenizer.eos_token_id,
|
113 |
-
max_length=256,
|
114 |
-
)
|
115 |
-
for seq in sequences:
|
116 |
-
print(seq['generated_text'][len(input_text):])
|
117 |
```
|
|
|
118 |
# Training
|
119 |
|
120 |
## Hardware
|
@@ -155,8 +159,12 @@ OpenCSG 的愿景是让每个行业、每个公司、每个人都拥有自己的
|
|
155 |
|
156 |
## 模型介绍
|
157 |
|
158 |
-
|
159 |
-
|
|
|
|
|
|
|
|
|
160 |
<br>
|
161 |
|
162 |
这是基于 [CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf) 进行微调的模型版本。
|
@@ -186,6 +194,11 @@ HumanEval 是评估模型在代码生成方面性能的最常见的基准,尤
|
|
186 |
|
187 |
| 模型 | HumanEval python pass@1 |
|
188 |
| --- |----------------------------------------------------------------------------- |
|
|
|
|
|
|
|
|
|
|
|
189 |
| CodeLlama-7b-hf | 30.5%|
|
190 |
| **opencsg-CodeLlama-7b-v0.1** | **43.9%** |
|
191 |
| **opencsg-CodeLlama-7b-v0.2** | **50.0%** |
|
@@ -206,34 +219,23 @@ HumanEval 是评估模型在代码生成方面性能的最常见的基准,尤
|
|
206 |
|
207 |
# 模型使用
|
208 |
|
209 |
-
|
210 |
-
```python
|
211 |
-
from transformers import AutoTokenizer
|
212 |
-
import transformers
|
213 |
import torch
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
214 |
|
215 |
-
|
216 |
-
|
217 |
-
|
218 |
-
pipeline = transformers.pipeline(
|
219 |
-
"text-generation",
|
220 |
-
model=model,
|
221 |
-
torch_dtype=torch.float16,
|
222 |
-
device_map="auto",
|
223 |
-
)
|
224 |
-
input_text = "#write a quick sort algorithm."
|
225 |
-
sequences = pipeline(
|
226 |
-
input_text,
|
227 |
-
do_sample=False,
|
228 |
-
top_k=10,
|
229 |
-
temperature=0.1,
|
230 |
-
top_p=0.95,
|
231 |
-
num_return_sequences=1,
|
232 |
-
eos_token_id=tokenizer.eos_token_id,
|
233 |
-
max_length=256,
|
234 |
-
)
|
235 |
-
for seq in sequences:
|
236 |
-
print(seq['generated_text'][len(input_text):])
|
237 |
```
|
238 |
# 训练
|
239 |
|
|
|
31 |
|
32 |
## Model Description
|
33 |
|
34 |
+
Phi-2 is a 2.7 billion-parameter Transformer model trained on augmented data sources, including synthetic NLP texts and filtered websites, alongside existing data used for Phi-1.5. It performs nearly state-of-the-art on benchmarks for common sense, language understanding, and logical reasoning, despite having fewer than 13 billion parameters.
|
35 |
+
Unlike some models, Phi-2 hasn't been fine-tuned through reinforcement learning from human feedback. The goal of this open-source model is to enable research into safety challenges like reducing toxicity, understanding biases, enhancing controllability, etc.
|
36 |
+
|
37 |
+
|
38 |
+
opencsg-phi-2-v0.1 is a series of models based on phi-2 that have been fine-tuned using full-parameter tuning methods.
|
39 |
<br>
|
40 |
|
41 |
This is the repository for the base 13B version finetuned based on [CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf).
|
|
|
43 |
| Model Size | Base Model |
|
44 |
| --- | ----------------------------------------------------------------------------- |
|
45 |
| phi-2 | [opencsg/Opencsg-phi-2-v0.1](https://huggingface.co/opencsg/opencsg-phi-2-v0.1) |
|
46 |
+
|
47 |
+
|
48 |
| 7B | [opencsg/Opencsg-CodeLlama-7b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-7b-v0.1) |
|
49 |
| 13B | [opencsg/Opencsg-CodeLlama-13b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-13b-v0.1) |
|
50 |
| 34B | [opencsg/Opencsg-CodeLlama-34b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-34b-v0.1) |
|
|
|
69 |
|
70 |
| Model | HumanEval python pass@1 |
|
71 |
| --- |----------------------------------------------------------------------------- |
|
72 |
+
|
73 |
+
| phi-2 | 48.2% |
|
74 |
+
| **opencsg-phi-2-v0.1** |**54.3**|
|
75 |
+
|
76 |
+
|
77 |
| CodeLlama-7b-hf | 30.5%|
|
78 |
| **opencsg-CodeLlama-7b-v0.1** | **43.9%** |
|
79 |
| **opencsg-CodeLlama-7b-v0.2** | **50.0%** |
|
|
|
90 |
|
91 |
|
92 |
|
93 |
+
|
94 |
+
|
95 |
**TODO**
|
96 |
- We will provide more benchmark scores on fine-tuned models in the future.
|
97 |
- We will provide different practical problems to evaluate the performance of fine-tuned models in the field of software engineering.
|
|
|
100 |
|
101 |
# Model Usage
|
102 |
|
103 |
+
```
|
|
|
|
|
104 |
import torch
|
105 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
106 |
+
|
107 |
+
torch.set_default_device("cuda")
|
108 |
+
|
109 |
+
model = AutoModelForCausalLM.from_pretrained("opencsg/opencsg-phi-2-v0.1", torch_dtype="auto", trust_remote_code=True)
|
110 |
+
tokenizer = AutoTokenizer.from_pretrained("opencsg/opencsg-phi-2-v0.1", trust_remote_code=True)
|
111 |
+
|
112 |
+
inputs = tokenizer('''def print_prime(n):
|
113 |
+
"""
|
114 |
+
Print all primes between 1 and n
|
115 |
+
"""''', return_tensors="pt", return_attention_mask=False)
|
116 |
|
117 |
+
outputs = model.generate(**inputs, max_length=200)
|
118 |
+
text = tokenizer.batch_decode(outputs)[0]
|
119 |
+
print(text)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
120 |
```
|
121 |
+
|
122 |
# Training
|
123 |
|
124 |
## Hardware
|
|
|
159 |
|
160 |
## 模型介绍
|
161 |
|
162 |
+
|
163 |
+
Phi-2是一个拥有27亿参数的Transformer模型,使用了经过增强的数据源进行训练,包括合成的NLP文本和经过筛选的网站,同时还使用了Phi-1.5使用的现有数据。尽管参数少于130亿,但它在常识、语言理解和逻辑推理的基准测试中表现出了接近最先进的水平。
|
164 |
+
与一些模型不同,Phi-2没有通过人类反馈的强化学习进行微调。这个开源模型的目标是促进对安全挑战的研究,如减少毒性、理解偏见、增强可控性等。
|
165 |
+
|
166 |
+
opencsg-phi-2-v0.1是是一系列基于phi-2的通过全参数微调方法进行调优的模型。
|
167 |
+
|
168 |
<br>
|
169 |
|
170 |
这是基于 [CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf) 进行微调的模型版本。
|
|
|
194 |
|
195 |
| 模型 | HumanEval python pass@1 |
|
196 |
| --- |----------------------------------------------------------------------------- |
|
197 |
+
|
198 |
+
| phi-2 | 48.2% |
|
199 |
+
| **opencsg-phi-2-v0.1** |**54.3**|
|
200 |
+
|
201 |
+
|
202 |
| CodeLlama-7b-hf | 30.5%|
|
203 |
| **opencsg-CodeLlama-7b-v0.1** | **43.9%** |
|
204 |
| **opencsg-CodeLlama-7b-v0.2** | **50.0%** |
|
|
|
219 |
|
220 |
# 模型使用
|
221 |
|
222 |
+
```
|
|
|
|
|
|
|
223 |
import torch
|
224 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
225 |
+
|
226 |
+
torch.set_default_device("cuda")
|
227 |
+
|
228 |
+
model = AutoModelForCausalLM.from_pretrained("opencsg/opencsg-phi-2-v0.1", torch_dtype="auto", trust_remote_code=True)
|
229 |
+
tokenizer = AutoTokenizer.from_pretrained("opencsg/opencsg-phi-2-v0.1", trust_remote_code=True)
|
230 |
+
|
231 |
+
inputs = tokenizer('''def print_prime(n):
|
232 |
+
"""
|
233 |
+
Print all primes between 1 and n
|
234 |
+
"""''', return_tensors="pt", return_attention_mask=False)
|
235 |
|
236 |
+
outputs = model.generate(**inputs, max_length=200)
|
237 |
+
text = tokenizer.batch_decode(outputs)[0]
|
238 |
+
print(text)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
239 |
```
|
240 |
# 训练
|
241 |
|