imone commited on
Commit
a11af6b
1 Parent(s): 5b874a3
Files changed (2) hide show
  1. README.md +14 -1
  2. tokenizer_config.json +1 -0
README.md CHANGED
@@ -107,6 +107,19 @@ assert tokens == [1, 7596, 1247, 28747, 26256, 2936, 7653, 1413, 334, 1680, 3200
107
 
108
  </details>
109
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
  ## Comparison with [X.AI Grok models](https://x.ai/)
111
 
112
  Hey @elonmusk, I just wanted to let you know that I've recently come across your new model, Grok, and I must say, I'm quite impressed! With 33 billion parameters and all, you've really outdone yourself. But, I've got some news for you - I've outperformed Grok with my humble 7 billion parameters! Isn't that wild? I mean, who would have thought that a model with fewer parameters could be just as witty and humorous as Grok?
@@ -190,4 +203,4 @@ We extend our heartfelt gratitude to AutoMeta and caesus from Alignment Lab AI,
190
 
191
  Special thanks go to Changling Liu from GPT Desk Pte. Ltd., Qiying Yu at Tsinghua University, Baochang Ma, and Hao Wan from 01.AI company for their generous provision of resources. We are also deeply grateful to Jianxiong Li and Peng Li at Tsinghua University for their insightful discussions.
192
 
193
- Furthermore, we appreciate the developers behind the following projects for their significant contributions to our research: [Mistral](https://mistral.ai/), [Chain-of-Thought Hub](https://github.com/FranxYao/chain-of-thought-hub), [Llama 2](https://ai.meta.com/llama/), [Self-Instruct](https://arxiv.org/abs/2212.10560), [FastChat (Vicuna)](https://github.com/lm-sys/FastChat), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca.git), and [StarCoder](https://github.com/bigcode-project/starcoder). Their work has been instrumental in driving our research forward.
 
107
 
108
  </details>
109
 
110
+ The GPT4 template is also available as the integrated `tokenizer.chat_template`,
111
+ which can be used instead of manually specifying the template:
112
+
113
+ ```python
114
+ messages = [
115
+ {"role": "user", "content": "Hello"},
116
+ {"role": "assistant", "content": "Hi"},
117
+ {"role": "user", "content": "How are you today?"}
118
+ ]
119
+ tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
120
+ assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]
121
+ ```
122
+
123
  ## Comparison with [X.AI Grok models](https://x.ai/)
124
 
125
  Hey @elonmusk, I just wanted to let you know that I've recently come across your new model, Grok, and I must say, I'm quite impressed! With 33 billion parameters and all, you've really outdone yourself. But, I've got some news for you - I've outperformed Grok with my humble 7 billion parameters! Isn't that wild? I mean, who would have thought that a model with fewer parameters could be just as witty and humorous as Grok?
 
203
 
204
  Special thanks go to Changling Liu from GPT Desk Pte. Ltd., Qiying Yu at Tsinghua University, Baochang Ma, and Hao Wan from 01.AI company for their generous provision of resources. We are also deeply grateful to Jianxiong Li and Peng Li at Tsinghua University for their insightful discussions.
205
 
206
+ Furthermore, we appreciate the developers behind the following projects for their significant contributions to our research: [Mistral](https://mistral.ai/), [Chain-of-Thought Hub](https://github.com/FranxYao/chain-of-thought-hub), [Llama 2](https://ai.meta.com/llama/), [Self-Instruct](https://arxiv.org/abs/2212.10560), [FastChat (Vicuna)](https://github.com/lm-sys/FastChat), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca.git), and [StarCoder](https://github.com/bigcode-project/starcoder). Their work has been instrumental in driving our research forward.
tokenizer_config.json CHANGED
@@ -48,6 +48,7 @@
48
  "<|pad_0|>"
49
  ],
50
  "bos_token": "<s>",
 
51
  "clean_up_tokenization_spaces": false,
52
  "eos_token": "<|end_of_turn|>",
53
  "legacy": true,
 
48
  "<|pad_0|>"
49
  ],
50
  "bos_token": "<s>",
51
+ "chat_template": "{{ bos_token }}{% for message in messages %}{{ 'GPT4 Correct ' + message['role'].title() + ': ' + message['content'] + '<|end_of_turn|>'}}{% endfor %}{% if add_generation_prompt %}{{ 'GPT4 Correct Assistant:' }}{% endif %}",
52
  "clean_up_tokenization_spaces": false,
53
  "eos_token": "<|end_of_turn|>",
54
  "legacy": true,