pom commited on
Commit
4ca17e7
1 Parent(s): 54e9ecc

upload models

Browse files
MODEL_LICENSE.pdf ADDED
Binary file (306 kB). View file
 
README.md CHANGED
@@ -1,3 +1,168 @@
1
  ---
2
  license: apache-2.0
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+
4
+ inference: false
5
+
6
  ---
7
+
8
+ # XVERSE-13B-256K
9
+
10
+ ## 更新信息
11
+ **[2024/01/15]** 发布长序列对话模型**XVERSE-13B-256K**,该版本模型最大支持256K窗口长度,约25w字的输入内容,可以协助进行文献总结、报告分析等任务。
12
+ **[2023/11/06]** 发布新版本的 **XVERSE-13B-2** 底座模型和 **XVERSE-13B-Chat-2** 对话模型,相较于原始版本,新版本的模型训练更加充分(从 1.4T 增加到 3.2T),各方面的能力均得到大幅提升,同时新增工具调用能力。
13
+ **[2023/09/26]** 发布 7B 尺寸的 [XVERSE-7B](https://github.com/xverse-ai/XVERSE-7B) 底座模型和 [XVERSE-7B-Chat](https://github.com/xverse-ai/XVERSE-7B) 对话模型,支持在单张消费级显卡部署运行,并保持高性能、全开源、免费可商用。
14
+ **[2023/08/22]** 发布经过指令精调的 XVERSE-13B-Chat 对话模型。
15
+ **[2023/08/07]** 发布 13B 尺寸的 XVERSE-13B 底座模型。
16
+
17
+ ## Update Information
18
+ **[2024/01/15]** Released the long-sequence model **XVERSE-13B-256K**. This model version supports a maximum window length of 256K, accommodating approximately 250,000 words for tasks such as literature summarization and report analysis.
19
+ **[2023/11/06]** The new versions of the **XVERSE-13B-2** base model and the **XVERSE-13B-Chat-2** model have been released. Compared to the original versions, the new models have undergone more extensive training (increasing from 1.4T to 3.2T), resulting in significant improvements in all capabilities, along with the addition of Function Call abilities.
20
+ **[2023/09/26]** Released the [XVERSE-7B](https://github.com/xverse-ai/XVERSE-7B) base model and [XVERSE-7B-Chat](https://github.com/xverse-ai/XVERSE-7B) instruct-finetuned model with 7B size, which support deployment and operation on a single consumer-grade graphics card while maintaining high performance, full open source, and free for commercial use.
21
+ **[2023/08/22]** Released the aligned instruct-finetuned model XVERSE-13B-Chat.
22
+ **[2023/08/07]** Released the XVERSE-13B base model.
23
+
24
+ ## 模型介绍
25
+
26
+ **XVERSE-13B-256K**是[**XVERSE-13B**](https://huggingface.co/xverse/XVERSE-13B)模型经过ABF+继续预训练、NTK+SFT微调后的版本。
27
+
28
+ **XVERSE-13B-256K** 是由深圳元象科技自主研发的支持多语言的大语言模型(Large Language Model),主要应用的技术如下:
29
+
30
+ - **ABF**:ABF的全称是Adjusted Base Frequency,表示将位置编码RoPE(Rotary Position Embedding)的频率从10000修改成500000。别小看这个数字的更改,它可以大幅减少前面序列attention的衰减速度,让后面的序列更好地获取所有序列的信息。
31
+ - **继续预训练**:在XVERSE-13B的基础上,使用20%的预训练数据进行32K的长序列继续预训练。通过少量长序列数据的继续预训练而不是从头开始的长序列预训练,可以大幅减少预训练的训练量。
32
+ - **NTK**:NTK的全称是Neural Tangent Kernel,翻译为神经正切核,是一种用于理解和分析深度神经网络行为的工具。使用了NTK的RoPE可以对RoPE的频率进行动态的插值。在保持分辨率的情况下(高频),进行频域空间的缩放(低频),从而实现位置空间的插值。
33
+ - **SFT数据**:自主构建包含单文本QA,多文本QA,摘要,代码补全等各类长序列数据,序列长度从32K到256K不等。
34
+
35
+ ## Model Introduction
36
+
37
+ **XVERSE-13B-256K** is the long-sequence version of model [**XVERSE-13B**](https://huggingface.co/xverse/XVERSE-13B),
38
+ updated by **Continual-Pre-Training** based on **ABF** and **supervised fine-tuning** based on **NTK**.
39
+
40
+ **XVERSE-13B-256K** is a multilingual large language model, independently developed by Shenzhen Yuanxiang Technology. Below are the main practical techniques:
41
+
42
+ - **ABF**: Adjusted Base Frequency means that changing the frequency of Rotary Position Embedding(RoPE) from 10,000 to 500,000.
43
+ - **Continual-Pre-Training**: Based on XVERSE-13B, 32K long sequence continuation pre-training is conducted using 20% of the pre-training data. This approach significantly reduces the training volume for pre-training by utilizing a small amount of long sequence data for continuation pre-training instead of starting from scratch with long sequence pre-training.
44
+ - **NTK**: Neural Tangent Kernel is a tool used for understanding and analyzing the behavior of deep neural networks. RoPE, employing NTK, enables dynamic interpolation of its frequencies. This involves scaling in the frequency domain while maintaining resolution, thereby achieving spatial interpolation in the positional domain.
45
+ - **Data for SFT**: We autonomously construct a diverse range of long sequence data, encompassing single-document question-answering (QA), multi-document QA, summarization, code completion, and other types. The sequence lengths vary from 32K to 256K.
46
+
47
+ ## 评测结果
48
+
49
+ 为了验证长序列的效果,这里我们使用了LongBench数据集。[LongBench](https://github.com/THUDM/LongBench)��第一个多任务、中英双语、针对大语言模型长文本理解能力的评测基准。LongBench由六大类、二十一个不同的任务组成,覆盖了单文档QA、多文档QA、摘要、Few-shot学习、合成任务和代码补全等关键的长文本应用场景。LongBench包含14个英文任务、5个中文任务和2个代码任务,多数任务的平均长度在5k-15k之间,共包含4750条测试数据。评估结果如下:
50
+
51
+
52
+ | 能力维度 | 数据集 | XVERSE-13B-256K | GPT-3.5-Turbo-16K | Llama2-7B-Chat-4K | LongChat-7B-16K | Yi-6B-200K |
53
+ | :--------: | :-------------------: | :----: | :----------: | :--------: | :-----------: | :--------: |
54
+ | 单文档QA | narrativeqa | 24.1 | 23.6 | 19.1 | 21.6 | 14.5 |
55
+ | | qasper | 30.2 | 43.3 | 19.6 | 21.6 | 21.6 |
56
+ | | multifieldqa_en | 43.5 | 52.3 | 35.8 | 44.6 | 36.6 |
57
+ | | multifieldqa_zh | 52.6 | 61.2 | 11.6 | 26.6 | 23.0 |
58
+ | 多文档QA | hotpotqa | 58.3 | 51.6 | 24.3 | 22.4 | 48.3 |
59
+ | | 2wikimqa | 49.6 | 37.7 | 31.4 | 16.8 | 39.2 |
60
+ | | musique | 31.4 | 26.9 | 8.6 | 9.1 | 25.3 |
61
+ | | dureader | 28.9 | 28.7 | 1.9 | 19.1 | 14.2 |
62
+ | 摘要 | gov_report | | 29.5 | 27.3 | 28.4 | 29.5 |
63
+ | | qmsum | 16.8 | 23.4 | 20.6 | 23.2 | 20.4 |
64
+ | | multinews | 21.1 | 26.7 | 25.8 | 26.4 | 26.2 |
65
+ | | vcsum | 11.3 | 16.0 | 0.2 | 14.0 | 8.2 |
66
+ | Few-shot | trec | 72.0 | 68.0 | 60.5 | 61.5 | 71.0 |
67
+ | | samsum | 34.6 | 41.7 | 31.4 | 44.8 | 11.3 |
68
+ | | triviaqa | 89.3 | 91.4 | 59.7 | 73.5 | 86.6 |
69
+ | | lsht | 35.0 | 29.2 | 19.8 | 20.8 | 38.0 |
70
+ | 合成任务 | passage_count | 6.0 | 4.5 | 2.5 | 4.5 | 2.0 |
71
+ | | passage_retrieval_en | 63.0 | 71.0 | 9.2 | 24.0 | 6.0 |
72
+ | | passage_retrieval_zh | 44.0 | 77.5 | 0.5 | 4.8 | 7.9 |
73
+ | 代码补全 | lcc | 55.2 | 54.7 | 52.3 | 59.2 | 64.6 |
74
+ | | repobench-p | | 53.6 | 42.4 | 54.7 | 61.5 |
75
+
76
+ 对于上述所有比较模型,我们优先汇报其官方公布的结果。在缺少官方结果的情况下,我们采用自行执行的评估流程所获得的数据。
77
+
78
+ ## Model Evaluation
79
+
80
+ To assess the performance of long sequences, we employed the LongBench dataset. [LongBench](https://github.com/THUDM/LongBench) stands as the inaugural multi-task, bilingual (English-Chinese), evaluation benchmark specifically designed to gauge the long-text comprehension capabilities of large language models. Comprising six major categories and twenty-one distinct tasks, LongBench encompasses critical long-text application scenarios such as single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. The dataset consists of 14 English tasks, 5 Chinese tasks, and 2 code tasks, with the majority of tasks having an average length ranging from 5,000 to 15,000 tokens, totaling 4,750 test instances. The evaluation results are presented below:
81
+
82
+ | Capability Dimension | Dataset | XVERSE-13B-256K | GPT-3.5-Turbo-16K | Llama2-7B-Chat-4K | LongChat-7B-16K | Yi-6B-200K |
83
+ | :--------: | :-------------------: | :----: | :----------: | :--------: | :-----------: | :--------: |
84
+ |single-document QA|narrativeqa | 24.1 | 23.6 | 19.1 | 21.6 | 14.5 |
85
+ | | qasper | 30.2 | 43.3 | 19.6 | 21.6 | 21.6 |
86
+ | | multifieldqa_en | 43.5 | 52.3 | 35.8 | 44.6 | 36.6 |
87
+ | | multifieldqa_zh | 52.6 | 61.2 | 11.6 | 26.6 | 23.0 |
88
+ |multi-document QA| hotpotqa | 58.3 | 51.6 | 24.3 | 22.4 | 48.3 |
89
+ | | 2wikimqa | 49.6 | 37.7 | 31.4 | 16.8 | 39.2 |
90
+ | | musique | 31.4 | 26.9 | 8.6 | 9.1 | 25.3 |
91
+ | | dureader | 28.9 | 28.7 | 1.9 | 19.1 | 14.2 |
92
+ |summarization | gov_report | | 29.5 | 27.3 | 28.4 | 29.5 |
93
+ | | qmsum | 16.8 | 23.4 | 20.6 | 23.2 | 20.4 |
94
+ | | multinews | 21.1 | 26.7 | 25.8 | 26.4 | 26.2 |
95
+ | | vcsum | 11.3 | 16.0 | 0.2 | 14.0 | 8.2 |
96
+ | Few-shot | trec | 72.0 | 68.0 | 60.5 | 61.5 | 71.0 |
97
+ | | samsum | 34.6 | 41.7 | 31.4 | 44.8 | 11.3 |
98
+ | | triviaqa | 89.3 | 91.4 | 59.7 | 73.5 | 86.6 |
99
+ | | lsht | 35.0 | 29.2 | 19.8 | 20.8 | 38.0 |
100
+ |synthetic tasks| passage_count | 6.0 | 4.5 | 2.5 | 4.5 | 2.0 |
101
+ | | passage_retrieval_en | 63.0 | 71.0 | 9.2 | 24.0 | 6.0 |
102
+ | | passage_retrieval_zh | 44.0 | 77.5 | 0.5 | 4.8 | 7.9 |
103
+ |code completion| lcc | 55.2 | 54.7 | 52.3 | 59.2 | 64.6 |
104
+ | | repobench-p | | 53.6 | 42.4 | 54.7 | 61.5 |
105
+
106
+
107
+ For all the comparison models mentioned above, we prioritize the disclosure of their officially published results. In the absence of official data, we refer to the results derived from our own evaluation pipline.
108
+
109
+ ### Loading with Transformers
110
+ 需要安装的:
111
+ Reqirements:
112
+
113
+ transformers==4.31.0
114
+ torch>=2.0
115
+ gradio>=3.39.0
116
+ accelerate>=0.21.0
117
+ cpm_kernels>=1.0.11
118
+ xformers
119
+
120
+
121
+ 可通过以下代码加载 XVERSE-13B-256K 模型进行对话:
122
+ The XVERSE-13B-256K model can be loaded for chat using the following code:
123
+
124
+ ```python
125
+ import torch
126
+ from transformers import AutoTokenizer, AutoModelForCausalLM
127
+ from transformers.generation.utils import GenerationConfig
128
+ model_path = "xverse/XVERSE-13B-256K"
129
+ tokenizer = AutoTokenizer.from_pretrained(model_path)
130
+ model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map='auto')
131
+ model.generation_config = GenerationConfig.from_pretrained(model_path)
132
+ model = model.eval()
133
+ history = [{"role": "user", "content": "1955年谁是美国总统?他是什么党派?"}]
134
+ response = model.chat(tokenizer, history)
135
+ print(response)
136
+ history.append({"role": "assistant", "content": response})
137
+ history.append({"role": "user", "content": "他任职了多少年"})
138
+ response = model.chat(tokenizer, history)
139
+ print(response)
140
+ ```
141
+
142
+ 更多细节,包括对话demo、模型微调及量化等,请参考我们的[Github](https://github.com/xverse-ai/XVERSE-13B)。
143
+
144
+ For more details, including chat demo, model fine-tuning and quantization, please refer to our [Github](https://github.com/xverse-ai/XVERSE-13B).
145
+
146
+ ## 局限性与免责申明
147
+
148
+ XVERSE-13B-256K 与其他所有 LLM 一样,在某些情况下可能会产生不准确、有偏见或其他令人反感的内容。因此,请谨慎使用模型生成的内容,请勿将生成的有害内容进行传播,在部署任何 XVERSE-13B-256K 的应用之前,开发人员应根据其具体应用对模型进行安全测试和调优。
149
+
150
+ 我们强烈警告不要将 XVERSE-13B-256K 模型用于制造或传播有害信息,或进行任何可能损害公众、国家、社会安全或违反法规的活动。如果使用 XVERSE-13B-256K 模型产生任何问题,无论是数据安全问题、公共舆论风险,还是模型被误解、滥用、传播或不合规使用所引发的任何风险和问题,我们将不承担任何责任。
151
+
152
+ ## Limitations and Disclaimer
153
+
154
+ Like all other Large Language Models (LLMs), XVERSE-13B-256K may produce inaccurate, biased, or otherwise offensive content under certain circumstances. Therefore, please use the content generated by the model with caution and refrain from disseminating harmful content. Before deploying any application of XVERSE-13B-256K, developers should conduct safety tests and optimization of the model according to its specific application.
155
+
156
+ We strongly warn against the use of the XVERSE-13B-256K model for producing or spreading harmful information, or conducting any activities that might harm the public, national, or social security, or violate regulations. We assume no responsibility for any problems arising from the use of the XVERSE-13B-256K model, whether it be data security issues, public opinion risks, or any risks and issues caused by misunderstanding, misuse, dissemination, or non-compliance with the model.
157
+
158
+ ## 模型开源协议
159
+
160
+ 使用本仓库的源码需要遵循 [Apache-2.0](https://github.com/xverse-ai/XVERSE-13B/blob/main/LICENSE) 开源协议,使用 XVERSE-13B-256K 的模型权重则需要遵循[模型许可协议](https://github.com/xverse-ai/XVERSE-13B/blob/main/MODEL_LICENSE.pdf)。
161
+
162
+ XVERSE-13B-256K 模型权重对学术研究**完全开放**,并且支持**免费商用**。如需申请商业许可证,请填写【[申请表](https://chat.xverse.cn/home/business.html)】,如有其他问题或合作,请联系 <opensource@xverse.cn>。
163
+
164
+ ## Open Source License
165
+
166
+ The use of the source code in this repository must follow the [Apache-2.0](https://github.com/xverse-ai/XVERSE-13B/blob/main/LICENSE) open-source license, while the use of the model weights of XVERSE-13B-256K needs to adhere to the [Model License Agreement](https://github.com/xverse-ai/XVERSE-13B/blob/main/MODEL_LICENSE.pdf).
167
+
168
+ The XVERSE-13B-256K model weights are **fully open** to academic research and support **free commercial use**. To apply for a commercial license, please fill in the [application form](https://chat.xverse.cn/home/business.html). For other questions or collaborations, please contact <opensource@xverse.cn>.
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "XverseForCausalLM"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_xverse.XverseConfig",
7
+ "AutoModelForCausalLM": "modeling_xverse.XverseForCausalLM"
8
+ },
9
+ "pad_token_id": 1,
10
+ "bos_token_id": 2,
11
+ "eos_token_id": 3,
12
+ "hidden_act": "silu",
13
+ "hidden_size": 5120,
14
+ "initializer_range": 0.02,
15
+ "intermediate_size": 13824,
16
+ "max_position_embeddings": 262144,
17
+ "max_tokenizer_truncation": 262144,
18
+ "model_type": "xverse",
19
+ "num_attention_heads": 40,
20
+ "num_hidden_layers": 40,
21
+ "rms_norm_eps": 1e-06,
22
+ "tie_word_embeddings": false,
23
+ "torch_dtype": "bfloat16",
24
+ "transformers_version": "4.28.1",
25
+ "use_cache": false,
26
+ "vocab_size": 100534
27
+ }
configuration.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "framework": "pytorch",
3
+ "task": "text-generation",
4
+ "model":
5
+ {
6
+ "type": "XVERSE-13B"
7
+ },
8
+ "pipeline":
9
+ {
10
+ "type": "XVERSE-13B-pipeline"
11
+ },
12
+ "allow_remote": true
13
+ }
configuration_xverse.py ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
5
+ # and OPT implementations in this library. It has been modified from its
6
+ # original forms to accommodate minor architectural differences compared
7
+ # to GPT-NeoX and OPT used by the Meta AI team that trained the model.
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ """ XVERSE model configuration"""
21
+
22
+ from transformers.configuration_utils import PretrainedConfig
23
+ from transformers.utils import logging
24
+
25
+
26
+ logger = logging.get_logger(__name__)
27
+
28
+ XVERSE_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
29
+
30
+
31
+ class XverseConfig(PretrainedConfig):
32
+ r"""
33
+ This is the configuration class to store the configuration of a [`XverseModel`]. It is used to instantiate an Xverse
34
+ model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
35
+ defaults will yield a similar configuration to that of the XVERSE-13B.
36
+
37
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
38
+ documentation from [`PretrainedConfig`] for more information.
39
+
40
+
41
+ Args:
42
+ vocab_size (`int`, *optional*, defaults to 100278):
43
+ Vocabulary size of the XVERSE model. Defines the number of different tokens that can be represented by the
44
+ `inputs_ids` passed when calling [`XverseModel`]
45
+ hidden_size (`int`, *optional*, defaults to 5120):
46
+ Dimension of the hidden representations.
47
+ intermediate_size (`int`, *optional*, defaults to 13824):
48
+ Dimension of the MLP representations.
49
+ num_hidden_layers (`int`, *optional*, defaults to 40):
50
+ Number of hidden layers in the Transformer encoder.
51
+ num_attention_heads (`int`, *optional*, defaults to 40):
52
+ Number of attention heads for each attention layer in the Transformer encoder.
53
+ hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
54
+ The non-linear activation function (function or string) in the decoder.
55
+ max_position_embeddings (`int`, *optional*, defaults to 8192):
56
+ The maximum sequence length that this model might ever be used with. Typically set this to something large
57
+ just in case (e.g., 512 or 1024 or 2048).
58
+ initializer_range (`float`, *optional*, defaults to 0.02):
59
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
60
+ rms_norm_eps (`float`, *optional*, defaults to 1e-6):
61
+ The epsilon used by the rms normalization layers.
62
+ use_cache (`bool`, *optional*, defaults to `True`):
63
+ Whether or not the model should return the last key/values attentions (not used by all models). Only
64
+ relevant if `config.is_decoder=True`.
65
+ tie_word_embeddings(`bool`, *optional*, defaults to `False`):
66
+ Whether to tie weight embeddings
67
+
68
+ Example:
69
+
70
+ ```python
71
+ >>> from transformers import XverseModel, XverseConfig
72
+
73
+ >>> # Initializing a Xverse XVERSE-13B style configuration
74
+ >>> configuration = XverseConfig()
75
+
76
+ >>> # Initializing a model from the XVERSE-13B style configuration
77
+ >>> model = XverseModel(configuration)
78
+
79
+ >>> # Accessing the model configuration
80
+ >>> configuration = model.config
81
+ ```"""
82
+ model_type = "xverse"
83
+ keys_to_ignore_at_inference = ["past_key_values"]
84
+
85
+ def __init__(
86
+ self,
87
+ vocab_size=100278,
88
+ hidden_size=5120,
89
+ intermediate_size=13824,
90
+ num_hidden_layers=40,
91
+ num_attention_heads=40,
92
+ hidden_act="silu",
93
+ max_position_embeddings=8192,
94
+ max_tokenizer_truncation=8192,
95
+ initializer_range=0.02,
96
+ rms_norm_eps=1e-6,
97
+ use_cache=True,
98
+ pad_token_id=None,
99
+ bos_token_id=1,
100
+ eos_token_id=2,
101
+ tie_word_embeddings=False,
102
+ **kwargs,
103
+ ):
104
+ self.vocab_size = vocab_size
105
+ self.max_position_embeddings = max_position_embeddings
106
+ self.hidden_size = hidden_size
107
+ self.intermediate_size = intermediate_size
108
+ self.num_hidden_layers = num_hidden_layers
109
+ self.num_attention_heads = num_attention_heads
110
+
111
+ self.hidden_act = hidden_act
112
+ self.initializer_range = initializer_range
113
+ self.rms_norm_eps = rms_norm_eps
114
+ self.use_cache = use_cache
115
+ self.max_tokenizer_truncation = max_tokenizer_truncation
116
+
117
+ super().__init__(
118
+ pad_token_id=pad_token_id,
119
+ bos_token_id=bos_token_id,
120
+ eos_token_id=eos_token_id,
121
+ tie_word_embeddings=tie_word_embeddings,
122
+ **kwargs,
123
+ )
generation_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "pad_token_id": 1,
3
+ "bos_token_id": 2,
4
+ "eos_token_id": 3,
5
+ "max_new_tokens": 2048,
6
+ "temperature": 0.5,
7
+ "top_k": 30,
8
+ "top_p": 0.85,
9
+ "repetition_penalty": 1.1,
10
+ "do_sample": true,
11
+ "transformers_version": "4.29.1"
12
+ }
modeling_xverse.py ADDED
@@ -0,0 +1,881 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
5
+ # and OPT implementations in this library. It has been modified from its
6
+ # original forms to accommodate minor architectural differences compared
7
+ # to GPT-NeoX and OPT used by the Meta AI team that trained the model.
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ """ PyTorch XVERSE model."""
21
+ import math
22
+ from typing import List, Optional, Tuple, Union
23
+
24
+ import torch
25
+ import torch.nn.functional as F
26
+ import torch.utils.checkpoint
27
+ from torch import nn
28
+ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
29
+
30
+ from transformers.activations import ACT2FN
31
+ from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
32
+ from transformers.modeling_utils import PreTrainedModel
33
+ from transformers.utils import add_start_docstrings, add_start_docstrings_to_model_forward, logging, replace_return_docstrings
34
+ from transformers.generation.utils import GenerationConfig
35
+ from .configuration_xverse import XverseConfig
36
+
37
+
38
+ logger = logging.get_logger(__name__)
39
+
40
+ _CONFIG_FOR_DOC = "XverseConfig"
41
+
42
+
43
+ # Copied from transformers.models.bart.modeling_bart._make_causal_mask
44
+ def _make_causal_mask(
45
+ input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0
46
+ ):
47
+ """
48
+ Make causal mask used for bi-directional self-attention.
49
+ """
50
+ bsz, tgt_len = input_ids_shape
51
+ mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min, device=device), device=device)
52
+ mask_cond = torch.arange(mask.size(-1), device=device)
53
+ mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
54
+ mask = mask.to(dtype)
55
+
56
+ if past_key_values_length > 0:
57
+ mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
58
+ return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)
59
+
60
+
61
+ # Copied from transformers.models.bart.modeling_bart._expand_mask
62
+ def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
63
+ """
64
+ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
65
+ """
66
+ bsz, src_len = mask.size()
67
+ tgt_len = tgt_len if tgt_len is not None else src_len
68
+
69
+ expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
70
+
71
+ inverted_mask = 1.0 - expanded_mask
72
+
73
+ return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)
74
+
75
+
76
+ class XverseRMSNorm(nn.Module):
77
+ def __init__(self, hidden_size, eps=1e-6):
78
+ """
79
+ XverseRMSNorm is equivalent to T5LayerNorm
80
+ """
81
+ super().__init__()
82
+ self.weight = nn.Parameter(torch.ones(hidden_size))
83
+ self.variance_epsilon = eps
84
+
85
+ def forward(self, hidden_states):
86
+ input_dtype = hidden_states.dtype
87
+ variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
88
+ hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
89
+
90
+ return (self.weight * hidden_states).to(input_dtype)
91
+
92
+
93
+ class XverseRotaryEmbedding(torch.nn.Module):
94
+ def __init__(self, dim, max_position_embeddings=2048, base=500000, device=None):
95
+ super().__init__()
96
+ inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float().to(device) / dim))
97
+ self.register_buffer("inv_freq", inv_freq)
98
+
99
+ # Build here to make `torch.jit.trace` work.
100
+ self.max_seq_len_cached = max_position_embeddings
101
+ t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
102
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
103
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
104
+ emb = torch.cat((freqs, freqs), dim=-1)
105
+ self.register_buffer("cos_cached", emb.cos()[None, None, :, :], persistent=False)
106
+ self.register_buffer("sin_cached", emb.sin()[None, None, :, :], persistent=False)
107
+
108
+ def forward(self, x, seq_len=None):
109
+ # x: [bs, num_attention_heads, seq_len, head_size]
110
+ # This `if` block is unlikely to be run after we build sin/cos in `__init__`. Keep the logic here just in case.
111
+ if seq_len > self.max_seq_len_cached:
112
+ self.max_seq_len_cached = seq_len
113
+ t = torch.arange(self.max_seq_len_cached, device=x.device, dtype=self.inv_freq.dtype)
114
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
115
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
116
+ emb = torch.cat((freqs, freqs), dim=-1).to(x.device)
117
+ self.register_buffer("cos_cached", emb.cos()[None, None, :, :], persistent=False)
118
+ self.register_buffer("sin_cached", emb.sin()[None, None, :, :], persistent=False)
119
+ return (
120
+ self.cos_cached[:, :, :seq_len, ...].to(dtype=x.dtype),
121
+ self.sin_cached[:, :, :seq_len, ...].to(dtype=x.dtype),
122
+ )
123
+
124
+
125
+ def rotate_half(x):
126
+ """Rotates half the hidden dims of the input."""
127
+ x1 = x[..., : x.shape[-1] // 2]
128
+ x2 = x[..., x.shape[-1] // 2 :]
129
+ return torch.cat((-x2, x1), dim=-1)
130
+
131
+
132
+ def apply_rotary_pos_emb(q, k, cos, sin, position_ids):
133
+ # The first two dimensions of cos and sin are always 1, so we can `squeeze` them.
134
+ cos = cos.squeeze(1).squeeze(0) # [seq_len, dim]
135
+ sin = sin.squeeze(1).squeeze(0) # [seq_len, dim]
136
+ cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
137
+ sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
138
+ q_embed = (q * cos) + (rotate_half(q) * sin)
139
+ k_embed = (k * cos) + (rotate_half(k) * sin)
140
+ return q_embed, k_embed
141
+
142
+
143
+ class XverseMLP(nn.Module):
144
+ def __init__(
145
+ self,
146
+ hidden_size: int,
147
+ intermediate_size: int,
148
+ hidden_act: str,
149
+ ):
150
+ super().__init__()
151
+ self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
152
+ self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False)
153
+ self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
154
+ self.act_fn = ACT2FN[hidden_act]
155
+
156
+ def forward(self, x):
157
+ return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
158
+
159
+
160
+ class XverseAttention(nn.Module):
161
+ """Multi-headed attention from 'Attention Is All You Need' paper"""
162
+
163
+ def __init__(self, config: XverseConfig):
164
+ super().__init__()
165
+ self.config = config
166
+ self.hidden_size = config.hidden_size
167
+ self.num_heads = config.num_attention_heads
168
+ self.head_dim = self.hidden_size // self.num_heads
169
+ self.max_position_embeddings = config.max_position_embeddings
170
+
171
+ if (self.head_dim * self.num_heads) != self.hidden_size:
172
+ raise ValueError(
173
+ f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}"
174
+ f" and `num_heads`: {self.num_heads})."
175
+ )
176
+ self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
177
+ self.k_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
178
+ self.v_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
179
+ self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False)
180
+ self.rotary_emb = XverseRotaryEmbedding(self.head_dim, max_position_embeddings=self.max_position_embeddings)
181
+
182
+ def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
183
+ return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()
184
+
185
+ def forward(
186
+ self,
187
+ hidden_states: torch.Tensor,
188
+ attention_mask: Optional[torch.Tensor] = None,
189
+ position_ids: Optional[torch.LongTensor] = None,
190
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
191
+ output_attentions: bool = False,
192
+ use_cache: bool = False,
193
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
194
+ bsz, q_len, _ = hidden_states.size()
195
+
196
+ query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
197
+ key_states = self.k_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
198
+ value_states = self.v_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
199
+
200
+ kv_seq_len = key_states.shape[-2]
201
+ if past_key_value is not None:
202
+ kv_seq_len += past_key_value[0].shape[-2]
203
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
204
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
205
+ # [bsz, nh, t, hd]
206
+
207
+ if past_key_value is not None:
208
+ # reuse k, v, self_attention
209
+ key_states = torch.cat([past_key_value[0], key_states], dim=2)
210
+ value_states = torch.cat([past_key_value[1], value_states], dim=2)
211
+
212
+ past_key_value = (key_states, value_states) if use_cache else None
213
+
214
+ attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
215
+
216
+ if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):
217
+ raise ValueError(
218
+ f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is"
219
+ f" {attn_weights.size()}"
220
+ )
221
+
222
+ if attention_mask is not None:
223
+ if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
224
+ raise ValueError(
225
+ f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
226
+ )
227
+ attn_weights = attn_weights + attention_mask
228
+ attn_weights = torch.max(
229
+ attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min, device=attn_weights.device)
230
+ )
231
+
232
+ # upcast attention to fp32
233
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
234
+ attn_output = torch.matmul(attn_weights, value_states)
235
+
236
+ if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
237
+ raise ValueError(
238
+ f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
239
+ f" {attn_output.size()}"
240
+ )
241
+
242
+ attn_output = attn_output.transpose(1, 2)
243
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
244
+
245
+ attn_output = self.o_proj(attn_output)
246
+
247
+ if not output_attentions:
248
+ attn_weights = None
249
+
250
+ return attn_output, attn_weights, past_key_value
251
+
252
+
253
+ class XverseDecoderLayer(nn.Module):
254
+ def __init__(self, config: XverseConfig):
255
+ super().__init__()
256
+ self.hidden_size = config.hidden_size
257
+ self.self_attn = XverseAttention(config=config)
258
+ self.mlp = XverseMLP(
259
+ hidden_size=self.hidden_size,
260
+ intermediate_size=config.intermediate_size,
261
+ hidden_act=config.hidden_act,
262
+ )
263
+ self.input_layernorm = XverseRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
264
+ self.post_attention_layernorm = XverseRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
265
+
266
+ def forward(
267
+ self,
268
+ hidden_states: torch.Tensor,
269
+ attention_mask: Optional[torch.Tensor] = None,
270
+ position_ids: Optional[torch.LongTensor] = None,
271
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
272
+ output_attentions: Optional[bool] = False,
273
+ use_cache: Optional[bool] = False,
274
+ ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
275
+ """
276
+ Args:
277
+ hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
278
+ attention_mask (`torch.FloatTensor`, *optional*): attention mask of size
279
+ `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
280
+ output_attentions (`bool`, *optional*):
281
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under
282
+ returned tensors for more detail.
283
+ use_cache (`bool`, *optional*):
284
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
285
+ (see `past_key_values`).
286
+ past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states
287
+ """
288
+
289
+ residual = hidden_states
290
+
291
+ hidden_states = self.input_layernorm(hidden_states)
292
+
293
+ # Self Attention
294
+ hidden_states, self_attn_weights, present_key_value = self.self_attn(
295
+ hidden_states=hidden_states,
296
+ attention_mask=attention_mask,
297
+ position_ids=position_ids,
298
+ past_key_value=past_key_value,
299
+ output_attentions=output_attentions,
300
+ use_cache=use_cache,
301
+ )
302
+ hidden_states = residual + hidden_states
303
+
304
+ # Fully Connected
305
+ residual = hidden_states
306
+ hidden_states = self.post_attention_layernorm(hidden_states)
307
+ hidden_states = self.mlp(hidden_states)
308
+ hidden_states = residual + hidden_states
309
+
310
+ outputs = (hidden_states,)
311
+
312
+ if output_attentions:
313
+ outputs += (self_attn_weights,)
314
+
315
+ if use_cache:
316
+ outputs += (present_key_value,)
317
+
318
+ return outputs
319
+
320
+
321
+ XVERSE_START_DOCSTRING = r"""
322
+ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
323
+ library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
324
+ etc.)
325
+
326
+ This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
327
+ Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
328
+ and behavior.
329
+
330
+ Parameters:
331
+ config ([`XverseConfig`]):
332
+ Model configuration class with all the parameters of the model. Initializing with a config file does not
333
+ load the weights associated with the model, only the configuration. Check out the
334
+ [`~PreTrainedModel.from_pretrained`] method to load the model weights.
335
+ """
336
+
337
+
338
+ @add_start_docstrings(
339
+ "The bare Xverse Model outputting raw hidden-states without any specific head on top.",
340
+ XVERSE_START_DOCSTRING,
341
+ )
342
+ class XversePreTrainedModel(PreTrainedModel):
343
+ config_class = XverseConfig
344
+ base_model_prefix = "model"
345
+ supports_gradient_checkpointing = True
346
+ _no_split_modules = ["XverseDecoderLayer"]
347
+ _skip_keys_device_placement = "past_key_values"
348
+ _keys_to_ignore_on_load_unexpected = [r"decoder\.version"]
349
+
350
+ def _init_weights(self, module):
351
+ std = self.config.initializer_range
352
+ if isinstance(module, nn.Linear):
353
+ module.weight.data.normal_(mean=0.0, std=std)
354
+ if module.bias is not None:
355
+ module.bias.data.zero_()
356
+ elif isinstance(module, nn.Embedding):
357
+ module.weight.data.normal_(mean=0.0, std=std)
358
+ if module.padding_idx is not None:
359
+ module.weight.data[module.padding_idx].zero_()
360
+
361
+ def _set_gradient_checkpointing(self, module, value=False):
362
+ if isinstance(module, XverseModel):
363
+ module.gradient_checkpointing = value
364
+
365
+
366
+ XVERSE_INPUTS_DOCSTRING = r"""
367
+ Args:
368
+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
369
+ Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
370
+ it.
371
+
372
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
373
+ [`PreTrainedTokenizer.__call__`] for details.
374
+
375
+ [What are input IDs?](../glossary#input-ids)
376
+ attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
377
+ Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
378
+
379
+ - 1 for tokens that are **not masked**,
380
+ - 0 for tokens that are **masked**.
381
+
382
+ [What are attention masks?](../glossary#attention-mask)
383
+
384
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
385
+ [`PreTrainedTokenizer.__call__`] for details.
386
+
387
+ If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see
388
+ `past_key_values`).
389
+
390
+ If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
391
+ and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
392
+ information on the default strategy.
393
+
394
+ - 1 indicates the head is **not masked**,
395
+ - 0 indicates the head is **masked**.
396
+ position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
397
+ Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
398
+ config.n_positions - 1]`.
399
+
400
+ [What are position IDs?](../glossary#position-ids)
401
+ past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
402
+ Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape
403
+ `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape
404
+ `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`.
405
+
406
+ Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
407
+ blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.
408
+
409
+ If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
410
+ don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
411
+ `decoder_input_ids` of shape `(batch_size, sequence_length)`.
412
+ inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
413
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
414
+ is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
415
+ model's internal embedding lookup matrix.
416
+ use_cache (`bool`, *optional*):
417
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
418
+ `past_key_values`).
419
+ output_attentions (`bool`, *optional*):
420
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
421
+ tensors for more detail.
422
+ output_hidden_states (`bool`, *optional*):
423
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
424
+ more detail.
425
+ return_dict (`bool`, *optional*):
426
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
427
+ """
428
+
429
+ @add_start_docstrings(
430
+ "The bare Xverse Model outputting raw hidden-states without any specific head on top.",
431
+ XVERSE_START_DOCSTRING,
432
+ )
433
+ class XverseModel(XversePreTrainedModel):
434
+ """
435
+ Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`XverseDecoderLayer`]
436
+
437
+ Args:
438
+ config: XverseConfig
439
+ """
440
+
441
+ def __init__(self, config: XverseConfig):
442
+ super().__init__(config)
443
+ self.padding_idx = config.pad_token_id
444
+ self.vocab_size = config.vocab_size
445
+
446
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
447
+ self.layers = nn.ModuleList([XverseDecoderLayer(config) for _ in range(config.num_hidden_layers)])
448
+ self.norm = XverseRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
449
+
450
+ self.gradient_checkpointing = False
451
+ # Initialize weights and apply final processing
452
+ self.post_init()
453
+
454
+ def get_input_embeddings(self):
455
+ return self.embed_tokens
456
+
457
+ def set_input_embeddings(self, value):
458
+ self.embed_tokens = value
459
+
460
+ # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask
461
+ def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):
462
+ # create causal mask
463
+ # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
464
+ combined_attention_mask = None
465
+ if input_shape[-1] > 1:
466
+ combined_attention_mask = _make_causal_mask(
467
+ input_shape,
468
+ inputs_embeds.dtype,
469
+ device=inputs_embeds.device,
470
+ past_key_values_length=past_key_values_length,
471
+ )
472
+
473
+ if attention_mask is not None:
474
+ # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
475
+ expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(
476
+ inputs_embeds.device
477
+ )
478
+ combined_attention_mask = (
479
+ expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
480
+ )
481
+
482
+ return combined_attention_mask
483
+
484
+ @add_start_docstrings_to_model_forward(XVERSE_INPUTS_DOCSTRING)
485
+ def forward(
486
+ self,
487
+ input_ids: torch.LongTensor = None,
488
+ attention_mask: Optional[torch.Tensor] = None,
489
+ position_ids: Optional[torch.LongTensor] = None,
490
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
491
+ inputs_embeds: Optional[torch.FloatTensor] = None,
492
+ use_cache: Optional[bool] = None,
493
+ output_attentions: Optional[bool] = None,
494
+ output_hidden_states: Optional[bool] = None,
495
+ return_dict: Optional[bool] = None,
496
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
497
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
498
+ output_hidden_states = (
499
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
500
+ )
501
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
502
+
503
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
504
+
505
+ # retrieve input_ids and inputs_embeds
506
+ if input_ids is not None and inputs_embeds is not None:
507
+ raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time")
508
+ elif input_ids is not None:
509
+ batch_size, seq_length = input_ids.shape
510
+ elif inputs_embeds is not None:
511
+ batch_size, seq_length, _ = inputs_embeds.shape
512
+ else:
513
+ raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")
514
+
515
+ seq_length_with_past = seq_length
516
+ past_key_values_length = 0
517
+
518
+ if past_key_values is not None:
519
+ past_key_values_length = past_key_values[0][0].shape[2]
520
+ seq_length_with_past = seq_length_with_past + past_key_values_length
521
+
522
+ if position_ids is None:
523
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
524
+ position_ids = torch.arange(
525
+ past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device
526
+ )
527
+ position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
528
+ else:
529
+ position_ids = position_ids.view(-1, seq_length).long()
530
+
531
+ if inputs_embeds is None:
532
+ inputs_embeds = self.embed_tokens(input_ids)
533
+ # embed positions
534
+ if attention_mask is None:
535
+ attention_mask = torch.ones(
536
+ (batch_size, seq_length_with_past), dtype=torch.bool, device=inputs_embeds.device
537
+ )
538
+ attention_mask = self._prepare_decoder_attention_mask(
539
+ attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length
540
+ )
541
+
542
+ hidden_states = inputs_embeds
543
+
544
+ if self.gradient_checkpointing and self.training:
545
+ if use_cache:
546
+ logger.warning_once(
547
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
548
+ )
549
+ use_cache = False
550
+
551
+ # decoder layers
552
+ all_hidden_states = () if output_hidden_states else None
553
+ all_self_attns = () if output_attentions else None
554
+ next_decoder_cache = () if use_cache else None
555
+
556
+ for idx, decoder_layer in enumerate(self.layers):
557
+ if output_hidden_states:
558
+ all_hidden_states += (hidden_states,)
559
+
560
+ past_key_value = past_key_values[idx] if past_key_values is not None else None
561
+
562
+ if self.gradient_checkpointing and self.training:
563
+
564
+ def create_custom_forward(module):
565
+ def custom_forward(*inputs):
566
+ # None for past_key_value
567
+ return module(*inputs, output_attentions, None)
568
+
569
+ return custom_forward
570
+
571
+ layer_outputs = torch.utils.checkpoint.checkpoint(
572
+ create_custom_forward(decoder_layer),
573
+ hidden_states,
574
+ attention_mask,
575
+ position_ids,
576
+ None,
577
+ )
578
+ else:
579
+ layer_outputs = decoder_layer(
580
+ hidden_states,
581
+ attention_mask=attention_mask,
582
+ position_ids=position_ids,
583
+ past_key_value=past_key_value,
584
+ output_attentions=output_attentions,
585
+ use_cache=use_cache,
586
+ )
587
+
588
+ hidden_states = layer_outputs[0]
589
+
590
+ if use_cache:
591
+ next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)
592
+
593
+ if output_attentions:
594
+ all_self_attns += (layer_outputs[1],)
595
+
596
+ hidden_states = self.norm(hidden_states)
597
+
598
+ # add hidden states from the last decoder layer
599
+ if output_hidden_states:
600
+ all_hidden_states += (hidden_states,)
601
+
602
+ next_cache = next_decoder_cache if use_cache else None
603
+ if not return_dict:
604
+ return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
605
+ return BaseModelOutputWithPast(
606
+ last_hidden_state=hidden_states,
607
+ past_key_values=next_cache,
608
+ hidden_states=all_hidden_states,
609
+ attentions=all_self_attns,
610
+ )
611
+
612
+
613
+ class XverseForCausalLM(XversePreTrainedModel):
614
+ def __init__(self, config):
615
+ super().__init__(config)
616
+ self.model = XverseModel(config)
617
+
618
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
619
+
620
+ # Initialize weights and apply final processing
621
+ self.post_init()
622
+
623
+ def get_input_embeddings(self):
624
+ return self.model.embed_tokens
625
+
626
+ def set_input_embeddings(self, value):
627
+ self.model.embed_tokens = value
628
+
629
+ def get_output_embeddings(self):
630
+ return self.lm_head
631
+
632
+ def set_output_embeddings(self, new_embeddings):
633
+ self.lm_head = new_embeddings
634
+
635
+ def set_decoder(self, decoder):
636
+ self.model = decoder
637
+
638
+ def get_decoder(self):
639
+ return self.model
640
+
641
+ @add_start_docstrings_to_model_forward(XVERSE_INPUTS_DOCSTRING)
642
+ @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
643
+ def forward(
644
+ self,
645
+ input_ids: torch.LongTensor = None,
646
+ attention_mask: Optional[torch.Tensor] = None,
647
+ position_ids: Optional[torch.LongTensor] = None,
648
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
649
+ inputs_embeds: Optional[torch.FloatTensor] = None,
650
+ labels: Optional[torch.LongTensor] = None,
651
+ use_cache: Optional[bool] = None,
652
+ output_attentions: Optional[bool] = None,
653
+ output_hidden_states: Optional[bool] = None,
654
+ return_dict: Optional[bool] = None,
655
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
656
+ r"""
657
+ Args:
658
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
659
+ Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
660
+ config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
661
+ (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
662
+
663
+ Returns:
664
+
665
+ Example:
666
+
667
+ ```python
668
+ >>> from transformers import AutoTokenizer, AutoModelForCausalLM
669
+
670
+ >>> model = AutoModelForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS, trust_remote_code=True)
671
+ >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)
672
+
673
+ >>> prompt = "Hey, are you conscious? Can you talk to me?"
674
+ >>> inputs = tokenizer(prompt, return_tensors="pt")
675
+
676
+ >>> # Generate
677
+ >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
678
+ >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
679
+ "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
680
+ ```"""
681
+
682
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
683
+ output_hidden_states = (
684
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
685
+ )
686
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
687
+
688
+ # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
689
+ outputs = self.model(
690
+ input_ids=input_ids,
691
+ attention_mask=attention_mask,
692
+ position_ids=position_ids,
693
+ past_key_values=past_key_values,
694
+ inputs_embeds=inputs_embeds,
695
+ use_cache=use_cache,
696
+ output_attentions=output_attentions,
697
+ output_hidden_states=output_hidden_states,
698
+ return_dict=return_dict,
699
+ )
700
+
701
+ hidden_states = outputs[0]
702
+ logits = self.lm_head(hidden_states)
703
+
704
+ loss = None
705
+ if labels is not None:
706
+ # Shift so that tokens < n predict n
707
+ shift_logits = logits[..., :-1, :].contiguous()
708
+ shift_labels = labels[..., 1:].contiguous()
709
+ # Flatten the tokens
710
+ loss_fct = CrossEntropyLoss()
711
+ shift_logits = shift_logits.view(-1, self.config.vocab_size)
712
+ shift_labels = shift_labels.view(-1)
713
+ # Enable model parallelism
714
+ shift_labels = shift_labels.to(shift_logits.device)
715
+ loss = loss_fct(shift_logits, shift_labels)
716
+
717
+ if not return_dict:
718
+ output = (logits,) + outputs[1:]
719
+ return (loss,) + output if loss is not None else output
720
+
721
+ return CausalLMOutputWithPast(
722
+ loss=loss,
723
+ logits=logits,
724
+ past_key_values=outputs.past_key_values,
725
+ hidden_states=outputs.hidden_states,
726
+ attentions=outputs.attentions,
727
+ )
728
+
729
+ def _build_chat_input(self, tokenizer, messages: List[dict], max_new_tokens: int=2048):
730
+ max_new_tokens = max_new_tokens or self.generation_config.max_new_tokens
731
+ max_input_tokens = self.config.max_position_embeddings - max_new_tokens
732
+ max_input_tokens = max(self.config.max_position_embeddings // 2, max_input_tokens)
733
+ max_input_tokens = min(self.config.max_tokenizer_truncation, max_input_tokens)
734
+
735
+ total_input, round_input = [], []
736
+ user_prompt_tokens = tokenizer.encode("Human: ", return_token_type_ids=False)
737
+ exec_prompt_tokens = tokenizer.encode("Exec: ", return_token_type_ids=False)
738
+ assist_prompt_tokens = tokenizer.encode("Assistant: ", return_token_type_ids=False)
739
+ assist_prompt_len = len(assist_prompt_tokens)
740
+
741
+ for i, message in enumerate(messages[::-1]):
742
+ if message['role'] == 'user' or message['role'] == 'exec':
743
+ user_content = f"{message['content']}\n\n"
744
+ content_tokens = user_prompt_tokens + tokenizer.encode(user_content, return_token_type_ids=False) if message['role'] == 'user' else \
745
+ exec_prompt_tokens + tokenizer.encode(user_content, return_token_type_ids=False)
746
+ if i == 0:
747
+ content_tokens = content_tokens[:max_input_tokens-assist_prompt_len]
748
+ content_tokens += assist_prompt_tokens
749
+ round_input = content_tokens + round_input
750
+
751
+ if i != 0:
752
+ if len(total_input) + len(round_input) > max_input_tokens:
753
+ break
754
+ else:
755
+ total_input = round_input + total_input
756
+ else:
757
+ total_input = round_input + total_input
758
+ if len(total_input) >= max_input_tokens:
759
+ break
760
+ round_input = []
761
+ elif message['role'] == 'assistant':
762
+ assist_content = f"{message['content']}"
763
+ content_tokens = assist_prompt_tokens + tokenizer.encode(assist_content, return_token_type_ids=False)
764
+ round_input = content_tokens + [self.generation_config.eos_token_id] + round_input
765
+ elif message['role'] == 'system':
766
+ assert i == len(messages) - 1
767
+ user_content = f"{message['content']}\n"
768
+ content_tokens = tokenizer.encode(user_content, return_token_type_ids=False)
769
+ round_input = user_prompt_tokens + content_tokens + round_input
770
+ if len(total_input) + len(round_input) > max_input_tokens:
771
+ break
772
+ else:
773
+ total_input = round_input + total_input
774
+ else:
775
+ raise ValueError(f"message role not supported yet: {message['role']}")
776
+ total_input = torch.LongTensor([total_input]).to(self.device)
777
+ return total_input
778
+
779
+ @torch.no_grad()
780
+ def chat(self, tokenizer, messages: List[dict], stream=False,
781
+ generation_config: Optional[GenerationConfig]=None):
782
+ generation_config = generation_config or self.generation_config
783
+ input_ids = self._build_chat_input(tokenizer, messages, generation_config.max_new_tokens)
784
+ if stream:
785
+ from transformers import TextIteratorStreamer
786
+ from threading import Thread
787
+ streamer = TextIteratorStreamer(tokenizer, skip_prompt=True)
788
+ self.__class__.generate = PreTrainedModel.generate
789
+
790
+ def stream_generator():
791
+ generation_kwargs = dict(inputs=input_ids, generation_config=generation_config, streamer=streamer)
792
+ thread = Thread(target=self.generate, kwargs=generation_kwargs)
793
+ thread.start()
794
+ for next_text in streamer:
795
+ yield next_text.replace(tokenizer.eos_token, "")
796
+
797
+ return stream_generator()
798
+ else:
799
+ self.__class__.generate = PreTrainedModel.generate # disable stream
800
+ outputs = self.generate(input_ids, generation_config=generation_config)
801
+ response = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True)
802
+ return response
803
+
804
+ def prepare_inputs_for_generation(
805
+ self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs
806
+ ):
807
+ if past_key_values:
808
+ input_ids = input_ids[:, -1:]
809
+
810
+ position_ids = kwargs.get("position_ids", None)
811
+ if attention_mask is not None and position_ids is None:
812
+ # create position_ids on the fly for batch generation
813
+ position_ids = attention_mask.long().cumsum(-1) - 1
814
+ position_ids.masked_fill_(attention_mask == 0, 1)
815
+ if past_key_values:
816
+ position_ids = position_ids[:, -1].unsqueeze(-1)
817
+
818
+ # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
819
+ if inputs_embeds is not None and past_key_values is None:
820
+ model_inputs = {"inputs_embeds": inputs_embeds}
821
+ else:
822
+ model_inputs = {"input_ids": input_ids}
823
+
824
+ model_inputs.update(
825
+ {
826
+ "position_ids": position_ids,
827
+ "past_key_values": past_key_values,
828
+ "use_cache": kwargs.get("use_cache"),
829
+ "attention_mask": attention_mask,
830
+ }
831
+ )
832
+ return model_inputs
833
+
834
+ @staticmethod
835
+ def _reorder_cache(past_key_values, beam_idx):
836
+ reordered_past = ()
837
+ for layer_past in past_key_values:
838
+ reordered_past += (tuple(past_state.index_select(0, beam_idx) for past_state in layer_past),)
839
+ return reordered_past
840
+
841
+ def quantize(self, bit_length: int):
842
+ from .quantization import QuantizationLinear
843
+
844
+ for layer in self.model.layers:
845
+ layer.self_attn.q_proj = QuantizationLinear(
846
+ bit_length=bit_length,
847
+ weight=layer.self_attn.q_proj.weight.to(torch.cuda.current_device()),
848
+ device=layer.self_attn.q_proj.weight.device,
849
+ )
850
+ layer.self_attn.k_proj = QuantizationLinear(
851
+ bit_length=bit_length,
852
+ weight=layer.self_attn.k_proj.weight.to(torch.cuda.current_device()),
853
+ device=layer.self_attn.k_proj.weight.device
854
+ )
855
+ layer.self_attn.v_proj = QuantizationLinear(
856
+ bit_length=bit_length,
857
+ weight=layer.self_attn.v_proj.weight.to(torch.cuda.current_device()),
858
+ device=layer.self_attn.v_proj.weight.device
859
+ )
860
+ layer.self_attn.o_proj = QuantizationLinear(
861
+ bit_length=bit_length,
862
+ weight=layer.self_attn.o_proj.weight.to(torch.cuda.current_device()),
863
+ device=layer.self_attn.o_proj.weight.device
864
+ )
865
+ layer.mlp.gate_proj = QuantizationLinear(
866
+ bit_length=bit_length,
867
+ weight=layer.mlp.gate_proj.weight.to(torch.cuda.current_device()),
868
+ device=layer.mlp.gate_proj.weight.device
869
+ )
870
+ layer.mlp.down_proj = QuantizationLinear(
871
+ bit_length=bit_length,
872
+ weight=layer.mlp.down_proj.weight.to(torch.cuda.current_device()),
873
+ device=layer.mlp.down_proj.weight.device
874
+ )
875
+ layer.mlp.up_proj = QuantizationLinear(
876
+ bit_length=bit_length,
877
+ weight=layer.mlp.up_proj.weight.to(torch.cuda.current_device()),
878
+ device=layer.mlp.up_proj.weight.device
879
+ )
880
+
881
+ return self
pytorch_model-00001-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:48e8548221d21e0c32c165ba1ecc5383e4b84f6c80acaf6d7a4806170f99c821
3
+ size 1873720633
pytorch_model-00002-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bb8ab8f005de2a4ad634cbe3c8573922dbf00d8f2ab95d416936ca2391e25916
3
+ size 1903235337
pytorch_model-00003-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6616978925903a85168951e8501b8106dc26b704554c834cc6639ae2b92c5bd9
3
+ size 1903235337
pytorch_model-00004-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b4984916ff0077d3fa9030340537ce1922d2d52fe0f625bfe6aad80fe78ea5e
3
+ size 1903235337
pytorch_model-00005-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d52953ccaa2e1e4240e6c894db09892a8512004301456b6012882def6a8939e0
3
+ size 1903235337
pytorch_model-00006-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:14fce147c474bacb81bd8893368e99f9f0d4178c9fb77eea40d5af8a3a8956c2
3
+ size 1903235337
pytorch_model-00007-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8ed7b8b944d2bcd3fb0a2a11bf15229202e1dc1b4020d63aaffdc0f509563f1f
3
+ size 1903235337
pytorch_model-00008-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:357ff4687a58cb05efa8ec3a57b7af06d374f363d95be0003cb4df2fbf7175b8
3
+ size 1903235337
pytorch_model-00009-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b6cc6a3727e4ea2bb00daad49924cf8f6413e43a03bb9b63f42d97a730c22cfb
3
+ size 1903235337
pytorch_model-00010-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:418f1f2dbc72e12c5e763834e30c12e85b134431ae90ea50b14820b94df2a3e7
3
+ size 1903235337
pytorch_model-00011-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:71bd96ec58f19a06b19940e7c61b181038d3eb3a30e15a611dfa8063c6fd964f
3
+ size 1903235337
pytorch_model-00012-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed0dc2db157e68e026c5909ae2b857663ccd4742f2ce5a1860f51f6b39624e23
3
+ size 1903235337
pytorch_model-00013-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c3d6d58fcbbeb0e1e28c8c99b75bcd76ce79ee4fdddd398260b3e39f69ad3fae
3
+ size 1903235337
pytorch_model-00014-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67ffa04b05c9f0154c2f5271e792edbb9b0959972ba1999b7a1e81e3d860417f
3
+ size 1693508289
pytorch_model-00015-of-00015.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bdfc3c7e72824494e553f71ef2984652d5b9e62d144f03bbe9be59bd5265b5bf
3
+ size 1029571498
pytorch_model.bin.index.json ADDED
@@ -0,0 +1,410 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 27435310080
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "pytorch_model-00015-of-00015.bin",
7
+ "model.embed_tokens.weight": "pytorch_model-00001-of-00015.bin",
8
+ "model.layers.0.input_layernorm.weight": "pytorch_model-00001-of-00015.bin",
9
+ "model.layers.0.mlp.down_proj.weight": "pytorch_model-00001-of-00015.bin",
10
+ "model.layers.0.mlp.gate_proj.weight": "pytorch_model-00001-of-00015.bin",
11
+ "model.layers.0.mlp.up_proj.weight": "pytorch_model-00001-of-00015.bin",
12
+ "model.layers.0.post_attention_layernorm.weight": "pytorch_model-00001-of-00015.bin",
13
+ "model.layers.0.self_attn.k_proj.weight": "pytorch_model-00001-of-00015.bin",
14
+ "model.layers.0.self_attn.o_proj.weight": "pytorch_model-00001-of-00015.bin",
15
+ "model.layers.0.self_attn.q_proj.weight": "pytorch_model-00001-of-00015.bin",
16
+ "model.layers.0.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00015.bin",
17
+ "model.layers.0.self_attn.v_proj.weight": "pytorch_model-00001-of-00015.bin",
18
+ "model.layers.1.input_layernorm.weight": "pytorch_model-00001-of-00015.bin",
19
+ "model.layers.1.mlp.down_proj.weight": "pytorch_model-00002-of-00015.bin",
20
+ "model.layers.1.mlp.gate_proj.weight": "pytorch_model-00002-of-00015.bin",
21
+ "model.layers.1.mlp.up_proj.weight": "pytorch_model-00002-of-00015.bin",
22
+ "model.layers.1.post_attention_layernorm.weight": "pytorch_model-00001-of-00015.bin",
23
+ "model.layers.1.self_attn.k_proj.weight": "pytorch_model-00001-of-00015.bin",
24
+ "model.layers.1.self_attn.o_proj.weight": "pytorch_model-00001-of-00015.bin",
25
+ "model.layers.1.self_attn.q_proj.weight": "pytorch_model-00001-of-00015.bin",
26
+ "model.layers.1.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00015.bin",
27
+ "model.layers.1.self_attn.v_proj.weight": "pytorch_model-00001-of-00015.bin",
28
+ "model.layers.10.input_layernorm.weight": "pytorch_model-00004-of-00015.bin",
29
+ "model.layers.10.mlp.down_proj.weight": "pytorch_model-00005-of-00015.bin",
30
+ "model.layers.10.mlp.gate_proj.weight": "pytorch_model-00005-of-00015.bin",
31
+ "model.layers.10.mlp.up_proj.weight": "pytorch_model-00005-of-00015.bin",
32
+ "model.layers.10.post_attention_layernorm.weight": "pytorch_model-00004-of-00015.bin",
33
+ "model.layers.10.self_attn.k_proj.weight": "pytorch_model-00004-of-00015.bin",
34
+ "model.layers.10.self_attn.o_proj.weight": "pytorch_model-00004-of-00015.bin",
35
+ "model.layers.10.self_attn.q_proj.weight": "pytorch_model-00004-of-00015.bin",
36
+ "model.layers.10.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00015.bin",
37
+ "model.layers.10.self_attn.v_proj.weight": "pytorch_model-00004-of-00015.bin",
38
+ "model.layers.11.input_layernorm.weight": "pytorch_model-00005-of-00015.bin",
39
+ "model.layers.11.mlp.down_proj.weight": "pytorch_model-00005-of-00015.bin",
40
+ "model.layers.11.mlp.gate_proj.weight": "pytorch_model-00005-of-00015.bin",
41
+ "model.layers.11.mlp.up_proj.weight": "pytorch_model-00005-of-00015.bin",
42
+ "model.layers.11.post_attention_layernorm.weight": "pytorch_model-00005-of-00015.bin",
43
+ "model.layers.11.self_attn.k_proj.weight": "pytorch_model-00005-of-00015.bin",
44
+ "model.layers.11.self_attn.o_proj.weight": "pytorch_model-00005-of-00015.bin",
45
+ "model.layers.11.self_attn.q_proj.weight": "pytorch_model-00005-of-00015.bin",
46
+ "model.layers.11.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00015.bin",
47
+ "model.layers.11.self_attn.v_proj.weight": "pytorch_model-00005-of-00015.bin",
48
+ "model.layers.12.input_layernorm.weight": "pytorch_model-00005-of-00015.bin",
49
+ "model.layers.12.mlp.down_proj.weight": "pytorch_model-00005-of-00015.bin",
50
+ "model.layers.12.mlp.gate_proj.weight": "pytorch_model-00005-of-00015.bin",
51
+ "model.layers.12.mlp.up_proj.weight": "pytorch_model-00005-of-00015.bin",
52
+ "model.layers.12.post_attention_layernorm.weight": "pytorch_model-00005-of-00015.bin",
53
+ "model.layers.12.self_attn.k_proj.weight": "pytorch_model-00005-of-00015.bin",
54
+ "model.layers.12.self_attn.o_proj.weight": "pytorch_model-00005-of-00015.bin",
55
+ "model.layers.12.self_attn.q_proj.weight": "pytorch_model-00005-of-00015.bin",
56
+ "model.layers.12.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00015.bin",
57
+ "model.layers.12.self_attn.v_proj.weight": "pytorch_model-00005-of-00015.bin",
58
+ "model.layers.13.input_layernorm.weight": "pytorch_model-00005-of-00015.bin",
59
+ "model.layers.13.mlp.down_proj.weight": "pytorch_model-00006-of-00015.bin",
60
+ "model.layers.13.mlp.gate_proj.weight": "pytorch_model-00006-of-00015.bin",
61
+ "model.layers.13.mlp.up_proj.weight": "pytorch_model-00006-of-00015.bin",
62
+ "model.layers.13.post_attention_layernorm.weight": "pytorch_model-00005-of-00015.bin",
63
+ "model.layers.13.self_attn.k_proj.weight": "pytorch_model-00005-of-00015.bin",
64
+ "model.layers.13.self_attn.o_proj.weight": "pytorch_model-00005-of-00015.bin",
65
+ "model.layers.13.self_attn.q_proj.weight": "pytorch_model-00005-of-00015.bin",
66
+ "model.layers.13.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00015.bin",
67
+ "model.layers.13.self_attn.v_proj.weight": "pytorch_model-00005-of-00015.bin",
68
+ "model.layers.14.input_layernorm.weight": "pytorch_model-00006-of-00015.bin",
69
+ "model.layers.14.mlp.down_proj.weight": "pytorch_model-00006-of-00015.bin",
70
+ "model.layers.14.mlp.gate_proj.weight": "pytorch_model-00006-of-00015.bin",
71
+ "model.layers.14.mlp.up_proj.weight": "pytorch_model-00006-of-00015.bin",
72
+ "model.layers.14.post_attention_layernorm.weight": "pytorch_model-00006-of-00015.bin",
73
+ "model.layers.14.self_attn.k_proj.weight": "pytorch_model-00006-of-00015.bin",
74
+ "model.layers.14.self_attn.o_proj.weight": "pytorch_model-00006-of-00015.bin",
75
+ "model.layers.14.self_attn.q_proj.weight": "pytorch_model-00006-of-00015.bin",
76
+ "model.layers.14.self_attn.rotary_emb.inv_freq": "pytorch_model-00006-of-00015.bin",
77
+ "model.layers.14.self_attn.v_proj.weight": "pytorch_model-00006-of-00015.bin",
78
+ "model.layers.15.input_layernorm.weight": "pytorch_model-00006-of-00015.bin",
79
+ "model.layers.15.mlp.down_proj.weight": "pytorch_model-00006-of-00015.bin",
80
+ "model.layers.15.mlp.gate_proj.weight": "pytorch_model-00006-of-00015.bin",
81
+ "model.layers.15.mlp.up_proj.weight": "pytorch_model-00006-of-00015.bin",
82
+ "model.layers.15.post_attention_layernorm.weight": "pytorch_model-00006-of-00015.bin",
83
+ "model.layers.15.self_attn.k_proj.weight": "pytorch_model-00006-of-00015.bin",
84
+ "model.layers.15.self_attn.o_proj.weight": "pytorch_model-00006-of-00015.bin",
85
+ "model.layers.15.self_attn.q_proj.weight": "pytorch_model-00006-of-00015.bin",
86
+ "model.layers.15.self_attn.rotary_emb.inv_freq": "pytorch_model-00006-of-00015.bin",
87
+ "model.layers.15.self_attn.v_proj.weight": "pytorch_model-00006-of-00015.bin",
88
+ "model.layers.16.input_layernorm.weight": "pytorch_model-00006-of-00015.bin",
89
+ "model.layers.16.mlp.down_proj.weight": "pytorch_model-00007-of-00015.bin",
90
+ "model.layers.16.mlp.gate_proj.weight": "pytorch_model-00007-of-00015.bin",
91
+ "model.layers.16.mlp.up_proj.weight": "pytorch_model-00007-of-00015.bin",
92
+ "model.layers.16.post_attention_layernorm.weight": "pytorch_model-00006-of-00015.bin",
93
+ "model.layers.16.self_attn.k_proj.weight": "pytorch_model-00006-of-00015.bin",
94
+ "model.layers.16.self_attn.o_proj.weight": "pytorch_model-00006-of-00015.bin",
95
+ "model.layers.16.self_attn.q_proj.weight": "pytorch_model-00006-of-00015.bin",
96
+ "model.layers.16.self_attn.rotary_emb.inv_freq": "pytorch_model-00006-of-00015.bin",
97
+ "model.layers.16.self_attn.v_proj.weight": "pytorch_model-00006-of-00015.bin",
98
+ "model.layers.17.input_layernorm.weight": "pytorch_model-00007-of-00015.bin",
99
+ "model.layers.17.mlp.down_proj.weight": "pytorch_model-00007-of-00015.bin",
100
+ "model.layers.17.mlp.gate_proj.weight": "pytorch_model-00007-of-00015.bin",
101
+ "model.layers.17.mlp.up_proj.weight": "pytorch_model-00007-of-00015.bin",
102
+ "model.layers.17.post_attention_layernorm.weight": "pytorch_model-00007-of-00015.bin",
103
+ "model.layers.17.self_attn.k_proj.weight": "pytorch_model-00007-of-00015.bin",
104
+ "model.layers.17.self_attn.o_proj.weight": "pytorch_model-00007-of-00015.bin",
105
+ "model.layers.17.self_attn.q_proj.weight": "pytorch_model-00007-of-00015.bin",
106
+ "model.layers.17.self_attn.rotary_emb.inv_freq": "pytorch_model-00007-of-00015.bin",
107
+ "model.layers.17.self_attn.v_proj.weight": "pytorch_model-00007-of-00015.bin",
108
+ "model.layers.18.input_layernorm.weight": "pytorch_model-00007-of-00015.bin",
109
+ "model.layers.18.mlp.down_proj.weight": "pytorch_model-00007-of-00015.bin",
110
+ "model.layers.18.mlp.gate_proj.weight": "pytorch_model-00007-of-00015.bin",
111
+ "model.layers.18.mlp.up_proj.weight": "pytorch_model-00007-of-00015.bin",
112
+ "model.layers.18.post_attention_layernorm.weight": "pytorch_model-00007-of-00015.bin",
113
+ "model.layers.18.self_attn.k_proj.weight": "pytorch_model-00007-of-00015.bin",
114
+ "model.layers.18.self_attn.o_proj.weight": "pytorch_model-00007-of-00015.bin",
115
+ "model.layers.18.self_attn.q_proj.weight": "pytorch_model-00007-of-00015.bin",
116
+ "model.layers.18.self_attn.rotary_emb.inv_freq": "pytorch_model-00007-of-00015.bin",
117
+ "model.layers.18.self_attn.v_proj.weight": "pytorch_model-00007-of-00015.bin",
118
+ "model.layers.19.input_layernorm.weight": "pytorch_model-00007-of-00015.bin",
119
+ "model.layers.19.mlp.down_proj.weight": "pytorch_model-00008-of-00015.bin",
120
+ "model.layers.19.mlp.gate_proj.weight": "pytorch_model-00008-of-00015.bin",
121
+ "model.layers.19.mlp.up_proj.weight": "pytorch_model-00008-of-00015.bin",
122
+ "model.layers.19.post_attention_layernorm.weight": "pytorch_model-00007-of-00015.bin",
123
+ "model.layers.19.self_attn.k_proj.weight": "pytorch_model-00007-of-00015.bin",
124
+ "model.layers.19.self_attn.o_proj.weight": "pytorch_model-00007-of-00015.bin",
125
+ "model.layers.19.self_attn.q_proj.weight": "pytorch_model-00007-of-00015.bin",
126
+ "model.layers.19.self_attn.rotary_emb.inv_freq": "pytorch_model-00007-of-00015.bin",
127
+ "model.layers.19.self_attn.v_proj.weight": "pytorch_model-00007-of-00015.bin",
128
+ "model.layers.2.input_layernorm.weight": "pytorch_model-00002-of-00015.bin",
129
+ "model.layers.2.mlp.down_proj.weight": "pytorch_model-00002-of-00015.bin",
130
+ "model.layers.2.mlp.gate_proj.weight": "pytorch_model-00002-of-00015.bin",
131
+ "model.layers.2.mlp.up_proj.weight": "pytorch_model-00002-of-00015.bin",
132
+ "model.layers.2.post_attention_layernorm.weight": "pytorch_model-00002-of-00015.bin",
133
+ "model.layers.2.self_attn.k_proj.weight": "pytorch_model-00002-of-00015.bin",
134
+ "model.layers.2.self_attn.o_proj.weight": "pytorch_model-00002-of-00015.bin",
135
+ "model.layers.2.self_attn.q_proj.weight": "pytorch_model-00002-of-00015.bin",
136
+ "model.layers.2.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00015.bin",
137
+ "model.layers.2.self_attn.v_proj.weight": "pytorch_model-00002-of-00015.bin",
138
+ "model.layers.20.input_layernorm.weight": "pytorch_model-00008-of-00015.bin",
139
+ "model.layers.20.mlp.down_proj.weight": "pytorch_model-00008-of-00015.bin",
140
+ "model.layers.20.mlp.gate_proj.weight": "pytorch_model-00008-of-00015.bin",
141
+ "model.layers.20.mlp.up_proj.weight": "pytorch_model-00008-of-00015.bin",
142
+ "model.layers.20.post_attention_layernorm.weight": "pytorch_model-00008-of-00015.bin",
143
+ "model.layers.20.self_attn.k_proj.weight": "pytorch_model-00008-of-00015.bin",
144
+ "model.layers.20.self_attn.o_proj.weight": "pytorch_model-00008-of-00015.bin",
145
+ "model.layers.20.self_attn.q_proj.weight": "pytorch_model-00008-of-00015.bin",
146
+ "model.layers.20.self_attn.rotary_emb.inv_freq": "pytorch_model-00008-of-00015.bin",
147
+ "model.layers.20.self_attn.v_proj.weight": "pytorch_model-00008-of-00015.bin",
148
+ "model.layers.21.input_layernorm.weight": "pytorch_model-00008-of-00015.bin",
149
+ "model.layers.21.mlp.down_proj.weight": "pytorch_model-00008-of-00015.bin",
150
+ "model.layers.21.mlp.gate_proj.weight": "pytorch_model-00008-of-00015.bin",
151
+ "model.layers.21.mlp.up_proj.weight": "pytorch_model-00008-of-00015.bin",
152
+ "model.layers.21.post_attention_layernorm.weight": "pytorch_model-00008-of-00015.bin",
153
+ "model.layers.21.self_attn.k_proj.weight": "pytorch_model-00008-of-00015.bin",
154
+ "model.layers.21.self_attn.o_proj.weight": "pytorch_model-00008-of-00015.bin",
155
+ "model.layers.21.self_attn.q_proj.weight": "pytorch_model-00008-of-00015.bin",
156
+ "model.layers.21.self_attn.rotary_emb.inv_freq": "pytorch_model-00008-of-00015.bin",
157
+ "model.layers.21.self_attn.v_proj.weight": "pytorch_model-00008-of-00015.bin",
158
+ "model.layers.22.input_layernorm.weight": "pytorch_model-00008-of-00015.bin",
159
+ "model.layers.22.mlp.down_proj.weight": "pytorch_model-00009-of-00015.bin",
160
+ "model.layers.22.mlp.gate_proj.weight": "pytorch_model-00009-of-00015.bin",
161
+ "model.layers.22.mlp.up_proj.weight": "pytorch_model-00009-of-00015.bin",
162
+ "model.layers.22.post_attention_layernorm.weight": "pytorch_model-00008-of-00015.bin",
163
+ "model.layers.22.self_attn.k_proj.weight": "pytorch_model-00008-of-00015.bin",
164
+ "model.layers.22.self_attn.o_proj.weight": "pytorch_model-00008-of-00015.bin",
165
+ "model.layers.22.self_attn.q_proj.weight": "pytorch_model-00008-of-00015.bin",
166
+ "model.layers.22.self_attn.rotary_emb.inv_freq": "pytorch_model-00008-of-00015.bin",
167
+ "model.layers.22.self_attn.v_proj.weight": "pytorch_model-00008-of-00015.bin",
168
+ "model.layers.23.input_layernorm.weight": "pytorch_model-00009-of-00015.bin",
169
+ "model.layers.23.mlp.down_proj.weight": "pytorch_model-00009-of-00015.bin",
170
+ "model.layers.23.mlp.gate_proj.weight": "pytorch_model-00009-of-00015.bin",
171
+ "model.layers.23.mlp.up_proj.weight": "pytorch_model-00009-of-00015.bin",
172
+ "model.layers.23.post_attention_layernorm.weight": "pytorch_model-00009-of-00015.bin",
173
+ "model.layers.23.self_attn.k_proj.weight": "pytorch_model-00009-of-00015.bin",
174
+ "model.layers.23.self_attn.o_proj.weight": "pytorch_model-00009-of-00015.bin",
175
+ "model.layers.23.self_attn.q_proj.weight": "pytorch_model-00009-of-00015.bin",
176
+ "model.layers.23.self_attn.rotary_emb.inv_freq": "pytorch_model-00009-of-00015.bin",
177
+ "model.layers.23.self_attn.v_proj.weight": "pytorch_model-00009-of-00015.bin",
178
+ "model.layers.24.input_layernorm.weight": "pytorch_model-00009-of-00015.bin",
179
+ "model.layers.24.mlp.down_proj.weight": "pytorch_model-00009-of-00015.bin",
180
+ "model.layers.24.mlp.gate_proj.weight": "pytorch_model-00009-of-00015.bin",
181
+ "model.layers.24.mlp.up_proj.weight": "pytorch_model-00009-of-00015.bin",
182
+ "model.layers.24.post_attention_layernorm.weight": "pytorch_model-00009-of-00015.bin",
183
+ "model.layers.24.self_attn.k_proj.weight": "pytorch_model-00009-of-00015.bin",
184
+ "model.layers.24.self_attn.o_proj.weight": "pytorch_model-00009-of-00015.bin",
185
+ "model.layers.24.self_attn.q_proj.weight": "pytorch_model-00009-of-00015.bin",
186
+ "model.layers.24.self_attn.rotary_emb.inv_freq": "pytorch_model-00009-of-00015.bin",
187
+ "model.layers.24.self_attn.v_proj.weight": "pytorch_model-00009-of-00015.bin",
188
+ "model.layers.25.input_layernorm.weight": "pytorch_model-00009-of-00015.bin",
189
+ "model.layers.25.mlp.down_proj.weight": "pytorch_model-00010-of-00015.bin",
190
+ "model.layers.25.mlp.gate_proj.weight": "pytorch_model-00010-of-00015.bin",
191
+ "model.layers.25.mlp.up_proj.weight": "pytorch_model-00010-of-00015.bin",
192
+ "model.layers.25.post_attention_layernorm.weight": "pytorch_model-00009-of-00015.bin",
193
+ "model.layers.25.self_attn.k_proj.weight": "pytorch_model-00009-of-00015.bin",
194
+ "model.layers.25.self_attn.o_proj.weight": "pytorch_model-00009-of-00015.bin",
195
+ "model.layers.25.self_attn.q_proj.weight": "pytorch_model-00009-of-00015.bin",
196
+ "model.layers.25.self_attn.rotary_emb.inv_freq": "pytorch_model-00009-of-00015.bin",
197
+ "model.layers.25.self_attn.v_proj.weight": "pytorch_model-00009-of-00015.bin",
198
+ "model.layers.26.input_layernorm.weight": "pytorch_model-00010-of-00015.bin",
199
+ "model.layers.26.mlp.down_proj.weight": "pytorch_model-00010-of-00015.bin",
200
+ "model.layers.26.mlp.gate_proj.weight": "pytorch_model-00010-of-00015.bin",
201
+ "model.layers.26.mlp.up_proj.weight": "pytorch_model-00010-of-00015.bin",
202
+ "model.layers.26.post_attention_layernorm.weight": "pytorch_model-00010-of-00015.bin",
203
+ "model.layers.26.self_attn.k_proj.weight": "pytorch_model-00010-of-00015.bin",
204
+ "model.layers.26.self_attn.o_proj.weight": "pytorch_model-00010-of-00015.bin",
205
+ "model.layers.26.self_attn.q_proj.weight": "pytorch_model-00010-of-00015.bin",
206
+ "model.layers.26.self_attn.rotary_emb.inv_freq": "pytorch_model-00010-of-00015.bin",
207
+ "model.layers.26.self_attn.v_proj.weight": "pytorch_model-00010-of-00015.bin",
208
+ "model.layers.27.input_layernorm.weight": "pytorch_model-00010-of-00015.bin",
209
+ "model.layers.27.mlp.down_proj.weight": "pytorch_model-00010-of-00015.bin",
210
+ "model.layers.27.mlp.gate_proj.weight": "pytorch_model-00010-of-00015.bin",
211
+ "model.layers.27.mlp.up_proj.weight": "pytorch_model-00010-of-00015.bin",
212
+ "model.layers.27.post_attention_layernorm.weight": "pytorch_model-00010-of-00015.bin",
213
+ "model.layers.27.self_attn.k_proj.weight": "pytorch_model-00010-of-00015.bin",
214
+ "model.layers.27.self_attn.o_proj.weight": "pytorch_model-00010-of-00015.bin",
215
+ "model.layers.27.self_attn.q_proj.weight": "pytorch_model-00010-of-00015.bin",
216
+ "model.layers.27.self_attn.rotary_emb.inv_freq": "pytorch_model-00010-of-00015.bin",
217
+ "model.layers.27.self_attn.v_proj.weight": "pytorch_model-00010-of-00015.bin",
218
+ "model.layers.28.input_layernorm.weight": "pytorch_model-00010-of-00015.bin",
219
+ "model.layers.28.mlp.down_proj.weight": "pytorch_model-00011-of-00015.bin",
220
+ "model.layers.28.mlp.gate_proj.weight": "pytorch_model-00011-of-00015.bin",
221
+ "model.layers.28.mlp.up_proj.weight": "pytorch_model-00011-of-00015.bin",
222
+ "model.layers.28.post_attention_layernorm.weight": "pytorch_model-00010-of-00015.bin",
223
+ "model.layers.28.self_attn.k_proj.weight": "pytorch_model-00010-of-00015.bin",
224
+ "model.layers.28.self_attn.o_proj.weight": "pytorch_model-00010-of-00015.bin",
225
+ "model.layers.28.self_attn.q_proj.weight": "pytorch_model-00010-of-00015.bin",
226
+ "model.layers.28.self_attn.rotary_emb.inv_freq": "pytorch_model-00010-of-00015.bin",
227
+ "model.layers.28.self_attn.v_proj.weight": "pytorch_model-00010-of-00015.bin",
228
+ "model.layers.29.input_layernorm.weight": "pytorch_model-00011-of-00015.bin",
229
+ "model.layers.29.mlp.down_proj.weight": "pytorch_model-00011-of-00015.bin",
230
+ "model.layers.29.mlp.gate_proj.weight": "pytorch_model-00011-of-00015.bin",
231
+ "model.layers.29.mlp.up_proj.weight": "pytorch_model-00011-of-00015.bin",
232
+ "model.layers.29.post_attention_layernorm.weight": "pytorch_model-00011-of-00015.bin",
233
+ "model.layers.29.self_attn.k_proj.weight": "pytorch_model-00011-of-00015.bin",
234
+ "model.layers.29.self_attn.o_proj.weight": "pytorch_model-00011-of-00015.bin",
235
+ "model.layers.29.self_attn.q_proj.weight": "pytorch_model-00011-of-00015.bin",
236
+ "model.layers.29.self_attn.rotary_emb.inv_freq": "pytorch_model-00011-of-00015.bin",
237
+ "model.layers.29.self_attn.v_proj.weight": "pytorch_model-00011-of-00015.bin",
238
+ "model.layers.3.input_layernorm.weight": "pytorch_model-00002-of-00015.bin",
239
+ "model.layers.3.mlp.down_proj.weight": "pytorch_model-00002-of-00015.bin",
240
+ "model.layers.3.mlp.gate_proj.weight": "pytorch_model-00002-of-00015.bin",
241
+ "model.layers.3.mlp.up_proj.weight": "pytorch_model-00002-of-00015.bin",
242
+ "model.layers.3.post_attention_layernorm.weight": "pytorch_model-00002-of-00015.bin",
243
+ "model.layers.3.self_attn.k_proj.weight": "pytorch_model-00002-of-00015.bin",
244
+ "model.layers.3.self_attn.o_proj.weight": "pytorch_model-00002-of-00015.bin",
245
+ "model.layers.3.self_attn.q_proj.weight": "pytorch_model-00002-of-00015.bin",
246
+ "model.layers.3.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00015.bin",
247
+ "model.layers.3.self_attn.v_proj.weight": "pytorch_model-00002-of-00015.bin",
248
+ "model.layers.30.input_layernorm.weight": "pytorch_model-00011-of-00015.bin",
249
+ "model.layers.30.mlp.down_proj.weight": "pytorch_model-00011-of-00015.bin",
250
+ "model.layers.30.mlp.gate_proj.weight": "pytorch_model-00011-of-00015.bin",
251
+ "model.layers.30.mlp.up_proj.weight": "pytorch_model-00011-of-00015.bin",
252
+ "model.layers.30.post_attention_layernorm.weight": "pytorch_model-00011-of-00015.bin",
253
+ "model.layers.30.self_attn.k_proj.weight": "pytorch_model-00011-of-00015.bin",
254
+ "model.layers.30.self_attn.o_proj.weight": "pytorch_model-00011-of-00015.bin",
255
+ "model.layers.30.self_attn.q_proj.weight": "pytorch_model-00011-of-00015.bin",
256
+ "model.layers.30.self_attn.rotary_emb.inv_freq": "pytorch_model-00011-of-00015.bin",
257
+ "model.layers.30.self_attn.v_proj.weight": "pytorch_model-00011-of-00015.bin",
258
+ "model.layers.31.input_layernorm.weight": "pytorch_model-00011-of-00015.bin",
259
+ "model.layers.31.mlp.down_proj.weight": "pytorch_model-00012-of-00015.bin",
260
+ "model.layers.31.mlp.gate_proj.weight": "pytorch_model-00012-of-00015.bin",
261
+ "model.layers.31.mlp.up_proj.weight": "pytorch_model-00012-of-00015.bin",
262
+ "model.layers.31.post_attention_layernorm.weight": "pytorch_model-00011-of-00015.bin",
263
+ "model.layers.31.self_attn.k_proj.weight": "pytorch_model-00011-of-00015.bin",
264
+ "model.layers.31.self_attn.o_proj.weight": "pytorch_model-00011-of-00015.bin",
265
+ "model.layers.31.self_attn.q_proj.weight": "pytorch_model-00011-of-00015.bin",
266
+ "model.layers.31.self_attn.rotary_emb.inv_freq": "pytorch_model-00011-of-00015.bin",
267
+ "model.layers.31.self_attn.v_proj.weight": "pytorch_model-00011-of-00015.bin",
268
+ "model.layers.32.input_layernorm.weight": "pytorch_model-00012-of-00015.bin",
269
+ "model.layers.32.mlp.down_proj.weight": "pytorch_model-00012-of-00015.bin",
270
+ "model.layers.32.mlp.gate_proj.weight": "pytorch_model-00012-of-00015.bin",
271
+ "model.layers.32.mlp.up_proj.weight": "pytorch_model-00012-of-00015.bin",
272
+ "model.layers.32.post_attention_layernorm.weight": "pytorch_model-00012-of-00015.bin",
273
+ "model.layers.32.self_attn.k_proj.weight": "pytorch_model-00012-of-00015.bin",
274
+ "model.layers.32.self_attn.o_proj.weight": "pytorch_model-00012-of-00015.bin",
275
+ "model.layers.32.self_attn.q_proj.weight": "pytorch_model-00012-of-00015.bin",
276
+ "model.layers.32.self_attn.rotary_emb.inv_freq": "pytorch_model-00012-of-00015.bin",
277
+ "model.layers.32.self_attn.v_proj.weight": "pytorch_model-00012-of-00015.bin",
278
+ "model.layers.33.input_layernorm.weight": "pytorch_model-00012-of-00015.bin",
279
+ "model.layers.33.mlp.down_proj.weight": "pytorch_model-00012-of-00015.bin",
280
+ "model.layers.33.mlp.gate_proj.weight": "pytorch_model-00012-of-00015.bin",
281
+ "model.layers.33.mlp.up_proj.weight": "pytorch_model-00012-of-00015.bin",
282
+ "model.layers.33.post_attention_layernorm.weight": "pytorch_model-00012-of-00015.bin",
283
+ "model.layers.33.self_attn.k_proj.weight": "pytorch_model-00012-of-00015.bin",
284
+ "model.layers.33.self_attn.o_proj.weight": "pytorch_model-00012-of-00015.bin",
285
+ "model.layers.33.self_attn.q_proj.weight": "pytorch_model-00012-of-00015.bin",
286
+ "model.layers.33.self_attn.rotary_emb.inv_freq": "pytorch_model-00012-of-00015.bin",
287
+ "model.layers.33.self_attn.v_proj.weight": "pytorch_model-00012-of-00015.bin",
288
+ "model.layers.34.input_layernorm.weight": "pytorch_model-00012-of-00015.bin",
289
+ "model.layers.34.mlp.down_proj.weight": "pytorch_model-00013-of-00015.bin",
290
+ "model.layers.34.mlp.gate_proj.weight": "pytorch_model-00013-of-00015.bin",
291
+ "model.layers.34.mlp.up_proj.weight": "pytorch_model-00013-of-00015.bin",
292
+ "model.layers.34.post_attention_layernorm.weight": "pytorch_model-00012-of-00015.bin",
293
+ "model.layers.34.self_attn.k_proj.weight": "pytorch_model-00012-of-00015.bin",
294
+ "model.layers.34.self_attn.o_proj.weight": "pytorch_model-00012-of-00015.bin",
295
+ "model.layers.34.self_attn.q_proj.weight": "pytorch_model-00012-of-00015.bin",
296
+ "model.layers.34.self_attn.rotary_emb.inv_freq": "pytorch_model-00012-of-00015.bin",
297
+ "model.layers.34.self_attn.v_proj.weight": "pytorch_model-00012-of-00015.bin",
298
+ "model.layers.35.input_layernorm.weight": "pytorch_model-00013-of-00015.bin",
299
+ "model.layers.35.mlp.down_proj.weight": "pytorch_model-00013-of-00015.bin",
300
+ "model.layers.35.mlp.gate_proj.weight": "pytorch_model-00013-of-00015.bin",
301
+ "model.layers.35.mlp.up_proj.weight": "pytorch_model-00013-of-00015.bin",
302
+ "model.layers.35.post_attention_layernorm.weight": "pytorch_model-00013-of-00015.bin",
303
+ "model.layers.35.self_attn.k_proj.weight": "pytorch_model-00013-of-00015.bin",
304
+ "model.layers.35.self_attn.o_proj.weight": "pytorch_model-00013-of-00015.bin",
305
+ "model.layers.35.self_attn.q_proj.weight": "pytorch_model-00013-of-00015.bin",
306
+ "model.layers.35.self_attn.rotary_emb.inv_freq": "pytorch_model-00013-of-00015.bin",
307
+ "model.layers.35.self_attn.v_proj.weight": "pytorch_model-00013-of-00015.bin",
308
+ "model.layers.36.input_layernorm.weight": "pytorch_model-00013-of-00015.bin",
309
+ "model.layers.36.mlp.down_proj.weight": "pytorch_model-00013-of-00015.bin",
310
+ "model.layers.36.mlp.gate_proj.weight": "pytorch_model-00013-of-00015.bin",
311
+ "model.layers.36.mlp.up_proj.weight": "pytorch_model-00013-of-00015.bin",
312
+ "model.layers.36.post_attention_layernorm.weight": "pytorch_model-00013-of-00015.bin",
313
+ "model.layers.36.self_attn.k_proj.weight": "pytorch_model-00013-of-00015.bin",
314
+ "model.layers.36.self_attn.o_proj.weight": "pytorch_model-00013-of-00015.bin",
315
+ "model.layers.36.self_attn.q_proj.weight": "pytorch_model-00013-of-00015.bin",
316
+ "model.layers.36.self_attn.rotary_emb.inv_freq": "pytorch_model-00013-of-00015.bin",
317
+ "model.layers.36.self_attn.v_proj.weight": "pytorch_model-00013-of-00015.bin",
318
+ "model.layers.37.input_layernorm.weight": "pytorch_model-00013-of-00015.bin",
319
+ "model.layers.37.mlp.down_proj.weight": "pytorch_model-00014-of-00015.bin",
320
+ "model.layers.37.mlp.gate_proj.weight": "pytorch_model-00014-of-00015.bin",
321
+ "model.layers.37.mlp.up_proj.weight": "pytorch_model-00014-of-00015.bin",
322
+ "model.layers.37.post_attention_layernorm.weight": "pytorch_model-00013-of-00015.bin",
323
+ "model.layers.37.self_attn.k_proj.weight": "pytorch_model-00013-of-00015.bin",
324
+ "model.layers.37.self_attn.o_proj.weight": "pytorch_model-00013-of-00015.bin",
325
+ "model.layers.37.self_attn.q_proj.weight": "pytorch_model-00013-of-00015.bin",
326
+ "model.layers.37.self_attn.rotary_emb.inv_freq": "pytorch_model-00013-of-00015.bin",
327
+ "model.layers.37.self_attn.v_proj.weight": "pytorch_model-00013-of-00015.bin",
328
+ "model.layers.38.input_layernorm.weight": "pytorch_model-00014-of-00015.bin",
329
+ "model.layers.38.mlp.down_proj.weight": "pytorch_model-00014-of-00015.bin",
330
+ "model.layers.38.mlp.gate_proj.weight": "pytorch_model-00014-of-00015.bin",
331
+ "model.layers.38.mlp.up_proj.weight": "pytorch_model-00014-of-00015.bin",
332
+ "model.layers.38.post_attention_layernorm.weight": "pytorch_model-00014-of-00015.bin",
333
+ "model.layers.38.self_attn.k_proj.weight": "pytorch_model-00014-of-00015.bin",
334
+ "model.layers.38.self_attn.o_proj.weight": "pytorch_model-00014-of-00015.bin",
335
+ "model.layers.38.self_attn.q_proj.weight": "pytorch_model-00014-of-00015.bin",
336
+ "model.layers.38.self_attn.rotary_emb.inv_freq": "pytorch_model-00014-of-00015.bin",
337
+ "model.layers.38.self_attn.v_proj.weight": "pytorch_model-00014-of-00015.bin",
338
+ "model.layers.39.input_layernorm.weight": "pytorch_model-00014-of-00015.bin",
339
+ "model.layers.39.mlp.down_proj.weight": "pytorch_model-00014-of-00015.bin",
340
+ "model.layers.39.mlp.gate_proj.weight": "pytorch_model-00014-of-00015.bin",
341
+ "model.layers.39.mlp.up_proj.weight": "pytorch_model-00014-of-00015.bin",
342
+ "model.layers.39.post_attention_layernorm.weight": "pytorch_model-00014-of-00015.bin",
343
+ "model.layers.39.self_attn.k_proj.weight": "pytorch_model-00014-of-00015.bin",
344
+ "model.layers.39.self_attn.o_proj.weight": "pytorch_model-00014-of-00015.bin",
345
+ "model.layers.39.self_attn.q_proj.weight": "pytorch_model-00014-of-00015.bin",
346
+ "model.layers.39.self_attn.rotary_emb.inv_freq": "pytorch_model-00014-of-00015.bin",
347
+ "model.layers.39.self_attn.v_proj.weight": "pytorch_model-00014-of-00015.bin",
348
+ "model.layers.4.input_layernorm.weight": "pytorch_model-00002-of-00015.bin",
349
+ "model.layers.4.mlp.down_proj.weight": "pytorch_model-00003-of-00015.bin",
350
+ "model.layers.4.mlp.gate_proj.weight": "pytorch_model-00003-of-00015.bin",
351
+ "model.layers.4.mlp.up_proj.weight": "pytorch_model-00003-of-00015.bin",
352
+ "model.layers.4.post_attention_layernorm.weight": "pytorch_model-00002-of-00015.bin",
353
+ "model.layers.4.self_attn.k_proj.weight": "pytorch_model-00002-of-00015.bin",
354
+ "model.layers.4.self_attn.o_proj.weight": "pytorch_model-00002-of-00015.bin",
355
+ "model.layers.4.self_attn.q_proj.weight": "pytorch_model-00002-of-00015.bin",
356
+ "model.layers.4.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00015.bin",
357
+ "model.layers.4.self_attn.v_proj.weight": "pytorch_model-00002-of-00015.bin",
358
+ "model.layers.5.input_layernorm.weight": "pytorch_model-00003-of-00015.bin",
359
+ "model.layers.5.mlp.down_proj.weight": "pytorch_model-00003-of-00015.bin",
360
+ "model.layers.5.mlp.gate_proj.weight": "pytorch_model-00003-of-00015.bin",
361
+ "model.layers.5.mlp.up_proj.weight": "pytorch_model-00003-of-00015.bin",
362
+ "model.layers.5.post_attention_layernorm.weight": "pytorch_model-00003-of-00015.bin",
363
+ "model.layers.5.self_attn.k_proj.weight": "pytorch_model-00003-of-00015.bin",
364
+ "model.layers.5.self_attn.o_proj.weight": "pytorch_model-00003-of-00015.bin",
365
+ "model.layers.5.self_attn.q_proj.weight": "pytorch_model-00003-of-00015.bin",
366
+ "model.layers.5.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00015.bin",
367
+ "model.layers.5.self_attn.v_proj.weight": "pytorch_model-00003-of-00015.bin",
368
+ "model.layers.6.input_layernorm.weight": "pytorch_model-00003-of-00015.bin",
369
+ "model.layers.6.mlp.down_proj.weight": "pytorch_model-00003-of-00015.bin",
370
+ "model.layers.6.mlp.gate_proj.weight": "pytorch_model-00003-of-00015.bin",
371
+ "model.layers.6.mlp.up_proj.weight": "pytorch_model-00003-of-00015.bin",
372
+ "model.layers.6.post_attention_layernorm.weight": "pytorch_model-00003-of-00015.bin",
373
+ "model.layers.6.self_attn.k_proj.weight": "pytorch_model-00003-of-00015.bin",
374
+ "model.layers.6.self_attn.o_proj.weight": "pytorch_model-00003-of-00015.bin",
375
+ "model.layers.6.self_attn.q_proj.weight": "pytorch_model-00003-of-00015.bin",
376
+ "model.layers.6.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00015.bin",
377
+ "model.layers.6.self_attn.v_proj.weight": "pytorch_model-00003-of-00015.bin",
378
+ "model.layers.7.input_layernorm.weight": "pytorch_model-00003-of-00015.bin",
379
+ "model.layers.7.mlp.down_proj.weight": "pytorch_model-00004-of-00015.bin",
380
+ "model.layers.7.mlp.gate_proj.weight": "pytorch_model-00004-of-00015.bin",
381
+ "model.layers.7.mlp.up_proj.weight": "pytorch_model-00004-of-00015.bin",
382
+ "model.layers.7.post_attention_layernorm.weight": "pytorch_model-00003-of-00015.bin",
383
+ "model.layers.7.self_attn.k_proj.weight": "pytorch_model-00003-of-00015.bin",
384
+ "model.layers.7.self_attn.o_proj.weight": "pytorch_model-00003-of-00015.bin",
385
+ "model.layers.7.self_attn.q_proj.weight": "pytorch_model-00003-of-00015.bin",
386
+ "model.layers.7.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00015.bin",
387
+ "model.layers.7.self_attn.v_proj.weight": "pytorch_model-00003-of-00015.bin",
388
+ "model.layers.8.input_layernorm.weight": "pytorch_model-00004-of-00015.bin",
389
+ "model.layers.8.mlp.down_proj.weight": "pytorch_model-00004-of-00015.bin",
390
+ "model.layers.8.mlp.gate_proj.weight": "pytorch_model-00004-of-00015.bin",
391
+ "model.layers.8.mlp.up_proj.weight": "pytorch_model-00004-of-00015.bin",
392
+ "model.layers.8.post_attention_layernorm.weight": "pytorch_model-00004-of-00015.bin",
393
+ "model.layers.8.self_attn.k_proj.weight": "pytorch_model-00004-of-00015.bin",
394
+ "model.layers.8.self_attn.o_proj.weight": "pytorch_model-00004-of-00015.bin",
395
+ "model.layers.8.self_attn.q_proj.weight": "pytorch_model-00004-of-00015.bin",
396
+ "model.layers.8.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00015.bin",
397
+ "model.layers.8.self_attn.v_proj.weight": "pytorch_model-00004-of-00015.bin",
398
+ "model.layers.9.input_layernorm.weight": "pytorch_model-00004-of-00015.bin",
399
+ "model.layers.9.mlp.down_proj.weight": "pytorch_model-00004-of-00015.bin",
400
+ "model.layers.9.mlp.gate_proj.weight": "pytorch_model-00004-of-00015.bin",
401
+ "model.layers.9.mlp.up_proj.weight": "pytorch_model-00004-of-00015.bin",
402
+ "model.layers.9.post_attention_layernorm.weight": "pytorch_model-00004-of-00015.bin",
403
+ "model.layers.9.self_attn.k_proj.weight": "pytorch_model-00004-of-00015.bin",
404
+ "model.layers.9.self_attn.o_proj.weight": "pytorch_model-00004-of-00015.bin",
405
+ "model.layers.9.self_attn.q_proj.weight": "pytorch_model-00004-of-00015.bin",
406
+ "model.layers.9.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00015.bin",
407
+ "model.layers.9.self_attn.v_proj.weight": "pytorch_model-00004-of-00015.bin",
408
+ "model.norm.weight": "pytorch_model-00014-of-00015.bin"
409
+ }
410
+ }
quantization.py ADDED
@@ -0,0 +1,124 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import bz2
2
+ import torch
3
+ import base64
4
+ import ctypes
5
+ from transformers.utils import logging
6
+ from typing import List
7
+
8
+ logger = logging.get_logger(__name__)
9
+
10
+ try:
11
+ from cpm_kernels.kernels.base import LazyKernelCModule, KernelFunction, round_up
12
+
13
+ class Kernel:
14
+ def __init__(self, code: bytes, function_names: List[str]):
15
+ self.code = code
16
+ self._function_names = function_names
17
+ self._cmodule = LazyKernelCModule(self.code)
18
+
19
+ for name in self._function_names:
20
+ setattr(self, name, KernelFunction(self._cmodule, name))
21
+
22
+ quantization_code = "QlpoOTFBWSZTWapgbn4ALTZ/////////9f/n9+/r/v//3/Tt7cDwfe5sdXXdZNR/9++P4BkfAfIVSVACQUFCEIJQoJUqkgAACJRSqAAAQgEqKqqoKoAAVINADQDTCMhoAGCZBpoNMg0Bk0AGRiGmQBghkNGmgAaGgGhkyAAADJoMhBoAaAaYRkNAAwTINNBpkGgMmgAyMQ0yAMEMho00ADQ0A0MmQAAAZNBkINADQDTCMhoAGCZBpoNMg0Bk0AGRiGmQBghkNGmgAaGgGhkyAAADJoMhBoAaAaYRkNAAwTINNBpkGgMmgAyMQ0yAMEMho00ADQ0A0MmQAAAZNBkCapJBTEyQ0nqeSZioDeqNlG1PU2o9Go9TTQ09QaPUD0CPSAA9Q9QPKA0AGQaANHqBo0GgNGgB6gaApSSCAJiIwmhlNGmjEmCYmTKbSnqP1J6m8gp56ppN6pp5R7U9Q9JMjxTbIk9Mj1BtUMCeoaAB+qNHqAyaNpHtoqfqcgV6L7jHRddxuUdeeq6Vy1s1q0PGdR98uY7Jv2di02u25eLvCuiUfprinG43A5DrtXzq4XdXQ7HSYXSK3F/dxtXgcbZptcVjbJtpppp/Jf7H57vPBXRiuNZXU0OXvTI3V0W5xmyvSd95rhM4s5HF6D4c/uT8etNOq3zqacJzz03vLmXS4Ti2aY77qnmeN2ze315l+oX4t89/x/uvSeN03beGb50V1vfv6lfNn12n6b0Nr8O/F31q1q6ToOydk+ONGjRhho7FPPyPwV4R1FsuRZeCbL18PZnyFyjkGq+Ies8bnvM1MaZeeT30nU/N9Pp+n+f+V+X3gHeXPR8p8ufwnqncyf4K4K2m5qaj1z2mi04YafvWjcbODfNTcysmVvfrTZjhVkHhejPQeOeZ4p5nknpOSZORxb5wdM783XnrL1K6lli2Oqt87d6wxoq9A/sudf+W6sMUfSaLwsnTN1/8P6Uyxj/fPzay3SR2UxR1BeGaFV7DFFecsUrcenP69fnv+pkybyK9J3V5ZpqYxqanlMnjm+OqO56ZxeWPhrS0vUt7DlmN1p3pckN+KtJYG+cJ3nHdBXlG5cTg3Ttraw5zi0vVz7jzst5xk3rRqVcjCLknW9BsR79+RYyYzSuLBWVvm9o+83EtLZ672k6tvu1Xo2MWpHxd6r2g8tdw5n8p/3GOByV4H8N+K7p+W5DDcuDhPi+JjMY6U+y/sP1GOntZY8C6+dj0maa1YYysvzXR0N1jWrucLof8TTZyB45+y6nFjg/z69n0TMdBu1kbTHvWR3mZvN+zc00PBYssS6HoPcW7g7bUzKwy1Y9c5ji4ufZYcurGDGGMzGMZTGVjLLBXdcro2lT5SwSfEmSqfBMpVxLGGTKxkwsr2E0T3yYkH6DPqsYy0zTGmO8ZS4Xv7G1YxjaloxJ3Y4vrmqo5m5qT7Jbm/aIca41org/rW6bJxTDgY7u31qhvMTpViiu/uvaK4LEptss0nNYk08D89snSsWLBPxFinvnjR7TGODdV8KyfMup+o2rmRjGLK/ErRxyWPKtRuYxh6DFteVdC1NnqHf1F02DtrErv3bdBsZeZP0VtpNLSxiNYV8BZDqLJ51g9ZXYmWJss2t9i4OJ8hbU3uNobm1cLibk4le1PGjxY2jnS8kcJP1JZLuhncnnjScVhX7Bf0TVXyy+CakeSMk9Bo1X0JyPKbQsrvLVLhWJTwGRXlsVdtcvwGMcN0nrsocD6Y9htSjvMReLCx9i9PaU+wwScpkj02EXpV51fKk2qHW7zRTUydk9k5W1epWXltGktGU6S7DMsmWWJcxkc1W0m6PhRhYsCxZL3C7btuqrcq3WL8A63U2WP1WO1tL4NZJvna1XblwdrlxtQ2d9p1f2WLBlqpi5Wi8U4LVjnY/SYXU3Rq4xpUPZuStUdjrvTi8StNHaOS7VVxVwrt2rmrrDTmS5jRWjE5phjDkrFwum1W9LhYai4OuNSca3JbzDYqWjAuPPedpdWzRlWWKbqV5NcDUjDI5JhoyXKsfnPOvymmOFW5RwmTzrLo78mvRdl+A4LhXCuC6vWrDsaNDNrSw4zDS9V6q3t0my3XDK1bB0VlTaHI5myxblfqn9ph0VwriA4jHMZYZRcito3m83W+sWTG4piuqytrJYLFtq1ZmWrWlixZLGGq4XdDRxOJqjz4vZWR58ZXfRlGoy/KlkNRqlvj8mS2W5ZZVe7NHKN9cpyngzbY2NWsWdWrzPM7/3H1f0/1nJV/9OXU6w0yw8NtNYGbJeJjZbXUZjzMeJuvKYvgNNGMaZuTE3G9h8dPBN28mo8L0a3E7lcK0NKyi98dIXNfoHSPUHGcbJ4U945n0jGn0V2D//d45VNjue9eC1k+Vew1Yw2N3FPZbq2sq3LFyOg8j77ctzkUeAxWLCauB1upch/ib0/JMowwyxNjIY5jUuaOjyVs6KsrnhzGczEtssWSy3WZjGWquxNDRgyMpjDEwWW1qWjKxZczLanB0vvuh6Det7gOpYzGctNLLdWaNMzTLS1WYmGWDbGGlq5lhpuatWtGpWI3GW0ppqqmcV2nfcW9Li52FojtWJXwcFeN7E/Dr48+G+292xp/hf5Z+M/4HRWXtpdq/jX+N5q8ReyK6tHoXyi1DsX0zwW1tlcHkN7H7C7l5a1au+sfwL4N/guN5li+BfvL3t7w+nPeP6Ttr6k/Xfqz91+3P3z2z+uXJWj9Gp9lZtXhpYeRf16RelfDx5tbPBs9IxVdowJ7d8Vgww0X23RWkyspfOYn4RMfiPntI+pVl819O+l+A+E3Mzsf5181q97fOMfUTc3ORjmhxXpjFi0dN4LXdsuOq0undqvur5Jj131ThWPoL9ReB1dK/E415JMC+Sb00nVdHXOqch5ZwTvpe7ucytpPsa61y1zLdMcNKfhmzcdLdGVkaYqaMF9A4tUri6XUxtI6jjNhyzlZvmuFbm0NWBqc60TVdl+df6LtVc9bzhGLmSxLtutNy6gwp1kuFcK61ul4w55LtqtVbE/kNzSvCyR4THEyI3MJyMFYeU1OUMlY7n6A69wOl2Hm2bRt2fRlz210T7Lc4hvrIOFXItTLIwadavpRz19Z+6uVe6/vLwq9B6DnH1ZNNy7l1suc1Xk8Tsq83CrvPCP5zXUaa7xyOLzPx3nfoOqNMqu5mzLXpOB/qHkk3sk00mzELVhJwq5G5vV3q3rV31jRjRpXKYlcjkYfTPVN0e+dTi7Ha6nA4Pf16L+0+S3vxntMeQ94dFs3PbPVaaY0eJ4n12LDmerNzmdVeo3jZ0Wn6TztPAY7nfr8r+XufE+rp/K9z+XyPC3zD0WGP902PXvh9ravQvM7fjz584OBXqDj4V5K9ls8TG3C9byv9tsbzDmcrGcWr23E5XqOven2Hqtzmb/OdBj4Drb3oZvxh81qTZstK3MaYx7xps9tonI3mqnla6q+BuWx2T+O/vv8b3vl9Zvdb+/XMruD0z4rpdzQxeVbOR3q5J3THBvPA3tzyP8BTnGH3jENMMyxlMsxg0sxi1aasZS633V9BdDpYx0nq+E6HvHS1fMdRp53c077vvVNjnnfdbZ52HQ6GnOx52mlxMrifkJ3P4H333X1n663ud4n8Jzl5TmdDyPmF+c/cd5xcgD8x0v9puXlOx0HfeL4ryD2fBXTXbX0a+jX5E9weUcTicTicTicTicFXA4HA4HA4HhXm+VetOVcxh4p3LV5Myp1UvGPRPcnqD8Q2NjY2NjY2NjhXuUserunePXK67236bxvkMfCcyu2uZ8dy258NqOlwrcHJZecxORjTpelVvbljocqbI53xWZjlcXl+W+Yuze35blS9+MVctlmKc2VdNh8rjTwHcX5rdXynpPYdFmPw30HBte5bNNm+1egy2vC8TqbNNmm1dK8VtX/s+e/i9jdU8BiDhed2um3V2saPM4ydlp2Mb2m+652Ox8KsbrLlnh5KPVGSded05W584ceNxVev7y3XJbo7DptKOplXrsE6WSfEN903xvyZR8B8R8R73dJyhwjtPcJ6NNyK8lWI3PXbm0bOq7/HpmrdUdC4GjvbLjYsWJ7z4J49j4Vo0f4U/bb6zMf9OGlu3zJ/2za/xP8fG4eAOc7E6ho+uWLqGPFxd727L8aq5f/b0n/MtrjX1r+Bafhm3Ce5WmVixZWNPuNGPdvj2x7rLgZcMMmrlO0Buc7fml0arVwuE47mYw0maYynSMORNTUbjhnI1bGN1XKP2zLGpfJb8ZlwXO0ZjGlz4f1l0LhG6MXzHYHcZVt+EaTxlj/cMm5h+cyHrvzGX8SenN09OZOVkYNPLfqk3e4+8WoXmr5NfbH0XD8iT435PY4uVgxo0000f8q/FvjG+vEzGSuxcLV75vrU8aT9K/GclTcTJMFjIc75taDx2DnXPbrRtjv5qnjxMrpco5Z+NacOENnW0bORzOsxwcH0HFcjneNpC/iri4t9cCfzq/erS70xHTX753jYBxrBisj1cU6LIO2bpNo2V9Qak1TF4H2k8rHsNMewfOXyvwr+McsncOT1j/vObbPYPsLvsHx33mn8R7r+ibzH9Ns7jZp8e/eb3zXBwc5+y24vmOZztqvkN6z2j13or4K9dXiV76vXeeGq9k4GyppTHvZuT4Qr5F+fW9Hxji+kux1Sb57OC+57rR7rBvmJMGTHTVonx3M7xsbN5pvPSq1VirJNVqrpbDtna3zYnfXzK/91fTVP8qb02TYjUFZGUskslT6Bf0WK9rHvY+wN1irdG6NVujaNo1G0exl3aXjnQuQdEyVo7vkktptbrL2024zUv+KZGSslZGRhixZYXVPcHSrpLIFi0/qIG5xjzv7xzvEY3SfivEafafzD+Y0MdLGnGuxWmzHQ/0Di2O9VXI2cF3pzXLdGTnzWtD4T5DsW1eCudDg4ty9qd4/sOJeivWODcvq1xbLmfj1qub6ldHQ8LsOCr126tLjw5utuV6ECx0XBrOH2XlfwMaae0556C/nmGMbDTD9+sDmI/BvcvktXvlb1v+adfWu46W4Z6S0cFvfra47b16DcvCcjg2bMq4OXfbQew9sYxjGLSfaXz13Lr21lllqy2twX75Vfu2AyH0YPf24+yjorQ5+8h0S7dR/rH2qVqDdXUk+Sk5IxXPQf9ir234LZzKm8dy41xkt0PgR76HjQ6JL3SqmSOtJ3UHKcRXRKn+Rsf8j7ir7kC6/UfLOA/TUPSgXdAu4V0VV1q4zxsP4xznEWxYnoSP1ivCrxq7wNVTdC0D0lVOeCvEqOi3R3sLnna9RMsY+MamYxjUf2V661P47v2nXtXzs5wetm+bTIdBwh6e+WvvbVc+bZP7Yy1YvBXOqxJ30PIV7fuqx9SMmMMY7LVjG+S6IyPiqp4Eakui7CvM7Kv2EnFUjjAvfFewVwVOSS7Mao9TDxTlVfClhkspcsB2PcfAe8nYOww6SmqwRlZCysNdLaej4FTwo9nwXBZaWl5uatmmrfbrZbHBYNmrFtbJ7xtbTG6xYerb7LL2UmT9ur54D+5V6tPU3ozGSyZPEV4S6JYfyT7qaNGzVNDs8Feer6i9gfBrFpbzjfYr0Z4huT5zJk5VWwfzX45uNljyNDG05zZZKdtWi/ecFpYbIu+m0cllZZVlZHQPYDnh469YW+YyzGOs1S5DzmlyGLc62l2rDyDKuQ9jai4VknV0tK2tS0sZjDWVzstxY3VaP4afXf6L6++rhYwx42nFYXeLpTqVp3DHVMeFwJplHv2RyzDe41pTmr9dquVk5WI+7tqN7xMY51NrCwxYxWMMMUODVd94IflreuiuDS4Q8B+01PdWIeR7Z2PSboDmZBzTZbm0vlm+YamDe71cHDeLe3NLQx35tNze/Zd4OtZJ4H5jleDtaO1dLTc/Yf6Tit0uDFV/VZHc77k3njf+BwXQY7l3VpE6HQ1bjTVdzLwTiWkXW61jv/bfz2MftP2mn8hsbNmnfe1dlci/cpeBZe9O9Mb3ad5J4ZMoeVvXnvE3Vf8IfuPGfXMf/K309N6h2v/Rji5HmfdXovWedzucx3N7e+s7GzZvK6n3jxvabNmzc+0/BWl3nefl8G5cr7RzMaaP8jFsvuXMcFe0713O7ZfWn3pPaq4rdKejPByrzPUN6cBH6LZsTzcXrHkrrdfMONXA5hhyybH4Ru4LktC3T4dvvdXM7ThXPdPp7m9yTlH6VdKvbzBd4ZjJll3nAi+2sX31iU4m52d+92rlq6g9cftu+HkO1bx/+LuSKcKEhVMDc/A="
23
+
24
+ kernels = Kernel(
25
+ bz2.decompress(base64.b64decode(quantization_code)),
26
+ [
27
+ "weightInt8_int4",
28
+ "weightInt4_fp16",
29
+ "weightInt4_bf16"
30
+ ],
31
+ )
32
+ except Exception as exception:
33
+ kernels = None
34
+ logger.warning("Failed to load cpm_kernels:" + str(exception))
35
+
36
+
37
+ def quantize_int8(weight: torch.Tensor, bit_length: int):
38
+ weight_scale = weight.abs().max(dim=-1).values / ((2 ** (bit_length - 1)) - 1)
39
+ weight_scale = weight_scale.to(torch.float32)
40
+
41
+ weight = torch.round(weight.to(weight_scale.dtype) / weight_scale[:, None]).to(torch.int8)
42
+ return weight, weight_scale
43
+
44
+
45
+ def compress_int4_weight(weight: torch.Tensor):
46
+ with torch.cuda.device(weight.device):
47
+ num_row, num_chan = weight.size(0), weight.size(1)
48
+ num_chan = num_chan // 2
49
+
50
+ int8_weight = torch.empty(num_row, num_chan, dtype=torch.int8, device="cuda")
51
+ stream = torch.cuda.current_stream()
52
+ dim_grid = (num_row, 1, 1)
53
+ dim_block = (min(round_up(num_chan, 32), 1024), 1, 1)
54
+
55
+ kernels.weightInt8_int4(
56
+ dim_grid,
57
+ dim_block,
58
+ 0,
59
+ stream,
60
+ [
61
+ ctypes.c_void_p(weight.data_ptr()),
62
+ ctypes.c_void_p(int8_weight.data_ptr()),
63
+ ctypes.c_int32(num_row),
64
+ ctypes.c_int32(num_chan)
65
+ ],
66
+ )
67
+
68
+ return int8_weight
69
+
70
+
71
+ def dequantize_float(weight: torch.Tensor, weight_scale: torch.Tensor, bit_length: int, input: torch.Tensor):
72
+ if bit_length == 8:
73
+ float_weight = weight.to(input.dtype) * weight_scale.to(input.dtype)[:, None]
74
+ return float_weight
75
+
76
+ assert bit_length == 4, f"unsupported bit length: {bit_length}"
77
+
78
+ func = (
79
+ kernels.weightInt4_fp16 if input.dtype == torch.half else kernels.weightInt4_bf16
80
+ )
81
+ with torch.cuda.device(weight.device):
82
+ num_row, num_chan = weight.size(0), weight.size(1)
83
+
84
+ float_weight = torch.empty(num_row, num_chan * 2, dtype=input.dtype, device="cuda")
85
+ stream = torch.cuda.current_stream()
86
+ dim_grid = (num_row, 1, 1)
87
+ dim_block = (min(round_up(num_chan, 32), 1024), 1, 1)
88
+
89
+ func(
90
+ dim_grid,
91
+ dim_block,
92
+ 0,
93
+ stream,
94
+ [
95
+ ctypes.c_void_p(weight.data_ptr()),
96
+ ctypes.c_void_p(weight_scale.data_ptr()),
97
+ ctypes.c_void_p(float_weight.data_ptr()),
98
+ ctypes.c_int32(num_row),
99
+ ctypes.c_int32(num_chan),
100
+ ],
101
+ )
102
+ return float_weight
103
+
104
+ class QuantizationLinear(torch.nn.Module):
105
+ def __init__(self, bit_length: int, weight: torch.Tensor, device="cuda"):
106
+ super().__init__()
107
+
108
+ self.bit_length = bit_length
109
+
110
+ weight, weight_scale = quantize_int8(weight=weight, bit_length=bit_length)
111
+ if bit_length == 4:
112
+ weight = compress_int4_weight(weight)
113
+
114
+ self.weight = torch.nn.Parameter(weight.to(device), requires_grad=False)
115
+ self.weight_scale = torch.nn.Parameter(weight_scale.to(device), requires_grad=False)
116
+
117
+ def forward(self, input: torch.Tensor):
118
+ input_size = input.size()
119
+
120
+ input = input.contiguous().view(-1, input.size(-1))
121
+ original_weight = dequantize_float(self.weight, self.weight_scale, self.bit_length, input)
122
+
123
+ output = torch.matmul(input, original_weight.t())
124
+ return output.view(*(input_size[:-1] + (self.weight.size(0),)))
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<|startoftext|>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|endoftext|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<pad>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "clean_up_tokenization_spaces": true,
3
+ "model_max_length": 1000000000000000019884624838656,
4
+ "tokenizer_class": "PreTrainedTokenizerFast"
5
+ }