hyx21 commited on
Commit
b99171b
1 Parent(s): 5686bac

Upload 10 files

Browse files
README.md CHANGED
@@ -1,3 +1,151 @@
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ - zh
5
+ tags:
6
+ - MiniCPM
7
+ - ModelBest
8
+ - THUNLP
9
  ---
10
+
11
+
12
+ <div align="center">
13
+ <h1>
14
+ MiniCPM
15
+ </h1>
16
+ </div>
17
+
18
+ <p align="center">
19
+ <a href="https://shengdinghu.notion.site/MiniCPM-c805a17c5c8046398914e47f0542095a?pvs=4" target="_blank">MiniCPM 技术报告</a><a href="https://shengdinghu.notion.site/MiniCPM-Unveiling-the-Potential-of-End-side-Large-Language-Models-d4d3a8c426424654a4e80e42a711cb20?pvs=4" target="_blank"> Technical Report</a> |
20
+ <a href="https://github.com/OpenBMB/OmniLMM/" target="_blank">OmniLMM 多模态模型 Multi-modal Model</a> |
21
+ <a href="https://luca.cn/" target="_blank">CPM-C 千亿模型试用 ~100B Model Trial </a>
22
+ </p>
23
+
24
+ MiniCPM 是面壁与清华大学自然语言处理实验室共同开源的系列端侧语言大模型,主体语言模型 MiniCPM-2B 仅有 24亿(2.4B)的非词嵌入参数量。
25
+ - 经过 SFT 后,MiniCPM 在公开综合性评测集上,MiniCPM 与 Mistral-7B相近(中文、数学、代码能力更优),整体性能超越 Llama2-13B、MPT-30B、Falcon-40B 等模型。
26
+ - 经过 DPO 后,MiniCPM 在当前最接近用户体感的评测集 MTBench上,MiniCPM-2B 也超越了 Llama2-70B-Chat、Vicuna-33B、Mistral-7B-Instruct-v0.1、Zephyr-7B-alpha 等众多代表性开源大模型。
27
+ - 以 MiniCPM-2B 为基础构建端侧多模态大模型 MiniCPM-V,整体性能在同规模模型中实现最佳,超越基于 Phi-2 构建的现有多模态大模型,在部分评测集上达到与 9.6B Qwen-VL-Chat 相当甚至更好的性能。
28
+ - 经过 Int4 量化后,MiniCPM 可在手机上进行部署推理,流式输出速度略高于人类说话速度。MiniCPM-V 也首次跑通了多模态大模型在手机上的部署。
29
+ - 一张1080/2080可高效参数微调,一张3090/4090可全参数微调,一台机器可持续训练 MiniCPM,二次开发成本较低。
30
+
31
+ 我们将完全开源MiniCPM-2B的模型参数供学术研究和有限商用,以及训练过程中的所有Checkpoint和大部分非专有数据供模型机理研究。
32
+
33
+ - 基于MiniCPM-2B的指令微调与人类偏好对**MiniCPM-2B-SFT/DPO。**
34
+ - 基于MiniCPM-2B的多模态模型**MiniCPM-V**,能力超越基于Phi-2的同参数级别多模态模型**。**
35
+ - MiniCPM-2B-SFT/DPO的Int4量化版**MiniCPM-2B-SFT/DPO-Int4。**
36
+ - 基于MLC-LLM、LLMFarm开发的MiniCPM手机端程序,**文本及多模态模型均可在手机端进行推理。**
37
+
38
+
39
+ MiniCPM is an End-Size LLM developed by ModelBest Inc. and TsinghuaNLP, with only 2.4B parameters excluding embeddings.
40
+
41
+ - MiniCPM has very close performance compared with Mistral-7B on open-sourced general benchmarks with better ability on Chinese, Mathmetics and Coding after SFT. The overall performance exceeds Llama2-13B, MPT-30B, Falcon-40B, etc.
42
+ - After DPO, MiniCPM outperforms Llama2-70B-Chat, Vicuna-33B, Mistral-7B-Instruct-v0.1, Zephyr-7B-alpha, etc. on MTBench.
43
+ - MiniCPM-V, based on MiniCPM-2B, achieves the best overall performance among multimodel models of the same scale, surpassing existing multimodal large models built on Phi-2 and achieving performance comparable to or even better than 9.6B Qwen-VL-Chat on some tasks.
44
+ - MiniCPM can be deployed and infer on smartphones, and the speed of streaming output is relatively higher than the verbal speed of human. MiniCPM-V is the first multi-modal models that can be deployed on smartphones.
45
+ - The cost of developing based on MiniCPM is low. Parameter efficient finetuning can be conducted with a single 1080/2080 GPU and full parameter finetuning can be conducted with a 3090/4090 GPU.
46
+
47
+ We release all model parameters for research and limited commercial use. We also release all the checkpoint during training and most public training data for research on model mechanism.
48
+
49
+ - SFT and DPO version based on MiniCPM-2B and human preference: **MiniCPM-2B-SFT/DPO**
50
+ - The multi-modal model **MiniCPM-V** based on MiniCPM-2B, which outperforms models with similar size, i.e., Phi-2
51
+ - The INT4 quantized version **MiniCPM-2B-SFT/DPO-Int4** based on MiniCPM-2B-SFT/DPO
52
+ - Mobile phone application based on MLC-LLM and LLMFarm. Both language model and multimodel model can conduct inference on smartphones.
53
+
54
+ ### 评测结果 Evaluation Results
55
+
56
+ 详细的评测结果位于[github仓库](https://github.com/OpenBMB/MiniCPM?tab=readme-ov-file#%E8%AF%84%E6%B5%8B%E7%BB%93%E6%9E%9C)
57
+
58
+ Detailed evaluation results are in [github repo](https://github.com/OpenBMB/MiniCPM/blob/main/README-en.md#evaluation-results)
59
+
60
+ 注意:我们发现使用Huggingface生成质量略差于vLLM,因此推荐使用vLLM进行测试。我们正在排查原因。
61
+
62
+ Notice: We discovered that the quality of Huggingface generation is slightly lower than vLLM, thus benchmarking using vLLM is recommended.
63
+ We are investigating the cause now.
64
+
65
+ ### 局限性 Limitations
66
+
67
+ - 受限于模型规模,模型可能出现幻觉性问题。其中由于DPO模型生成的回复内容更长,更容易出现幻觉。我们也将持续进行MiniCPM模型的迭代改进;
68
+ - 为了保证在学术研究用途上模型的通用性,我们未对模型进行任何身份认同训练。同时由于我们用ShareGPT开源语料作为部分训练数据,模型可能会输出类似GPT系列模型的身份认同信息;
69
+ - 受限于模型规模,模型的输出受到提示词(prompt)的影响较大,可能多次尝试产生不一致的结果;
70
+ - 受限于模型容量,模型的知识记忆较不准确,后续我们将结合RAG方法来增强模型的知识记忆能力。
71
+
72
+ - Due to limitations in model size, the model may experience hallucinatory issues. As DPO model tend to generate longer response, hallucinations are more likely to occur. We will also continue to iterate and improve the MiniCPM model.
73
+ - To ensure the universality of the model for academic research purposes, we did not conduct any identity training on the model. Meanwhile, as we use ShareGPT open-source corpus as part of the training data, the model may output identity information similar to the GPT series models.
74
+ - Due to the limitation of model size, the output of the model is greatly influenced by prompt words, which may result in inconsistent results from multiple attempts.
75
+ - Due to limited model capacity, the model's knowledge memory is not accurate. In the future, we will combine the RAG method to enhance the model's knowledge memory ability.
76
+
77
+ ## 模型下载 Download
78
+
79
+ | HuggingFace | ModelScope | WiseModel |
80
+ |-------------|------------|-----------|
81
+ |[sft-bf16](https://huggingface.co/openbmb/MiniCPM-2B-sft-bf16)|[sft-bf16](https://modelscope.cn/models/OpenBMB/miniCPM-bf16)|[sft-bf16](https://wisemodel.cn/models/OpenBMB/miniCPM-bf16)
82
+ |[sft-fp32](https://huggingface.co/openbmb/MiniCPM-2B-sft-fp32)|[sft-fp32](https://modelscope.cn/models/OpenBMB/MiniCPM-2B-sft-fp32)|[sft-fp32](https://wisemodel.cn/models/OpenBMB/miniCPM-dpo-fp32)
83
+ |[dpo-bf16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-bf16)|[dpo-bf16](https://modelscope.cn/models/OpenBMB/MiniCPM-2B-dpo-bf16/summary)|[dpo-bf16](https://wisemodel.cn/models/OpenBMB/MiniCPM-2B-dpo-bf16)
84
+ |[dpo-fp16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16)|[dpo-fp16](https://modelscope.cn/models/OpenBMB/MiniCPM-2B-dpo-fp16/)|[dpo-fp16](https://wisemodel.cn/models/OpenBMB/MiniCPM-2B-dpo-fp16)
85
+ |[dpo-fp32](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp32)|[dpo-fp32](https://modelscope.cn/models/OpenBMB/MiniCPM-2B-dpo-fp32)|[dpo-fp32](https://wisemodel.cn/models/OpenBMB/miniCPM-dpo-fp32)
86
+
87
+ ## 模型使用 Usage
88
+
89
+ * 安装`transformers>=4.36.0`以及`accelerate`后,运行以下代码
90
+ * 注意:需要在`from_pretrained`中明确指明模型的数据类型,否则会引起较大计算误差
91
+ * Run the following code after install `transformers>=4.36.0` and `accelerate`
92
+ * Warning: It is necessary to specify the data type of the model clearly in 'from_pretrained', otherwise large calculation errors will be caused
93
+ ```python
94
+ from transformers import AutoModelForCausalLM, AutoTokenizer
95
+ import torch
96
+ torch.manual_seed(0)
97
+
98
+ path = 'openbmb/MiniCPM-2B-sft-bf16'
99
+ tokenizer = AutoTokenizer.from_pretrained(path)
100
+ model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)
101
+
102
+ responds, history = model.chat(tokenizer, "山东省最高的山是哪座山, 它比黄山高还是矮?差距多少?", temperature=0.8, top_p=0.8)
103
+ print(responds)
104
+ ```
105
+
106
+ * 期望输出 Expected Output
107
+ ```shell
108
+ 山东省最高的山是泰山,海拔1545米。
109
+
110
+ 相对于黄山(海拔1864米),泰山海拔较低,相差约319米。
111
+ ```
112
+
113
+ ## 开源协议 LICENSE
114
+
115
+ #### 模型协议 Model LICENSE
116
+
117
+ * 本仓库中代码依照 [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) 协议开源
118
+ * MiniCPM 模型权重的使用则需要遵循 [“通用模型许可协议-来源说明-宣传限制-商业授权”](https://github.com/OpenBMB/General-Model-License/blob/main/%E9%80%9A%E7%94%A8%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE-%E6%9D%A5%E6%BA%90%E8%AF%B4%E6%98%8E-%E5%AE%A3%E4%BC%A0%E9%99%90%E5%88%B6-%E5%95%86%E4%B8%9A%E6%8E%88%E6%9D%83.md)。
119
+ * MiniCPM 模型权重对学术研究完全开放。
120
+ * 如需将模型用于商业用途,请联系cpm@modelbest.cn来获取书面授权,在登记后亦允许免费商业使用。
121
+
122
+ * This repository is released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License.
123
+ * The usage of MiniCPM model weights must strictly follow [the General Model License (GML)](https://github.com/OpenBMB/General-Model-License/blob/main/%E9%80%9A%E7%94%A8%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E5%8D%8F%E8%AE%AE-%E6%9D%A5%E6%BA%90%E8%AF%B4%E6%98%8E-%E5%AE%A3%E4%BC%A0%E9%99%90%E5%88%B6-%E5%95%86%E4%B8%9A%E6%8E%88%E6%9D%83.md).
124
+ * The models and weights of MiniCPM are completely free for academic research.
125
+ * If you intend to utilize the model for commercial purposes, please reach out to cpm@modelbest.cn to obtain the certificate of authorization.
126
+
127
+ #### 声明 Statement
128
+
129
+ * 作为一个语言模型,MiniCPM 通过学习大量的文本来生成内容,但它无法理解、表达个人观点或价值判断,它所输出的任何内容都不代表模型开发者的观点和立场。
130
+ * 因此用户在使用 MiniCPM 生成的内容时,应自行负责对其进行评估和验证。
131
+ * 如果由于使用 MinCPM 开源模型而导致的任何问题,包括但不限于数据安全问题、公共舆论风险,或模型被误导、滥用、传播或不当利用所带来的任何风险和问题,我们将不承担任何责任。
132
+
133
+ * As a language model, MiniCPM generates content by learning from a vast amount of text.
134
+ * However, it does not possess the ability to comprehend or express personal opinions or value judgments.
135
+ * Any content generated by MiniCPM does not represent the viewpoints or positions of the model developers.
136
+ * Therefore, when using content generated by MiniCPM, users should take full responsibility for evaluating and verifying it on their own.
137
+
138
+ <p id="8"></p>
139
+
140
+ ## 工作引用 Citation
141
+
142
+ * 如果觉得MiniCPM有助于您的工作,请考虑引用下列[技术报告](https://shengdinghu.notion.site/MiniCPM-c805a17c5c8046398914e47f0542095a?pvs=4)
143
+ * Please cite our [techinical report](https://shengdinghu.notion.site/MiniCPM-Unveiling-the-Potential-of-End-side-Large-Language-Models-d4d3a8c426424654a4e80e42a711cb20?pvs=4) if you find our work valuable.
144
+
145
+ ```
146
+ @inproceedings{minicpm2024,
147
+ title={MiniCPM:Unveiling the Potential of End-side Large Language Models},
148
+ booktitle={OpenBMB Blog},
149
+ year={2024}
150
+ }
151
+ ```
config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "openbmb/CPM-2B",
3
+ "architectures": [
4
+ "MiniCPMForCausalLM"
5
+ ],
6
+ "auto_map": {
7
+ "AutoConfig": "configuration_minicpm.MiniCPMConfig",
8
+ "AutoModel": "modeling_minicpm.MiniCPMModel",
9
+ "AutoModelForCausalLM": "modeling_minicpm.MiniCPMForCausalLM",
10
+ "AutoModelForSeq2SeqLM": "modeling_minicpm.MiniCPMForCausalLM",
11
+ "AutoModelForSequenceClassification": "modeling_minicpm.MiniCPMForSequenceClassification"
12
+ },
13
+ "bos_token_id": 1,
14
+ "eos_token_id": 2,
15
+ "hidden_act": "silu",
16
+ "hidden_size": 1536,
17
+ "initializer_range": 0.1,
18
+ "intermediate_size": 3840,
19
+ "max_position_embeddings": 4096,
20
+ "num_attention_heads": 24,
21
+ "num_hidden_layers": 52,
22
+ "num_key_value_heads": 8,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_scaling": null,
25
+ "torch_dtype": "bfloat16",
26
+ "transformers_version": "4.36.0",
27
+ "use_cache": true,
28
+ "vocab_size": 73440,
29
+ "scale_emb": 12,
30
+ "dim_model_base": 256,
31
+ "scale_depth": 1.4
32
+ }
configuration_minicpm.py ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
5
+ # and OPT implementations in this library. It has been modified from its
6
+ # original forms to accommodate minor architectural differences compared
7
+ # to GPT-NeoX and OPT used by the Meta AI team that trained the model.
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ """ MiniCPM model configuration"""
21
+
22
+ from transformers.configuration_utils import PretrainedConfig
23
+ from transformers.utils import logging
24
+
25
+
26
+ logger = logging.get_logger(__name__)
27
+
28
+ MINICPM_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
29
+
30
+
31
+ class MiniCPMConfig(PretrainedConfig):
32
+ r"""
33
+ This is the configuration class to store the configuration of a [`MiniCPMModel`]. It is used to instantiate an MiniCPM
34
+ model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
35
+ defaults will yield a similar configuration to that of the MiniCPM-7B.
36
+
37
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
38
+ documentation from [`PretrainedConfig`] for more information.
39
+
40
+
41
+ Args:
42
+ vocab_size (`int`, *optional*, defaults to 32000):
43
+ Vocabulary size of the MiniCPM model. Defines the number of different tokens that can be represented by the
44
+ `inputs_ids` passed when calling [`MiniCPMModel`]
45
+ hidden_size (`int`, *optional*, defaults to 4096):
46
+ Dimension of the hidden representations.
47
+ intermediate_size (`int`, *optional*, defaults to 11008):
48
+ Dimension of the MLP representations.
49
+ num_hidden_layers (`int`, *optional*, defaults to 32):
50
+ Number of hidden layers in the Transformer decoder.
51
+ num_attention_heads (`int`, *optional*, defaults to 32):
52
+ Number of attention heads for each attention layer in the Transformer decoder.
53
+ num_key_value_heads (`int`, *optional*):
54
+ This is the number of key_value heads that should be used to implement Grouped Query Attention. If
55
+ `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
56
+ `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
57
+ converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
58
+ by meanpooling all the original heads within that group. For more details checkout [this
59
+ paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
60
+ `num_attention_heads`.
61
+ hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
62
+ The non-linear activation function (function or string) in the decoder.
63
+ max_position_embeddings (`int`, *optional*, defaults to 2048):
64
+ The maximum sequence length that this model might ever be used with. MiniCPM 1 supports up to 2048 tokens,
65
+ MiniCPM 2 up to 4096, CodeMiniCPM up to 16384.
66
+ initializer_range (`float`, *optional*, defaults to 0.02):
67
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
68
+ rms_norm_eps (`float`, *optional*, defaults to 1e-06):
69
+ The epsilon used by the rms normalization layers.
70
+ use_cache (`bool`, *optional*, defaults to `True`):
71
+ Whether or not the model should return the last key/values attentions (not used by all models). Only
72
+ relevant if `config.is_decoder=True`.
73
+ pad_token_id (`int`, *optional*):
74
+ Padding token id.
75
+ bos_token_id (`int`, *optional*, defaults to 1):
76
+ Beginning of stream token id.
77
+ eos_token_id (`int`, *optional*, defaults to 2):
78
+ End of stream token id.
79
+ pretraining_tp (`int`, *optional*, defaults to 1):
80
+ Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
81
+ document](https://huggingface.co/docs/transformers/parallelism) to understand more about it. This value is
82
+ necessary to ensure exact reproducibility of the pretraining results. Please refer to [this
83
+ issue](https://github.com/pytorch/pytorch/issues/76232).
84
+ tie_word_embeddings (`bool`, *optional*, defaults to `False`):
85
+ Whether to tie weight embeddings
86
+ rope_theta (`float`, *optional*, defaults to 10000.0):
87
+ The base period of the RoPE embeddings.
88
+ rope_scaling (`Dict`, *optional*):
89
+ Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
90
+ strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
91
+ `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
92
+ `max_position_embeddings` to the expected new maximum. See the following thread for more information on how
93
+ these scaling strategies behave:
94
+ https://www.reddit.com/r/LocalMiniCPM/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This is an
95
+ experimental feature, subject to breaking API changes in future versions.
96
+ attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
97
+ Whether to use a bias in the query, key, value and output projection layers during self-attention.
98
+ attention_dropout (`float`, *optional*, defaults to 0.0):
99
+ The dropout ratio for the attention probabilities.
100
+
101
+ ```python
102
+ >>> from transformers import MiniCPMModel, MiniCPMConfig
103
+
104
+ >>> # Initializing a MiniCPM minicpm-7b style configuration
105
+ >>> configuration = MiniCPMConfig()
106
+
107
+ >>> # Initializing a model from the minicpm-7b style configuration
108
+ >>> model = MiniCPMModel(configuration)
109
+
110
+ >>> # Accessing the model configuration
111
+ >>> configuration = model.config
112
+ ```"""
113
+
114
+ model_type = "minicpm"
115
+ keys_to_ignore_at_inference = ["past_key_values"]
116
+
117
+ def __init__(
118
+ self,
119
+ vocab_size=32000,
120
+ hidden_size=4096,
121
+ intermediate_size=11008,
122
+ num_hidden_layers=32,
123
+ num_attention_heads=32,
124
+ num_key_value_heads=None,
125
+ hidden_act="silu",
126
+ max_position_embeddings=2048,
127
+ initializer_range=0.02,
128
+ rms_norm_eps=1e-6,
129
+ use_cache=True,
130
+ pad_token_id=None,
131
+ bos_token_id=1,
132
+ eos_token_id=2,
133
+ pretraining_tp=1,
134
+ tie_word_embeddings=True,
135
+ rope_theta=10000.0,
136
+ rope_scaling=None,
137
+ attention_bias=False,
138
+ attention_dropout=0.0,
139
+ scale_emb=1,
140
+ dim_model_base=1,
141
+ scale_depth=1,
142
+ **kwargs,
143
+ ):
144
+ self.vocab_size = vocab_size
145
+ self.max_position_embeddings = max_position_embeddings
146
+ self.hidden_size = hidden_size
147
+ self.intermediate_size = intermediate_size
148
+ self.num_hidden_layers = num_hidden_layers
149
+ self.num_attention_heads = num_attention_heads
150
+
151
+ # for backward compatibility
152
+ if num_key_value_heads is None:
153
+ num_key_value_heads = num_attention_heads
154
+
155
+ self.num_key_value_heads = num_key_value_heads
156
+ self.hidden_act = hidden_act
157
+ self.initializer_range = initializer_range
158
+ self.rms_norm_eps = rms_norm_eps
159
+ self.pretraining_tp = pretraining_tp
160
+ self.use_cache = use_cache
161
+ self.rope_theta = rope_theta
162
+ self.rope_scaling = rope_scaling
163
+ self._rope_scaling_validation()
164
+ self.attention_bias = attention_bias
165
+ self.attention_dropout = attention_dropout
166
+ self.scale_emb = scale_emb
167
+ self.dim_model_base = dim_model_base
168
+ self.scale_depth = scale_depth
169
+
170
+ super().__init__(
171
+ pad_token_id=pad_token_id,
172
+ bos_token_id=bos_token_id,
173
+ eos_token_id=eos_token_id,
174
+ tie_word_embeddings=tie_word_embeddings,
175
+ **kwargs,
176
+ )
177
+ try:
178
+ import flash_attn
179
+ self._attn_implementation = "flash_attention_2"
180
+ except:
181
+ pass
182
+
183
+ def _rope_scaling_validation(self):
184
+ """
185
+ Validate the `rope_scaling` configuration.
186
+ """
187
+ if self.rope_scaling is None:
188
+ return
189
+
190
+ if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
191
+ raise ValueError(
192
+ "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
193
+ f"got {self.rope_scaling}"
194
+ )
195
+ rope_scaling_type = self.rope_scaling.get("type", None)
196
+ rope_scaling_factor = self.rope_scaling.get("factor", None)
197
+ if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
198
+ raise ValueError(
199
+ f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
200
+ )
201
+ if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
202
+ raise ValueError(f"`rope_scaling`'s factor field must be a float > 1, got {rope_scaling_factor}")
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_sample": true,
3
+ "top_p": 0.8,
4
+ "temperature": 0.8,
5
+ "bos_token_id": 1,
6
+ "eos_token_id": 2
7
+ }
modeling_minicpm.py ADDED
@@ -0,0 +1,1453 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
5
+ # and OPT implementations in this library. It has been modified from its
6
+ # original forms to accommodate minor architectural differences compared
7
+ # to GPT-NeoX and OPT used by the Meta AI team that trained the model.
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ """ PyTorch MiniCPM model."""
21
+ import math
22
+ import warnings
23
+ from typing import List, Optional, Tuple, Union, Dict
24
+
25
+ import torch
26
+ import torch.nn.functional as F
27
+ import torch.utils.checkpoint
28
+ from torch import nn
29
+ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
30
+
31
+ from transformers.activations import ACT2FN
32
+ from transformers.cache_utils import Cache, DynamicCache
33
+ from transformers.modeling_attn_mask_utils import (
34
+ AttentionMaskConverter,
35
+ _prepare_4d_attention_mask,
36
+ _prepare_4d_causal_attention_mask,
37
+ _prepare_4d_causal_attention_mask_for_sdpa,
38
+ )
39
+ from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast, SequenceClassifierOutputWithPast
40
+ from transformers.modeling_utils import PreTrainedModel
41
+ from transformers.pytorch_utils import ALL_LAYERNORM_LAYERS, is_torch_greater_or_equal_than_1_13
42
+ from transformers.utils import (
43
+ add_start_docstrings,
44
+ add_start_docstrings_to_model_forward,
45
+ is_flash_attn_2_available,
46
+ is_flash_attn_greater_or_equal_2_10,
47
+ logging,
48
+ replace_return_docstrings,
49
+ )
50
+ from transformers.utils.import_utils import is_torch_fx_available
51
+ from .configuration_minicpm import MiniCPMConfig
52
+ import re
53
+
54
+ try:
55
+ from flash_attn import flash_attn_func, flash_attn_varlen_func
56
+ from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input # noqa
57
+ except:
58
+ pass
59
+
60
+
61
+ # This makes `_prepare_4d_causal_attention_mask` a leaf function in the FX graph.
62
+ # It means that the function will not be traced through and simply appear as a node in the graph.
63
+ if is_torch_fx_available():
64
+ if not is_torch_greater_or_equal_than_1_13:
65
+ import torch.fx
66
+
67
+ _prepare_4d_causal_attention_mask = torch.fx.wrap(_prepare_4d_causal_attention_mask)
68
+
69
+
70
+ logger = logging.get_logger(__name__)
71
+
72
+ _CONFIG_FOR_DOC = "MiniCPMConfig"
73
+
74
+
75
+ def _get_unpad_data(attention_mask):
76
+ seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
77
+ indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
78
+ max_seqlen_in_batch = seqlens_in_batch.max().item()
79
+ cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.torch.int32), (1, 0))
80
+ return (
81
+ indices,
82
+ cu_seqlens,
83
+ max_seqlen_in_batch,
84
+ )
85
+
86
+
87
+ def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
88
+ warnings.warn(
89
+ "Calling `transformers.models.minicpm.modeling_minicpm._prepare_4d_attention_mask` is deprecated and will be removed in v4.37. Use `transformers.modeling_attn_mask_utils._prepare_4d_attention_mask"
90
+ )
91
+ return _prepare_4d_attention_mask(mask=mask, dtype=dtype, tgt_len=tgt_len)
92
+
93
+
94
+ def _make_causal_mask(
95
+ input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0
96
+ ):
97
+ warnings.warn(
98
+ "Calling `transformers.models.minicpm.modeling_minicpm._make_causal_mask` is deprecated and will be removed in v4.37. Use `transformers.models.minicpm.modeling_minicpm.AttentionMaskConverter._make_causal_mask"
99
+ )
100
+ return AttentionMaskConverter._make_causal_mask(
101
+ input_ids_shape=input_ids_shape, dtype=dtype, device=device, past_key_values_length=past_key_values_length
102
+ )
103
+
104
+ # @torch.jit.script # type: ignore
105
+ def rms_layernorm(hidden: torch.Tensor, weight: torch.Tensor, eps: float):
106
+ old_dtype = hidden.dtype
107
+ variance = hidden.to(torch.float32).pow(2).mean(dim=-1, keepdim=True)
108
+ hidden = (hidden * torch.rsqrt(variance + eps)).to(old_dtype)
109
+ return hidden * weight
110
+
111
+
112
+ class MiniCPMRMSNorm(nn.Module):
113
+ def __init__(self, hidden_size, eps=1e-6):
114
+ """
115
+ MiniCPMRMSNorm is equivalent to T5LayerNorm
116
+ """
117
+ super().__init__()
118
+ self.weight = nn.Parameter(torch.ones(hidden_size))
119
+ self.variance_epsilon = eps
120
+
121
+ def forward(self, hidden_states):
122
+ return rms_layernorm(hidden_states, self.weight, self.variance_epsilon)
123
+
124
+
125
+ ALL_LAYERNORM_LAYERS.append(MiniCPMRMSNorm)
126
+
127
+
128
+ class MiniCPMRotaryEmbedding(nn.Module):
129
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None):
130
+ super().__init__()
131
+
132
+ self.dim = dim
133
+ self.max_position_embeddings = max_position_embeddings
134
+ self.base = base
135
+ inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
136
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
137
+
138
+ # Build here to make `torch.jit.trace` work.
139
+ self._set_cos_sin_cache(
140
+ # seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype()
141
+ seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.float32
142
+ )
143
+
144
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
145
+ self.max_seq_len_cached = seq_len
146
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
147
+ freqs = torch.outer(t, self.inv_freq)
148
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
149
+ emb = torch.cat((freqs, freqs), dim=-1)
150
+
151
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
152
+ self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
153
+
154
+ def forward(self, x, seq_len=None):
155
+ # x: [bs, num_attention_heads, seq_len, head_size]
156
+ if seq_len > self.max_seq_len_cached:
157
+ self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype)
158
+
159
+ return (
160
+ self.cos_cached[:seq_len].to(dtype=x.dtype),
161
+ self.sin_cached[:seq_len].to(dtype=x.dtype),
162
+ )
163
+
164
+
165
+ class MiniCPMLinearScalingRotaryEmbedding(MiniCPMRotaryEmbedding):
166
+ """MiniCPMRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev"""
167
+
168
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
169
+ self.scaling_factor = scaling_factor
170
+ super().__init__(dim, max_position_embeddings, base, device)
171
+
172
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
173
+ self.max_seq_len_cached = seq_len
174
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
175
+ t = t / self.scaling_factor
176
+
177
+ freqs = torch.outer(t, self.inv_freq)
178
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
179
+ emb = torch.cat((freqs, freqs), dim=-1)
180
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
181
+ self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
182
+
183
+
184
+ class MiniCPMDynamicNTKScalingRotaryEmbedding(MiniCPMRotaryEmbedding):
185
+ """MiniCPMRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla"""
186
+
187
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
188
+ self.scaling_factor = scaling_factor
189
+ super().__init__(dim, max_position_embeddings, base, device)
190
+
191
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
192
+ self.max_seq_len_cached = seq_len
193
+
194
+ if seq_len > self.max_position_embeddings:
195
+ base = self.base * (
196
+ (self.scaling_factor * seq_len / self.max_position_embeddings) - (self.scaling_factor - 1)
197
+ ) ** (self.dim / (self.dim - 2))
198
+ inv_freq = 1.0 / (base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
199
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
200
+
201
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
202
+
203
+ freqs = torch.outer(t, self.inv_freq)
204
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
205
+ emb = torch.cat((freqs, freqs), dim=-1)
206
+
207
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
208
+ self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
209
+
210
+
211
+ def rotate_half(x):
212
+ """Rotates half the hidden dims of the input."""
213
+ x1 = x[..., : x.shape[-1] // 2]
214
+ x2 = x[..., x.shape[-1] // 2 :]
215
+ return torch.cat((-x2, x1), dim=-1)
216
+
217
+
218
+ def apply_rotary_pos_emb(q, k, cos, sin, position_ids, unsqueeze_dim=1):
219
+ """Applies Rotary Position Embedding to the query and key tensors.
220
+
221
+ Args:
222
+ q (`torch.Tensor`): The query tensor.
223
+ k (`torch.Tensor`): The key tensor.
224
+ cos (`torch.Tensor`): The cosine part of the rotary embedding.
225
+ sin (`torch.Tensor`): The sine part of the rotary embedding.
226
+ position_ids (`torch.Tensor`):
227
+ The position indices of the tokens corresponding to the query and key tensors. For example, this can be
228
+ used to pass offsetted position ids when working with a KV-cache.
229
+ unsqueeze_dim (`int`, *optional*, defaults to 1):
230
+ The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
231
+ sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
232
+ that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
233
+ k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
234
+ cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
235
+ the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
236
+ Returns:
237
+ `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
238
+ """
239
+ # cos = cos[position_ids].unsqueeze(unsqueeze_dim)
240
+ # sin = sin[position_ids].unsqueeze(unsqueeze_dim)
241
+ # q_embed = (q * cos) + (rotate_half(q) * sin)
242
+ # k_embed = (k * cos) + (rotate_half(k) * sin)
243
+ orig_dtype = k.dtype
244
+ cos = cos[position_ids].unsqueeze(unsqueeze_dim) # [bs, 1, seq_len, dim]
245
+ sin = sin[position_ids].unsqueeze(unsqueeze_dim) # [bs, 1, seq_len, dim]
246
+ q_fp32 = q.to(dtype=torch.float32, device=q.device)
247
+ k_fp32 = k.to(dtype=torch.float32, device=k.device)
248
+ q_embed = (q_fp32 * cos) + (rotate_half(q_fp32) * sin)
249
+ k_embed = (k_fp32 * cos) + (rotate_half(k_fp32) * sin)
250
+ return q_embed.to(dtype=orig_dtype), k_embed.to(dtype=orig_dtype)
251
+
252
+ class MiniCPMMLP(nn.Module):
253
+ def __init__(self, config):
254
+ super().__init__()
255
+ self.config = config
256
+ self.hidden_size = config.hidden_size
257
+ self.intermediate_size = config.intermediate_size
258
+ self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
259
+ self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
260
+ self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
261
+ self.act_fn = ACT2FN[config.hidden_act]
262
+
263
+ def forward(self, x):
264
+ if self.config.pretraining_tp > 1:
265
+ slice = self.intermediate_size // self.config.pretraining_tp
266
+ gate_proj_slices = self.gate_proj.weight.split(slice, dim=0)
267
+ up_proj_slices = self.up_proj.weight.split(slice, dim=0)
268
+ down_proj_slices = self.down_proj.weight.split(slice, dim=1)
269
+
270
+ gate_proj = torch.cat(
271
+ [F.linear(x, gate_proj_slices[i]) for i in range(self.config.pretraining_tp)], dim=-1
272
+ )
273
+ up_proj = torch.cat([F.linear(x, up_proj_slices[i]) for i in range(self.config.pretraining_tp)], dim=-1)
274
+
275
+ intermediate_states = (self.act_fn(gate_proj) * up_proj).split(slice, dim=2)
276
+ down_proj = [
277
+ F.linear(intermediate_states[i], down_proj_slices[i]) for i in range(self.config.pretraining_tp)
278
+ ]
279
+ down_proj = sum(down_proj)
280
+ else:
281
+ down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
282
+
283
+ return down_proj
284
+
285
+
286
+ def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
287
+ """
288
+ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
289
+ num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
290
+ """
291
+ batch, num_key_value_heads, slen, head_dim = hidden_states.shape
292
+ if n_rep == 1:
293
+ return hidden_states
294
+ hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
295
+ return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
296
+
297
+
298
+
299
+ class MiniCPMAttention(nn.Module):
300
+ """Multi-headed attention from 'Attention Is All You Need' paper"""
301
+
302
+ def __init__(self, config: MiniCPMConfig, layer_idx: Optional[int] = None):
303
+ super().__init__()
304
+ self.config = config
305
+ self.layer_idx = layer_idx
306
+ if layer_idx is None:
307
+ logger.warning_once(
308
+ f"Instantiating {self.__class__.__name__} without passing `layer_idx` is not recommended and will "
309
+ "to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` "
310
+ "when creating this class."
311
+ )
312
+
313
+ self.attention_dropout = config.attention_dropout
314
+ self.hidden_size = config.hidden_size
315
+ self.num_heads = config.num_attention_heads
316
+ self.head_dim = self.hidden_size // self.num_heads
317
+ self.num_key_value_heads = config.num_key_value_heads
318
+ self.num_key_value_groups = self.num_heads // self.num_key_value_heads
319
+ self.max_position_embeddings = config.max_position_embeddings
320
+ self.rope_theta = config.rope_theta
321
+ self.is_causal = True
322
+
323
+ if (self.head_dim * self.num_heads) != self.hidden_size:
324
+ raise ValueError(
325
+ f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}"
326
+ f" and `num_heads`: {self.num_heads})."
327
+ )
328
+
329
+ self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=config.attention_bias)
330
+ self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=config.attention_bias)
331
+ self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=config.attention_bias)
332
+ self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=config.attention_bias)
333
+ self._init_rope()
334
+
335
+ def _init_rope(self):
336
+ if self.config.rope_scaling is None:
337
+ self.rotary_emb = MiniCPMRotaryEmbedding(
338
+ self.head_dim,
339
+ max_position_embeddings=self.max_position_embeddings,
340
+ base=self.rope_theta,
341
+ )
342
+ else:
343
+ scaling_type = self.config.rope_scaling["type"]
344
+ scaling_factor = self.config.rope_scaling["factor"]
345
+ if scaling_type == "linear":
346
+ self.rotary_emb = MiniCPMLinearScalingRotaryEmbedding(
347
+ self.head_dim,
348
+ max_position_embeddings=self.max_position_embeddings,
349
+ scaling_factor=scaling_factor,
350
+ base=self.rope_theta,
351
+ )
352
+ elif scaling_type == "dynamic":
353
+ self.rotary_emb = MiniCPMDynamicNTKScalingRotaryEmbedding(
354
+ self.head_dim,
355
+ max_position_embeddings=self.max_position_embeddings,
356
+ scaling_factor=scaling_factor,
357
+ base=self.rope_theta,
358
+ )
359
+ else:
360
+ raise ValueError(f"Unknown RoPE scaling type {scaling_type}")
361
+
362
+ def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
363
+ return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()
364
+
365
+ def forward(
366
+ self,
367
+ hidden_states: torch.Tensor,
368
+ attention_mask: Optional[torch.Tensor] = None,
369
+ position_ids: Optional[torch.LongTensor] = None,
370
+ past_key_value: Optional[Cache] = None,
371
+ output_attentions: bool = False,
372
+ use_cache: bool = False,
373
+ **kwargs,
374
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
375
+ if "padding_mask" in kwargs:
376
+ warnings.warn(
377
+ "Passing `padding_mask` is deprecated and will be removed in v4.37. Please make sure use `attention_mask` instead.`"
378
+ )
379
+
380
+ bsz, q_len, _ = hidden_states.size()
381
+
382
+ if self.config.pretraining_tp > 1:
383
+ key_value_slicing = (self.num_key_value_heads * self.head_dim) // self.config.pretraining_tp
384
+ query_slices = self.q_proj.weight.split(
385
+ (self.num_heads * self.head_dim) // self.config.pretraining_tp, dim=0
386
+ )
387
+ key_slices = self.k_proj.weight.split(key_value_slicing, dim=0)
388
+ value_slices = self.v_proj.weight.split(key_value_slicing, dim=0)
389
+
390
+ query_states = [F.linear(hidden_states, query_slices[i]) for i in range(self.config.pretraining_tp)]
391
+ query_states = torch.cat(query_states, dim=-1)
392
+
393
+ key_states = [F.linear(hidden_states, key_slices[i]) for i in range(self.config.pretraining_tp)]
394
+ key_states = torch.cat(key_states, dim=-1)
395
+
396
+ value_states = [F.linear(hidden_states, value_slices[i]) for i in range(self.config.pretraining_tp)]
397
+ value_states = torch.cat(value_states, dim=-1)
398
+
399
+ else:
400
+ query_states = self.q_proj(hidden_states)
401
+ key_states = self.k_proj(hidden_states)
402
+ value_states = self.v_proj(hidden_states)
403
+
404
+ query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
405
+ key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
406
+ value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
407
+
408
+ kv_seq_len = key_states.shape[-2]
409
+ if past_key_value is not None:
410
+ if self.layer_idx is None:
411
+ raise ValueError(
412
+ f"The cache structure has changed since version v4.36. If you are using {self.__class__.__name__} "
413
+ "for auto-regressive decoding with k/v caching, please make sure to initialize the attention class "
414
+ "with a layer index."
415
+ )
416
+ kv_seq_len += past_key_value.get_usable_length(kv_seq_len, self.layer_idx)
417
+ cos, sin = self.rotary_emb(value_states.to(torch.float32), seq_len=kv_seq_len)
418
+
419
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
420
+
421
+ if past_key_value is not None:
422
+ cache_kwargs = {"sin": sin, "cos": cos} # Specific to RoPE models
423
+ key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
424
+
425
+ key_states = repeat_kv(key_states, self.num_key_value_groups)
426
+ value_states = repeat_kv(value_states, self.num_key_value_groups)
427
+
428
+ attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
429
+ if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):
430
+ raise ValueError(
431
+ f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is"
432
+ f" {attn_weights.size()}"
433
+ )
434
+
435
+ if attention_mask is not None:
436
+ if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
437
+ raise ValueError(
438
+ f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
439
+ )
440
+ attn_weights = attn_weights + attention_mask
441
+
442
+ # upcast attention to fp32
443
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
444
+ attn_weights = nn.functional.dropout(attn_weights, p=self.attention_dropout, training=self.training)
445
+ attn_output = torch.matmul(attn_weights, value_states)
446
+
447
+ if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
448
+ raise ValueError(
449
+ f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
450
+ f" {attn_output.size()}"
451
+ )
452
+
453
+ attn_output = attn_output.transpose(1, 2).contiguous()
454
+
455
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
456
+
457
+ if self.config.pretraining_tp > 1:
458
+ attn_output = attn_output.split(self.hidden_size // self.config.pretraining_tp, dim=2)
459
+ o_proj_slices = self.o_proj.weight.split(self.hidden_size // self.config.pretraining_tp, dim=1)
460
+ attn_output = sum([F.linear(attn_output[i], o_proj_slices[i]) for i in range(self.config.pretraining_tp)])
461
+ else:
462
+ attn_output = self.o_proj(attn_output)
463
+
464
+ if not output_attentions:
465
+ attn_weights = None
466
+
467
+ return attn_output, attn_weights, past_key_value
468
+
469
+
470
+ class MiniCPMFlashAttention2(MiniCPMAttention):
471
+ """
472
+ MiniCPM flash attention module. This module inherits from `MiniCPMAttention` as the weights of the module stays
473
+ untouched. The only required change would be on the forward pass where it needs to correctly call the public API of
474
+ flash attention and deal with padding tokens in case the input contains any of them.
475
+ """
476
+
477
+ def __init__(self, *args, **kwargs):
478
+ super().__init__(*args, **kwargs)
479
+
480
+ # TODO: Should be removed once Flash Attention for RoCm is bumped to 2.1.
481
+ # flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. Reference: https://github.com/Dao-AILab/flash-attention/releases/tag/v2.1.0.
482
+ # Beware that with flash_attn<2.1, using q_seqlen != k_seqlen (except for the case q_seqlen == 1) produces a wrong mask (top-left).
483
+ self._flash_attn_uses_top_left_mask = not is_flash_attn_greater_or_equal_2_10()
484
+
485
+ def forward(
486
+ self,
487
+ hidden_states: torch.Tensor,
488
+ attention_mask: Optional[torch.LongTensor] = None,
489
+ position_ids: Optional[torch.LongTensor] = None,
490
+ past_key_value: Optional[Cache] = None,
491
+ output_attentions: bool = False,
492
+ use_cache: bool = False,
493
+ **kwargs,
494
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
495
+ # MiniCPMFlashAttention2 attention does not support output_attentions
496
+ if "padding_mask" in kwargs:
497
+ warnings.warn(
498
+ "Passing `padding_mask` is deprecated and will be removed in v4.37. Please make sure use `attention_mask` instead.`"
499
+ )
500
+
501
+ # overwrite attention_mask with padding_mask
502
+ attention_mask = kwargs.pop("padding_mask")
503
+
504
+ output_attentions = False
505
+
506
+ bsz, q_len, _ = hidden_states.size()
507
+
508
+ query_states = self.q_proj(hidden_states)
509
+ key_states = self.k_proj(hidden_states)
510
+ value_states = self.v_proj(hidden_states)
511
+
512
+ # Flash attention requires the input to have the shape
513
+ # batch_size x seq_length x head_dim x hidden_dim
514
+ # therefore we just need to keep the original shape
515
+ query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
516
+ key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
517
+ value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
518
+
519
+ kv_seq_len = key_states.shape[-2]
520
+ if past_key_value is not None:
521
+ kv_seq_len += past_key_value.get_usable_length(kv_seq_len, self.layer_idx)
522
+ cos, sin = self.rotary_emb(value_states.to(torch.float32), seq_len=kv_seq_len)
523
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
524
+
525
+ if past_key_value is not None:
526
+ cache_kwargs = {"sin": sin, "cos": cos} # Specific to RoPE models
527
+ key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
528
+
529
+ # TODO: These transpose are quite inefficient but Flash Attention requires the layout [batch_size, sequence_length, num_heads, head_dim]. We would need to refactor the KV cache
530
+ # to be able to avoid many of these transpose/reshape/view.
531
+ query_states = query_states.transpose(1, 2)
532
+ key_states = key_states.transpose(1, 2)
533
+ value_states = value_states.transpose(1, 2)
534
+
535
+ dropout_rate = self.attention_dropout if self.training else 0.0
536
+
537
+ # In PEFT, usually we cast the layer norms in float32 for training stability reasons
538
+ # therefore the input hidden states gets silently casted in float32. Hence, we need
539
+ # cast them back in the correct dtype just to be sure everything works as expected.
540
+ # This might slowdown training & inference so it is recommended to not cast the LayerNorms
541
+ # in fp32. (MiniCPMRMSNorm handles it correctly)
542
+
543
+ input_dtype = query_states.dtype
544
+ if input_dtype == torch.float32:
545
+ # Handle the case where the model is quantized
546
+ if hasattr(self.config, "_pre_quantization_dtype"):
547
+ target_dtype = self.config._pre_quantization_dtype
548
+ else:
549
+ target_dtype = self.q_proj.weight.dtype
550
+
551
+ logger.warning_once(
552
+ f"The input hidden states seems to be silently casted in float32, this might be related to"
553
+ f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in"
554
+ f" {target_dtype}."
555
+ )
556
+
557
+ query_states = query_states.to(target_dtype)
558
+ key_states = key_states.to(target_dtype)
559
+ value_states = value_states.to(target_dtype)
560
+
561
+ attn_output = self._flash_attention_forward(
562
+ query_states, key_states, value_states, attention_mask, q_len, dropout=dropout_rate
563
+ )
564
+
565
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
566
+ attn_output = self.o_proj(attn_output)
567
+
568
+ if not output_attentions:
569
+ attn_weights = None
570
+
571
+ return attn_output, attn_weights, past_key_value
572
+
573
+ def _flash_attention_forward(
574
+ self, query_states, key_states, value_states, attention_mask, query_length, dropout=0.0, softmax_scale=None
575
+ ):
576
+ """
577
+ Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token
578
+ first unpad the input, then computes the attention scores and pad the final attention scores.
579
+
580
+ Args:
581
+ query_states (`torch.Tensor`):
582
+ Input query states to be passed to Flash Attention API
583
+ key_states (`torch.Tensor`):
584
+ Input key states to be passed to Flash Attention API
585
+ value_states (`torch.Tensor`):
586
+ Input value states to be passed to Flash Attention API
587
+ attention_mask (`torch.Tensor`):
588
+ The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the
589
+ position of padding tokens and 1 for the position of non-padding tokens.
590
+ dropout (`int`, *optional*):
591
+ Attention dropout
592
+ softmax_scale (`float`, *optional*):
593
+ The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim)
594
+ """
595
+ if not self._flash_attn_uses_top_left_mask:
596
+ causal = self.is_causal
597
+ else:
598
+ # TODO: Remove the `query_length != 1` check once Flash Attention for RoCm is bumped to 2.1. For details, please see the comment in MiniCPMFlashAttention2 __init__.
599
+ causal = self.is_causal and query_length != 1
600
+ # Contains at least one padding token in the sequence
601
+ if attention_mask is not None:
602
+ batch_size = query_states.shape[0]
603
+ query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = self._upad_input(
604
+ query_states, key_states, value_states, attention_mask, query_length
605
+ )
606
+
607
+ cu_seqlens_q, cu_seqlens_k = cu_seq_lens
608
+ max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens
609
+ attn_output_unpad = flash_attn_varlen_func(
610
+ query_states,
611
+ key_states,
612
+ value_states,
613
+ cu_seqlens_q=cu_seqlens_q,
614
+ cu_seqlens_k=cu_seqlens_k,
615
+ max_seqlen_q=max_seqlen_in_batch_q,
616
+ max_seqlen_k=max_seqlen_in_batch_k,
617
+ dropout_p=dropout,
618
+ softmax_scale=softmax_scale,
619
+ causal=causal,
620
+ )
621
+
622
+ attn_output = pad_input(attn_output_unpad, indices_q, batch_size, query_length)
623
+ else:
624
+ attn_output = flash_attn_func(
625
+ query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=causal
626
+ )
627
+
628
+ return attn_output
629
+
630
+ def _upad_input(self, query_layer, key_layer, value_layer, attention_mask, query_length):
631
+ indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(attention_mask)
632
+ batch_size, kv_seq_len, num_key_value_heads, head_dim = key_layer.shape
633
+
634
+ key_layer = index_first_axis(
635
+ key_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
636
+ )
637
+ value_layer = index_first_axis(
638
+ value_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
639
+ )
640
+ if query_length == kv_seq_len:
641
+ query_layer = index_first_axis(
642
+ query_layer.reshape(batch_size * kv_seq_len, self.num_heads, head_dim), indices_k
643
+ )
644
+ cu_seqlens_q = cu_seqlens_k
645
+ max_seqlen_in_batch_q = max_seqlen_in_batch_k
646
+ indices_q = indices_k
647
+ elif query_length == 1:
648
+ max_seqlen_in_batch_q = 1
649
+ cu_seqlens_q = torch.arange(
650
+ batch_size + 1, dtype=torch.int32, device=query_layer.device
651
+ ) # There is a memcpy here, that is very bad.
652
+ indices_q = cu_seqlens_q[:-1]
653
+ query_layer = query_layer.squeeze(1)
654
+ else:
655
+ # The -q_len: slice assumes left padding.
656
+ attention_mask = attention_mask[:, -query_length:]
657
+ query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(query_layer, attention_mask)
658
+
659
+ return (
660
+ query_layer,
661
+ key_layer,
662
+ value_layer,
663
+ indices_q,
664
+ (cu_seqlens_q, cu_seqlens_k),
665
+ (max_seqlen_in_batch_q, max_seqlen_in_batch_k),
666
+ )
667
+
668
+
669
+ class MiniCPMSdpaAttention(MiniCPMAttention):
670
+ """
671
+ MiniCPM attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from
672
+ `MiniCPMAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to
673
+ SDPA API.
674
+ """
675
+
676
+ # Adapted from MiniCPMAttention.forward
677
+ def forward(
678
+ self,
679
+ hidden_states: torch.Tensor,
680
+ attention_mask: Optional[torch.Tensor] = None,
681
+ position_ids: Optional[torch.LongTensor] = None,
682
+ past_key_value: Optional[Cache] = None,
683
+ output_attentions: bool = False,
684
+ use_cache: bool = False,
685
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
686
+ if output_attentions:
687
+ # TODO: Improve this warning with e.g. `model.config.attn_implementation = "manual"` once this is implemented.
688
+ logger.warning_once(
689
+ "MiniCPMModel is using MiniCPMSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, "
690
+ 'but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.'
691
+ )
692
+ return super().forward(
693
+ hidden_states=hidden_states,
694
+ attention_mask=attention_mask,
695
+ position_ids=position_ids,
696
+ past_key_value=past_key_value,
697
+ output_attentions=output_attentions,
698
+ use_cache=use_cache,
699
+ )
700
+
701
+ bsz, q_len, _ = hidden_states.size()
702
+
703
+ query_states = self.q_proj(hidden_states)
704
+ key_states = self.k_proj(hidden_states)
705
+ value_states = self.v_proj(hidden_states)
706
+
707
+ query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
708
+ key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
709
+ value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
710
+
711
+ kv_seq_len = key_states.shape[-2]
712
+ if past_key_value is not None:
713
+ kv_seq_len += past_key_value.get_usable_length(kv_seq_len, self.layer_idx)
714
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
715
+
716
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
717
+
718
+ if past_key_value is not None:
719
+ cache_kwargs = {"sin": sin, "cos": cos} # Specific to RoPE models
720
+ key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
721
+
722
+ key_states = repeat_kv(key_states, self.num_key_value_groups)
723
+ value_states = repeat_kv(value_states, self.num_key_value_groups)
724
+
725
+ if attention_mask is not None:
726
+ if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
727
+ raise ValueError(
728
+ f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
729
+ )
730
+
731
+ # SDPA with memory-efficient backend is currently (torch==2.1.2) bugged with non-contiguous inputs with custom attn_mask,
732
+ # Reference: https://github.com/pytorch/pytorch/issues/112577.
733
+ if query_states.device.type == "cuda" and attention_mask is not None:
734
+ query_states = query_states.contiguous()
735
+ key_states = key_states.contiguous()
736
+ value_states = value_states.contiguous()
737
+
738
+ attn_output = torch.nn.functional.scaled_dot_product_attention(
739
+ query_states,
740
+ key_states,
741
+ value_states,
742
+ attn_mask=attention_mask,
743
+ dropout_p=self.attention_dropout if self.training else 0.0,
744
+ # The q_len > 1 is necessary to match with AttentionMaskConverter.to_causal_4d that does not create a causal mask in case q_len == 1.
745
+ is_causal=self.is_causal and attention_mask is None and q_len > 1,
746
+ )
747
+
748
+ attn_output = attn_output.transpose(1, 2).contiguous()
749
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
750
+
751
+ attn_output = self.o_proj(attn_output)
752
+
753
+ return attn_output, None, past_key_value
754
+
755
+
756
+ MINICPM_ATTENTION_CLASSES = {
757
+ "eager": MiniCPMAttention,
758
+ "flash_attention_2": MiniCPMFlashAttention2,
759
+ "sdpa": MiniCPMSdpaAttention,
760
+ }
761
+
762
+
763
+ class MiniCPMDecoderLayer(nn.Module):
764
+ def __init__(self, config: MiniCPMConfig, layer_idx: int):
765
+ super().__init__()
766
+ self.hidden_size = config.hidden_size
767
+ self.self_attn = MINICPM_ATTENTION_CLASSES[config._attn_implementation](config=config, layer_idx=layer_idx)
768
+
769
+ self.mlp = MiniCPMMLP(config)
770
+ self.input_layernorm = MiniCPMRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
771
+ self.post_attention_layernorm = MiniCPMRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
772
+
773
+ self.scale_depth = config.scale_depth
774
+ self.num_hidden_layers = config.num_hidden_layers
775
+
776
+ def forward(
777
+ self,
778
+ hidden_states: torch.Tensor,
779
+ attention_mask: Optional[torch.Tensor] = None,
780
+ position_ids: Optional[torch.LongTensor] = None,
781
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
782
+ output_attentions: Optional[bool] = False,
783
+ use_cache: Optional[bool] = False,
784
+ **kwargs,
785
+ ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
786
+ """
787
+ Args:
788
+ hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
789
+ attention_mask (`torch.FloatTensor`, *optional*):
790
+ attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1,
791
+ query_sequence_length, key_sequence_length)` if default attention is used.
792
+ output_attentions (`bool`, *optional*):
793
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under
794
+ returned tensors for more detail.
795
+ use_cache (`bool`, *optional*):
796
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
797
+ (see `past_key_values`).
798
+ past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states
799
+ """
800
+ if "padding_mask" in kwargs:
801
+ warnings.warn(
802
+ "Passing `padding_mask` is deprecated and will be removed in v4.37. Please make sure use `attention_mask` instead.`"
803
+ )
804
+
805
+ residual = hidden_states
806
+ hidden_states = self.input_layernorm(hidden_states)
807
+ # Self Attention
808
+ hidden_states, self_attn_weights, present_key_value = self.self_attn(
809
+ hidden_states=hidden_states,
810
+ attention_mask=attention_mask,
811
+ position_ids=position_ids,
812
+ past_key_value=past_key_value,
813
+ output_attentions=output_attentions,
814
+ use_cache=use_cache,
815
+ **kwargs,
816
+ )
817
+
818
+ hidden_states = residual + hidden_states * (self.scale_depth / math.sqrt(self.num_hidden_layers))
819
+
820
+ # Fully Connected
821
+ residual = hidden_states
822
+ hidden_states = self.post_attention_layernorm(hidden_states)
823
+
824
+ hidden_states = self.mlp(hidden_states)
825
+ hidden_states = residual + hidden_states * (self.scale_depth / math.sqrt(self.num_hidden_layers))
826
+
827
+ outputs = (hidden_states,)
828
+
829
+ if output_attentions:
830
+ outputs += (self_attn_weights,)
831
+
832
+ if use_cache:
833
+ outputs += (present_key_value,)
834
+
835
+ return outputs
836
+
837
+
838
+ MINICPM_START_DOCSTRING = r"""
839
+ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
840
+ library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
841
+ etc.)
842
+
843
+ This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
844
+ Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
845
+ and behavior.
846
+
847
+ Parameters:
848
+ config ([`MiniCPMConfig`]):
849
+ Model configuration class with all the parameters of the model. Initializing with a config file does not
850
+ load the weights associated with the model, only the configuration. Check out the
851
+ [`~PreTrainedModel.from_pretrained`] method to load the model weights.
852
+ """
853
+
854
+
855
+ @add_start_docstrings(
856
+ "The bare MiniCPM Model outputting raw hidden-states without any specific head on top.",
857
+ MINICPM_START_DOCSTRING,
858
+ )
859
+ class MiniCPMPreTrainedModel(PreTrainedModel):
860
+ config_class = MiniCPMConfig
861
+ base_model_prefix = "model"
862
+ supports_gradient_checkpointing = True
863
+ _no_split_modules = ["MiniCPMDecoderLayer"]
864
+ _skip_keys_device_placement = "past_key_values"
865
+ _supports_flash_attn_2 = True
866
+ _supports_sdpa = True
867
+ _supports_cache_class = True
868
+
869
+ def _init_weights(self, module):
870
+ std = self.config.initializer_range
871
+ if isinstance(module, nn.Linear):
872
+ module.weight.data.normal_(mean=0.0, std=std)
873
+ if module.bias is not None:
874
+ module.bias.data.zero_()
875
+ elif isinstance(module, nn.Embedding):
876
+ module.weight.data.normal_(mean=0.0, std=std)
877
+ if module.padding_idx is not None:
878
+ module.weight.data[module.padding_idx].zero_()
879
+
880
+
881
+ MINICPM_INPUTS_DOCSTRING = r"""
882
+ Args:
883
+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
884
+ Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
885
+ it.
886
+
887
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
888
+ [`PreTrainedTokenizer.__call__`] for details.
889
+
890
+ [What are input IDs?](../glossary#input-ids)
891
+ attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
892
+ Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
893
+
894
+ - 1 for tokens that are **not masked**,
895
+ - 0 for tokens that are **masked**.
896
+
897
+ [What are attention masks?](../glossary#attention-mask)
898
+
899
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
900
+ [`PreTrainedTokenizer.__call__`] for details.
901
+
902
+ If `past_key_values` is used, optionally only the last `input_ids` have to be input (see
903
+ `past_key_values`).
904
+
905
+ If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
906
+ and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
907
+ information on the default strategy.
908
+
909
+ - 1 indicates the head is **not masked**,
910
+ - 0 indicates the head is **masked**.
911
+ position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
912
+ Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
913
+ config.n_positions - 1]`.
914
+
915
+ [What are position IDs?](../glossary#position-ids)
916
+ past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*):
917
+ Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
918
+ blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values`
919
+ returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.
920
+
921
+ Two formats are allowed:
922
+ - a [`~cache_utils.Cache`] instance;
923
+ - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of
924
+ shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy
925
+ cache format.
926
+
927
+ The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the
928
+ legacy cache format will be returned.
929
+
930
+ If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't
931
+ have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids`
932
+ of shape `(batch_size, sequence_length)`.
933
+ inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
934
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
935
+ is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
936
+ model's internal embedding lookup matrix.
937
+ use_cache (`bool`, *optional*):
938
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
939
+ `past_key_values`).
940
+ output_attentions (`bool`, *optional*):
941
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
942
+ tensors for more detail.
943
+ output_hidden_states (`bool`, *optional*):
944
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
945
+ more detail.
946
+ return_dict (`bool`, *optional*):
947
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
948
+ """
949
+
950
+
951
+ @add_start_docstrings(
952
+ "The bare MiniCPM Model outputting raw hidden-states without any specific head on top.",
953
+ MINICPM_START_DOCSTRING,
954
+ )
955
+ class MiniCPMModel(MiniCPMPreTrainedModel):
956
+ """
957
+ Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MiniCPMDecoderLayer`]
958
+
959
+ Args:
960
+ config: MiniCPMConfig
961
+ """
962
+
963
+ def __init__(self, config: MiniCPMConfig):
964
+ super().__init__(config)
965
+ self.padding_idx = config.pad_token_id
966
+ self.vocab_size = config.vocab_size
967
+
968
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
969
+ self.layers = nn.ModuleList(
970
+ [MiniCPMDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
971
+ )
972
+ self._use_sdpa = config._attn_implementation == "sdpa"
973
+ self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2"
974
+
975
+ self.norm = MiniCPMRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
976
+
977
+ self.gradient_checkpointing = False
978
+ # Initialize weights and apply final processing
979
+ self.post_init()
980
+
981
+ def get_input_embeddings(self):
982
+ return self.embed_tokens
983
+
984
+ def set_input_embeddings(self, value):
985
+ self.embed_tokens = value
986
+
987
+ @add_start_docstrings_to_model_forward(MINICPM_INPUTS_DOCSTRING)
988
+ def forward(
989
+ self,
990
+ input_ids: torch.LongTensor = None,
991
+ attention_mask: Optional[torch.Tensor] = None,
992
+ position_ids: Optional[torch.LongTensor] = None,
993
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
994
+ inputs_embeds: Optional[torch.FloatTensor] = None,
995
+ use_cache: Optional[bool] = None,
996
+ output_attentions: Optional[bool] = None,
997
+ output_hidden_states: Optional[bool] = None,
998
+ return_dict: Optional[bool] = None,
999
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
1000
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
1001
+ output_hidden_states = (
1002
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
1003
+ )
1004
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
1005
+
1006
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1007
+
1008
+ # retrieve input_ids and inputs_embeds
1009
+ if input_ids is not None and inputs_embeds is not None:
1010
+ raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
1011
+ elif input_ids is not None:
1012
+ batch_size, seq_length = input_ids.shape[:2]
1013
+ elif inputs_embeds is not None:
1014
+ batch_size, seq_length = inputs_embeds.shape[:2]
1015
+ else:
1016
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
1017
+
1018
+ if self.gradient_checkpointing and self.training:
1019
+ if use_cache:
1020
+ logger.warning_once(
1021
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
1022
+ )
1023
+ use_cache = False
1024
+
1025
+ past_key_values_length = 0
1026
+ if use_cache:
1027
+ use_legacy_cache = not isinstance(past_key_values, Cache)
1028
+ if use_legacy_cache:
1029
+ past_key_values = DynamicCache.from_legacy_cache(past_key_values)
1030
+ past_key_values_length = past_key_values.get_usable_length(seq_length)
1031
+
1032
+ if position_ids is None:
1033
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
1034
+ position_ids = torch.arange(
1035
+ past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device
1036
+ )
1037
+ position_ids = position_ids.unsqueeze(0)
1038
+
1039
+ if inputs_embeds is None:
1040
+ inputs_embeds = self.embed_tokens(input_ids) * self.config.scale_emb
1041
+
1042
+ if self._use_flash_attention_2:
1043
+ # 2d mask is passed through the layers
1044
+ attention_mask = attention_mask if (attention_mask is not None and 0 in attention_mask) else None
1045
+ elif self._use_sdpa and not output_attentions:
1046
+ # output_attentions=True can not be supported when using SDPA, and we fall back on
1047
+ # the manual implementation that requires a 4D causal mask in all cases.
1048
+ attention_mask = _prepare_4d_causal_attention_mask_for_sdpa(
1049
+ attention_mask,
1050
+ (batch_size, seq_length),
1051
+ inputs_embeds,
1052
+ past_key_values_length,
1053
+ )
1054
+ else:
1055
+ # 4d mask is passed through the layers
1056
+ attention_mask = _prepare_4d_causal_attention_mask(
1057
+ attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length
1058
+ )
1059
+
1060
+ # embed positions
1061
+ hidden_states = inputs_embeds
1062
+
1063
+ # decoder layers
1064
+ all_hidden_states = () if output_hidden_states else None
1065
+ all_self_attns = () if output_attentions else None
1066
+ next_decoder_cache = None
1067
+
1068
+ for decoder_layer in self.layers:
1069
+ if output_hidden_states:
1070
+ all_hidden_states += (hidden_states,)
1071
+
1072
+ if self.gradient_checkpointing and self.training:
1073
+ layer_outputs = self._gradient_checkpointing_func(
1074
+ decoder_layer.__call__,
1075
+ hidden_states,
1076
+ attention_mask,
1077
+ position_ids,
1078
+ past_key_values,
1079
+ output_attentions,
1080
+ use_cache,
1081
+ )
1082
+ else:
1083
+ layer_outputs = decoder_layer(
1084
+ hidden_states,
1085
+ attention_mask=attention_mask,
1086
+ position_ids=position_ids,
1087
+ past_key_value=past_key_values,
1088
+ output_attentions=output_attentions,
1089
+ use_cache=use_cache,
1090
+ )
1091
+
1092
+ hidden_states = layer_outputs[0]
1093
+
1094
+ if use_cache:
1095
+ next_decoder_cache = layer_outputs[2 if output_attentions else 1]
1096
+
1097
+ if output_attentions:
1098
+ all_self_attns += (layer_outputs[1],)
1099
+
1100
+ hidden_states = self.norm(hidden_states)
1101
+
1102
+ # add hidden states from the last decoder layer
1103
+ if output_hidden_states:
1104
+ all_hidden_states += (hidden_states,)
1105
+
1106
+ next_cache = None
1107
+ if use_cache:
1108
+ next_cache = next_decoder_cache.to_legacy_cache() if use_legacy_cache else next_decoder_cache
1109
+ if not return_dict:
1110
+ return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
1111
+ return BaseModelOutputWithPast(
1112
+ last_hidden_state=hidden_states,
1113
+ past_key_values=next_cache,
1114
+ hidden_states=all_hidden_states,
1115
+ attentions=all_self_attns,
1116
+ )
1117
+
1118
+
1119
+ class MiniCPMForCausalLM(MiniCPMPreTrainedModel):
1120
+ _tied_weights_keys = ["lm_head.weight"]
1121
+
1122
+ def __init__(self, config):
1123
+ super().__init__(config)
1124
+ self.model = MiniCPMModel(config)
1125
+ self.vocab_size = config.vocab_size
1126
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
1127
+
1128
+ # Initialize weights and apply final processing
1129
+ self.post_init()
1130
+
1131
+ def get_input_embeddings(self):
1132
+ return self.model.embed_tokens
1133
+
1134
+ def set_input_embeddings(self, value):
1135
+ self.model.embed_tokens = value
1136
+
1137
+ def get_output_embeddings(self):
1138
+ return self.lm_head
1139
+
1140
+ def set_output_embeddings(self, new_embeddings):
1141
+ self.lm_head = new_embeddings
1142
+
1143
+ def set_decoder(self, decoder):
1144
+ self.model = decoder
1145
+
1146
+ def get_decoder(self):
1147
+ return self.model
1148
+
1149
+ @add_start_docstrings_to_model_forward(MINICPM_INPUTS_DOCSTRING)
1150
+ @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
1151
+ def forward(
1152
+ self,
1153
+ input_ids: torch.LongTensor = None,
1154
+ attention_mask: Optional[torch.Tensor] = None,
1155
+ position_ids: Optional[torch.LongTensor] = None,
1156
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
1157
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1158
+ labels: Optional[torch.LongTensor] = None,
1159
+ use_cache: Optional[bool] = None,
1160
+ output_attentions: Optional[bool] = None,
1161
+ output_hidden_states: Optional[bool] = None,
1162
+ return_dict: Optional[bool] = None,
1163
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
1164
+ r"""
1165
+ Args:
1166
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
1167
+ Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
1168
+ config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
1169
+ (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
1170
+
1171
+ Returns:
1172
+
1173
+ Example:
1174
+
1175
+ ```python
1176
+ >>> from transformers import AutoTokenizer, MiniCPMForCausalLM
1177
+
1178
+ >>> model = MiniCPMForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)
1179
+ >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)
1180
+
1181
+ >>> prompt = "Hey, are you conscious? Can you talk to me?"
1182
+ >>> inputs = tokenizer(prompt, return_tensors="pt")
1183
+
1184
+ >>> # Generate
1185
+ >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
1186
+ >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
1187
+ "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
1188
+ ```"""
1189
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
1190
+ output_hidden_states = (
1191
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
1192
+ )
1193
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1194
+
1195
+ # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
1196
+ outputs = self.model(
1197
+ input_ids=input_ids,
1198
+ attention_mask=attention_mask,
1199
+ position_ids=position_ids,
1200
+ past_key_values=past_key_values,
1201
+ inputs_embeds=inputs_embeds,
1202
+ use_cache=use_cache,
1203
+ output_attentions=output_attentions,
1204
+ output_hidden_states=output_hidden_states,
1205
+ return_dict=return_dict,
1206
+ )
1207
+
1208
+ hidden_states = outputs[0]
1209
+ if self.config.pretraining_tp > 1:
1210
+ lm_head_slices = self.lm_head.weight.split(self.vocab_size // self.config.pretraining_tp, dim=0)
1211
+ logits = [F.linear(hidden_states, lm_head_slices[i]) for i in range(self.config.pretraining_tp)]
1212
+ logits = torch.cat(logits, dim=-1)
1213
+ else:
1214
+ logits = self.lm_head(hidden_states / (self.config.hidden_size / self.config.dim_model_base))
1215
+ logits = logits.float()
1216
+
1217
+ loss = None
1218
+ if labels is not None:
1219
+ # Shift so that tokens < n predict n
1220
+ shift_logits = logits[..., :-1, :].contiguous()
1221
+ shift_labels = labels[..., 1:].contiguous()
1222
+ # Flatten the tokens
1223
+ loss_fct = CrossEntropyLoss()
1224
+ shift_logits = shift_logits.view(-1, self.config.vocab_size)
1225
+ shift_labels = shift_labels.view(-1)
1226
+ # Enable model parallelism
1227
+ shift_labels = shift_labels.to(shift_logits.device)
1228
+ loss = loss_fct(shift_logits, shift_labels)
1229
+
1230
+ if not return_dict:
1231
+ output = (logits,) + outputs[1:]
1232
+ return (loss,) + output if loss is not None else output
1233
+
1234
+ return CausalLMOutputWithPast(
1235
+ loss=loss,
1236
+ logits=logits,
1237
+ past_key_values=outputs.past_key_values,
1238
+ hidden_states=outputs.hidden_states,
1239
+ attentions=outputs.attentions,
1240
+ )
1241
+
1242
+ def prepare_inputs_for_generation(
1243
+ self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs
1244
+ ):
1245
+ if past_key_values is not None:
1246
+ if isinstance(past_key_values, Cache):
1247
+ cache_length = past_key_values.get_seq_length()
1248
+ past_length = past_key_values.seen_tokens
1249
+ max_cache_length = past_key_values.get_max_length()
1250
+ else:
1251
+ cache_length = past_length = past_key_values[0][0].shape[2]
1252
+ max_cache_length = None
1253
+
1254
+ # Keep only the unprocessed tokens:
1255
+ # 1 - If the length of the attention_mask exceeds the length of input_ids, then we are in a setting where
1256
+ # some of the inputs are exclusivelly passed as part of the cache (e.g. when passing input_embeds as
1257
+ # input)
1258
+ if attention_mask is not None and attention_mask.shape[1] > input_ids.shape[1]:
1259
+ input_ids = input_ids[:, -(attention_mask.shape[1] - past_length) :]
1260
+ # 2 - If the past_length is smaller than input_ids', then input_ids holds all input tokens. We can discard
1261
+ # input_ids based on the past_length.
1262
+ elif past_length < input_ids.shape[1]:
1263
+ input_ids = input_ids[:, past_length:]
1264
+ # 3 - Otherwise (past_length >= input_ids.shape[1]), let's assume input_ids only has unprocessed tokens.
1265
+
1266
+ # If we are about to go beyond the maximum cache length, we need to crop the input attention mask.
1267
+ if (
1268
+ max_cache_length is not None
1269
+ and attention_mask is not None
1270
+ and cache_length + input_ids.shape[1] > max_cache_length
1271
+ ):
1272
+ attention_mask = attention_mask[:, -max_cache_length:]
1273
+
1274
+ position_ids = kwargs.get("position_ids", None)
1275
+ if attention_mask is not None and position_ids is None:
1276
+ # create position_ids on the fly for batch generation
1277
+ position_ids = attention_mask.long().cumsum(-1) - 1
1278
+ position_ids.masked_fill_(attention_mask == 0, 1)
1279
+ if past_key_values:
1280
+ position_ids = position_ids[:, -input_ids.shape[1] :]
1281
+
1282
+ # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
1283
+ if inputs_embeds is not None and past_key_values is None:
1284
+ model_inputs = {"inputs_embeds": inputs_embeds}
1285
+ else:
1286
+ model_inputs = {"input_ids": input_ids}
1287
+
1288
+ model_inputs.update(
1289
+ {
1290
+ "position_ids": position_ids,
1291
+ "past_key_values": past_key_values,
1292
+ "use_cache": kwargs.get("use_cache"),
1293
+ "attention_mask": attention_mask,
1294
+ }
1295
+ )
1296
+ return model_inputs
1297
+
1298
+ @staticmethod
1299
+ def _reorder_cache(past_key_values, beam_idx):
1300
+ reordered_past = ()
1301
+ for layer_past in past_key_values:
1302
+ reordered_past += (
1303
+ tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past),
1304
+ )
1305
+ return reordered_past
1306
+
1307
+ @torch.inference_mode()
1308
+ def chat(self, tokenizer, query: str, history: List[Dict] = None, role: str = "user",
1309
+ max_length: int = 4096, num_beams=1, do_sample=True, top_p=0.8, temperature=0.3, logits_processor=None,
1310
+ **kwargs):
1311
+ if history is None:
1312
+ history = []
1313
+ if logits_processor:
1314
+ gen_kwargs = {"max_length": max_length, "num_beams": num_beams, "do_sample": do_sample, "top_p": top_p,
1315
+ "temperature": temperature, "logits_processor": logits_processor, **kwargs}
1316
+ else:
1317
+ gen_kwargs = {"max_length": max_length, "num_beams": num_beams, "do_sample": do_sample, "top_p": top_p,
1318
+ "temperature": temperature, "logits_processor": logits_processor, **kwargs}
1319
+
1320
+ history.append({"role": role, "content": query})
1321
+ history_str = tokenizer.apply_chat_template(history, tokenize=False, add_generation_prompt=False)
1322
+ inputs = tokenizer(history_str, return_tensors='pt').to(self.device)
1323
+ outputs = self.generate(**inputs, **gen_kwargs)
1324
+ outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):-1]
1325
+ response = tokenizer.decode(outputs)
1326
+ pattern = re.compile(r".*?(?=<AI>|<用户>)", re.DOTALL)
1327
+ matches = pattern.findall(response)
1328
+ if len(matches) > 0:
1329
+ response = matches[0]
1330
+ history.append({"role": "assistant", "content": response})
1331
+ return response, history
1332
+
1333
+
1334
+ @add_start_docstrings(
1335
+ """
1336
+ The MiniCPM Model transformer with a sequence classification head on top (linear layer).
1337
+
1338
+ [`MiniCPMForSequenceClassification`] uses the last token in order to do the classification, as other causal models
1339
+ (e.g. GPT-2) do.
1340
+
1341
+ Since it does classification on the last token, it requires to know the position of the last token. If a
1342
+ `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If
1343
+ no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the
1344
+ padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in
1345
+ each row of the batch).
1346
+ """,
1347
+ MINICPM_START_DOCSTRING,
1348
+ )
1349
+ class MiniCPMForSequenceClassification(MiniCPMPreTrainedModel):
1350
+ def __init__(self, config):
1351
+ super().__init__(config)
1352
+ self.num_labels = config.num_labels
1353
+ self.model = MiniCPMModel(config)
1354
+ self.score = nn.Linear(config.hidden_size, self.num_labels, bias=False)
1355
+
1356
+ # Initialize weights and apply final processing
1357
+ self.post_init()
1358
+
1359
+ def get_input_embeddings(self):
1360
+ return self.model.embed_tokens
1361
+
1362
+ def set_input_embeddings(self, value):
1363
+ self.model.embed_tokens = value
1364
+
1365
+ @add_start_docstrings_to_model_forward(MINICPM_INPUTS_DOCSTRING)
1366
+ def forward(
1367
+ self,
1368
+ input_ids: torch.LongTensor = None,
1369
+ attention_mask: Optional[torch.Tensor] = None,
1370
+ position_ids: Optional[torch.LongTensor] = None,
1371
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
1372
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1373
+ labels: Optional[torch.LongTensor] = None,
1374
+ use_cache: Optional[bool] = None,
1375
+ output_attentions: Optional[bool] = None,
1376
+ output_hidden_states: Optional[bool] = None,
1377
+ return_dict: Optional[bool] = None,
1378
+ ) -> Union[Tuple, SequenceClassifierOutputWithPast]:
1379
+ r"""
1380
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1381
+ Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
1382
+ config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
1383
+ `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
1384
+ """
1385
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1386
+
1387
+ transformer_outputs = self.model(
1388
+ input_ids,
1389
+ attention_mask=attention_mask,
1390
+ position_ids=position_ids,
1391
+ past_key_values=past_key_values,
1392
+ inputs_embeds=inputs_embeds,
1393
+ use_cache=use_cache,
1394
+ output_attentions=output_attentions,
1395
+ output_hidden_states=output_hidden_states,
1396
+ return_dict=return_dict,
1397
+ )
1398
+ hidden_states = transformer_outputs[0]
1399
+ logits = self.score(hidden_states)
1400
+
1401
+ if input_ids is not None:
1402
+ batch_size = input_ids.shape[0]
1403
+ else:
1404
+ batch_size = inputs_embeds.shape[0]
1405
+
1406
+ if self.config.pad_token_id is None and batch_size != 1:
1407
+ raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
1408
+ if self.config.pad_token_id is None:
1409
+ sequence_lengths = -1
1410
+ else:
1411
+ if input_ids is not None:
1412
+ sequence_lengths = (torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1).to(
1413
+ logits.device
1414
+ )
1415
+ else:
1416
+ sequence_lengths = -1
1417
+
1418
+ pooled_logits = logits[torch.arange(batch_size, device=logits.device), sequence_lengths]
1419
+
1420
+ loss = None
1421
+ if labels is not None:
1422
+ labels = labels.to(logits.device)
1423
+ if self.config.problem_type is None:
1424
+ if self.num_labels == 1:
1425
+ self.config.problem_type = "regression"
1426
+ elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
1427
+ self.config.problem_type = "single_label_classification"
1428
+ else:
1429
+ self.config.problem_type = "multi_label_classification"
1430
+
1431
+ if self.config.problem_type == "regression":
1432
+ loss_fct = MSELoss()
1433
+ if self.num_labels == 1:
1434
+ loss = loss_fct(pooled_logits.squeeze(), labels.squeeze())
1435
+ else:
1436
+ loss = loss_fct(pooled_logits, labels)
1437
+ elif self.config.problem_type == "single_label_classification":
1438
+ loss_fct = CrossEntropyLoss()
1439
+ loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1))
1440
+ elif self.config.problem_type == "multi_label_classification":
1441
+ loss_fct = BCEWithLogitsLoss()
1442
+ loss = loss_fct(pooled_logits, labels)
1443
+ if not return_dict:
1444
+ output = (pooled_logits,) + transformer_outputs[1:]
1445
+ return ((loss,) + output) if loss is not None else output
1446
+
1447
+ return SequenceClassifierOutputWithPast(
1448
+ loss=loss,
1449
+ logits=pooled_logits,
1450
+ past_key_values=transformer_outputs.past_key_values,
1451
+ hidden_states=transformer_outputs.hidden_states,
1452
+ attentions=transformer_outputs.attentions,
1453
+ )
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a8d7818cfcde8fdfd1a0703616551b78bdf57883066dbee78185a2b0abb759a4
3
+ size 2720644645
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "unk_token": {
17
+ "content": "<unk>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bb74d51116831c3bf65db812c553f94ab0c88dcf97a5bbb37e3504f6d359c530
3
+ size 1181204
tokenizer_config.json ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "added_tokens_decoder": {
5
+ "0": {
6
+ "content": "<unk>",
7
+ "lstrip": false,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": true
12
+ },
13
+ "1": {
14
+ "content": "<s>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": true
20
+ },
21
+ "2": {
22
+ "content": "</s>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": true
28
+ }
29
+ },
30
+ "bos_token": "<s>",
31
+ "clean_up_tokenization_spaces": false,
32
+ "eos_token": "</s>",
33
+ "legacy": true,
34
+ "model_max_length": 1000000000000000019884624838656,
35
+ "pad_token": null,
36
+ "sp_model_kwargs": {},
37
+ "spaces_between_special_tokens": false,
38
+ "tokenizer_class": "LlamaTokenizer",
39
+ "unk_token": "<unk>",
40
+ "use_default_system_prompt": false
41
+ }