Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+gptq_model-8bit-128g.safetensors.aa filter=lfs diff=lfs merge=lfs -text
+gptq_model-8bit-128g.safetensors.ab filter=lfs diff=lfs merge=lfs -text
+gptq_model-8bit-128g.safetensors.ac filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,149 @@
 ---
 license: apache-2.0
+
+inference: false
+
 ---
+
+# XVERSE-65B-Chat-GPTQ-Int4
+
+## 更新信息
+
+**[2024/03/25]** 发布XVERSE-65B-Chat-GPTQ-Int4量化模型，支持vLLM推理xverse-65b量化模型。
+
+**[2023/12/08]** 发布 **XVERSE-65B-2** 底座模型，该模型在前一版本的基础上进行了 **Continual Pre-Training**，训练总 token 量达到 **3.2** 万亿；模型各方面的能力均得到提升，尤其是数学和代码能力，在 GSM8K 上提升 **20**%，HumanEval 上提升 **41**%。 
+**[2023/11/29]** 更新模型架构及更多底座数据的相关信息。  
+
+**[2023/11/24]** 更新预训练数据的相关信息。  
+
+**[2023/11/06]** 发布 65B 尺寸的 XVERSE-65B 底座模型。  
+
+## Update Information
+
+**[2024/03/25] ** Release the XVERSE-65B-Chat-GPTQ-Int4 quantification model, supporting vLLM inference for the xverse-65b quantification model.
+
+**[2023/12/08]** Released the **XVERSE-65B-2** base model. This model builds upon its predecessor through **Continual Pre-Training**, reaching a total training volume of **3.2** trillion tokens. It exhibits enhancements in all capabilities, particularly in mathematics and coding skills, with a **20%** improvement on the GSM8K benchmark and a **41%** increase on HumanEval. 
+
+**[2023/11/29]** Update model architecture and additional pre-training data information.  
+
+**[2023/11/24]** Update the related information of the pre-training data.  
+
+**[2023/11/06]** Released the XVERSE-65B base model.  
+
+## 模型介绍
+
+**XVERSE-65B** 是由深圳元象科技自主研发的支持多语言的大语言模型（Large Language Model），参数规模为 650 亿，本次开源的模型为底座模型 **XVERSE-65B**，主要特点如下：
+
+- **模型结构**：XVERSE-65B 使用主流 Decoder-only 的标准 Transformer 网络结构，支持 16K 的上下文长度（Context Length），能满足更长的多轮对话、知识问答与摘要等需求，模型应用场景更广泛。
+- **训练数据**：构建了 2.6 万亿 token 的高质量、多样化的数据对模型进行充分训练，包含中、英、俄、西等 40 多种语言，通过精细化设置不同类型数据的采样比例，使得中英两种语言表现优异，也能兼顾其他语言效果。
+- **分词**：基于 BPE（Byte-Pair Encoding）算法，使用上百 GB 语料训练了一个词表大小为 100,534 的分词器，能够同时支持多语言，而无需额外扩展词表。
+- **训练框架**：训练中采用 FlashAttention2 加速计算，3D 并行基础上采用虚拟流水线（virtual pipeline）技术，降低较长流水线和 16k 上下文窗口产生的过高气泡率，在千卡集群的峰值算力利用率达到业界前列。同时通过集群基础设施运营、资源调度、训练框架和调度平台协同等持续优化，打造出高稳定、低中断、强容错的训练系统，将每周有效训练率提升至 98.6%。
+
+**XVERSE-65B**的模型大小、架构和学习率如下：
+
+| params | d_model | n_heads | n_layers |  d_ff | learning rate |
+|:------:|:-------:|:-------:|:--------:|:-----:|:-------------:|
+|  65B |   8192  |    64   |    80    | 22016 |     1.5e−4    |
+
+## Model Introduction
+
+**XVERSE-65B** is a multilingual large language model, independently developed by Shenzhen Yuanxiang Technology. The models released this time is the base model **XVERSE-65B**. Its key features are as follows:
+
+- **Model Structure**: XVERSE-65B uses the mainstream Decoder-only Transformer network structure, supports 16k context length, which can meet the need of longer multi-round dialogues, knowledge question-answering, and summarization. This makes the model more versatile in application scenarios.
+- **Training Data**: The model has been thoroughly trained on a diversified and high-quality dataset consisting of 2.6 trillion of tokens, including more than 40 languages such as Chinese, English, Russian, and Spanish. The sampling ratio of different types of data is finely set, which makes the performance of Chinese and English excellent, and also takes into account the effect of other languages.
+- **Tokenization**: Based on the BPE (Byte-Pair Encoding) algorithm, a tokenizer with a vocabulary size of 100,534 has been trained using hundreds of gigabytes of language data. This tokenizer is capable of supporting multilingual without the need for additional vocabulary expansion.
+- **Training Framework**: The training utilizes FlashAttention2 for accelerated computation, and on top of 3D parallelism, virtual pipeline technology is applied to reduce the excessive bubble rate caused by longer pipelines and 16k context windows. This achieves a peak computational efficiency within the industry-leading range in the petaflop-scale cluster. Concurrently, through continuous optimization of cluster infrastructure operations, resource scheduling, training frameworks, and the scheduling platform, a highly stable, low-interruption, and robust fault-tolerant training system has been developed, enhancing the effective weekly training rate to 98.6%.
+
+The models sizes, architectures and learning rate of **XVERSE-65B** are showed as follows:
+
+| params | d_model | n_heads | n_layers |  d_ff | learning rate |
+|:------:|:-------:|:-------:|:--------:|:-----:|:-------------:|
+|  65B |   8192  |    64   |    80    | 22016 |     1.5e−4    |
+
+## 环境准备
+
+我们建议您克隆[`vllm`](https://github.com/vllm-project/vllm.git)并按照官方指南进行安装。
+
+## Environment Setup
+
+We advise you to clone [`vllm`](https://github.com/vllm-project/vllm.git) and install it following the official guide.
+
+## 使用方法
+
+我们演示了如何使用 `vllm` 来运行XVERSE-65B-Chat-GPTQ-Int4量化模型:
+
+```python
+from vllm import LLM, SamplingParams
+
+model_dir = "xverse/XVERSE-65B-Chat-GPTQ-Int4/"
+
+# Create an LLM.
+llm = LLM(model_dir, 
+          trust_remote_code=True)
+
+# Create a sampling params object.
+sampling_params = SamplingParams(temperature=0.5, top_p=0.85, max_tokens=2048, repetition_penalty=1.1)
+
+# Generate texts from the prompts. The output is a list of RequestOutput objects
+# that contain the prompt, generated text, and other information.
+prompts = ["Human: 请你写一篇关于环保的文章，题材是从个人做起。\n\nAssistant: ",]
+outputs = llm.generate(prompts, sampling_params)
+
+# Print the outputs.
+for output in outputs:
+    prompt = output.prompt
+    generated_text = output.outputs[0].text
+    print(f"Generated text:\n{generated_text}")
+```
+
+## Usage
+
+We demonstrated how to use 'vllm' to run the XVERSE-65B-Chat GPTQ Int4 quantization model:
+
+```python
+from vllm import LLM, SamplingParams
+
+model_dir = "xverse/XVERSE-65B-Chat-GPTQ-Int4/"
+
+# Create an LLM.
+llm = LLM(model_dir, 
+          trust_remote_code=True)
+
+# Create a sampling params object.
+sampling_params = SamplingParams(temperature=0.5, top_p=0.85, max_tokens=2048, repetition_penalty=1.1)
+
+# Generate texts from the prompts. The output is a list of RequestOutput objects
+# that contain the prompt, generated text, and other information.
+prompts = ["Human: 请你写一篇关于环保的文章，题材是从个人做起。\n\nAssistant: ",]
+outputs = llm.generate(prompts, sampling_params)
+
+# Print the outputs.
+for output in outputs:
+    prompt = output.prompt
+    generated_text = output.outputs[0].text
+    print(f"Generated text:\n{generated_text}")
+```
+
+## 局限性与免责申明
+
+XVERSE-65B 与其他所有 LLM 一样，在某些情况下可能会产生不准确、有偏见或其他令人反感的内容。因此，请谨慎使用模型生成的内容，请勿将生成的有害内容进行传播，在部署任何 XVERSE-65B 的应用之前，开发人员应根据其具体应用对模型进行安全测试和调优。
+
+我们强烈警告不要将 XVERSE-65B 模型用于制造或传播有害信息，或进行任何可能损害公众、国家、社会安全或违反法规的活动。如果使用 XVERSE-65B 模型产生任何问题，无论是数据安全问题、公共舆论风险，还是模型被误解、滥用、传播或不合规使用所引发的任何风险和问题，我们将不承担任何责任。
+
+## Limitations and Disclaimer
+
+Like all other Large Language Models (LLMs), XVERSE-65B may produce inaccurate, biased, or otherwise offensive content under certain circumstances. Therefore, please use the content generated by the model with caution and refrain from disseminating harmful content. Before deploying any application of XVERSE-65B, developers should conduct safety tests and optimization of the model according to its specific application.
+
+We strongly warn against the use of the XVERSE-65B model for producing or spreading harmful information, or conducting any activities that might harm the public, national, or social security, or violate regulations. We assume no responsibility for any problems arising from the use of the XVERSE-65B model, whether it be data security issues, public opinion risks, or any risks and issues caused by misunderstanding, misuse, dissemination, or non-compliance with the model.
+
+## 模型开源协议
+
+使用本仓库的源码需要遵循 [Apache-2.0](https://github.com/xverse-ai/XVERSE-65B/blob/main/LICENSE) 开源协议，使用 XVERSE-65B 的模型权重则需要遵循[模型许可协议](https://github.com/xverse-ai/XVERSE-65B/blob/main/MODEL_LICENSE.pdf)。
+
+XVERSE-65B 模型权重对学术研究**完全开放**，并且支持**免费商用**。如需申请商业许可证，请填写【[申请表](https://chat.xverse.cn/home/business.html)】，如有其他问题或合作，请联系 <opensource@xverse.cn>。
+
+## Open Source License
+
+The use of the source code in this repository must follow the [Apache-2.0](https://github.com/xverse-ai/XVERSE-65B/blob/main/LICENSE) open-source license, while the use of the model weights of XVERSE-65B needs to adhere to the [Model License Agreement](https://github.com/xverse-ai/XVERSE-65B/blob/main/MODEL_LICENSE.pdf).
+
+The XVERSE-65B model weights are **fully open** to academic research and support **free commercial use**.  To apply for a commercial license, please fill in the [application form](https://chat.xverse.cn/home/business.html). For other questions or collaborations, please contact <opensource@xverse.cn>.

config.json

@@ -0,0 +1,41 @@
+{
+  "_name_or_path": "/mnt/llm_dataset/libinbin/xverse_65b/LLaMA-Megatron/checkpoint/65b-2.5t-stage2-turbo_40w_clean/pretrain_PP1_new/iter_0000639/mp_rank_00/",
+  "architectures": [
+    "XverseForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_xverse.XverseConfig",
+    "AutoModelForCausalLM": "modeling_xverse.XverseForCausalLM"
+  },
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "hidden_act": "silu",
+  "hidden_size": 8192,
+  "initializer_range": 0.02,
+  "intermediate_size": 22016,
+  "max_position_embeddings": 16384,
+  "max_tokenizer_truncation": 16384,
+  "model_type": "xverse",
+  "num_attention_heads": 64,
+  "num_hidden_layers": 80,
+  "pad_token_id": 1,
+  "quantization_config": {
+    "bits": 8,
+    "damp_percent": 0.01,
+    "desc_act": true,
+    "group_size": 128,
+    "is_marlin_format": false,
+    "model_file_base_name": null,
+    "model_name_or_path": null,
+    "quant_method": "gptq",
+    "static_groups": true,
+    "sym": true,
+    "true_sequential": true
+  },
+  "rms_norm_eps": 1e-06,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.39.1",
+  "use_cache": true,
+  "vocab_size": 100534
+}

configuration_xverse.py

@@ -0,0 +1,123 @@
+# coding=utf-8
+# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
+# and OPT implementations in this library. It has been modified from its
+# original forms to accommodate minor architectural differences compared
+# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" XVERSE model configuration"""
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+XVERSE_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
+
+
+class XverseConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`XverseModel`]. It is used to instantiate an Xverse
+    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
+    defaults will yield a similar configuration to that of the XVERSE-13B.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 100278):
+            Vocabulary size of the XVERSE model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`XverseModel`]
+        hidden_size (`int`, *optional*, defaults to 5120):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 13824):
+            Dimension of the MLP representations.
+        num_hidden_layers (`int`, *optional*, defaults to 40):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 40):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 8192):
+            The maximum sequence length that this model might ever be used with. Typically set this to something large
+            just in case (e.g., 512 or 1024 or 2048).
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-6):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        tie_word_embeddings(`bool`, *optional*, defaults to `False`):
+            Whether to tie weight embeddings
+
+        Example:
+
+    ```python
+    >>> from transformers import XverseModel, XverseConfig
+
+    >>> # Initializing a Xverse XVERSE-13B style configuration
+    >>> configuration = XverseConfig()
+
+    >>> # Initializing a model from the XVERSE-13B style configuration
+    >>> model = XverseModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+    model_type = "xverse"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=100278,
+        hidden_size=5120,
+        intermediate_size=13824,
+        num_hidden_layers=40,
+        num_attention_heads=40,
+        hidden_act="silu",
+        max_position_embeddings=8192,
+        max_tokenizer_truncation=8192,
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=1,
+        eos_token_id=2,
+        tie_word_embeddings=False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.max_tokenizer_truncation = max_tokenizer_truncation
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )

gptq_model-8bit-128g.safetensors.aa

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dec09ee26a3dc02d7b9640aacd7400d2aa96cf799bb8ec1998693d1c3125553f
+size 23203847901

gptq_model-8bit-128g.safetensors.ab

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84e2e1f70f193abe6f726d3e501e0790be8e937c7122d3aaff9709fc01dc61b1
+size 23203847901

gptq_model-8bit-128g.safetensors.ac

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94b9f65b79053005bac0a0af4f59fa1bf4d730ba43dd2a86ae89fa55eb5dc340
+size 23203847902

quantize_config.json

@@ -0,0 +1,13 @@
+{
+  "bits": 8,
+  "group_size": 128,
+  "damp_percent": 0.01,
+  "desc_act": true,
+  "static_groups": true,
+  "sym": true,
+  "true_sequential": true,
+  "model_name_or_path": null,
+  "model_file_base_name": null,
+  "is_marlin_format": false,
+  "quant_method": "gptq"
+}

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+    "bos_token": {
+      "content": "<|startoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    "eos_token": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    "pad_token": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  }

tokenizer_config.json

@@ -0,0 +1,5 @@
+{
+  "clean_up_tokenization_spaces": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}