infly/INF-34B-Chat-GPTQ-8bit

1. Introduction

INF-34B has 34 billion parameters with a context window length of 32K, and is trained on about 3.5T well-processed tokens from English and Chinese bilingual corpus. Compared with open source models of the comparable size, INF-34B not only provides competitive performance in the OpenCompass evaluation, but also has impressive potential on both finance and healthcare domains. Besides, the quantized INF-34B runs on graphics cards of 24GB VRAM with negligible accuracy loss, which facilitates commercial applications, especially low-resource scenarios.

Detailed for Training GPT Model: We provide comprehensive details about our model pretraining and alignment, including high-quality data pipeline, instruction data preparation, and quantization results etc.
Superior Performance on Benchmarks: We demonstrate superior performance of the INF-34B models by comparing against two competitors with comparable model size, Qwen1.5-32B and Yi1.5-34B, on the public OpenCompass benchmarks.

2. Models

We release the base and chat models with 34B parameters based on the LLaMA framework, while using LayerNorm with zero-centered gamma instead of RMSNorm for training stability. Please note that you could use our models for commercial applications under the terms outlined in License section.

Huggingface

Model	Sequence Length	Download
INF-34B-Base	32K	🤗 HuggingFace
Inf-34B-Chat	32K	🤗 HuggingFace
Inf-34B-Chat-GPTQ-4bits	32K	🤗 HuggingFace
Inf-34B-Chat-GPTQ-8bits	32K	🤗 HuggingFace
Inf-34B-Chat-AWQ	32K	🤗 HuggingFace

3. Benchmarks

Note: If you want to reproduce the evaluation results, please refer to the details of evaluation, including prompts, postprocess scripts and version of inference frameworks.

Base Model

We evaluate our model on several academic benchmarks then compare with other similar-sized open access model. INF-34B has stronger performance in the fields that we chose to optimize while simultaneously preserves the general capabilities of LLM such as commonsense, world knowledge, math and coding.

model	QWen1.5-32B	Yi1.5-34B	INF-34B
MMLU(5-shot)	73.60	77.86	76.11
CMMLU(5-shot)	81.87	81.85	80.08
GSM8K(4-shot)	72.86	80.06	83.02
MATH(4-shot)	36.80	33.88	38.34
HumanEval(0-shot)	44.51	47.56	65.24
MBPP(3-shot)	51.00	65.60	64.00
BBH(3-shot)	70.60	74.83	71.20
HellaSwag(0-shot)	82.03	81.57	83.32

Note: To facilitate reproduction, the results of common benchmarks are generated by OpenCompass except humaneval and mbpp as we experience code timeout and postprocess issues.

Chat Model

We present the performance results of our chat model and other LLM on various standard benchmarks, as well as two domain-specific benchmarks.

model	QWen1.5-32B-Chat	Yi1.5-34B-Chat	INF-34B-Chat
MT-bench	8.3	8.5	8.3
AlignBench	7.1	7.2	7.1
IFEval	49.54	58.04	59.70
Arena-Hard	24.2	42.6	43.1
GSM8K	81.42	79.45	84.04
MATH	42.28	54.06	51.48

Long Context

We employed a long context SFT dataset of various length. Specifically, 37.7% shorter than 8k tokens, 40.5% falling within 8k to 16k tokens and 21.8% ranging from 16k to 32k tokens. And Our model has demonstrated superior performance on LongBench(via OpenCompass) tasks compared to Qwen1.5-32B.

model	Single-Doc QA	Multi-Doc QA	Summari- zation	Few-shot Learning	Synthetic	Code
QWen1.5-32B-Chat	45.6	40.4	23.1	52.6	67.3	43.8
INF-34B-Chat	47.4	43.2	24.1	66.0	66.8	57.2

Note: All the reported results on the table are the average of sub-tasks for different categories.

INF-34B-32k also performs well across context window lengths up to 32k on Single-Needle RetrievalTask(S-RT) as visualized below.

4. Training Details

Data Pipeline

We propose different data pipelines for general, domain and code data to ensure the richness, variety and quality of training samples. The general data pipeline involves general processing methods. For the domain data of interest, e.g., math, wiki, code, we propose a domain-specific data pipeline to extract the domain data from Common Crawl (CC). We also devise a code-specific pipeline to handle massive code data, since the code data has proven its effectiveness in improving the model’s reasoning and comprehension ability.

General Data Pipeline: Our text cleaning pipeline mainly includes two stages: filtering and deduplication. The filtering involves language identification, URL filtering, and heuristic filtering rules. The deduplication includes both fuzzy deduplication and exact deduplication techniques.
Domain Data Pipeline: We propose an iterative high-quality data retrieval method that recalls relevant data from the Common Crawl (CC) dataset for various target domains. It comprises three main component: FastText training, Performing recall and Human annotation.
Code Data Pipeline: The code-specific data processing pipeline includes modules of preprocessing, heuristic filtering, deduplication, transformation and data mixture.

Training Settings

The architecture choices of INF-34B follows LLaMA framework. Specifically, we opt Rotary Embedding for positional encoding, SwiGLU for activation function, Grouped Query Attention (GQA) and LayerNorm with zero-centered gamma instead of RMSNorm for training stability.

Motivated by the idea of first training on relatively large but less polished corpus to equip the model with language understanding and world knowledge and then improves model’s domain knowledge and reasoning ability, our training process is split into 3 stages:

Stage 1: The dataset mainly includes web text, paper, Wikipedia and source code. In this early stage, we aim at larger data and higher diversity.
Stage 2: For second stage we seek to gradually challenge the model with longer and more complex texts. We up-weight long texts in the same data distribution of stage 1. We tune the rope base and extend our context window to 32k for more sophisticated comprehension of human knowledge.
Stage 3: The final stage is composed of domain data recalled from Web text and synthetic data.

5. Inference

This repo contains GPTQ model file, which are made with AutoGPTQ.

Explanation of GPTQ parameters

Bits: The bit size of the quantised model.
GS: GPTQ group size. Higher numbers use less VRAM, but have lower quantisation accuracy. "None" is the lowest possible value.
Act Order: True or False. Also known as desc_act. True results in better quantisation accuracy. Some GPTQ clients have had issues with models that use Act Order plus Group Size, but this is generally resolved now.
Damp %: A GPTQ parameter that affects how samples are processed for quantisation. 0.01 is default, but 0.1 results in slightly better accuracy.

We provide the inference examples from Python code. You can then use the following code.

Installation

Requires: The environment of the Hugging Face transformers is:

Pytorch 2.3.0+cu121
Flash Attention 2.5.0
Transformers 4.42.4

pip3 install transformers optimum
pip3 uninstall -y auto-gptq
git clone https://github.com/infly-ai/AutoGPTQ
cd AutoGPTQ
git checkout inflm
pip3 install .
# If you are compiling on an A800/H800 GPU, you can add the environment 'export TORCH_CUDA_ARCH_LIST="8.0;9.0"'

Inference with Huggingface's Transformers

We provide the inference examples with Huggingface's Transformers.

from transformers import AutoModelForCausalLM, AutoTokenizer
import time

model_path="path/to/model/"
device = "cuda" # the device to load the model onto
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    trust_remote_code=True
)

prompt = "Write a resume for a fresh high school graduate who is seeking their first job. Make sure to include at least 12 placeholder represented by square brackets, such as [address], [name]."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(device)

context_time = 0
start = time.time()
generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=200
)
context_time += time.time() - start

generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print("=> response: ", response)

Inference with AutoGPTQ

The above files provided are tested to work with Transformers. AutoGPTQ can also be used directly. Note: If you encounter RuntimeError: probability tensor contains either inf, nan or element < 0 during inference with transformers, or the generation token contains <unk>, we strongly recommend that you use the following method to run the code.

import torch
from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer

model_path = "/path/to/model/"

device = "cuda:0"
tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

model = AutoGPTQForCausalLM.from_quantized(
    model_path,
    inject_fused_attention=False,
    inject_fused_mlp=False,
    device=device,
    trust_remote_code=True,
    use_marlin=True, # marlin kernel slove <unk> for gptq-4bit, for gptq-8bit set 'use_marlin=False'
    use_triton=False, # for gptq-8bit set 'use_triton=True'
)
model.eval()

prompts = [
    "I would like to",
    "北京的冬天很冷，广州的夏天很热。",
    "I have a dream that",
    "To be or not to be, that is the question.",
]

token_dict = tokenizer(prompts, return_tensors="pt", padding="longest").to(device)
with torch.inference_mode():
    output_ids = model.generate(**token_dict, max_new_tokens=200)
output_ids_cut = output_ids[:, token_dict["input_ids"].shape[1] :]

for nb, output_id in enumerate(output_ids_cut):
    print(f"Prompt {nb}: {prompts[nb]}")
    print(f"Generated: {tokenizer.decode(output_id, skip_special_tokens=False)}")
    print('*'*40)

6. License

INF-34B series (including Base and Chat) support commercial applications under a permissive License.

7. Citation

@article{inf-llm,
  author = {INF-Team},
  title = {INF’s Open-Source Large Language Models},
  year = {2024},
  url = {https://github.com/infly-ai/INF-LLM}
}

8. Contact

If you have any questions or seek for cooperation, please contact us at bd@infteach.ai.