Edit model card

This is an experimental HQQ 1-bit quantized (binary weights) Llama2-7B-chat model using a low-rank adapter to improve the performance (referred to as HQQ+).


Quantizing small models at extreme low-bits is a challenging task. The purpose of this model is to show the community what to expect when fine-tuning such models. We notice that, 1-bit quantization doesn't work well when applied directly on small models such as the Llama2-7B. However, when fine-tuned, the model's ouput significantly improves. In fact, the 1-bit base model outperforms Quip# 2-bit after fine-tuning on ~2.8K samples.

Note that the weights here are unsigned 1-bit (0 or 1), not ternary like the recent 1.58-bit work . This is a more challenging task since we lose the sign of the weights and only fine-tune a small fraction of the parameters (~94MB worth of weights). The dequantization step can be rewriten as a 1-bit matmul which could potentially require only additions + a very low-rank matmul which is fast to compute.

This version offloads the meta-data to the CPU, so only the binary weights and the low-rank adapters are stored in the GPU memory.


The adapter was trained via SFT on random subsets of the following:

Base Model

Chat Model


Models Llama2-7B (fp16) Llama2-7B (HQQ 1-bit) Llama2-7B (HQQ+ 1-bit) Quip# (2-bit)
Wiki Perpexlity 5.18 9866 8.53 8.54
VRAM (GB) 13.5 1.76 1.85 2.72
forward time (sec) 0.1 0.231 0.257 0.353
Models Llama2-7B-chat (fp16) Llama2-7B-chat (HQQ 1-bit) Llama2-7B-chat (HQQ+ 1-bit)
ARC (25-shot) 53.67 21.59 31.14
HellaSwag (10-shot) 78.56 25.66 52.96
MMLU (5-shot) 48.16 25.08 26.54
TruthfulQA-MC2 45.32 47.81 43.16
Winogrande (5-shot) 72.53 49.72 60.54
GSM8K (5-shot) 23.12 0 11
Average 53.56 28.31 37.56


First, install the latest version of HQQ:

pip install git+https://github.com/mobiusml/hqq.git

Then you can use the sample code below:

from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer

#Load the model
model_id = 'mobiuslabsgmbh/Llama-2-7b-chat-hf_1bitgs8_hqq' 
model     = HQQModelForCausalLM.from_quantized(model_id, adapter='adapter_v0.1.lora')
tokenizer = AutoTokenizer.from_pretrained(model_id)

#Setup Inference Mode
tokenizer.add_bos_token = False
tokenizer.add_eos_token = False
if not tokenizer.pad_token: tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model.config.use_cache  = True

# Optional: torch compile for faster inference
# model = torch.compile(model)

#Streaming Inference
import torch, transformers
from threading import Thread

def chat_processor(chat, max_new_tokens=100, do_sample=True, device='cuda'):
    tokenizer.use_default_system_prompt = False
    streamer = transformers.TextIteratorStreamer(tokenizer, timeout=10.0, skip_prompt=True, skip_special_tokens=True)

    generate_params = dict(
        tokenizer("<s> [INST] " + chat + " [/INST] ", return_tensors="pt").to(device),
        top_p=0.90 if do_sample else None,
        top_k=50 if do_sample else None,
        temperature= 0.6 if do_sample else None,

    t = Thread(target=model.generate, kwargs=generate_params)
    print("User: ", chat); 
    print("Assistant: ");
    outputs = ""
    for text in streamer:
        outputs += text
        print(text, end="", flush=True)

    return outputs


outputs = chat_processor("What is the solution to x^2 - 1 = 0", max_new_tokens=1000, do_sample=False)
User:  What is the solution to x^2 - 1 = 0
The equation $x^2 - 1 = 0$ can be factored as $(x-1)(x+1) = 0$.
You want to find a value of $x$ that makes this true for all values of $x$. This means that either $x=1$ or $-1$, or $x=-1$. So, there are two solutions: $x=\boxed{1}$ and $x=\boxed{-1}$. The answer is: 1
Downloads last month
Inference API
Input a message to start chatting with mobiuslabsgmbh/Llama-2-7b-chat-hf_1bitgs8_hqq.
Inference API (serverless) has been turned off for this model.

Space using mobiuslabsgmbh/Llama-2-7b-chat-hf_1bitgs8_hqq 1

Collection including mobiuslabsgmbh/Llama-2-7b-chat-hf_1bitgs8_hqq