Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

OpenVINO Weight-Quantized LLMs

This repo contains binary of weight quantized by OpenVINO.

| LLM             	| ratio 	| group_size 	|
|-----------------	|-------	|------------	|
| llama-2-chat-7b 	| 0.8   	| 128        	|
| mistral-7b      	| 0.6   	| 64         	|
| gemma-2b-it     	| 0.6   	| 64         	|

Notes:

  • ratio=0.8 means 80% of FC (linear) layers are 4-bit weight quantized and the rest in 8-bit.
  • group_size refers to number of elements being considered to be quantized. e.g. group size of 128 means each output channel of the weight is split into groups of 128 for quantization.
  • q_weight is in uint8 even it is 4-bit.
  • scripts are for internal use only.

Example usage of the saved blob

import torch

blob_path = "./mistral-7b_r0.6_g64.pth"

blob = torch.load(blob_path)

for layer, attr in blob.items():
    print(f"{layer:30} | q_dtype: {attr['q_dtype']:5} | orig. shape: {str(attr['original_shape']):15} | quantized_shape: {str(attr['q_weight'].shape):15}")
# Sample outputs:
.
.
layers.14.mlp.gate_proj        | q_dtype: u4    | orig. shape: (11008, 4096)   | quantized_shape: (11008, 32, 128)
layers.14.mlp.down_proj        | q_dtype: u4    | orig. shape: (4096, 11008)   | quantized_shape: (4096, 86, 128)
layers.15.self_attn.k_proj     | q_dtype: u8    | orig. shape: (4096, 4096)    | quantized_shape: (4096, 4096)   
layers.15.self_attn.v_proj     | q_dtype: u8    | orig. shape: (4096, 4096)    | quantized_shape: (4096, 4096)   
layers.15.self_attn.q_proj     | q_dtype: u4    | orig. shape: (4096, 4096)    | quantized_shape: (4096, 32, 128)
layers.15.self_attn.o_proj     | q_dtype: u4    | orig. shape: (4096, 4096)    | quantized_shape: (4096, 32, 128)
layers.15.mlp.up_proj          | q_dtype: u4    | orig. shape: (11008, 4096)   | quantized_shape: (11008, 32, 128)
layers.15.mlp.gate_proj        | q_dtype: u4    | orig. shape: (11008, 4096)   | quantized_shape: (11008, 32, 128)
layers.15.mlp.down_proj        | q_dtype: u4    | orig. shape: (4096, 11008)   | quantized_shape: (4096, 86, 128)
layers.16.self_attn.k_proj     | q_dtype: u8    | orig. shape: (4096, 4096)    | quantized_shape: (4096, 4096)   
layers.16.self_attn.v_proj     | q_dtype: u8    | orig. shape: (4096, 4096)    | quantized_shape: (4096, 4096)   
.
.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Unable to determine this model's library. Check the docs .