yujiepan/meta-llama-3.1-tiny-random-hidden128-awq-w4g64

This model is for debugging. It is randomly initialized using the config from meta-llama/Meta-Llama-3.1-70B-Instruct but with smaller size.

Codes:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "yujiepan/meta-llama-3.1-tiny-random-hidden128"
quant_config = {
    "zero_point": True,
    "q_group_size": 64,
    "w_bit": 4,
    "version": "GEMM",
}
# Load model
model = AutoAWQForCausalLM.from_pretrained(
    model_path, low_cpu_mem_usage=True, use_cache=False, device_map='cuda',
)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Quantize
model.quantize(tokenizer, quant_config=quant_config)