oxfrug/Llama-3-8B-instruct-awq

AI-Sweden Llama-3-8B-instruct — AWQ W4A16

4-bit AWQ (W4A16, asymmetric) of AI-Sweden-Models/Llama-3-8B-instruct.

Official Hub files are BF16 and GGUF. EXL2 lives at oxfrug/Llama-3-8B-instruct-exl2. This repo is the vLLM / TGI pack.

Load

# vLLM
from vllm import LLM
llm = LLM("oxfrug/Llama-3-8B-instruct-awq")

hf download oxfrug/Llama-3-8B-instruct-awq --local-dir ./Llama-3-8B-instruct-awq

Format is compressed-tensors from llmcompressor, not classic AutoAWQ GEMM files.

Convert

• Tool: llmcompressor AWQModifier + QuantizationModifier(scheme="W4A16_ASYM")
• lm_head left in higher precision
• Calibration: 256 rows × 512 tokens from a Nordic/English wiki mix already used for EXL2 Swedish-heavy cal (sv_heavy.rows.jsonl). Not the frozen eval suite.
• Hardware: RTX 3090

License

Meta Llama 3 Community License. Keep NOTICE. Derivative of Meta Llama 3 via AI Sweden’s instruct tune.

Smoke (greedy)

Prompt	Output
Vad heter Sveriges huvudstad? Ett ord.	Huvudstaden i Sverige är Stockholm.
Name the capital of Sweden in one word.	Huvudstaden i Sverige är Stockholm.

Same Swedish-first habit as the BF16 parent. This is not a suite score.

Sibling

Pack	Repo
EXL2 4.0–6.0 bpw	oxfrug/Llama-3-8B-instruct-exl2
Eval suite (do not train)	oxfrug/nordic-instruct-eval-v2