oxfrug/Llama-3-8B-instruct-exl2

AI-Sweden Llama-3-8B-instruct — EXL2

ExLlamaV2 / EXL2 quants of AI-Sweden-Models/Llama-3-8B-instruct.
That checkpoint is a Swedish/Nordic instruct tune of Llama 3 8B. Official Hub files are BF16 (and a separate GGUF); there were no EXL2 uploads.

Converted with ExLlamaV2 0.3.2, lm_head at 6-bit, built-in default calibration (same recipe as turboderp-style quants). One measurement pass, then each bitrate from measurement.json.

Branches

Revision	Target bpw	Size (approx.)
4.0bpw	4.0	4.7 GB
4.5bpw	4.5	5.1 GB
5.0bpw	5.0	5.5 GB
5.5bpw	5.5	5.9 GB
6.0bpw	6.0	6.3 GB

measurement.json is on main if you want to roll another bitrate yourself.

Download a revision:

hf download oxfrug/Llama-3-8B-instruct-exl2 --revision 5.0bpw --local-dir ./Llama-3-8B-instruct-exl2-5.0bpw

TabbyAPI / ExUI: point the model path at a checked-out branch, or set the HF revision to 5.0bpw.

Notes

• License: Meta Llama 3 Community License. Keep the NOTICE file. This is a derivative of Meta Llama 3 via AI Sweden’s instruct tune.
• Loader: ExLlamaV2 (TabbyAPI, text-generation-webui exllamav2, ExUI). Not GGUF / llama.cpp.
• On PyTorch 2.13 without Flash Attention 2.5.7+, set config.no_sdpa = True before load. Default SDPA + causal_lower_right produced collapsed output in our tests; the explicit matmul path matched BF16 (6.0bpw raw-EN first 16 tokens identical). Older torch + flash-attn (paged attention) is the usual community stack and was not the convert path.
• These are not the experimental Swedish-heavy calibration files. Default EXL2 cal is what other EXL2 repos ship.

Also: AWQ W4A16 for vLLM/TGI.

Source

AI-Sweden-Models/Llama-3-8B-instruct
  ← AI-Sweden-Models/Llama-3-8B
    ← meta-llama/Meta-Llama-3-8B

Convert: ExLlamaV2 convert.py 0.3.2, RTX 3090.

Smoke (greedy, no_sdpa=True)

6.0bpw, same chat template as the BF16 card:

Prompt	Output
Vad heter Sveriges huvudstad? Ett ord.	Huvudstaden i Sverige är Stockholm.
Name the capital of Sweden in one word.	Stockholm

4.0bpw is coherent on the same prompts (Swedish-first on the English question, like the BF16 base). This is not a leaderboard score.