Llama-Poro-2-70B-Instruct-oQ3e

A 3-bit (mixed-precision, imatrix-calibrated) MLX quantization of LumiOpen/Llama-Poro-2-70B-Instruct, produced with oQ (oMLX v0.6.4) importance-matrix quantization. It exists to make the 70B model runnable on Apple Silicon Macs with as little as ~48 GB of unified memory.

3-bit is an aggressive quantization. Expect some quality loss relative to a 4-bit or 8-bit quant of the same model. The importance-matrix calibration and the higher-precision embedding, attention and output layers (see below) are there to limit the damage, but if you have the memory, prefer a 4-bit build. Pairing this model with a small speculative draft model (e.g. an 8-bit Poro 2 8B quant) recovers a good deal of interactive speed.

Original model

Poro 2 70B Instruct is a Finnish/English instruction-following model built on the Llama 3.1 70B architecture (70.55B parameters, 80 layers, 128256 vocab, 8192 context) through continued pretraining, SFT and DPO. It was created by AMD Silo AI, the TurkuNLP group at the University of Turku, and HPLT, and trained on the LUMI supercomputer.

All model behaviour, capabilities, evaluation results, training data, intended use and limitations are documented in the original model card. This repository only changes the weight precision and storage format — please refer to the original for everything else.

Quantization details

• Base model: LumiOpen/Llama-Poro-2-70B-Instruct
• Tool: oQ / oMLX v0.6.4, enhanced (imatrix) path
• Architecture: llama (70.55B params, 80 layers, 128256 vocab, 8192 context)
• Precision: 3-bit affine base, mixed-precision:
  • token embeddings and the output projection (lm_head) kept at 8-bit
  • a subset of attention projections (k_proj, v_proj, o_proj, q_proj) and a few mlp.down_proj layers boosted to 4–6-bit by measured sensitivity
  • all other linear weights at 3-bit
• Group size: 64
• Calibration: importance matrix (imatrix / oQe) enabled, oqe_code_multilingual calibration set, 128 samples at sequence length 512. Because the bf16 model does not fit in memory, calibration was run against a temporary 4-bit proxy of the model.
• Format: MLX safetensors (7 shards)
• Size on disk: 32.6 GB (3.7 bits per weight effective)

Per-tensor bit assignments are in config.json; calibration metadata is in oq_imatrix_report.json.

Usage

Requires mlx-lm on an Apple Silicon Mac.

pip install mlx-lm

mlx_lm.generate --model janihal/Llama-Poro-2-70B-Instruct-oQ3e \
  --prompt "Mikä on Suomen pääkaupunki?"

from mlx_lm import load, generate

model, tokenizer = load("janihal/Llama-Poro-2-70B-Instruct-oQ3e")
messages = [{"role": "user", "content": "Kerro lyhyesti poroista."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

The chat template is bundled in tokenizer_config.json.

Speculative decoding

Generation is memory-bandwidth bound. Drafting with the much smaller Poro 2 8B (same family, same tokenizer) speeds up token generation with no change to outputs:

mlx_lm.generate --model janihal/Llama-Poro-2-70B-Instruct-oQ3e \
  --draft-model janihal/Llama-Poro-2-8B-Instruct-oQ8e \
  --num-draft-tokens 4 \
  --prompt "Kirjoita lyhyt runo talvesta."

Related quantizations

• janihal/Llama-Poro-2-8B-Instruct-oQ4e — 4-bit 8B
• janihal/Llama-Poro-2-8B-Instruct-oQ8e — 8-bit 8B (good speculative draft model for this repo)

License

Released under the Llama 3.3 Community License, inherited from the base model. Built with Llama.

Attribution

If you use this model, please cite the original Poro 2 work by LumiOpen / AMD Silo AI / TurkuNLP / HPLT.