Quark-150M-128K-Zero β€” GGUF

GGUF conversions of empero-ai/Quark-150M-128K-Zero, a 165M-parameter research chat model (RL/GRPO from base, no SFT) built on the custom Quark architecture.

This repo ships the quantized weights plus a patched llama.cpp source tree that teaches llama.cpp to recognize the quark architecture (needed to produce and load these files).

Files

File Quant Size BPW
quark-150m-128k-zero-bf16.gguf BF16 318 MiB 16.15
quark-150m-128k-zero-q8_0.gguf Q8_0 173 MiB 8.72
quark-150m-128k-zero-q6_k.gguf Q6_K 135 MiB 6.80
quark-150m-128k-zero-q4_k_m.gguf Q4_K_M 101 MiB 5.07
quark-150m-128k-zero-q3_k_m.gguf Q3_K_M 84 MiB 4.18
quark-150m-128k-zero-q2_k.gguf Q2_K 70 MiB 3.50

Source snapshot of the patched inference/quantization runtime: llama-cpp-src.tar.gz.

⚠️ Architecture & runtime status β€” read this first

QuarkForCausalLM is a novel, custom architecture that upstream llama.cpp, Ollama and LM Studio do not support out of the box. It is a hybrid decoder:

  • Local layers (12): sliding-window attention (window 256) with RoPE, QK-norm and a learned per-head attention sink, blended per channel with a double-gated short causal convolution (kernel 4).
  • Global layers (4, every 4th): full causal grouped-query attention (12 Q heads / 4 KV heads), RoPE, QK-norm.
  • Multi-head gated value residuals anchored to layer 0, depth-scaled "sandwich" residuals, tied embeddings, 131,072-token context, 32,896-entry vocabulary.

What this means for these GGUF files:

  • βœ… They are valid GGUF v3 containers, correctly shaped and quantized (produced with llama-quantize using real k-quant mixtures, not naive re-packing).
  • βœ… They carry full metadata (arch = quark, hyper-params, tokenizer, chat template) and can be inspected, quantized further, or consumed by tooling.
  • ⚠️ Full inference is not yet implemented. The bundled llama.cpp patch registers the quark architecture so the model loads/quantizes, but the forward graph (conv blend, value residuals, sinks) is not wired up yet. So stock llama.cpp / Ollama / LM Studio cannot generate text from these files today.

These artifacts are intended for research, model-distribution, and as a foundation for a future quark inference implementation. If you need runnable inference, the next step is implementing the Quark graph in llama.cpp (the arch is already registered in the bundled source).

What's in llama-cpp-src.tar.gz

A source snapshot of ggml-org/llama.cpp (shallow, tip commit) with a minimal quark architecture registration patch applied β€” 4 files, +13 lines:

  • src/llama-arch.h β€” add LLM_ARCH_QUARK
  • src/llama-arch.cpp β€” map "quark" β†’ LLM_ARCH_QUARK
  • src/llama-model.cpp β€” factory case for llama_model_quark
  • src/models/models.h β€” minimal llama_model_quark model class

Packaged like git clone --depth 1 (shallow history β€” only the tip commit, no full upstream history) and with no build/ directory. Build it with CMake to get a llama-quantize (and friends) that understand quark.

How these were produced

  1. Downloaded the F32 model.safetensors from the source model.
  2. Converted to GGUF with a custom writer (arch quark, GPT-2 byte-level BPE tokenizer, chat template preserved).
  3. Quantized the F32 master to each target with llama-quantize from the bundled patched llama.cpp: BF16, Q8_0, Q6_K, Q4_K_M, Q3_K_M, Q2_K.

License

Apache-2.0 β€” same as the source model empero-ai/Quark-150M-128K-Zero. See LICENSE in the source repository and the llama.cpp license (MIT) for the bundled runtime source.

Downloads last month
191
GGUF
Model size
0.2B params
Architecture
quark
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for usermma/test

Quantized
(1)
this model