orcarouter/Flint-abliterated-GGUF

Flint-abliterated GGUF

🛡️ New from Orca — OrcaCyber Zero 1.5 Post-trained for cybersecurity: vulnerability analysis, vulnerability reproduction, red teaming & offensive-security research. 98.01% pass@1 on CyberGym Level 1 · 1M context · function calling · OpenAI-compatible API.

🔌 New from Orca — Orca Connect Running this GGUF in llama.cpp? Orca Connect puts the rig under your desk behind your usual OrcaRouter API key, next to 200+ hosted models, reachable from a laptop, a phone or a cloud agent. Outbound only: no port forwarding, no tunnel, no firewall rule. Requests your own hardware serves cost $0, and orca-connect claude points Claude Code at your rig for one session.

The uncensored Flint coding-agent model on one 16 GB or 32 GB GPU.

12 GB and 24 GB builds · on par with unsloth's UD files of the same size · llama.cpp

OrcaRouter AI Gateway · X · Discord · GitHub · All Models

---

Abliterated Flint, run locally.

These are GGUF builds of an abliterated (refusal-removed) RecursiveSI/flint. Flint is a 39.8B MoE for coding agents, with 256 routed experts (top-6) and one shared expert, about 2.5B active per token. It uses hybrid sliding-window / full attention with learned attention sinks, and a 262K-token context.

One refusal direction was removed from the BF16 weights before quantization. Harmful-prompt refusal drops from 86% to about 10%, and general capability stays within noise.

Routed experts are 95% of the weights, so that is where the bits were spent. Each group of expert layers got its own precision from measured KL divergence. Every tensor was then rounded with GPTQ-style error compensation into stock llama.cpp types, so no custom kernels are needed.

At about the same file size, the builds match unsloth's UD files on held-out text the search never saw. Mix-24G scores slightly lower KLD than UD-Q4_K_XL while being 1.1 GB smaller, and Mix-12G sits between UD-IQ1_M and UD-IQ2_XXS.

---

⚠️ Disclaimer: read before use

This model is abliterated: its safety alignment has been substantially reduced. It will comply with many harmful, unethical, offensive, or illegal requests that the original Flint would refuse.

• It is released strictly for legitimate research: interpretability, AI safety and refusal-mechanism study, red teaming, and robustness evaluation.
• You assume full responsibility for how you use it and for everything it generates. Add your own safety, moderation and abuse-prevention layers before any deployment.
• Use must comply with the Apache 2.0 license inherited from the base model, and with all applicable law.
• The authors accept no liability for misuse. Its outputs do not reflect the views of the uploaders or of Recursive.

---

Pick a build

Build	File	Experts	KLD ↓	Top-1 ↑	PPL vs. BF16
Mix-12G	11.99 GB	2.27 bpw	0.2023	88.1%	+20.0%
Mix-24G	23.99 GB	4.65 bpw	0.0186	96.5%	+1.1%

• Mix-24G fits a 32 GB GPU with room for a long context. It is close to the BF16: 96.5% of next tokens match, and perplexity is 1.1% higher.
• Mix-12G fits a 16 GB GPU. It averages 2.3 bits per expert weight, and that costs quality: 88% of next tokens match, and perplexity is 20% higher.

Use Mix-24G if you have the memory. Use Mix-12G when 16 GB is what you have.

The KV cache is small. Only the 14 full-attention layers keep the whole context, at about 28 KB per token in F16, and the 35 sliding-window layers keep 513 tokens each. So 64K tokens of context take about 1.9 GB, and 128K tokens take about 3.7 GB.

---

Evaluation

Fidelity to the BF16

All builds are scored against the BF16 GGUF they were quantized from, using llama.cpp's full-vocabulary KLD.

The text is Flint's own answers. Prompts come from four sources:

• code (Magicoder OSS-Instruct), 35%
• English chat (UltraChat 200k), 25%
• math reasoning (OpenR1-Math), 25%
• non-English chat (WildChat), 15%

The abliterated BF16 answered each one with Flint's recommended sampling (temperature 1.0, top_p 0.95, top_k 20). Each conversation was rendered through the chat template with its reasoning as a <think> block. KLD is counted on the assistant tokens, reasoning and answer, over 128 windows of 2048 tokens. The precision search used a separate draw and never saw this text.

Each build is compared with unsloth's dynamic GGUFs around the same size, from unsloth/flint-GGUF. Unsloth quantized the original Flint, so their files are scored against the original BF16, with the same text and llama.cpp build. Each KLD therefore measures the quantization error alone.

	File	KLD ↓	KLD p99 ↓	Top-1 ↑	PPL vs. BF16
unsloth UD-IQ1_M	11.35 GB	0.2203	3.05	86.76%	+20.4%
Mix-12G (this repo)	11.99 GB	0.2023	3.19	88.13%	+20.0%
unsloth UD-IQ2_XXS	11.86 GB	0.1961	2.77	87.55%	+17.8%
unsloth UD-Q2_K_XL	13.56 GB	0.1500	2.24	89.26%	+12.6%
unsloth UD-Q3_K_XL	18.49 GB	0.0632	1.04	93.36%	+5.0%
Mix-24G (this repo)	23.99 GB	0.01864	0.318	96.45%	+1.1%
unsloth UD-Q4_K_XL	25.05 GB	0.01871	0.302	96.30%	+1.1%
unsloth UD-Q5_K_XL	29.92 GB	0.00915	0.146	97.39%	+0.4%

• Mix-24G matches UD-Q4_K_XL in mean KLD and beats it in next-token agreement, in a file 1.1 GB smaller. Its tail (KLD p99) is 5% higher.
• Mix-12G has 8% less KLD than UD-IQ1_M and 3% more than UD-IQ2_XXS, with the highest next-token agreement of the three. Its tail is the weakest of the three.

Why the model's own text. Flint always reasons before it answers. On other models' answers (no reasoning block, someone else's style) a model makes confident misses even at high precision, and which of those misses a quant happens to flip dominates the KLD. So the score covers what the model itself writes.

Teachers. The abliterated BF16 is itself 0.015 KLD from the original on this text. Scored against the abliterated BF16 instead of its own, UD-IQ2_XXS reads 0.2005, so the comparison does not hinge on the choice of teacher.

Method note. These are full-vocabulary KLD and next-token agreement at 2K context, not a downstream benchmark.

Abliteration

These numbers were measured on the abliterated BF16 before quantization, against the original Flint, with identical prompts, splits and seed. Generation used Flint's official SGLang kit and sampling (temperature 1.0, top-p 0.95, top-k 20). Refusal is judged on the final answer by a rule-based classifier, not an LLM judge, so treat them as indicative.

Harmful-prompt refusal (lower = more uncensored). Average 10.4%, against 86.3% for the original.

Benchmark	n	Original	Abliterated
AdvBench	100	94.0%	13.0%
JailbreakBench (harmful)	100	84.0%	6.0%
StrongREJECT	150	95.3%	10.7%
HarmBench (standard)	150	79.3%	19.3%
MaliciousInstruct	100	93.0%	2.0%
SimpleSafetyTests	50	98.0%	14.0%
ForbiddenQuestions	150	60.0%	15.3%

Over-refusal on benign prompts (lower = better).

Benchmark	n	Original	Abliterated
XSTest-safe	250	2.8%	1.6%
JailbreakBench (benign)	100	6.0%	1.0%

Capability. Every delta is within ±2 pp, which is sampling noise at these sizes.

Benchmark	n	Original	Abliterated
MMLU	300	83.7%	83.3%
MMLU-Pro (CoT)	250	72.4%	73.2%
GSM8K (CoT)	150	92.0%	90.0%
CMMLU	500	67.2%	67.0%

---

How the builds were made

Where the bits go

The budget is the whole file: 12.0 GB and 24.0 GB.

1. Measure. For every group of tensors, the search measures the real KLD of moving it one precision level up and one level down. The 85 groups are:

   • routed experts: one projection (gate, up or down) over about two layers, 25 groups each
   • attention: q, k, v and output, one group each over all layers
   • shared expert: gate, up and down, one group each
   • dense FFN: layers 0–1
   • embedding and output head: one group each

   Each measurement runs llama.cpp's own kernels on 64 windows of 2048 tokens of the model's own text.

2. Choose. A knapsack picks the combination of moves that fits the budget exactly.

3. Accept. Only steps whose candidate KLD was actually measured lower are kept.

Mix-12G puts its bits as follows:

Part	Type
Routed experts gate	IQ2_XXS; IQ1_S on layers 2–5, IQ2_XS on 10–11, 19–22 and 42–43, IQ2_S on 40–41
Routed experts up	IQ2_XXS and IQ2_XS; IQ2_S on layers 19–20, 25–28 and 42–43, IQ1_S on 2–7, IQ3_XXS on 48
Routed experts down	IQ2_S on 18 layers, IQ2_XXS on 11, IQ2_XS on 8; IQ3_XXS on 12–13, 19–22 and 34–35, IQ3_S on 46–47
Attention q / k / v / output	Q4_K / Q5_K / Q5_K / Q4_K
Shared expert gate / up / down	Q4_K / Q5_K / Q6_K
Dense FFN (layers 0–1)	Q4_K
Token embedding / output head	Q3_K / Q6_K

Mix-24G puts its bits as follows:

Part	Type
Routed experts gate	IQ4_XS; Q5_K on layers 6–7, 16–17 and 34–35, IQ3_XXS on 4–5, IQ2_S on 2–3
Routed experts up	IQ4_XS; Q5_K on layers 10–17, 21–26, 34–35 and 48, IQ3_XXS on 2–3
Routed experts down	Q5_K on 29 layers, IQ4_XS on 18
Attention q / k / v / output	Q8_0
Shared expert gate / up / down	Q8_0 / Q6_K / Q8_0
Dense FFN (layers 0–1)	Q6_K
Token embedding / output head	Q5_K / Q8_0

In both builds the router, the norms and the attention sinks stay F32.

How each tensor is rounded

llama-quantize rounds each block of 256 weights on its own, weighted by the importance matrix. Rebuilt that way from the same BF16, with the same per-tensor types, unsloth's UD-IQ2_XXS scored 0.27 KLD against 0.196 for their own file, with either importance matrix. The gap was in the rounding, not in the allocation or the importance matrix.

So each tensor is rounded the GPTQ way instead. It is quantized one GGUF block of columns at a time, with llama.cpp's own quantizer for that type. Each block's rounding error is then pushed onto the columns not yet quantized, through the inverse Hessian of the layer's inputs, so later columns absorb what earlier ones lost. The output keeps stock types and layout, so any llama.cpp build reads it.

• The Hessians come from 424K tokens of Flint's own conversations, run through the BF16 model.
• A routed expert sees only about 2.3% of those tokens, so its Hessian is shrunk toward its layer's mean, with a prior of 2048 tokens. The prior and the damping (0.3) were tuned on held-out KLD.

With this rounding and the search rerun on top of it, the 12 GB build went from 0.262 to 0.202 KLD, and the 24 GB build from 0.0222 to 0.0186.

The importance matrix comes from ~1.2M tokens:

• the exllamav3 calibration mix: web text (C4), code, multilingual text, technical text, Wikipedia, tiny stories and random token rows
• Flint's own conversations, so that it sees the chat format and <think> blocks

None of it overlaps the evaluation text.

---

What's in this repo

Path	Size	Contents
Flint-abliterated-Mix-12G.gguf	11.99 GB	the 12 GB build
Flint-abliterated-Mix-24G.gguf	23.99 GB	the 24 GB build
imatrix.gguf	0.45 GB	the importance matrix the builds were quantized with
LICENSE		Apache 2.0, from the base model

Each build is a single file with the chat template embedded. These are text-only, like Flint.

---

Serving (llama.cpp)

Flint support landed in llama.cpp master with ggml-org/llama.cpp#30928, and its chat template needs #30929. These files were made and tested at master commit e0f142d:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout e0f142d   # or any later master
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server llama-cli
# Mac: cmake -B build (Metal is on by default) · CPU only: cmake -B build

hf download orcarouter/Flint-abliterated-GGUF Flint-abliterated-Mix-24G.gguf --local-dir ./flint-gguf

# one 32 GB GPU (Mix-24G), 128K context
llama-server -m flint-gguf/Flint-abliterated-Mix-24G.gguf -c 131072 --jinja

# one 16 GB GPU (Mix-12G), 32K context
llama-server -m flint-gguf/Flint-abliterated-Mix-12G.gguf -c 32768 --jinja

• Reasoning is always on. With --jinja, llama-server returns the reasoning in message.reasoning_content and the answer in message.content. Give it a large max_tokens: Flint's own card recommends at least 32768 for hard problems, reasoning included.
• Tool calls work through the OpenAI-compatible API with --jinja, and come back in message.tool_calls.
• Sampling. Recursive recommends temperature=1.0, top_p=0.95, top_k=20. The files carry these as their default sampling metadata.
• Mac. macOS lets the GPU wire only about 75% of RAM by default. Mix-24G on a 32 GB Mac needs the limit raised, for example sudo sysctl iogpu.wired_limit_mb=28000, and other apps closed.

---

Model details

Base model	RecursiveSI/flint, abliterated
Architecture	flint in llama.cpp. 49 layers: 2 dense + 47 MoE with 256 routed experts top-6 + 1 shared expert; hidden 2048, 32 query / 4 KV heads; 35 sliding-window (513) RoPE layers and 14 full-attention layers without positional encoding; learned attention sinks
Weights	39.8B: 37.8B routed experts, 2.0B attention / shared expert / dense / embeddings / head. About 2.5B active per token
Abliteration	One refusal direction (diff-of-means, hidden-state layer 17) orthogonalized out of the attention o_proj, the routed-expert down_proj and the token embedding, in BF16
Quantization	Per-group mixed precision found by measured-KLD search, from the abliterated BF16. GPTQ-style error-compensated rounding into stock llama.cpp types, with an importance matrix
Context	262,144 tokens (inherited; evaluated at 2K) · Vocabulary 150,144
Runtime	llama.cpp master (PR #30928 or later); text in, text out

---

Bias, risks, and limitations

• Safety guardrails are substantially reduced. See the disclaimer.
• Mix-12G is noticeably degraded. About 2.3 bits per expert weight is a large cut. It is the best we found at this size, not a match for the BF16.
• Refusal and capability were measured on the BF16, not on these files. The quantized builds were checked for fidelity (KLD) only.
• Not measured: coding and agentic benchmarks (Flint's card reports Terminal-Bench and SWE-bench; none of that was rerun here), and long context. All fidelity measurements are at 2K context.
• Tested on CUDA only. CPU and Apple Metal have not been run with these files.
• Inherits the biases and limitations of RecursiveSI/flint.
• Requires llama.cpp with PR #30928 and #30929. Tools built on older llama.cpp, such as older Ollama and LM Studio releases, will not load these files.

License

Apache 2.0, inherited from the base model RecursiveSI/flint. Abliteration and quantization do not change the license obligations.

---

One Gateway. Every Model.

Route Smarter · Ship Safer · Spend Less

Website · Models · GitHub · Discord · X