sizzlebop/flame-27m-instruct-GGUF

flame-27m-instruct GGUF

This repository provides GGUF quantizations for ifx-pse-sys-ml/flame-27m-instruct.

flame-27m-instruct is a compact 27.1M-parameter English instruction-tuned model trained on FineWeb-Edu-dedup, Cosmopedia-v2, ClimbMix, and FineMath, fine-tuned on SmolTalk. At ~5x smaller than SmolLM-135M-Instruct, it serves as an ultra-lightweight decoder for edge devices, embedded environments, and rapid local experimentation.

All GGUF files were converted from the original safetensors weights using llama.cpp at native F16 precision, followed by quantization into standard k-quant variants.

---

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
flame-27m-instruct-F16.gguf	F16	52.16 MB	Full precision base conversion. Highest fidelity reference weights.
flame-27m-instruct-Q8_0.gguf	Q8_0	27.91 MB	Near-lossless 8-bit quantization. Recommended for best quality.
flame-27m-instruct-Q6_K.gguf	Q6_K	21.65 MB	High quality retention with minimal degradation. Excellent balance.
flame-27m-instruct-Q5_K_M.gguf	Q5_K_M	19.36 MB	Solid balance between memory usage and generation accuracy.
flame-27m-instruct-Q4_K_M.gguf	Q4_K_M	17.20 MB	Fast, lightweight 4-bit quant. Recommended default for edge setups.
flame-27m-instruct-Q3_K_M.gguf	Q3_K_M	14.85 MB	Ultra-compact footprint under 15 MB.
flame-27m-instruct-Q2_K.gguf	Q2_K	12.64 MB	Maximum compression (~12.6 MB) for severely constrained microcontrollers.

---

Architecture Specifications

• Parameters: 27.1M
• Layers: 8
• Hidden size: 512
• Attention heads: 8 (Query) / 2 (Key/Value) with Grouped Query Attention (GQA)
• Head dimension: 64
• Intermediate size: 1280 (SwiGLU)
• Query/Key Normalization: Per-head RMSNorm on Q and K projections
• Context length: 2048 tokens
• RoPE base frequency: 1,000,000 (NeoX style)
• Vocabulary: 12,000 (English ByteLevel BPE)
• Embeddings: Tied word embeddings (tie_word_embeddings: true)

---

Benchmarks

Accuracy (%) evaluated via lm-evaluation-harness 0.4:

Benchmark	Chance	flame-27m-instruct	SmolLM-135M-Instruct
HellaSwag	25.0	30.4	41.9
ARC-Easy	25.0	38.3	43.9
ARC-Challenge	25.0	24.0	27.4
PIQA	50.0	61.1	67.0
WinoGrande	50.0	50.6	51.3
OpenBookQA	25.0	28.6	33.6
CommonsenseQA	20.0	18.5	20.3
MMLU	25.0	26.3	24.4
Average	—	34.7	38.7

---

Prompt Format

flame-27m-instruct uses standard ChatML markup:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Tell me about the moon in one sentence.<|im_end|>
<|im_start|>assistant

---

How to Use

1. With llama.cpp

Run interactive generation with llama-cli:

llama-cli -m ./flame-27m-instruct-Q4_K_M.gguf \
    -p "<|im_start|>user\nTell me about the moon in one sentence.<|im_end|>\n<|im_start|>assistant\n" \
    -n 64 --temp 0.3

Launch a local HTTP server:

llama-server -m ./flame-27m-instruct-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 -c 2048

2. With Ollama

Create a Modelfile:

FROM ./flame-27m-instruct-Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.3

Then create and run:

ollama create flame-27m -f Modelfile
ollama run flame-27m

3. With LM Studio

1. Copy flame-27m-instruct-Q4_K_M.gguf into your LM Studio models directory.
2. Select ChatML as the prompt preset.
3. Load the model and begin testing.