sizzlebop/Nebula-320M-GGUF

Nebula (320M) GGUF

This repository provides GGUF quantizations for OrionLLM/Nebula.

Nebula is a 320M-parameter generalist Small Reasoning Model trained on 200B+ tokens, designed for edge AI, embedded systems, and on-device deployment. It is optimized for strong memory, general reasoning, arithmetic, and retrieval-friendly tasks while operating with compact compute and memory overhead.

All GGUF files were converted from the original safetensors weights using llama.cpp at native BF16 precision, then quantized into standard k-quant variants.

---

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
Nebula-320M-BF16.gguf	BF16	614.61 MB	Full precision base conversion. Highest fidelity reference weights.
Nebula-320M-Q8_0.gguf	Q8_0	327.74 MB	Near-lossless 8-bit quantization. Recommended for best quality.
Nebula-320M-Q6_K.gguf	Q6_K	311.39 MB	High quality retention with minimal degradation. Excellent balance.
Nebula-320M-Q5_K_M.gguf	Q5_K_M	244.94 MB	Solid balance between memory usage and generation accuracy.
Nebula-320M-Q4_K_M.gguf	Q4_K_M	228.60 MB	Fast, lightweight 4-bit quant. Recommended default for mobile and edge setups.
Nebula-320M-Q3_K_M.gguf	Q3_K_M	198.01 MB	Compact footprint when memory headroom is strictly limited.
Nebula-320M-Q2_K.gguf	Q2_K	185.35 MB	Maximum compression. Noticeable quality loss, intended for ultra-constrained environments.

---

Reasoning Format and Stenographic Tokens

Nebula produces reasoning traces with an intentionally compact, stenographic style designed for efficiency under limited parameter capacity. Traces use structured shorthand notations within <think>...</think> blocks:

Logical Markers

Token	Meaning	Usage
→	derivation / implication	Short causal and logical flow
↺	iterative return / refinement loop	Backtracking, reconsidering priors, RAG re-querying
?	uncertainty / questions to resolve	Appended to short expressions and words
!/※	insight / breakthrough	Marked knowledge discovery
≈	approximation / estimate	Intermediary hypothesis and preliminary statements
∴	therefore / final step	Stable conclusion marker

Uncertainty Markers

Token	Meaning	Usage
●	high confidence	Well-supported empirical or theoretical anchor points
◐	medium / partial confidence	Incomplete data, plausible but unverified links
○	low confidence	Speculation, missing context, weak inference chain
⚠	bias / premise risk	Domain mismatch, cultural assumptions, language-switch artifacts
?maybe?	soft speculation	Tentative branch that may collapse later

Verification Process

Token	Meaning	Usage
☐	unverified hypothesis	Raw claim without cross-check
☑	intermediate verification	Supported by at least one source or argument
✓	confirmed / validated	Multiple independent supports

---

Prompt Format

Nebula uses ChatML markup with generation prompts opening into <think> blocks:

<|im_start|>system
You are a helpful reasoning assistant.<|im_end|>
<|im_start|>user
Calculate the square root of 1764 and explain your steps.<|im_end|>
<|im_start|>assistant
<think>

---

How to Use

1. With llama.cpp

Run interactive generation with llama-cli:

llama-cli -m ./Nebula-320M-Q4_K_M.gguf \
    -p "<|im_start|>system\nYou are a helpful reasoning assistant.<|im_end|>\n<|im_start|>user\nWhat is the capital of France and why is it culturally significant?<|im_end|>\n<|im_start|>assistant\n<think>\n" \
    -n 512 \
    -c 4096 \
    --temp 0.6

Launch a local OpenAI-compatible API server:

llama-server -m ./Nebula-320M-Q4_K_M.gguf \
    --host 127.0.0.1 \
    --port 8080 \
    -c 4096

2. With Ollama

Create a file named Modelfile in the same directory:

FROM ./Nebula-320M-Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
<think>
"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|end_of_text|>"
PARAMETER temperature 0.6

Build and run the model:

ollama create nebula-320m -f Modelfile
ollama run nebula-320m

3. With LM Studio

1. Place the desired .gguf file into your local LM Studio models directory.
2. Select Nebula-320M in the model loader tab.
3. Use the ChatML preset as the prompt template.

---

Model Architecture Details

• Original weights: OrionLLM/Nebula
• Architecture: LlamaForCausalLM (model_type: llama)
• Parameters: ~320M
• Layers: 80
• Hidden size: 576
• Intermediate size: 1536
• Attention heads: 9 query heads, 3 key/value heads (GQA)
• Head dimension: 64
• Context window: 4,096 tokens
• Vocabulary size: 65,536 tokens
• Native precision: bfloat16
• License: Apache 2.0

---

Attribution & Credits

Original model training and architecture by OrionLLM.

GGUF conversions produced by Pink Pixel.

Made with 💖 by Pink Pixel