ramankrishna10/npc-agentic-7b-v3-gguf

NPC Agentic 7B — GGUF

GGUF quants of ramankrishna10/npc-agentic-7b-v3 for llama.cpp / Ollama / LM Studio / local CPU+GPU inference.

See the FP16 reference card for the full training recipe, eval numbers, and known limitations (notably a GSM8K regression vs base — use base Qwen2.5 or Qwen2.5-Math-7B for math-heavy workflows).

Files

File	Quant	Size	Use case
npc-agentic-7b-Q4_K_M.gguf	Q4_K_M	~4.4 GB	default for Ollama / laptop CPU+GPU
npc-agentic-7b-Q5_K_M.gguf	Q5_K_M	~5.1 GB	higher-fidelity local inference
npc-agentic-7b-Q8_0.gguf	Q8_0	~7.7 GB	near-fp16 quality, consumer-GPU friendly

Build by llama.cpp's convert_hf_to_gguf.py + llama-quantize.

Inference

llama.cpp

./llama-cli -m npc-agentic-7b-Q4_K_M.gguf \
    -p "Design an event-sourced microservice with exactly-once command handling." \
    -n 1024 --temp 0.7 --top-p 0.9

Ollama

# Pull the Q4_K_M quant into a local Ollama modelfile
echo "FROM ./npc-agentic-7b-Q4_K_M.gguf" > Modelfile
ollama create npc-agentic:7b -f Modelfile
ollama run npc-agentic:7b "Explain photosynthesis step by step."

LM Studio / Jan / Koboldcpp

Drop any of the .gguf files into the app's model directory. Use chat template: Qwen2 / ChatML (<|im_start|> / <|im_end|>).

See also

• ramankrishna10/npc-agentic-7b-v3 — FP16 reference
• ramankrishna10/npc-agentic-7b-v3-gptq-4bit — GPTQ 4-bit for vLLM
• ramankrishna10/npc-agentic-7b-v3-lora — LoRA adapter

---

Built by Bottensor.

Citation

If you use NPC Agentic 7B in your work, please cite:

@misc{bachu2026npcagentic7b,
  title        = {NPC Agentic 7B: A Single-GPU QLoRA Recipe for a Laptop-Scale Conversational Model},
  author       = {Bachu, Rama Krishna},
  year         = {2026},
  month        = may,
  publisher    = {Zenodo},
  version      = {v1},
  doi          = {10.5281/zenodo.19954103},
  url          = {https://doi.org/10.5281/zenodo.19954103},
  note         = {Preprint}
}

Paper: https://doi.org/10.5281/zenodo.19954103