AgentCraft-GGUF

QLoRA fine-tuned Qwen2.5-Coder-7B-Instruct GGUF quantized to Q4_K_M, Q5_K_M, and Q8_0, trained for coding, analysis reasoning, and tool/function calling (terminal, filesystem, web search) in Qwen's native tool format.

Model Details

Attribute	Value
Base model	Qwen2.5-Coder-7B-Instruct
Format	GGUF (llama.cpp)
Quantizations	Q4_K_M, Q5_K_M, Q8_0
Size (Q4_K_M)	~4.36 GB
Size (Q5_K_M)	~5.07 GB
Size (Q8_0)	~7.10 GB
Parameters	7.6B (dense)
Context length	32,768 tokens
License	Apache-2.0
Hugging Face	https://huggingface.co/skbhati199/AgentCraft-GGUF

Available GGUF Files

File	Quantization	Size	Use Case
AgentCraft-Q4_K_M.gguf	Q4_K_M (4-bit)	4.36 GB	Best speed/quality balance; recommended for most users
AgentCraft-Q5_K_M.gguf	Q5_K_M (5-bit)	5.07 GB	Higher quality; good for coding/reasoning tasks
AgentCraft-Q8_0.gguf	Q8_0 (8-bit)	7.10 GB	Near-F16 quality; maximum fidelity

Quick Start

Ollama (Recommended)

# Pull and run (auto-selects best quantization)
ollama run skbhati199/AgentCraft-GGUF

# Or create from local GGUF
ollama create agentcraft -f Modelfile
ollama run agentcraft

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="AgentCraft-Q4_K_M.gguf",
    n_ctx=8192,
    n_threads=8,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are AgentCraft, an expert coding assistant."},
        {"role": "user", "content": "Write a Python function to fibonacci with memoization."}
    ],
    temperature=0.2,
    max_tokens=512,
)
print(response["choices"][0]["message"]["content"])

REST API (Ollama)

curl http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{
  "model": "agentcraft:latest",
  "messages": [
    {"role": "user", "content": "Explain JWT authentication in FastAPI"}
  ],
  "stream": false
}'

Training

• Method: QLoRA (4-bit NF4)
• LoRA config: r=32, alpha=64, dropout=0.05
• Learning rate: 2e-4
• Epochs: 1
• Effective batch: gradient accumulation
• Max sequence length: 8192
• Hardware: 2x T4 (DDP) on Kaggle

Training Data Mix

Skill	Source	Examples
Terminal / filesystem / web search	Synthetic trajectories	N/A
Function calling	NousResearch/hermes-function-calling-v1	N/A
Coding	ise-uiuc/Magicoder-OSS-Instruct-75K	N/A
Reasoning	HuggingFaceH4/Bespoke-Stratos-17k + GSM8K	N/A

Loss computed on assistant tokens only. Sequences longer than max length were dropped, not truncated.

Evaluation

Run the evaluation scripts in G:\my-llms\eval\:

# Basic chat test
python test_chat.py

# Tool calling test
python test_tools.py

# Code generation test
python test_code.py

# Full batch evaluation
python eval_batch.py

Benchmark Results (LoRA Adapter, Unmerged)

Benchmark	Metric	Score	95% CI	Samples
GSM8K	Accuracy (exact match, 0-shot)	83.3%	[66.4%, 92.7%]	30
HumanEval	Pass@1 (greedy)	90.0%	[76.9%, 96.0%]	40
MBPP (sanitized)	Pass@1 (greedy)	76.7%	[59.1%, 88.2%]	30
Tool-call Probe	Tool-call Accuracy	25.0%	[8.9%, 53.2%]	12

Evaluated with n_samples from config.json. Tool calling shows 0% valid JSON rate — needs investigation.

Evaluation Dataset Counts

Benchmark	Source	Total Available	Used in Eval
GSM8K	openai/gsm8k (main, test)	1,319	30
HumanEval	openai/openai_humaneval (test)	164	40
MBPP	google-research-datasets/mbpp (sanitized, test)	257	30
Tool-call Probe	Hand-written (9 tool-needed + 3 no-tool)	12	12

Limitations

• Much of the agent data is synthetic; real-world tool use may be weaker than benchmarks suggest.
• Always sandbox any tools you connect to the model.
• Q4_K_M quantization introduces minor quality loss vs F16; Q5_K_M and Q8_0 are higher fidelity.
• Tool calling benchmark shows 0% valid JSON rate — needs investigation and improvement.

Citation

@model{agentcraft-gguf,
  author = {skbhati199},
  title = {AgentCraft-GGUF},
  year = {2026},
  url = {https://huggingface.co/skbhati199/AgentCraft-GGUF}
}