shr3y/Qwen3-0.6B-Agentic-Terminal-GGUF

🚀 Qwen3-0.6B Agentic Terminal (GGUF)

A highly optimized, ultra-lightweight (0.6B parameter) AI agent designed specifically for terminal automation and structured function calling.

This model was fine-tuned to bridge the gap between natural language and command-line execution, making it perfect for local homelabs, edge devices, and lightweight local assistants. It has been mathematically fused and quantized to Q4_K_M via llama.cpp for blazing-fast CPU inference.

🧠 Model Details

• Base Model: Qwen/Qwen3-0.6B
• Architecture: Qwen3 (Transformer)
• Parameter Count: 0.6 Billion
• Quantization Format: GGUF (Q4_K_M) - Ultra-fast, optimized for standard CPUs/RAM.
• Context Length: 40,960 tokens
• Author: @shr3y

🛠️ Training & Methodology

This model was trained using a custom Multi-GPU Distributed Data Parallel (DDP) pipeline via Hugging Face Accelerate.

• Fine-tuning method: LoRA (Rank 16, Alpha 32) targeting q_proj and v_proj.
• Datasets Mixed: 1. NousResearch/hermes-function-calling-v1 (For structured tool use/JSON outputs)
  2. nvidia/Nemotron-Terminal-Corpus (For multi-turn terminal CLI trajectories)
• Prompt Format: ChatML

💻 How to Use

Because this model is in the GGUF format, it can be run seamlessly using llama.cpp, LM Studio, Ollama, or GPT4All.

Option 1: CLI (llama.cpp)

# Download the model
wget [https://huggingface.co/shr3y/Qwen3-0.6B-Agentic-Terminal-GGUF/resolve/main/qwen3-agentic-terminal-q4_k_m.gguf](https://huggingface.co/shr3y/Qwen3-0.6B-Agentic-Terminal-GGUF/resolve/main/qwen3-agentic-terminal-q4_k_m.gguf)

# Run interactive chat with the ChatML template
./llama-cli -m qwen3-agentic-terminal-q4_k_m.gguf -p "<|im_start|>system\nYou are a helpful terminal agent.<|im_end|>\n<|im_start|>user\nList all files in the current directory.<|im_end|>\n<|im_start|>assistant\n" -n 512

Option 2: Python (llama-cpp-python)

from llama_cpp import Llama

# Load the model
llm = Llama(
  model_path="./qwen3-agentic-terminal-q4_k_m.gguf",  
  n_ctx=4096,
  n_threads=4,
)

# Generate a response
response = llm.create_chat_completion(
    messages = [
        {"role": "system", "content": "You are a specialized terminal automation agent. Provide exact CLI commands."},
        {"role": "user", "content": "How do I find all python files modified in the last 7 days?"}
    ]
)

print(response['choices'][0]['message']['content'])

📝 Prompt Template (ChatML)

This model natively uses the ChatML format. If you are writing custom integration code, ensure your prompts are wrapped accordingly:

<|im_start|>system
You are a helpful terminal agent.<|im_end|>
<|im_start|>user
Write a bash script to backup my documents.<|im_end|>
<|im_start|>assistant

⚠️ Limitations

While highly capable for its size, this is a 0.6B parameter model. It excels at targeted, specific terminal commands and function calling formats, but may struggle with highly abstract reasoning or complex, multi-stage logic puzzles compared to 7B+ models. Keep system prompts strict and focused for best results.