qwen-agent-scripts — OpenAI-compatible local engine for Qwen3 code-agent models (Kaggle-ready)

A single-file, dependency-light inference engine (llm_engine.py) that wraps harshagowtham/qwen3-14b-code-agent (Qwen3-14B fine-tune) with:

• OpenAI-style API — llm.chat.completions.create(...) works in-process (streaming, tool calls, reasoning_content for Qwen3 thinking mode).
• OpenAI-compatible HTTP server — serve(llm, port=8000) exposes /v1/chat/completions and /v1/models; point the openai python client or any OpenAI SDK at http://localhost:8000/v1.
• Weight-surgery patch support — automatically detects and applies the companion patch repo harshagowtham/qwen3-14b-code-agent-wsurgery (byte-exact, torch-free safetensors splicing) before loading the model. Pass the patch repo directly as the model for one-liner usage.
• Auto 4-bit NF4 quantization when GPU VRAM < 24 GB (Kaggle T4/P4 friendly).
• LoRA adapter support via PEFT — e.g. the MCP tool-calling adapter harshagowtham/qwen3-14b-code-agent-mcp-lora.
• max_tokens auto-clamped to the model context (e.g. 50000 becomes ~31k on Qwen3-14B) so long generations never crash.

---

Kaggle setup (3 steps)

1. Create a Kaggle Dataset named qwen-agent-scripts containing this folder's files (upload llm_engine.py, requirements.txt, examples/, tests/). It will be mounted at /kaggle/input/qwen-agent-scripts. Shortcut: the same files live in the HF repo harshagowtham/qwen-agent-scripts — you can also git clone or snapshot_download it in the notebook.
2. Kaggle notebook: Settings, Accelerator: GPU T4 x2 (or P100). Internet must be ON (needed once, to download the weights).
3. First cell:

!pip -q install -U transformers accelerate peft bitsandbytes sentencepiece
!pip -q install -U huggingface_hub fastapi uvicorn

import sys
sys.path.append("/kaggle/input/qwen-agent-scripts")

from llm_engine import LocalLLM, serve

llm = LocalLLM(
    "harshagowtham/qwen3-14b-code-agent",                    # fine-tuned weights
    adapter="harshagowtham/qwen3-14b-code-agent-mcp-lora",   # MCP LoRA (private -> token)
    hf_token="hf_YOUR_TOKEN",                                # your HF token
    patch="auto",                                            # auto-apply ...-wsurgery patch
)

patch="auto" finds the sibling repo harshagowtham/qwen3-14b-code-agent-wsurgery, splices the surgical tensors into the downloaded safetensors shards (sha256-verified, resumable, ~1 min), and only then loads the model.

Prefer the patched model in one argument? This is equivalent:

llm = LocalLLM(
    "harshagowtham/qwen3-14b-code-agent-wsurgery",  # patch repo (manifest points at base)
    adapter="harshagowtham/qwen3-14b-code-agent-mcp-lora",
    hf_token="hf_YOUR_TOKEN",
)

In-process usage (no server)

resp = llm.chat.completions.create(
    model="qwen3-14b-code-agent",
    messages=[
        {"role": "system", "content": "You are an expert web developer."},
        {"role": "user", "content": "Build a complete single-file HTML page for a bakery."},
    ],
    max_tokens=50000,     # auto-clamped to the context window
    temperature=0.7,
)
print(resp.choices[0].message.content)
# Qwen3 thinking mode (on by default):
print(resp.choices[0].message.reasoning_content)   # the internal reasoning
print(resp.usage)                                   # prompt/completion tokens

Streaming

for chunk in llm.chat.completions.create(
        messages=[{"role": "user", "content": "Explain recursion in 3 levels."}],
        max_tokens=2048, stream=True, include_usage=True):
    d = chunk.choices[0].delta if chunk.choices else None
    if d and d.get("reasoning_content"):
        print("[think]", d["reasoning_content"], end="", flush=True)
    elif d and d.get("content"):
        print(d["content"], end="", flush=True)

Tool calling (Qwen3 native)

tools = [{
    "type": "function",
    "function": {
        "name": "run_command",
        "description": "Run a shell command in the sandbox and return stdout.",
        "parameters": {
            "type": "object",
            "properties": {"cmd": {"type": "string"}},
            "required": ["cmd"],
        },
    },
}]
resp = llm.chat.completions.create(
    messages=[{"role": "user", "content": "List the files in /tmp using ls."}],
    tools=tools, max_tokens=2048,
)
tc = resp.choices[0].message.tool_calls
print(tc[0].function.name, tc[0].function.arguments)

Disable thinking (faster answers)

resp = llm.chat.completions.create(
    messages=[{"role": "user", "content": "2+2?"}],
    max_tokens=256, enable_thinking=False,
)

Serve an OpenAI-compatible API

import threading
threading.Thread(target=lambda: serve(llm, port=8000), daemon=True).start()

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-noop")
resp = client.chat.completions.create(
    model="qwen3-14b-code-agent",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    max_tokens=200,
)
print(resp.choices[0].message.content)

curl works too:

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"hi"}],"max_tokens":64}'

The Linux Mastery website test (the benchmark this engine was tuned for)

See examples/02_linux_mastery_site.py — asks the model for a complete multi-page Linux Mastery website (HTML/CSS/JS) with max_tokens=50000 (clamped automatically). Run on Kaggle:

!python /kaggle/input/qwen-agent-scripts/examples/02_linux_mastery_site.py

What the wsurgery patch does

The fine-tune moved 280 projection tensors uniformly (~0.3-0.7% relative energy) — a light, LoRA-style merge with no single dominant direction. The surgery keeps that learned delta where it is useful (deep-layer residual writers) but trims its magnitude toward the base model to reduce drift:

W_surgery = W_base + 0.70 * (W_ft - W_base) on the top-8 o_proj and top-5 down_proj tensors (13 tensors, 1.31 GB, chosen by delta energy).

The patch is 13 raw bf16 tensor files + wsurgery/manifest.json. Because patched tensors have identical byte length, they are spliced directly into the safetensors shards on disk — no torch, no re-serialization, sha256 verified both directions. Full details in harshagowtham/qwen3-14b-code-agent-wsurgery.

Repository layout

llm_engine.py        # the engine (single file, imports clean without torch)
requirements.txt
examples/            # quickstart, Linux Mastery site, OpenAI client, tools
tests/               # offline tests (no GPU, no torch, no network)
kaggle_notebooks/    # ready-made Kaggle notebooks (run engine / build v2 weights)

Offline tests / CLI

python llm_engine.py selftest                  # parsing, types, patch logic
python tests/test_offline_plumbing.py          # full offline suite
python llm_engine.py info --patch-repo harshagowtham/qwen3-14b-code-agent-wsurgery
python llm_engine.py apply-patch --model-dir /path/to/snapshot \
    --patch-repo harshagowtham/qwen3-14b-code-agent-wsurgery

VRAM cheat-sheet (Qwen3-14B)

Mode	VRAM	Quality
4-bit NF4 (auto on <24 GB GPUs, e.g. T4)	~9-10 GB	slightly lossy
bf16 (24 GB+ GPUs, e.g. A100/L4 x2)	~28 GB	full

Notes: with device_map="auto" on T4 x2 the 4-bit model splits across both GPUs. Context is 32k (native); max_tokens is clamped to context - prompt_len - 8.

License

Apache-2.0 (code). Model weights follow their upstream licenses (Qwen3: Apache-2.0; the fine-tune/patch/adapter: same).