esilva/SlopCoder-Mongo-0.5B-ONNX

SlopCoder-Mongo-0.5B · ONNX (CPU and DirectML GPU)

ONNX Runtime GenAI builds of esilva/SlopCoder-Mongo-0.5B, a compact MongoDB autocomplete / editor-rewrite model distilled from a DeepSeek Coder 6.7B teacher into Qwen2.5-Coder-0.5B. Prompt format, training data, and limitations are described in the main model card.

Folder	Hardware	Quantization	Size	Use it for
int4/	CPU	INT4 k-quant (MatMulNBits, block 32, accuracy level 4), lm_head in INT8	399 MB	CPU default: smallest and fastest
int8/	CPU	INT8 (MatMulNBits, block 32)	1.11 GB	slightly higher autocomplete quality on CPU
dml-fp16/	GPU (DirectML)	FP16	994 MB	GPU default: bf16 quality, lowest latency
dml-int4/	GPU (DirectML)	INT4 RTN on MatMul only (embeddings kept in FP16)	558 MB	GPUs with little free memory

Every folder is self-contained: model, genai_config.json (which selects the execution provider), tokenizer, slopstudio-model.json (declares hardware: cpu or gpu) and a copy of the license files. Exported with the ONNX Runtime GenAI 0.15.2 builder; CPU builds use the cpu execution provider, dml-* builds the dml provider with DirectML graph capture (all nodes on the GPU, so the INT4 build does not quantize the tied embeddings).

Slop Studio: Preferências → Autocomplete → Baixar modelo lists these four folders, downloads the chosen one and verifies every file against the hashes published here.

Evaluation

2,000-example isolated benchmark (IDE prompt contract, greedy). The bf16 model on the same examples: APT 3.57, rewrite intent 93.6%; on 120 handwritten free-form requests: 72. CPU: ONNX Runtime GenAI 0.15.2, AMD Ryzen 9 7900. GPU: ONNX Runtime GenAI WinML 0.15.2, DirectML, AMD Radeon RX 7800 XT, one model per process.

Variant	APT	Syntax valid	MongoDB valid	Rewrite intent	Free-form (120)	TTFT mean / p95	Tokens/s
int4 (CPU)	3.51	92.0%	91.5%	93.3%	67	215 / 341 ms	109
int8 (CPU)	3.65	90.4%	90.4%	93.9%	—	201 / 319 ms	86
dml-fp16 (GPU)	3.65	90.9%	90.8%	93.7%	72	113 / 133 ms	103
dml-int4 (GPU)	3.40	90.6%	90.2%	93.0%	70	151 / 177 ms	115

• Peak process RAM: 1.53 GB (int4), 2.07 GB (int8), ~0.6 GB for the DirectML builds (weights live in GPU memory, roughly the folder size; GPU memory was not measured by the benchmark).
• On GPU prefer dml-fp16: dml-int4 loses autocomplete accuracy (APT −0.17 vs bf16) without lowering latency.
• CPU int4 quantization hurts free-form requests more than the template benchmark suggests (67 vs 72).

Usage — CPU

hf download esilva/SlopCoder-Mongo-0.5B-ONNX --include "int4/*" --local-dir SlopCoder-Mongo-0.5B-ONNX
pip install onnxruntime-genai==0.15.2

import onnxruntime_genai as og

model = og.Model("SlopCoder-Mongo-0.5B-ONNX/int4")
tokenizer = og.Tokenizer(model)
STOP = {151643, 151645, 151659, 151660, 151661, 151662}  # endoftext, im_end, fim_prefix, fim_middle, fim_suffix, fim_pad

prefix = 'db.getCollection("orders").find({ status: "paid", total: { $gte: '
suffix = " } })"
prompt = "<|fim_prefix|>" + prefix + "<|fim_suffix|>" + suffix + "<|fim_middle|>"
ids = tokenizer.encode(prompt)

params = og.GeneratorParams(model)
params.set_search_options(do_sample=False, max_length=len(ids) + 32)
generator = og.Generator(model, params)
generator.append_tokens(ids)
out = []
while not generator.is_done():
    generator.generate_next_token()
    token = generator.get_next_tokens()[0]
    if token in STOP:
        break
    out.append(token)
print(tokenizer.decode(out) if out else "")  # decode([]) raises in 0.15.2

Usage — GPU (DirectML, Windows)

hf download esilva/SlopCoder-Mongo-0.5B-ONNX --include "dml-fp16/*" --local-dir SlopCoder-Mongo-0.5B-ONNX
pip install onnxruntime-genai-winml==0.15.2

Use a clean environment: if the CPU onnxruntime Python package is also importable, the WinML package may load its DLL and refuse DirectML. genai_config.json already selects DirectML, so the code is the same as on CPU:

import onnxruntime_genai as og

model = og.Model("SlopCoder-Mongo-0.5B-ONNX/dml-fp16")
tokenizer = og.Tokenizer(model)
STOP = {151643, 151645, 151659, 151660, 151661, 151662}

prompt = '<|fim_prefix|>db.getCollection("orders").find({ status: "paid", total: { $gte: <|fim_suffix|> } })<|fim_middle|>'
ids = tokenizer.encode(prompt)
params = og.GeneratorParams(model)
params.set_search_options(do_sample=False, max_length=len(ids) + 32)
generator = og.Generator(model, params)
generator.append_tokens(ids)
out = []
while not generator.is_done():
    generator.generate_next_token()
    token = generator.get_next_tokens()[0]
    if token in STOP:
        break
    out.append(token)
print(tokenizer.decode(out) if out else "")

Known issue (ONNX Runtime GenAI WinML 0.15.2): loading a second DirectML model in the same process after releasing the first one crashed the process in our tests; loading it in a new process works.

For the editor-context header and the rewrite prompt, see the main model card.

License

This is a quantized copy of a Derivative of DeepSeek Coder under the DeepSeek License Agreement.

• DeepSeek License Agreement — LICENSE: its use-based restrictions (paragraph 5 and Attachment A) apply and must be passed on to anyone you redistribute to, together with a copy of the agreement.
• Apache License 2.0 — LICENSE-APACHE-2.0, for the Qwen2.5-Coder-0.5B initial weights.
• Attribution details: NOTICE.md. Each variant folder carries the same three files.

---

Resumo em português

Versões ONNX Runtime GenAI do SlopCoder-Mongo-0.5B (autocomplete MongoDB e reescrita de código em PT-BR/EN):

• CPU: int4/ (399 MB, padrão, TTFT ~215 ms) e int8/ (1,11 GB).
• GPU DirectML: dml-fp16/ (994 MB, padrão na GPU, mesma qualidade do bf16, TTFT ~113 ms) e dml-int4/ (558 MB, perde precisão no autocomplete sem ganhar latência).
• No Slop Studio, Preferências → Autocomplete → Baixar modelo lista as quatro pastas; slopstudio-model.json declara o hardware.

O modelo foi destilado de um professor baseado no DeepSeek Coder 6.7B e por isso segue a DeepSeek License Agreement, com as restrições de uso do Anexo A, além da Apache-2.0 do Qwen2.5-Coder; cada pasta traz uma cópia das licenças. Formato de prompt e limitações: card principal esilva/SlopCoder-Mongo-0.5B.