XORTRON-CriminalComputing-RICO-v4-W4A16

4-bit weight-only (W4A16) quantization of darkc0de/XORTRON-CriminalComputing-RICO-v4, produced with LLM Compressor in the compressed-tensors format. vLLM loads it directly, so no --quantization flag is needed.

The goal is faster cold starts and a smaller memory footprint for serverless deployments (for example RunPod): roughly 15-20 GB on disk instead of ~52 GB for the bf16 original.

Quantization details

Item	Value
Method	<GPTQ / RTN>, scheme W4A16 (4-bit weights, 16-bit activations)
Tool	llmcompressor (oneshot, sequential pipeline)
Format	compressed-tensors
Quantized	Linear layers of the language model
Kept in original precision	lm_head, vision tower (visual), mtp, <linear_attn.in_proj_ba / all linear_attn if CONSERVATIVE>
Calibration data	UltraChat 200k (96 samples), OpenThoughts-114k (288 samples, thought tags mapped to <think>)
Calibration size	<384> samples, max sequence length <2048 / 4096>
Chat template	applied to every calibration sample
Hardware	<RTX PRO 6000 96GB>

The vision tower is not quantized, so image input keeps the base model's behavior. Calibration used text only.

Usage

vLLM

vllm serve ngloc2411/XORTRON-CriminalComputing-RICO-v4-W4A16 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="ngloc2411/XORTRON-CriminalComputing-RICO-v4-W4A16",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=512,
)
print(r.choices[0].message.content)

RunPod serverless (worker-vllm) environment variables

MODEL_NAME=ngloc2411/XORTRON-CriminalComputing-RICO-v4-W4A16
MAX_MODEL_LEN=32768
GPU_MEMORY_UTILIZATION=0.90
REASONING_PARSER=qwen3
TOOL_CALL_PARSER=qwen3_coder
ENABLE_AUTO_TOOL_CHOICE=true
HF_HOME=/runpod-volume/huggingface-cache

Leave QUANTIZATION empty; vLLM reads it from the model config.

Notes

• The chat template accepts reasoning_effort values low, medium and xhigh (default). The value high is rejected with HTTP 400.
• Sampling defaults come from generation_config.json (temperature 1.0, top_k 20, top_p 0.95).

Evaluation

Quantization lowers quality slightly compared with the bf16 original. <Fill in after running the evaluation, or delete this section.>

Benchmark	Base (bf16)	W4A16
GSM8K	<...>	<...>
ARC-Challenge	<...>	<...>
HellaSwag	<...>	<...>
WikiText perplexity	<...>	<...>

Limitations and responsible use

• This model inherits all behavior, biases and limitations of the base model, plus small additional errors from 4-bit quantization (weaker on long reasoning chains, tool calls and low-resource languages).
• The base model is described by its author as having reduced safety restrictions. It may produce harmful, inaccurate or inappropriate content. Review the base model card, apply your own safeguards, and use it only in compliance with applicable laws and policies.
• No guarantees of accuracy, safety or fitness for any purpose are provided.

License

This quantized model is a derivative work and is subject to the license and usage terms of the base model darkc0de/XORTRON-CriminalComputing-RICO-v4 and of the model it was built from. <Replace license: other in the metadata with the exact license of the base model, for example apache-2.0.>

Credits

• Base model: darkc0de
• Quantization: ngloc2411, using LLM Compressor