litigerking/Qwen3.8-27B-OrcaRouter-NInfer

Qwen3.8-27B OrcaRouter — NInfer artifact

This repository provides a converted NInfer artifact for the OrcaRouter Qwen3.8-27B Uncensored checkpoint. It is intended for inference with the NInfer engine and is not a Transformers checkpoint.

Artifact

• File: qwen3_8_27b_orcarouter-v2.ninfer
• Size: approximately 18.2 GB
• Architecture: Qwen3.8-27B / Qwen3.5 multimodal architecture
• Context length: up to 262,144 tokens
• Quantization: NInfer groupwise integer recipe (Q4G64_F16S, Q5G64_F16S, Q6G64_F16S, and W8G32_F16S components)
• MTP tensors: preserved for speculative decoding
• Conversion device: NVIDIA A40, CUDA

Running with NInfer

git clone https://github.com/tctiger/ninfer-A40.git
cd ninfer-A40

ninfer-serve /models/qwen3_8_27b_orcarouter-v2.ninfer \
  --host 0.0.0.0 \
  --port 8080 \
  --max-context 262144 \
  --kv-dtype bf16 \
  --spec mtp \
  --draft-tokens 3 \
  --lm-head-draft

The server exposes an OpenAI-compatible API. Exact memory use and throughput depend on GPU, KV-cache type, batch size, prompt length, and sampling settings.

Provenance and license

This artifact is derived from the upstream OrcaRouter/Qwen3.8-27B-Uncensored checkpoint. Please review and comply with the upstream model license and usage requirements. The NInfer conversion does not change the upstream model's intended attribution or restrictions.

Limitations

This file can only be loaded by an NInfer build that supports the Qwen3.8 artifact format. It cannot be loaded directly with transformers or llama.cpp. Validate outputs and tool-use behavior in your own environment before production deployment.