willyesiam/Qwen38Jrell-MTP-256K

qwen 38Jrell - mtp 256k

Overview

This repository is an Ollama deployment package for Qwen3.8 27B. It combines the jrell IQ4_XS-style main quantization with a separate Q4_0 MTP draft model and targets a 262,144-token (256K) context window.

This is a packaging and runtime-configuration project, not a new training run, fine-tune, merge, or official Qwen release. It does not claim ownership of the upstream model or quantizations.

The repository contains scripts, configuration, attribution, and documentation. It does not contain the large GGUF weights. install.ps1 downloads those artifacts from their canonical upstream repositories at install time, so this repository can be renamed and updated without re-uploading multi-GB files.

Files / Components

The current repository contains:

File	Purpose
install.ps1	Downloads the two upstream GGUF files and creates the Ollama model.
run.ps1	Runs qwen38-jrell-mtp-256k.
verify.ps1	Prints Ollama, process, GPU, and local-file diagnostics.
Modelfile.template	Exact local-path Modelfile structure used after installation.
ATTRIBUTION.md	Upstream sources, credits, and licensing notes.
SHA256SUMS.txt	Instructions for hashing locally downloaded artifacts.
.gitattributes	Git LFS rules for any large binaries added in the future.

No .gguf file is stored in this repository. The filenames used by the installer are:

• Main model, downloaded from jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller: Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf
• MTP draft, downloaded from unsloth/Qwen3.8-27B-GGUF: MTP/mtp-Qwen3.8-27B-Q4_0.gguf

The draft file is stored locally as mtp-Qwen3.8-27B-Q4_0.gguf under models/mtp/ by the installer.

Ollama Usage

On Windows, with Ollama on PATH:

powershell -ExecutionPolicy Bypass -File .\install.ps1
ollama run qwen38-jrell-mtp-256k

The installer skips non-empty local GGUF files, generates Modelfile.generated, runs ollama create, and calls ollama show for validation. It does not alter Ollama's internal blob store or set permanent environment variables.

The generated Modelfile uses the actual filenames downloaded by the installer:

FROM ./models/main/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf
DRAFT ./models/mtp/mtp-Qwen3.8-27B-Q4_0.gguf
PARAMETER num_ctx 262144
PARAMETER draft_num_predict 2

The paths are relative to the Modelfile. Do not replace them with Ollama's internal blob SHA paths.

MTP Configuration

• Main model: jrell's Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf IQ4_XS-style quantization.
• Draft model: Unsloth's MTP/mtp-Qwen3.8-27B-Q4_0.gguf Q4_0 MTP draft.
• Context target: num_ctx 262144.
• Draft setting: draft_num_predict 2.
• Ollama model name: qwen38-jrell-mtp-256k.

draft_num_predict controls the number of speculative draft tokens proposed per step when a draft model is available. Actual acceleration depends on the installed Ollama version, hardware, context length, quantization, and workload.

Benchmark

Runtime context:

• Ollama local inference.
• Qwen3.8 27B with the IQ4_XS main quantization.
• MTP draft enabled for the MTP configuration.
• Same prompt and generation length for the A/B comparison.
• num_predict = 512.

Configuration	Throughput	512-token time	Token latency
MTP OFF	41.02 tok/s	12.48 s	24.38 ms/token
MTP ON	54.62 tok/s	9.37 s	18.31 ms/token

Observed throughput speedup: approximately 1.33x.

Observed tok/s improvement: approximately 33.2%.

Observed generation-time reduction in this test: approximately 24.9%.

Benchmark Methodology / Limitations

This is a single-run matched 512-token A/B test, not a statistically rigorous benchmark. On this local Ollama configuration, the run measured 54.62 tok/s with MTP versus 41.02 tok/s without MTP. Results depend on hardware, Ollama version, context length, quantization, runtime configuration, and workload; the observed ratio is not a universal performance claim.

No MTP acceptance-rate data or multi-run median/mean is reported here. The earlier rough "effective TOPS" proxy is intentionally omitted: it was calculated from parameter count and tok/s, not measured GPU hardware TOPS.

Upstream Models

• Qwen/Qwen3.8-27B — base model and architecture.
• jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller — main quantization.
• unsloth/Qwen3.8-27B-GGUF — GGUF lineage and MTP draft source.

The upstream repositories remain the source of the model artifacts. This repository does not claim to have created or trained Qwen3.8.

License / Attribution

The original scripts and documentation in this repository are Apache-2.0 licensed; see LICENSE. The Qwen and Unsloth upstream repositories identify Apache-2.0 licensing in their metadata. The jrell repository does not explicitly state an independent license in its model-card metadata, so its main GGUF is downloaded from upstream and is not redistributed here. See ATTRIBUTION.md for detailed credits and terms.

This is not an official Qwen release and is not affiliated with Qwen, Alibaba Cloud, Unsloth, or jrell. The maintainer receives no revenue from this repository.

Notes

• A 256K context is memory intensive; lower -NumCtx in install.ps1 if the local system cannot load the full context.
• The installer accepts -NumCtx and -DraftNumPredict overrides.
• The current package is intended for local Ollama use on Windows and assumes stock Ollama behavior.
• Hash locally downloaded GGUF files with the commands in SHA256SUMS.txt if you need a reproducibility record.