Uncensored and NVMe offload Request
This looks very promising for single-GPU deployment, especially on 64GB Ampere cards like the CMP 170HX / A100 64GB.
I have two feature requests that I think would make this model much more useful for local coding workloads:
An uncensored / abliterated / low-refusal variant
My main use case is coding and agentic coding, so a weight-level uncensored version of this pruned W4A16 checkpoint would be extremely interesting. Ideally this would preserve the same expert-pruning profile and coding-oriented routing behavior, instead of starting from a separately quantized uncensored model.
NVMe mmap for the PLE / n-gram table
It would be great to have an option similar to the recent Flash-Next NVMe-PLE implementations, where the PLE table is stored as a sidecar file on local NVMe and accessed via mmap / page cache instead of requiring ~100GB of resident system RAM.
The combination I’m especially interested in is:
Qwen3.8-Flash-Next + coding-oriented expert pruning + W4A16 + uncensored weights + NVMe-mmap PLE
That could potentially make this a very strong fit for a single 64GB SM80 GPU, while preserving much more coding quality than aggressive 3-bit quantization.
If you’re interested in supporting this, even publishing the pruning/export scripts or some guidance on how to apply the same expert mask to an uncensored checkpoint would be very helpful.
Thanks for releasing this — the coding-traffic-based expert pruning approach is particularly interesting.
Your feedback is invaluable; I understand that implementation shouldn't be too difficult, so I'll proceed to create a version incorporating your suggestions. This may involve model reweighting/optimization or simply adjusting the inference engine accordingly.
Your feedback is invaluable; I understand that implementation shouldn't be too difficult, so I'll proceed to create a version incorporating your suggestions. This may involve model reweighting/optimization or simply adjusting the inference engine accordingly.
This is incredible, I can't express how grateful I am. Thank you.