Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud (GGUF)

繁體中文 | English below

由云碩科技(xCloudinfo)以 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 為基礎,移除其過度拒絕傾向後所產生的語言模型,並轉為llama.cpp可用的GGUF量化格式。 本模型在云碩自有AI算力資源池(云碩算力中心)上完成處理與量化。

這是什麼

  • 基礎模型:Nemotron 3.5 Lightning 30B-A3B(混合架構:Mamba-2 + MoE + Attention,MoE總量30B、啟用約3B, 含Multi-Token Prediction,上下文最長1M,授權OpenMDW-1.1,可商用)。
  • 處理方式:方向消融(directional ablation / abliteration),非重新訓練。依Arditi et al. (2024), 將「拒絕方向」從殘差寫入矩陣(attention的 o_proj 與所有MoE專家的 down_proj,共2,973個矩陣)正交化移除, 強度0.8。Mamba(SSM) 的 out_proj 與MTP草稿頭保持原狀,以維持混合架構的連貫性。
    • 註:混合/SSM模型收「拒絕方向」時必須單一裝置載入,跨多卡切分會破壞SSM遞迴而得到壞方向;本模型於單機統一記憶體上處理。

版本對照

量化 檔案大小 說明
Q8_0 33 GB 近乎無損
Q6_K 33 GB 高品質(見下方註)
Q5_K_M 26 GB 品質與體積平衡
Q4_K_M 24 GB 一般部署建議
IQ4_XS 18 GB 以importance matrix量化
IQ2_M 18 GB 最小可用,以importance matrix量化(見下方註)

另附 imatrix.dat(量化用的importance matrix)。

註(MoE量化特性):本模型hidden維度為2688,非256的整數倍,部分專家張量在量化時會回退到較高位元, 因此Q8_0與Q6_K、IQ4_XS與IQ2_M的體積相近。各版本皆可正常使用;追求最小體積請選IQ4_XS或IQ2_M, 追求品質請選Q8_0或Q6_K。

使用方式(llama.cpp)

llama-server -m Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-Q4_K_M.gguf \
  --jinja -ngl 99 -c 8192
  • 思考預算:本模型的reasoning(thinking)預設開啟,且屬「預算問題」而非開關問題。要求事實正確的回答時, 請給足 max_tokens(建議 ≥ 2000),思考才收斂、答案才會出現;避免使用 medium_effort(思考更長且不收斂)。
  • 注入當前日期:在system prompt明確給出今天日期,否則模型的年份會預設為較早的年份。

授權與責任

  • 授權:OpenMDW-1.1(沿用基礎模型;Linux Foundation制定,可商用,散布時保留授權書與著作權聲明,對產出無限制)。
  • 本模型移除了安全對齊層的拒絕行為,可能對敏感或雙用途請求直接作答。使用者須自行負責合法、合規、合乎倫理地使用本模型。 云碩不對本模型的輸出或其後續使用承擔責任。
  • 本模型為內部研發/技術驗證用途。

關於云碩科技

云碩科技(xCloudinfo Corp. Limited)專注於地端私有化人工智慧解決方案。xVITA是云碩自研模型的品牌名稱。

其他模型請見 https://hf.co/xCloudinfo。


Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud (GGUF) — English

A language model produced by xCloudinfo, based on nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, with its over-refusal behaviour removed, converted to llama.cpp GGUF quantizations. Processing and quantization were performed on xCloudinfo's own AI compute pool.

What this is

  • Base model: Nemotron 3.5 Lightning 30B-A3B (hybrid: Mamba-2 + MoE + Attention, 30B total / ~3B active, with Multi-Token Prediction, up to 1M context, OpenMDW-1.1 license, commercial use permitted).
  • Method: directional ablation (abliteration), not retraining. Following Arditi et al. (2024), the refusal direction is orthogonalized out of the residual-writing matrices (attention o_proj and every MoE expert's down_proj; 2,973 matrices) at strength 0.8. The Mamba (SSM) out_proj and the MTP draft head are left intact to preserve the coherence of the hybrid architecture.
    • Note: for hybrid/SSM models the refusal direction must be collected with the model on a single device; splitting across GPUs corrupts the SSM recurrence and yields a bad direction. This model was processed on a single unified-memory device.

Versions

Quant Size Notes
Q8_0 33 GB near-lossless
Q6_K 33 GB high quality (see note)
Q5_K_M 26 GB quality/size balance
Q4_K_M 24 GB recommended for deployment
IQ4_XS 18 GB importance-matrix quantized
IQ2_M 18 GB smallest usable, importance-matrix quantized (see note)

Also included: imatrix.dat.

Note (MoE quantization): this model's hidden size is 2688, not a multiple of 256, so some expert tensors fall back to higher bit-widths during quantization. As a result Q8_0/Q6_K and IQ4_XS/IQ2_M are close in size. All variants work normally; pick IQ4_XS/IQ2_M for the smallest footprint, Q8_0/Q6_K for the highest quality.

Usage (llama.cpp)

llama-server -m Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-Q4_K_M.gguf \
  --jinja -ngl 99 -c 8192
  • Thinking budget: reasoning is on by default and is a budget matter, not an on/off switch. For factually correct answers, give enough max_tokens (>= 2000) so the thinking converges and the answer appears; avoid medium_effort (longer, non-converging thinking).
  • Inject the current date in the system prompt; otherwise the model defaults the year to an earlier value.

License and responsibility

  • License: OpenMDW-1.1 (inherited from the base model; Linux Foundation, commercial use permitted, keep the license and copyright notice when redistributing, no restrictions on outputs).
  • This model has had its safety-alignment refusal behaviour removed and may respond directly to sensitive or dual-use requests. Users are solely responsible for using it lawfully, in compliance, and ethically. xCloudinfo assumes no responsibility for the outputs or their downstream use.
  • Released for internal research and technical validation.

About xCloudinfo

xCloudinfo Corp. Limited (云碩科技) builds on-premises, private AI solutions. xVITA is xCloudinfo's in-house model brand.

More models: https://hf.co/xCloudinfo

Downloads last month
1,146
GGUF
Model size
33B params
Architecture
nemotron_h_moe
Hardware compatibility
Log In to add your hardware

2-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF

Quantized
(128)
this model

Collection including xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF