Instructions to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M # Run inference directly in the terminal: llama cli -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M # Run inference directly in the terminal: llama cli -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
Use Docker
docker model run hf.co/xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
- Ollama
How to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with Ollama:
ollama run hf.co/xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
- Unsloth Desktop
- Pi
How to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with Docker Model Runner:
docker model run hf.co/xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
- Lemonade
How to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
Run and chat with the model
lemonade run user.Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF-UD-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "xCloudinfo/Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-GGUF:UD-Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud (GGUF)
繁體中文 | English below
由云碩科技(xCloudinfo)以 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 為基礎,移除其過度拒絕傾向後所產生的語言模型,並轉為llama.cpp可用的GGUF量化格式。 本模型在云碩自有AI算力資源池(云碩算力中心)上完成處理與量化。
這是什麼
- 基礎模型:Nemotron 3.5 Lightning 30B-A3B(混合架構:Mamba-2 + MoE + Attention,MoE總量30B、啟用約3B, 含Multi-Token Prediction,上下文最長1M,授權OpenMDW-1.1,可商用)。
- 處理方式:方向消融(directional ablation / abliteration),非重新訓練。依Arditi et al. (2024),
將「拒絕方向」從殘差寫入矩陣(attention的
o_proj與所有MoE專家的down_proj,共2,973個矩陣)正交化移除, 強度0.8。Mamba(SSM) 的out_proj與MTP草稿頭保持原狀,以維持混合架構的連貫性。- 註:混合/SSM模型收「拒絕方向」時必須單一裝置載入,跨多卡切分會破壞SSM遞迴而得到壞方向;本模型於單機統一記憶體上處理。
版本對照
| 量化 | 檔案大小 | 說明 |
|---|---|---|
| Q8_0 | 33 GB | 近乎無損 |
| Q6_K | 33 GB | 高品質(見下方註) |
| Q5_K_M | 26 GB | 品質與體積平衡 |
| Q4_K_M | 24 GB | 一般部署建議 |
| IQ4_XS | 18 GB | 以importance matrix量化 |
| IQ2_M | 18 GB | 最小可用,以importance matrix量化(見下方註) |
另附 imatrix.dat(量化用的importance matrix)。
註(MoE量化特性):本模型hidden維度為2688,非256的整數倍,部分專家張量在量化時會回退到較高位元, 因此Q8_0與Q6_K、IQ4_XS與IQ2_M的體積相近。各版本皆可正常使用;追求最小體積請選IQ4_XS或IQ2_M, 追求品質請選Q8_0或Q6_K。
使用方式(llama.cpp)
llama-server -m Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-Q4_K_M.gguf \
--jinja -ngl 99 -c 8192
- 思考預算:本模型的reasoning(thinking)預設開啟,且屬「預算問題」而非開關問題。要求事實正確的回答時,
請給足
max_tokens(建議 ≥ 2000),思考才收斂、答案才會出現;避免使用medium_effort(思考更長且不收斂)。 - 注入當前日期:在system prompt明確給出今天日期,否則模型的年份會預設為較早的年份。
授權與責任
- 授權:OpenMDW-1.1(沿用基礎模型;Linux Foundation制定,可商用,散布時保留授權書與著作權聲明,對產出無限制)。
- 本模型移除了安全對齊層的拒絕行為,可能對敏感或雙用途請求直接作答。使用者須自行負責合法、合規、合乎倫理地使用本模型。 云碩不對本模型的輸出或其後續使用承擔責任。
- 本模型為內部研發/技術驗證用途。
關於云碩科技
云碩科技(xCloudinfo Corp. Limited)專注於地端私有化人工智慧解決方案。xVITA是云碩自研模型的品牌名稱。
其他模型請見 https://hf.co/xCloudinfo。
Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud (GGUF) — English
A language model produced by xCloudinfo, based on nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, with its over-refusal behaviour removed, converted to llama.cpp GGUF quantizations. Processing and quantization were performed on xCloudinfo's own AI compute pool.
What this is
- Base model: Nemotron 3.5 Lightning 30B-A3B (hybrid: Mamba-2 + MoE + Attention, 30B total / ~3B active, with Multi-Token Prediction, up to 1M context, OpenMDW-1.1 license, commercial use permitted).
- Method: directional ablation (abliteration), not retraining. Following Arditi et al. (2024), the refusal
direction is orthogonalized out of the residual-writing matrices (attention
o_projand every MoE expert'sdown_proj; 2,973 matrices) at strength 0.8. The Mamba (SSM)out_projand the MTP draft head are left intact to preserve the coherence of the hybrid architecture.- Note: for hybrid/SSM models the refusal direction must be collected with the model on a single device; splitting across GPUs corrupts the SSM recurrence and yields a bad direction. This model was processed on a single unified-memory device.
Versions
| Quant | Size | Notes |
|---|---|---|
| Q8_0 | 33 GB | near-lossless |
| Q6_K | 33 GB | high quality (see note) |
| Q5_K_M | 26 GB | quality/size balance |
| Q4_K_M | 24 GB | recommended for deployment |
| IQ4_XS | 18 GB | importance-matrix quantized |
| IQ2_M | 18 GB | smallest usable, importance-matrix quantized (see note) |
Also included: imatrix.dat.
Note (MoE quantization): this model's hidden size is 2688, not a multiple of 256, so some expert tensors fall back to higher bit-widths during quantization. As a result Q8_0/Q6_K and IQ4_XS/IQ2_M are close in size. All variants work normally; pick IQ4_XS/IQ2_M for the smallest footprint, Q8_0/Q6_K for the highest quality.
Usage (llama.cpp)
llama-server -m Nemotron-3.5-Lightning-30B-A3B-Uncensored-xCloud-Q4_K_M.gguf \
--jinja -ngl 99 -c 8192
- Thinking budget: reasoning is on by default and is a budget matter, not an on/off switch. For factually
correct answers, give enough
max_tokens(>= 2000) so the thinking converges and the answer appears; avoidmedium_effort(longer, non-converging thinking). - Inject the current date in the system prompt; otherwise the model defaults the year to an earlier value.
License and responsibility
- License: OpenMDW-1.1 (inherited from the base model; Linux Foundation, commercial use permitted, keep the license and copyright notice when redistributing, no restrictions on outputs).
- This model has had its safety-alignment refusal behaviour removed and may respond directly to sensitive or dual-use requests. Users are solely responsible for using it lawfully, in compliance, and ethically. xCloudinfo assumes no responsibility for the outputs or their downstream use.
- Released for internal research and technical validation.
About xCloudinfo
xCloudinfo Corp. Limited (云碩科技) builds on-premises, private AI solutions. xVITA is xCloudinfo's in-house model brand.
More models: https://hf.co/xCloudinfo
- Downloads last month
- 1,146
2-bit
4-bit
5-bit
6-bit
8-bit