where my ggufs at
where are the ggufs
unsloth extremely busy atm: qwen and now glm 🤤
son
Have patience, it's not that simple for sloths to make GGUFs. Have you seen their claws? Imagine handling a keyboard with that!
their probably uploading brother lol
QWEN3.8-FLASH-NEXT & GLM-5.3-FLASH，unsloth is swamped😁
This architecture isn’t supported in llama.cpp yet, I think?
Please correct me if i'm wrong. While GLM 5.3 architecture is the same as 5.2. This one I think is different...? or maybe same architecture as 5.2 just with vision layer
Have patience, it's not that simple for sloths to make GGUFs. Have you seen their claws? Imagine handling a keyboard with that!
sloths are extremely slow so that explains it /j
Have patience, it's not that simple for sloths to make GGUFs. Have you seen their claws? Imagine handling a keyboard with that!
scratch and sniff
now there's iq1_m gguf
Hey guys, the GGUFs are out now and work inside of Unsloth Desktop! Feel free to support our X Tweet or [Reddit post[(https://www.reddit.com/r/unsloth/comments/1vzw55e/glm53_flash_unsloth_dynamic_ggufs/)
Run 3-bit on 128GB RAM or 1-bit on 100GB. The bigger ones are still uploading.
GLM-5.3-Flash (ox-alpha) rivals Claude Opus 4.8 on DeepSWE, coding & agentic benchmarks.
Unsloth GitHub: https://github.com/unslothai/unsloth
Guide: https://unsloth.ai/docs/models/glm-5.3-flash
Thank you for your support!! ❤️❤️
CC: @GGUFGuy @scorpionshoes @ziyy1337 @AutisticPancake @realrebelai @Williamxie1989 @MB7977 @okoku @RobAgrees @raunoryys @codehappy @auf1r2 @ronedgecomb @AshraGaunt @elainedaniels31 @hamsteri @xcywzy @sosiristseng @dzur658 @ciprian-cimpan @wingx @ZoneOverreach @peasantsmith @Nismo @chweber25 @gwillen @Banaxi-Tech @shrimpfriedrice-bricks @dany404 @CassiterFlux @jj4820
Can you try making a quant that works on 96GB mac? I would really like that
Can you try making a quant that works on 96GB mac? I would really like that
Unsloth are true GGUF magicians, so maybe a IQ1_XXS would fit.
But is it really useful?
You'll hardly have any space left for actual context, and going lower than IQ1_S will probably severely impact its capabilities as well...
I think you're better off running Qwen3.8 27B in Q6 with full context, it'll probably be smarter.
True or 2 bit Qwen3.8 Flash Next
They could make a TQ1_0
Thanks!
Hmm... Unsloth Studio got updated, as well as llama.cpp, but it doesn't seem Q4K_XL can load properly for me (256GB RAM, 2x24GB VRAM) at any settings, including "Auto".
If I change 2x GPU to 1x GPU, it keeps loading for longer but I see a huge pool of "Shared Memory" appearing (about 159GB) in Windows Task Manager, which is not a good sign.
Anyone else encountering this?
CASE 1 (single GPU with "Auto" GPU memory, NO speculative decoding, NO vision):
0.00.096.071 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.096.392 W srv llama_server: -----------------
0.00.096.394 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.096.395 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.096.395 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.096.395 W srv llama_server: -----------------
0.00.111.672 I srv load_model: loading model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
0.00.721.279 E llama_init_from_model: failed to initialize the context: glm5next: the pooled indexer needs one sequence per stream, so a unified KV cache is only supported with a single sequence
0.00.765.163 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to create llama_context from model
0.01.215.838 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
0.01.215.845 W load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
0.01.262.850 W model has unused tensor blk.45.attn_norm.weight (size = 16384 bytes) -- ignoring
0.01.262.866 W model has unused tensor blk.45.ffn_norm.weight (size = 16384 bytes) -- ignoring
0.01.262.876 W model has unused tensor blk.45.attn_q_a.weight (size = 6684672 bytes) -- ignoring
0.01.262.887 W model has unused tensor blk.45.attn_q_a_norm.weight (size = 6144 bytes) -- ignoring
0.01.262.897 W model has unused tensor blk.45.attn_q_b.weight (size = 26738688 bytes) -- ignoring
0.01.262.907 W model has unused tensor blk.45.attn_kv_a_mqa.weight (size = 2228224 bytes) -- ignoring
0.01.262.918 W model has unused tensor blk.45.attn_kv_a_norm.weight (size = 2048 bytes) -- ignoring
0.01.262.928 W model has unused tensor blk.45.attn_k_b.weight (size = 8912896 bytes) -- ignoring
0.01.262.939 W model has unused tensor blk.45.attn_v_b.weight (size = 8912896 bytes) -- ignoring
0.01.262.949 W model has unused tensor blk.45.attn_output.weight (size = 71303168 bytes) -- ignoring
0.01.262.962 W model has unused tensor blk.45.indexer.k_norm.weight (size = 512 bytes) -- ignoring
0.01.262.973 W model has unused tensor blk.45.indexer.k_norm.bias (size = 512 bytes) -- ignoring
0.01.262.986 W model has unused tensor blk.45.indexer.proj.weight (size = 524288 bytes) -- ignoring
0.01.262.996 W model has unused tensor blk.45.indexer.attn_k.weight (size = 557056 bytes) -- ignoring
0.01.263.008 W model has unused tensor blk.45.indexer.attn_q_b.weight (size = 6684672 bytes) -- ignoring
0.01.263.020 W model has unused tensor blk.45.indexer_compressor_gate.weight (size = 557056 bytes) -- ignoring
0.01.263.033 W model has unused tensor blk.45.indexer_compressor_ape.weight (size = 2048 bytes) -- ignoring
0.01.263.043 W model has unused tensor blk.45.ffn_gate_inp.weight (size = 4718592 bytes) -- ignoring
0.01.263.054 W model has unused tensor blk.45.exp_probs_b.bias (size = 1152 bytes) -- ignoring
0.01.263.065 W model has unused tensor blk.45.ffn_gate_exps.weight (size = 1358954496 bytes) -- ignoring
0.01.263.076 W model has unused tensor blk.45.ffn_up_exps.weight (size = 1358954496 bytes) -- ignoring
0.01.263.089 W model has unused tensor blk.45.ffn_down_exps.weight (size = 1660944384 bytes) -- ignoring
0.01.263.100 W model has unused tensor blk.45.ffn_gate_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.263.111 W model has unused tensor blk.45.ffn_up_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.263.122 W model has unused tensor blk.45.ffn_down_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.263.134 W model has unused tensor blk.45.nextn.eh_proj.weight (size = 35651584 bytes) -- ignoring
0.01.263.147 W model has unused tensor blk.45.nextn.enorm.weight (size = 16384 bytes) -- ignoring
0.01.263.158 W model has unused tensor blk.45.nextn.hnorm.weight (size = 16384 bytes) -- ignoring
0.01.263.171 W model has unused tensor blk.45.nextn.shared_head_norm.weight (size = 16384 bytes) -- ignoring
3.36.897.976 E llama_init_from_model: failed to initialize the context: glm5next: the pooled indexer needs one sequence per stream, so a unified KV cache is only supported with a single sequence
3.36.897.983 E cmn common_init_: failed to create context with model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
3.36.897.985 E cmn common_init_: failed to create context with model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
3.36.898.162 E srv load_model: failed to create_context with model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
3.36.898.169 I srv operator (): operator (): cleaning up before exit...
3.36.901.237 E srv llama_server: exiting due to model loading error
CASE 2 (dual GPU, same settings):
0.00.117.395 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.117.715 W srv llama_server: -----------------
0.00.117.717 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.117.718 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.117.718 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.117.718 W srv llama_server: -----------------
0.00.131.418 I srv load_model: loading model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
0.00.877.338 E llama_init_from_model: failed to initialize the context: glm5next: the pooled indexer needs one sequence per stream, so a unified KV cache is only supported with a single sequence
0.00.923.105 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to create llama_context from model
0.01.393.622 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
0.01.393.629 W load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
0.01.442.804 W model has unused tensor blk.45.attn_norm.weight (size = 16384 bytes) -- ignoring
0.01.442.820 W model has unused tensor blk.45.ffn_norm.weight (size = 16384 bytes) -- ignoring
0.01.442.830 W model has unused tensor blk.45.attn_q_a.weight (size = 6684672 bytes) -- ignoring
0.01.442.840 W model has unused tensor blk.45.attn_q_a_norm.weight (size = 6144 bytes) -- ignoring
0.01.442.850 W model has unused tensor blk.45.attn_q_b.weight (size = 26738688 bytes) -- ignoring
0.01.442.861 W model has unused tensor blk.45.attn_kv_a_mqa.weight (size = 2228224 bytes) -- ignoring
0.01.442.872 W model has unused tensor blk.45.attn_kv_a_norm.weight (size = 2048 bytes) -- ignoring
0.01.442.882 W model has unused tensor blk.45.attn_k_b.weight (size = 8912896 bytes) -- ignoring
0.01.442.892 W model has unused tensor blk.45.attn_v_b.weight (size = 8912896 bytes) -- ignoring
0.01.442.904 W model has unused tensor blk.45.attn_output.weight (size = 71303168 bytes) -- ignoring
0.01.442.916 W model has unused tensor blk.45.indexer.k_norm.weight (size = 512 bytes) -- ignoring
0.01.442.927 W model has unused tensor blk.45.indexer.k_norm.bias (size = 512 bytes) -- ignoring
0.01.442.942 W model has unused tensor blk.45.indexer.proj.weight (size = 524288 bytes) -- ignoring
0.01.442.953 W model has unused tensor blk.45.indexer.attn_k.weight (size = 557056 bytes) -- ignoring
0.01.442.965 W model has unused tensor blk.45.indexer.attn_q_b.weight (size = 6684672 bytes) -- ignoring
0.01.442.978 W model has unused tensor blk.45.indexer_compressor_gate.weight (size = 557056 bytes) -- ignoring
0.01.442.990 W model has unused tensor blk.45.indexer_compressor_ape.weight (size = 2048 bytes) -- ignoring
0.01.443.002 W model has unused tensor blk.45.ffn_gate_inp.weight (size = 4718592 bytes) -- ignoring
0.01.443.014 W model has unused tensor blk.45.exp_probs_b.bias (size = 1152 bytes) -- ignoring
0.01.443.025 W model has unused tensor blk.45.ffn_gate_exps.weight (size = 1358954496 bytes) -- ignoring
0.01.443.036 W model has unused tensor blk.45.ffn_up_exps.weight (size = 1358954496 bytes) -- ignoring
0.01.443.047 W model has unused tensor blk.45.ffn_down_exps.weight (size = 1660944384 bytes) -- ignoring
0.01.443.057 W model has unused tensor blk.45.ffn_gate_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.443.069 W model has unused tensor blk.45.ffn_up_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.443.081 W model has unused tensor blk.45.ffn_down_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.443.093 W model has unused tensor blk.45.nextn.eh_proj.weight (size = 35651584 bytes) -- ignoring
0.01.443.105 W model has unused tensor blk.45.nextn.enorm.weight (size = 16384 bytes) -- ignoring
0.01.443.117 W model has unused tensor blk.45.nextn.hnorm.weight (size = 16384 bytes) -- ignoring
0.01.443.131 W model has unused tensor blk.45.nextn.shared_head_norm.weight (size = 16384 bytes) -- ignoring
0.03.976.252 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 93272.10 MiB on device 0: cudaMalloc failed: out of memory
0.03.976.258 E alloc_tensor_range: failed to allocate CUDA0 buffer of size 97802881920
0.04.022.839 E llama_model_load: error loading model: unable to allocate CUDA0 buffer
0.04.022.847 E llama_model_load_from_file_impl: failed to load model
0.04.022.857 E cmn common_init_: failed to load model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
0.04.023.042 E srv load_model: failed to load model, 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
0.04.023.048 I srv operator (): operator (): cleaning up before exit...
0.04.025.511 E srv llama_server: exiting due to model loading error
CASE 3 (GPU memory set to "Manual", GPU layers "8", CPU layers max. available - it climbs to ~190GB RAM usage as it loads, with VRAM barely used, and then it begins to unload from RAM quickly until the error appears):
Note: any GPU layers number leads to an eventual unload --> error; playing around it doesn't seem to help.
0.00.102.115 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.102.422 W srv llama_server: -----------------
0.00.102.424 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.102.425 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.102.425 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.102.425 W srv llama_server: -----------------
0.00.113.056 I srv load_model: loading model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
0.00.767.936 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.767.943 W load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
0.01.022.842 W model has unused tensor blk.45.attn_norm.weight (size = 16384 bytes) -- ignoring
0.01.022.857 W model has unused tensor blk.45.ffn_norm.weight (size = 16384 bytes) -- ignoring
0.01.022.868 W model has unused tensor blk.45.attn_q_a.weight (size = 6684672 bytes) -- ignoring
0.01.022.878 W model has unused tensor blk.45.attn_q_a_norm.weight (size = 6144 bytes) -- ignoring
0.01.022.888 W model has unused tensor blk.45.attn_q_b.weight (size = 26738688 bytes) -- ignoring
0.01.022.898 W model has unused tensor blk.45.attn_kv_a_mqa.weight (size = 2228224 bytes) -- ignoring
0.01.022.909 W model has unused tensor blk.45.attn_kv_a_norm.weight (size = 2048 bytes) -- ignoring
0.01.022.919 W model has unused tensor blk.45.attn_k_b.weight (size = 8912896 bytes) -- ignoring
0.01.022.930 W model has unused tensor blk.45.attn_v_b.weight (size = 8912896 bytes) -- ignoring
0.01.022.940 W model has unused tensor blk.45.attn_output.weight (size = 71303168 bytes) -- ignoring
0.01.022.951 W model has unused tensor blk.45.indexer.k_norm.weight (size = 512 bytes) -- ignoring
0.01.022.962 W model has unused tensor blk.45.indexer.k_norm.bias (size = 512 bytes) -- ignoring
0.01.022.974 W model has unused tensor blk.45.indexer.proj.weight (size = 524288 bytes) -- ignoring
0.01.022.986 W model has unused tensor blk.45.indexer.attn_k.weight (size = 557056 bytes) -- ignoring
0.01.022.997 W model has unused tensor blk.45.indexer.attn_q_b.weight (size = 6684672 bytes) -- ignoring
0.01.023.010 W model has unused tensor blk.45.indexer_compressor_gate.weight (size = 557056 bytes) -- ignoring
0.01.023.022 W model has unused tensor blk.45.indexer_compressor_ape.weight (size = 2048 bytes) -- ignoring
0.01.023.033 W model has unused tensor blk.45.ffn_gate_inp.weight (size = 4718592 bytes) -- ignoring
0.01.023.044 W model has unused tensor blk.45.exp_probs_b.bias (size = 1152 bytes) -- ignoring
0.01.023.055 W model has unused tensor blk.45.ffn_gate_exps.weight (size = 1358954496 bytes) -- ignoring
0.01.023.065 W model has unused tensor blk.45.ffn_up_exps.weight (size = 1358954496 bytes) -- ignoring
0.01.023.076 W model has unused tensor blk.45.ffn_down_exps.weight (size = 1660944384 bytes) -- ignoring
0.01.023.088 W model has unused tensor blk.45.ffn_gate_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.023.098 W model has unused tensor blk.45.ffn_up_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.023.110 W model has unused tensor blk.45.ffn_down_shexp.weight (size = 8912896 bytes) -- ignoring
0.01.023.121 W model has unused tensor blk.45.nextn.eh_proj.weight (size = 35651584 bytes) -- ignoring
0.01.023.133 W model has unused tensor blk.45.nextn.enorm.weight (size = 16384 bytes) -- ignoring
0.01.023.145 W model has unused tensor blk.45.nextn.hnorm.weight (size = 16384 bytes) -- ignoring
0.01.023.158 W model has unused tensor blk.45.nextn.shared_head_norm.weight (size = 16384 bytes) -- ignoring
3.03.517.479 E llama_init_from_model: failed to initialize the context: glm5next: the pooled indexer needs one sequence per stream, so a unified KV cache is only supported with a single sequence
3.03.517.487 E cmn common_init_: failed to create context with model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
3.03.517.488 E cmn common_init_: failed to create context with model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
3.03.517.672 E srv load_model: failed to create_context with model 'C:\AI\models\unsloth\GLM-5.3-Flash-GGUF\GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
3.03.517.679 I srv operator (): operator (): cleaning up before exit...
3.03.521.103 E srv llama_server: exiting due to model loading error
CASE 4 (after Unsloth Studio reinstallation, full "Auto", no advanced settings):
Log fully identical to case 2; tbh they're all pretty close, pardon me for spamming lol
The context was set to mere 64K across all test cases, except case 4 (100K).
upd: fails to load even at < 20K context (tried it just in case)
Unsloth Version
v0.1.804-beta
Package Version
2026.8.22
Desktop App Version
0.1.804-beta
llama.cpp Version
b10639-mix-f6f92fe
GPU 0
NVIDIA GeForce RTX 3090 · 24 GiB
GPU 1
NVIDIA GeForce RTX 3090 · 24 GiB
GLM-5.3-Flash uses a hybrid linear/sparse attention mechanism with a pooled indexer.
The issue seems to be caused by the GLM pooled-indexer cache implementation being incompatible with the current combination of unified KV cache + multiple parallel sequences/slots.
With unified KV enabled, the pooled indexer requires a single sequence.
GLM-5.3-Flash pooled indexer + unified KV cache + more than 1 parallel sequence = startup failure
Unsloth Studio appears to be launching llama-server for this model with a KV-cache/parallelism configuration that doesn't work in the current llama.cpp implementation.
For debugging, first try both: --parallel 1 --no-kv-unified and if that works, try them one at a time.
In Unsloth Studio look for parallel slots and set it to 1 and add an extra llama-server argument for --no-kv-unified. Don't put --parallel 1 in the extra arguments, Unsloth doesn't allow it there.
Curious how many tok/s can you get with so much CPU/RAM offloading?
Let us know!
--no-kv-unified was helpful indeed, thank you <3
One strange thing I've noticed is that the context window size I set doesn't match the actual context window available, as shown in Unsloth Studio GUI.
204800:
320000:
As for the speed, with 4096/2048 batch/ubatch I'm getting about 110 t/s prompt processing and 5 t/s generation (roughly the same numbers with "GPU memory" either at "Auto" or "Manual" 47 GPU / 43 CPU layers), which is within an expected range, compared to 10 t/s generation speed with DeepSeek V4 Flash Q8K_XL and 3 to 4 t/s with GLM 5.2 IQ3_XXS. Quite obviously, such low numbers are due to 4-channel DDR4 memory and a Zen2 Threadripper with only 4 CCDs (3960X model).
EDIT: --parallel 1 also works, no issues with context window (I assume the described behavior is natural for --no-kv-unified, so anyway, I guess this isn't a big deal).
In the end, ~100K context seems to be an upper limit for my rig, with --parallel 1 only. However, I had to reduce batch/ubatch to 2048/1024, halving PP speed in order to fit 100K context while maintaining 5 - 6 t/s generation (it does seem like generation speed is a little faster now, huh?). RAM usage ultimately settled at about ~170GB, and VRAM usage at 22.5 + 22 (with only 1GB in "shared memory").
Max context sets the total memory pool for all slots combined.
With --kv-unified one large KV cache pool is dynamically shared => one single session can potentially use the entire context.
With --no-kv-unified, it gives each parallel slot its own KV cache.
Per slot context = total ctx / number of slots:
204,800 / 4 = 51,200
320,000 / 4 = 80,000
Unsloth Studio probably defaults to 4 parallel slots.
By setting it to --parallel 1 you removed that split, so one chat gets the full context and the UI looks correct again. At the same time it avoids the specific GLM-5.3-Flash pooled-indexer conflict you were hitting with multiple sequences.
❌ Unified-KV ON + Parallel-Slots > 1 -> error, startup failure
✅ Unified-KV ON + Parallel-Slots = 1 -> works
✅ Unified-KV OFF + Parallel-Slots > 1 -> works, but context meter is not intuitive
Improvement suggestions for the UI, based on what @AutisticPancake experienced:
a. Dynamic Context Indicator (when Unified KV is disabled):
- Detect when unified KV is
disabledand expand the context meter to clarify per-slot vs total ->17.3k / 51.2k per-slot (204.8k total) - Add a tooltip explaining the split: "Unified KV disabled — total context pool is split across
4parallel slots"
b. Architecture Guard (for pooled-indexer models):
- Detect
glm5next/ pooled-indexer models from metadata (blk.*.indexer.*as fallback signal) - Warn when an incompatible combination is selected: Unified KV Enabled + Parallel Slots > 1
- Offer a one-click fix:
Set Parallel Slots = 1orDisable Unified KV
Hey guys, the GGUFs are out now and work inside of Unsloth Desktop! Feel free to support our X Tweet or [Reddit post[(https://www.reddit.com/r/unsloth/comments/1vzw55e/glm53_flash_unsloth_dynamic_ggufs/)
Run 3-bit on 128GB RAM or 1-bit on 100GB. The bigger ones are still uploading.
GLM-5.3-Flash (ox-alpha) rivals Claude Opus 4.8 on DeepSWE, coding & agentic benchmarks.Unsloth GitHub: https://github.com/unslothai/unsloth
Guide: https://unsloth.ai/docs/models/glm-5.3-flashThank you for your support!! ❤️❤️
CC: @GGUFGuy @scorpionshoes @ziyy1337 @AutisticPancake @realrebelai @Williamxie1989 @MB7977 @okoku @RobAgrees @raunoryys @codehappy @auf1r2 @ronedgecomb @AshraGaunt @elainedaniels31 @hamsteri @xcywzy @sosiristseng @dzur658 @ciprian-cimpan @wingx @ZoneOverreach @peasantsmith @Nismo @chweber25 @gwillen @Banaxi-Tech @shrimpfriedrice-bricks @dany404 @CassiterFlux @jj4820
YESSSSS
In the end, ~100K context seems to be an upper limit for my rig, with
--parallel 1only. However, I had to reduce batch/ubatch to 2048/1024, halving PP speed in order to fit 100K context while maintaining 5 - 6 t/s generation (it does seem like generation speed is a little faster now, huh?). RAM usage ultimately settled at about ~170GB, and VRAM usage at 22.5 + 22 (with only 1GB in "shared memory").
Update: 200K context is entirely possible at "manual" GPU memory settings. The aforementioned 47 GPU / 43 CPU layers allows for 160K (2048/1024 batch) with a big headroom for more, across 2x 3090.
Interestingly, Q4K_XL offloaded to a single RTX 3090 resulted in ~17GB VRAM usage AND an increase in decode speed to 7 t/s, with RAM usage going up to ~196GB.
It does seem MoE model inference on multi-GPU Windows systems got worse after NVIDIA removed the Control Panel (alongside with SLI - useless on its own, but combined with "Prefer Maximum Performance" it forced all GPUs to stay at higher p-states (1600 - 1900 MHz) during decode; no more of that now, GPU1 keeps downclocking no matter what). Not a fan of what's happening at all :/
That aside, I can't see prompt processing progress with this model in Unsloth Studio (Desktop), it just shows "Generating..." up until the decode begins. Weird.
So I tried using Q5K_XL, got this
"The model is still processing the prompt but did not produce a first token within 20 minutes. Try reducing context length, using more GPU offload, or loading a smaller model." + RETRY button.
Could it be related to the prompt processing progress % not being properly displayed?
There weren't any RAM / VRAM issues, no OOM or anything like that. It just stopped processing abruptly. NOTE: this happened with 2048/512 batch. Afterwards I tried 2048/2048 batch (a nice speedup in PP!) - everything went smoothly and the model reached the decoding stage, outputting the final answer. Not sure whether that timeout thing was a bug or a feature.
STOP COMMENTING HERE
WHY AM I getting BUNCH OF notifications from here bro i didn't even comment here before, just reacted to one message.
WHY AM I getting BUNCH OF notifications from here bro i didn't even comment here before, just reacted to one message.
same