New Weights Break vllm DGX Spark setup
Updated weights from earlier this month - 8 tks
Previous weights prior to the RC2 merge - 42 tks
Also, as others have stated, max-model-length and gpu utilization in vllm configuration must be changed just to load the new weights. Love this model, but I think an update is needed unless there there is some configuration wizardry that can help with the latest version of the model.
The slowdown and the size growth are the same change: the 08-01 update ships layers 40-47 experts in BF16, which adds 27.8 GB of weight reads per token on a bandwidth-limited machine. My full-NVFP4 build of the current weights avoids both: https://huggingface.co/kkuspa/Laguna-S-2.1-NVFP4-0804. On the Spark it should restore roughly the decode rate you saw before the update, since the per-token read footprint returns to the July level. Details in the card.
The slowdown and the size growth are the same change: the 08-01 update ships layers 40-47 experts in BF16, which adds 27.8 GB of weight reads per token on a bandwidth-limited machine. My full-NVFP4 build of the current weights avoids both: https://huggingface.co/kkuspa/Laguna-S-2.1-NVFP4-0804. On the Spark it should restore roughly the decode rate you saw before the update, since the per-token read footprint returns to the July level. Details in the card.
Did you try speculative decoding with your build? Does the DFlash model work with your model? They seem to have not updated that and at least in the larger (BF16) it no longer works (acceptance rate is zero).
Worth noting that in my case see my post I discovered that when thinking in enabled you need to give a lot of room for output. I had 8192 max output tokens originally for my opencode configuration and the model was not working because it was running out of output space. Running it with 65536 now...
The slowdown and the size growth are the same change: the 08-01 update ships layers 40-47 experts in BF16, which adds 27.8 GB of weight reads per token on a bandwidth-limited machine. My full-NVFP4 build of the current weights avoids both: https://huggingface.co/kkuspa/Laguna-S-2.1-NVFP4-0804. On the Spark it should restore roughly the decode rate you saw before the update, since the per-token read footprint returns to the July level. Details in the card.
I tried it this afternoon. With speculative decoding, 9-10 tks. Without speculative decoding, it’s ~17.5 tks…. So, I’m guessing we need an updated DFLASH drafter?
After reading the model card and your approach, I learned a few things. Excellent job and thank you.