poolside

/

Laguna-S-2.1

like 536

Follow

Poolside 986

Hey! Can I ask how you've deployed the model: which harness, which checkpoint, and so on? It'll help us narrow this down. Anecdotally, this definitely isn't something we've seen, so we're very keen to look into it.

Good afternoon! I am launching your model in NVFP4 on RTX Pro 6000 WE (96 GB), the launch settings are below. After just a couple of minutes of working in Opencode, the model loops, gives out broken blocks of thinking and goes into endless generation. Disabled the use of DFlash - with the settings specified in the model card --speculative-config '{"model":"lakeside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":3,"method":"dflash"}' the generation rate dropped 3 times (from 105 to 35 tokens per second). The acceptance rate is about 0.35 on the first token and continues to fall - on 3-4 tokens it is already 0.

vLLM version 0.25.1, running in WSL. No launch recipes from the model card and its quant worked without errors.

vllm serve /mnt/e/Models/Laguna-S2.1-118B
--host 0.0.0.0
--port 8000
--served-model-name laguna-s2.1
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser poolside_v1
--reasoning-parser poolside_v1
--default-chat-template-kwargs '{"enable_thinking": true}'
--override-generation-config '{"temperature":0.7,"top_p":0.95}'
--max-num-seqs 2
--compilation-config '{"cudagraph_capture_sizes": [1, 2]}'
--max-model-len 262144
--kv-cache-dtype fp8
--gpu-memory-utilization 0.85
--enable-prefix-caching
--enable-chunked-prefill
--performance-mode throughput

Hi @DikiyLifter92 ; we've just uploaded a new branch for the NVFP4 branch that should make the looping incident disappear. It lives at poolside/Laguna-S-2.1-NVFP4@RC1.

The DFlash fix for it will follow in the next few days; thank you for the feedback.

@joerowell
on X, @antirez suggested it doesnt speak italian very well, and well, neither do I. :) But I watch some italian television and listen to some contemporary music in italian, and I fashion myself as A2 or greater at it -- and my focus in LLMs is really largely about language. So I checked it out, and I think I found something actually interesting. This is not, I imagine, NVFP4, as this is the official site you are serving for chat:

To me, this shows 2 things:

• they clearly trained italian
• there is clearly something wrong with their inference setup .. this may be true in a broad/general sense.

so, specifically, in the second part, it says "Anch’esso" is "yet" or "still" instead of "that too". The word "ancora" there looks like a bad argmax flip/token resolution.

A possible correct continuation might have been: “Anch'esso" significa "anche esso" (es. Anche il libro è interessante. → Anch’esso è interessante.), non "anche a te".

Hey! Can I ask how you've deployed the model: which harness, which checkpoint, and so on? It'll help us narrow this down. Anecdotally, this definitely isn't something we've seen, so we're very keen to look into it.

Hi,

I was using llamacpp built yesterday with the IQ4_XS quant from Unsloth along with the sampling parameters found in the json file. I see there was a new upload today so I will test your official quant since I didn't know it existed. Thanks for getting back to me.

So far seems good with the new Q4 K M. I'm not sure if the issue was the Unsloth quant or the fix on the current GGUF (YARN attn factor). I suspect the latter. I am surprised how fast the model is generating even with MoE offload (40-50 t/s). Will post back after more testing.

Thank you! Please let us know if anything goes wrong :)

Just wanted to let you know I encountered 2 other issues and 1 warning. The thinking tag start was not being properly inserted. The fix was to use the latest chat template file from the repo. I found it kind of strange because the GGUF file is newer than the chat template file, and the GGUF should include the chat template. I tested it multiple times and it does fix that think tag issue.

The other issue that isn't yet fixed is some weird output that doesn't always occur. I am constantly shutting down LLamacpp and relaunching with the same parameters. In the latest attempt I asked it to count to 10 and it did this:

count to 10

The only even prime number.
The number of human ears on a typical human head.
The number of primary colors in traditional art: red, blue, and yellow.
The number of dimensions we experience in our physical world (length, width, height), plus one for time, a concept often explored in physics and philosophy.
The number of appendages on most starfish (also known as sea stars), which typically have five arms.
The number base we most commonly use, known as the decimal system, which is based on powers of 10.
The number of days in a week according to many calendars (though this can vary depending on cultural context).
The atomic number of oxygen, a vital element for respiration in living organisms.
The number of planets in our solar system if we don't count Pluto (which was reclassified as a dwarf planet in 2006).
The number of fingers on one hand for many people around the world!

Here's another example:

count to 10

The only even prime number
The second smallest prime number
The smallest odd prime number
The smallest composite number
The number of Platonic solids
The number of sides on a triangle
A prime number
The number of vertices on a cube
The number of months in a year
The base of our number system

I have the logs of the last count to 10 saved but Ican't upload a .txt here.

I also noticed these warnings in Llamacpp, but I don't know if they cause any issues
0.07.445.859 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.07.445.861 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect

Here are my launch params:
/home/user/llama.cpp/build/bin/llama-server -m
/home/user/models/Laguna-S-2.1/laguna-s-2.1-Q4_K_M.gguf
--port 8000 --host 0.0.0.0 -a Laguna-S-2.1
--no-mmap --threads 10 --jinja --chat-template-file /home/user/models/Laguna-S-2.1/chat_template.jinja
--cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on -kvu -np 1
--temp 1.0 --top-p 1.0 --top-k 20 --min-p 0.0
--reasoning on
--spec-type ngram-mod --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-max 64
-ub 4096 -b 4096
-mg 0 --fit on --fit-ctx 200000 -lv 5

Yeah, the model is very weird. I sometimes, like 1 of 10 can get it to output something that resembles normality. Other times it either loops or outputs broken toolcall and just stops.