poolside

/

Laguna-S-2.1-INT4

like 16

Follow

Poolside 1.05k

Hi there! Thanks for your comment.

We'll upload a new INT4 checkpoint in the next few hours that, in our internal tests, we've been unable to make loop.

I'll post here once it's uploaded.

Hi, thank you :) In the meantime I temporarily increased the repetition/presence penalties it seems to be good or maybe it is a coincidence, the repetition was less present on the int4, it is much easier to reproduce in nvfp4

Hi there! The updated INT4 weights are on the repo now. If you have any problems with them (or any feedback at all), please let us know: if I don't hear from you in the next 24 hours or so, I'll close this issue :)

Following up, does the DFlash Speculator also need to be updated? Acceptance rates have crashed to 0% after deploying the updated model.

Same here, I have 0 acceptance with the updated weights.

It keeps looping on new weights. Confirmed commit hash is 67dbeda .

Here is docker compose file (Dflash is not worked on this weights so I commented out )

  vllm-laguna:
    image: vllm/vllm-openai:vllm-x86_64-cu13-0.25.1-7a33ba9
    restart: unless-stopped
    shm_size: '48gb'
    ports:
      - "8000:8080"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: ${QWEN_GPU_COUNT:-1}
              capabilities:
                - gpu
    environment:
      HF_HOME: /root/.cache/huggingface
      HF_TOKEN: ${HF_TOKEN}
      HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface
      TRANSFORMERS_CACHE: /root/.cache/huggingface
      VLLM_CACHE_ROOT: /root/.cache/huggingface
      VLLM_LOGGING_LEVEL: ${VLLM_LOGGING_LEVEL:-info}
    volumes:
      - /mnt/data/models/vllm:/root/.cache
    security_opt:
      - seccomp=unconfined
    command:
      - --model
      - poolside/Laguna-S-2.1-INT4
      - --served-model-name
      - laguna-s2.1
      - --trust-remote-code
      - --max-model-len
      - "262144"
      - --enable-auto-tool-choice
      - --tool-call-parser
      - "poolside_v1"
      - --reasoning-parser
      - "poolside_v1"
      - --max-num-seqs
      - "16"
      - --gpu-memory-utilization
      - "0.95"
      - --cpu-offload-gb
      - "0"
      - --default-chat-template-kwargs
      - '{"enable_thinking":true}'
      - --override-generation-config
      - '{ "temperature": 0.7, "top_p":0.95}'
        # - --speculative-config
        # - '{"model":"poolside/Laguna-S-2.1-DFlash-INT4","num_speculative_tokens":15,"method":"dflash"}'

@TOSUKUi , just pointing out that the documentation specifies temp 0.7 top_p 0.95 only for the 1M context configuration. For your 256K configuration the config defaults to 1.0/1.0.

still looping after new weight, vllm nightly 0.23.1rc1.dev1432+g0231dd546

@jakehlee Thank you.

My understanding from README is that temperature <= 0.7 and top_p <= 0.95 are recommended when using the extended 1M-token context window, while temperature = 0.7 and top_p = 0.95 are recommended for general use.

I think this because the first guidance is in the Context Length section, while the second is in the Recommended Sampling section. Since both sections are at the same level under Usage, they seem to describe different use cases.

I also still have looping, I just removed the temperature settings but nothing changed.

        args:
            - --model
            - /models/Laguna-S-2.1-INT4
            - --served-model-name
            - laguna
            - --reasoning-parser
            - poolside_v1
            - --tensor-parallel-size
            - "4"
            - --max-num-batched-tokens
            - "2048"
            - --gpu-memory-utilization
            - "0.88"
            - --max-num-seqs
            - "8"
            - --max-model-len
            - "262144"
            - --enable-auto-tool-choice
            - --tool-call-parser
            - poolside_v1
            - --default-chat-template-kwargs
            - '{"enable_thinking": true}'
            - --enforce-eager

I think it’s useless to try, Ive retested the nvfp4 and it’s the same, all models are bad, when it doesn't go to infinity it can't even get out a simple HTML code he writes noob errors, for exemple for html comments he writes <!--..., in JS and PHP he uses the wrong concat symbols..., I don't know if it’s an issue in all quantisations or on the base model but for now it's impossible to do anything...