yashsutorichat/Lotus-1-GGUF

Lotus-1 GGUF

GGUF quantizations of Lotus-1, the 35B-A3B roleplay model behind Sutorichat, for KoboldCpp, llama.cpp, LM Studio and text-generation-webui. Only about 3B parameters are active per token, so the model decodes at small-model speed: the quant you pick is about what fits in memory, not about speed.

Converted with llama.cpp convert_hf_to_gguf.py (bf16) and llama-quantize, no importance matrix. The embedded chat template defaults to non-thinking, so every loader that reads the GGUF template produces the right prompt with no flags.

file	bits	size	fits
Lotus-1-Q3_K_M.gguf	3	~16 GB	16 GB VRAM, or 24 GB RAM
Lotus-1-Q4_K_S.gguf	4	~21 GB	24 GB VRAM
Lotus-1-Q4_K_M.gguf	4	~22 GB	24 GB VRAM (recommended)
Lotus-1-Q5_K_M.gguf	5	~26 GB	32 GB
Lotus-1-Q6_K.gguf	6	~29 GB	32 GB, near-lossless
Lotus-1-Q8_0.gguf	8	~37 GB	48 GB
Lotus-1-BF16-*.gguf (split)	16	~70 GB	re-quantizing

Sizes are before the KV cache; at 32k context budget another 2 to 3 GB. Because experts are offloaded well, partial GPU offload works: a 12 GB card plus 32 GB of RAM runs Q4_K_M at usable speed with --n-gpu-layers set to what fits.

SillyTavern

1. Load the GGUF with a 32k context:
   • KoboldCpp: koboldcpp --model Lotus-1-Q4_K_M.gguf --contextsize 32768 --gpulayers 99
   • llama.cpp: llama-server -m Lotus-1-Q4_K_M.gguf -c 32768 -ngl 99 --jinja
   • LM Studio: load the file, set context to 32768 in the model settings.
2. SillyTavern, API: Text Completion, pick your backend.
3. Advanced Formatting: import sillytavern/Lotus-1.context.json (Context Template) and sillytavern/Lotus-1.instruct.json (Instruct Template). Turn the System Prompt off; the preamble the model expects is in the context template.
4. Samplers: import sillytavern/Lotus-1.textgen.json. That is temperature 0.9, top_p 0.95, top_k 20, repetition penalty 1.05, 700 response tokens.

Keep the temperature at 0.8 or above. Below that the model starts repeating across turns; repetition penalty, not temperature, is what controls loops here.

Prompt format

ChatML with thinking disabled. What the instruct preset produces:

<|im_start|>system
You are {char} in an ongoing roleplay conversation.

Stay in character, respond naturally to the user's latest message, preserve continuity, and do not describe or control the user's actions.

Character and scenario context:
{card}<|im_end|>
<|im_start|>user
{message}<|im_end|>
<|im_start|>assistant
<think>

</think>

The empty <think> block is part of the format: the model was trained with thinking off, and the assistant prefix must include it in Text Completion mode. The instruct preset does this for you.

The context preset renders the card the way Sutorichat does (Name: tagline, then the details as prose, then your persona). Write the card's Description as a one-line tagline and put everything else in Personality as prose with no headings. The exact production prompt, including the mid-chat memory block, is in PROMPT_TEMPLATE.md in this repo, with lotus_prompt.py to build it.

See the Lotus-1 card for training details, limitations and the license.