oldhag88/Qwen3.8-27B-json_captioner-basefp8-Rank64

Model Card for Model ID

FP8 base + BF16 LoRA on vLLM

This LoRA has been trained on json BBOXes for erotic images (NSFW).

Setup for serving an FP8-quantized Qwen3.8 VL model with this LoRA:

Config

LLM(
    model="Qwen/Qwen3.8-27B-FP8",
    dtype="bfloat16",
    trust_remote_code=True,

    # multimodal
    mm_processor_kwargs={"min_pixels": 256 * 1024, "max_pixels": 1_649_664},
    limit_mm_per_prompt={"image": 1},

    # memory / throughput budget
    max_model_len=7168,
    max_num_batched_tokens=8192,
    max_num_seqs=170,
    gpu_memory_utilization=0.96,
    tensor_parallel_size=1,

    # reasoning
    reasoning_parser="qwen3",

    # LoRA
    enable_lora=True,
    max_lora_rank=64,
    max_loras=1,
    enable_tower_connector_lora=True,
    lora_dtype="bfloat16",
)

lora = LoRARequest("combined", 1, "oldhag88/Qwen3.8-27B-json_captioner-basefp8-Rank64")
llm.chat(..., lora_request=lora)

Notes

• FP8 base + BF16 LoRA: the base weights are FP8-quantized checkpoints (loaded directly, no scaling config needed), while the LoRA adapter runs in bfloat16 via lora_dtype. This avoids quantizing the adapter and typically trains/ serves fine with a rank up to max_lora_rank=64.
• enable_tower_connector_lora=True: required for VL models — it trains LoRA on the vision-to-LLM (tower) connector projection in addition to the language model attention/MLP layers. Without it, adapter weights on the connector are silently dropped.
• max_loras=1: only one adapter is resident; lora_request id 1 maps to the "combined" adapter. Increase if you want to serve multiple adapters concurrently (costs extra KV/gPU memory per resident adapter).
• reasoning_parser="qwen3" + chat kwargs enable_thinking=True, reasoning_effort="medium": enables Qwen3 thinking mode; the parser splits <think>...</think> out of the completion text.
• Prefix caching note: enable_prefix_caching=False was set here even though the workload shares an identical system+user text prefix across requests. Leave it enabled (default) unless it conflicts with your version's LoRA + FP8 support.
• Prompt-mm resize clamp: min_pixels/max_pixels cap the per-image token budget (1,649,664 px² ≈ ~2100 image tokens); pre-resize large images client-side as well to cut decode/preprocess CPU time.