Qwen3-VL-8B AURA Focal

Final model from one epoch of AURA full fine-tuning of Qwen/Qwen3-VL-8B-Instruct with token-level Focal loss (gamma=2.0). This repository contains the final BF16 model weights, tokenizer, custom model and processor code, and training summary. External evaluation has not been performed.

Training

• Base revision: 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b.
• 173,553 cached training examples; 2,712 optimizer steps; one epoch.
• 32 GPUs; per-device batch 1; gradient accumulation 2; global batch 64.
• Learning rate 1e-5; cosine scheduler; warmup ratio 0.1; seed 42.
• BF16, FlashAttention 2, DeepSpeed ZeRO-3; sequence limit 11,000 tokens.
• AURA answer selection and silent-token weighting are preserved. Focal modulation applies to all retained supervised tokens, with the original weighted-token denominator.
• Visual patch embedding, transformer blocks, deepstack mergers and projector are frozen. model.visual.pos_embed.weight remains trainable to match the CE baseline. The language model and LM head are trainable.

Mean training Focal loss: 0.493920. Final logged step loss: 0.3440. Focal loss values cannot be directly compared with CE values as model-quality scores.

Loading

Tested with Transformers 4.57.1 and PyTorch 2.6.0.

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "leobo09263013/focal"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto"
)

The checkpoint retains the custom <|silent|> token (ID 151669). The saved Fast tokenizer removes it when skip_special_tokens=True; account for that when evaluating silence decisions.

For continued training, Focal is a runtime loss hook and is not restored by loading model weights alone. Reattach it from training/aura_loss.py using attach_aura_loss(model, processor.tokenizer, "focal", focal_gamma=2.0). The original CE model implementation is retained in the custom model file. This export does not contain the distributed optimizer state.