yuyu199741/raya-decision-v1

Raya: Multimodal Fast Decision Model

Raya is a multimodal System-1 fast decision model that replicates and extends the Jev decision paradigm from TypeSafe AI to multimodal inputs (text + image + video). It is built on Qwen3.5-2B and trained with a three-stage pipeline: SFT-LoRA → Full Fine-tuning → RLCD (RL-based Calibration Distillation with GRPO).

Model Hub: yuyu199741/raya-decision-v1

---

What is Raya?

Raya is not a chat model. It does not generate text. Instead, it performs single-forward-pass decision making: given a context (text/image/video), a question, and a set of candidate options, it outputs a probability distribution over the candidates in one forward pass — no autoregressive generation, no text output.

Key Design Principles (Jev Paradigm)

1. No autoregression: Zero generation steps. A single forward pass produces the decision. Latency in milliseconds.
2. Masked projection: Last-position logits are projected onto candidate label tokens only (local softmax over candidates, not the full vocabulary). Output is structurally verifiable — the model physically cannot produce anything outside the candidate set.
3. RLCD calibration: Post-training with strictly proper scoring rules (Brier/log score) + GRPO for confidence calibration. Confidence scores are meaningful and can be used for gating (e.g., escalate to System-2 when uncertain).
4. Dynamic candidate space: Candidates are passed as structured JSON with each request. The model is agnostic to the number/content of options — 3 or 10 options is just a matter of how many logit positions to extract.

What Makes Raya Different

Model	Modality	Calibration	Decision Format
Laya	Text only	No	Text generation
OpenJev	Text only	No	Text generation
jev-visual	Image only	No	Text generation
Raya	Text + Image + Video	Yes (RLCD)	Masked projection, no generation

---

Model Architecture

• Base: Qwen3.5-2B (natively multimodal, 2026-02 release, Apache 2.0)
• Architecture: Hybrid Gated DeltaNet + Gated Attention (3:1 ratio, 24 layers) + sparse MoE FFN
• Context window: 262K tokens
• Parameters: 2.2B (all parameters fine-tuned in Stage 2)

Training Pipeline

Stage	Method	Parameters	Data	Goal
Step 0	Teacher distillation (MiniMax-M3)	—	Soft labels for ambiguous cases	Distribution estimation
Step 1	SFT-LoRA	17M LoRA params	race + koniq + gqa + nextqa (~250K)	Learn decision format
Step 2	Full fine-tuning	All 2.2B params	Same as Step 1	Deepen judgment ability
Step 3	RLCD (GRPO)	All 2.2B params	Same + Brier reward	Confidence calibration

---

Performance

Held-out Evaluation (90 samples, never seen during training)

Modality	Accuracy	vs Zero-shot Baseline
Text (RACE)	86.67%	+50.0pt
Image (KonIQ + GQA)	95.00%	+42.5pt
Video (NExT-QA)	90.00%	+25.0pt
Overall	91.11%	+41.1pt

Calibration (ECE)

Stage	ECE
Before RLCD (Full FT)	0.1221
After RLCD	0.0855

Target: ECE ≤ 0.08. High-confidence buckets (0.9-1.0) are nearly perfectly calibrated (gap = 0.2%).

Latency (measured on NVIDIA H20)

Modality	Latency	Bottleneck
Text	43ms	✅ Production-ready
Image	61ms	✅ Production-ready
Video	1373ms	⚠️ 64% is CPU video decoding, not model forward

---

Usage

Installation

pip install torch transformers pillow
# For video inference:
pip install av

Text Decision

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

model = AutoModelForImageTextToText.from_pretrained(
    "yuyu199741/raya-decision-v1", dtype=torch.bfloat16
).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained("yuyu199741/raya-decision-v1")

text = """User: 帮我把明天下午3点的会议改到4点。这个请求的意图是什么?

A. 修改会议时间
B. 创建新会议
C. 取消会议
D. 查询会议详情

Answer with exactly one label from the options above."""

inputs = tokenizer(text, return_tensors="pt").to("cuda")
with torch.no_grad():
    logits = model(**inputs).logits

# Masked projection: extract logits at candidate label token positions
last_logits = logits[0, -1, :]
label_token_ids = {
    "A": tokenizer.encode(" A", add_special_tokens=False)[0],
    "B": tokenizer.encode(" B", add_special_tokens=False)[0],
    "C": tokenizer.encode(" C", add_special_tokens=False)[0],
    "D": tokenizer.encode(" D", add_special_tokens=False)[0],
}

selected = torch.tensor([last_logits[tid].item() for tid in label_token_ids.values()])
probs = torch.softmax(selected, dim=-1)

for label, prob in zip(label_token_ids.keys(), probs):
    print(f"{label}: {prob:.2%}")

Image Decision

from PIL import Image
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-2B")  # Load from base repo

image = Image.open("photo.jpg").convert("RGB")
text = """<|vision_start|><|image_pad|><|vision_end|>

Rate the overall technical quality of this image.

A. very poor
B. poor
C. fair
D. good
E. excellent

Answer with exactly one label from the options above."""

enc = processor(text=[text], images=[image], return_tensors="pt").to("cuda")
with torch.no_grad():
    logits = model(**enc).logits
# ... same masked projection logic

Video Decision

text = """<|vision_start|><|video_pad|><|vision_end|>

What did the person do first in this video?

A. opened the door
B. sat down
C. picked up the cup
D. left the room

Answer with exactly one label from the options above."""

enc = processor(
    text=[text], videos=["clip.mp4"], return_tensors="pt",
    fps=None, num_frames=8, cap_pixels_per_frame=True,  # Fixed frame count, not fps-based
).to("cuda")
with torch.no_grad():
    logits = model(**enc).logits
# ... same masked projection logic

Critical Rules

1. Do NOT use model.generate(): This model does not generate text. It produces logits at the last position, and you extract the candidate label probabilities via masked projection.
2. Candidate labels are A/B/C/D...: The prompt must render options as "A. option text" format. The model only looks at the logits of these label tokens.
3. Video must use fixed frame count: Use num_frames=8, cap_pixels_per_frame=True. Do NOT use fps parameter (fps-based sampling explodes token count for long videos).
4. Processor loads from base repo: AutoProcessor.from_pretrained("Qwen/Qwen3.5-2B"), not from the model directory (save_pretrained doesn't save processor config).

---

Training Data

Dataset	Modality	Split	Size	License
RACE	Text	train	40,000	research-only
KonIQ-10k	Image	train	7,058	research-only (KonIQ-10k EULA / YFCC100M)
GQA	Image	train_balanced	200,000	research-only (GQA / Visual Genome)
NExT-QA	Video	train (from MC/test parquet)	7,209	research-only (NExT-QA / VidOR)

Note: All training data is research-only licensed. This model is for research use only.

---

Limitations

1. Not a general-purpose chat model: It only does decision-making from candidate options. It cannot answer open-ended questions or generate text.
2. Video frame sampling: Videos are sampled at 8 fixed frames (not fps-based). This is sufficient for action/sequence understanding but may miss frame-level details in long videos.
3. Research license: All training datasets are research-only. Commercial use requires re-training with commercially licensed data.
4. Calibration near target but not exact: ECE = 0.0855, slightly above the 0.08 target. High-confidence predictions are well-calibrated; low-confidence predictions have fewer samples and less statistical significance.
5. 2B model ceiling: Video understanding is inherently limited by the 2B parameter budget. For more demanding video tasks, consider a larger base model.

---

Citation

If you use Raya in your research, please cite:

@misc{raya2026,
  title={Raya: Multimodal Fast Decision Model with Jev Paradigm},
  author={yuyu199741},
  year={2026},
  howpublished={\url{https://huggingface.co/yuyu199741/raya-decision-v1}},
}

---

Acknowledgments

• Base model: Qwen3.5-2B by Alibaba Cloud
• Jev paradigm: TypeSafe AI's Jev decision paradigm
• Training framework: PyTorch + Hugging Face Transformers + PEFT + Accelerate