SciVision-5B: Technical Report & Model Card

SciVision-5B is a specialized multimodal vision-language model (VLM) optimized for high-density STEM visual reasoning, technical diagram interpretation, and step-by-step problem-solving. Built on top of Qwen/Qwen3.5-4B (AutoModelForMultimodalLM), the model was fine-tuned using a 4-bit NormalFloat (NF4) QLoRA architecture over a highly curated, domain-stratified subset derived from the MathV360K benchmark ecosystem.

---

1. Abstract & Executive Summary

While large vision-language models excel at general visual question answering, their accuracy degrades on complex STEM diagrams requiring multi-step logical deduction, symbolic manipulation, and technical knowledge integration. SciVision-5B addresses this challenge through a parameter-efficient fine-tuning methodology guided by the LIMA (Less Is More for Alignment) hypothesis.

By applying an automated Multi-Criteria Quality & Diversity Scoring (MCQDS) algorithm to the candidate dataset kerembb123/MathV360K-Candidate-20k, we extracted an elite $1,000$-sample subset ($N=1,000$) prioritizing Chain-of-Thought (CoT) depth, LaTeX formula density, and structured STEM visual problems. The resulting model achieves steady optimization convergence (final step loss $1.636$, average loss $3.07$) while maintaining a lightweight deployment footprint compatible with consumer hardware.

---

2. Dataset Analysis & Provenance

2.1 Underlying Corpus Provenance

The training baseline originates from kerembb123/MathV360K-Candidate-20k, a $20,000$-sample candidate subset of the full MathV360K dataset (Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models, arXiv:2406.17294).

An audit of the corpus reveals that MathV360K is a heterogeneous benchmark aggregating 24 distinct vision-language datasets across broader STEM visual modalities:

• STEM Concepts & Technical Systems (ScienceQA): Technical system dynamics, state interactions, energy structures, and formal process schematics.
• Diagrammatic & Symbolic Mechanics: Vector systems, structural schematics, signal paths, and optical/spatial representations.
• Document & Tabular Intelligence (DocVQA, DVQA): Plot/chart visual question answering, bar graphs, function plots, and technical document tables.
• Geometric & Symbolic Mathematics (Geometry3K, UniGeo, CLEVR-Math): Formal geometric proofs, coordinate geometry, and spatial-symbolic visual reasoning.

2.2 Quality Curation Methodology (MCQDS)

To prevent model degradation from low-information samples (e.g., direct single-letter option answers lacking reasoning steps), we constructed a mathematical scoring function $S(i)$ for each candidate sample $i$:

$$S(i)=S_{\text{base}}+S_{\text{CoT}}(i)+S_{\text{LaTeX}}(i)+S_{\text{kw}}(i)+S_{\text{prompt}}(i)S(i)\; =\; S\_\{\backslash text\{base\}\}\; +\; S\_\{\backslash text\{CoT\}\}(i)\; +\; S\_\{\backslash text\{LaTeX\}\}(i)\; +\; S\_\{\backslash text\{kw\}\}(i)\; +\; S\_\{\backslash text\{prompt\}\}(i)$$

Scoring Criteria Definitions:

$$S_{\text{base}}=1.0S\_\{\backslash text\{base\}\}\; =\; 1.0$$ Baseline score for validated image-text pair presence on disk.

$$S_{\text{CoT}}(i)=\min (\mathrm{\mid }{W}_i\mathrm{\mid }\times 0.1,6.0)S\_\{\backslash text\{CoT\}\}(i)\; =\; \backslash min(|W\_i|\; \backslash times\; 0.1,\; 6.0)$$ Where $|W_i|$ is the word count of the assistant response (rewarding extended Chain-of-Thought reasoning, applied when $|W_i| \ge 15$).

$$S_{\text{LaTeX}}(i)=\min (L_i\times 0.25,5.0)S\_\{\backslash text\{LaTeX\}\}(i)\; =\; \backslash min(L\_i\; \backslash times\; 0.25,\; 5.0)$$ Where $L_i$ represents the frequency of LaTeX structural characters (\, ^, _, {, }, =, +, -, *, /).

$$S_{\text{kw}}(i)=0.5\times \sum _{k\in K}\mathbf{1}(k\in T_i)S\_\{\backslash text\{kw\}\}(i)\; =\; 0.5\; \backslash times\; \backslash sum\_\{k\; \backslash in\; K\}\; \backslash mathbf\{1\}(k\; \backslash in\; T\_i)$$ Where $K = {\text{step}, \text{therefore}, \text{because}, \text{calculate}, \text{since}, \text{solution}, \text{hence}, \text{equal}}$ and $T_i$ is the target response text.

$$S_{\text{prompt}}(i)=1.0S\_\{\backslash text\{prompt\}\}(i)\; =\; 1.0$$ Applied if structured prompt markers (Hint:, Choices:, Question:) are present in the user query.

Following scoring, Stratified Visual Sampling was applied across the candidate sources to yield $N = 1,000$ gold-standard training instances, preserving a balanced distribution across technical visual categories.

3. Model Architecture & Fine-Tuning Pipeline

3.1 Base Model Specifications

• Base Model: Qwen/Qwen3.5-4B
• Fine-Tuned Model Identifier: SciVision-5B
• Model Class: AutoModelForMultimodalLM
• Positional Embeddings: 3D Multimodal Rotary Position Embeddings (M-RoPE), decomposing positional indices across temporal, height, and width axes ($t, h, w$) to enable fine-grained visual feature alignment.

3.2 Parameter-Efficient Fine-Tuning (QLoRA)

Quantization and low-rank adaptation parameters were selected to balance memory efficiency on dual NVIDIA Tensor Core T4 GPUs ($16\text{ GB}$ VRAM per device) with high model expressivity:

$${\mathbf{W}}_{\text{effective}}={\mathbf{W}}_0+\frac{\alpha }{r}\mathbf{B}\mathbf{A}\backslash mathbf\{W\}\_\{\backslash text\{effective\}\}\; =\; \backslash mathbf\{W\}\_\{0\}\; +\; \backslash frac\{\backslash alpha\}\{r\}\; \backslash mathbf\{B\}\backslash mathbf\{A\}$$

• Base Model Quantization: 4-bit NormalFloat (NF4) with double quantization and FP16 compute dtype.
• LoRA Rank ($r$): $16$
• LoRA Alpha ($\alpha$): $32$
• LoRA Dropout: $0.05$
• Target Modules: All linear projections within the self-attention and feed-forward blocks (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).

3.3 Visual Processing & Memory Optimization

To avoid Out-of-Memory (OOM) errors caused by unbounded visual patch allocation during M-RoPE attention computation, visual input dimensions were strictly bounded:

• Resolution Caps: $\text{min_pixels} = 256 \times 28 \times 28$ ($200,704$ px), $\text{max_pixels} = 384 \times 28 \times 28$ ($301,056$ px). This caps visual tokens at approximately $\sim 384$ tokens per image.
• Sequence Truncation: Maximum input sequence length $L_{\text{max}} = 1,536$ tokens.
• Memory Management: PyTorch allocator setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to mitigate memory fragmentation during the backward pass.

---

4. Training Dynamics & Convergence

The model was trained for $3$ full epochs ($189$ total optimization steps) with an effective global batch size of $16$ ($1 \text{ sample/GPU} \times 8 \text{ accumulation steps} \times 2 \text{ GPUs}$).

4.1 Loss Progression

The model exhibited rapid early adaptation during learning rate warmup, stabilizing into a smooth loss plateau.

4.2 Training Metrics Summary

Epoch	Step	Training Loss	Gradient Norm ($\Vert{}g\Vert{}_2$)	Learning Rate
$0.08$	$5$	$17.31$	NaN (Warmup)	$1.958 \times 10^{-4}$
$0.24$	$15$	$13.59$	$11.35$	$1.852 \times 10^{-4}$
$0.48$	$30$	$2.755$	$1.941$	$1.693 \times 10^{-4}$
$0.80$	$50$	$1.765$	$0.358$	$1.481 \times 10^{-4}$
$1.03$	$65$	$1.727$	$0.273$	$1.323 \times 10^{-4}$
$2.06$	$130$	$1.662$	$0.227$	$6.349 \times 10^{-5}$
$2.94$	$185$	$1.636$	$0.248$	$5.291 \times 10^{-6}$

• Overall Run Loss: $3.07$
• Total Training Runtime: $3,994\text{ seconds}$ ($\sim 66.5\text{ minutes}$)
• Throughput: $0.751\text{ samples/sec}$, $0.047\text{ steps/sec}$

---

5. Hyperparameter Reference Table

Specification / Hyperparameter	Operational Setting
Base Architecture	Qwen/Qwen3.5-4B
Target Model Identifier	SciVision-5B
Model Framework	PyTorch $2.5.1$, Hugging Face Transformers 4.49.0+
Quantization Precision	4-bit NF4 (load_in_4bit=True, bnb_4bit_quant_type="nf4")
Compute Dtype	torch.float16
LoRA Rank ($r$) / Alpha ($\alpha$)	$r = 16$, $\alpha = 32$
Target Linear Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Peak Learning Rate	$2.0 \times 10^{-4}$
LR Scheduler	Linear Decay with $3%$ Warmup Ratio
Optimizer	AdamW (beta1=0.9, beta2=0.999, weight_decay=0.01)
Micro Batch Size per GPU	$1$
Gradient Accumulation Steps	$8$
Total Effective Batch Size	$16$ ($1 \text{ batch} \times 8 \text{ accum} \times 2 \text{ GPUs}$)
Hardware	$2 \times \text{NVIDIA Tensor Core T4}$ ($16\text{ GB}$ VRAM each)

---

6. Inference & Deployment

6.1 PyTorch / Transformers Integration

The model requires mm_token_type_ids handling provided automatically by AutoProcessor.

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "salihfurkaan/SciVision-5B"

# 1. Initialize Processor and Model
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)

# 2. Load STEM Visual Input
image_path = "stem_diagram.png"
image = Image.open(image_path).convert("RGB")

prompt_text = (
    "Analyze the provided STEM diagram step-by-step. "
    "Identify key components, explain their relationships, "
    "and derive the final quantitative or logical result."
)

# 3. Format Input via Chat Template
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": prompt_text}
        ]
    }
]

formatted_prompt = processor.apply_chat_template(
    messages, 
    tokenize=False, 
    add_generation_prompt=True
)

inputs = processor(
    text=[formatted_prompt], 
    images=[image], 
    return_tensors="pt"
).to("cuda")

# 4. Generate Response
with torch.no_grad():
    generated_ids = model.generate(
        **inputs, 
        max_new_tokens=512, 
        do_sample=False
    )

# Trim input prompt tokens from generation output
trimmed_ids = [
    out[len(inp):] for inp, out in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(trimmed_ids, skip_special_tokens=True)[0]

print("--- SciVision-5B Response ---")
print(response)