itsawaysguthing/gemma-4-4b-it-latex-ocr-lora

Gemma 4 (4B) - LaTeX OCR LoRA

Model Description

LoRA adapter fine-tuned on Gemma 4 (4B) for LaTeX OCR — converting math formula screenshots into LaTeX code.

Fine-tuned with RsLoRA + DoRA strategy on the unsloth/LaTeX_OCR dataset (68,686 samples).

Training Details

Item	Value
Base Model	google/gemma-4-4b-it
Strategy	RsLoRA + DoRA
LoRA rank	8
Target Modules	q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj
Training Data	unsloth/LaTeX_OCR (68,686 samples)
Precision	bfloat16
Booster	Auto
Validation Size	0.002 (0.2%)
Hardware	NVIDIA RTX 3090 Ti (24GB)
Training Time	~5.6 hours (GPU time, 1 epoch, 4285 steps)

Performance

Evaluation on 100 random test samples:

Metric	Before Fine-tuning	After Fine-tuning
Levenshtein Similarity (avg)	0.5284	0.9154
WS-Free Exact Match (avg)	0.02	0.42
WS-Free Exact Match (hits)	2/100	42/100

Training Curves

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, Gemma4ForConditionalGeneration
from peft import PeftModel

# Load base model + LoRA adapter
model = Gemma4ForConditionalGeneration.from_pretrained(
    "google/gemma-4-4b-it",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, "itsawaysguthing/gemma-4-4b-it-latex-ocr-lora")

processor = AutoProcessor.from_pretrained("google/gemma-4-4b-it")

# Inference
image = Image.open("math_formula.png")
text = f"<bos><|turn|>user\n<|image|>Please output the LaTeX code for this math formula.<turn|><|turn|>model\n"
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=256, do_sample=False, num_beams=1)
out = out[:, inputs["input_ids"].shape[1]:]
result = processor.tokenizer.decode(out[0], skip_special_tokens=True).strip()
print(result)