julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer

Occitan Gemma 4 E2B IT - rs-LoRA (SFTTrainer) v2

This model is a fine-tuned version of unsloth/gemma-4-e2b-it-unsloth-bnb-4bit on Occitan language text data using rs-LoRA (Rank-Stabilized LoRA) with the SFTTrainer from the TRL library.

v2 improvements over v1: Larger rank (32 vs 16), lower learning rate with cosine schedule and warmup, weight decay, and more training steps — resulting in significantly lower loss.

Model Description

• Base Model: Gemma 4 E2B IT (Instruction-Tuned)
• Fine-tuning Method: rs-LoRA (Rank-Stabilized Low-Rank Adaptation)
• Training Framework: TRL SFTTrainer
• Language: Occitan (Lengadocian dialect)
• Model Type: Causal Language Model

Training Details

Hyperparameters (v2)

Parameter	Value
LoRA Rank (r)	32
LoRA Alpha	32
Training Steps	2535
Block Size	384 tokens
Learning Rate	1e-4
LR Scheduler	cosine
Warmup Steps	200
Weight Decay	0.01
Epochs	5
Gradient Accumulation	8
Optimizer	paged_adamw_8bit
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Dropout	0
Bias	none

Changes from v1

Parameter	v1	v2
Rank	16	32
Steps	1270	2535
Learning Rate	2e-4	1e-4
Scheduler	linear	cosine
Warmup	0	200
Weight Decay	0	0.01
Gradient Accum.	16	8
Optimizer	AdamW	paged_adamw_8bit

Training Loss Progression

Step	Loss
10	1.571
100	0.539
500	0.415
1000	0.373
1500	0.322
2000	0.275
2530	0.247

Loss decreases steadily throughout training with a smooth cosine decay schedule, converging to ~0.25.

Repository Structure

├── adapter/                    # LoRA adapter weights (for loading with base model)
│   ├── adapter_config.json
│   ├── adapter_model.safetensors
│   ├── chat_template.jinja
│   ├── tokenizer.json
│   └── tokenizer_config.json
├── gguf/                       # GGUF quantized models (for llama.cpp)
│   ├── occitan-gemma-4-e2b-it-rslora-merged-Q4_K_M.gguf  (3.43 GB)
│   ├── occitan-gemma-4-e2b-it-rslora-merged-Q5_K_M.gguf  (3.63 GB)
│   ├── occitan-gemma-4-e2b-it-rslora-merged-Q8_0.gguf    (4.97 GB)
│   └── occitan-gemma-4-e2b-it-rslora-merged-f16.gguf    (9.31 GB)
├── config.json               # Model configuration
├── model.safetensors        # Merged 16-bit model (10.2 GB)
├── tokenizer.json
└── README.md

Usage

Option 1: Use the Merged Model (Recommended)

Load the full merged model directly:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer"
)

prompt = "Conta m'en una istueta brèva en occitan"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Option 2: Use with Base Model + LoRA Adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/gemma-4-e2b-it-unsloth-bnb-4bit",
    load_in_4bit=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "unsloth/gemma-4-e2b-it-unsloth-bnb-4bit"
)

model = PeftModel.from_pretrained(
    base_model,
    "julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer",
    subfolder="adapter"
)

Option 3: GGUF with llama.cpp

./llama-server -m occitan-gemma-4-e2b-it-rslora-merged-Q4_K_M.gguf -c 512 --port 8081

Quantization Options

File	Size	Description
Q4_K_M	3.43 GB	Recommended: Best quality/size tradeoff
Q5_K_M	3.63 GB	Higher quality, slightly larger
Q8_0	4.97 GB	Near-lossless quality
f16	9.31 GB	Full 16-bit precision (lossless)

Generated Text Examples

Prompt: Lo marin

Generated continuation:

òm auriá vist.
Lo marin òm auriá vist.
Mas la marè n'es tota luènha.
Al mens per far venir l'aiga,
Cal anar a la plaja,
E la marè n'es tota luènha.
La marè n'es tota luènha
Per la far venir,
Per la far venir.
Vèni, marin, vèni,
Porta-

Quality Notes

The model generates high-quality Lengadocian Occitan text with:

• ✅ Correct Lengadocian vocabulary (marin, auriá, marè, luènha, plaja)
• ✅ Proper grammar (conditional, imperfect subjunctive)
• ✅ No French/Catalan interference
• ✅ Natural narrative style

Training Data

Trained on a corpus of Occitan text in the Lengadocian dialect, including:

• Traditional literature
• Contemporary texts
• Wikipedia articles
• Cultural documents

Limitations

• The model is specialized for Occitan text generation
• Chat template support may vary depending on inference framework
• For best results with instruction-following, use the text completion endpoint rather than chat endpoint in some frameworks (e.g., current llama.cpp)

Citation

If you use this model, please cite:

@misc{occitan-gemma-4-e2b-it-rslora-sfttrainer-v2,
  author = {Julien},
  title = {Occitan Gemma 4 E2B IT - rs-LoRA SFTTrainer v2},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer}
}