muzaffercky/Sorjin2-LoRA

Sorjîn2-LoRA

⚠️ Merged at full strength into gemma-4-12B-it, this adapter makes the model answer in Turkish

Added at full strength to gemma-4-12B-it, it makes the model answer Kurmanji questions in Turkish.

For Kurmanji chat, use Sorjîn1.1-7B-Instruct or gemma-4-12B-it until a corrected version is available.

• Known issue: Sorjîn2-12B-Instruct, gemma-4-12B-it with this adapter merged in, answered all 8 test prompts in Turkish. gemma-4-12B-it alone answered them in Kurmanji
• Cause: The adapter was trained by continual pretraining on the base model google/gemma-4-12B, with no instruction data. Added to the instruct model at full strength, it overrides the instruct model's habit of answering in the language of the question
• Strength test: At half strength or less (LoRA scale 0.5 or lower), gemma-4-12B-it with this adapter answered the same prompts in Kurmanji. At 0.75 strength 6 of 8 answers were in Turkish
• On the base model: Sorjîn2-12B is google/gemma-4-12B with this adapter merged in, the model it was trained for. It is a text completion model, not a chat model, and its output has not been evaluated yet

A LoRA adapter for google/gemma-4-12B, trained by continual pretraining on Kurdish Kurmanji (Latin script) text. It is the second version of Sorjîn, after Sorjin1.1-7B-Instruct, which was built on Qwen2.5-7B.

This adapter is a base model adapter: it continues text, it does not follow instructions. For chat, use Sorjin2-12B-Instruct (the adapter merged into google/gemma-4-12B-it) once it is published.

Models

Model	Description
muzaffercky/Sorjin2-LoRA	This adapter
muzaffercky/Sorjin2-12B	google/gemma-4-12B with the adapter merged in (planned)
muzaffercky/Sorjin2-12B-Instruct	google/gemma-4-12B-it with the adapter merged in (planned)

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("google/gemma-4-12B", dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(base_model, "muzaffercky/Sorjin2-LoRA")
tokenizer = AutoTokenizer.from_pretrained("muzaffercky/Sorjin2-LoRA")

inputs = tokenizer("Kurdistan welatekî", return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

The tokenizer is the unchanged Gemma 4 tokenizer.

Training Data

One epoch over 585M tokens, counted with the Gemma 4 tokenizer:

Dataset	Part used	Documents	Tokens	Role
HuggingFaceFW/fineweb-2	kmr_Latn train	444,677	454.6M	Main Kurmanji corpus
muzaffercky/kurdish-theses-yok	Kurmanji theses, 97% of theses	398	42.7M	Academic register
HuggingFaceFW/fineweb-edu	sample-10BT	84,667	87.8M	English replay against forgetting, 15% of the mix

Documents shorter than 200 characters and exact duplicate texts were removed. Train and eval sets are split by document, so no evaluation text was seen in training. Documents are packed into 2,048-token sequences that start with <bos>, with <eos> between documents.

Training

Setting	Value
LoRA rank / alpha / dropout	64 / 128 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (328 modules, all in the 48 text layers)
Trainable parameters	262M (2.1% of 12.2B)
Precision	bfloat16, no quantization
Sequence length	2,048 tokens
Batch size	16 sequences (4 × 4 gradient accumulation), about 33k tokens per step
Steps	17,865 (one epoch)
Learning rate	1e-4, cosine schedule, 2% warmup
Weight decay / gradient clipping	0.01 / 1.0
Hardware	One H100 80GB, about 66 hours

The saved adapter is the checkpoint with the lowest Kurmanji (fineweb-2) eval loss, which was the last step.

Evaluation

Eval loss (cross-entropy per token) on held-out text, measured on the same 200 sequences per source before and after training:

Eval set	Before	After
fineweb-2 kmr_Latn test split	2.274	1.620
Kurmanji theses (8 held-out theses)	2.347	1.862
fineweb-edu (English)	2.213	2.159

A Kurmanji word is about 2.3 Gemma tokens. Per word, the fineweb-2 perplexity fell from about 185 to about 41.

Per-token losses cannot be compared across languages or tokenizers, so these numbers are not comparable with Sorjîn1 or with English. Benchmark results (FLORES-200, Belebele) and comparisons with gemma-4-12B-it and Sorjîn1.1 will be added after the merge.

Limitations

• Not instruction-tuned: it continues text and does not answer questions in chat form.
• Kurmanji in Latin script only. Sorani and other Kurdish varieties were not trained.
• Most of the Kurmanji data is web text. The model reproduces its style, including inconsistent spelling and Turkish-influenced phrasing, and may produce wrong or biased content.
• The theses eval set has only 8 theses, so its loss is a rough number.
• English ability is mostly kept, but the English eval loss ended about 0.04 above its level after the first few hundred steps, so some mild forgetting took place.
• Use is subject to the license and terms of the base model, google/gemma-4-12B.