muzaffercky/Sorjin2-12B-Instruct-LoRA

Sorjîn2-12B-Instruct-LoRA

A LoRA adapter for muzaffercky/Sorjin2-12B-Instruct, trained by instruction tuning on Kurdish Kurmanji (Latin script) questions and answers:

Sorjin2-12B-Instruct   = gemma-4-12B-it + Sorjin2-LoRA
Sorjin2.1-12B-Instruct = Sorjin2-12B-Instruct + Sorjin2-12B-Instruct-LoRA

Sorjin2-LoRA was trained by continual pretraining on 585M tokens of mostly Kurmanji text. Merged into google/gemma-4-12B-it at full strength, it gives Sorjin2-12B-Instruct, which answers Kurmanji chat questions in Turkish. This adapter was trained on top of Sorjin2-12B-Instruct to make it answer in Kurmanji.

This adapter does not apply to google/gemma-4-12B-it directly, only to Sorjin2-12B-Instruct.

Models

Model	Description
muzaffercky/Sorjin2-LoRA	Continual pretraining adapter, trained on google/gemma-4-12B
muzaffercky/Sorjin2-12B-Instruct	google/gemma-4-12B-it with Sorjin2-LoRA merged in, the base of this adapter
muzaffercky/Sorjin2-12B-Instruct-LoRA	This adapter
muzaffercky/Sorjin2.1-12B-Instruct	Sorjin2-12B-Instruct with this adapter merged in (planned)

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "muzaffercky/Sorjin2-12B-Instruct", dtype=torch.bfloat16, device_map="cuda"
)
model = PeftModel.from_pretrained(model, "muzaffercky/Sorjin2-12B-Instruct-LoRA")
tokenizer = AutoTokenizer.from_pretrained("muzaffercky/Sorjin2-12B-Instruct")

messages = [{"role": "user", "content": "LoRA çawa dixebite?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=300, do_sample=True, temperature=1.0, top_p=0.95, top_k=64)
print(tokenizer.decode(output_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The tokenizer and chat template are the unchanged ones of gemma-4-12B-it. The model was trained without thinking: every training prompt ends with the template's empty thinking block.

Training Data

A cleaned copy of muzaffercky/kurdish-ins: Kurmanji questions written by Gemini, with answers copied verbatim from fineweb-2 kmr_Latn and Kurmanji theses. The answers are human-written Kurmanji; their style is that of web text and academic writing, not of an assistant.

The source text has garbled letters (þ for ş, for example). The cleaning script repairs the letters whose correct form is certain and drops the pairs whose garbling has no single correct form, keeping 30,927 of 32,991 pairs. Of those, 338 duplicates and 67 conversations longer than 2,048 tokens were dropped; a long answer is dropped whole rather than cut, so the model does not learn to stop mid-sentence.

Split	Conversations	Tokens
Train	29,985	17.5M
Eval (2%, split by conversation)	537	0.3M

A quarter of the conversations start with one of two Kurmanji system prompts ("Tu alîkarekî zîrek î. Bi kurmancî bersiv bide.", "Bersivên xwe her tim bi kurmancî binivîse."), so the model also sees system prompts. Each conversation is one training sequence: the prompt as the chat template renders it for generation, followed by the answer and the end-of-turn marker <turn|>. Only the answer and the end-of-turn marker are trained on.

Training

Setting	Value
LoRA rank / alpha / dropout	32 / 64 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (328 modules, all in the 48 text layers)
Trainable parameters	131M (1.1% of 12.2B)
Precision	bfloat16, no quantization
Batch size	16 conversations (4 × 4 gradient accumulation)
Steps	3,750 (two epochs)
Learning rate	1e-4, cosine schedule, 3% warmup
Weight decay / gradient clipping	0.01 / 1.0
Hardware	One H100 80GB, about 4.5 hours

The saved adapter is the checkpoint with the lowest eval loss, step 3,700.

Evaluation

Eval loss (cross-entropy per answer token) on the 537 held-out conversations, before and after training: 3.235 → 1.932.

The eval loss is computed on the given answer, so it cannot show which language the model chooses when it writes its own answer. After every evaluation, the model being trained answered seven Kurmanji questions with greedy decoding, and the answers were scored by their share of letters that only Turkish has (ı, ğ, ö, ü) and letters that only Kurmanji has (ê, î, û). Before this adapter, Sorjin2-12B-Instruct's answers had 8.1% Turkish-only letters and 0.1% Kurmanji-only letters. During training, the answers had 0% Turkish-only letters and 10–12% Kurmanji-only letters, about the same Kurmanji share as gemma-4-12B-it itself (12.0%).

Benchmark results and comparisons with gemma-4-12B-it and Sorjîn1.1 will be added later.

Limitations

• With greedy decoding, some answers fall into a loop and repeat one phrase until the token limit. Use sampling, and a mild repetition penalty if needed.
• The training answers are web and thesis passages, not assistant answers. The model tends to answer in that register: long, run-on paragraphs rather than short, structured replies.
• The questions were written by a model and the answers were not written for them, so some answers only partly answer their question. The model may do the same.
• Kurmanji in Latin script only. Sorani and other Kurdish varieties were not trained.
• The model may produce wrong or biased content.
• Use is subject to the license and terms of google/gemma-4-12B-it.