Sorjîn1-7B

Sorjîn1-7B is a Kurdish Kurmanji language model produced by continually pretraining Qwen/Qwen2.5-7B on academic theses written in Kurdish Kurmanji.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "muzaffercky/Sorjin1-7B",
    torch_dtype="bfloat16",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("muzaffercky/Sorjin1-7B")

inputs = tokenizer("Zimanê kurdî zimanekî", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Evaluation

Model	Eval Loss	Perplexity
Qwen/Qwen2.5-7B (base)	3.42	30.57
Sorjîn1-7B	1.86	6.42

5× perplexity reduction after a single epoch of continual pretraining.

Training

Base model: Qwen/Qwen2.5-7B
Dataset: muzaffercky/kurdish-kurmanji-theses — 389 academic theses, 57.6 MB of cleaned Kurmanji text
Method: QLoRA continual pretraining (4-bit NF4 quantization during training, merged to bfloat16)
LoRA rank: 64, alpha: 128, target modules: all attention and MLP projections
Sequence length: 2048 tokens
Epochs: 1
Hardware: 1× NVIDIA RTX 4090

Intended Use

This is a base model — it is not instruction-tuned. It is intended for:

Further fine-tuning on Kurdish downstream tasks
Merging with instruction-tuned models (see Sorjîn1-7B-Instruct)
Kurdish language research

Limitations

Trained on academic theses only — language style is formal and academic
Single epoch training — further pretraining on broader Kurdish data would improve general language coverage
Not suitable for direct use as a chat assistant — use Sorjîn1-7B-Instruct instead