Qwen2.5-1.5B-Instruct, QLoRA fine-tuned for Indonesian instructions

This is Qwen2.5-1.5B-Instruct fine-tuned with QLoRA on about 47k Indonesian instruction/response pairs. It is the generator model in indonesian-labor-law-rag, a retrieval-augmented QA system over Indonesian labor regulations.

The model was fine-tuned for general Indonesian instruction following. It was not trained on legal text: legal knowledge in the RAG system comes from retrieved documents, not from the model weights.

Model details

Base model unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit, a 4-bit build of Qwen/Qwen2.5-1.5B-Instruct
Method QLoRA (4-bit NF4 with double quantization); LoRA r=16, alpha=16, dropout 0, on q/k/v/o_proj and gate/up/down_proj
Weights LoRA adapter merged into the base model and saved in float16 (~3.1 GB)
Chat format ChatML (<|im_start|> / <|im_end|>), with no default system prompt
Language Indonesian
License Apache 2.0, inherited from the base model

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "dedyirama/qwen2.5-1.5b-indonesian-qlora"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Jelaskan secara singkat apa itu energi terbarukan."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=256, repetition_penalty=1.1)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

The model also loads with Unsloth's FastLanguageModel.from_pretrained(..., load_in_4bit=True), which is how the RAG notebook runs it on a T4 GPU.

Training

Data Ichsan2895/alpaca-gpt4-indonesian: 49,969 single-turn pairs reformatted from FreedomIntelligence/alpaca-gpt4-indonesian (CC BY-SA 4.0), split 95/5 into train (47,470) and validation (2,499) with seed 3407
Formatting Each pair mapped to a ChatML user/assistant conversation
Trainer TRL SFTTrainer via Unsloth, no sequence packing, max sequence length 2,048
Schedule 800 steps (~0.13 epoch), effective batch size 8 (2 ร— 4 gradient accumulation), learning rate 2e-4, linear decay, 50 warmup steps
Optimizer adamw_8bit, weight decay 0.01, fp16
Hardware One NVIDIA Tesla T4 (Google Colab)

Hyperparameter comparison

Two configurations were trained with the same schedule, and the one with the lower validation loss was published:

Run LoRA r / alpha Learning rate Trainable params Final validation loss
This model 16 / 16 2e-4 18.5M (1.18%) 1.3726
Alternative 32 / 32 1e-4 36.9M (2.34%) 1.3784

Each configuration was trained only once, and rank and learning rate were changed together, so the small gap does not show that either setting is better. Validation loss is the only metric reported: the model was not evaluated on downstream benchmarks or compared against the base model.

Intended use and limitations

  • Intended use: learning and experimentation with Indonesian instruction following, and as a small generator inside a RAG pipeline that supplies the facts.
  • Not for factual or legal advice on its own. At 1.5B parameters the model can hallucinate, and it sometimes repeats itself. Use retrieval and a repetition penalty.
  • Limited training. The model saw only ~13% of one epoch of the dataset, so the changes from the base model are modest.
  • Quantization error. The adapter was trained against 4-bit base weights and merged into a dequantized copy of them, so the float16 weights differ slightly from the original Qwen/Qwen2.5-1.5B-Instruct.
  • Inherited biases. The training data is an Indonesian version of Alpaca-GPT4, whose responses were generated by GPT-4. The model may inherit artifacts and biases from GPT-4, from the dataset's conversion to Indonesian, and from the base model.

Links

Downloads last month
94
Safetensors
Model size
2B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for dedyirama/qwen2.5-1.5b-indonesian-qlora

Finetuned
(199)
this model

Dataset used to train dedyirama/qwen2.5-1.5b-indonesian-qlora