junwatu/candi-sailor2-8b-lora

Candi Sailor2 8B LoRA

candi-sailor2-8b-lora is a LoRA adapter for sail/Sailor2-8B, fine-tuned for Indonesian cultural heritage, temples (candi), and Javanese history.

This repository contains the adapter weights, tokenizer files, chat template, and training arguments. It is not a standalone full model. To use it, load the Sailor2 base model first, then apply this adapter with PEFT.

What This Adapter Is For

The adapter is trained to improve the behavior of a candi-focused assistant, especially:

• answering in concise Indonesian
• explaining Indonesian and Javanese cultural heritage topics
• handling candi records with uncertain geographic validation
• avoiding overconfident claims when geo_flag is not ok
• preserving uncertainty across multi-turn conversations

For exact factual answers, use this model with RAG or another trusted source. The adapter should guide style and behavior; it should not be used as the only database for coordinates, addresses, elevation, or validation status.

Related Repositories

Repo	Purpose
junwatu/candi-sailor2-8b-lora	This LoRA adapter
junwatu/candi-sailor2-8b	Full merged Transformers model
junwatu/candi-sailor2-8b-gguf	GGUF quantized local inference release

Repository Contents

File	Description
adapter_model.safetensors	LoRA adapter weights
adapter_config.json	PEFT LoRA configuration
tokenizer.json	Tokenizer file
tokenizer_config.json	Tokenizer settings
chat_template.jinja	ChatML-style template
training_args.bin	Saved trainer arguments

Adapter Configuration

Setting	Value
PEFT type	LoRA
Task type	Causal language modeling
Rank r	16
LoRA alpha	32
LoRA dropout	0.05
Bias	none
Base model	sail/Sailor2-8B

Target modules:

q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Setup

The project training notes describe the setup as:

Parameter	Value
Method	SFT with TRL
Training file	candi_chat_1k.jsonl
Epochs	2
Learning rate	2e-4
Batch size	1
Gradient accumulation	8
Max sequence length	2048
Training quantization	4-bit NF4
Optimizer	paged_adamw_8bit
Scheduler	cosine with warmup
GPU	L40S

Quick Start

Install dependencies:

pip install torch transformers peft accelerate

Load the base model and apply the adapter:

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "sail/Sailor2-8B"
adapter_id = "junwatu/candi-sailor2-8b-lora"

tokenizer = AutoTokenizer.from_pretrained(
    adapter_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

messages = [
    {
        "role": "system",
        "content": "You are Candi, a helpful assistant specialized in Indonesian cultural heritage, temples (candi), and Javanese history. Answer in clear Indonesian.",
    },
    {
        "role": "user",
        "content": "Apa itu Candi Borobudur?",
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
    )

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)

print(response.strip())

Merge The Adapter

If you want a standalone Transformers model, merge the adapter into the base model:

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "sail/Sailor2-8B"
adapter_id = "junwatu/candi-sailor2-8b-lora"
output_dir = "candi-sailor2-8b-merged"

tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(model, adapter_id)
model = model.merge_and_unload()

model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)

Example Prompts

Jelaskan Candi Borobudur dengan bahasa sederhana.

Apa perbedaan candi Hindu dan candi Buddha?

Kalau data lokasi candi punya geo_flag reverse_geo_needs_review, apakah aman ditulis valid tanpa catatan?

Ceritakan legenda Roro Jonggrang secara singkat.

Evaluation Context

Before fine-tuning, the local Sailor2 + RAG baseline was strong for factual retrieval but weaker on multi-turn uncertainty handling.

Observed pre-fine-tune behavior baseline:

Category	Pass rate
Indonesian style	100.00%
Geo-flag discipline	83.33%
Multi-turn consistency	50.00%
Unknown candi refusal	100.00%
Overall behavior	72.73%

The fine-tuning target was behavior improvement, not memorizing facts. For production use, pair the model with retrieval from trusted candi records.

Limitations

• This is a LoRA adapter, not a complete model.
• It requires the correct Sailor2 base model.
• It may still hallucinate if used without RAG.
• It should not be trusted as the sole source for exact locations, coordinates, or historical facts.
• It is specialized for Indonesian candi and cultural-heritage assistance, not general-purpose expert reasoning.

License

Apache 2.0, following the Sailor2 base model license.

Citation

@misc{junwatu2026candi_sailor2_lora,
  title={Candi Sailor2 8B LoRA},
  author={junwatu},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/junwatu/candi-sailor2-8b-lora}
}

@article{sailor2,
  title={Sailor2: Advancing Multilingual Large Language Models for Southeast Asian Languages},
  author={Sea AI Lab},
  journal={arXiv preprint arXiv:2502.12982},
  year={2025}
}