Haru v1.1

A newer release is available. Haru v2.0 is the current model: 17.0M parameters, and the first Haru to score above chance on KoBEST (mean 0.469 against a chance mean of 0.450, where v1.1 scores 0.442).

v2.0 is not a drop-in replacement. It is a different architecture and a different tokenizer:

v1.1 reuses three decoder cells twice; v2.0 has six independent cells.

v2.0 uses a separate 12k BPE tokenizer trained on a different corpus, so loss and perplexity are not comparable between the two.

v1.1 supports recurrent depths 2, 4 and 6. v2.0 runs at depth 6 only. If you rely on shallow-depth inference, stay on v1.1.

This repository remains available for reproducibility and existing users.

Haru is a compact Korean story continuation model built with the custom CFRD causal architecture. It has 11,634,459 parameters and supports recurrent inference depths 2, 4, and 6.

Usage

Review the included Python files before enabling remote custom code.

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "gaon12/haru_1.1"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, trust_remote_code=True)

inputs = tokenizer("작은 마을에 아침이 찾아왔어요.", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    top_k=40,
    repetition_penalty=1.08,
    use_cache=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Model details

Parameters: 11,634,459
Context length: 512
Recurrent depths supervised during training: (2, 4, 6)
Exported checkpoint step: 5750
Training tokens seen: 753,664,000

Evaluation

Recurrent depth	Validation loss	Perplexity
2	2.95689	19.238
4	2.12258	8.353
6	1.92052	6.825

Training data attribution

Tiny-Ko-Stories by psymon, licensed under CC BY 4.0. The dataset is not redistributed with this model.

Limitations

Haru is a continuation model, not an instruction-following assistant.
Longer generations can repeat ideas or drift between entities.
The model is not suitable for factual or safety-critical use.
There is no inference cache yet, so generation recomputes the active context.

License

Haru model weights and included code are released under the MIT License. The training dataset remains under its separate CC BY 4.0 license.

Downloads last month: 369

Safetensors

Model size

12.7M params

Tensor type

F32

BOOL

gaon12
/

haru_1.1