Model Card for Kiseki-1.1-0.3B

An updated iteration of Kiseki-0.3B with a revised dataset mix, yielding substantial improvements across EN↔ZH and JA↔EN directions.

Note: Kiseki-1.1 is not fine-tuned from Kiseki-0.3B; it is trained independently with a different dataset configuration.

---

Model Details

Model Description

• Developed by: telecomadm1145
• Model type: Transformer-Mamba2 Seq2Seq
• Language(s) (NLP): Japanese (ja), Chinese (zh), English (en)
• License: MIT
• Parameters: 0.3B

---

⚠️ Important: UL2 Paradigm Token Required

Due to an oversight during seq2seq fine-tuning, the encoder was trained with a UL2 paradigm sentinel token prepended to every input. You must prepend token ID 65662 to the encoder input_ids at inference time. Omitting it will produce significantly degraded output.

ul2_token_id = 65662
input_ids = tokenizer.encode("<|2zh|>" + text + "<eos>")
input_ids = [ul2_token_id] + input_ids  # required prepend

---

How to Get Started

Prompt format: <|2zh|>/<|2ja|>/<|2en|>{source text}<eos>

import torch
from transformers import AutoModelForSeq2SeqLM, PreTrainedTokenizerFast

repo_id = "telecomadm1145/Kiseki-1.1-0.3B"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=torch.float32
)
model.to(device)

text = "おはようございます。"

ul2_token_id = 65662
input_ids = [ul2_token_id] + tokenizer.encode("<|2zh|>" + text + "<eos>")
input_tensor = torch.tensor([input_ids]).to(device)

output_ids = model.generate(input_tensor, max_new_tokens=256)
result = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(f"Translation: {result}")

---

Training Data

Dataset	Direction
telecomadm1145/ccmatrix_clean_en_ja	ja ↔ en
telecomadm1145/sakuragpt_synthetic_ja_zh	ja ↔ zh
telecomadm1145/ccmatrix_clean_en_zh	zh ↔ en

---

Metrics

Table 1: Multi-Model Comparison (ja → zh)

Metric	opus-mt-ja-zh (~73M)	NanoSakura-2-0.2B	NanoSakura-2.2-0.2B	NanoSakura-2.3-0.2B	NanoSakura-0.3B	nllb-200-1.3B	Qwen3-0.6B (fp16)	Qwen3-0.6B (fp16, thinking)	Qwen3-1.7B (fp16)	Qwen3-1.7B (fp16, thinking)	Kiseki-0.3B	Kiseki-1.1-0.3B
FLORES-200 BLEU	25.67	23.27	26.55	28.67	22.36	20.87	12.58	21.13	27.12	27.56	22.51	23.47
FLORES-200 spBLEU	-	-	20.08	22.02	16.13	-	-	-	-	-	16.09	16.88
FLORES-200 COMET	0.8371	0.8380	0.8494	0.8563	0.8307	0.7805	0.8020	0.8220	0.8561	0.8571	0.8272	0.8416
shard_00134 BLEU	8.07	58.13	57.55	49.32	58.71	5.73	6.89	14.57	23.37	24.60	53.29	54.07
shard_00134 spBLEU	-	-	54.12	45.10	55.57	-	-	-	-	-	49.75	50.36
shard_00134 COMET [Note 1]	-	-	0.9016	0.8899	0.9044	-	-	-	-	-	0.8910	0.8951

[Note 1] Methodology Variance in SHARD-00134 COMET Evaluation: Evaluated on SHARD-00134 at the sentence level (the 200 raw document pairs were split into 3,197 individual sentences). Sentence-level evaluation reduces length penalties and aligns semantic embeddings more tightly, resulting in a higher absolute score than document-level evaluation.

---

Table 2: Kiseki-0.3B vs. Kiseki-1.1-0.3B — All Directions

Values shown as 0.3B / 1.1-0.3B; bold marks the better score per cell.

Dataset	Direction	SacreBLEU	spBLEU	COMET (Ref)	COMET-QE (Ref-less)
FLORES-200_JA-ZH	ja → zh	22.51 / 23.47	16.09 / 16.88	0.8272 / 0.8416	0.7894 / 0.8061
FLORES-200_ZH-JA	zh → ja	14.99 / 16.17	13.49 / 14.20	0.8479 / 0.8636	0.7738 / 0.7950
FLORES-200_JA-EN	ja → en	14.35 / 18.58	15.30 / 21.27	0.7620 / 0.8268	0.7564 / 0.8142
FLORES-200_EN-JA	en → ja	21.73 / 24.28	17.61 / 21.40	0.8179 / 0.8625	0.7868 / 0.8317
FLORES-200_ZH-EN	zh → en	11.01 / 20.31	11.91 / 23.30	0.6655 / 0.8226	0.6467 / 0.8012
FLORES-200_EN-ZH	en → zh	21.28 / 31.17	14.21 / 26.28	0.7017 / 0.8190	0.6733 / 0.7922
SHARD-00134_JA-ZH	ja → zh	53.29 / 54.07	49.75 / 50.36	0.8910 / 0.8951	0.6873 / 0.6915
SHARD-00134_ZH-JA	zh → ja	32.85 / 33.32	30.18 / 30.83	0.9070 / 0.9085	0.7270 / 0.7267