summerAIO/sda-lm-ja

SDA-LM JA — pipeline-v2 candidate

Private experimental model, 164,767,776 parameters, 16 layers, d_model=512, 8 heads, context limit 1024 tokens. Not a production-ready assistant.

Original checkpoint revision: 8643ef9b257ab263a35d9672f09a1d90f7d863aa. This candidate was explicitly promoted to main by the repository owner. The previous main is preserved at backup-main-before-pipeline-v2-20261031-014354.

Training and implementation

Merged rank-16 LoRA (scale=2), learning rate 2e-4, short-answer English/Japanese Dolly instruction data. Validation-selected checkpoint: step 350. All adapters are merged: no PEFT installation or external adapter is needed.

Custom SDA/full-attention stack. Output logits reuse the embedding weight (there is no separate lm_head). The tokenizer is llm-jp-derived; this does NOT transfer the upstream large model's knowledge.

Includes fp32 delta-state algebra under autocast, attention-mask-aware padded rows, and explicit no-cache generation. These fixes passed fp32 logit equivalence, padding and causality checks. KV/SDA archive caches are not implemented; generation recomputes the prefix each step.

Loading

Custom Python code executes when trust_remote_code=True; review it before use. Supply your own Hugging Face token through your environment or HF login for this private repository.

import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = 'summerAIO/sda-lm-ja'
revision = 'main'
kwargs = dict(revision=revision, token=os.environ['HF_TOKEN'], trust_remote_code=True)
tok = AutoTokenizer.from_pretrained(repo, **kwargs)
model = AutoModelForCausalLM.from_pretrained(repo, **kwargs).eval()
if torch.cuda.is_available():
    model = model.cuda()
tok.padding_side = 'left'
text = tok.apply_chat_template(
    [{'role': 'user', 'content': '日本の首都はどこですか？'}],
    tokenize=False, add_generation_prompt=True,
)
x = tok(text, return_tensors='pt', add_special_tokens=False).to(model.device)
with torch.inference_mode():
    y = model.generate(**x, max_new_tokens=48, do_sample=False, use_cache=False)
print(tok.decode(y[0, x.input_ids.shape[1]:], skip_special_tokens=True))

The inherited chat template is a single-turn instruction formatter: assistant history and system messages are not faithfully represented. Use a single user message. For generation use left padding; masked batch rows are processed separately and may be slow. Explicit use_cache=True is rejected. Context including generated tokens must stay within 1024 tokens.

Evaluation — honest limitations

All scores below use candidate-answer mean token log-likelihood, not free-form answer accuracy. Zero-shot, repository chat template. Not identical to lm-evaluation-harness acc_norm and not directly leaderboard-comparable.

Two candidates were selected using a 64-question development set (32 Japanese + 32 English), excluding the previous audit sample. Development correct: original 27/64, selected candidate 35/64; other candidate 32/64.

Only AFTER selection, the selected candidate was evaluated once on a separate, disjoint 64-question final subset:

Final subset	Original	Candidate
JCommonsenseQA validation (32)	12/32 (37.50%)	14/32 (43.75%)
ARC-Easy test (32)	8/32 (25.00%)	9/32 (28.13%)
Total	20/64 (31.25%)	23/64 (35.94%)

Three extra correct answers are insufficient to establish a statistically reliable general improvement. Small samples have wide confidence intervals. Datasets were not used for training this trial; unknown original pretraining contamination cannot be excluded. These now-observed final questions should not be reused as a pristine final set in future optimization.

General instruction validation NLL: 6.06977 -> 5.30144 on 32 short-answer examples (mean per-example token NLL). The reused 22-question strict free-generation development suite remains 0/22 -> 0/22. Thus lower loss and modest multiple-choice gains do NOT demonstrate a useful chat assistant.

A separate, in-memory single-example overfit diagnostic succeeded; those diagnostic-trained weights were NOT saved or included.

License and use

The original repository specifies license: other; no new license grant is made. Confirm permitted use with the owner and check applicable upstream tokenizer and dataset terms (including Japanese Dolly and JCommonsenseQA attribution/share-alike requirements). This is not a representation that all upstream license obligations have been audited.

Research only. Repetition, wrong arithmetic, hallucinations and poor instruction following persist. Not for high-risk medical/legal/financial decisions. Evaluation reports and hf-smoke-test.json are included for reproducibility.