Wader-100M 🐟

Wader-100M is a lightweight (~110M parameter) conversational Indonesian language model implementing the DeepSeek-V4 architecture (MLA, MoE, Hyper-Connections). It is trained from scratch on Indonesian corpus and instruction-tuned (SFT) to act as a responsive and natural chat assistant.

For the pre-trained base model, see neosantara/wader-100m-base.

Architecture & Innovations

This model implements key DeepSeek-V4 innovations at a miniature scale for extreme edge efficiency:

• Parameters: ~110M total
• Multi-head Latent Attention (MLA): Compressed KV cache via latent projections.
• Mixture of Experts (MoE): Sparse activation — only 2 of 4 routed experts active per token.
• Hyper-Connections: Multi-copy hidden states with learned Sinkhorn routing.
• Context Length: 2,048 tokens (Vocab: 129,280).

Training Pipeline

1. Pretraining (5,000 steps):
   • Trained on eli5_id (70%) and twitter_indonesia_sarcastic (30%).
   • Achieved 97.81% token accuracy.
2. Supervised Fine-Tuning (3,000 steps):
   • Trained on intisari-indonesian-chat-v4 (50,000 multi-turn dialogs) + Indonesian slang/formalization lexicons.
   • Final SFT Loss: 2.27.
   • Perplexity: 139.68.
   • Chat Template: Native DeepSeek-V4 (<｜begin▁of▁sentence｜>, <｜User｜>, <｜Assistant｜>, <｜end▁of▁sentence｜>).

Usage

import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

repo_id = "neosantara/wader-100m"

# Load config and model
config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_config(config, trust_remote_code=True).float()

# Download and load weights
weights_path = hf_hub_download(repo_id, "model.safetensors")
state_dict = load_file(weights_path)
model.load_state_dict(state_dict, strict=True)
model = model.cuda().eval() if torch.cuda.is_available() else model.eval()

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)

# Generate conversational response
messages = [
    {"role": "user", "content": "Bro, ada rekomendasi tempat nongkrong asik ga di Jakarta?"}
]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=100,
        temperature=0.7,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
    )

# Slice the generated tokens (exclude input prompt)
generated_tokens = output[0][input_ids.shape[1]:]
print(tokenizer.decode(generated_tokens, skip_special_tokens=True))

Limitations

• Small Capacity: At 110M parameters, the model's factual knowledge is limited. It is best used for conversational flow, translation, and edge API experimentation rather than complex reasoning.
• Custom Architecture: Requires trust_remote_code=True to run correctly.

License

Apache-2.0