orcheflow/Purebase-Instruct-MoE-1B

JunsuAI-Purebase-Instruct-MoE (1B)

This is a Korean-Specialized Mixture-of-Experts (MoE) model based on Cerebras-256M.

Architecture

• Base: Cerebras-GPT-256M (Upcycled)
• Structure: 8 Experts, Top-2 Gating
• Params: ~1B (Total activation)
• Vocab: Custom Korean Tokenizer (32k)

Training Stages

1. Tokenization & Embedding Adaptation: Adapted to Korean vocabulary.
2. MoE Upcycling & Pre-training: Trained on 2B tokens (mixed) with Aux Loss.
3. SFT & Annealing: Fine-tuned on Balanced Dataset (KULLM-v2 + 10x Dolly-15k) to ensure multilingual capability.

Usage

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
# Note: Requires custom MoE layer definition or `trust_remote_code=True` if supported in future.
# See repo for modeling code.