DragonData-FinMoE-8x3B (ROADMAP)

8 experts, ~3B active per token, one public finance brain. Built on the DragonData-FinDense-3B base: experts specialize by language/domain (incl. a CN expert), routed per sample. Same training budget as dense 3B, far more capacity.

Status: PLANNED (S5 — starts after flagship pretrain + distill). This repo is the public roadmap so the trajectory is auditable from day one.

Locked method

1. Dense 3B pretrains first (no SFT/DPO before MoE).
2. 8 experts freshly init on the dense base; MoE pretraining with sample-expert-masking (each sample routes to a subset; one permissive-license sub-domain per expert).
3. After MoE pretrain + evals: joint SFT → DPO (masking OFF, all experts + base update).
4. Defect scan → safetensors + GGUF → full token/GPU-hour records on this card.

Where to find things (lands at S5)

├── README.md ├── RUN.md ├── eval/ ├── checkpoints/ └── gguf/

Base: DragonData-FinDense-3B. Data: DragonData-Finance-Corpus.

License

Apache 2.0.