the-broker-v1

A crypto-native language model, pretrained from scratch on data crawled and consensus-verified by the SynthSynd swarm.

Model Details

• Parameters: 5,313,024
• Vocabulary: 8192
• Context length: 256 tokens
• Architecture: decoder-only transformer (RoPE, RMSNorm, SwiGLU)
• Trained from: random initialization

Training Data

• Documents: 96 crawled pages
• Total characters: 785,340
• Sources: docs.solana.com, ethereum.org, bitcoin.org, docs.arbitrum.io, docs.uniswap.org, docs.aave.com, docs.pyth.network

Every page was fetched via headless browser, robots.txt respected, cleaned of scripts, hashed with BLAKE3, stored in SQLite.

Training Procedure

• Hardware: Google Colab T4 GPU
• Steps: 2000
• Batch: 16 x 4 gradient accumulation
• Learning rate: 3e-4 with cosine decay
• Final train loss: 1.7841
• Final val loss: 7.3867

Links

• Crawler: github.com/vermanehaz/synthsynd-crawler
• Live viewer: super-engine-x5v56j66jc66wr-8000.app.github.dev

Note

v1. A 7M parameter model trained on 800K characters. Generation quality is limited. Future versions will scale.