Clyx — 600M+ BASE
A new step in the Clyx project: a compact language model planned for training from scratch on Russian, English, and code.
Status: in development — weights not released. This card describes the current design and training plans, not a completed model. Architecture, data, and training settings may change following GPU tests. No benchmark scores or demonstrated language capabilities are claimed yet.
Overview
Clyx is an independent language-model project built around a custom decoder-only Transformer. The next planned BASE model targets approximately 635.51 million parameters, using Grouped-Query Attention, RoPE, RMSNorm, and SwiGLU.
The intended training approach is random initialization and next-token prediction, without importing external pretrained weights. The goal is a Russian–English base model with exposure to programming languages, while keeping the architecture and training process documented.
This is a BASE model project, not an instruction-tuned assistant. Any later chat or coding-assistant fine-tune would be a separate training stage and release.
Planned Architecture
The following values describe the current development configuration, not a released checkpoint.
| Parameter | Current design |
|---|---|
| Architecture | Custom decoder-only Transformer |
| Parameters | 635,512,320 with a 32,768-token vocabulary |
| Hidden size | 1,536 |
| Transformer layers | 24 |
| Query attention heads | 24 |
| Key/value attention heads | 4 |
| Attention type | Grouped-Query Attention (GQA) |
| Head dimension | 64 |
| MLP intermediate size | 4,096 |
| Positional encoding | RoPE, base 10,000 |
| Normalization | RMSNorm |
| Activation | SwiGLU |
| Configured context capacity | 2,048 tokens; not yet validated by training |
| Initial pretraining sequence length | 1,024 tokens, subject to GPU testing |
| Tokenizer | ByteLevel BPE |
| Target vocabulary | 32,768 entries, including special tokens |
| Input/output embeddings | Tied |
The final parameter count depends on the actual trained tokenizer vocabulary. Vocabulary size is independent of FP16/BF16 arithmetic precision. A configured context capacity is not evidence of reliable performance at that context length.
Languages and Data
The intended corpus will combine:
- Russian and English natural-language text.
- Python and C++ source code.
- Luau, where suitable data is available; Luau coverage is a goal, not a confirmed capability.
Dataset selection and mixture proportions are still being finalized. Dataset identifiers, pinned revisions, filtering details, usage restrictions, and the number of tokens actually consumed should accompany the eventual release. No final corpus size or language distribution is reported yet.
Training Plan
| Item | Current plan |
|---|---|
| Objective | Causal language modeling / next-token prediction |
| Initialization | From scratch |
| Optimizer | AdamW; fused implementation where supported |
| Learning-rate schedule | Cosine decay with warmup |
| Initial learning-rate range | 3e-4 → 3e-5; to be finalized for the actual run |
| Precision | Mixed precision: FP16 or native BF16, depending on hardware |
| Memory management | Gradient checkpointing and chunked language-model loss |
| Preliminary GPU testing | Google Colab, NVIDIA T4, FP16 |
| Main training environment | RunPod; GPU and final configuration not yet selected |
The T4 is intended for preliminary compatibility, memory, and checkpoint-resume tests. It is not being reported as the hardware used to train a released 635M model. Short synthetic tests are not model-quality evaluations.
Evaluation
Not yet evaluated. There are no reported validation-loss, perplexity, reasoning, or coding benchmark results for this planned model.
Future results should identify the exact checkpoint, dataset version, evaluation code, and settings. Results from earlier Clyx releases do not describe this model, and a larger parameter count alone does not establish better quality.
Availability and Usage
There are no downloadable weights or verified loading instructions for this planned model yet. This announcement cannot currently be used for inference.
A release is intended to include the trained weights, matching tokenizer, architecture configuration, training summary, and tested inference instructions. Compatibility with Transformers, inference providers, or existing ClyxBox releases should not be assumed before it is explicitly tested and documented.
The intended BASE release will be designed for text continuation, research, and further fine-tuning. It should not be presented as instruction-following or tool-using merely because the tokenizer includes conversation or tool-related special tokens.
Limitations
- Russian, English, and programming-language coverage are training goals, not verified capabilities.
- A base model is not automatically a reliable conversational assistant.
- Future generated text may contain factual errors, bias, offensive content, or memorized material; these risks have not yet been evaluated.
- Generated code may be incorrect or insecure and will require review and testing.
- No suitability for production or high-stakes decisions has been established.
Previous Release
Clyx 0.2 — 115.67M BASE is an earlier release in the Clyx project.
The planned 600M+ model is a new from-scratch training effort, not a fine-tune or weight expansion of Clyx 0.2. The earlier model's training-token count, validation loss, hardware, tokenizer, and usage examples do not apply to this development announcement.
Release License
The license for the future weights will be confirmed before release. No weight license is assigned by this announcement.
Built from scratch as part of the Clyx project. Training details and measured results will be documented when available.