About Aether
Aether is a small language model built and trained from scratch to explore the end-to-end language-model development process—from Transformer architecture and tokenization to pretraining, supervised fine-tuning, checkpoint merging, and evaluation.
Aether uses a decoder-only Transformer with grouped-query attention, rotary positional embeddings, RMSNorm, and SwiGLU feed-forward layers. The project also includes experiments with LoRA fine-tuning and explicit end-of-sequence (EOS) supervision.
Research status: Aether is an experimental project, not a production-ready assistant. Generated responses may be repetitive, incoherent, incomplete, or incorrect.
Model specifications
| Specification | Details |
|---|---|
| Architecture | Decoder-only Transformer |
| Parameters | 100,092,672 (~100M) |
| Transformer layers | 12 |
| Hidden dimension | 768 |
| Query attention heads | 12 |
| Key/value heads | 4 |
| Attention | Grouped-query attention (GQA) |
| Head dimension | 64 |
| Feed-forward network | SwiGLU |
| Feed-forward dimension | 2,048 |
| Normalization | RMSNorm |
| Positional encoding | RoPE |
| Vocabulary | 32,000 tokens |
| Configured context length | 2,048 tokens |
| Embeddings | Tied input/output embeddings |
| Tokenizer | SentencePiece |
| Training objective | Autoregressive next-token prediction |
Training
Pretraining
Aether was pretrained with the autoregressive next-token prediction objective.
| Metric | Result |
|---|---|
| Tokens processed | 1,800,011,776 |
| Training duration | Approximately 11.31 hours |
| Average throughput | Approximately 20,379 tokens/second |
| Best checkpoint step | 13,600 |
| Tokens processed at best checkpoint | 1,782,579,200 |
| Best recorded validation loss | 4.3108 |
Fine-tuning experiments
- SFT1: A 20,000-example subset of UltraChat 200K; one epoch using LoRA.
- SFT2: 4,844 mixed-domain examples covering general instructions, Python, SQL, technical questions, and formatting.
- SFT3: 12,000 prepared examples spanning coding, general instructions, mathematics, and formatting.
- EOS experiment: Fine-tuning from SFT1 with explicit EOS tokens appended to assistant messages during training.
The LoRA experiments used rank 16, alpha 32, dropout 0.05, and targeted query, key, value, and output attention projections.
Evaluation
Base-model benchmarks
| Benchmark | Result |
|---|---|
| HellaSwag | 26.59% |
| ARC-Easy | 39.18% |
Held-out instruction-tuning evaluation
All checkpoints were evaluated on the same held-out set of 500 UltraChat examples, containing 442,453 supervised assistant tokens.
| Checkpoint | Token loss ↓ | Perplexity ↓ | Token accuracy ↑ |
|---|---|---|---|
| Base model | 6.8429 | 937.16 | 2.29% |
| SFT1 | 2.9120 | 18.39 | 44.41% |
| SFT2 | 2.9253 | 18.64 | 44.19% |
| SFT3 | 2.9664 | 19.42 | 43.69% |
| SFT1 + EOS supervision | 2.8229 | 16.83 | 45.64% |
The EOS-supervision checkpoint achieved the best token loss, perplexity, and token accuracy among the evaluated checkpoints. These metrics measure token prediction on this particular held-out dataset and do not establish general assistant quality.
Architecture highlights
- Grouped-query attention: 12 query heads share 4 key/value heads.
- Rotary positional embeddings: encodes token positions in attention.
- RMSNorm: normalizes hidden representations.
- SwiGLU: used in the feed-forward sublayers.
- Tied embeddings: shares input and output token embeddings.
- SentencePiece: uses a 32,000-token vocabulary.
Limitations
Generation tests involving Python code, prime-number functions, and process/thread explanations showed repetition, incomplete code, and incoherent passages. In the short tests performed, the EOS experiment did not eliminate these issues.
Aether should be treated as a learning and research artifact. It is not recommended for production chat systems, high-stakes decisions, security-critical code generation, or tasks requiring dependable factual accuracy.
Intended use
Aether is intended for:
- Learning and experimenting with small language models.
- Studying decoder-only Transformer architecture and training.
- Exploring LoRA fine-tuning and checkpoint merging.
- Investigating EOS supervision, decoding, and repetition.
- Educational demonstrations and reproducible research.
Repository contents
The release should include the files required by the chosen checkpoint, such as:
model.safetensors— model weights.config.json— architecture configuration.tokenizer/— tokenizer assets.model/— custom architecture implementation, if included.README.md— this model card.banner.png— repository banner shown above.
The exact contents depend on the files uploaded. Verify that the weights, configuration, tokenizer, and implementation are mutually compatible.
Loading and inference
Aether uses a custom Transformer implementation and is not guaranteed to load directly through AutoModelForCausalLM without a compatible model implementation.
For inference, use the matching architecture code and tokenizer, load model.safetensors into the model, verify tensor-key compatibility, encode the prompt, and generate tokens autoregressively. Stop when the actual EOS token ID is produced or the generation limit is reached.
License and acknowledgements
The redistribution and usage license for this release still needs to be confirmed. The license: other metadata is intentionally conservative until the release license is selected.
Before redistribution, review the licenses and terms of the training datasets, tokenizer assets, and third-party code. Do not assume every source dataset permits unrestricted redistribution or commercial use.
Aether · Exploring the engineering of small language models
- Downloads last month
- 51