Talos Tiny — 254K
An experimental language model built from scratch by Promethean Studios.
Talos is an open-source language-model research project focused on exploring how capable AI systems can be built from the ground up.
Talos Tiny is the first experimental checkpoint in the project. At just 254,272 parameters, it is not intended to compete with modern large language models. Instead, it serves as a small, reproducible testbed for developing and validating Talos's architecture, tokenizer, and training pipeline.
Model Details
| Property | Value |
|---|---|
| Parameters | 254,272 |
| Architecture | Dense Transformer |
| Hidden size | 64 |
| Layers | 2 |
| Attention heads | 4 |
| KV heads | 2 |
| Vocabulary size | 1,024 |
| Maximum sequence length | 512 |
| Training hardware | NVIDIA Tesla T4 |
| Training steps | 200 |
Initial Training Results
The first end-to-end training experiment successfully demonstrated that Talos can learn from a training corpus.
| Metric | Result |
|---|---|
| Initial loss | 6.9177 |
| Final loss | 0.2117 |
| Loss reduction | 6.7060 |
| Training steps | 200 |
| Result | PASS |
The model's loss decreased substantially during training, confirming that the forward pass, backward pass, optimizer, and training pipeline were functioning together.
What This Demonstrates
- Model initialization
- Forward propagation
- Backpropagation
- Gradient-based optimization
- GPU training
- Loss calculation
- Learning from training data
- End-to-end training
Dataset
The experimental training data was derived from:
OpenAssistant/oasst1
The dataset was selected because it contains human-generated conversational examples and provides a useful starting point for experimenting with language-model training.
Please review the dataset's licensing and usage terms before redistributing derivative artifacts.
Current Status
Experimental / Early Research
Talos Tiny is a very small prototype. It should not be expected to produce coherent general-purpose conversation.
The project is currently focused on building and validating the underlying infrastructure before significantly increasing model size.
Known Issues
The current tokenizer is still under development.
Early testing identified an issue where certain trained-tokenizer configurations do not correctly preserve ordinary text during encoding/decoding. A tokenizer rebuild and memory optimization are currently being investigated.
The project therefore considers the current checkpoint an engineering milestone rather than a production language model.
What's Next
The next stages of Talos development include:
- Fix and optimize the tokenizer
- Improve tokenizer memory efficiency
- Establish reproducible training benchmarks
- Train a larger prototype
- Improve conversational training
- Add standardized evaluation
- Improve inference and generation tooling
- Release future Talos checkpoints
Reproducibility
Talos is open source and the training code is available in the project's GitHub repository.
The goal of this release is not simply to provide a model checkpoint, but to document the process of building Talos from the ground up.
About Promethean Studios
Promethean Studios is an independent technology project focused on exploring accessible, locally deployable AI systems.
Talos is our ongoing language-model research project.
This is only the beginning.
Disclaimer
Talos Tiny is an experimental research model. It has not undergone extensive safety evaluation, factuality evaluation, or standardized language-model benchmarking.
Outputs may be incorrect, nonsensical, repetitive, or otherwise unreliable.
Use for research and experimentation.