Model Description
MzModelV1 is a decoder-only Transformer language model built from scratch using PyTorch and Hugging Face Transformers. It was created as an experiment to explore language model training on consumer hardware in a home lab environment.
The project demonstrates that it is possible to train a language model with approximately 110 million parameters on a single consumer GPU.
Model Details
| Property | Value |
|---|---|
| Model name | MzModelV1 |
| Architecture | Decoder-only Transformer |
| Model type | mz_model_v1 |
| Parameters | 109.65 million |
| Hidden size | 768 |
| Number of layers | 12 |
| Attention heads | 12 |
| Feed-forward dimension | 3,072 |
| Vocabulary size | 32,000 |
| Maximum sequence length | 512 tokens |
| Positional encoding | Learned positional embeddings |
| Attention mechanism | Causal multi-head self-attention |
| Activation function | GELU |
| Normalization | LayerNorm |
| Weight tying | Input embeddings and output projection |
| Training hardware | NVIDIA RTX 5060 Ti (16 GB VRAM) |
| Training time | Approximately 38 hours |
| Pretraining dataset | Wikipedia Structured (7.6 million articles) |
| Fine-tuning dataset | SmolTalk (20,000 samples) |
| Framework | PyTorch, Hugging Face Transformers |
Training
The model was trained on a single NVIDIA RTX 5060 Ti with 16 GB of VRAM in approximately 38 hours.
The training process consisted of two stages:
- Pretraining: Wikipedia Structured dataset containing approximately 7.6 million articles.
- Fine-tuning: 20,000 samples from the SmolTalk dataset to improve the model's conversational abilities.
The goal of this project is to demonstrate that training a language model from scratch and fine-tuning it afterward is achievable in a home lab, without enterprise-grade hardware or a large GPU cluster.
MzModelV1 is an experimental project focused on learning, experimentation, and making language model development more accessible on consumer hardware.
Inference
Use inference script with GUI from https://github.com/marekzel/mz-model-v1
Disclaimer
This model is experimental and provided "as is" for research purposes only. The authors make no guarantees regarding its performance, safety, or accuracy. Use at your own risk.
- Downloads last month
- -