Frosya-1 (Micro-GPT 76M)

The first generation of local Artificial Intelligence developed by Fortaki (DTF). This is an ultra-lightweight language model of the Micro-GPT class, trained completely from scratch (from scratch) on a custom Russian text dataset. The project was built as a core component of the private Frosya (FreeSearch) desktop browser ecosystem.

The main philosophy of the project is to provide a fully functional local AI assistant that runs on absolutely any hardware without needing an internet connection or external APIs.

📊 Technical Specifications

• Architecture: GPT-2 LM Head (16 layers, 8 attention heads, hidden dimension n_embd = 512)
• Parameter Count: ~76.45 million (saved in FP32 format)
• Model Weights File Size: ~298.6 MB (micro_gpt_170m.bin)
• Dataset Size: 500 MB of cleaned text (~150+ million tokens)
• Tokenizer: Custom BPE (GPT2Tokenizer structure) optimized for the Russian language
• Vocabulary Size: 50,257 tokens
• Context Window: 512 tokens (strictly managed via rolling memory buffer)

📁 Project Structure

├── .venv/                 # Python virtual environment
├── tokenizer/             # Custom BPE tokenizer matrix files
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   └── vocab.json
├── micro_gpt_170m.bin     # Model weights file (76M parameters in FP32)
└── chat.py                # Text inference script & context manager

⚙️ Core Implementation Details

Small language models (under 1B parameters) are highly prone to rapid context loss, hallucinations, and infinite loops. To maximize the output quality of the 76M weights, several clever heuristics are implemented directly in chat.py:

1. Environment Bug Isolation: The script explicitly bypasses torchvision modules to prevent critical import errors with torch on newer environments (such as Python 3.13+).
2. Hardcoded System Prompt Injection: The core instruction (Information: The user's name is Arsen...) is hard-glued to the very beginning of the context array on every generation step, preventing the model from "forgetting" its primary directives.
3. Sliding Context Window: The algorithm automatically discards (.pop(0)) the oldest Q&A pairs from the dialogue history as soon as the total length approaches the strict 512-token limit, keeping the system prompt intact.
4. Conservative Sampling: The generation temperature is set to a low 0.4 to keep responses coherent and deterministic, while a repetition_penalty=1.1 effectively mitigates phrase-looping bugs.

🛠️ Quick Start

1. Ensure you have Python installed (Python 3.11 - 3.13 recommended).
2. Activate the pre-configured virtual environment:

   # For Windows
   .venv\Scripts\activate
   # For Linux/macOS
   source .venv/bin/activate
   

3. Install the core requirements (if setting up a new environment):

   pip install torch transformers
   

4. Initialize the local chat:

   python chat.py
   

---

Developed by Fortaki for the DTF indie-dev community. It might be small, but it runs entirely on your own terms!