Frosya-2-Fast (LGTT-Mini 40M)

The second generation of local Artificial Intelligence developed by Fortaki (DTF). Moving away from traditional dense architectures, Frosya-2-Fast utilizes a modern MoE (Mixture of Experts) architecture, fine-tuned for high efficiency, code generation, and complex logical reasoning on lightweight client hardware.

By shrinking the vocabulary size and adopting advanced Transformer components (RoPE, SwiGLU), this 40-million parameter engine dramatically outperforms its predecessor in programming syntax, mathematics, and natural dialogue.

📊 Technical Specifications

• Architecture: LGTT Mixture of Experts (MoE) + SwiGLU + RoPE
• Layer Configuration: 8 layers, 8 attention heads, hidden dimension embed_dim = 256
• MoE Routing: 4 Experts total, Top-2 activated per token (num_experts = 4, top_k = 2)
• Parameter Count: ~40 million active parameters
• Dataset Size: 650 MB of highly cleaned and filtered text data
• Tokenizer: Custom compressed Byte-Pair Encoding (BPE) optimized for Russian/English balance
• Vocabulary Size: 8,229 tokens (Engineered for high training data density)
• Context Window: 2,048 tokens (Full-scale context support with long-term retention)

📁 Project Structure

├── LGTT_mini.py           # Core MoE engine architecture (NumPy-based execution)
├── chat.py          # Google Colab inference CLI with sampling heuristics
├── lgtt_tokenizer.json    # Compressed 8k vocabulary matrix configuration
└── micro_lgtt_40m.bin     # Pre-trained PyTorch weights checkpoint (Convertible to .pkl)

⚙️ Architectural & Logic Enhancements

Compared to the 1st generation, Frosya-2-Fast introduces massive Data Science upgrades:

1. True 2048 Context Support: Thanks to Rotary Position Embeddings (_apply_rope), the model natively processes long-form prompts up to 2048 tokens without losing coherence, making it highly capable of reading full scripts and long math problem descriptions.
2. Vocabulary Optimization: Reducing the token count from 50k to 8,229 forced the 40M parameters to focus entirely on core syntax, morphology, and high-probability connections instead of wasting weight capacity on rare words.
3. Sparse MoE Execution: The routing gate feeds tokens exclusively to the best 2 out of 4 specialist experts, mimicking the behavioral benefits of a much larger language model while keeping memory footprints extremely low.
4. Context Heuristics: Features runtime repetition penalties (scaled dynamic logits) and a strict Top-K sampling filter (top_k_sampling = 6) combined with a conservative temperature (0.2) to eliminate code-looping bugs.

🛠️ Quick Start (Google Colab / Local CLI)

1. Place your model weights file (micro_lgtt_40m.bin or micro_lgtt_40m.pkl) and lgtt_tokenizer.json in the root workspace directory.
2. Execute the inference assistant launcher:

   python chat.py
   

3. Type your query into the console prompt. System prompts adapt automatically based on keywords (e.g., triggering specialized code constraints when asking for Python functions).

---

Developed by Fortaki. Optimized for the DTF AI-dev community. Fast, adaptive, and running locally.