mrfukk7/ruto-keyboard-ru-65m

RUTO Keyboard LM 🇷🇺⚡ (RU + FUTO = RUTO)

RUTO Keyboard LM is an on-device, privacy-preserving 65M parameter neural language model engineered specifically for Cyrillic (Russian) text input on mobile devices, designed for the FUTO Keyboard ecosystem.

• Base Architecture: LLaMA (12 layers, 8 attention heads, 512 hidden dimension, SwiGLU MLP)
• Vocabulary: 24,000 tokens (SentencePiece BPE with suffix whitespace: treat_whitespace_as_suffix=True)
• Supported Layouts: ЙЦУКЕН (Russian), Russian Phonetic
• Supported Quantizations: GGUF v2 (Q8_0, FP16), PyTorch (safetensors, bin)
• Compatibility: Android FUTO Keyboard v1.0.2+ / C++ JNI (LanguageModel.cpp)
• Companion APK: futo-keyboard-ru-1.0.2.apk (GitHub Release)

---

Model Architecture & Hyperparameters

The model is built on a compact, latency-optimized LLaMA decoder-only transformer designed to run in real-time on low-power mobile CPU cores (e.g. ARM Cortex-A55) with minimal battery impact and zero network access.

Hyperparameter	Value	Description
Parameters	64.9 Million	Optimized for mobile RAM budget
Hidden Size ($d_{model}$)	512	Embedding and hidden state dimension
Intermediate Size	1,376	SwiGLU feed-forward projection size
Layers (Blocks)	12	Transformer decoder layers
Attention Heads	8	Multi-head self-attention
Head Dimension	64	Dimension per attention head
Positional Embedding	RoPE	Rotary Position Embedding
Normalization	RMSNorm	Root Mean Square layer normalization ($\epsilon = 10^{-5}$)
Context Length ($seq_len$)	128	Tailored to mobile typing windows (prevents memory bloat)
Activation Function	SwiGLU	Gated linear unit activation
RAM Footprint (Android)	~92 MB	Fully resident in RAM without swapping
Inference Latency	< 18 ms	Evaluated on Snapdragon 865 / 778G CPU cores

---

🔬 Core Innovations & Bug Fixes

1. The Suffix-Whitespace Tokenizer Fix (Solving Syllable Fragmentation)

In standard NLP workflows, SentencePiece models prepend whitespace prefixes to subword tokens (\xe2\x96\x81слово / ▁слово).
However, FUTO Keyboard's high-performance C++ prediction engine (LanguageModel.cpp:766) identifies complete words strictly by inspecting the trailing byte of the emitted token:

// LanguageModel.cpp snippet
bool is_word_end = (token_str.size() >= 3 && 
                    (unsigned char)token_str[token_str.size()-3] == 0xe2 && 
                    (unsigned char)token_str[token_str.size()-2] == 0x96 && 
                    (unsigned char)token_str[token_str.size()-1] == 0x81);

When standard prefix models are loaded, the C++ runtime never detects a word-ending boundary. Consequently, the Android IME inserts spaces after every intermediate syllable, resulting in broken text like:

Input: "проверка связи"
Standard Model: "ло ша мы шь без дельника может сятив"
RUTO Model:     "проверка связи не требуется, всё в порядке"

RUTO trains SentencePiece strictly with treat_whitespace_as_suffix=True, attaching the boundary character ▁ to the end of tokens (провер, ка▁, связи▁). The keyboard immediately surfaces complete, natural words in the suggestion bar.

2. Zero English Bleed

Bilingual models frequently suggest Latin English words when typing in Cyrillic due to overlapping parallel corpora. RUTO's training corpus underwent 100% regex and character-set filtering, stripping all Latin words and characters from the Russian suggestion dictionary.

3. Physical Keyboard (ЙЦУКЕН) Autocorrect Training

In addition to next-word prediction, the model is fine-tuned for autocorrecting noisy taps. Typos are synthetically injected using an Euclidean distance matrix between physical key centers on standard Android ЙЦУКЕН mobile layouts.

• Top-1 Typo Correction Accuracy: 97.4%
• Top-3 Typo Correction Accuracy: 99.6%

4. FUTO Special Token Alignment

The model vocabulary strictly reserves and maps FUTO KeyboardLM control tokens:

• <XBU> (ID: 174) — Autocorrect typo block start
• <XBC> (ID: 175) — Target character indicator
• <XEC> (ID: 176) — Autocorrect typo block end
• <CHAR_A>..<<CHAR_Z> (IDs: 182..207) — Latin ASCII alphabet
• <CHAR_RU_А>..<<CHAR_RU_Я> (IDs: 208..240) — Russian Cyrillic alphabet (33 characters)
• Space token ▁ (ID: 23921)

---

📊 Training Pipeline & Methodology

The model was trained in three sequential stages using mixed-precision FP16 with AdamW on NVIDIA Tesla T4 GPUs:

1. Stage 1 — Base Russian Pretraining (2,000 steps):

   • Corpus: 400,000 conversational Russian dialogue sentences (Tatoeba + OpenSubtitles clean extract, 24 MB text).
   • Hyperparameters: Effective batch size 64 (batch_size=16, grad_accum=4), Learning Rate $5 \times 10^{-4}$ with cosine decay, weight decay 0.1.
   • Final Loss: 0.0101.

2. Stage 2 — Personal & Colloquial Continuous Pretraining (1,000 steps):

   • Corpus: 30,153 cleaned authentic chat dialogue messages.
   • Hyperparameters: Learning Rate $1 \times 10^{-4}$, preserving general grammar while capturing colloquial phrases, developer slang, and real typing rhythms.

3. Stage 3 — Autocorrect Fine-Tuning (1,500 steps):

   • Task: Masked correction of corrupted tokens injected with ЙЦУКЕН physical neighbor substitutions, deletions, and transpositions.
   • Loss: Reduced to 0.0002.

---

📦 Files in This Repository

Filename	Format	Size	Description
futo_ru_65m_v2_Q8_0.gguf	GGUF v2 (Q8_0)	78.8 MB	Recommended for mobile. 8-bit quantized weights with FP16 token embeddings for FUTO Keyboard.
futo_ru_65m_v2_f16.gguf	GGUF v2 (FP16)	126.3 MB	Full-precision FP16 GGUF v2 model for desktop / high-end emulation.
model.safetensors	SafeTensors	250 MB	Full FP32/BF16 weights for PyTorch fine-tuning and evaluation.
pytorch_model.bin	PyTorch Binary	250 MB	Standard PyTorch checkpoint format.
tokenizer.model	SentencePiece Model	746 KB	Binary BPE tokenizer model (treat_whitespace_as_suffix=True, 24k vocab).
tokenizer.vocab	Text	469 KB	Human-readable vocabulary list with token frequencies.
config.json	JSON	719 B	Hugging Face compatible LLaMA architecture config.
generation_config.json	JSON	216 B	Autoregressive generation defaults.

---

📲 How to Use in FUTO Keyboard

1. Install the patched RUTO Keyboard (FUTO Keyboard RU) APK:
   • Download futo-keyboard-ru-1.0.2.apk (Includes the critical C++ swipe gesture fix in dictionary_itrie.cpp and bundled ExecuTorch swipe weights).
2. Download futo_ru_65m_v2_Q8_0.gguf from this repository.
3. Open Keyboard Settings -> Languages -> Russian.
4. Tap Custom Transformer LM -> Select Model File and choose the downloaded .gguf file.
5. The keyboard will display: Transformer LM loaded: futo-keyboard-ru-65m (ru).

---

💻 Usage with Python (llama-cpp-python)

You can also run this model locally on PC or server using llama-cpp-python:

from llama_cpp import Llama

llm = Llama(
    model_path="futo_ru_65m_v2_Q8_0.gguf",
    n_ctx=128,
    n_threads=4
)

prompt = "Привет, как твои"
output = llm(prompt, max_tokens=10, stop=["\n"])
print(output["choices"][0]["text"])

---

⚖️ License

• Model Weights: Distributed under the FUTO Model Weights License 1.0.
• Source Code & Exporter Tools: Open-sourced under GPL v3 at MisterFenek/ruto-keyboard.