Bashkir Diacritics Restorer v1.2
A compact, local-first ONNX model that restores Bashkir-specific letters in partially undiacritized text while deliberately preserving foreign and technical tokens.
Overview
This is a character-level Bashkir diacritics restorer for the nine Bashkir
letters: Ә Ғ Ҙ Ҡ Ң Ө Ҫ Ү Һ. It combines a 1.18M-parameter ConvBiGRU model
with a conservative safety gate. The neural model is self-contained and runs
with only model.onnx, vocab.json and substitution_map.json. The default
runtime does not require a lexicon, KenLM, an internet connection, or a GPU.
| At a glance | |
|---|---|
| Task | Restore Bashkir-specific Cyrillic letters |
| Architecture | Multi-scale Conv1D + 2-layer BiGRU |
| Parameters | 1,184,032 |
| Runtime | ONNX Runtime on CPU |
| Default mode | Safe neural restoration, no external lexical or language model |
| License | CC BY 4.0 |
Contents
Files and Configurations
| File | Purpose |
|---|---|
model.onnx |
CPU-ready neural model |
vocab.json |
Character vocabulary |
substitution_map.json |
Allowed base-to-diacritic substitutions |
runtime.py |
Minimal safe Python / ONNX runtime |
config.json |
Runtime defaults and package metadata |
SHA256SUMS |
SHA-256 checksums for public release files |
The public package intentionally excludes the derived lexicon.json and the
optional phrase-level KenLM model. This keeps the release self-contained and
avoids redistributing a corpus-derived lexical resource. A research/full variant
with optional components may be requested from the author through the Hugging
Face profile.
Model Architecture
The model predicts a character distribution at each input position. It uses parallel 1D convolution features (kernel sizes 3 and 5) followed by a two-layer bidirectional GRU with hidden size 192. A substitution whitelist permits only valid base-letter to Bashkir-letter changes; it cannot invent arbitrary symbols.
Examples
| Input | Restored output |
|---|---|
мин башкортса хойлэшэм |
мин башҡортса һөйләшәм |
казакстан |
ҡаҙаҡстан |
БАШКОРТ ТЕЛЕ |
БАШҠОРТ ТЕЛЕ |
hello world iPhone |
unchanged |
XI IX VI |
unchanged |
Method
input text → token safety gate → character ONNX predictions → confidence threshold
→ restored text
- Safety gate: skips URLs, email-like fragments, Roman numerals, ordinary Latin tokens, common Russian function words and unknown foreign quoted text.
- Controlled replacements: only observed base-to-specific-letter substitutions
from
substitution_map.jsoncan be applied. - Existing diacritics: already typed Bashkir-specific letters are retained.
Benchmark
The original held-out evaluation reports sentence exact match of 93.42% and
Bashkir-specific-letter F1 of 99.67%. The public standalone release —
without both lexicon.json and KenLM — reaches 87.8% restoration exact
match and 97.6% untouched clean input on a mixed 1,000-item safety check.
That check includes foreign and technical strings that the v1.2 safety gate
intentionally leaves unchanged. It also uses automatically generated targets,
so individual disagreements can include noisy or historical source forms. See
META.json for the machine-readable record.
Component Smoke Check
A 15-item behavioural check, ordered from simple word restoration through mixed-script input, casing, ambiguity and protected foreign text, was run with three local configurations. This is a transparent integration check, not a representative benchmark.
| Configuration | Expected behaviours passed | What it shows |
|---|---|---|
| Public release: neural-only, threshold 0.40 | 14 / 15 | Runs independently; protects English and Roman numerals, but misses mixed-script hойлэшэм. |
| Optional hybrid: model + lexicon, threshold 0.40 | 15 / 15 | Recovers the mixed-script form as well. |
| Optional hybrid, stricter threshold 0.60 | 14 / 15 | More conservative neural fallback can leave a valid change unapplied. |
The context-free form кул is deliberately retained in all modes: it can
mean more than one Bashkir word, and this release does not guess without a
phrase-level language model.
| Mode | Exact match | Clean text left unchanged |
|---|---|---|
| Neural-only (public release) | 87.8% | 97.6% |
| Neural + optional lexicon | 89.1% | 97.8% |
| Neural + optional lexicon + KenLM | 92.3% | 98.5% |
On this automatic test the lexical component changes the overall result by 1.3 percentage points. It is most useful for known forms and mixed-script keyboard input; its impact on real-world usage should be measured separately with a human-checked test set.
Quality and Use
This release is suitable for local text cleanup, input-method suggestions and post-processing of Bashkir text. Its small ONNX model is practical on CPU-only systems. For a keyboard, call it only for the current token or short composing text, and present a candidate rather than silently replacing uncertain text.
Limitations
- This is a restoration model, not a general spellchecker or text generator.
- A text with no Bashkir lexical evidence may be intentionally left unchanged.
- Without the optional context language model, homographs can remain ambiguous.
- The model is trained for Bashkir Cyrillic; code-mixed, dialectal, novel and OCR-noisy forms need application-level review.
- Quality scores do not establish linguistic normativity for every output.
Related Resources
- Bashkir FastText Embeddings — compact subword embeddings for similar-form lookup and Bashkir lexical signals.
- Bashkir Word N-gram Index — phrase-level statistics for optional context-aware candidate ranking in keyboards.
Usage
Install the two runtime dependencies:
pip install -r requirements.txt
Run the included example from this release directory:
python example.py
Or use the runtime directly:
from pathlib import Path
from runtime import BashkirDiacriticsRestorer
restorer = BashkirDiacriticsRestorer(Path("path/to/bashkir-diacritics-restorer"))
print(restorer.restore("мин башкортса хойлэшэм"))
# мин башҡортса һөйләшәм
The default threshold is 0.40. Raise it to make neural substitutions more
conservative, for example restorer.restore(text, min_confidence=0.60).
License
Distributed under the CC BY 4.0 license. The release contains model artifacts and derived substitution rules, not source texts. Upstream source licenses and attribution requirements still apply to the underlying materials.
Citation
@software{failed09_bashkir_diacritics_restorer_2026,
title = {Bashkir Diacritics Restorer v1.2},
author = {failed09},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/failed09/bashkir-diacritics-restorer},
note = {Compact ONNX model for safe Bashkir diacritics restoration}
}
Open Bashkir Data and Sources 🐝
This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.
The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.
- Downloads last month
- 88