Upload all models and assets for ady (20251001)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/ngram_coverage.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.453
+    value: 4.231
   - name: best_isotropy
     type: isotropy
-    value: 0.6831
+    value: 0.4730
   - name: vocabulary_size
     type: vocab
-    value: 8988
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # ADY - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,58 +70,53 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.223x | 3.18 | 0.1016% | 189,909 |
-| **16k** | 3.621x | 3.57 | 0.1142% | 169,055 |
-| **32k** | 4.071x | 4.02 | 0.1284% | 150,370 |
-| **64k** | 4.453x 🏆 | 4.39 | 0.1404% | 137,476 |
+| **8k** | 3.442x | 3.45 | 0.1638% | 134,283 |
+| **16k** | 3.798x | 3.80 | 0.1808% | 121,676 |
+| **32k** | 4.231x 🏆 | 4.24 | 0.2014% | 109,215 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Киото — Японием и къалэ.
-
-Category:Къалэхэр
-Category:Японие`
+**Sample 1:** `Шъхьафит — Ашэ псыхъо иджабгъу нэпкъы тес Адыгэ къуадж. районым хахьэ. Хым зы пэ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁к и от о ▁— ▁японием ▁и ▁къалэ . ▁category ... (+5 more)` | 15 |
-| 16k | `▁ки ото ▁— ▁японием ▁и ▁къалэ . ▁category : къалэхэр ... (+3 more)` | 13 |
-| 32k | `▁ки ото ▁— ▁японием ▁и ▁къалэ . ▁category : къалэхэр ... (+3 more)` | 13 |
-| 64k | `▁киото ▁— ▁японием ▁и ▁къалэ . ▁category : къалэхэр ▁category ... (+2 more)` | 12 |
+| 8k | `▁шъхьафит ▁— ▁ашэ ▁псыхъо ▁иджабгъу ▁нэпкъы ▁тес ▁адыгэ ▁къуадж . ... (+7 more)` | 17 |
+| 16k | `▁шъхьафит ▁— ▁ашэ ▁псыхъо ▁иджабгъу ▁нэпкъы ▁тес ▁адыгэ ▁къуадж . ... (+7 more)` | 17 |
+| 32k | `▁шъхьафит ▁— ▁ашэ ▁псыхъо ▁иджабгъу ▁нэпкъы ▁тес ▁адыгэ ▁къуадж . ... (+7 more)` | 17 |
 
-**Sample 2:** `Ереван () – Армение и къэлэшъхьаI. Нэбгырэ млн 1,06 фэдиз дэс. Къалэм и лIышъхьэ...`
+**Sample 2:** `thumb Америкэ - чӀынэлъэшхухэр Iут зэхэт (Къыблэ Америкэмрэ, Ишъхъэрэмрэ) Тыгъэк...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁е ре ван ▁() ▁– ▁армение ▁и ▁къэлэшъхьа i . ... (+28 more)` | 38 |
-| 16k | `▁ереван ▁() ▁– ▁армение ▁и ▁къэлэшъхьа i . ▁нэбгырэ ▁млн ... (+25 more)` | 35 |
-| 32k | `▁ереван ▁() ▁– ▁армение ▁и ▁къэлэшъхьа i . ▁нэбгырэ ▁млн ... (+25 more)` | 35 |
-| 64k | `▁ереван ▁() ▁– ▁армение ▁и ▁къэлэшъхьа i . ▁нэбгырэ ▁млн ... (+20 more)` | 30 |
-
-**Sample 3:** `thumb
-thumb
-Ишъхъэрэ Америкэ — континент.
+| 8k | `▁thumb ▁америкэ ▁- ▁чӏы нэлъэ шхухэр ▁i ут ▁зэхэт ▁( ... (+17 more)` | 27 |
+| 16k | `▁thumb ▁америкэ ▁- ▁чӏынэлъэшхухэр ▁i ут ▁зэхэт ▁( къыблэ ▁америкэмрэ ... (+13 more)` | 23 |
+| 32k | `▁thumb ▁америкэ ▁- ▁чӏынэлъэшхухэр ▁i ут ▁зэхэт ▁( къыблэ ▁америкэмрэ ... (+11 more)` | 21 |
 
-ЧIырэу млн 24,7 км² фэдиз еубыты. ЦIы...`
+**Sample 3:** `thumb Мамуныр мэз псэушъхьэхэмэ а щыщ. Мамунхэр чыг дэпшэиэным лъэшэу Мамуным и ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁thumb ▁thumb ▁ишъхъэрэ ▁америкэ ▁— ▁континент . ▁ч i ырэу ... (+27 more)` | 37 |
-| 16k | `▁thumb ▁thumb ▁ишъхъэрэ ▁америкэ ▁— ▁континент . ▁ч i ырэу ... (+27 more)` | 37 |
-| 32k | `▁thumb ▁thumb ▁ишъхъэрэ ▁америкэ ▁— ▁континент . ▁ч i ырэу ... (+27 more)` | 37 |
-| 64k | `▁thumb ▁thumb ▁ишъхъэрэ ▁америкэ ▁— ▁континент . ▁ч i ырэу ... (+27 more)` | 37 |
+| 8k | `▁thumb ▁мамун ыр ▁мэз ▁псэушъхьэхэмэ ▁а ▁щыщ . ▁мамун хэр ... (+22 more)` | 32 |
+| 16k | `▁thumb ▁мамуныр ▁мэз ▁псэушъхьэхэмэ ▁а ▁щыщ . ▁мамунхэр ▁ч ыг ... (+14 more)` | 24 |
+| 32k | `▁thumb ▁мамуныр ▁мэз ▁псэушъхьэхэмэ ▁а ▁щыщ . ▁мамунхэр ▁чыг ▁дэпшэиэным ... (+10 more)` | 20 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.453x compression
-- **Lowest UNK Rate:** 8k with 0.1016% unknown tokens
+- **Best Compression:** 32k achieves 4.231x compression
+- **Lowest UNK Rate:** 8k with 0.1638% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -128,57 +125,89 @@ thumb
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 927 🏆 | 9.86 | 1,856 | 38.3% | 83.1% |
-| **2-gram** | 486 🏆 | 8.92 | 2,656 | 53.5% | 95.5% |
-| **3-gram** | 1,521 | 10.57 | 2,744 | 28.3% | 71.0% |
-| **3-gram** | 3,351 | 11.71 | 15,024 | 23.1% | 61.6% |
-| **4-gram** | 4,981 | 12.28 | 7,604 | 14.3% | 42.5% |
-| **4-gram** | 12,700 | 13.63 | 44,900 | 11.7% | 37.6% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 418 | 8.71 | 593 | 45.3% | 100.0% |
+| **2-gram** | Subword | 399 🏆 | 8.64 | 2,072 | 57.0% | 97.4% |
+| **3-gram** | Word | 706 | 9.46 | 922 | 33.9% | 100.0% |
+| **3-gram** | Subword | 2,788 | 11.44 | 11,614 | 24.5% | 65.1% |
+| **4-gram** | Word | 2,848 | 11.48 | 3,264 | 13.1% | 44.3% |
+| **4-gram** | Subword | 10,651 | 13.38 | 35,316 | 12.4% | 39.6% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `нэбгырэ млн` | 169 |
+| 2 | `къехъу щэпсэу` | 104 |
+| 3 | `картым тетэу` | 100 |
+| 4 | `м къехъу` | 89 |
+| 5 | `дло м` | 87 |
+
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `category :` | 662 |
-| 2 | `- рэ` | 638 |
-| 3 | `- м` | 464 |
-| 4 | `рэ илъэсым` | 335 |
-| 5 | `. category` | 276 |
+| 1 | `м къехъу щэпсэу` | 76 |
+| 2 | `къехъу щэпсэу хэгэгум` | 70 |
+| 3 | `адыгэ республикэм и` | 48 |
+| 4 | `дло м хахьэ` | 44 |
+| 5 | `м хахьэ хэгъэгу` | 39 |
 
-**3-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `- рэ илъэсым` | 333 |
-| 2 | `. category :` | 276 |
-| 3 | `category : !` | 179 |
-| 4 | `: ! main` | 179 |
-| 5 | `! main category` | 179 |
+| 1 | `м къехъу щэпсэу хэгэгум` | 45 |
+| 2 | `дло м хахьэ хэгъэгу` | 39 |
+| 3 | `еуропэм хэт къэралыгъу къэлэ` | 23 |
+| 4 | `америкэм ит къэралыгъу къэлэ` | 19 |
+| 5 | `азием ит къэралыгъу къэлэ` | 18 |
 
-**4-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `category : ! main` | 179 |
-| 2 | `: ! main category` | 179 |
-| 3 | `. category : !` | 132 |
-| 4 | `. хэгэгум чiырэу иiэр` | 101 |
-| 5 | `. дло - м` | 87 |
+| 1 | `г ъ` | 9,349 |
+| 2 | `ъ э` | 9,255 |
+| 3 | `э _` | 8,719 |
+| 4 | `м _` | 7,823 |
+| 5 | `э р` | 6,778 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `г ъ э` | 4,967 |
+| 2 | `_ к ъ` | 4,149 |
+| 3 | `э м _` | 3,582 |
+| 4 | `ы г ъ` | 3,357 |
+| 5 | `э р _` | 3,016 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ы г ъ э` | 1,903 |
+| 2 | `х э р _` | 1,450 |
+| 3 | `а г ъ э` | 1,351 |
+| 4 | `х э м _` | 1,305 |
+| 5 | `_ к ъ э` | 1,289 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 486
+- **Best Perplexity:** 2-gram (subword) with 399
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~38% of corpus
+- **Coverage:** Top-1000 patterns cover ~40% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -186,55 +215,86 @@ thumb
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.3643 | 1.287 | 2.28 | 28,827 | 63.6% |
-| **1** | 1.5343 | 2.896 | 12.27 | 463 | 0.0% |
-| **2** | 0.1248 | 1.090 | 1.24 | 65,637 | 87.5% |
-| **2** | 1.1477 | 2.216 | 5.66 | 5,679 | 0.0% |
-| **3** | 0.0452 | 1.032 | 1.07 | 80,882 | 95.5% |
-| **3** | 0.7357 | 1.665 | 2.89 | 32,122 | 26.4% |
-| **4** | 0.0244 🏆 | 1.017 | 1.04 | 86,492 | 97.6% |
-| **4** | 0.4145 🏆 | 1.333 | 1.83 | 92,841 | 58.5% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.4365 | 1.353 | 2.10 | 22,306 | 56.3% |
+| **1** | Subword | 1.4909 | 2.811 | 10.56 | 410 | 0.0% |
+| **2** | Word | 0.0764 | 1.054 | 1.12 | 46,305 | 92.4% |
+| **2** | Subword | 1.1481 | 2.216 | 5.61 | 4,325 | 0.0% |
+| **3** | Word | 0.0240 | 1.017 | 1.03 | 51,243 | 97.6% |
+| **3** | Subword | 0.7541 | 1.687 | 2.97 | 24,260 | 24.6% |
+| **4** | Word | 0.0128 🏆 | 1.009 | 1.02 | 52,387 | 98.7% |
+| **4** | Subword | 0.4304 | 1.348 | 1.86 | 72,077 | 57.0% |
+
+### Generated Text Samples (Word-based)
 
-### Generated Text Samples
+Below are text samples generated from each word-based Markov chain model:
 
-Below are text samples generated from each Markov chain model:
+**Context Size 1:**
+
+1. `и 13 мэ ащыщэу адыгэр сыдигъокіи адыгэ къуаж ипшъэ итхьапӏэ иблэгъожъхэм афэгъэхьыгъэ мифхэр къызэра...`
+2. `адыгэ хьатыкъуай унагъохэр тыркуем и плакат ныбэрынхьэблэ адыгэбзэ жэбзэ къабзэ ежь ныпым зызиушъомб...`
+3. `м ахахьэ хэгъэгу шавкат мирзияев къэрал лӏышъхьэр кӏокӏо къызбэч кавказ заом ыпэкӏэ щыӏагъэхэмрэ якъ...`
+
+**Context Size 2:**
+
+1. `нэбгырэ млн 10 фэдиз тешӏагъэу анатолием ахэр агъэкощыгъэх тхыгъэ зэфэшъхьафхэм мэхьанэу каноничност...`
+2. `къехъу щэпсэу я 84 хэгэгум 93 030 км я 26 испаныбзэр ащ нэмыкӏэу регионыбзэхэр иӏэх дло м`
+3. `картым тетэу бразилие къыблэ америкэм ыгу ит германиер аустриер словакиер руманиер украинэр сербиер ...`
+
+**Context Size 3:**
+
+1. `м къехъу щэпсэу хэгэгум 2 149 690 км арапыбз сауд арабиер арап къэралыгъомэ ащыщмэ анахь хэгъэгу ащы...`
+2. `къехъу щэпсэу хэгэгум 140 800 км непали дло м хахьэ хэгъэгу хассанал болкиах географие азием и гъунэ...`
+3. `адыгэ республикэм и къэралыгъо премие илауреат дунэе адыгэ академием иакадемик къалэу шъачэ поселкэу...`
+
+**Context Size 4:**
+
+1. `м къехъу щэпсэу хэгэгум 9 596 960 км китаибзэр дло м хахьэ хэгъэгу эмомали рахмон къэрал тхьэматэр к...`
+2. `дло м хахьэ хэгъэгу джоко видодо гуадзэр юсуф калла географие океан шъэфымымрэ инд океанымрэ азфагу ...`
+3. `еуропэм хэт къэралыгъу къэлэ париж нэбгырэ млн 66 м къехъу щэпсэу хэгэгум 9 984 670 км я 2 англыбзэ`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. ахъщэр iэгурыхьэ - 14 . ы ↔ ӏей ; пшэс 1угжъу category : / даутэ`
-2. `, коц , " ids " " тэiошъ , батэ вгъэшым ��э силъэпкъ шъукъыхахьэу , къэрабгъэр`
-3. `- рэ лӏэшӏэгъуршапсыгъэхэршапсыгъэ ныпгорэ . географие гъунэгъухэр : " kaynar , каракас ) къо зиiэм ...`
+1. `_фим_хъэрикъолам`
+2. `эм_илъу_-м_бэхь_`
+3. `ышъэпсым_илнине_`
 
 **Context Size 2:**
 
-1. `category : район category : ! main category зэпыщэхэр`
-2. `- рэ щэпсэу . гулъытэгъуэ техьэпӏэхэр къайсэр адыгэ хасэм ( дах - м хахьэ . хэгъэгу лiышъхьэр`
-3. `- м инароднэ тхакiу . илэжьэнхэр 1960 - рэ ислъэсхэм – адыгэ къэралыгъо университетым студентхэр щег...`
+1. `гъэгъэ_асэу_ɡʲadə`
+2. `ъэхьэухэм_епхъухь`
+3. `э_хэгьэмрэ_щыпӏэ-`
 
 **Context Size 3:**
 
-1. `- рэ илъэсым къыщегъэжьагъэу щэiэфэ гуманитар ушэтынхэмкiэ адыгэ республикэ институтым литературэмкi...`
-2. `. category : ! main category зэпыщэхэр`
-3. `category : ! main category зэпыщэхэр`
+1. `гъэ_уахэмрэ,_къыуи`
+2. `_къалэбилэжъ_зэпхъ`
+3. `эм_ыгугъэкон_къаук`
 
 **Context Size 4:**
 
-1. `category : ! main category зэпыщэхэр`
-2. `. category : ! main category зэпыщэхэр`
-3. `. хэгэгум чiырэу иiэр 9 984 670 км² ( дунаемкiэ я - 11 ) . хэгэгум чiырэу иiэр 283`
+1. `ыгъэуцохэр_чэзыу-чэ`
+2. `хэр_нэхъин_динхэр_з`
+3. `агъэр_гъэп,_англыбз`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 97.6% predictability
+- **Best Predictability:** Context-4 (word) with 98.7% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (92,841 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (72,077 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -250,64 +310,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 8,988 |
-| Total Tokens | 57,159 |
-| Mean Frequency | 6.36 |
+| Vocabulary Size | 7,032 |
+| Total Tokens | 44,503 |
+| Mean Frequency | 6.33 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 23.47 |
+| Frequency Std Dev | 22.13 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | и | 1,019 |
-| 2 | category | 841 |
-| 3 | адыгэ | 701 |
-| 4 | рэ | 641 |
-| 5 | м | 541 |
-| 6 | илъэсым | 407 |
-| 7 | ащ | 392 |
-| 8 | я | 349 |
-| 9 | ары | 276 |
-| 10 | а | 259 |
+| 1 | и | 1,013 |
+| 2 | адыгэ | 666 |
+| 3 | м | 489 |
+| 4 | илъэсым | 398 |
+| 5 | ащ | 391 |
+| 6 | я | 309 |
+| 7 | ары | 271 |
+| 8 | нэбгырэ | 247 |
+| 9 | а | 243 |
+| 10 | ыкӏи | 211 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | britishpedia | 2 |
-| 2 | encyklopedia | 2 |
-| 3 | osobistości | 2 |
-| 4 | rzeczypospolitej | 2 |
-| 5 | polskiej | 2 |
-| 6 | bph | 2 |
-| 7 | british | 2 |
-| 8 | publishing | 2 |
-| 9 | ltd | 2 |
-| 10 | 912100 | 2 |
+| 1 | рсфср | 2 |
+| 2 | серийнэ | 2 |
+| 3 | ныбжьыкӏэхэри | 2 |
+| 4 | зэратебэнагъэр | 2 |
+| 5 | хираганэ | 2 |
+| 6 | катаканэ | 2 |
+| 7 | сербыбзэм | 2 |
+| 8 | къыздикӏыгъэр | 2 |
+| 9 | тыванбзэ | 2 |
+| 10 | къызыл | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.7855 |
-| R² (Goodness of Fit) | 0.976491 |
+| Zipf Coefficient | 0.7821 |
+| R² (Goodness of Fit) | 0.977951 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 26.7% |
-| Top 1,000 | 56.9% |
-| Top 5,000 | 86.0% |
+| Top 100 | 29.3% |
+| Top 1,000 | 60.6% |
+| Top 5,000 | 90.9% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9765 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 26.7% of corpus
-- **Long Tail:** -1,012 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9780 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 29.3% of corpus
+- **Long Tail:** -2,968 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -320,24 +380,129 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 1,830 | 32 | 3.764 | 0.663 | 0.6831 🏆 |
-| **mono_64d** | 1,830 | 64 | 3.806 | 0.668 | 0.2517 |
-| **mono_128d** | 1,830 | 128 | 3.824 | 0.669 | 0.0484 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.4730 🏆 | 0.4239 | N/A | N/A |
+| **mono_64d** | 64 | 0.2201 | 0.4040 | N/A | N/A |
+| **mono_128d** | 128 | 0.0372 | 0.3952 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.6831 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 1,830 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.4730 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.4077. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-къ` | къыщыхъу, къуаджэхэу, къэбарым |
+| `-зэ` | зэман, зэдаштэгъэ, зэпэух |
+| `-къы` | къыщыхъу, къыщыфэфедэщтхэу, къызыхэкӏыгъэр |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-э` | ятхьэ, урысыбзэ, чылэ |
+| `-м` | такъырым, шапхъэхэм, къэбарым |
+| `-р` | латвиер, сыхьатыр, министр |
+| `-эр` | курдхэр, щыгъынхэр, мэхъошхэр |
+| `-эм` | шапхъэхэм, япэм, урымыбзэм |
+| `-эу` | алфавитэу, илъхэу, игъэкӏотыгъэу |
+| `-хэр` | курдхэр, щыгъынхэр, мэхъошхэр |
+| `-рэ` | къагъэлъагъуэрэ, зыгорэ, цӏэмрэ |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `тыгъ` | 1.78x | 28 contexts | тыгъэ, итыгъ, тыгъу |
+| `ъагъ` | 2.15x | 14 contexts | пчъагъ, лъагъо, пчъагъэ |
+| `агъэ` | 1.54x | 41 contexts | тхагъэ, благъэ, пчагъэ |
+| `эпкъ` | 1.74x | 25 contexts | нэпкъ, тхэпкъ, лъэпкъ |
+| `къуа` | 2.16x | 10 contexts | къуае, къуажэ, къуадж |
+| `ъхьэ` | 1.78x | 16 contexts | шъхьэ, пшъхьэ, шъхьэм |
+| `дыгэ` | 1.82x | 14 contexts | адыгэ, адыгэм, иадыгэ |
+| `эхэр` | 1.56x | 21 contexts | бэхэр, усэхэр, ынэхэр |
+| `шъхь` | 1.49x | 24 contexts | шъхьэ, пшъхьэ, шъхьэм |
+| `псэу` | 1.57x | 19 contexts | щыпсэу, щэпсэу, сыпсэу |
+| `ыгъо` | 1.56x | 19 contexts | цыгъо, мыгъо, пщыгъо |
+| `гъэх` | 1.65x | 14 contexts | багъэх, хъугъэх, ежагъэх |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-къ` | `-э` | 96 words | къэлэмымкӏэ, къалэмэ |
+| `-къ` | `-р` | 64 words | къо��, къуаджэхэр |
+| `-къ` | `-м` | 56 words | къалэм, къумбылым |
+| `-къ` | `-эр` | 52 words | къуаджэхэр, къэбархэр |
+| `-зэ` | `-р` | 42 words | зэготхэр, зэхэтхэр |
+| `-зэ` | `-м` | 41 words | зэхэзгъэуцуагъэхэм, зэӏукӏэгъум |
+| `-къ` | `-эм` | 36 words | къалэм, къуаджэхэм |
+| `-зэ` | `-эр` | 34 words | зэготхэр, зэхэтхэр |
+| `-къ` | `-эу` | 34 words | къыхэкӏыгъэу, къэгъэлъэгъонэу |
+| `-зэ` | `-э` | 31 words | зэ, зэригъэфэгъэ |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| щыпсэухэрэр | **`щыпс-эу-хэр-эр`** | 7.5 | `щыпс` |
+| америкэмрэ | **`америк-эм-рэ`** | 6.0 | `америк` |
+| океанымрэ | **`океан-ым-рэ`** | 6.0 | `океан` |
+| литературэмрэ | **`литератур-эм-рэ`** | 6.0 | `литератур` |
+| бзылъфыгъэмрэ | **`бзылъфыгъ-эм-рэ`** | 6.0 | `бзылъфыгъ` |
+| адыгабзэмрэ | **`адыгабз-эм-рэ`** | 6.0 | `адыгабз` |
+| хыплъыжьымрэ | **`хыплъыжь-ым-рэ`** | 6.0 | `хыплъыжь` |
+| алфавитэу | **`алфавит-эу`** | 4.5 | `алфавит` |
+| цӏыкӏухэр | **`цӏыкӏу-хэр`** | 4.5 | `цӏыкӏу` |
+| исурэтхэр | **`исурэт-хэр`** | 4.5 | `исурэт` |
+| шӏыпӏэхэр | **`шӏыпӏэ-хэр`** | 4.5 | `шӏыпӏэ` |
+| шӏэныгъэм | **`шӏэныгъ-эм`** | 4.5 | `шӏэныгъ` |
+| къыпыщылъ | **`къы-пыщылъ`** | 4.5 | `пыщылъ` |
+| пэблагъэу | **`пэблагъ-эу`** | 4.5 | `пэблагъ` |
+| ишъхъэрэмрэ | **`ишъхъ-эр-эм-рэ`** | 4.5 | `ишъхъ` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language ADY appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -345,11 +510,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.45x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (486) |
-| Markov | **Context-4** | Highest predictability (97.6%) |
+| Tokenizer | **32k BPE** | Best compression (4.23x) |
+| N-gram | **2-gram** | Lowest perplexity (399) |
+| Markov | **Context-4** | Highest predictability (98.7%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -539,7 +705,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -555,7 +722,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 04:34:00*
+*Report Date: 2026-01-03 05:00:02*

models/embeddings/monolingual/ady_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:27f96cb22cce1dccf3f5a68afba58270c9a3d30c33c99e0b985a0160dc08be41
-size 1025913894
+oid sha256:79239fb8f1d5516f7259d49637e01b3851134cf4dc232d72d6bbc383171bd360
+size 1025621365

models/embeddings/monolingual/ady_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 1830
+  "vocab_size": 1551
 }

models/embeddings/monolingual/ady_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4df91a4ae1b68f41d1ebc698b3b79863f82558368b99039be6879f7219c9dfab
-size 256508454
+oid sha256:35a83ab3b148734fe90d463a14bb03ac25aa53bebe1cd930e6ee275b421f2b9e
+size 256430197

models/embeddings/monolingual/ady_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 1830
+  "vocab_size": 1551
 }

models/embeddings/monolingual/ady_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e39765ffd2c8eb6ef0e3b1854af129b7b0260b2e70a54156a81a9a099b1fa27b
-size 512976934
+oid sha256:f26b79bc9f27a68c68e56e0b9e83bef6a8d25662bd63c7a490adce062b153229
+size 512827253

models/embeddings/monolingual/ady_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 1830
+  "vocab_size": 1551
 }

models/subword_markov/ady_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:eb658a5696f0194212546a4afabf40b1859c08800bbd6f1ffecda37fddfd3c58
-size 45335
+oid sha256:326d1899bc21c88f5d2ecbb60df90091a3f81877b05f9f3880179e64fc82bc39
+size 35559

models/subword_markov/ady_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ady",
-  "unique_contexts": 463,
-  "total_transitions": 611427
+  "unique_contexts": 410,
+  "total_transitions": 461186
 }

models/subword_markov/ady_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b5ebd13fcb32a70f1eeeb3c7ec3df3945466bdb98d3539405ae08a602378e310
-size 217004
+oid sha256:439580e1d85b149b7a4e12434dcacbd8c63ce4f422949f7787d12f51a3b06768
+size 169546

models/subword_markov/ady_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ady",
-  "unique_contexts": 5679,
-  "total_transitions": 610662
+  "unique_contexts": 4325,
+  "total_transitions": 460515
 }

models/subword_markov/ady_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b9df0d86e436385c2b2494f0af89b0ca67a6a9d8aa7e5706dadd31de6649a3d8
-size 714083
+oid sha256:ba32cd485f139de6037badf290db15170159de5ebb63242fabe6473e50963984
+size 554556

models/subword_markov/ady_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ady",
-  "unique_contexts": 32122,
-  "total_transitions": 609897
+  "unique_contexts": 24260,
+  "total_transitions": 459844
 }

models/subword_markov/ady_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:14192c7fe2413ebd6957d9d1e1bf08f20d2cf28008cf6b161e3c9ff26a079c9b
-size 1599436
+oid sha256:a23a4022efd4fb6c5360771eb3945cadabce7d4e08d9a04895e4535a30c12dad
+size 1240010

models/subword_markov/ady_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ady",
-  "unique_contexts": 92841,
-  "total_transitions": 609132
+  "unique_contexts": 72077,
+  "total_transitions": 459173
 }

models/subword_ngram/ady_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:016367258813db8565f69ab7cb29945baccaac7e94321afc25a4b6c3c9f79f58
-size 32965
+oid sha256:f2744392ac6f30902dc24ef71be21ed3d0027816ff58c51cc117dbe79b60b6eb
+size 26615

models/subword_ngram/ady_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ady",
-  "unique_ngrams": 2656,
-  "total_ngrams": 611427
+  "unique_ngrams": 2072,
+  "total_ngrams": 461186
 }

models/subword_ngram/ady_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:19fa61eac0061901eb9506a6a556a9261919e5b319eaf1999aa663587ce22014
-size 187929
+oid sha256:3e27920b4a896f7b33778a4da60561fc4bf2c40fe17414d4721a023a815cc6e6
+size 144946

models/subword_ngram/ady_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ady",
-  "unique_ngrams": 15024,
-  "total_ngrams": 610662
+  "unique_ngrams": 11614,
+  "total_ngrams": 460515
 }

models/subword_ngram/ady_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c90862733e2c9b9b0d0d50264ea47e26c1bdce42f31aa0ab349bff62b10cc5fe
-size 580214
+oid sha256:a1189db5bc7c400c2d162eabf55e0ca7ce6b8403c020f3330fe200b437eab55c
+size 467994

models/subword_ngram/ady_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ady",
-  "unique_ngrams": 44900,
-  "total_ngrams": 609897
+  "unique_ngrams": 35316,
+  "total_ngrams": 459844
 }

models/tokenizer/ady_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e6120c55e21bf381032672a3067bcf46760e2c4d359e940e77b9f3f746421b69
-size 564362
+oid sha256:aa7339e704e163ae3ef399bf56ca2b041b58ad1cead7a66d4fef2204b2af8435
+size 582264

models/tokenizer/ady_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8e432f1beef1ea7d335d31918e5e88f5751d4fc577f2231b239c92518f39dc9c
-size 955536
+oid sha256:dd22631bb665397f8acd6aab07c2522b6ca5fc9532f157143320bbb205558552
+size 924924

models/tokenizer/ady_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cfc7334d75b921cca2c9e616b995ac20d0728c4f9b3c54f3c5cb0b2ba2b5267c
-size 398334
+oid sha256:8a4cd8210cf36092a1e5cc2950c6a2e334677c7d204476c1e9ac6f464fda33e5
+size 396510

models/vocabulary/ady_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:733b017356844758ce21db6ab8269f1bff9141c2d41430f547302391bc541d7e
-size 161889
+oid sha256:0394f3c6c2f9871c8756de87f7c0ab00cba32facf111d0c2027f0844c932241b
+size 124070

models/vocabulary/ady_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "ady",
-  "vocabulary_size": 8988,
+  "vocabulary_size": 7032,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.374012371348373,
+    "type_token_ratio": 0.37416908841901325,
     "coverage": {
-      "top_100": 0.1982014762449319,
-      "top_1000": 0.42253612641646743,
-      "top_5000": 0.6391386838548706,
-      "top_10000": 0.7559387670235991
+      "top_100": 0.21755686942579416,
+      "top_1000": 0.4501954103617597,
+      "top_5000": 0.6754016768547283,
+      "top_10000": 0.7928483147944015
     },
-    "hapax_count": 19793,
-    "hapax_ratio": 0.6877106424377193,
-    "total_documents": 765
+    "hapax_count": 15371,
+    "hapax_ratio": 0.6861134669463911,
+    "total_documents": 671
   }
 }

models/word_markov/ady_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0faaa6cb0c52d77ac3f86b442e03b6cdd0646165c5b5467afaa42799100c5c08
-size 1013255
+oid sha256:e529d92df652217192b1a53b43d340b1b2d134ff78b718c2331e8296f5e8a183
+size 772451

models/word_markov/ady_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ady",
-  "unique_contexts": 28827,
-  "total_transitions": 105741
+  "unique_contexts": 22306,
+  "total_transitions": 59203
 }

models/word_markov/ady_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cdf814da92430ed71f7261a45abbc27d6bc4f679afc5556ff21d158586b5709f
-size 1738893
+oid sha256:949fea0fcff222936353244ef6edc60951c50ac50290dc3b857a31ce73658051
+size 1310186

models/word_markov/ady_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ady",
-  "unique_contexts": 65637,
-  "total_transitions": 104976
+  "unique_contexts": 46305,
+  "total_transitions": 58532
 }

models/word_markov/ady_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c430f624230776fb0bed9ceec3edfcc58d221b9f682aa377294e7faf952a7f33
-size 2203636
+oid sha256:0edd29aaf52ac72ad56819e317223cc422f1de2fd93feb950a6fd0c9d7772f20
+size 1585798

models/word_markov/ady_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ady",
-  "unique_contexts": 80882,
-  "total_transitions": 104212
+  "unique_contexts": 51243,
+  "total_transitions": 57861
 }

models/word_markov/ady_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f994f94aaa5faf04bea4da903f474a186f4d719203b671b4c8538fa7e3d6c323
-size 2577702
+oid sha256:d3e2c27ffec6ab82c94cdc5ee7375d9e46705feb8849d5c1a1b44f49a1934653
+size 1845825

models/word_markov/ady_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ady",
-  "unique_contexts": 86492,
-  "total_transitions": 103449
+  "unique_contexts": 52387,
+  "total_transitions": 57190
 }

models/word_ngram/ady_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e36a3e7022b130b28b923c0f0edfaff7cb6f8c269b793337bd6e999b68fc36cd
-size 35981
+oid sha256:844f6778f268cd035e5b1a16fd38a5a6039c763c63cfaee2c5da98c36decbf9d
+size 15361

models/word_ngram/ady_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "ady",
-  "unique_ngrams": 1856,
-  "total_ngrams": 105741
+  "unique_ngrams": 593,
+  "total_ngrams": 59203
 }

models/word_ngram/ady_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a2ce08e36db31a04439c365d753c2684b44924398908b6f7a2883305f54f1f0c
-size 61177
+oid sha256:2d7eb072c0df1857a28734446befa43d91f7bec4523744835d1f96b7ef4ea52f
+size 26381

models/word_ngram/ady_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "ady",
-  "unique_ngrams": 2744,
-  "total_ngrams": 104976
+  "unique_ngrams": 922,
+  "total_ngrams": 58532
 }

models/word_ngram/ady_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:43176b575286a7924ea5d8764dc0f6dfc283f757937cfa2d648424a65326ca9f
-size 203246
+oid sha256:554ec1d49a6c147c67f6b66d5ec37c9b9ad812cce293118aea436980088ed1b3
+size 106335

models/word_ngram/ady_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "ady",
-  "unique_ngrams": 7604,
-  "total_ngrams": 104212
+  "unique_ngrams": 3264,
+  "total_ngrams": 57861
 }