Upload all models and assets for av (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.583
+    value: 4.697
   - name: best_isotropy
     type: isotropy
     value: 0.8716
   - name: vocabulary_size
     type: vocab
-    value: 38576
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # AV - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,61 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.534x | 3.49 | 0.0801% | 219,599 |
-| **16k** | 3.897x | 3.84 | 0.0884% | 199,103 |
-| **32k** | 4.254x | 4.20 | 0.0965% | 182,410 |
-| **64k** | 4.583x 🏆 | 4.52 | 0.1039% | 169,325 |
+| **8k** | 3.636x | 3.64 | 0.0717% | 252,363 |
+| **16k** | 4.040x | 4.04 | 0.0797% | 227,147 |
+| **32k** | 4.391x | 4.40 | 0.0866% | 208,961 |
+| **64k** | 4.697x 🏆 | 4.70 | 0.0927% | 195,348 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Кванирукъ (латиназул мацӀалда ventriculus) — гӀадамасул лага-черх. 
-
-Категория:Г...`
+**Sample 1:** `Хъипчахъ () гъорлъе уна жибго Хъипчахъ росу. Гьеб росулъ гьабула . росаби`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁кван ир укъ ▁( латиназул ▁мацӏалда ▁v ent ric ul ... (+14 more)` | 24 |
-| 16k | `▁кван ир укъ ▁( латиназул ▁мацӏалда ▁v ent ric ulus ... (+13 more)` | 23 |
-| 32k | `▁кванирукъ ▁( латиназул ▁мацӏалда ▁v ent ric ulus ) ▁— ... (+11 more)` | 21 |
-| 64k | `▁кванирукъ ▁( латиназул ▁мацӏалда ▁vent ric ulus ) ▁— ▁гӏадамасул ... (+10 more)` | 20 |
+| 8k | `▁хъ ип ч ахъ ▁() ▁гъорлъе ▁уна ▁жибго ▁хъ ип ... (+9 more)` | 19 |
+| 16k | `▁хъ ип ч ахъ ▁() ▁гъорлъе ▁уна ▁жибго ▁хъ ип ... (+9 more)` | 19 |
+| 32k | `▁хъипчахъ ▁() ▁гъорлъе ▁уна ▁жибго ▁хъипчахъ ▁росу . ▁гьеб ▁росулъ ... (+3 more)` | 13 |
+| 64k | `▁хъипчахъ ▁() ▁гъорлъе ▁уна ▁жибго ▁хъипчахъ ▁росу . ▁гьеб ▁росулъ ... (+3 more)` | 13 |
 
-**Sample 2:** `Гудермес ( ) — Россиялъул Буртиялъ жумхӀурияталда бугеб шагьар.
-Сунж-хъалаялдаса...`
+**Sample 2:** `26-абилеб июль — грегорианияб календаралда рекъон къо (високоснияб соналъ — свер...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁г уд ер мес ▁( ▁) ▁— ▁россиялъул ▁бурт иялъ ... (+36 more)` | 46 |
-| 16k | `▁гуд ер мес ▁( ▁) ▁— ▁россиялъул ▁бурт иялъ ▁жум ... (+33 more)` | 43 |
-| 32k | `▁гудермес ▁( ▁) ▁— ▁россиялъул ▁буртиялъ ▁жумхӏ урият алда ▁бугеб ... (+25 more)` | 35 |
-| 64k | `▁гудермес ▁( ▁) ▁— ▁россиялъул ▁буртиялъ ▁жумхӏурият алда ▁бугеб ▁шагьар ... (+22 more)` | 32 |
-
-**Sample 3:** `Лъугьа-бахъинал 
+| 8k | `▁ 2 6 - абилеб ▁июль ▁— ▁грегорианияб ▁календаралда ▁рекъон ... (+19 more)` | 29 |
+| 16k | `▁ 2 6 - абилеб ▁июль ▁— ▁грегорианияб ▁календаралда ▁рекъон ... (+19 more)` | 29 |
+| 32k | `▁ 2 6 - абилеб ▁июль ▁— ▁грегорианияб ▁календаралда ▁рекъон ... (+19 more)` | 29 |
+| 64k | `▁ 2 6 - абилеб ▁июль ▁— ▁грегорианияб ▁календаралда ▁рекъон ... (+19 more)` | 29 |
 
- Гьаруна 
-
- Хвана 
-
-
-Категория:1927`
+**Sample 3:** `() ккола Билкан районалда гъорлъе унеб росу. росаби`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁лъугьа - бахъинал ▁гьаруна ▁хвана ▁категория : 1 9 2 ... (+1 more)` | 11 |
-| 16k | `▁лъугьа - бахъинал ▁гьаруна ▁хвана ▁категория : 1 9 2 ... (+1 more)` | 11 |
-| 32k | `▁лъугьа - бахъинал ▁гьаруна ▁хвана ▁категория : 1 9 2 ... (+1 more)` | 11 |
-| 64k | `▁лъугьа - бахъинал ▁гьаруна ▁хвана ▁категория : 1 9 2 ... (+1 more)` | 11 |
+| 8k | `▁() ▁ккола ▁билкан ▁районалда ▁гъорлъе ▁унеб ▁росу . ▁росаби` | 9 |
+| 16k | `▁() ▁ккола ▁билкан ▁районалда ▁гъорлъе ▁унеб ▁росу . ▁росаби` | 9 |
+| 32k | `▁() ▁ккола ▁билкан ▁районалда ▁гъорлъе ▁унеб ▁росу . ▁росаби` | 9 |
+| 64k | `▁() ▁ккола ▁билкан ▁районалда ▁гъорлъе ▁унеб ▁росу . ▁росаби` | 9 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.583x compression
-- **Lowest UNK Rate:** 8k with 0.0801% unknown tokens
+- **Best Compression:** 64k achieves 4.697x compression
+- **Lowest UNK Rate:** 8k with 0.0717% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -131,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 5,221 🏆 | 12.35 | 14,725 | 20.8% | 49.4% |
-| **2-gram** | 502 🏆 | 8.97 | 5,314 | 55.0% | 94.9% |
-| **3-gram** | 8,074 | 12.98 | 19,718 | 16.9% | 42.5% |
-| **3-gram** | 4,078 | 11.99 | 36,896 | 22.5% | 60.1% |
-| **4-gram** | 18,096 | 14.14 | 39,973 | 12.6% | 31.2% |
-| **4-gram** | 18,482 | 14.17 | 151,649 | 12.4% | 35.7% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 3,247 | 11.66 | 6,413 | 22.5% | 54.2% |
+| **2-gram** | Subword | 428 🏆 | 8.74 | 4,133 | 57.8% | 96.7% |
+| **3-gram** | Word | 2,834 | 11.47 | 6,427 | 26.0% | 57.0% |
+| **3-gram** | Subword | 3,424 | 11.74 | 28,949 | 23.6% | 62.9% |
+| **4-gram** | Word | 8,629 | 13.07 | 17,392 | 16.9% | 37.2% |
+| **4-gram** | Subword | 15,875 | 13.95 | 119,337 | 12.4% | 36.8% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `категория :` | 6,060 |
-| 2 | `) .` | 2,431 |
-| 3 | `) ,` | 2,098 |
-| 4 | `) —` | 1,555 |
-| 5 | `. —` | 1,376 |
+| 1 | `росу буго` | 509 |
+| 2 | `лъугьа бахъинал` | 496 |
+| 3 | `география росу` | 461 |
+| 4 | `цо цо` | 455 |
+| 5 | `of the` | 441 |
 
-**3-grams:**
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `. география росу` | 645 |
-| 2 | `география росу буго` | 645 |
-| 3 | `. категория :` | 622 |
-| 4 | `мугъчӏваял категория :` | 614 |
-| 5 | `лъугьа - бахъинал` | 597 |
+| 1 | `география росу буго` | 448 |
+| 2 | `лъугьа бахъинал гьаруна` | 368 |
+| 3 | `бахъинал гьаруна хвана` | 358 |
+| 4 | `байрамал лъугьа бахъинал` | 353 |
+| 5 | `гьаруна хвана ишараби` | 352 |
 
-**4-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `. география росу буго` | 630 |
-| 2 | `география росу буго мухъалъул` | 513 |
-| 3 | `. мугъчӏваял категория :` | 483 |
-| 4 | `лъугьа - бахъинал гьаруна` | 471 |
-| 5 | `- бахъинал гьаруна хвана` | 461 |
+| 1 | `лъугьа ��ахъинал гьаруна хвана` | 358 |
+| 2 | `байрамал лъугьа бахъинал гьаруна` | 352 |
+| 3 | `къо байрамал лъугьа бахъинал` | 351 |
+| 4 | `бахъинал гьаруна хвана ишараби` | 349 |
+| 5 | `демография ккола моноэтникияб авар` | 329 |
+
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `а л` | 82,724 |
+| 2 | `л _` | 63,062 |
+| 3 | `л ъ` | 52,236 |
+| 4 | `а _` | 52,185 |
+| 5 | `у л` | 49,900 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `у л _` | 33,240 |
+| 2 | `л ъ у` | 30,603 |
+| 3 | `ъ у л` | 25,387 |
+| 4 | `а л ъ` | 23,574 |
+| 5 | `_ г ь` | 22,295 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `л ъ у л` | 23,988 |
+| 2 | `ъ у л _` | 21,518 |
+| 3 | `а л ъ у` | 16,083 |
+| 4 | `а л д а` | 11,383 |
+| 5 | `_ г ь е` | 11,094 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 502
+- **Best Perplexity:** 2-gram (subword) with 428
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~36% of corpus
+- **Coverage:** Top-1000 patterns cover ~37% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -189,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5741 | 1.489 | 3.65 | 105,500 | 42.6% |
-| **1** | 1.3715 | 2.587 | 12.28 | 1,091 | 0.0% |
-| **2** | 0.1898 | 1.141 | 1.41 | 384,678 | 81.0% |
-| **2** | 1.0636 | 2.090 | 6.00 | 13,391 | 0.0% |
-| **3** | 0.0614 | 1.043 | 1.11 | 542,652 | 93.9% |
-| **3** | 0.8006 | 1.742 | 3.64 | 80,309 | 19.9% |
-| **4** | 0.0249 🏆 | 1.017 | 1.04 | 599,240 | 97.5% |
-| **4** | 0.5368 🏆 | 1.451 | 2.24 | 292,181 | 46.3% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.6602 | 1.580 | 3.57 | 91,234 | 34.0% |
+| **1** | Subword | 1.1781 | 2.263 | 9.32 | 1,145 | 0.0% |
+| **2** | Word | 0.1256 | 1.091 | 1.21 | 324,656 | 87.4% |
+| **2** | Subword | 0.9990 | 1.999 | 5.68 | 10,664 | 0.1% |
+| **3** | Word | 0.0281 | 1.020 | 1.04 | 392,645 | 97.2% |
+| **3** | Subword | 0.7935 | 1.733 | 3.66 | 60,534 | 20.6% |
+| **4** | Word | 0.0114 🏆 | 1.008 | 1.02 | 406,500 | 98.9% |
+| **4** | Subword | 0.5614 | 1.476 | 2.33 | 221,628 | 43.9% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. хундерилифастандартгьо ̄ ⲛ ̄ л . costumes caucasus circassians caucasus . — anatidae хъизан патагӏ...`
-2. `, къагiидаби . амма руго . байрамал лъугьа - бакъбаккул кавказияб календар категория : гардарики ,`
-3. `- абилеб ) мугъчӏваял категория : « вечера на хадидже категория : пко « моноклеr »`
+1. `ва бищун хирияб рокьуе рецц гьабун росулӏ историкияб кьучӏги x гіасру ккола гъуниб округалъул цӏигӏу...`
+2. `буго гьединго гьолокьги бекьизабун буго шумеразул ги къасимехалъ бачӏингун чагӏазда макьаби контрола...`
+3. `ккола гьижрияб соналъул 29 август цояб гьелъул буго гьанже батизе бегьула 1 гуржистаналъул бищун бор...`
 
 **Context Size 2:**
 
-1. `категория : гӏанди - гӏорул жанилъуда , ралъдал гьумералдаса 1869 метраялъ тіадегіан . хіоралъул тіа...`
-2. `) . эратосфениде ( iii гӏ . байбихьи ) букіана патрикиясул титул , гьелдаса хадуб дагъистаналде .`
-3. `) , лачен ( falco peregrinus ) , продолжительность 2 ч . ii – i гіасрабазул гіорхъода`
+1. `росу буго лъарагӏлъиялда хасавхъала мухъалда хасавхъалаялдаса 24 км ялъ жанубиябгин бакъбаккудехун а...`
+2. `лъугьа бахъинал гьаруна хвана ишараби мугъчӏ��аял гь балагье хіужаби иццал адабият гіурус маціалда бу...`
+3. `география росу буго лъарагӏлъиялда дибирилросу мухъалда дибирилросуялдаса 10 км ялъ шималиябгин бакъ...`
 
 **Context Size 3:**
 
-1. `география росу буго мухъалъул центер уркарахъалдаса 50 км - лъ жанубияб бакътӏерхьудехун . демографи...`
-2. `. география росу буго ралъдал гьурматӏама 606 метралъ борхалъуда , мухъалъул центер хунзахъа шималия...`
-3. `. категория : ираналъул останал * категория : азиялъул исламиял хіаракатчагіи категория : тіалибан`
+1. `география росу буго мухъалъул марказ хӏебдаса 15 километралъ бакъбаккудехун халкъ мугъчӏваял регӏела...`
+2. `лъугьа бахъинал гьаруна хвана ишараби мугъчӏваял гь балагье трактат адабият тайпаби изданиял`
+3. `байрамал лъугьа бахъинал гьаруна хвана ишараби мугъчӏваял гь балагье трактат адабият тайпаби издания...`
 
 **Context Size 4:**
 
-1. `. география росу буго мухъалъул марказ лъаратӏаса 11 км - алъ шималалдехун . демография ккола моноэт...`
-2. `география росу буго мухъалъул центер дешлахӏаралдаса 13 км - лъ рикӏкӏад . история 1886 соналъул бая...`
-3. `. мугъчӏваял категория : гӏандалазул бол чагӏи категория : кавказалъул имамзаби`
+1. `байрамал лъугьа бахъинал гьаруна хвана ишараби мугъчӏваял гь балагье`
+2. `къо байрамал лъугьа бахъинал гьаруна хвана ишараби мугъчӏваял гь балагье`
+3. `лъугьа бахъинал гьаруна хвана ишараби мугъчӏваял гь балагье`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_iv–_гъугіо_usth`
+2. `абиза_и._д_2%_ке`
+3. `л_—_ilissoldan_|`
+
+**Context Size 2:**
+
+1. `алъахъану_кконие_`
+2. `л_адекалабаяракӏ)`
+3. `лъул_кіаялъул_на_`
+
+**Context Size 3:**
+
+1. `ул_къотӏагораний_в`
+2. `лъул_реал_карт_гӏа`
+3. `ъулгун_ар-рип_хъал`
+
+**Context Size 4:**
+
+1. `лъул_хіалалда_чӏали`
+2. `ъул_большая_и_казбе`
+3. `алъул_руго_9:_мугъч`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 97.5% predictability
+- **Best Predictability:** Context-4 (word) with 98.9% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (292,181 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (221,628 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -253,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 38,576 |
-| Total Tokens | 474,364 |
-| Mean Frequency | 12.30 |
+| Vocabulary Size | 34,392 |
+| Total Tokens | 405,867 |
+| Mean Frequency | 11.80 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 81.10 |
+| Frequency Std Dev | 73.46 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ва | 7,190 |
-| 2 | категория | 6,086 |
-| 3 | буго | 5,703 |
-| 4 | бугеб | 2,911 |
-| 5 | ккола | 2,903 |
-| 6 | росу | 2,847 |
-| 7 | мухъалъул | 2,671 |
-| 8 | гьеб | 2,187 |
-| 9 | дагъистаналъул | 1,923 |
-| 10 | росдал | 1,903 |
+| 1 | ва | 7,245 |
+| 2 | буго | 5,074 |
+| 3 | ккола | 2,830 |
+| 4 | бугеб | 2,699 |
+| 5 | гьеб | 2,222 |
+| 6 | росу | 2,175 |
+| 7 | мухъалъул | 2,030 |
+| 8 | цо | 1,833 |
+| 9 | the | 1,815 |
+| 10 | соналъ | 1,799 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | уркутамахьи | 2 |
-| 2 | континуумалде | 2 |
-| 3 | къулецӏмаги | 2 |
-| 4 | гьаркӏасуниб | 2 |
-| 5 | махӏарги | 2 |
-| 6 | пилибхиталъул | 2 |
-| 7 | заповедникалда | 2 |
-| 8 | пилибхит | 2 |
-| 9 | лъалъадул | 2 |
-| 10 | хӏанчӏи | 2 |
+| 1 | долтул | 2 |
+| 2 | кӏалалдаса | 2 |
+| 3 | шаргі | 2 |
+| 4 | харитӏун | 2 |
+| 5 | луткунги | 2 |
+| 6 | беглъуда | 2 |
+| 7 | къацӏар | 2 |
+| 8 | мичегь | 2 |
+| 9 | хъурукал | 2 |
+| 10 | мягьле | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9487 |
-| R² (Goodness of Fit) | 0.992879 |
+| Zipf Coefficient | 0.9506 |
+| R² (Goodness of Fit) | 0.993368 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 22.2% |
-| Top 1,000 | 49.8% |
-| Top 5,000 | 72.6% |
-| Top 10,000 | 82.2% |
+| Top 100 | 22.5% |
+| Top 1,000 | 50.8% |
+| Top 5,000 | 73.6% |
+| Top 10,000 | 83.3% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9929 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 22.2% of corpus
-- **Long Tail:** 28,576 words needed for remaining 17.8% coverage
+- **Zipf Compliance:** R²=0.9934 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 22.5% of corpus
+- **Long Tail:** 24,392 words needed for remaining 16.7% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -323,24 +384,129 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 12,900 | 32 | 4.114 | 0.854 | 0.8716 🏆 |
-| **mono_64d** | 12,900 | 64 | 4.625 | 0.771 | 0.7752 |
-| **mono_128d** | 12,900 | 128 | 4.775 | 0.759 | 0.3123 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8716 🏆 | 0.3278 | N/A | N/A |
+| **mono_64d** | 64 | 0.7240 | 0.2821 | N/A | N/A |
+| **mono_128d** | 128 | 0.2461 | 0.2702 | N/A | N/A |
 
 ### Key Findings
 
 - **Best Isotropy:** mono_32d with 0.8716 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 12,900 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Semantic Density:** Average pairwise similarity of 0.2934. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-гь` | гьамчукъотӏи, гьечӏони, гьаркьалги |
+| `-гӏ` | гӏасру, гӏаракъи, гӏелмуялде |
+| `-ма` | материялъул, машгьадалда, магіарухъ |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-л` | сабабал, материялъул, рикӏкӏиналъул |
+| `-а` | елена, современника, шагьаралда |
+| `-ул` | материялъул, рикӏкӏиналъул, хӏажиевасул |
+| `-да` | шагьаралда, машгьадалда, флорида |
+| `-ъул` | материялъул, рикӏкӏиналъул, медициналъул |
+| `-лъул` | материялъул, рикӏкӏиналъул, медициналъул |
+| `-ал` | сабабал, кьурахарал, къезавидал |
+| `-лда` | шагьаралда, машгьадалда, борталда |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `алъу` | 1.82x | 100 contexts | алъул, далъун, ралъуе |
+| `агьа` | 1.89x | 59 contexts | дагьа, багьа, загьаб |
+| `ялъу` | 2.04x | 43 contexts | ялъул, аялъул, ялъуни |
+| `ьабу` | 2.16x | 29 contexts | гьабу, гьабун, кьабун |
+| `иялъ` | 1.96x | 36 contexts | абиялъе, химиялъ, лъиялъе |
+| `иялд` | 1.83x | 35 contexts | сиялда, азиялда, азиялде |
+| `анал` | 1.42x | 70 contexts | данал, канал, ханал |
+| `ралъ` | 1.49x | 53 contexts | ралъад, ралъуе, хералъ |
+| `буге` | 2.00x | 17 contexts | бугел, бугез, бугеб |
+| `иста` | 2.02x | 16 contexts | систан, христа, лазистан |
+| `лдас` | 2.06x | 15 contexts | лдаса, алдаса, ялдаса |
+| `азда` | 1.62x | 32 contexts | мазда, раздан, ишазда |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-гь` | `-л` | 44 words | гьавамухъал, гьудулзабазул |
+| `-ма` | `-л` | 40 words | мажлисалъул, маринил |
+| `-гӏ` | `-л` | 38 words | гӏурусазул, гӏалиевалъул |
+| `-ма` | `-а` | 35 words | макъалоялда, малъана |
+| `-гӏ` | `-а` | 29 words | гӏуцӏиялда, гӏодула |
+| `-гь` | `-а` | 28 words | гьада, гьала |
+| `-гӏ` | `-ул` | 25 words | гӏурусазул, гӏалиевалъул |
+| `-гь` | `-ул` | 24 words | гьудулзабазул, гьезул |
+| `-ма` | `-ул` | 21 words | мажлисалъул, мактабалъул |
+| `-ма` | `-да` | 16 words | макъалоялда, макъалаялда |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| руччабаздаги | **`руччабаз-да-ги`** | 6.0 | `руччабаз` |
+| хронологиялъул | **`хронология-лъул`** | 4.5 | `хронология` |
+| теориялда | **`теория-лда`** | 4.5 | `теория` |
+| къавмазул | **`къавмаз-ул`** | 4.5 | `къавмаз` |
+| къанагӏатал | **`къанагӏат-ал`** | 4.5 | `къанагӏат` |
+| групалъул | **`група-лъул`** | 4.5 | `група` |
+| ракьалъул | **`ракьа-лъул`** | 4.5 | `ракьа` |
+| такрарлъул | **`такрар-лъул`** | 4.5 | `такрар` |
+| алвеолариялги | **`алвеолариял-ги`** | 4.5 | `алвеолариял` |
+| европалъул | **`европа-лъул`** | 4.5 | `европа` |
+| гьабулаго | **`гь-абулаго`** | 4.5 | `абулаго` |
+| рахъалъги | **`рахъалъ-ги`** | 4.5 | `рахъалъ` |
+| пассажирги | **`пассажир-ги`** | 4.5 | `пассажир` |
+| партиялъул | **`партия-лъул`** | 4.5 | `партия` |
+| оппозициялъул | **`оппозиция-лъул`** | 4.5 | `оппозиция` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language AV appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -348,11 +514,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.58x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (502) |
-| Markov | **Context-4** | Highest predictability (97.5%) |
+| Tokenizer | **64k BPE** | Best compression (4.70x) |
+| N-gram | **2-gram** | Lowest perplexity (428) |
+| Markov | **Context-4** | Highest predictability (98.9%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -542,7 +709,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -558,7 +726,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 20:39:38*
+*Report Date: 2026-01-03 05:23:28*

models/embeddings/monolingual/av_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e9b6638db57121afe5ec40e5d975a427a9f37c786498e2a2d32874648d6e940b
-size 1037515926
+oid sha256:086998639c3328d1f88a224eb653bef49f8aa011b5880d7feb85792cbe742361
+size 1036208926

models/embeddings/monolingual/av_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 12900
+  "vocab_size": 11654
 }

models/embeddings/monolingual/av_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a659bfb895d802a760b82e4bfa4c8d06af1ca7e7e48c9edcbe7eea312922c4c5
-size 259608726
+oid sha256:6290250d27a46d72b90ede94292690c45a0a3f14bdbf05ab7aa5d07aa2093541
+size 259258654

models/embeddings/monolingual/av_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 12900
+  "vocab_size": 11654
 }

models/embeddings/monolingual/av_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:42a486c86184075e6fbcad6c04646238858ba9785c971dda95529c259785636b
-size 518911126
+oid sha256:5b709ff729efabdd809f80746b620dba08607a5b3377d6237ee6f2434e1eb3c2
+size 518242078

models/embeddings/monolingual/av_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 12900
+  "vocab_size": 11654
 }

models/subword_markov/av_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:80d9f829d3520985c3ec2e555c4f56f88029ba50c0f8e0a8d143025208979f1b
-size 96598
+oid sha256:4517ac9631ea8dbfee38f5e0a123dfdfec510e3a9c1219d351bc1fd509b60c17
+size 81084

models/subword_markov/av_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "av",
-  "unique_contexts": 1091,
-  "total_transitions": 4387638
+  "unique_contexts": 1145,
+  "total_transitions": 3671343
 }

models/subword_markov/av_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e31801ab66f88e5b5e45a7c74584663475f8a158e42122c638b42657aca01fe
-size 605158
+oid sha256:2851c37f629eaaf5084dca27968d6cf91fd8843aa330d489a4440f208d3e60cd
+size 486043

models/subword_markov/av_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "av",
-  "unique_contexts": 13391,
-  "total_transitions": 4383567
+  "unique_contexts": 10664,
+  "total_transitions": 3667770
 }

models/subword_markov/av_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2e9124d975e5792b36fc393927806d32e658235b480893ccef7d0dde1592724b
-size 2084996
+oid sha256:e93984ebab9588d00463857f800a79e61319598925a11d5785c24e04809162f3
+size 1681476

models/subword_markov/av_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "av",
-  "unique_contexts": 80309,
-  "total_transitions": 4379496
+  "unique_contexts": 60534,
+  "total_transitions": 3664197
 }

models/subword_markov/av_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1e309e2636831f32a2161c4e5b0ad24463660acaea5b1b23d507678e2b7e207f
-size 5912238
+oid sha256:df78890cb55850cde1e4fbcdb06710bcbf3c0a04113c2fb5a81c53eb29e9a8bc
+size 4643917

models/subword_markov/av_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "av",
-  "unique_contexts": 292181,
-  "total_transitions": 4375425
+  "unique_contexts": 221628,
+  "total_transitions": 3660624
 }

models/subword_ngram/av_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4efe6ef223dd1f6472674537d1c596680156ddacece7718273c5a6229338ee6f
-size 67795
+oid sha256:933b0de1b63cb20550e97fa2419bacf817141c74827bc4e101e5d76c7780509c
+size 54611

models/subword_ngram/av_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "av",
-  "unique_ngrams": 5314,
-  "total_ngrams": 4387638
+  "unique_ngrams": 4133,
+  "total_ngrams": 3671343
 }

models/subword_ngram/av_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fe60524de25890ebbb4579aeb1d0dc4e16f60cf7c9a21aadae938fe7d8a651c0
-size 482417
+oid sha256:ca32b51e264e1da8fb8ad605682f565e4a0fd97590d147b080946a6cf8993da2
+size 370254

models/subword_ngram/av_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "av",
-  "unique_ngrams": 36896,
-  "total_ngrams": 4383567
+  "unique_ngrams": 28949,
+  "total_ngrams": 3667770
 }

models/subword_ngram/av_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0969deb3aaf25140d6a8f6829e34559fe563aa08e2863a2d91328f85f8f16ac0
-size 1846055
+oid sha256:1511e4aa7f03fe2637243fa72bb121ccae146a0024ee7a81f233a708cc2778bb
+size 1469426

models/subword_ngram/av_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "av",
-  "unique_ngrams": 151649,
-  "total_ngrams": 4379496
+  "unique_ngrams": 119337,
+  "total_ngrams": 3664197
 }

models/tokenizer/av_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:37ab57b7217643d6cfb077a937974ffd4dcebcfabf852aa82bfdfe4ce2a0500d
-size 576619
+oid sha256:95f05447a801c104acaac9e4a45ef1ade9a750529b7be0b69f4ee702b40bdd0f
+size 579855

models/tokenizer/av_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:28d374a06b6b7aa8a4d3941bcd4695e5533544ad636d5d0bb610fb7793bc4354
-size 936914
+oid sha256:bcc76ebd732e7de0cc3f1d1a7445d6e9812fc5861347bfbdb764a34ab378093a
+size 943739

models/tokenizer/av_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d5521eccdbb9f46291a027fc56f120b592ff6dfdafcd09218747f5dfebcdc505
-size 1716704
+oid sha256:e25c37c305a3f71acc7fde1f68cea5d1b999e9cc76e1a9ae7e946535b348e7eb
+size 1709450

models/tokenizer/av_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0fe4689b0c6b4beca83bfdeb9649a3570f36538f43ca7cf4f02c432b94214c16
-size 403337
+oid sha256:69224dbe8521456dbe0800e5c74893d56ecd3d316ce5785f5ea5922ff2f1af24
+size 404399

models/vocabulary/av_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3288037a303bb285d697f0b1691c0479ba6d9e29db6034c70012c05716d329f1
-size 738845
+oid sha256:a7310d663e46f21c8df63d5030b41ef164690182783e77aaf70f0bb2efff7bb0
+size 661413

models/vocabulary/av_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "av",
-  "vocabulary_size": 38576,
+  "vocabulary_size": 34392,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.19482218346291424,
+    "type_token_ratio": 0.1973921591928009,
     "coverage": {
-      "top_100": 0.1948757649215124,
-      "top_1000": 0.43634707482188784,
-      "top_5000": 0.6366234812427942,
-      "top_10000": 0.7208073432465191
+      "top_100": 0.19702269707347111,
+      "top_1000": 0.4456231702442555,
+      "top_5000": 0.6456771851739821,
+      "top_10000": 0.7303661131936867
     },
-    "hapax_count": 66868,
-    "hapax_ratio": 0.6341565191001859,
-    "total_documents": 4071
+    "hapax_count": 56968,
+    "hapax_ratio": 0.623555166374781,
+    "total_documents": 3573
   }
 }

models/word_markov/av_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:63f373412e062fd6835774cb1266eaa3e245850dba38a8e6151d4ac7a4759ede
-size 5274705
+oid sha256:e06648f80de88e4993ac542ab602cc9aac6b01b06f56ff96c14658afaf6be279
+size 4391092

models/word_markov/av_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "av",
-  "unique_contexts": 105500,
-  "total_transitions": 728256
+  "unique_contexts": 91234,
+  "total_transitions": 459262
 }

models/word_markov/av_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fb88d7a9d6079d49efd1ad1b30db00e9d3c96aab81c84a0f8d0ef679e10fe3d4
-size 11187735
+oid sha256:c68afd2ecd43feece23866755466060024bed9599ffd5b4fb7f0eebba9e44b7c
+size 9606282

models/word_markov/av_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "av",
-  "unique_contexts": 384678,
-  "total_transitions": 724185
+  "unique_contexts": 324656,
+  "total_transitions": 455689
 }

models/word_markov/av_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:907faf6f55ecd04d74aa3d7f21d0111dda47ad2d999e00d289b0665dfe132da5
-size 14841527
+oid sha256:2f09b691f7a4f20607265b8fbe10331f64de10a5d0492129209c70e0fac37837
+size 11942432

models/word_markov/av_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "av",
-  "unique_contexts": 542652,
-  "total_transitions": 720116
+  "unique_contexts": 392645,
+  "total_transitions": 452116
 }

models/word_markov/av_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:910fed69ecfd672367e2d66b7147fecc45ddedd0acda63f49e00c5b1add64c18
-size 17509163
+oid sha256:d983a9226228a9202fb65e4494c1671eacd7872fafdff4b618a0f3e99c703539
+size 14036981

models/word_markov/av_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "av",
-  "unique_contexts": 599240,
-  "total_transitions": 716050
+  "unique_contexts": 406500,
+  "total_transitions": 448543
 }

models/word_ngram/av_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6cb354b1f97cbdfc6d757a9c19d07428b4d51d46a086e2f6d4633877e66a449f
-size 309626
+oid sha256:ca8324fa6f1298f13453cb69c80bda1c34da42ca4d16390fc0f21e33c68f286c
+size 164610

models/word_ngram/av_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "av",
-  "unique_ngrams": 14725,
-  "total_ngrams": 728256
+  "unique_ngrams": 6413,
+  "total_ngrams": 459262
 }

models/word_ngram/av_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d2a68230c75c38fad71bc4e8c7c054c54a5ac1bae080c287811284115b39aee8
-size 461245
+oid sha256:d863fc44b978ead18ea6982afba0be260335838763c4435ad08e18f827ded683
+size 202995

models/word_ngram/av_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "av",
-  "unique_ngrams": 19718,
-  "total_ngrams": 724185
+  "unique_ngrams": 6427,
+  "total_ngrams": 455689
 }

models/word_ngram/av_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a2c985bdc681c13eadcce7267033132b7282125a774fd8af76143f2674a46d5
-size 1007425
+oid sha256:ca081b844faf3f648da186cea540ee1d8a8481a461588ae8b2a7f220a80f8acb
+size 568270

models/word_ngram/av_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "av",
-  "unique_ngrams": 39973,
-  "total_ngrams": 720116
+  "unique_ngrams": 17392,
+  "total_ngrams": 452116
 }