Upload all models and assets for an (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.796
+    value: 4.267
   - name: best_isotropy
     type: isotropy
-    value: 0.8139
+    value: 0.8193
   - name: vocabulary_size
     type: vocab
-    value: 217616
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # AN - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,53 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.240x | 3.20 | 0.1217% | 1,479,330 |
-| **16k** | 3.471x | 3.43 | 0.1304% | 1,380,762 |
-| **32k** | 3.657x | 3.61 | 0.1374% | 1,310,812 |
-| **64k** | 3.796x 🏆 | 3.75 | 0.1426% | 1,262,658 |
+| **8k** | 3.551x | 3.55 | 0.1259% | 1,215,405 |
+| **16k** | 3.845x | 3.85 | 0.1363% | 1,122,403 |
+| **32k** | 4.084x | 4.08 | 0.1448% | 1,056,930 |
+| **64k** | 4.267x 🏆 | 4.27 | 0.1513% | 1,011,533 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Schnaitsee (en bavaro Schnoatsee) ye un municipio d'o districto de Traunstein en...`
+**Sample 1:** `Anyos: - - Decenios: Anyos - Anyos - Anyos Sieglos: Sieglo X - Sieglo XI - Siegl...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁sch na it see ▁( en ▁bavaro ▁sch no at ... (+25 more)` | 35 |
-| 16k | `▁sch na it see ▁( en ▁bavaro ▁sch no at ... (+25 more)` | 35 |
-| 32k | `▁sch na it see ▁( en ▁bavaro ▁sch no at ... (+25 more)` | 35 |
-| 64k | `▁sch na it see ▁( en ▁bavaro ▁sch no at ... (+25 more)` | 35 |
+| 8k | `▁anyos : ▁- ▁- ▁decenios : ▁anyos ▁- ▁anyos ▁- ... (+15 more)` | 25 |
+| 16k | `▁anyos : ▁- ▁- ▁decenios : ▁anyos ▁- ▁anyos ▁- ... (+15 more)` | 25 |
+| 32k | `▁anyos : ▁- ▁- ▁decenios : ▁anyos ▁- ▁anyos ▁- ... (+15 more)` | 25 |
+| 64k | `▁anyos : ▁- ▁- ▁decenios : ▁anyos ▁- ▁anyos ▁- ... (+15 more)` | 25 |
 
-**Sample 2:** `Chesa puet estar:
-Chesa terreno con muito cheso u chacimiento de cheso.
-Chesa, m...`
+**Sample 2:** `Lo Buçon (en francés Aubusson) ye una localidat y comuna francesa situada en o d...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ch esa ▁puet ▁estar : ▁ch esa ▁terreno ▁con ▁muito ... (+23 more)` | 33 |
-| 16k | `▁ch esa ▁puet ▁estar : ▁ch esa ▁terreno ▁con ▁muito ... (+21 more)` | 31 |
-| 32k | `▁chesa ▁puet ▁estar : ▁chesa ▁terreno ▁con ▁muito ▁cheso ▁u ... (+18 more)` | 28 |
-| 64k | `▁chesa ▁puet ▁estar : ▁chesa ▁terreno ▁con ▁muito ▁cheso ▁u ... (+18 more)` | 28 |
+| 8k | `▁lo ▁bu ç on ▁( en ▁francés ▁a ub us ... (+28 more)` | 38 |
+| 16k | `▁lo ▁bu çon ▁( en ▁francés ▁a ub us son ... (+25 more)` | 35 |
+| 32k | `▁lo ▁bu çon ▁( en ▁francés ▁aub us son ) ... (+24 more)` | 34 |
+| 64k | `▁lo ▁bu çon ▁( en ▁francés ▁aub us son ) ... (+22 more)` | 32 |
 
-**Sample 3:** `L'ibón de ra Sartén ye un ibón situau chunto o garmo de ra Mina (2.581 m) en l'A...`
+**Sample 3:** `Holzmann ye un lugar d'o municipio de Chieming en o sud-este de Bavera, Alemanya...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁l ' ibón ▁de ▁ra ▁sar t én ▁ye ▁un ... (+41 more)` | 51 |
-| 16k | `▁l ' ibón ▁de ▁ra ▁sar t én ▁ye ▁un ... (+33 more)` | 43 |
-| 32k | `▁l ' ibón ▁de ▁ra ▁sar t én ▁ye ▁un ... (+33 more)` | 43 |
-| 64k | `▁l ' ibón ▁de ▁ra ▁sar tén ▁ye ▁un ▁ibón ... (+31 more)` | 41 |
+| 8k | `▁hol z mann ▁ye ▁un ▁lugar ▁d ' o ▁municipio ... (+28 more)` | 38 |
+| 16k | `▁hol z mann ▁ye ▁un ▁lugar ▁d ' o ▁municipio ... (+28 more)` | 38 |
+| 32k | `▁holz mann ▁ye ▁un ▁lugar ▁d ' o ▁municipio ▁de ... (+26 more)` | 36 |
+| 64k | `▁holz mann ▁ye ▁un ▁lugar ▁d ' o ▁municipio ▁de ... (+26 more)` | 36 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 3.796x compression
-- **Lowest UNK Rate:** 8k with 0.1217% unknown tokens
+- **Best Compression:** 64k achieves 4.267x compression
+- **Lowest UNK Rate:** 8k with 0.1259% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -123,57 +129,89 @@ Chesa, m...`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 18,567 🏆 | 14.18 | 299,504 | 21.6% | 42.2% |
-| **2-gram** | 321 🏆 | 8.33 | 8,390 | 62.9% | 98.8% |
-| **3-gram** | 65,327 | 16.00 | 654,301 | 13.6% | 29.7% |
-| **3-gram** | 2,716 | 11.41 | 70,003 | 23.4% | 68.9% |
-| **4-gram** | 178,250 | 17.44 | 1,306,932 | 8.6% | 21.3% |
-| **4-gram** | 14,930 | 13.87 | 389,532 | 12.2% | 38.1% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 25,531 | 14.64 | 231,538 | 16.8% | 37.4% |
+| **2-gram** | Subword | 258 🏆 | 8.01 | 7,000 | 68.7% | 99.3% |
+| **3-gram** | Word | 86,752 | 16.40 | 456,958 | 8.4% | 23.0% |
+| **3-gram** | Subword | 2,153 | 11.07 | 52,839 | 25.8% | 73.4% |
+| **4-gram** | Word | 208,198 | 17.67 | 890,645 | 6.8% | 17.2% |
+| **4-gram** | Subword | 12,189 | 13.57 | 289,840 | 12.6% | 39.7% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `d a` | 106,922 |
+| 2 | `d o` | 105,903 |
+| 3 | `en a` | 60,539 |
+| 4 | `en o` | 45,891 |
+| 5 | `de l` | 37,271 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `a provincia de` | 17,407 |
+| 2 | `d a provincia` | 13,389 |
+| 3 | `una superficie de` | 12,709 |
+| 4 | `suya población ye` | 12,409 |
+| 5 | `población ye de` | 12,354 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `suya población ye de` | 12,288 |
+| 2 | `en una superficie de` | 12,121 |
+| 3 | `d a provincia de` | 12,081 |
+| 4 | `habitants en una superficie` | 11,267 |
+| 5 | `a suya población ye` | 11,250 |
+
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `d '` | 513,061 |
-| 2 | `| |` | 216,222 |
-| 3 | `categoría :` | 138,788 |
-| 4 | `' a` | 107,710 |
-| 5 | `' o` | 106,730 |
+| 1 | `a _` | 1,856,533 |
+| 2 | `_ d` | 1,594,808 |
+| 3 | `e _` | 1,531,455 |
+| 4 | `s _` | 1,293,859 |
+| 5 | `n _` | 1,201,430 |
 
-**3-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `d ' a` | 107,463 |
-| 2 | `d ' o` | 106,609 |
-| 3 | `| | |` | 81,137 |
-| 4 | `categoría : cintas` | 47,877 |
-| 5 | `| - |` | 47,248 |
+| 1 | `_ d e` | 886,389 |
+| 2 | `d e _` | 768,800 |
+| 3 | `_ d '` | 488,988 |
+| 4 | `e n _` | 472,591 |
+| 5 | `_ e n` | 449,047 |
 
-**4-grams:**
+**4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `| | | |` | 40,524 |
-| 2 | `- | | |` | 39,996 |
-| 3 | `| | socorro |` | 25,824 |
-| 4 | `| linear | -` | 25,824 |
-| 5 | `| socorro | |` | 25,824 |
+| 1 | `_ d e _` | 734,534 |
+| 2 | `_ e n _` | 392,856 |
+| 3 | `_ d ' a` | 233,826 |
+| 4 | `a _ d e` | 183,991 |
+| 5 | `_ c o n` | 176,849 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 321
+- **Best Perplexity:** 2-gram (subword) with 258
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~38% of corpus
+- **Coverage:** Top-1000 patterns cover ~40% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -181,55 +219,86 @@ Chesa, m...`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.6636 | 1.584 | 5.61 | 525,914 | 33.6% |
-| **1** | 1.0327 | 2.046 | 7.36 | 3,250 | 0.0% |
-| **2** | 0.3652 | 1.288 | 2.21 | 2,944,964 | 63.5% |
-| **2** | 0.8991 | 1.865 | 6.02 | 23,920 | 10.1% |
-| **3** | 0.1730 | 1.127 | 1.41 | 6,515,045 | 82.7% |
-| **3** | 0.8416 | 1.792 | 4.55 | 143,854 | 15.8% |
-| **4** | 0.0911 🏆 | 1.065 | 1.19 | 9,175,931 | 90.9% |
-| **4** | 0.7384 🏆 | 1.668 | 3.44 | 654,131 | 26.2% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.9747 | 1.965 | 7.58 | 367,064 | 2.5% |
+| **1** | Subword | 0.7827 | 1.720 | 5.89 | 3,499 | 21.7% |
+| **2** | Word | 0.3412 | 1.267 | 2.01 | 2,775,765 | 65.9% |
+| **2** | Subword | 0.8316 | 1.780 | 5.33 | 20,583 | 16.8% |
+| **3** | Word | 0.1546 | 1.113 | 1.33 | 5,573,548 | 84.5% |
+| **3** | Subword | 0.7758 | 1.712 | 4.33 | 109,746 | 22.4% |
+| **4** | Word | 0.0738 🏆 | 1.052 | 1.14 | 7,417,165 | 92.6% |
+| **4** | Subword | 0.7143 | 1.641 | 3.37 | 474,426 | 28.6% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
 
-### Generated Text Samples
+**Context Size 1:**
+
+1. `de barcelona cuan obtiene un municipio espanyol o suyo segundo punto alchido d o brien moore`
+2. `d a z vs üü alto penedés información de chunio de argañán a camera a population`
+3. `a risilleta a suya identidat y bi ha una economía como dancing in europe bbc d`
+
+**Context Size 2:**
+
+1. `d a provincia de barcelona iste matrimonio naixoron 2 fillas a on a hansa montó as suyas`
+2. `d o far west anexionando muitos territorios que componeban a corona d aragón que explicitament denom...`
+3. `en a suya población ye de 643 habitants en una superficie de 16 01 491 teruel torralba`
+
+**Context Size 3:**
+
+1. `a provincia de uesca d as penyas de riglos solano n 42 27 29 21 e 0 27`
+2. `d a provincia de zaragoza en la suya part d o conchunto d o sud con a huerta`
+3. `una superficie de 88 4 km y una densidat de población de 10 44 hab km cifras oficiales`
+
+**Context Size 4:**
 
-Below are text samples generated from each Markov chain model:
+1. `suya población ye de en una superficie de 20 1 km y una densidat de población de 5 24`
+2. `en una superficie de 82 09 km con una densidat d hab km cheografía a localidat de gonnosnò ye`
+3. `d a provincia de chirona y partiu chudicial de teruel catálogo de pueblos y municipios de aragón est...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, pedanía . por aon serviban pa permitir que garra problema d ' arquitectura . bibliografía`
-2. `de febrero , ugariticas y legendarios del xúcar que lo país exerce como l ' octubre`
-3. `. isbn 978 - | 30 de roses en a estudiants mes recientment plegada d '`
+1. `_ptonena_le_a_ev`
+2. `atarro_d'a_erra.`
+3. `e_ula_coril._qus`
 
 **Context Size 2:**
 
-1. `d ' aventuras dramaticas de 1992 22px | border espanya 22px francia 263 . je serai allée`
-2. `| | | | 13 d ' el plantío como equipo local dica l ' ataque .`
-3. `categoría : cintas interpretadas por joe baker categoría : cintas interpretadas por mort engelberg c...`
+1. `a_forangacia_ingà`
+2. `_de_le_d'isten_co`
+3. `e_ye_suff_una_fes`
 
 **Context Size 3:**
 
-1. `d ' a empresa atac . redolada villa lazzaroni via appia nuova , circa d ' a mar`
-2. `d ' o fisico y incheniero estausunidense karl jansky ( † 1950 ) . - naixencia en hamburgo`
-3. `| | | | 12 de setiembre , 1998 loneos 52773 - 17 d ' octubre , 1977`
+1. `_de_barrisor_intas`
+2. `de_a_prencias)._th`
+3. `_d'a_latín_a_lo_si`
 
 **Context Size 4:**
 
-1. `| | | | 5 de chulio , 2000 loneos 33965 - 10 de chulio , 2000 linear 61258`
-2. `- | | | | 2 de febrero , 2000 linear 50315 - 2 de febrero , 2000 linear`
-3. `| | socorro | | linear | - | 87422 - | | | | 16 de setiembre ,`
+1. `_de_l'animent_y_pol`
+2. `_en_tiembre_de_davi`
+3. `_d'a_por_o_nuesta_y`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 90.9% predictability
+- **Best Predictability:** Context-4 (word) with 92.6% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (654,131 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (474,426 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -245,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 217,616 |
-| Total Tokens | 13,131,016 |
-| Mean Frequency | 60.34 |
+| Vocabulary Size | 182,961 |
+| Total Tokens | 11,551,476 |
+| Mean Frequency | 63.14 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 2696.90 |
+| Frequency Std Dev | 2812.87 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | de | 776,198 |
-| 2 | d | 517,135 |
-| 3 | a | 444,267 |
-| 4 | en | 414,383 |
-| 5 | o | 303,229 |
-| 6 | y | 249,159 |
-| 7 | categoría | 139,745 |
-| 8 | que | 128,308 |
-| 9 | l | 111,482 |
-| 10 | ye | 109,905 |
+| 1 | de | 738,428 |
+| 2 | d | 494,628 |
+| 3 | a | 437,762 |
+| 4 | en | 406,259 |
+| 5 | o | 301,745 |
+| 6 | y | 244,743 |
+| 7 | que | 126,444 |
+| 8 | l | 109,014 |
+| 9 | ye | 108,632 |
+| 10 | una | 104,144 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | rochut | 2 |
-| 2 | méchaly | 2 |
-| 3 | wiedemann | 2 |
-| 4 | limotte | 2 |
-| 5 | wlodkowski | 2 |
-| 6 | taos | 2 |
-| 7 | slovis | 2 |
-| 8 | samaha | 2 |
-| 9 | seros | 2 |
-| 10 | cookeville | 2 |
+| 1 | koftinoff | 2 |
+| 2 | landlocked | 2 |
+| 3 | hamidi | 2 |
+| 4 | tangy | 2 |
+| 5 | sélignac | 2 |
+| 6 | cômene | 2 |
+| 7 | varneton | 2 |
+| 8 | mackelway | 2 |
+| 9 | wigutow | 2 |
+| 10 | críspulo | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0856 |
-| R² (Goodness of Fit) | 0.997842 |
+| Zipf Coefficient | 1.0685 |
+| R² (Goodness of Fit) | 0.998276 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 43.1% |
-| Top 1,000 | 66.3% |
-| Top 5,000 | 80.4% |
-| Top 10,000 | 85.6% |
+| Top 100 | 44.9% |
+| Top 1,000 | 66.9% |
+| Top 5,000 | 80.7% |
+| Top 10,000 | 85.9% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9978 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 43.1% of corpus
-- **Long Tail:** 207,616 words needed for remaining 14.4% coverage
+- **Zipf Compliance:** R²=0.9983 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 44.9% of corpus
+- **Long Tail:** 172,961 words needed for remaining 14.1% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -315,24 +384,127 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 130,625 | 32 | 3.989 | 1.163 | 0.8139 🏆 |
-| **mono_64d** | 130,625 | 64 | 4.561 | 1.144 | 0.8131 |
-| **mono_128d** | 130,625 | 128 | 5.245 | 1.094 | 0.8002 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8177 | 0.3514 | N/A | N/A |
+| **mono_64d** | 64 | 0.8193 🏆 | 0.2716 | N/A | N/A |
+| **mono_128d** | 128 | 0.8061 | 0.2141 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8139 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 130,625 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_64d with 0.8193 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2790. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-co` | colleges, consagratos, consumible |
+| `-ca` | camelot, caldero, cabellera |
+| `-ma` | mardy, manaba, maeztu |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-s` | peraleios, noveciercos, engueradas |
+| `-a` | gina, pátria, aquileya |
+| `-as` | engueradas, febas, alimentadas |
+| `-os` | peraleios, noveciercos, consagratos |
+| `-an` | reflectan, goodman, recolectan |
+| `-es` | detalles, colleges, valses |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `ento` | 1.86x | 126 contexts | lento, rento, vento |
+| `cion` | 1.91x | 110 contexts | scion, nacion, accion |
+| `ient` | 1.67x | 177 contexts | aient, dient, oient |
+| `rago` | 2.01x | 59 contexts | ragot, drago, crago |
+| `ranc` | 1.63x | 140 contexts | ranch, rancó, rance |
+| `enci` | 1.55x | 164 contexts | encia, renci, venciu |
+| `laci` | 2.02x | 49 contexts | lacio, glacio, placid |
+| `nter` | 1.55x | 144 contexts | anter, unter, enter |
+| `obla` | 1.85x | 56 contexts | pobla, dobla, robla |
+| `renc` | 1.66x | 81 contexts | arenc, renci, rencor |
+| `ació` | 1.87x | 47 contexts | ación, nació, fació |
+| `mbre` | 1.66x | 75 contexts | ambre, ombre, mbret |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-co` | `-s` | 61 words | conformes, columbus |
+| `-co` | `-a` | 58 words | comunicaba, conservata |
+| `-ca` | `-s` | 54 words | caseilles, cavalls |
+| `-ca` | `-a` | 46 words | carezza, carabaza |
+| `-ma` | `-a` | 38 words | maquinista, mavumengwana |
+| `-ma` | `-s` | 31 words | mamuts, mads |
+| `-ca` | `-as` | 17 words | carpaticas, castanyas |
+| `-co` | `-as` | 16 words | conillas, coladas |
+| `-ca` | `-os` | 12 words | carpos, capuzamientos |
+| `-co` | `-es` | 11 words | conformes, contimparables |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| roncaleses | **`roncal-es-es`** | 6.0 | `roncal` |
+| coinciden | **`co-inciden`** | 4.5 | `inciden` |
+| processos | **`process-os`** | 4.5 | `process` |
+| productoras | **`productor-as`** | 4.5 | `productor` |
+| cobianchi | **`co-bianchi`** | 4.5 | `bianchi` |
+| elefantes | **`elefant-es`** | 4.5 | `elefant` |
+| musicales | **`musical-es`** | 4.5 | `musical` |
+| capelleta | **`ca-pelleta`** | 4.5 | `pelleta` |
+| lumerosas | **`lumer-os-as`** | 3.0 | `lumer` |
+| cantalojas | **`ca-ntaloj-as`** | 3.0 | `ntaloj` |
+| confiscatos | **`co-nfiscat-os`** | 3.0 | `nfiscat` |
+| concentraban | **`co-ncentrab-an`** | 3.0 | `ncentrab` |
+| cavanilles | **`ca-vanill-es`** | 3.0 | `vanill` |
+| aldeyanos | **`aldey-an-os`** | 3.0 | `aldey` |
+| cascavillos | **`ca-scavill-os`** | 3.0 | `scavill` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language AN appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -340,11 +512,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.80x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (321) |
-| Markov | **Context-4** | Highest predictability (90.9%) |
+| Tokenizer | **64k BPE** | Best compression (4.27x) |
+| N-gram | **2-gram** | Lowest perplexity (258) |
+| Markov | **Context-4** | Highest predictability (92.6%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -534,7 +707,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -550,7 +724,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 06:02:07*
+*Report Date: 2026-01-03 05:38:36*

models/embeddings/monolingual/an_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:df977b20d5a3138783d7bf5a29de1bc1087ae8adcf91357a08b6df3286a11c57
-size 1160074298
+oid sha256:3b8c8d2f8e8ebd6d647e7092d20c2a5ed3abae1a07a6be40fc84c516534b000d
+size 1148906690

models/embeddings/monolingual/an_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 130625
+  "vocab_size": 119918
 }

models/embeddings/monolingual/an_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a91f101e748ee0ad7dfaa908fad3faa7fc759c6736c6983d93227a6782d7011a
-size 291754298
+oid sha256:227c09966b020164fd97d17f0e971bef21897af0662c9891e19c26eef18f815d
+size 288809666

models/embeddings/monolingual/an_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 130625
+  "vocab_size": 119918
 }

models/embeddings/monolingual/an_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:68087cbf97ce11afd0878309f550fb79e3d9de063a23e6bcf07049c3996b50e8
-size 581194298
+oid sha256:168f92e58a097a484a5136a652e39c7b532d3e83d40af551f9b434d64c32dcf2
+size 575508674

models/embeddings/monolingual/an_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 130625
+  "vocab_size": 119918
 }

models/subword_markov/an_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:72a3cba7b373771551a54b5cf2cbc6318340e1a3501f9b25f3d8d2f48c472a23
-size 184463
+oid sha256:8d40de84446df679c4cebd27cec50dd996c043fbace52eee0439eb6badc7bd7c
+size 168642

models/subword_markov/an_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "an",
-  "unique_contexts": 3250,
-  "total_transitions": 82537174
+  "unique_contexts": 3499,
+  "total_transitions": 69109076
 }

models/subword_markov/an_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9fd27d05dfe93d7d9c282432331494cc61942215e42be9ce61f99ac74a7cf3f7
-size 1142612
+oid sha256:49319842d520d6479015d48b0398bc41858ef988fef862a150994662fcc06982
+size 916009

models/subword_markov/an_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "an",
-  "unique_contexts": 23920,
-  "total_transitions": 82486435
+  "unique_contexts": 20583,
+  "total_transitions": 69059397
 }

models/subword_markov/an_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1253e4ca1ee70bcf8b2ad676ddf40fff69b1220391f20884e1357bcafdecddc8
-size 5043098
+oid sha256:706cb13def2cb1c8316e030700d89c8a6b8edcdaa283466b09d5fe99ab709294
+size 3807898

models/subword_markov/an_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "an",
-  "unique_contexts": 143854,
-  "total_transitions": 82435696
+  "unique_contexts": 109746,
+  "total_transitions": 69009718
 }

models/subword_markov/an_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b7ca1fb83528f0a8a6947d8f7636bc221319b40e1be50b7b392667b6a626ecb3
-size 17868053
+oid sha256:5eba517deec4e418e88d7689e4203c7c287c5ff4a05060f0fbb2342c404dccc8
+size 13264812

models/subword_markov/an_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "an",
-  "unique_contexts": 654131,
-  "total_transitions": 82384957
+  "unique_contexts": 474426,
+  "total_transitions": 68960039
 }

models/subword_ngram/an_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e45d8c731d40f03a95d2d4171f1362f8c423ab80b642aec7287ad38018e6251d
-size 112683
+oid sha256:555bb0a9950d43d338c420105f45d6e40844826ad59879138e4521a5e4f7bc0c
+size 93665

models/subword_ngram/an_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "an",
-  "unique_ngrams": 8390,
-  "total_ngrams": 82537174
+  "unique_ngrams": 7000,
+  "total_ngrams": 69109076
 }

models/subword_ngram/an_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9e8f3eb6d597d9d55c84231a3112486a81249b52737f684c64905bb039763c82
-size 875250
+oid sha256:f78a3129dc57d3cdb79e700b2f8fd170e91b6a063ad7ecc8c3970aa700eb2eba
+size 691123

models/subword_ngram/an_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "an",
-  "unique_ngrams": 70003,
-  "total_ngrams": 82486435
+  "unique_ngrams": 52839,
+  "total_ngrams": 69059397
 }

models/subword_ngram/an_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fd8db606ff5dd4ef0b5fb54501a844e424ac0fab787cef5b599cdde9d02d1e43
-size 4436397
+oid sha256:2c2ad5d14d9c914207a8289e32db63c141e19a1ca414e8a0d26bb2bd2d7c2251
+size 3358033

models/subword_ngram/an_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "an",
-  "unique_ngrams": 389532,
-  "total_ngrams": 82435696
+  "unique_ngrams": 289840,
+  "total_ngrams": 69009718
 }

models/tokenizer/an_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a69d504bfe94be166ec6bfb3523a637720073719779a402e09380c3be46f5ed1
-size 510191
+oid sha256:7e6718ee086df807a9b8e62706d7c765c76896efe91b3e0c87de4d756afbca43
+size 511370

models/tokenizer/an_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e8d18b8e75d8486ad231a507c071537bae5b2fe7257de2c6b01e9d4b1bc9f37f
-size 789754
+oid sha256:3a3ebce2687e88197aaa7a091e4f04cb995c75343de7c69b2a452444c0e4ec12
+size 791243

models/tokenizer/an_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d2e6056e5c18ce40dfc1c918e4ffa96c311c860c6d18ff1eb23841c68899cfcd
-size 1358196
+oid sha256:7137d416046b5467fc533de58fb10d2dfc01761285b5489a24416366e18446ce
+size 1364521

models/tokenizer/an_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bfbfee5efa99c394ac891a3f732716a915eead3966ae1e24a13c2951303196e8
-size 373704
+oid sha256:b6b38bd103587bec698ef8bfa6fc1eca198cf371a05427efd0b9100615abdbd5
+size 374587

models/vocabulary/an_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4707a0b58ba6a4c9c5677662e07dabe17ddde1b83806f3d2f55abb4beaf51637
-size 3310355
+oid sha256:899ce817bf3d0cd63c52370382d0c8f7e06c5beb0c78ed09f131cf1aafd3d182
+size 2932502

models/vocabulary/an_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "an",
-  "vocabulary_size": 217616,
+  "vocabulary_size": 182961,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.03914079689391589,
+    "type_token_ratio": 0.03129810058501218,
     "coverage": {
-      "top_100": 0.4213646709719088,
-      "top_1000": 0.6475247834171538,
-      "top_5000": 0.7852817418595878,
-      "top_10000": 0.8360937926682902
+      "top_100": 0.4415954090654393,
+      "top_1000": 0.6581393006575422,
+      "top_5000": 0.793948937884549,
+      "top_10000": 0.8454613838789675
     },
-    "hapax_count": 308414,
-    "hapax_ratio": 0.5863049635952322,
-    "total_documents": 50739
+    "hapax_count": 184348,
+    "hapax_ratio": 0.50188805610535,
+    "total_documents": 49679
   }
 }

models/word_markov/an_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e4e3be111b565571799df73c6bc31d0d51b6e089aa29260ceffb95042a709c64
-size 25436455
+oid sha256:24ca3e38b9546b9a06f5df3b46ba1a09d874251e2d187e8bdb92e634b9a97756
+size 23502911

models/word_markov/an_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "an",
-  "unique_contexts": 525914,
-  "total_transitions": 17591589
+  "unique_contexts": 367064,
+  "total_transitions": 11686145
 }

models/word_markov/an_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7fa7fbbcf2879c8d8d4b0187b08a2c130097d304f1651be06b0290d592c31292
-size 69286803
+oid sha256:c73b675412ec88a37c27f3ed414e5087949843a731d927309077f06614513901
+size 65110098

models/word_markov/an_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "an",
-  "unique_contexts": 2944964,
-  "total_transitions": 17540850
+  "unique_contexts": 2775765,
+  "total_transitions": 11636466
 }

models/word_markov/an_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5b280d2444a1e04abe65378050ad54cd0a9e8454e02100eaa0530275d565379f
-size 118851189
+oid sha256:e609aeb7661d18f4043b4e6b0ed80afed876c3c30105b27506fba72732e47b0e
+size 103695006

models/word_markov/an_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "an",
-  "unique_contexts": 6515045,
-  "total_transitions": 17490111
+  "unique_contexts": 5573548,
+  "total_transitions": 11586787
 }

models/word_markov/an_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d44caecd25449243adac00d70be5caf8377f1fa6e9d2b68eabe22a90eabe0bd8
-size 156148146
+oid sha256:a1a638efc93bf5156a281bc3520210801cecab11b50bfc0049d2319bf73f3192
+size 130961488

models/word_markov/an_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "an",
-  "unique_contexts": 9175931,
-  "total_transitions": 17439372
+  "unique_contexts": 7417165,
+  "total_transitions": 11537108
 }

models/word_ngram/an_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7b5e5d9c55b22bc997a7bc412142ff833f50070bba74fc40661e41738efd2e8f
-size 3974678
+oid sha256:61e57673546517ca4e0a925bb3a888da339f48eba31c8c84442546c480d14b96
+size 3221474

models/word_ngram/an_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "an",
-  "unique_ngrams": 299504,
-  "total_ngrams": 17591589
+  "unique_ngrams": 231538,
+  "total_ngrams": 11686145
 }

models/word_ngram/an_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:520fd6eb5f53501aefe99f6d10405529fe4fc67a873bf2c192b942f70d1e3333
-size 9375840
+oid sha256:da9caf8a85ac19844e4a241494138ca37ea47684fde967f32db04f2aa4b9a824
+size 6789536

models/word_ngram/an_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "an",
-  "unique_ngrams": 654301,
-  "total_ngrams": 17540850
+  "unique_ngrams": 456958,
+  "total_ngrams": 11636466
 }

models/word_ngram/an_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6d0e04ea7106db32719462b5a9b93c4318c4769ea2c2071995be760c72248952
-size 19565625
+oid sha256:54224a8ae68f476eda16f0412c6b35eed02a6aeaf371db0788c6e009b177d091
+size 13943741

models/word_ngram/an_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "an",
-  "unique_ngrams": 1306932,
-  "total_ngrams": 17490111
+  "unique_ngrams": 890645,
+  "total_ngrams": 11586787
 }