Upload all models and assets for bs (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.389
+    value: 4.707
   - name: best_isotropy
     type: isotropy
-    value: 0.6746
+    value: 0.6837
   - name: vocabulary_size
     type: vocab
-    value: 558155
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BS - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,60 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.441x | 3.41 | 0.1222% | 1,452,315 |
-| **16k** | 3.798x | 3.76 | 0.1349% | 1,315,871 |
-| **32k** | 4.122x | 4.08 | 0.1464% | 1,212,400 |
-| **64k** | 4.389x 🏆 | 4.35 | 0.1559% | 1,138,483 |
+| **8k** | 3.624x | 3.62 | 0.1219% | 1,310,495 |
+| **16k** | 4.031x | 4.03 | 0.1355% | 1,178,214 |
+| **32k** | 4.403x | 4.40 | 0.1481% | 1,078,528 |
+| **64k** | 4.707x 🏆 | 4.71 | 0.1583% | 1,008,868 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Bovan je naseljeno mjesto u općini Rudo, Bosna i Hercegovina.
-
- Stanovništvo 
-
- ...`
+**Sample 1:** `Marija Amalija Austrijska se može odnositi na: Mariju Amaliju Josipu Anu caricu ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁bo van ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁rudo , ▁bosna ... (+11 more)` | 21 |
-| 16k | `▁bo van ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁rudo , ▁bosna ... (+11 more)` | 21 |
-| 32k | `▁bo van ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁rudo , ▁bosna ... (+11 more)` | 21 |
-| 64k | `▁bo van ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁rudo , ▁bosna ... (+11 more)` | 21 |
+| 8k | `▁marija ▁ama lija ▁austri jska ▁se ▁može ▁odnositi ▁na : ... (+24 more)` | 34 |
+| 16k | `▁marija ▁ama lija ▁austrijska ▁se ▁može ▁odnositi ▁na : ▁mari ... (+21 more)` | 31 |
+| 32k | `▁marija ▁ama lija ▁austrijska ▁se ▁može ▁odnositi ▁na : ▁mariju ... (+19 more)` | 29 |
+| 64k | `▁marija ▁ama lija ▁austrijska ▁se ▁može ▁odnositi ▁na : ▁mariju ... (+16 more)` | 26 |
 
-**Sample 2:** `Rovna je naseljeno mjesto u općini Bugojno, Bosna i Hercegovina.
-
- Stanovništvo ...`
+**Sample 2:** `Margetići su naseljeno mjesto u općini Novi Travnik, Bosna i Hercegovina. Stanov...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ro vna ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁bu go jno ... (+15 more)` | 25 |
-| 16k | `▁ro vna ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁bugo jno , ... (+13 more)` | 23 |
-| 32k | `▁ro vna ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁bugojno , ▁bosna ... (+11 more)` | 21 |
-| 64k | `▁ro vna ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁bugojno , ▁bosna ... (+11 more)` | 21 |
-
-**Sample 3:** `Hrvatska:
+| 8k | `▁mar ge ti ći ▁su ▁naseljeno ▁mjesto ▁u ▁općini ▁novi ... (+15 more)` | 25 |
+| 16k | `▁mar ge tići ▁su ▁naseljeno ▁mjesto ▁u ▁općini ▁novi ▁travnik ... (+14 more)` | 24 |
+| 32k | `▁mar ge tići ▁su ▁naseljeno ▁mjesto ▁u ▁općini ▁novi ▁travnik ... (+14 more)` | 24 |
+| 64k | `▁marge tići ▁su ▁naseljeno ▁mjesto ▁u ▁općini ▁novi ▁travnik , ... (+13 more)` | 23 |
 
- Novoselo Bilajsko, naselje grada Gospića, Ličko-senjska županija
- No...`
+**Sample 3:** `Lilić je naseljeno mjesto u općini Srbac, Bosna i Hercegovina. Stanovništvo Refe...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁hrvatska : ▁novo se lo ▁bila jsko , ▁naselje ▁grada ... (+29 more)` | 39 |
-| 16k | `▁hrvatska : ▁novo se lo ▁bila jsko , ▁naselje ▁grada ... (+27 more)` | 37 |
-| 32k | `▁hrvatska : ▁novo selo ▁bila jsko , ▁naselje ▁grada ▁gospi ... (+21 more)` | 31 |
-| 64k | `▁hrvatska : ▁novo selo ▁bila jsko , ▁naselje ▁grada ▁gospića ... (+18 more)` | 28 |
+| 8k | `▁li lić ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁sr bac , ... (+13 more)` | 23 |
+| 16k | `▁li lić ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁sr bac , ... (+13 more)` | 23 |
+| 32k | `▁li lić ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁srbac , ▁bosna ... (+11 more)` | 21 |
+| 64k | `▁li lić ▁je ▁naseljeno ▁mjesto ▁u ▁općini ▁srbac , ▁bosna ... (+11 more)` | 21 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.389x compression
-- **Lowest UNK Rate:** 8k with 0.1222% unknown tokens
+- **Best Compression:** 64k achieves 4.707x compression
+- **Lowest UNK Rate:** 8k with 0.1219% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -130,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 65,516 🏆 | 16.00 | 866,401 | 11.3% | 32.8% |
-| **2-gram** | 391 🏆 | 8.61 | 13,004 | 58.5% | 98.1% |
-| **3-gram** | 147,841 | 17.17 | 1,540,282 | 7.3% | 27.1% |
-| **3-gram** | 3,957 | 11.95 | 136,436 | 19.2% | 60.6% |
-| **4-gram** | 238,725 | 17.86 | 2,663,711 | 7.3% | 27.1% |
-| **4-gram** | 25,994 | 14.67 | 960,072 | 8.3% | 29.6% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 79,811 | 16.28 | 657,818 | 9.9% | 28.8% |
+| **2-gram** | Subword | 329 🏆 | 8.36 | 10,932 | 62.1% | 98.9% |
+| **3-gram** | Word | 98,469 | 16.59 | 914,717 | 11.8% | 30.2% |
+| **3-gram** | Subword | 3,226 | 11.66 | 100,952 | 20.8% | 64.4% |
+| **4-gram** | Word | 131,360 | 17.00 | 1,461,546 | 13.0% | 31.0% |
+| **4-gram** | Subword | 21,068 | 14.36 | 689,562 | 8.6% | 31.6% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `spiralna galaksija` | 91,081 |
+| 2 | `vanjski linkovi` | 67,671 |
+| 3 | `se u` | 45,349 |
+| 4 | `reference vanjski` | 43,829 |
+| 5 | `ngc ic` | 40,015 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `reference vanjski linkovi` | 43,767 |
+| 2 | `prečkasta spiralna galaksija` | 32,672 |
+| 3 | `zavod za statistiku` | 22,677 |
+| 4 | `popisu stanovništva godine` | 20,724 |
+| 5 | `na popisu stanovništva` | 20,183 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `kategorija :` | 276,348 |
-| 2 | `. godine` | 129,207 |
-| 3 | `0 ,` | 126,589 |
-| 4 | `) ,` | 106,390 |
-| 5 | `galaksija (` | 105,937 |
+| 1 | `na popisu stanovništva godine` | 20,088 |
+| 2 | `državni zavod za statistiku` | 14,619 |
+| 3 | `broj stanovnika po popisima` | 13,853 |
+| 4 | `reference vanjski linkovi u` | 13,661 |
+| 5 | `pogledajte novi opći katalog` | 13,518 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `spiralna galaksija (` | 75,074 |
-| 2 | `reference vanjski linkovi` | 44,816 |
-| 3 | `objekti kategorija :` | 43,868 |
-| 4 | `. godine .` | 40,322 |
-| 5 | `ngc / ic` | 40,009 |
+| 1 | `a _` | 5,676,715 |
+| 2 | `e _` | 4,422,458 |
+| 3 | `j e` | 3,860,834 |
+| 4 | `i _` | 3,755,142 |
+| 5 | `_ s` | 3,354,838 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `prečkasta spiralna galaksija (` | 26,935 |
-| 2 | `kategorija : naselja u` | 26,641 |
-| 3 | `vanjski linkovi kategorija :` | 21,329 |
-| 4 | `reference vanjski linkovi kategorija` | 19,302 |
-| 5 | `spiralna galaksija ( s0` | 15,207 |
+| 1 | `j e _` | 1,718,703 |
+| 2 | `n a _` | 1,228,627 |
+| 3 | `_ n a` | 1,166,020 |
+| 4 | `_ j e` | 1,117,037 |
+| 5 | `_ p o` | 1,073,431 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ j e _` | 915,938 |
+| 2 | `i j a _` | 454,224 |
+| 3 | `_ n a _` | 449,657 |
+| 4 | `_ s e _` | 393,812 |
+| 5 | `i j e _` | 313,056 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 391
+- **Best Perplexity:** 2-gram (subword) with 329
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~30% of corpus
+- **Coverage:** Top-1000 patterns cover ~32% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -188,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7933 | 1.733 | 7.93 | 1,327,715 | 20.7% |
-| **1** | 1.1910 | 2.283 | 9.62 | 3,680 | 0.0% |
-| **2** | 0.3580 | 1.282 | 2.20 | 10,525,382 | 64.2% |
-| **2** | 1.0115 | 2.016 | 7.35 | 35,380 | 0.0% |
-| **3** | 0.1428 | 1.104 | 1.32 | 23,101,514 | 85.7% |
-| **3** | 1.0489 | 2.069 | 5.94 | 260,071 | 0.0% |
-| **4** | 0.0616 🏆 | 1.044 | 1.12 | 30,570,937 | 93.8% |
-| **4** | 0.9074 🏆 | 1.876 | 3.99 | 1,544,375 | 9.3% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.9822 | 1.975 | 9.95 | 1,092,943 | 1.8% |
+| **1** | Subword | 1.0154 | 2.021 | 7.75 | 3,822 | 0.0% |
+| **2** | Word | 0.3063 | 1.237 | 1.90 | 10,856,748 | 69.4% |
+| **2** | Subword | 0.9503 | 1.932 | 6.62 | 29,614 | 5.0% |
+| **3** | Word | 0.1026 | 1.074 | 1.20 | 20,575,067 | 89.7% |
+| **3** | Subword | 0.9528 | 1.936 | 5.48 | 195,969 | 4.7% |
+| **4** | Word | 0.0377 🏆 | 1.026 | 1.06 | 24,704,289 | 96.2% |
+| **4** | Subword | 0.9418 | 1.921 | 4.19 | 1,073,568 | 5.8% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. tako noseći na strani , 9 . karijera muzička adaptacija sezone , danska . u`
-2. `, vinište se može značajno veći gradovi domaćini , koju je nešto što povećava kako se`
-3. `i imigranti su uzrokovali su naseljeno mjesto u glavu , pamuk se sastojale su počeli provaljivati`
+1. `i iskazivano pod bizantijsku teritoriju poljski 2 kola za zvornik i hercegovine a nastavljaju sa dru...`
+2. `je reference vanjski linkovi www portalanalitika me 31 okt 3 kalisz david friedrich ehrendorfer ehre...`
+3. `u plazmi parcijalni derivati otkriveni napad na evropskom prvenstvu su klasifikovani kao tipično do ...`
 
 **Context Size 2:**
 
-1. `kategorija : nukleinske kiseline . kasnije je ( ) jedan od 50 američkih saveznih država , teritorija`
-2. `. godine . najbliži tome je podjela na : mehri - muedždžel ( odgođeni ) . datum`
-3. `0 , 000291 * p , gdje je m = 12 , 9 l nanolitar nl nl`
+1. `spiralna galaksija s0 a ic 0 66 spiralna galaksija s također pogledajte orah čvor`
+2. `vanjski linkovi na hromosomu 13 proteini sindrom`
+3. `se u gorskom kotaru velebitu lici i član glavnog odbora stranka je bila 1 nositeljica u 1`
 
 **Context Size 3:**
 
-1. `spiralna galaksija ( sbc ) ngc 672 0 , 99 % bakterija u crijevima buba . oskar je`
-2. `objekti kategorija : iras objekti kategorija : astronomski objekti otkriveni 1885 . ‎`
-3. `reference vanjski linkovi copa américa na zvaničnoj stranici uefa - e european championship 1980 , r...`
+1. `reference vanjski linkovi skakaonica paul ausserleitner izgrađena je u periodu od do godine je tu i ...`
+2. `prečkasta spiralna galaksija koja je udaljena oko 162 miliona sg od zemlje i nalazi se u sazviježđu ...`
+3. `zavod za statistiku republike hrvatske reference vanjski linkovi u sloveniji u primorsko notranjskoj...`
 
 **Context Size 4:**
 
-1. `prečkasta spiralna galaksija ( sbb ) također pogledajte novi opći katalog spisak ic objekata spisak ...`
-2. `kategorija : naselja u gorenjskoj regiji kategorija : naselja u podravskoj regiji kategorija : nasel...`
-3. `vanjski linkovi kategorija : naselja u savinjskoj regiji kategorija : naselja u splitsko - dalmatins...`
+1. `na popisu stanovništva godine črešnjevec je imao 19 stanovnika broj stanovnika po popisima 553 492 5...`
+2. `državni zavod za statistiku naselja i stanovništvo republike hrvatske 118 128 172 210 219 245 266 26...`
+3. `broj stanovnika po popisima 89 120 123 reference vanjski linkovi u sloveniji u posavskoj regiji hist...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_ebiću_hakorona_`
+2. `araćiskano_nit_l`
+3. `i_d_portajemojse`
+
+**Context Size 2:**
+
+1. `a_augućina_bi_med`
+2. `e_vrhipa,_ceaka_s`
+3. `jeglazrobelikimal`
+
+**Context Size 3:**
+
+1. `je_poznac_ženerga_`
+2. `na_galaksija_sa_ce`
+3. `_najblijača_objavl`
+
+**Context Size 4:**
+
+1. `_je_u_složenja_dell`
+2. `ija_roadbez_von_lew`
+3. `_na_pozici_bosnu!_t`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 93.8% predictability
+- **Best Predictability:** Context-4 (word) with 96.2% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (1,544,375 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (1,073,568 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -252,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 558,155 |
-| Total Tokens | 35,712,013 |
-| Mean Frequency | 63.98 |
+| Vocabulary Size | 502,911 |
+| Total Tokens | 32,206,003 |
+| Mean Frequency | 64.04 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 2712.06 |
+| Frequency Std Dev | 2755.29 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | i | 952,586 |
-| 2 | je | 934,949 |
-| 3 | u | 930,575 |
-| 4 | na | 461,115 |
-| 5 | se | 405,379 |
-| 6 | su | 294,193 |
-| 7 | 1 | 286,740 |
-| 8 | kategorija | 277,314 |
-| 9 | od | 273,253 |
-| 10 | za | 268,536 |
+| 1 | i | 934,658 |
+| 2 | je | 922,929 |
+| 3 | u | 915,148 |
+| 4 | na | 453,346 |
+| 5 | se | 397,234 |
+| 6 | su | 288,366 |
+| 7 | od | 268,408 |
+| 8 | za | 263,873 |
+| 9 | 1 | 253,982 |
+| 10 | ngc | 206,398 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | 14678519 | 2 |
-| 2 | esac | 2 |
-| 3 | dkp256 | 2 |
-| 4 | catchshortfilm | 2 |
-| 5 | martirosyan | 2 |
-| 6 | neuzimanje | 2 |
-| 7 | spekarski | 2 |
-| 8 | probabilizamski | 2 |
-| 9 | setap | 2 |
-| 10 | visoravani | 2 |
+| 1 | polikristale | 2 |
+| 2 | ṁ | 2 |
+| 3 | bikristal | 2 |
+| 4 | polikristal | 2 |
+| 5 | misesov | 2 |
+| 6 | abstractmethod | 2 |
+| 7 | ugođen | 2 |
+| 8 | unifilarni | 2 |
+| 9 | neomurani | 2 |
+| 10 | arhebakterije | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9865 |
-| R² (Goodness of Fit) | 0.998665 |
+| Zipf Coefficient | 0.9663 |
+| R² (Goodness of Fit) | 0.999465 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 30.9% |
-| Top 1,000 | 53.0% |
-| Top 5,000 | 68.7% |
-| Top 10,000 | 75.5% |
+| Top 100 | 32.1% |
+| Top 1,000 | 53.2% |
+| Top 5,000 | 68.8% |
+| Top 10,000 | 75.7% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9987 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 30.9% of corpus
-- **Long Tail:** 548,155 words needed for remaining 24.5% coverage
+- **Zipf Compliance:** R²=0.9995 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 32.1% of corpus
+- **Long Tail:** 492,911 words needed for remaining 24.3% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -322,24 +384,128 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 371,696 | 32 | 4.146 | 1.829 | 0.6746 🏆 |
-| **mono_64d** | 371,696 | 64 | 4.579 | 1.739 | 0.6582 |
-| **mono_128d** | 371,696 | 128 | 5.121 | 1.629 | 0.6288 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.6837 🏆 | 0.3607 | N/A | N/A |
+| **mono_64d** | 64 | 0.6836 | 0.2874 | N/A | N/A |
+| **mono_128d** | 128 | 0.6518 | 0.2275 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.6746 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 371,696 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.6837 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2919. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-pr` | pričvršćenom, prihova, prethodnica |
+| `-po` | pokusu, povljana, polimerskim |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-a` | kolomina, puferska, prihova |
+| `-e` | ostatke, akademske, dominantnotype |
+| `-i` | ristovski, ukrajinski, dovođeni |
+| `-om` | pričvršćenom, nekontrolisanom, polupustinjskom |
+| `-na` | kolomina, povljana, financirana |
+| `-ja` | ašiklija, preimenovanja, grobalja |
+| `-ma` | polusestrama, metaloenzima, falconsima |
+| `-im` | tvrtkovim, polimerskim, briljantnim |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `kovi` | 1.59x | 619 contexts | kovin, kovič, ković |
+| `anov` | 1.57x | 625 contexts | hanov, kanov, banov |
+| `selj` | 2.07x | 82 contexts | seljo, selja, seljak |
+| `alak` | 2.52x | 33 contexts | malak, talak, stalak |
+| `vanj` | 1.71x | 170 contexts | vanje, kvanj, vanja |
+| `renc` | 1.98x | 75 contexts | renci, renco, renca |
+| `acij` | 1.55x | 220 contexts | lacij, acije, acija |
+| `alna` | 1.97x | 58 contexts | šalna, malna, valna |
+| `jekt` | 1.82x | 78 contexts | objekt, subjekt, trajekt |
+| `iral` | 1.50x | 164 contexts | miral, ziral, viral |
+| `njsk` | 1.56x | 134 contexts | vnjski, anjski, vanjsk |
+| `egov` | 1.55x | 114 contexts | negov, begov, begovo |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-pr` | `-a` | 63 words | prementuma, protivpožarna |
+| `-po` | `-a` | 56 words | potomcima, pobunila |
+| `-pr` | `-i` | 49 words | pravokutni, propuštajući |
+| `-pr` | `-e` | 49 words | primijećuje, prirasle |
+| `-po` | `-i` | 48 words | poručivši, poliribosomi |
+| `-po` | `-e` | 32 words | poupée, popularizacije |
+| `-po` | `-ma` | 12 words | potomcima, poslodavcima |
+| `-pr` | `-om` | 12 words | preporodnom, preuranjenom |
+| `-pr` | `-ja` | 12 words | protozoologija, pribavlja |
+| `-pr` | `-na` | 11 words | protivpožarna, protimozina |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| umoljanima | **`umol-ja-ni-ma`** | 7.5 | `umol` |
+| melkumani | **`melku-ma-ni`** | 6.0 | `melku` |
+| postepenim | **`po-stepen-im`** | 6.0 | `stepen` |
+| skečevima | **`skečevi-ma`** | 4.5 | `skečevi` |
+| nesigurnostima | **`nesigurnosti-ma`** | 4.5 | `nesigurnosti` |
+| balansiranje | **`balansiran-je`** | 4.5 | `balansiran` |
+| ignoriranje | **`ignoriran-je`** | 4.5 | `ignoriran` |
+| pobjesnio | **`po-bjesnio`** | 4.5 | `bjesnio` |
+| integrirani | **`integrira-ni`** | 4.5 | `integrira` |
+| rutherfordovom | **`rutherfordov-om`** | 4.5 | `rutherfordov` |
+| karlingom | **`karling-om`** | 4.5 | `karling` |
+| kriopirinom | **`kriopirin-om`** | 4.5 | `kriopirin` |
+| šezdesetim | **`šezdeset-im`** | 4.5 | `šezdeset` |
+| pojašnjena | **`po-jašn-je-na`** | 4.5 | `jašn` |
+| akreditiranje | **`akreditiran-je`** | 4.5 | `akreditiran` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BS appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -347,11 +513,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.39x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (391) |
-| Markov | **Context-4** | Highest predictability (93.8%) |
+| Tokenizer | **64k BPE** | Best compression (4.71x) |
+| N-gram | **2-gram** | Lowest perplexity (329) |
+| Markov | **Context-4** | Highest predictability (96.2%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -541,7 +708,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -557,7 +725,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 09:16:14*
+*Report Date: 2026-01-03 10:03:26*

models/embeddings/monolingual/bs_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f6c7842094c5d54ec997bd4f8206b5f27aca331f5a1c1dff76da9e6cefa40e21
-size 1411374464
+oid sha256:150a739475c8b2802aaaa35679a6ec5946f23a4489669763729d419b2cf329bf
+size 1381600692

models/embeddings/monolingual/bs_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 371696
+  "vocab_size": 343142
 }

models/embeddings/monolingual/bs_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6fe01a31b5c93cf415cace6e7b02aa8239e718e60b37cd4c2b060a8eb251a318
-size 357911936
+oid sha256:5db060f6dd5915bae4076dfd0dea4755e1f4664360e6713434947ebf5dd49b94
+size 350067636

models/embeddings/monolingual/bs_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 371696
+  "vocab_size": 343142
 }

models/embeddings/monolingual/bs_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ff1c0f9d89bb40f97868ceeb132ac53c6484a2f8fdcc5c6cbaf6f2868d696c7f
-size 709066112
+oid sha256:09f0ccc9cb7ef218444a015a72c973a1aa64b7424c24eac2b2b5f999e7bae986
+size 693911988

models/embeddings/monolingual/bs_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 371696
+  "vocab_size": 343142
 }

models/subword_markov/bs_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7b5f27f2e1460c1e87d820c8c92dd81b1f82c6d52d39730f07ad0343240f4c4b
-size 258521
+oid sha256:c413639e86aff70a1e366f239702d90f57805452729b30f47b449e1b50e9d0d1
+size 236628

models/subword_markov/bs_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "bs",
-  "unique_contexts": 3680,
-  "total_transitions": 247647212
+  "unique_contexts": 3822,
+  "total_transitions": 217153057
 }

models/subword_markov/bs_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:691f6350f2c87d8ba672a74a63f3856baba3e586a695e4f8cc1e00bb093febdb
-size 2030096
+oid sha256:07af575a720ce0333065d1365fd6f0dc1e9c196dbbdf47dae5915e384670002e
+size 1556026

models/subword_markov/bs_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "bs",
-  "unique_contexts": 35380,
-  "total_transitions": 247550979
+  "unique_contexts": 29614,
+  "total_transitions": 217057401
 }

models/subword_markov/bs_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f7bb33fe8a1b2b35089fb879359b5ccdee6a2dc4a6fa92764f5318e4c2c051b9
-size 11370107
+oid sha256:8734e9ff242be1673110fbadecec4921d18b749fab506afd58084c6454f10179
+size 8759226

models/subword_markov/bs_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "bs",
-  "unique_contexts": 260071,
-  "total_transitions": 247454746
+  "unique_contexts": 195969,
+  "total_transitions": 216961745
 }

models/subword_markov/bs_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:282f70dd99bb2b2125b622ee85b9719605135a2ae0a545ff1ec079fde4a31cd0
-size 44185313
+oid sha256:fcd4e07fd4b57bbf32facc8ddf98017e371bfdad9f2a2932def87de4cad2a336
+size 32234978

models/subword_markov/bs_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "bs",
-  "unique_contexts": 1544375,
-  "total_transitions": 247358513
+  "unique_contexts": 1073568,
+  "total_transitions": 216866089
 }

models/subword_ngram/bs_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b3b171086fda38b2cacca4736ca9721e98ec82df7ea537a0bf4450e1fa0f4270
-size 177940
+oid sha256:5088ff625c4275eba000b58efc27a957d0b2fe00df41f79272702d5740b6aa5a
+size 149061

models/subword_ngram/bs_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "bs",
-  "unique_ngrams": 13004,
-  "total_ngrams": 247647212
+  "unique_ngrams": 10932,
+  "total_ngrams": 217153057
 }

models/subword_ngram/bs_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f68e0d14a83843d048ae0bab5f16f3972dd2f5f10a586eef6dfeda247caf1686
-size 1633760
+oid sha256:994dcf177eaabe0675b5f4a1b329598035e90a3b5e97cc7f73d373979f3cdd23
+size 1254949

models/subword_ngram/bs_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "bs",
-  "unique_ngrams": 136436,
-  "total_ngrams": 247550979
+  "unique_ngrams": 100952,
+  "total_ngrams": 217057401
 }

models/subword_ngram/bs_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d2a25a4668d29995039cf80d734d71341a21ef16c346a7e489b00f61ef609606
-size 10910928
+oid sha256:f9e9075117ccaa25ec707915fefe1acd5957f11ad9c94308474e7238513466d0
+size 7917058

models/subword_ngram/bs_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "bs",
-  "unique_ngrams": 960072,
-  "total_ngrams": 247454746
+  "unique_ngrams": 689562,
+  "total_ngrams": 216961745
 }

models/tokenizer/bs_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ab84f7758210739a88220d86d23d3f9a99422e8156920a2449641ddb4a839cbf
-size 510609
+oid sha256:6009fbe251dc69f2b102003863ae8d942c6cdb255917c51ca2f5748f2025e0b0
+size 511740

models/tokenizer/bs_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a03a861c43e4b168d6f569981254f69bb1e97114610f73801f36979f16349797
-size 795530
+oid sha256:f77069e783a7216648e25a6ebc87d0c8bc974ab6c4df1509692cd565a485b703
+size 795090

models/tokenizer/bs_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4c47d94f1d0144c562baecc82da4a80a381b89d1140ae6cb88626bfec834c555
-size 1379541
+oid sha256:6b340f78c1c205596e8e78b4b37abd3a0af06a9dcb987769156d87e8ae5bc64f
+size 1373883

models/tokenizer/bs_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:705f090f1b6380f4c29e1b3dca853a74ffde76d3d0261edd9635bc0110f452fc
-size 373194
+oid sha256:a827c60064fd24d771a22fb0ed577a444662b37250d01ef94952af5e5f0ea59f
+size 374247

models/vocabulary/bs_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fe14a8d1b945f3bd03d3b1b788afc4218e96b19f5b80a557ba00106fefbc47b5
-size 8317107
+oid sha256:49eafd3b4bbbba80874eda2ec39ac7b2e78de842040a668fddae30d8d872a494
+size 7547245

models/vocabulary/bs_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "bs",
-  "vocabulary_size": 558155,
+  "vocabulary_size": 502911,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.036391908235982366,
+    "type_token_ratio": 0.033338017781857625,
     "coverage": {
-      "top_100": 0.30248203410776353,
-      "top_1000": 0.5185736251061464,
-      "top_5000": 0.6729546318682332,
-      "top_10000": 0.7393625298572655
+      "top_100": 0.3148584842736538,
+      "top_1000": 0.5222226568897175,
+      "top_5000": 0.6752045594187739,
+      "top_10000": 0.7431974443827949
     },
-    "hapax_count": 769476,
-    "hapax_ratio": 0.5795857433277771,
-    "total_documents": 96233
+    "hapax_count": 590458,
+    "hapax_ratio": 0.5400354317709758,
+    "total_documents": 95656
   }
 }

models/word_markov/bs_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bc9e7078d4df81ec6a101ebf8a0197bb7a7092abee7f59acc2d370c1f5159e06
-size 98493993
+oid sha256:ec8d96e1bb6fe5416cce99b5c6d8559355a902c39366ee4bac65c211144112c5
+size 99684168

models/word_markov/bs_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "bs",
-  "unique_contexts": 1327715,
-  "total_transitions": 46611373
+  "unique_contexts": 1092943,
+  "total_transitions": 32700805
 }

models/word_markov/bs_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3f63562dc7c7ccdc0d7d8e972c6f458762ec1a33d7517c853122ce5c8ddc993e
-size 286466586
+oid sha256:2b7fcee8bc4b37f470ffa1f616fa833e1a90112ecac1f412ad0a81c5ac29b58b
+size 277373494

models/word_markov/bs_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "bs",
-  "unique_contexts": 10525382,
-  "total_transitions": 46515140
+  "unique_contexts": 10856748,
+  "total_transitions": 32605149
 }

models/word_markov/bs_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9582dc3de637812d3da30d19b03a1c884f4ce0da175db42ba6d51639dd88fedf
-size 463368487
+oid sha256:7ebc2e93fc71d313258da50f1964cefec942b76af63de84a7b0bb64c24a43d04
+size 410403619

models/word_markov/bs_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "bs",
-  "unique_contexts": 23101514,
-  "total_transitions": 46418911
+  "unique_contexts": 20575067,
+  "total_transitions": 32509493
 }

models/word_markov/bs_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fbe7ccc8981a9260457038e49973e858baf5ee815f9daa5f221d9d3bb26ec2f0
-size 573670650
+oid sha256:2f2ff5216e6ef640bafa4dd312ef8dad995a25afe35a2f47ff855d105c151152
+size 484392746

models/word_markov/bs_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "bs",
-  "unique_contexts": 30570937,
-  "total_transitions": 46322686
+  "unique_contexts": 24704289,
+  "total_transitions": 32413837
 }

models/word_ngram/bs_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:99c97b22a0c0a5c5edfa9d625815ac911c555be1f1a7e0be67f52902dc66cd97
-size 11862865
+oid sha256:f83fc5f11c0f8326e29ac35309cbdfb73699a8a949c72b421e41c46db43f6106
+size 9448613

models/word_ngram/bs_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "bs",
-  "unique_ngrams": 866401,
-  "total_ngrams": 46611373
+  "unique_ngrams": 657818,
+  "total_ngrams": 32700805
 }

models/word_ngram/bs_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e1b8015ac65d6f173d3d93954bcf4ad6d729f5339475ea9f46ab0eade57ec72
-size 22954892
+oid sha256:3babe8520d1c50503b96745109a27db90a3f3ca40452f51c64ae4e06fafd37c8
+size 14627614

models/word_ngram/bs_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "bs",
-  "unique_ngrams": 1540282,
-  "total_ngrams": 46515140
+  "unique_ngrams": 914717,
+  "total_ngrams": 32605149
 }

models/word_ngram/bs_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:94d9815bd2fecb0a318ed09618e190927c12a0ac232ecc3854041c6e954684c8
-size 41299300
+oid sha256:762b45944f4641b6630a17f47d8317ebdd6446f7959328117df5bc157a98c126
+size 24754896

models/word_ngram/bs_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "bs",
-  "unique_ngrams": 2663711,
-  "total_ngrams": 46418911
+  "unique_ngrams": 1461546,
+  "total_ngrams": 32509493
 }