Upload all models and assets for am (latest)

README.md

@@ -99,32 +99,32 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `እኔ እውነት እናገራለሁ ሌላውን አስኮንናለሁ የአማርኛ ምሳሌ ነው። ትርጉሙ መደብ: ያልተተረጎመ ምሳሌ`
+**Sample 1:** `ናውሩ በሰላማዊ ውቅያኖስ የሚገኝ ደሴት አገር ነው። ዋና ከተማ የለውም፣ ትልቁ ከተማ ግን ያሬን ነው።`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁እኔ ▁እውነት ▁እና ገራ ለሁ ▁ሌላ ውን ▁አስ ኮ ንና ... (+9 more)` | 19 |
-| 16k | `▁እኔ ▁እውነት ▁እና ገራ ለሁ ▁ሌላውን ▁አስ ኮ ንና ለሁ ... (+8 more)` | 18 |
-| 32k | `▁እኔ ▁እውነት ▁እናገራ ለሁ ▁ሌላውን ▁አስ ኮ ንና ለሁ ▁የአማርኛ ... (+7 more)` | 17 |
-| 64k | `▁እኔ ▁እውነት ▁እናገራለሁ ▁ሌላውን ▁አስ ኮንና ለሁ ▁የአማርኛ ▁ምሳሌ ▁ነው። ... (+5 more)` | 15 |
+| 8k | `▁ና ው ሩ ▁በሰ ላማዊ ▁ውቅያኖስ ▁የሚገኝ ▁ደሴት ▁አገር ▁ነው። ... (+10 more)` | 20 |
+| 16k | `▁ና ውሩ ▁በሰላማዊ ▁ውቅያኖስ ▁የሚገኝ ▁ደሴት ▁አገር ▁ነው። ▁ዋና ▁ከተማ ... (+8 more)` | 18 |
+| 32k | `▁ናውሩ ▁በሰላማዊ ▁ውቅያኖስ ▁የሚገኝ ▁ደሴት ▁አገር ▁ነው። ▁ዋና ▁ከተማ ▁የለውም፣ ... (+6 more)` | 16 |
+| 64k | `▁ናውሩ ▁በሰላማዊ ▁ውቅያኖስ ▁የሚገኝ ▁ደሴት ▁አገር ▁ነው። ▁ዋና ▁ከተማ ▁የለውም፣ ... (+5 more)` | 15 |
 
-**Sample 2:** `እንኳን ለገንፎ ለሙቅም አልደነግጥ የአማርኛ ምሳሌ ነው። ትርጉሙ መደብ: ያልተተረጎመ ምሳሌ`
+**Sample 2:** `አሾካ ከ277 ስከ 240 ዓክልበ. ድረስ የሕንድ አገር ማውርያ መንግሥት ንጉሥ ነበር። በ271 ዓክልበ. ግድም የቡዲስም ተከታይ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁እንኳን ▁ለ ገን ፎ ▁ለ ሙ ቅም ▁አል ደ ነግ ... (+9 more)` | 19 |
-| 16k | `▁እንኳን ▁ለ ገን ፎ ▁ለሙ ቅም ▁አል ደነግ ጥ ▁የአማርኛ ... (+7 more)` | 17 |
-| 32k | `▁እንኳን ▁ለ ገንፎ ▁ለሙ ቅም ▁አል ደነግጥ ▁የአማርኛ ▁ምሳሌ ▁ነው። ... (+5 more)` | 15 |
-| 64k | `▁እንኳን ▁ለገንፎ ▁ለሙ ቅም ▁አል ደነግጥ ▁የአማርኛ ▁ምሳሌ ▁ነው። ▁ትርጉሙ ... (+4 more)` | 14 |
+| 8k | `▁አ ሾ ካ ▁ከ 2 7 7 ▁ስ ከ ▁ ... (+42 more)` | 52 |
+| 16k | `▁አ ሾ ካ ▁ከ 2 7 7 ▁ስ ከ ▁ ... (+39 more)` | 49 |
+| 32k | `▁አሾ ካ ▁ከ 2 7 7 ▁ስ ከ ▁ 2 ... (+38 more)` | 48 |
+| 64k | `▁አሾካ ▁ከ 2 7 7 ▁ስከ ▁ 2 4 0 ... (+34 more)` | 44 |
 
-**Sample 3:** `ሞፈር ረገጠ በአማርኛ ፈሊጣዊ አነጋገር የሆነ ዘይቤ ነው። ትርጉም እራሱን ቻለ። ከቤተሰብ ቁጥጥር ውጭ ሆነ። ምሳሌ ደበበ ዕድሜ...`
+**Sample 3:** `ኔትፍሊክስ (እንግሊዝኛ: Netflix) በመስመር ላይ ፊልሞችን እና የቴሌቪዥን ፕሮግራሞችን ለመመልከት የሚያስችል የዥረት አገል...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ሞ ፈር ▁ረ ገ ጠ ▁በአማርኛ ▁ፈሊጣዊ ▁አነጋገር ▁የሆነ ▁ዘይቤ ... (+29 more)` | 39 |
-| 16k | `▁ሞ ፈር ▁ረገ ጠ ▁በአማርኛ ▁ፈሊጣዊ ▁አነጋገር ▁የሆነ ▁ዘይቤ ▁ነው። ... (+24 more)` | 34 |
-| 32k | `▁ሞፈር ▁ረገ ጠ ▁በአማርኛ ▁ፈሊጣዊ ▁አነጋገር ▁የሆነ ▁ዘይቤ ▁ነው። ▁ትርጉም ... (+21 more)` | 31 |
-| 64k | `▁ሞፈር ▁ረገ ጠ ▁በአማርኛ ▁ፈሊጣዊ ▁አነጋገር ▁የሆነ ▁ዘይቤ ▁ነው። ▁ትርጉም ... (+21 more)` | 31 |
+| 8k | `▁ኔ ት ፍ ሊ ክስ ▁( እንግሊዝኛ : ▁n et ... (+36 more)` | 46 |
+| 16k | `▁ኔ ትፍ ሊ ክስ ▁( እንግሊዝኛ : ▁n et fl ... (+29 more)` | 39 |
+| 32k | `▁ኔ ትፍ ሊክስ ▁( እንግሊዝኛ : ▁net fl ix ) ... (+23 more)` | 33 |
+| 64k | `▁ኔ ትፍ ሊክስ ▁( እንግሊዝኛ : ▁net flix ) ▁በመስመር ... (+16 more)` | 26 |
 
 
 ### Key Findings
@@ -274,27 +274,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ነው ወደዚህም የተሳበው ተፈጥሮን ፀባዮች የሚመረኮዝ ፆታዊ ውሳኔ ተላልፏል ውድድሩን በ ኤርትራውያን የኩራት ምንጭ የበለጠ የተገደበ`
-2. `እና ሳይንሶችን እንዲሁም ከላስታና ከላሊበላ ከፍተኛ የመብት ጥሰቶች እ ኤ አ የተካሄደውን መፈንቅለ መንግሥት በጌሤም የአሦርም`
-3. `ላይ እንዲገኝ ስለሚያስገድድ ነው ኬንያ ወደሚገኘው ማይ ጎጋ የተባለ የህንድ ጥቃቶች የተጠበቀ እና በችግር ጊዜ የተረጋገጠ`
+1. `ነው ያኽዱን ሊም ዓክልበ የነገሠ የሊፒት እሽታርን እርዳታ የማግኘት መብቱ የተጠበቀ ስለሆነ ፈጽሞ ይበላል ፍሬው ሳይበስል`
+2. `እና ኢኮኖሚያዊ እና አመለካከቶችን ለመግለጽ ይወዳል የወዳጅሽ የመሠወሪያው ማዕበልም ያማታዋል ዳግመኛም የከበረውን የመልክተኛዎን የቃል ትርጉም ሊያዳብር`
+3. `ላይ አፈፃፀምን በራስ መተማመን አይችሉም ከሚለው ቃል በሲቪል ደግሞ ለየተለያዩ በአፍሪካ ውስጥ የተረጋገጠ ይመስላል ከዚያም የሶቪየት`
 
 **Context Size 2:**
 
-1. `ዓ ም የዊስቡር ልጅ ዶማር 300 307 ዓ ም ተከለከለ ታጂኪስታን ዓ ም በነሐሴ ወር 450 ዓ`
-2. `ምሳሌ ነው ጦጣ መጀመሪያ የመቀመጫዬን አለች አሉ ጦጣ ባለቤቱን ታስወጣ የአማርኛ ምሳሌ ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ`
-3. `የአማርኛ ምሳሌ ነው ለላሙ መንጃ ለሸማው መቅደጃ የአማርኛ ምሳሌ ነው ትርጉሙ መደብ ተረትና ምሳሌ መደብ ተረትና ምሳሌ`
+1. `ዓ ም በኋላ ለሆኑት ዓመታት ግን በሌላ ቀን ላይ መሆኑን ይገንዘቡ ለእነዚያ ዓመቶች ይህ የቀን መለወጫ መሣርያ`
+2. `ምሳሌ ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ መደብ ተረትና ምሳሌ መደብ ተረትና ምሳሌ መደብ ተረትና ምሳሌ ምናልባትም ከቤ`
+3. `የአማርኛ ምሳሌ ነው ትርጉሙ ሚስጥር አይደበቅ ይመስላል ትርጉሙ መደብ ተረትና ምሳሌ መደብ ተረትና ምሳሌ መደብ ተረትና ምሳሌ`
 
 **Context Size 3:**
 
-1. `የአማርኛ ምሳሌ ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ መደብ ተረትና ምሳሌ wiz`
-2. `እ ኤ አ ቦራስ ስዊድን የግሪክ ዘፋኝ ነች አልበሞች protereotita my number one iparhi logos the game of`
-3. `ምሳሌ ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ መደብ ተረትና ምሳሌ መደብ ተረትና ምሳሌ መደብ ተረትና ምሳሌ ምሳሌ`
+1. `የአማርኛ ምሳሌ ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ መደብ ተረትና ምሳሌ ምግባር ሳይኖር ስም እንደማለት ነዉ`
+2. `እ ኤ አ የእንግሊዝ ካላንደር ማሻሻያ ተከትሎ የንግሥቲቱን ሞት መመዝገብ የተለመደ ቢሆንም እንግሊዝ መጋቢት 25 ቀን ማለት ነው`
+3. `ምሳሌ ነው ትርጉሙ የተያያዙ ነገሮችን ለመለየት የሚያገለግል ፈሊጥ መደብ ተረትና ምሳሌ ምሳሌ`
 
 **Context Size 4:**
 
-1. `የአማርኛ ምሳሌ ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ መደብ ተረትና ምሳሌ wiz`
-2. `ምሳሌ ነው ትርጉሙ መደብ ተረትና ምሳሌ በሬ ካራጁ ይዉላል`
-3. `ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ መደብ ተረትና ምሳሌ መደብ ያልተተረጎመ ምሳሌ`
+1. `የአማርኛ ምሳሌ ነው ትርጉሙ መደብ ተረትና ምሳሌ በሬ ካራጁ ይዉላል`
+2. `ምሳሌ ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ መደብ ተረትና ምሳሌ መደብ ያልተተረጎመ ምሳሌ`
+3. `ነው ትርጉሙ መደብ ያልተተረጎመ ምሳሌ መደብ ተረትና ምሳሌ ሴት ሁሉን ቻይ ናት`
 
 
 ### Generated Text Samples (Subword-based)
@@ -303,27 +303,27 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_493_የለቀድ_አት_በተገ`
-2. `ንዳዎች_20_የሳት_ወቀን_`
-3. `ት_ገኙ_ነበትላን_ጆች_በዚ`
+1. `_በይምነዉ፡ቢቢትር_የተፅሀ`
+2. `ን_እንዋጮችት_crcue_አ`
+3. `ት_ው_ፈርዕስክሎ_አስ_po`
 
 **Context Size 2:**
 
-1. `_የጠፈ_እጅጉ_ሙከራ_ተፈጥሮ`
-2. `ት_ከተማ_እቃ_ለማብራዶሮ_ሶ`
-3. `_በሁለተቸት_ስለ_ተመሳር_ከ`
+1. `_የኢትዮጵያ_ዘን_ሳይንስ_ተ`
+2. `ት_ነው።_እንግሥታት_ናይትድ`
+3. `_በወራ_ህብረ_የሳምን_ወይ_`
 
 **Context Size 3:**
 
-1. `_እንዲሁም_ዘር።_ከነዚህ_ጊዜ`
-2. `_ነው_፡፡_አየሩ_በኋላም_ብዙ`
-3. `_እና_ጁላይ_ጥይቶቹ_ላይ_(2`
+1. `_እንዲህ፡መልክ_ፈላሁ_ዐንሁ_`
+2. `_ነው።_እንዲህም፡ዅሉ፡_ደግሞ`
+3. `_እና_ከተያዙ_እንዲሸከሙአቸው`
 
 **Context Size 4:**
 
-1. `_��ና_የከተማ፡-_ጎንደርና_አገ`
-2. `_ነው።_ባብዛኛው_ህይወት_ውስጥ`
-3. `ነው።_ዓ.ም_ኪዮሺ_ሱጊዩራ_(1`
+1. `_እና_ቁሳዊ_ነገሥታት_መሽኛ_ት`
+2. `_ነው።_ከግብጽ_ዘውድ_ጭነው_ነ`
+3. `ነው።_ሁሉም_የተነሳ_በኋላም_ያ`
 
 
 ### Key Findings
@@ -428,18 +428,18 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.9080 | 0.3255 | N/A | N/A |
-| **mono_64d** | 64 | 0.9137 | 0.2344 | N/A | N/A |
-| **mono_128d** | 128 | 0.8453 | 0.1726 | N/A | N/A |
-| **aligned_32d** | 32 | 0.9080 | 0.3232 | 0.0220 | 0.1700 |
-| **aligned_64d** | 64 | 0.9137 🏆 | 0.2323 | 0.0420 | 0.1840 |
-| **aligned_128d** | 128 | 0.8453 | 0.1725 | 0.0680 | 0.2480 |
+| **mono_32d** | 32 | 0.9098 | 0.3240 | N/A | N/A |
+| **mono_64d** | 64 | 0.9137 🏆 | 0.2319 | N/A | N/A |
+| **mono_128d** | 128 | 0.8452 | 0.1755 | N/A | N/A |
+| **aligned_32d** | 32 | 0.9098 | 0.3259 | 0.0200 | 0.1420 |
+| **aligned_64d** | 64 | 0.9137 | 0.2299 | 0.0480 | 0.1860 |
+| **aligned_128d** | 128 | 0.8452 | 0.1764 | 0.0840 | 0.2800 |
 
 ### Key Findings
 
-- **Best Isotropy:** aligned_64d with 0.9137 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.2434. Lower values indicate better semantic separation.
-- **Alignment Quality:** Aligned models achieve up to 6.8% R@1 in cross-lingual retrieval.
+- **Best Isotropy:** mono_64d with 0.9137 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2439. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 8.4% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
@@ -467,18 +467,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `እንደሚ` | 2.39x | 158 contexts | እንደሚሉ, እንደሚሻ, እንደሚል |
-| `ርስቲያ` | 2.46x | 61 contexts | ክርስቲያ, ክርስቲያኗ, ክርስቲያኑ |
-| `ትዮጵያ` | 2.23x | 57 contexts | ኢትዮጵያ, እትዮጵያ, ኢትዮጵያና |
-| `መንግስ` | 2.21x | 49 contexts | መንግስቱ, መንግስት, መንግስተ |
-| `ግዚአብ` | 2.66x | 23 contexts | እግዚአብሔር, እግዚአብሐር, እግዚአብሄር |
-| `ኢትዮጵ` | 2.18x | 46 contexts | ኢትዮጵያ, ኢትዮጵያና, የኢትዮጵያ |
-| `እንግሊ` | 2.08x | 52 contexts | እንግሊኛ, እንግሊዙ, እንግሊዝ |
-| `መንግሥ` | 2.12x | 46 contexts | መንግሥት, መንግሥተ, መንግሥቱ |
-| `ጀመሪያ` | 2.29x | 33 contexts | መጀመሪያ, በመጀመሪያ, ለመጀመሪያ |
-| `ፈረንሳ` | 2.27x | 34 contexts | ፈረንሳይ, ፈረንሳዊ, የፈረንሳዩ |
-| `tion` | 2.77x | 17 contexts | nation, action, section |
-| `መጀመሪ` | 2.29x | 31 contexts | መጀመሪአ, መጀመሪያ, የመጀመሪ |
+| `እንደሚ` | 2.30x | 158 contexts | እንደሚሹ, እንደሚሻ, እንደሚል |
+| `ርስቲያ` | 2.39x | 61 contexts | ክርስቲያ, ከርስቲያን, ክርስቲያን |
+| `ትዮጵያ` | 2.17x | 57 contexts | ኢትዮጵያ, እትዮጵያ, ኢትዮጵያን |
+| `መንግስ` | 2.10x | 49 contexts | መንግስቱ, መንግስተ, መንግስት |
+| `ግዚአብ` | 2.58x | 23 contexts | እግዚአብሐር, እግዚአብሔር, እግዚአብሄር |
+| `ኢትዮጵ` | 2.08x | 46 contexts | ኢትዮጵያ, ኢትዮጵያን, ኢትዮጵያና |
+| `እንግሊ` | 2.00x | 52 contexts | እንግሊዝ, እንግሊዙ, እንግሊኛ |
+| `ፈረንሳ` | 2.23x | 34 contexts | ፈረንሳዊ, ፈረንሳይ, ከፈረንሳዩ |
+| `መንግሥ` | 2.04x | 46 contexts | መንግሥቱ, መንግሥት, መንግሥተ |
+| `tion` | 2.71x | 17 contexts | action, nation, section |
+| `አስተዳ` | 2.21x | 33 contexts | አስተዳደጉ, አስተዳደሪ, አስተዳደጓ |
+| `ግሊዝኛ` | 2.54x | 19 contexts | እንግሊዝኛ, በእንግሊዝኛ, ኢንግሊዝኛው |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -726,4 +726,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 14:11:24*
+*Report Date: 2026-01-03 16:28:42*

models/embeddings/aligned/am_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3964ee0c4f9ca092d9f74907f56f9a3d93b19752347882a64e552d576a095e2b
+oid sha256:05365ec12b2c2fb2d8df995aee24119fee6ddb57a678bd862fd40de10e193cb1
 size 1064306440

models/embeddings/aligned/am_128d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cfbe39d8562cf38339d4ba377db708ff89f8f76337965c05da5c47a5511cd90d
+oid sha256:deb24c9e18725a3ee6f696f5f490d96e499e66d1aef469fc3b67ca71da6c2d47
 size 65664

models/embeddings/aligned/am_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:43acd2454c73ec6a90e9b75922f4a4ecc9db024320951b2c88c136aaaf3e57dc
+oid sha256:8345b199adcc25abea3de9a656a6b112dd0b6a39d6361db11f5e425bf8e004bd
 size 266727688

models/embeddings/aligned/am_32d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b3d0d5ccf51dddf65743a5123bfc0ecd1944f4cfce415736d0e147acf3d55f4b
+oid sha256:634cf159edda5ce6a8152867dd7521b71f1c434d47e85d17f55d1e0d6feb3316
 size 4224

models/embeddings/aligned/am_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ad86c60bc61db296fa76aedb6ab90d476fc19f98d61421e743d16270d0805cf5
+oid sha256:93e5df8ef91db2272bca59309b0c79fd6feaa7e932515206d8059b8330fa5e72
 size 532587272

models/embeddings/aligned/am_64d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:56b5628675cc1634a4f9405dd4c0ad1d8ef1da74827604d4d4f4a7c37742850a
+oid sha256:627287628b6e540e4db14e30e43e8a0a62ea37c30be3ad5ce1ab5d656c55c446
 size 16512

models/embeddings/monolingual/am_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3964ee0c4f9ca092d9f74907f56f9a3d93b19752347882a64e552d576a095e2b
+oid sha256:05365ec12b2c2fb2d8df995aee24119fee6ddb57a678bd862fd40de10e193cb1
 size 1064306440

models/embeddings/monolingual/am_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:43acd2454c73ec6a90e9b75922f4a4ecc9db024320951b2c88c136aaaf3e57dc
+oid sha256:8345b199adcc25abea3de9a656a6b112dd0b6a39d6361db11f5e425bf8e004bd
 size 266727688

models/embeddings/monolingual/am_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ad86c60bc61db296fa76aedb6ab90d476fc19f98d61421e743d16270d0805cf5
+oid sha256:93e5df8ef91db2272bca59309b0c79fd6feaa7e932515206d8059b8330fa5e72
 size 532587272

models/subword_markov/am_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8839cf6220dd43e6dea9746686fa8a81a9a14dd02c6132ff81405220cf661652
-size 350051
+oid sha256:a0819e23adea11e4b2d437cb14f2eb3a8095f9305eca7295c22f45869cc2398c
+size 354543

models/subword_markov/am_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:27974fda5f8693d4915e9b657526e84bd0fb36fde9d966f19e785d000aedbe5c
-size 2143373
+oid sha256:b6243c7c58a847e7d8a7e59e0d6cf893519ad61e931d6182b4fd19948111d9cf
+size 2169861

models/subword_markov/am_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:82c019b23a13a08dd018f89e36328fb55a539c91738572761bc8164c5041b3f4
-size 8416653
+oid sha256:ab2bfec0c73c9d92c58f9ffed32659a2ea045ce31cbf4ae474be8c7609c2d247
+size 8421663

models/subword_markov/am_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f07c64385b4c55e182b6ffee304e1810b2d71ef22b3c5a0b251758b633fca59c
-size 23309934
+oid sha256:f389ec100097a598046200d415e719f5c015e20313c572c31733c198bd66825b
+size 23300653

models/subword_ngram/am_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5eabd2483a9b43b8f8fd572f71957cc349ed926f2f419138f2f3a188ed78c42f
-size 300472
+oid sha256:ad5c361634d826f85a00a861a3aac431b960daae8e30d389a26b10f223079031
+size 300438

models/subword_ngram/am_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3c9b592f96c76cc3101fee1a715cf6c4149dfe51039e391d6098096f9714abe5
-size 1885405
+oid sha256:bb1a1174c9a4c80aefbd37753d4553c8a756c0598012bc0df0044d2a23423c4d
+size 1911161

models/subword_ngram/am_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:899527071ee5237fc41c2fdb6de104208b69f563db321792d2b61794349f7f99
-size 7120084
+oid sha256:a82c126bddc5c2665815030f550a0890e69f0a9cfbd4eaac08c79d884183a4d1
+size 7098220

models/subword_ngram/am_5gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d6256142f289450d718eafc3693efd234fd5e20489751f38071b275a053df337
-size 12113547
+oid sha256:c35770939585fe557d5ccf7666948844ecbbbeab5b5bd7ec5cde2dc5905be254
+size 12087680

models/tokenizer/am_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:802c7e23f92ccb7959b0feb6dc8f82635d55d846dbd9f4570913915ce17d5785
+oid sha256:a111ad7a5e4c6cde164649095c769c75312c7ad72aae53c5ee2f8ef7838721c4
 size 559625

models/tokenizer/am_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:15d15bc01b2e176dbce09f1705536a89afa2737570c8c47d3353634f9f68a94a
+oid sha256:ab016e2ea38f8d8c7b267cf37f85d2bb7c17bda4ed8c6c8b690fb0324f3335dd
 size 902568

models/tokenizer/am_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5022a070bcee7d18664184b36241ade2425fae873f2d51e1b98af97932ce4f68
+oid sha256:961377d69eb3cec24c9c718382b0b467b9a1afa24c2dc238af1e0fa8ea2122c9
 size 1589838

models/tokenizer/am_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5a8188f6c50ce22b57642fe6d5b7e098ba217e95117378d4c365656422f23b18
+oid sha256:dc01c040ea8751cdd811ff0c122044d022da068ea1f778e5d6105fed66d9c6e2
 size 394741

models/word_markov/am_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95ccb7912954a141c9b8b7ca95e221a9fccec6dd51f1c444544019f2ac02b83d
-size 13813306
+oid sha256:b31d703fb4cebc4b7a0c50356f366a54d6244d97c871ead6e6e93a4fb68fbb1f
+size 13829693

models/word_markov/am_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d1434179ed6804e7277902166d67afa9d340c443633dfac8c4f6574077bc3705
-size 30016914
+oid sha256:cc2ff4ea0c2d68217181f37e11af662feba1f560f09d89557360c878b261cc0e
+size 30071868

models/word_markov/am_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:780fe369483db11af3d104778e7d5bb9b9a5f148283b4a56821d1108c225a8f1
-size 37857525
+oid sha256:de52d6ed2d3cf8ba885adb29815096e6f0eaf92d7f4420e09402db1799affe4e
+size 37843385

models/word_markov/am_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e728b8a8a0a36069b78abb0671af76227c43bf867e138b921d8bc54cbf49eefb
-size 43426773
+oid sha256:26c1f4fea34061dd7bf3ac5175c36ba8e8c0eecf6bf70e7a3b3673f18dfef779
+size 43401561

models/word_ngram/am_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ec5b536313f116b0bde636f400cc6da30021220a55d66b04d2d921a46afac00c
-size 546977
+oid sha256:3253eb36f46641335616a4d569c3e3db153afd4a90452fd3c6b3a43277097773
+size 548774

models/word_ngram/am_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a4b1a9110621fff9f70cff208a0fd53443188c369db1a7931d916d8df983445d
-size 758494
+oid sha256:9e2da18fb13944faacfb0c55bc5dbe5f5aff6172a504ee740427290dda7f2ee6
+size 761215

models/word_ngram/am_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b1d7bbbf331a49536843f3ee8f30a563ca757570bee2a718000a58404f82e7e8
-size 2116335
+oid sha256:21a05c63fd25438fb235f39c907ad834a6f3b4a58c825f0913049f3461b196f7
+size 2121749

models/word_ngram/am_5gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4df340c03cec3e7a88e1f02a417db7326be9b7576ed70faa5e9eba5a5bad79b6
-size 2009853
+oid sha256:42543fabd233c4bac0f24bbbd665c94db17fdf8859ee492de75dc0a80d2dfc9e
+size 2016123