File size: 1,420 Bytes
28a4293 2922974 28a4293 c4ac097 28a4293 84ee4a0 291ed41 84ee4a0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
---
license: bsd-3-clause
language:
- ja
tags:
- vibrato
---
# Vibrato Model Archive (BSD License Only)
This archive contains all models identified under the BSD license from the Vibrato([url](https://github.com/daac-tools/vibrato)) GitHub releases.
Models previously compressed using zstd have been decompressed, allowing for immediate download and use.
## Available Models
- bccwj-suw+unidic-cwj-3_1_1+compact-dual
- bccwj-suw+unidic-cwj-3_1_1+compact
- bccwj-suw+unidic-cwj-3_1_1-extracted+compact-dual
- bccwj-suw+unidic-cwj-3_1_1-extracted+compact
- bccwj-suw+unidic-cwj-3_1_1
- jumandic-mecab-7_0
- unidic-cwj-3_1_1+compact-dual
- unidic-cwj-3_1_1+compact
- unidic-cwj-3_1_1
- unidic-mecab-2_1_2
## Usage
```python
from huggingface_hub import hf_hub_download
import vibrato
# Load tokenizer from `.cache/hf`
model_path = hf_hub_download("ryan-minato/vibrato-models-bsdonly", "<<model_name>>/system.dic")
with open(model_path, "rb") as f:
tokenizer = vibrato.Vibrato(f.read())
text = """\
ใๅๅไบใ ใจ!ใใซใผใณใฏใฉใผใซใๅซใใ ใ
ใไธ็พไบๅไธๅนดใใใฆใใใใ ใใ?ใ
ใไฝๅบฆใๅพนๅบ็ใซๆค็ฎใใพใใใใณใณใใฅใผใฟใๅฟใใใ
ใใพใกใใใชใใใใ็ญใใงใใ็็ดใชใจใใใใฟใชใใใฎใปใใง็ฉถๆฅตใฎ็ๅใไฝใงใใใใใใฃใฆใใชใใฃใใจใใใซๅ้กใใใใฎใงใใ
"""
tokenizer.tokenize(text)
```
|