ryan-minato commited on
Commit
84ee4a0
โ€ข
1 Parent(s): c4ac097

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +18 -0
README.md CHANGED
@@ -26,3 +26,21 @@ Models previously compressed using zstd have been decompressed, allowing for imm
26
 
27
  ## Usage
28
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
 
27
  ## Usage
28
 
29
+ ```python
30
+ from huggingface_hub import hf_hub_download
31
+ import vibrato
32
+
33
+ # Load tokenizer from `.cache/hf`
34
+ model_path = hf_hub_download("aha-org/vibrato-models-bsdonly", "<<model_name>>/system.dic")
35
+ with open(model_path, "rb") as f:
36
+ tokenizer = vibrato.Vibrato(f.read())
37
+
38
+ text = """\
39
+ ใ€Œๅ››ๅไบŒใ ใจ!ใ€ใƒซใƒผใƒณใ‚ฏใ‚ฉใƒผใƒซใŒๅซใ‚“ใ ใ€‚
40
+ ใ€Œไธƒ็™พไบ”ๅไธ‡ๅนดใ‹ใ‘ใฆใ€ใใ‚Œใ ใ‘ใ‹?ใ€
41
+ ใ€Œไฝ•ๅบฆใ‚‚ๅพนๅบ•็š„ใซๆคœ็ฎ—ใ—ใพใ—ใŸใ€ใ‚ณใƒณใƒ”ใƒฅใƒผใ‚ฟใŒๅฟœใ˜ใŸใ€‚
42
+ ใ€ŒใพใกใŒใ„ใชใใใ‚ŒใŒ็ญ”ใˆใงใ™ใ€‚็Ž‡็›ดใชใจใ“ใ‚ใ€ใฟใชใ•ใ‚“ใฎใปใ†ใง็ฉถๆฅตใฎ็–‘ๅ•ใŒไฝ•ใงใ‚ใ‚‹ใ‹ใ‚ใ‹ใฃใฆใ„ใชใ‹ใฃใŸใจใ“ใ‚ใซๅ•้กŒใŒใ‚ใ‚‹ใฎใงใ™ใ€
43
+ """
44
+
45
+ tokenizer.tokenize(text)
46
+ ```