iloko_model_new / vocab.json
denden's picture
add tokenizer
6cbef3f
{"h": 0, "n": 1, "‘": 2, "p": 3, "l": 4, "u": 5, "t": 6, "g": 7, "m": 8, "b": 9, "\n": 10, "d": 11, "k": 13, "e": 14, "y": 15, "i": 16, "w": 17, "s": 18, "o": 19, "a": 20, "r": 21, "’": 22, "'": 23, "|": 12, "[UNK]": 24, "[PAD]": 25}