hrdipto's picture
add tokenizer
2effb9d
{"right": 0, "left": 1, "shuru": 3, "|": 2, "[UNK]": 4, "[PAD]": 5}