gary109's picture
Add tokenizer
92c47f5
raw
history blame contribute delete
177 Bytes
{
"#": 1,
"1": 2,
"2": 3,
"3": 4,
"4": 5,
"5": 6,
"[PAD]": 15,
"[UNK]": 14,
"a": 7,
"b": 8,
"c": 9,
"d": 10,
"e": 11,
"f": 12,
"g": 13,
"|": 0
}