emre's picture
add tokenizer
8cfa350
raw
history blame
628 Bytes
{"s": 0, "щ": 1, "м": 2, "'": 3, ")": 4, "й": 5, "r": 6, "x": 7, "t": 8, "»": 9, "(": 10, "n": 11, "з": 12, "o": 13, "e": 14, "я": 16, "g": 17, "ч": 18, "p": 19, "к": 20, "‑": 21, "ь": 22, "a": 23, "ю": 24, "у": 25, "н": 26, "…": 27, "т": 28, "ц": 29, "k": 30, "и": 31, "ё": 32, "ш": 33, "b": 34, "г": 35, "−": 36, "п": 37, "l": 38, "h": 39, "m": 40, "д": 41, "х": 42, "—": 43, "«": 44, "c": 45, "ф": 46, "б": 47, "р": 48, "f": 49, "ж": 50, "а": 51, "о": 52, "i": 53, "э": 54, "ы": 55, "е": 56, "ъ": 57, "л": 58, "z": 59, "в": 60, "с": 61, "–": 62, "|": 15, "[UNK]": 63, "[PAD]": 64}