## vocab_file - ice_text.model - 二进制文件 - num_image_tokens = 20000 词典大小 150528 ``` tokens: ['▁good', '▁morning'] ; id: [20315, 21774] ; text: good morning tokens: ['▁good', '<|blank_2|>', 'morning'] ; id: [20315, 150009, 60813] ; text: good morning tokens: ['▁', 'goog', '▁morning', 'abc'] ; id: [20005, 46456, 21774, 27415] ; text: goog morningabc tokens: ['▁', '你是谁'] ; id: [20005, 128293] ; text: 你是谁 ``` `▁` 是啥,空格吗?注意区分 `_` ## ``` tokenizer = TextTokenizer(self.vocab_file) ```