vocab/chatglm_6b/README.md

vocab_file

• ice_text.model
  • 二进制文件
  • num_image_tokens = 20000 文本词典大小=150528-20000

tokens:  ['▁good', '▁morning'] ;	            id:  [20315, 21774] ;	            text:  good morning
tokens:  ['▁good', '<|blank_2|>', 'morning'] ;	id:  [20315, 150009, 60813] ;	    text:  good  morning
tokens:  ['▁', 'goog', '▁morning', 'abc'] ;     id:  [20005, 46456, 21774, 27415] ;	text:  goog morningabc
tokens:  ['▁', '你是谁'] ;	                    id:  [20005, 128293] ;	            text:  你是谁

▁ 是啥，空格吗？注意区分 _

    tokenizer = TextTokenizer(self.vocab_file)