vocab/chatglm_6b/README.md

## vocab_file

- ice_text.model
  - 二进制文件
  - num_image_tokens = 20000
文本词典大小=150528-20000


```
tokens:  ['▁good', '▁morning'] ;	            id:  [20315, 21774] ;	            text:  good morning
tokens:  ['▁good', '<|blank_2|>', 'morning'] ;	id:  [20315, 150009, 60813] ;	    text:  good  morning
tokens:  ['▁', 'goog', '▁morning', 'abc'] ;     id:  [20005, 46456, 21774, 27415] ;	text:  goog morningabc
tokens:  ['▁', '你是谁'] ;	                    id:  [20005, 128293] ;	            text:  你是谁
```

`▁` 是啥，空格吗？注意区分 `_`


## 

```
    tokenizer = TextTokenizer(self.vocab_file)
```