pivoine-7b / README.md
keminglu's picture
add tokenizer
16242c4
metadata
license: bigscience-bloom-rail-1.0

Tokenizer used for all BLOOM models

Tokenizer information are provided at https://huggingface.co/bigscience/bloom#preprocessing

TODO: point to paper once it comes out with extra details on the tokenizer