sail
/

scaling-with-vocab-trained-tokenizers

Model card Files Files and versions Community

scaling-with-vocab-trained-tokenizers / README.md

tcftrees's picture

Create README.md

47d7788 verified 7 months ago

|

145 Bytes

The trained BPE tokenziers with various vocabulary sizes, which uses to study how the vocabulary size affects the performance of language models.