@nroggendorff on Hugging Face: "I still think whitespace in tokenizers are so dumb. Congrats, you just doubled…"

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

nroggendorff

posted an update Nov 5

Post

2259

I still think whitespace in tokenizers are so dumb.
Congrats, you just doubled your vocab size for no reason.

Nov 7

Any alternative ideas?🤔

·

Nov 20

•

merges.txt :spinnyhat:

In this post