Vocab size of tokenizer is not equal to vocab size of the model

by dinhanhx - opened Dec 2, 2023

Dec 2, 2023

•

edited Dec 2, 2023

I notice that in config.json, the vocab size of the model is 256512. However, the vocab size of the tokenizer is 256000.

dinhanhx changed discussion title from Vocab size is 256512 or 25600? to Vocab size of tokenizer is not equal to vocab size of the model Dec 2, 2023

jbochi

Owner Dec 4, 2023

The vocab size is set to 256512 in the original gin file. That matches the dimensions of the weights.

I believe the difference is due to EXTRA_IDS = 512, also from the gin file. You can fine-tune those for your own needs.

jbochi changed discussion status to closed Dec 4, 2023

dinhanhx

Dec 4, 2023

Ah I see thanks for clarifying

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment