How can I extend the tokenizer to create a multi-lingual model of Phi-2?

#71

by mochaneko - opened Jan 7, 2024

Jan 7, 2024

I don't know how to extend the tokenizer for Phi-2 because I don't know unigram, bpe, etc.

mochaneko changed discussion status to closed Jan 8, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment