Better Tokenization (Uses the correct token for padding)

by Ehsanjahanbakhsh - opened Nov 16, 2023

←

Nov 16, 2023

Created a better tokenizer from sentencepiece model. please test it.

jbochi

Owner Nov 17, 2023

Thank you! I have confirmed this fixed the issue with padding.

jbochi changed pull request status to merged Nov 17, 2023

jbochi changed pull request title from Better Tokenization to Better Tokenization (Uses the correct token for padding) Nov 17, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment