gpt2-medium-indonesian / tokenizer.json

Commit History

update <|endoftext|> tokenizer id from 50257 to 50256
34c19b7

alvin commited on

refactor tokenizer related files with eos token
08d39dc

alvin commited on

fixed mismatched vocab_size between model and tokenizer
eb84e5f

alvin commited on

add tokenizer
8b888a4

cahya commited on