revert to 28k vocab

update to new 43k tokenizer for experiment

fix id issue

fix main tokenizer file

add end of text token

28k vocab size, prefix_space=False, truecase

50k vocab, prefix_space=false,trained on PubMed Abstracts

experiment with 50k vocab

add of |endoftext|

add of |endoftext|

use lowercase normalizer

does bert normalizer work

change lowercase key

add normalizer

add lowercase normalizer

add merges.txt

add vocab.json

add vocab file

do lower case

config for tokenizer

tokenizer model

initial commit