stanford-crfm
/

pubmed_gpt_tokenizer

Model card Files Files and versions Community

pubmed_gpt_tokenizer / merges.txt

Commit History

revert to 28k vocab

3d6a170

J38 commited on Oct 31, 2022

update to new 43k tokenizer for experiment

b101750

J38 commited on Oct 29, 2022

28k vocab size, prefix_space=False, truecase

c8684a4

J38 commited on Sep 16, 2022

50k vocab, prefix_space=false,trained on PubMed Abstracts

39545d2

J38 commited on Sep 15, 2022

experiment with 50k vocab

9d29e9b

J38 commited on Sep 14, 2022

add merges.txt

c19b598

J38 commited on Sep 5, 2022