Update batch encode plus first argument to match HF convention. 28fe393 verified pragaash commited on Feb 27
Fix auto tokenizer import reference format in auto map as list for slow and fast. a9a132f Pragaash commited on Feb 26
Replace tokenizer class with auto map for relative import of byte tokenizer. 72e0b3e verified pragaash commited on Feb 26
Add tokenizer import reference to auto_map in config.json. 29194e9 verified pragaash commited on Feb 26
Remove tokenizer.json and replace tokenizer.py with correct version. ebac1e8 verified pragaash commited on Feb 26
Update byte tokenizer to be compatible with auto tokenizer and clean-up. 158e9d1 verified pragaash commited on Feb 26