databricks

/

dbrx-instruct

like 1.07k

You would have to get the tokenizer files to the machine using it, somehow, at least once. After that, by default transformers caches the files locally. You can also copy the files by other means, then load a tokenizer by pointing at the local directory containing the files. In this respect it's not different than other tokenizers. This model uses tiktoken, in particular, if that helps.

Hi @Rnake , we recently updated the tokenizer to use the GPT2Tokenizer class and no longer require tiktoken. Please give it a try and let us know how it goes!

thanks