TokenizerFast with tokenizer.json

by vubiqus - opened Nov 8, 2023

Nov 8, 2023

Hello,

Will it be possible to release a tokenizer.json config for TokenizerFast like in whisper-large-v2 ?

Thanks

lmz

Nov 8, 2023

Also interested by a tokenizer.json config, this would make it easy to use the model with candle.

Nov 16, 2023

Closed by #16

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment