How is falcon able to generate indic token , when it is not trained on Indic languages. even tokenizer.json doest have any Indic tokens.

#90

by Sibadatta - opened Oct 3, 2023

Oct 3, 2023

@sanchit-gandhi boss can you please answer this doubt.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment