facebook

/

nllb-200-distilled-600M

like 379

Hello,
I am finetuning nllb-200 for a new low corpora language.
For the new language to be recognized by the tokenizer, i'm modifying the source code of the library (tokenisation_nllb.py) by adding the token of my new language to the ""FAIRSEQ_LANGUAGE_CODES" list. After that, I load the nllb tokenizer with the additional_special_token=["new_token"].
Is there any library included function/attribute to do it directly without modifying source code ?
Thanks for your help !