olob0/nllb-200-distilled-600M-ct2-int8_float16

NLLB-200-distilled-600M, CTranslate2 int8_float16

facebook/nllb-200-distilled-600M converted to the CTranslate2 format and quantised to int8_float16. The same model, 0.6 GB instead of 2.5 GB, and it loads without a conversion step.

The tokenizer files are included, so nothing else has to be downloaded.

Use

import ctranslate2
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

path = snapshot_download("olob0/nllb-200-distilled-600M-ct2-int8_float16")
tokenizer = AutoTokenizer.from_pretrained(path)
translator = ctranslate2.Translator(path, device="cuda", compute_type="int8_float16")

tokenizer.src_lang = "por_Latn"
tokens = tokenizer.convert_ids_to_tokens(tokenizer("Bom dia.").input_ids)
result = translator.translate_batch([tokens], target_prefix=[["eng_Latn"]])
print(tokenizer.decode(tokenizer.convert_tokens_to_ids(result[0].hypotheses[0][1:])))

The target language goes in as a decoder prefix, and the first output token is that same language code, which is why it is dropped before decoding.

How it was built

ct2-transformers-converter --model facebook/nllb-200-distilled-600M \
  --quantization int8_float16 --output_dir nllb-200-distilled-600M-ct2-int8_float16 \
  --copy_files tokenizer.json tokenizer_config.json special_tokens_map.json sentencepiece.bpe.model

License

CC-BY-NC-4.0, inherited from the original model. Non-commercial use only. Original work by Meta AI, this is a format conversion of it.