michaelfeil

/

ct2fast-mpt-30b-chat

like 2

I'm getting this error:

RuntimeError: Cannot load the vocabulary from the model directory

Source:

python3.8/dist-packages/hf_hub_ctranslate2/translate.py:48

This is the code I'm running:

from transformers import AutoTokenizer
from hf_hub_ctranslate2 import GeneratorCT2fromHfHub

tokenizer = AutoTokenizer.from_pretrained('mosaicml/mpt-30b')

model_name = "michaelfeil/ct2fast-mpt-30b-chat"

model = GeneratorCT2fromHfHub(
        # load in int8 on CUDA
        model_name_or_path=model_name,
        device="cuda",
        compute_type="int8_float16",
        tokenizer=tokenizer
)

outputs = model.generate(
    text=["def fibonnaci(", "User: How are you doing? Bot:"],
    max_length=64,
    include_prompt_in_result=False
)

print(outputs)