meta-llama

/

Meta-Llama-3-8B-Instruct

like 2.72k

I am using this sample code from this official blog of hugging face: https://huggingface.co/blog/llama3

I already am using the latest version '4.41.1' of transformers library, still I am getting the following error:

The model 'LlamaModel' is not supported for text-generation. Supported models are ['BartForCausalLM', 'BertLMHeadModel', 'BertGenerationDecoder', 'BigBirdForCausalLM', 'BigBirdPegasusForCausalLM', .......].

Code from Hugging Face Blog (with a modification that I am using locally downloaded model)

import torch

# model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained("./models/meta-llama/Meta-Llama-3-8B-Instruct")
model = AutoModel.from_pretrained("./models/meta-llama/Meta-Llama-3-8B-Instruct")


pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer= tokenizer,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
)

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = pipeline(
    prompt,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
print(outputs[0]["generated_text"][len(prompt):])