CohereForAI

/

c4ai-command-r-plus

like 1.44k

Hello, I am installing c4ai-command-r-plus locally, but whenever I run the following code, it only gives the warning 'Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained' and does not produce any other output. How can I solve this issue?

PYTHON CODE;

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "CohereForAI/c4ai-command-r-plus"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "Hello!"
generated_text = model.generate(tokenizer.encode(prompt, return_tensors="pt"))
print(generated_text[0])