Model response concatenated
#4
by
inuwamobarak
- opened
Why is the response concatenated like this:
# Define the input prompt
chat = [
{
"role": "user",
"content": "Please list one IBM Research laboratory located in the United States. You should only output its name and location."
},
]
# Tokenize input
chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
input_tokens = tokenizer(chat, return_tensors="pt").to(device)
# Generate model output
output = model.generate(**input_tokens, max_new_tokens=100)
# Decode and print the response
response = tokenizer.batch_decode(output, skip_special_tokens=True)
print(response)
RESPONSE:
userPlease list one IBM Research laboratory located in the United States. You should only output its name and location.
assistant1. IBM Research - Austin, Texas
Why is there no space between the user or assistant and the string?
Hi
@inuwamobarak
, thanks for your interest in this model! You've provided the skip_special_tokens=True flag to batch_decode which removes the special tokens that would normally separate the role and content fields in the model response. If you set that to False, it should insert the special separator tokens.
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("granite-3.0-2b-instruct")
model = AutoModelForCausalLM.from_pretrained("granite-3.0-2b-instruct")
chat = [
{
"role": "user",
"content": "Please list one IBM Research laboratory located in the United States. You should only output its name and location."
},
]
chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
input_tokens = tokenizer(chat, return_tensors="pt")
output = model.generate(**input_tokens, max_new_tokens=100)
print(tokenizer.batch_decode(output, skip_special_tokens=True))
# Output: ['userPlease list one IBM Research laboratory located in the United States. You should only output its name and location.\nassistant1. IBM Research - Austin, Texas']
print(tokenizer.batch_decode(output, skip_special_tokens=False))
# Output: ['<|start_of_role|>user<|end_of_role|>Please list one IBM Research laboratory located in the United States. You should only output its name and location.<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>1. IBM Research - Austin, Texas<|end_of_text|>']