How to set the attention mask for inference ?

#13
by jgsmcmahon - opened

MSG appears at Inference : The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's attention_mask to obtain reliable results. Setting pad_token_id to eos_token_id:2 for open-end generation.

Please let me know what I need to do to get past this issue ? Many thanks :-)

Just pass the full inputs from the tokenizer. inputs = tokenizer(["Hey how are you?"], ['Good'], padding = True) will return the attention mask. But if there is no padding token you should specify tokenizer.pad_token = tokenizer.eos_token

Ok thanks will try this :-)

Add this after loading in the tokenizer from pretrained:

if tokenizer.pad_token is None:
            tokenizer.pad_token = tokenizer.eos_token

And then change the generate line to this:

generated_ids = model.generate(model_inputs, pad_token_id=tokenizer.pad_token_id, max_new_tokens=2000, do_sample=True)

Sign up or log in to comment