Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.

#72
by tianke0711 - opened

I following the code provided by example:

import transformers
import torch

base_model = "meta-llama/Meta-Llama-3-8B"
pipeline = transformers.pipeline("text-generation", model=base_model, 
                                 model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto")
pipeline("Hey how are you doing today?")

But there is error:

Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Unexpected exception formatting exception. Falling back to standard exception
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3526, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/var/tmp/ipykernel_28044/1028330205.py", line 1, in <module>
    pipeline("Hey how are you doing today?")
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 241, in __call__
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1196, in __call__
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1203, in run_single
    if self.call_count > 10 and self.framework == "pt" and self.device.type == "cuda":
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1102, in forward
    It is not meant to be called directly, it will be automatically called and the final parameters resolved by
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 328, in _forward
    out_b = generated_sequence.shape[0]
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1597, in generate
    logits_processor=prepared_logits_processor,
  File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 2715, in sample
    ['Today is a beautiful day, and we must do everything possible to make it a day of celebration.']
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1208, in forward
    outputs = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 992, in forward
    causal_mask = self._update_causal_mask(attention_mask, inputs_embeds, cache_position, past_seen_tokens)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1076, in _update_causal_mask
    if AttentionMaskConverter._ignore_causal_mask_sdpa(
AttributeError: type object 'AttentionMaskConverter' has no attribute '_ignore_causal_mask_sdpa'
Meta Llama org

Mmm this is super weird, seems like a versioning issue. This function is defined here: https://github.com/huggingface/transformers/blob/9df8b301ceb6697a713fbe4034dc02395e5b28cd/src/transformers/modeling_attn_mask_utils.py#L238 .

The error does not come from the pad token id, but rather from transformers. Unfortunately I cannot reproduce your issue:

image.png

Meta Llama org

I suspect there is some environnement issue at play

I suspect there is some environnement issue at play

Thanks, it is version issue.

I suspect there is some environnement issue at play

Thanks, it is version issue.

Hi ,thanks for your question and I met the same problem here, may I ask how you solved the problem?
And if there is no pad_id, how can I pad the sentence to the max length?

I suspect there is some environnement issue at play

Thanks, it is version issue.

Hi ,thanks for your question and I met the same problem here, may I ask how you solved the problem?
And if there is no pad_id, how can I pad the sentence to the max length?

I think you need to change transformer version. please try different version, if want my installed transformer version, please let me know.

I suspect there is some environnement issue at play

Thanks, it is version issue.

Hi ,thanks for your question and I met the same problem here, may I ask how you solved the problem?
And if there is no pad_id, how can I pad the sentence to the max length?

I think you need to change transformer version. please try different version, if want my installed transformer version, please let me know.

Thank you very much and I might really need your help for the version(●'◡'●)

I suspect there is some environnement issue at play

Thanks, it is version issue.

Hi ,thanks for your question and I met the same problem here, may I ask how you solved the problem?
And if there is no pad_id, how can I pad the sentence to the max length?

I think you need to change transformer version. please try different version, if want my installed transformer version, please let me know.

Thank you very much and I might really need your help for the version(●'◡'●)

My library version:

accelerate ==0.21.0
transformer ==4.34.0
torch == 2.0.1
trl ==0.4.7
peft==0.4.0

I suspect there is some environnement issue at play

Thanks, it is version issue.

Hi ,thanks for your question and I met the same problem here, may I ask how you solved the problem?
And if there is no pad_id, how can I pad the sentence to the max length?

I think you need to change transformer version. please try different version, if want my installed transformer version, please let me know.

Thank you very much and I might really need your help for the version(●'◡'●)

My library version:

accelerate ==0.21.0
transformer ==4.34.0
torch == 2.0.1
trl ==0.4.7
peft==0.4.0

Thanks a lot for your helping! I will have a try later.

In the instruct version the eos token was recently changed to "128009", while the generation config defines both "[128001, 128009]".
Is this supposed to be done in this base version too @ArthurZ ?

Sign up or log in to comment