AttributeError: 'YayiTokenizer' object has no attribute 'sp_model'
#4
by
Yhyu13
- opened
Hi,
I tried to load Yiyi2 with transformer 4.36.2, and here is an error with tokenizer class
│ /home/hangyu5/Documents/Git-repoMy/text-generation-webui/server.py:241 in <module> │
│ │
│ 240 # Load the model │
│ ❱ 241 shared.model, shared.tokenizer = load_model(model_name) │
│ 242 if shared.args.lora: │
│ │
│ /home/hangyu5/Documents/Git-repoMy/text-generation-webui/modules/models.py:98 in load_model │
│ │
│ 97 else: │
│ ❱ 98 tokenizer = load_tokenizer(model_name, model) │
│ 99 │
│ │
│ /home/hangyu5/Documents/Git-repoMy/text-generation-webui/modules/models.py:126 in load_tokenizer │
│ │
│ 125 │
│ ❱ 126 tokenizer = AutoTokenizer.from_pretrained( │
│ 127 path_to_model, │
│ │
│ /home/hangyu5/anaconda3/envs/textgen/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py:774 in from_pretrained │
│ │
│ 773 tokenizer_class.register_for_auto_class() │
│ ❱ 774 return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *input │
│ 775 elif config_tokenizer_class is not None: │
│ │
│ /home/hangyu5/anaconda3/envs/textgen/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:2028 in from_pretrained │
│ │
│ 2027 │
│ ❱ 2028 return cls._from_pretrained( │
│ 2029 resolved_vocab_files, │
│ │
│ ... 1 frames hidden ... │
│ │
│ /home/hangyu5/.cache/huggingface/modules/transformers_modules/yayi2-30b/tokenization_yayi.py:74 in __init__ │
│ │
│ 73 pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_ │
│ ❱ 74 super().__init__( │
│ 75 bos_token=bos_token, │
│ │
│ /home/hangyu5/anaconda3/envs/textgen/lib/python3.11/site-packages/transformers/tokenization_utils.py:367 in __init__ │
│ │
│ 366 # the order of addition is the same as self.SPECIAL_TOKENS_ATTRIBUTES following │
│ ❱ 367 self._add_tokens( │
│ 368 [token for token in self.all_special_tokens_extended if token not in self._a │
│ │
│ /home/hangyu5/anaconda3/envs/textgen/lib/python3.11/site-packages/transformers/tokenization_utils.py:467 in _add_tokens │
│ │
│ 466 # TODO this is fairly slow to improve! │
│ ❱ 467 current_vocab = self.get_vocab().copy() │
│ 468 new_idx = len(current_vocab) # only call this once, len gives the last index + │
│ │
│ /home/hangyu5/.cache/huggingface/modules/transformers_modules/yayi2-30b/tokenization_yayi.py:111 in get_vocab │
│ │
│ 110 """Returns vocab as a dict""" │
│ ❱ 111 vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} │
│ 112 vocab.update(self.added_tokens_encoder) │
│ │
│ /home/hangyu5/.cache/huggingface/modules/transformers_modules/yayi2-30b/tokenization_yayi.py:107 in vocab_size │
│ │
│ 106 """Returns vocab size""" │
│ ❱ 107 return self.sp_model.get_piece_size() │
│ 108 │
╰─────────────────────────────────────────────────
AttributeError: 'YayiTokenizer' object has no attribute 'sp_model'
Yhyu13
changed discussion title from
Tokenizer does not have sp_model
to AttributeError: 'YayiTokenizer' object has no attribute 'sp_model'
Also, hf flash attan2 does not support this model yet. Would you like to update this model to be part of hf transformer?
Thank you for your attention to YAYI2! In order to avoid version conflicts, it is recommended to follow the transformer version in config.json. You can also refer to the solutions provided by mzbac in discussion #5.
We will adapt to the latest transformer version as soon as possible.
wenge-research
changed discussion status to
closed