Description

Issue

The tokenizer_config.json in this repository has extra_special_tokens as a list, but the transformers library expects it to be a dict. This causes an AttributeError when loading the tokenizer.

Error

  File "transformers/tokenization_utils_base.py", line 1210, in _set_model_specific_special_tokens
    self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
                                                                           ^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'keys'

Root Cause

In tokenizer_config.json, extra_special_tokens is defined as:

"extra_special_tokens": [
    "<|im_start|>",
    "<|im_end|>",
    "<|object_ref_start|>",
    ...
]

The transformers library (tokenization_utils_base.py) calls .keys() on this field, which requires a dict, not a list. For example, Qwen/Qwen3-VL-4B-Instruct (the base model) does not have this issue because it does not include extra_special_tokens in its tokenizer config.

This appears to have been introduced by saving the tokenizer with transformers==5.0.0rc0 (as noted in config.json), which may serialize extra_special_tokens differently than stable releases.

Reproduction

pip install transformers>=4.57
python -c "from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('nvidia/nemotron-colembed-vl-4b-v2')"

Suggested Fix

Change extra_special_tokens in tokenizer_config.json from a list to a dict:

"extra_special_tokens": {
    "<|im_start|>": "<|im_start|>",
    "<|im_end|>": "<|im_end|>",
    "<|object_ref_start|>": "<|object_ref_start|>",
    "<|object_ref_end|>": "<|object_ref_end|>",
    "<|box_start|>": "<|box_start|>",
    "<|box_end|>": "<|box_end|>",
    "<|quad_start|>": "<|quad_start|>",
    "<|quad_end|>": "<|quad_end|>",
    "<|vision_start|>": "<|vision_start|>",
    "<|vision_end|>": "<|vision_end|>",
    "<|vision_pad|>": "<|vision_pad|>",
    "<|image_pad|>": "<|image_pad|>",
    "<|video_pad|>": "<|video_pad|>"
}

Or simply remove the extra_special_tokens field, as these tokens are already registered via added_tokens in tokenizer.json.

Environment

• transformers: 4.57.1
• Python: 3.12
• Affects: nvidia/nemotron-colembed-vl-4b-v2, nvidia/nemotron-colembed-vl-8b-v2