Repository: EleutherAI/lm-evaluation-harness
Issue: Issue with the Tokenizer of Pixtral-12B-2409
Root cause: Validation Gap

I am trying to evaluate `mistralai/Pixtral-12B-2409` on a few text-only tasks using lm-evaluation-harness. But, I see that there are issues with the tokeniser of this model.
Below you can find the commands and the logs of my runs. I've tried with both hugging face (`--model hf`) and vllm (`--model vllm`) and got different errors with each setting. But, they both seem to be related to the tokenizer of the model.

**HuggingFace:**
```
CUDA_VISIBLE_DEVICES=0 accelerate launch --main_process_port 18001 -m lm_eval \
    --model hf \
    --model_args pretrained='mistralai/Pixtral-12B-2409',dtype=auto \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --tasks leaderboard_mmlu_pro \
    --batch_size auto \
    --output_path OUTPUT_DIR
```

with this setting, I get the OSError, saying that it can not load the tokenizer. See the full log below:
```
2025-02-24:16:44:48,090 INFO     [lm_eval.models.huggingface:136] Using device 'cuda'
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/__main__.py", line 467, in <module>
    cli_evaluate()
  File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/__main__.py", line 388, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/utils.py", line 422, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/evaluator.py", line 209, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/api/model.py", line 151, in create_from_arg_string
    return cls(**args, **args2)
  File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/models/huggingface.py", line 180, in __init__
    self._create_tokenizer(
  File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/models/huggingface.py", line 726, in _create_tokenizer
    self.tokenizer = transformers.AutoTokenizer.from_pretrained(
  File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 963, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2036, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'mistralai/Pixtral-12B-2409'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'mistralai/Pixtral-12B-2409' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.
Traceback (most recent call last):
  File "/mnt/data/amin/venvs/harness.pixtral/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1199, in launch_command
    simple_launcher(args)
  File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/accelerate/commands/launch.py", line 778, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/mnt/data/amin/venvs/harness.pixtral/bin/python', '-m', 'lm_eval', '--model', 'hf', '--model_args', 'pretrained=mistralai/Pixtral-12B-2409,dtype=auto', '--apply_chat_template', '--fewshot_as_multiturn', '--tasks', 'leaderboard_mmlu_pro', '--batch_size', 'auto', '--output_path', '/mnt/data/amin/NLP/widn/projects/tower-vision/results/TEXT-ONLY/leaderboard_mmlu_pro']' returned non-zero exit status 1.
```

**VLLM:**
and here is the command that I use to run the evaluation with vllm.
```
CUDA_VISIBLE_DEVICES=0 accelerate launch --main_process_port 18001 -m lm_eval \
    --model vllm \
    --model_args pretrained='mistralai/Pixtral-12B-2409',dtype=auto,gpu_memory_utilization=0.9,tokenizer_mode=mistral \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --tasks leaderboard_mmlu_pro \
    --batch_size auto \
    --output_path OUTPUT_DIR
```

with this setting, I get the `NotImplementedError` for `tokenizer.pad_token`. See the full log below:
```
2025-02-24:16:49:51,378 INFO     [mistral_common.tokens.tokenizers.tekken:114] Adding special tokens <SPECIAL_20>, ..., <SPECIAL_999>
2025-02-24:16:49:51,378 INFO     [mistral_common.tokens.tokenizers.tekken:293] Vocab size: 150000
2025-02-24:16:49:51,379 INFO     [mistral_common.tokens.tokenizers.tekken:297] Cutting vocab to first 130072 tokens.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/__main__.py", line 467, in <module>
[rank0]:     cli_evaluate()
[rank0]:   File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/__main__.py", line 388, in cli_evaluate
[rank0]:     results = evaluator.simple_evaluate(
[rank0]:   File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/utils.py", line 422, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/evaluator.py", line 209, in simple_evaluate
[rank0]:     lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
[rank0]:   File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/api/model.py", line 151, in create_from_arg_string
[rank0]:     return cls(**args, **args2)
[rank0]:   File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 127, in __init__
[rank0]:     self.tokenizer = configure_pad_token(self.tokenizer)
[rank0]:   File "/mnt/data/amin/tools/lm-evaluation-harness/lm_eval/models/utils.py", line 644, in configure_pad_token
[rank0]:     if tokenizer.pad_token:
[rank0]:   File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 291, in pad_token
[rank0]:     raise NotImplementedError()
[rank0]: NotImplementedError
[rank0]:[W224 16:49:52.779066435 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
Traceback (most recent call last):
  File "/mnt/data/amin/venvs/harness.pixtral/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1199, in launch_command
    simple_launcher(args)
  File "/mnt/data/amin/venvs/harness.pixtral/lib/python3.10/site-packages/accelerate/commands/launch.py", line 778, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/mnt/data/amin/venvs/harness.pixtral/bin/python', '-m', 'lm_eval', '--model', 'vllm', '--model_args', 'pretrained=mistralai/Pixtral-12B-2409,dtype=auto,dtype=auto,gpu_memory_utilization=0.8,tokenizer_mode=mistral', '--apply_chat_template', '--fewshot_as_multiturn', '--tasks', 'leaderboard_mmlu_pro', '--batch_size', 'auto', '--output_path', '/mnt/data/amin/NLP/widn/projects/tower-vision/results/TEXT-ONLY/leaderboard_mmlu_pro']' returned non-zero exit status 1.
```


Looking at the code of the `MistralTokenizer` class in https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/tokenizers/mistral.py  I see that `pad_token`  and `sep_token` are not implemented:

```
    @property
    def sep_token(self) -> str:
        raise NotImplementedError()


    @property
    def pad_token(self) -> str:
        breakpoint()
        raise NotImplementedError()
```

The strange part to me is that by following the examples provided in the model card of this model on HuggingFace (https://huggingface.co/mistralai/Pixtral-12B-2409) I can easily run the inference using vllm without any issues. 

Can this be related to the fact that Mistral tokenizer doesn't support guided decoding yet, as reported here: https://github.com/vllm-project/vllm/issues/9359#issuecomment-2412840803

In this case, what do you recommend? What would be the best way to evaluate this model using lm-evaluation-harness? 


Any updates here? Facing exactly the same problem while evaluating, vllm==0.9.2, transformers==4.53.2, lm_eval[vllm]==0.4.9

lm_eval --model vllm \
 --model_args pretrained=mistralai/Mistral-Small-3.2-24B-Instruct-2506,tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=1,tokenizer_mode=mistral,config_format=mistral,load_format=mistral \
 --tasks lambada_openai,hellaswag \
 --batch_size auto


Same here for `mistralai/Mistral-Small-24B-Instruct-2501` and `mistralai/Mistral-Small-3.2-24B-Instruct-2506`

I'm using vllm==0.11.0, lm-eval[vllm]==0.4.9 and transformers==4.55.4

Here's the command I'm running:
```
lm-eval --model vllm
  --model_args pretrained=mistralai/Mistral-Small-24B-Instruct-2501,tokenizer_mode=mistral,config_format=mistral,load_format=mistral,dtype=auto
  --tasks multimedqa
  --device cuda
  --batch_size auto
  --seed 42
  --apply_chat_template
  --num_fewshot 0
  --output_path results
  --trust_remote_code
```

Yet, I can successfully serve Mistral 3.2 with:
```
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 \
  --tokenizer_mode mistral --config_format mistral \
  --load_format mistral --tool-call-parser mistral \
  --enable-auto-tool-choice --limit-mm-per-prompt '{"image":0}' \
  --tensor-parallel-size 1
```