Repository: stanford-crfm/helm
Issue: Bug: ValueError: tokenizer_name is None when adding a new Hugging Face model due to prod_env behavior
Root cause: Configuration Error

I am reporting a persistent and difficult-to-debug issue where helm-run fails with a ValueError: tokenizer_name is None when adding a new, standard Hugging Face model.

After a very lengthy debugging session, we have concluded that this is not a user configuration error but rather an unexpected behavior in how HELM handles the prod_env directory, which seems to be auto-created and then overrides otherwise correct configurations.

The Problem in Detail
When a new model is correctly defined in src/helm/config/model_deployments.yaml, running helm-run still fails. The traceback points to the WindowService receiving a None value for tokenizer_name.

The logs show that even if the prod_env directory is deleted or renamed beforehand, the helm-run script appears to find or recreate it, and then enters a "local mode with base path: prod_env". This action seems to invalidate the configurations loaded from src/helm/config/.

Environment
OS: v100 machine

Python Version: 3.10

HELM Installation Method: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct

Steps to Reproduce
Start with a clean clone of the crfm-helm repository.

Bash

git clone https://github.com/stanford-crfm/helm.git
cd helm
Set up the environment (e.g., pip install -e .).

Ensure no prod_env directory exists.

Bash

rm -rf prod_env
Add the following complete and correct model deployment configuration to the end of src/helm/config/model_deployments.yaml:

YAML

- name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  model: meta-llama/Llama-4-Scout-17B-16E-Instruct
  tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  client_spec:
    class_name: helm.proxy.clients.huggingface_client.HuggingFaceClient
    args:
      pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
      trust_remote_code: true
Use any standard run spec, for example, by creating a file named my_eval.conf with the following content:

entries: [
  {description: "MMLU test", priority: 1, groups: ["mmlu"]},
]
Execute the helm-run command:

Bash

helm-run   --conf-paths my_eval.conf   --suite my_suite   --models-to-run meta-llama/Llama-4-Scout-17B-16E-Instruct   --max-eval-instances 1
Expected Behavior
The helm-run command should start successfully, load the specified model, and run the evaluation without configuration errors.

Actual Behavior (The Bug)
The command fails with the ValueError: tokenizer_name is None traceback. Crucially, a prod_env directory is observed to be present during or after the run, and the log shows Running in local mode with base path: prod_env.

(Please paste the full traceback from your failed helm-run command here)

Evidence and Diagnostics (The "Smoking Gun")
To prove that the YAML configuration itself is loaded correctly by HELM's registry system, I ran a separate diagnostic script. This script successfully loads the configuration and finds the correct tokenizer_name. This proves the configuration file is correct, but the information is lost or ignored later in the helm-run process.

Output of the successful diagnostic script:

-Starting Final HELM Debug Script -
-----------------------------------------------------------------
Here is the FINAL configuration HELM actually sees for your model:
{'name': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'client_spec': ClientSpec(class_name='helm.proxy.clients.huggingface_client.HuggingFaceClient', args={'pretrained_model_name_or_path': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'trust_remote_code': True}), 'model_name': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'tokenizer_name': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'window_service_spec': None, 'max_sequence_length': None, 'max_request_length': None, 'max_sequence_and_generated_tokens_length': None, 'deprecated': False}
------------------------------------------------------------------------------
Confirmed Workaround
The only way to make the run succeed is to manually disable the prod_env directory right before running the command:

Bash

mv prod_env prod_env_disabled
helm-run ... # This now works correctly
This confirms the issue is tied to the prod_env override logic.

Thank you for looking into this complex issue.




-----------------------------------------------------------------------------------------------------------------------------------------

(helm) junyao@goofy-1:~/helm$ cat src/helm/config/model_deployments.yaml
model_deployments:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    model: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    client_spec:
      class_name: helm.proxy.clients.huggingface_client.HuggingFaceClient
      args:
        pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
        trust_remote_code: true
    window_service_spec:
      class_name: helm.benchmark.window_services.local_window_service.LocalWindowService
      args:
        tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
(helm) junyao@goofy-1:~/helm$ cat src/helm/benchmark/run_specs/my_llama4_eval.conf
entries: [
  {
    name: "mmlu:computer_security",
    description: "mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama/Llama-4-Scout-17B-16E-Instruct",
    priority: 1,
    scenario_spec: {
      class_name: "helm.benchmark.scenarios.mmlu_scenario.MMLUScenario",
      args: {
        subject: "computer_security"
      }
    },
    adapter_spec: {
      method: "multiple_choice_joint",
      model: "meta-llama/Llama-4-Scout-17B-16E-Instruct",
      model_deployment: "meta-llama/Llama-4-Scout-17B-16E-Instruct"
    },
    metric_specs: [
      {
        class_name: "helm.benchmark.metrics.basic_metrics.BasicGenerationMetric",
        args: {
          names: ["exact_match", "quasi_exact_match", "prefix_exact_match", "quasi_prefix_exact_match"]
        }
      },
      {
        class_name: "helm.benchmark.metrics.basic_metrics.BasicReferenceMetric"
      },
      {
        class_name: "helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric"
      }
    ]
  }
]
(helm) junyao@goofy-1:~/helm$ cat src/helm/config/tokenizer_configs.yaml
tokenizer_configs:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tokenizer: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tokenizer_spec:
      class_name: AutoTokenizer
      tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct

(helm) junyao@goofy-1:~/helm$ cat src/helm/config/model_metadata.yaml
models:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    display_name: Llama‑4 Scout 17B
    creator_organization_name: Meta
    description: |
      Instruction‑tuned Llama‑4 Scout 17B model (16 experts), multimodal (text+image input, text output).
    access: gated
    release_date: 2024-01-30
    license: Llama 4 Community License
    tags:
      - TEXT_MODEL_TAG
      - INSTRUCTION_FOLLOWING_MODEL_TAG
    tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
(helm) junyao@goofy-1:~/helm$ 




















Hi @ztmyjs, thanks for opening an issue.

There are two things going on here:

1. Mode deployment override order. This recent commit 213b918d703885ea5bae70dee53194c564833f6e / pull request #3694 changed the logic for which model deployment gets chosen as the default. If you are on a more recent commit, you should put your model deployment at the _end_ of the file, or you can put it in `prod_env/model_deployments.yaml`. If if you are on an older commit, you should put your model deployment at the _start_ of the file.
2. Tokenizer names. The HELM tokenizer name does not necessarily match the Hugging Face tokenizer name. In this case, the tokenizer name is actually `meta/llama-4-scout-17b-16e-instruct` (see the code snippet below). There are two ways to fix this - either create a new entry in `tokenizer_configs.yaml` with the tokenizer name `meta-llama/Llama-4-Scout-17B-16E-Instruct `, or change the `tokenizer_name` in your model deployments entry in `model_deployments.yaml` to  `meta/llama-4-scout-17b-16e-instruct`. https://github.com/stanford-crfm/helm/blob/cc701a7ec9d01358ffeb228c302d51f08881d9ba/src/helm/config/tokenizer_configs.yaml#L428-L434

Please also refer to the documentation on adding [model deployments](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/) and [tokenizers](https://crfm-helm.readthedocs.io/en/latest/adding_new_tokenizers/).

> Please also refer to the documentation on adding [model deployments](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/) and [tokenizers](https://crfm-helm.readthedocs.io/en/latest/adding_new_tokenizers/).

Hello Yifanmai:

Thanks so much on your kind help!

This issue is unresolvable through any configuration changes. We have now tried every possible combination, including the exact methods suggested in the previous comment.

Final Proof:

I created a "perfect" configuration, strictly following all official advice (using the internal tokenizer_name, placing configs in the correct files, etc.).

I ran the command in a single thread (--num-threads 1) to rule out any parallel processing issues.

The run still fails with the exact same ValueError: tokenizer_name is None traceback. The logs clearly show the correct tokenizer name being loaded by the main thread, but it is lost somewhere before the WindowService needs it.

Here are the following configs i tried the second solution you mentioned:
```
(helm) junyao@goofy-1:~/helm$ git checkout -- src/helm/config/tokenizer_configs.yaml
(helm) junyao@goofy-1:~/helm$ git checkout -- src/helm/config/model_metadata.yaml
(helm) junyao@goofy-1:~/helm$ git checkout -- src/helm/config/model_deployments.yaml
```
--------------------------------------------------------------------------------------------------------------------

(helm) junyao@goofy-1:~/helm$ git log -n 1
```
commit 36eba5ae02df3f2f64313931da8b19f8f388b168 (HEAD -> main, origin/main, origin/HEAD)
Author: Yifan Mai <yifan@cs.stanford.edu>
Date:   Mon Jun 30 15:54:30 2025 -0700
    Add MMMLU scenario (#3699)
```
--------------------------------------------------------------------------------------------------------------------

**my_llama4_eval.conf:**

(helm) junyao@goofy-1:~/helm$ **cat src/helm/benchmark/run_specs/my_llama4_eval.conf**
```
entries: [
  {
    name: "mmlu:computer_security",
    description: "mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama/Llama-4-Scout-17B-16E-Instruct",
    priority: 1,
    adapter_spec: {
      method: "multiple_choice_joint",
      model: "meta-llama/Llama-4-Scout-17B-16E-Instruct",
      model_deployment: "meta-llama/Llama-4-Scout-17B-16E-Instruct",
      tokenizer: "meta-llama/Llama-4-Scout-17B-16E-Instruct"
    }
  }
]
```
--------------------------------------------------------------------------------------------------------------------
**model_metadata.yaml**

(helm) junyao@goofy-1:~/helm$ **cat prod_env/model_metadata.yaml***
```
models:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    display_name: "Llama-4 Scout 17B Instruct"
    creator_organization_name: "Meta"
    description: "Llama-4 Scout 17B Instruct model"
    access: "gated"
    tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tags: ["text-generation"]
```
--------------------------------------------------------------------------------------------------------------------
**tokenizer_configs.yaml**

(helm) junyao@goofy-1:~/**helm$ cat prod_env/tokenizer_configs.yaml**
```
tokenizer_configs:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tokenizer_spec:
      class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
      args:
        pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
```
--------------------------------------------------------------------------------------------------------------------
**model_deployments.yaml**

(helm) junyao@goofy-1:~/**helm$ cat prod_env/model_deployments.yaml**
```
model_deployments:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
        trust_remote_code: true
```
--------------------------------------------------------------------------------------------------------------------
**Error Message**
(helm) junyao@goofy-1:~/helm$ helm-run   --conf-paths src/helm/benchmark/run_specs/my_llama4_eval.conf   --suite my_llama4_suite   --models-to-run meta-llama/Llama-4-Scout-17B-16E-Instruct   --max-eval-instances 100   --num-threads 1
2025-07-05T05:03:38 INFO     helm_run {
2025-07-05T05:03:39 INFO       Reading tokenizer configs from /home/junyao/helm/src/helm/config/tokenizer_configs.yaml...
2025-07-05T05:03:39 INFO       Reading model deployments from /home/junyao/helm/src/helm/config/model_deployments.yaml...
2025-07-05T05:03:39 INFO       Reading tokenizer configs from prod_env/tokenizer_configs.yaml...
2025-07-05T05:03:39 INFO       Reading model deployments from prod_env/model_deployments.yaml...
2025-07-05T05:03:39 INFO       Read 1 run entries from src/helm/benchmark/run_specs/my_llama4_eval.conf
2025-07-05T05:03:41 INFO       1 entries produced 1 run specs
2025-07-05T05:03:41 INFO       run_specs {
2025-07-05T05:03:41 INFO         RunSpec(name='mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama_Llama-4-Scout-17B-16E-Instruct', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.mmlu_scenario.MMLUScenario', args={'subject': 'computer_security'}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about computer security.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', chain_of_thought_prefix='', chain_of_thought_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=100, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='meta-llama/Llama-4-Scout-17B-16E-Instruct', model='meta-llama/Llama-4-Scout-17B-16E-Instruct', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, reeval_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['mmlu'], annotators=None)
2025-07-05T05:03:41 INFO       } [0.0s]
2025-07-05T05:03:41 INFO       Running in local mode with base path: prod_env
Looking in path: prod_env
2025-07-05T05:03:41 INFO       AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
2025-07-05T05:03:41 INFO       AutoClient: file_storage_path = prod_env/cache
2025-07-05T05:03:41 INFO       AutoClient: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
2025-07-05T05:03:41 INFO       AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
Looking in path: prod_env
2025-07-05T05:03:41 INFO       AnnotatorFactory: file_storage_path = prod_env/cache
2025-07-05T05:03:41 INFO       AnnotatorFactory: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
  0%|                                                                                                                                                                                                                                              | 0/1 [00:00<?, ?it/s]2025-07-05T05:03:41 INFO       Running mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama_Llama-4-Scout-17B-16E-Instruct {
2025-07-05T05:03:41 INFO         scenario.get_instances {
2025-07-05T05:03:41 INFO           ensure_file_downloaded {
2025-07-05T05:03:41 INFO             Not downloading https://people.eecs.berkeley.edu/~hendrycks/data.tar because benchmark_output/scenarios/mmlu/data already exists
2025-07-05T05:03:41 INFO           } [0.0s]
2025-07-05T05:03:41 INFO           benchmark_output/scenarios/mmlu/data/auxiliary_train/computer_security_auxiliary_train.csv doesn't exist, skipping
2025-07-05T05:03:41 INFO           Reading benchmark_output/scenarios/mmlu/data/dev/computer_security_dev.csv
2025-07-05T05:03:41 INFO           Reading benchmark_output/scenarios/mmlu/data/val/computer_security_val.csv
2025-07-05T05:03:41 INFO           Reading benchmark_output/scenarios/mmlu/data/test/computer_security_test.csv
2025-07-05T05:03:41 INFO         } [0.002s]
2025-07-05T05:03:41 INFO         116 instances, 5 train instances, 100/111 eval instances
2025-07-05T05:03:41 INFO         DataPreprocessor.preprocess {
2025-07-05T05:03:41 INFO         } [0.0s]
2025-07-05T05:03:41 INFO         MultipleChoiceJointAdapter.adapt {
2025-07-05T05:03:41 INFO           105 instances, choosing 5/5 train instances, 100 eval instances
2025-07-05T05:03:41 INFO           Adapting with train_trial_index=0 {
2025-07-05T05:03:41 INFO             Sampled 5 examples for trial #0.
2025-07-05T05:03:41 INFO             Parallelizing computation on 100 items over 1 threads {
                                                                                                                                                                                                                                                                        [DEBUG] local_context.tokenize() got request.tokenizer = meta-llama/Llama-4-Scout-17B-16E-Instruct                                                                                                                                                | 0/100 [00:00<?, ?it/s]
[DEBUG] _get_tokenizer called with tokenizer_name = meta-llama/Llama-4-Scout-17B-16E-Instruct
  0%|                                                                                                                                                                                                                                            | 0/100 [00:00<?, ?it/s]
2025-07-05T05:03:41 INFO             } [0.0s]
2025-07-05T05:03:41 INFO           } [0.0s]
2025-07-05T05:03:41 INFO         } [0.001s]
2025-07-05T05:03:41 INFO       } [0.005s]
2025-07-05T05:03:41 INFO       Error when running mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama_Llama-4-Scout-17B-16E-Instruct:
Traceback (most recent call last):
  File "/home/junyao/helm/src/helm/benchmark/runner.py", line 216, in run_all
    self.run_one(run_spec)
  File "/home/junyao/helm/src/helm/benchmark/runner.py", line 283, in run_one
    request_states: List[RequestState] = adapter.adapt(instances, self.executor.execution_spec.parallelism)
  File "/home/junyao/helm/src/helm/common/hierarchical_logger.py", line 127, in wrapper
    return fn(*args, **kwargs)
  File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 63, in adapt
    self._adapt_trial_index(all_train_instances, train_trial_index, eval_instances, parallelism)
  File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 86, in _adapt_trial_index
    results: List[List[RequestState]] = parallel_map(
  File "/home/junyao/helm/src/helm/common/general.py", line 239, in parallel_map
    results = list(tqdm(map(process, items), total=len(items), disable=None))
  File "/home/junyao/miniconda3/envs/helm/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 83, in generate_requests_for_training_trial
    return self.generate_requests(eval_instance, train_trial_index, training_instances)
  File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/multiple_choice_joint_adapter.py", line 57, in generate_requests
    prompt = self.construct_prompt(training_instances, eval_instance, include_output=False, reference_index=None)
  File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 242, in construct_prompt
    prompt = self._make_prompt_fit(prompt)
  File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 302, in _make_prompt_fit
    if self.window_service.fits_within_context_window(
  File "/home/junyao/helm/src/helm/benchmark/window_services/local_window_service.py", line 94, in fits_within_context_window
    return self.get_num_tokens(text) + expected_completion_token_length <= self.max_request_length
  File "/home/junyao/helm/src/helm/benchmark/window_services/local_window_service.py", line 87, in get_num_tokens
    return len(self.encode(text).tokens)
  File "/home/junyao/helm/src/helm/benchmark/window_services/local_window_service.py", line 45, in encode
    response: TokenizationRequestResult = self.service.tokenize(
  File "/home/junyao/helm/src/helm/benchmark/window_services/tokenizer_service.py", line 21, in tokenize
    return self._context.tokenize(request)
  File "/home/junyao/helm/src/helm/common/local_context.py", line 93, in tokenize
    return self.tokenizer.tokenize(request)
  File "/home/junyao/helm/src/helm/tokenizers/auto_tokenizer.py", line 72, in tokenize
    tokenizer: Tokenizer = self._get_tokenizer(request.tokenizer)
  File "/home/junyao/helm/src/helm/tokenizers/auto_tokenizer.py", line 38, in _get_tokenizer
    raise ValueError("tokenizer_name is None. Please check your model_metadata.yaml and model_deployments.yaml")
ValueError: tokenizer_name is None. Please check your model_metadata.yaml and model_deployments.yaml

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 144.96it/s]
2025-07-05T05:03:41 INFO     } [2.726s]
Traceback (most recent call last):
  File "/home/junyao/miniconda3/envs/helm/bin/helm-run", line 8, in <module>
    sys.exit(main())
  File "/home/junyao/helm/src/helm/benchmark/run.py", line 371, in main
    return helm_run(args)
  File "/home/junyao/helm/src/helm/common/hierarchical_logger.py", line 127, in wrapper
    return fn(*args, **kwargs)
  File "/home/junyao/helm/src/helm/benchmark/run.py", line 274, in helm_run
    run_benchmarking(
  File "/home/junyao/helm/src/helm/benchmark/run.py", line 127, in run_benchmarking
    runner.run_all(run_specs)
  File "/home/junyao/helm/src/helm/benchmark/runner.py", line 225, in run_all
    raise RunnerError(f"Failed runs: [{failed_runs_str}]")
helm.benchmark.runner.RunnerError: Failed runs: ["mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama_Llama-4-Scout-17B-16E-Instruct"]
(helm) junyao@goofy-1:~/helm$ git checkout -- src/helm/config/tokenizer_configs.yaml







@yifanmai thanks