Repository: stanford-crfm/helm
Issue: Turning off thinking for Qwen3 series
Root cause: Documentation Deficiency

I'm experiencing an issue when running MedHELM evaluations with Qwen3 models. The models are producing low scores because many metrics only extract the <think> token and not the actual response. This should be solved by turning the thinking off for the model. The closest I found to a solution from model_deployment file for a qwen3 series model was a param parse_thinking that was used, but when I used it I got below error. The closest solution I found was in the model_deployment file for a qwen3 series model, which used a parameter called parse_thinking, but when I tried using it, I got the below error.

helm.benchmark.executor.ExecutorError: OpenAIClient.__init__() got an unexpected keyword argument 'parse_thinking' Request: Request(model_deployment=...



You may need a custom client for Qwen3 on your inference platform. Currently we only support Together AI. Are you using Hugging Face local inference, or a inference API?

I'm using a locally hosted vLLM server. That's not the issue - it's giving fine results for some metrics like MMIC-RS, MedicationQA, etc., but for some metrics like HeadQA and Medbullets it somehow doesn't detect or parse any response, or mostly just shows " < think > " in the response section. 

- For Qwen3 on vLLM, refer to https://github.com/vllm-project/vllm/issues/17327
- If the response is being truncated due to max tokens, you can increase tokens by using the `increase_max_tokens` run expander e.g. `increase_max_tokens=20000`
- If the metrics are not working due to the extra thinking tokens, you can strip the tokens by using the `process_output` run expander e.g. `process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking`

Putting this all together, this means that instead of running `head_qa:language=en,category=medicine`, you would run `head_qa:language=en,category=medicine,increase_max_tokens=20000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking`

Hi great this solved almost 70 percent of the solution, there was no issue related to Qwen3 usage however this helped a lot
head_qa:language=en,category=medicine,increase_max_tokens=20000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking


I had to run that at 12000 max other wise it would give below error 


Traceback (most recent call last):
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/tokenizers/auto_tokenizer.py", line 66, in tokenize_with_retry
    return tokenizer.tokenize(request)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/tokenizers/caching_tokenizer.py", line 123, in tokenize
    raise ValueError(f"Failed to tokenize text with {self.__class__.__name__} tokenizer: {error}") from error
ValueError: Failed to tokenize text with HuggingFaceTokenizer tokenizer: can't convert negative int to unsigned

          Request failed. Retrying (attempt #5) in 16 seconds... (See above for error details)





However now headqa and med bullets are detecting inference, there are some still metrics like medec medcalc that even with this variable are not detecting inference of model i;m also attaching image how they are only detecting one token.

<img width="1448" alt="Image" src="https://github.com/user-attachments/assets/4f8e138e-4a01-45b7-b04e-5a709a352e20" />

Could you provide the complete run entry that you're using for MedCalc?

its not medcalc same issue is in medec ,pubmed and was previously in headqa, med bullet. iam running this through 

helm-run -c ./med_tasks.conf \
    --suite Qwen3/Qwen3-14B  \
    --max-eval-instances 1 \
    --local-path ./config \
    --models-to-run Qwen3/Qwen3-14B




where med tasks are as below if you need anything let me know

entries: [
  {description: "aci_bench:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "ehr_sql:increase_max_tokens=16000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "head_qa:language=en,category=medicine,increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "head_qa:language=en,category=nursing,increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "head_qa:language=en,category=psychology,increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "head_qa:language=en,category=chemistry,increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "head_qa:language=en,category=pharmacology,increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "head_qa:language=en,category=biology,increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "med_dialog:subset=healthcaremagic,increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "med_dialog:subset=icliniq,increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "medbullets:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "medcalc_bench:increase_max_tokens=16000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "medec:increase_max_tokens=16000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "medhallu:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "medication_qa:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "mtsamples_procedures:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "mtsamples_replicate:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "pubmed_qa:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "mimic_bhc:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1},
  {description: "mimic_rrs:increase_max_tokens=12000,process_output=helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking", priority: 1}
]

(client_env) ubuntu@152-67-127-3:~/glm/MedHELM$ helm-run -c ./run_entries.conf     --suite Qwen/Qwen3-14B      --max-eval-instances 1     --local-path ./config     --models-to-run Qwen/Qwen3-14B
main {
  Reading tokenizer configs from /home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/config/tokenizer_configs.yaml...
  Reading model deployments from /home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/config/model_deployments.yaml...
  Reading tokenizer configs from ./config/tokenizer_configs.yaml...
  Reading model deployments from ./config/model_deployments.yaml...
  Read 1 run entries from ./run_entries.conf
  1 entries produced 1 run specs
  run_specs {
    RunSpec(name='medcalc_bench:model=Qwen_Qwen3-14B', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.medcalc_bench_scenario.MedCalcBenchScenario', args={}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Given a patient note and a clinical question, compute the requested medical value.\n', input_prefix='', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', chain_of_thought_prefix='', chain_of_thought_suffix='\n', output_prefix='Answer only the requested quantity without units. No explanation needed: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=0, max_eval_instances=1, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='Qwen/Qwen3-14B', model='Qwen/Qwen3-14B', temperature=0.0, max_tokens=12010, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, reeval_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.output_processing_metric.OutputProcessingMetric', args={'processor': 'helm.benchmark.metrics.output_processors.remove_deepseek_r1_thinking', 'metric_specs': [{'class_name': 'helm.benchmark.metrics.medcalc_bench_metrics.MedCalcBenchMetric', 'args': {}}, {'class_name': 'helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', 'args': {'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}}, {'class_name': 'helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', 'args': {}}, {'class_name': 'helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', 'args': {}}]})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['medcalc_bench'], annotators=None)
  } [0.0s]
  Running in local mode with base path: ./config
Looking in path: ./config
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  AutoClient: file_storage_path = ./config/cache
  AutoClient: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  Found 1 account(s).
Looking in path: ./config
  AnnotatorFactory: file_storage_path = ./config/cache
  AnnotatorFactory: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  0%|                                                                                                        | 0/1 [00:00<?, ?it/s]  Running medcalc_bench:model=Qwen_Qwen3-14B {
    scenario.get_instances {
      Processing data for test split
    } [4.757s]
    1047 instances, 0 train instances, 1/1047 eval instances
    DataPreprocessor.preprocess {
    } [0.0s]
    GenerationAdapter.adapt {
      1 instances, choosing 0/0 train instances, 1 eval instances
      Adapting with train_trial_index=0 {
        Sampled 0 examples for trial #0.
        Parallelizing computation on 1 items over 4 threads {
                                                                                                                                            Created cache with config: SqliteCacheConfig(path='./config/cache/Qwen.sqlite')                     | 0/1 [00:00<?, ?it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.14s/it]
        } [1.144s]███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.14s/it]
        Sample prompts {
          reference index = None, request_mode = None {
            Given a patient note and a clinical question, compute the requested medical value.
            
            Patient note: A 70-year-old female was rushed into the ICU due to respiratory distress, following which she was promptly put on mechanical ventilation. Her delivered oxygen fell to 51 % FiO₂; meanwhile, her partial pressure of oxygen (PaO₂) registered at 74 mm Hg. She was conscious but visibly disoriented with a functional Glasgow Coma Score of 12. She was hypotensive with blood pressure of 91/70 mm Hg. Multiple vasopressors are being administered simultaneously including DOPamine at 4 mcg/kg/min, norEPINEPHrine at 0.06 mcg/kg/min, DOBUTamine at 3 mcg/kg/min, and EPINEPHrine at 0.03 mcg/kg/min. Laboratory evaluations revealed mild renal impairment with creatinine levels slightly elevated at 1.6 mg/dL and a bilirubin level of 1.9 mg/dL. Her platelet count was found to be 165,000/µL. Her daily urine output of 950 mL.
            
            Question: What is the patient's Sequential Organ Failure Assessment (SOFA) Score?
            
            Answer only the requested quantity without units. No explanation needed:
          } [0.0s]
        } [0.0s]
      } [1.144s]
      1 requests
    } [1.144s]
    Executor.execute {
      Parallelizing computation on 1 items over 4 threads {
                                                                                                                                          Created cache with config: SqliteCacheConfig(path='./config/cache/Qwen.sqlite')                       | 0/1 [00:00<?, ?it/s]
        Created cache with config: SqliteCacheConfig(path='./config/cache/Qwen.sqlite')
        WARNING: truncate_sequence needs to strip "\n"
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.12s/it]
      } [2.118s]█████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.12s/it]
      Processed 1 requests
    } [2.118s]
    AnnotationExecutor.execute {
      No annotators to run.
    } [0.0s]
    1 metrics {
      <helm.benchmark.metrics.output_processing_metric.OutputProcessingMetric object at 0x70ef187e4d90> {
        Parallelizing computation on 1 items over 4 threads {
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 21845.33it/s]
        } [0.0s]                                                                                             | 0/1 [00:00<?, ?it/s]
        Parallelizing computation on 1 items over 4 threads {
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 119.57it/s]
        } [0.01s]                                                                                            | 0/1 [00:00<?, ?it/s]
        Skipping computing calibration metrics because logprobs were not available.
        Parallelizing computation on 1 items over 4 threads {
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 20560.31it/s]
        } [0.0s]                                                                                             | 0/1 [00:00<?, ?it/s]
      } [0.015s]
    } [0.015s]
    Generated 72 stats.
    Writing 2522 characters to benchmark_output/runs/Qwen/Qwen3-14B/medcalc_bench:model=Qwen_Qwen3-14B/run_spec.json
    Writing 374 characters to benchmark_output/runs/Qwen/Qwen3-14B/medcalc_bench:model=Qwen_Qwen3-14B/scenario.json
    Writing 4413 characters to benchmark_output/runs/Qwen/Qwen3-14B/medcalc_bench:model=Qwen_Qwen3-14B/scenario_state.json
    Writing 23205 characters to benchmark_output/runs/Qwen/Qwen3-14B/medcalc_bench:model=Qwen_Qwen3-14B/stats.json
    Writing 6755 characters to benchmark_output/runs/Qwen/Qwen3-14B/medcalc_bench:model=Qwen_Qwen3-14B/per_instance_stats.json
    CacheStats.print_status {
      ./config/cache/Qwen.sqlite: 6 queries, 0 computes
    } [0.0s]
  } [9.724s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:09<00:00,  9.72s/it]
  Done.
} [14.621s]