Repository: promptfoo/promptfoo
Issue: Unable to disable thinking for qwen3-235b-a22b-thinking-2507
Root cause: Configuration Error

**Describe the bug**
Similar to https://github.com/promptfoo/promptfoo/issues/5182

All my function call assertions fail because I am unable to suppress the thinking output of qwen3-235b-a22b-thinking-2507

When trying to use qwen/qwen3-235b-a22b-thinking-2507 I am unable to suppress thinking by using the `showThinking: false` flag as the Promptfoo docs describe

Is there really no other choice than using qwen/qwen3-235b-a22b-2507? I would rather use the general thinking model 

**To Reproduce**
Use this as a provider to reproduce

- id: openrouter:qwen/qwen3-coder
    config:
      temperature: 0.1
      showThinking: false
      top_p: 0.25
      tool_choice: auto
      apiKeyEnvar: OPENROUTER_API_KEY
      apiBaseUrl: https://openrouter.ai/api/v1
      tools: *tools
`;

**Expected behavior**
Thinking output should be suppressed--only a tool call should be visible if triggering a tool call

**Screenshots**

NA

**System information:**

=============================================================================================
Promptfoo Debug Information
=============================================================================================
{
  "version": "0.117.5",
  "platform": {
    "os": "darwin",
    "release": "24.6.0",
    "arch": "arm64",
    "nodeVersion": "v20.18.1"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
=============================================================================================
Please include this output when reporting issues on GitHub: https://github.com/promptfoo/promptfoo/issues

- Promptfoo version:

0.117.5

**Additional context**
NA

Hi! I've addressed this bug in https://github.com/promptfoo/promptfoo/pull/5263 and this should be fixed in the next release.

> Hi! I've addressed this bug in [#5263](https://github.com/promptfoo/promptfoo/pull/5263) and this should be fixed in the next release.

👑 

Hey, this is now available in the latest version- 0.117.6. Let me know if you run into any more issues!

> Hey, this is now available in the latest version- 0.117.6. Let me know if you run into any more issues!

hey @faizanminhas I upgraded both in my package.json and in the CLI but still seeing this issue for multiple providers

here's an example provider

```
- id: openrouter:gpt-5-mini
    config:
      temperature: 0.1
      showThinking: false
      top_p: 0.25
      response_format: text
      reasoning_effort: high
      tool_choice: auto
      apiKeyEnvar: OPENROUTER_API_KEY
      apiBaseUrl: https://openrouter.ai/api/v1
      tools: *tools
```

here's another example provider

```
- id: openrouter:qwen/qwen3-235b-a22b-thinking-2507
    config:
      temperature: 0.1
      showThinking: false
      top_p: 0.25
      max_tokens: 1024
      tool_choice: auto
      apiKeyEnvar: OPENROUTER_API_KEY
      apiBaseUrl: https://openrouter.ai/api/v1
      tools: *tools
```

both of these are throwing errors because it's not hiding the thinking. Maybe this is an issue with the OpenRouter integration?

I validated I am on the latest version


<img width="278" height="58" alt="Image" src="https://github.com/user-attachments/assets/e612f034-2b0c-4fd4-8153-c6e022c44d61" />

Here's what the promptfoo dashboard looks like (sorry for the redactions, but just to give you an idea)

<img width="1468" height="582" alt="Image" src="https://github.com/user-attachments/assets/24451511-a56b-4f23-a5aa-66c953fd7ffe" />







hey @jahabeebs, just double checked and I'm pretty sure that the thinking tokens aren't being returned in the output. Can you try running with `--no-cache` in case you're looking at cached results? If that doesn't work, can you double check with the `--verbose` flag running that the `message.reasoning` is what's being included? If you're looking for a particular format, you can also try adding a response_format, like this, to force structured output:

```
  config:
      response_format:
        type: json_schema
        json_schema:
          name: answer
          strict: true
          schema:
            {
              "type": "object",
              "properties": {
                "answer": {
                  "type": "integer",
                },
              },
              "required": ["answer"],
              "additionalProperties": false
            }
```

Let me know if that doesn't work- I could be testing incorrectly 👀 

@faizanminhas thanks for the tips. I have indeed been running with --no-cache and I used the verbose flag to save the output. As you suggest all of the reasoning values show "0"

```
  "completionDetails": {
          "reasoning": 0,
          "acceptedPrediction": 0,
          "rejectedPrediction": 0
        },
```

However, I strongly question whether this number is correct/if the tests are taking this into account. I've tried with multiple providers (Deepseek, Gemini, Claude, Kimi...) and still getting this same issue

The issue is especially prevalent with google/gemini-2.5-pro, which ALWAYS reasons heavily before doing a function call and so I get a 100% failure rate when trying to validate function calls 

for example,

```yaml
 - id: openrouter:google/gemini-2.5-pro
    config:
      temperature: 0.1
      showThinking: false
      top_p: 0.25
      max_tokens: 1024
      tool_choice: auto
      apiKeyEnvar: OPENROUTER_API_KEY
      apiBaseUrl: https://openrouter.ai/api/v1
      tools: *tool_array
```

You can see the model has the _intention_ of doing a function call but all we get in promptfoo is Gemini's chain of thought reasoning. Note that I've specified `showThinking: false` and verified that in my promptfooconfig.yaml that showThinking is indeed false under all my providers

<img width="755" height="306" alt="Image" src="https://github.com/user-attachments/assets/7a192ef3-823a-49c1-b861-a52bad003b61" />

<img width="667" height="569" alt="Image" src="https://github.com/user-attachments/assets/33274efe-a889-4aec-941e-3c70f740b1c3" />

I considered that the issue may be that the tests are timing out before getting to the function call step, but even when I change the timeout time to some extreme number it doesn't affect the performance 

You can see that we still have this issue, but much less, when using minimal reasoning, such as with this provider:

```
- id: openrouter:gpt-5-mini
    config:
      temperature: 0.1
      showThinking: false
      top_p: 0.25
      response_format: text
      reasoning_effort: minimal
      tool_choice: auto
      apiKeyEnvar: OPENROUTER_API_KEY
      apiBaseUrl: https://openrouter.ai/api/v1
      tools: *tool_array
```

<img width="725" height="322" alt="Image" src="https://github.com/user-attachments/assets/97f5985c-7d30-4b9d-bd8b-9c5851bf1d8b" />

Any idea why this is happening? shouldn't showThinking completely hide the thinking, and instead show _nothing_ if there was no tool call? Instead we're getting _just_ the thinking and _no tool call_

I would say adding a `response_format` is not ideal in this situation because I have test scenarios that asert that a function call happens, and test scenarios that assert that a function call _did not_ happen

@jahabeebs Thanks for the details- I will look into this and get a fix out soon 👀 

> [@jahabeebs](https://github.com/jahabeebs) Thanks for the details- I will look into this and get a fix out soon 👀

@faizanminhas I asked codex to solve it three different ways--it might be of some help

https://chatgpt.com/s/cd_68a69147cff8819190f95727996c0a97

@faizanminhas Hey Faizan,

Thanks a lot for adding that OpenRouter thinking fix yesterday

I tested the fix and it resolves the issue for _certain_ providers

For OpenAI providers this issue is now resolved

Example:

- id: openrouter:gpt-5-mini
    config:
      temperature: 0.1
      showThinking: false
      top_p: 0.25
      response_format: text
      reasoning_effort: high
      tool_choice: auto
      apiKeyEnvar: OPENROUTER_API_KEY
      apiBaseUrl: https://openrouter.ai/api/v1
      tools: *illa_tools


I'm running 100 evals and not one has the thinking bug anymore--either we get the text response or a function call when showThinking is false

However, Qwen is still broken

Example:

- id: openrouter:qwen/qwen3-235b-a22b-thinking-2507
    config:
      temperature: 0.1
      showThinking: false
      top_p: 0.25
      max_tokens: 1024
      tool_choice: auto
      apiKeyEnvar: OPENROUTER_API_KEY
      apiBaseUrl: https://openrouter.ai/api/v1
      tools: *illa_tools


This leads to a large discrepancy in error rates in evals

<img width="998" height="202" alt="Image" src="https://github.com/user-attachments/assets/71e0c8d3-10f9-4840-b5c1-847d6aa5a42f" />