Repository: promptfoo/promptfoo
Issue: promptfoo return reason too simple
Root cause: Documentation Deficiency

**Describe the bug**
After updating promptfoo to version 0.117.5 and passing the evaluation, the only reason given for the assessment is "Grading passed". For specific details, please refer to the screenshot.

**To Reproduce**
1. use promptfoo version is [0.117.5](https://www.npmjs.com/package/promptfoo/v/0.117.5)
2. prepare the test.csv and promptfooconfig.yaml
3. run promptfoo eval -o result.json --no-cache

**Expected behavior**
Before updating promptfoo 0.117.5, regardless of whether the assessment was passed or not, we could obtain the specific reasons for the assessment.

**Screenshots**

<img width="2840" height="1230" alt="Image" src="https://github.com/user-attachments/assets/3eca54d0-e9cd-40d7-8cbd-8bade05f023d" />


- Promptfoo version:
0.117.5



@zachma-820 can you please include a promptfooconfig I can use to replicate the issue? 

@mldangelo Thanks
[promptfooconfig.yaml](https://github.com/user-attachments/files/23491134/promptfooconfig.yaml)
[test.csv](https://github.com/user-attachments/files/23491135/test.csv)

In test.csv， I put the llm return to column __providerOutput 

RUN:
promptfoo eval -o result.json --no-cache

Hi @zachma-820, thank you for reporting this! I've investigated the issue and believe I've identified the root cause.

The "Grading passed" message is a **fallback behavior** that only appears when the LLM doesn't return a `reason` field in its JSON response (or returns an empty/null reason). Looking at your configuration, I suspect your LLM (gemini-2.5-pro through the Alibaba endpoint) is returning JSON without a detailed `reason` field, something like:

```json
{"pass": true, "score": 1}
```

instead of:

```json
{"pass": true, "score": 1, "reason": "详细的评估理由..."}
```

To confirm, please run your evaluation with debug logging enabled to see exactly what the LLM is returning:

```bash
PROMPTFOO_LOG_LEVEL=debug promptfoo eval -o result.json --no-cache
```

Look for log lines that show the parsed LLM response. This will reveal whether the `reason` field is missing or empty.

## Recommended Solution

The issue is likely with your rubric prompt not being explicit enough about requiring detailed reasons. Try updating your `rubricPrompt` to be more explicit:

```yaml
defaultTest:
  options:
    rubricPrompt:
      - role: system
        content: |-
          你是一个专业的输出评估专家。你必须返回包含以下三个字段的JSON对象：

          {
            "reason": "必填！详细的中文分析理由（必须至少30字，说明评估的具体依据）",
            "pass": true或false,
            "score": 0到100的数值
          }

          重要规则：
          1. reason字段绝对不能为空或省略
          2. reason必须包含具体的分析内容，不能只写"通过"或"评估通过"等简单短语
          3. reason必须说明你观察到输出中的哪些具体内容导致了你的判断

          示例：

          <输出>你好世界</输出>
          <评估标准>内容包含问候语</评估标准>
          {"reason": "输出中包含了'你好'这个问候语，符合评估标准的要求", "pass": true, "score": 100}

          <输出>船长，准备战斗！</输出>
          <评估标准>不使用海盗式语言</评估标准>
          {"reason": "输出使用了'船长'等海盗风格的表达方式，不符合评估标准", "pass": false, "score": 0}
      - role: user
        content: |-
          <待评估输出>{{ output }}</待评估输出>
          <评估标准>{{ rubric }}</评估标准>
          <参考输出>{{ rawData }}</参考输出>

          请严格按照上述JSON格式返回评估结果，确保reason字段包含详细的中文分析。
```

## Why This Happens

The fallback logic is in `src/matchers.ts`:

```typescript
const reason = parsed.reason || (pass ? 'Grading passed' : `Score ${score} below threshold ${threshold}`);
```

This is intentional - it prevents the evaluation from crashing when LLMs return malformed responses. However, it means if your LLM doesn't provide a `reason`, you'll see the generic fallback message. If improving the prompt doesn't work, test with a different model to isolate if it's model-specific.

Please try the improved rubric prompt above and let me know if you still see "Grading passed" as the only reason. If the issue persists after making the prompt more explicit, please share the debug logs so we can see exactly what the LLM is returning.

Closing for now, but please let me know how it goes.