Repository: promptfoo/promptfoo
Issue: Description Field in Test.yaml Causes Assertions to Pass Incorrectly
Root cause: Documentation Deficiency

When defining assertions in a separate yaml file, the `description` key is useful as, without it, the `input` and `assert` fields do not seem to appear in the output of the eval command. However, including the `description` field causes assertions to pass when they should fail. For example, I have the following config:

```
description: "Test PromptFoo Setup"
prompts:
  - |-
    Respond to me with JSON in the the following format
    {
      "message": "Hello"
    }
providers: [openai:gpt-3.5-turbo-0125]
tests: test.yaml
```
And a test.yaml file defined like so:

```
- description: "Returns json"
  vars:
    input: "Hello!"

    assert:
      - type: is-json
        value: "true"
```

when I run `yarn promptfoo eval -c ./<path_to_config>/promptfooconfig.yaml --no-cache` I see no errors (expected) 
<img width="1434" alt="Screenshot 2024-05-09 at 12 06 47 PM" src="https://github.com/promptfoo/promptfoo/assets/40777383/b75aaef4-1bd7-4025-8ebe-b4b4926bacf8">


however, if I change the assertion to expect in-valid json (`value: "false"`) I still see no errors (**not expected**). 
<img width="1428" alt="Screenshot 2024-05-09 at 12 07 10 PM" src="https://github.com/promptfoo/promptfoo/assets/40777383/d81e6b47-c2e8-4d12-b7bf-8a9b1717c4ab">


By removing the `description` key, the assertions behave as expected, but the output is less desirable. This test.yaml
 
```
- vars:
  input: "Hello!"

  assert:
    - type: is-json
      value: "false"
```

produces the following, expected output:
<img width="1438" alt="Screenshot 2024-05-09 at 12 09 50 PM" src="https://github.com/promptfoo/promptfoo/assets/40777383/e226e94b-917b-4426-9004-99d600f63cd4">

Is the expectation that the description key should be supported?

 Big fan of this tool and any insight you could provide would be most appreciated!

currently running v.0.55.0, as a project dependency (not a global dependency)



Hi @SimonPringleWallace,

Looks like your indentation is off in both test.yaml examples.

In this one, you need to unindent `assert`.  It should not be under `vars` (that is why `assert` shows up as a var in the CLI table output)
```yaml
- description: "Returns json"
  vars:
    input: "Hello!"

    assert:
      - type: is-json
        value: "true"
```

In this one, `input` needs to be indented:
```yaml
- vars:
  input: "Hello!"

  assert:
    - type: is-json
      value: "false"
```

Well that is a bit embarrassing. Thanks for the correction there @typpo, works like a charm 🚀 

No worries, it happens!

On Fri, May 10, 2024 at 7:19 AM SimonPringleWallace <
***@***.***> wrote:

> Well that is a bit embarrassing. Thanks for the correction there @typpo
> <https://github.com/typpo>, works like a charm 🚀
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/767#issuecomment-2104695491>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJXM6JHTZL37LDMRWVDZBTJQZAVCNFSM6AAAAABHPCZILCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBUGY4TKNBZGE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>