Repository: promptfoo/promptfoo
Issue: Request: extending the `equals` assertion to work with JSON
Root cause: Unimplemented Feature Gap

Hi, thank you very much for the wonderful software!

I am working on a task to output information contained in images in JSON format using a multimodal LLM. I would like to evaluate whether the output JSON matches the expected JSON.

Currently, I am placing test values in `vars`.

```yaml
- vars:
    ...
    expected: {}  // expected json here
  assert: 
    - type: python
      value: context["vars"]["expected"] == json.loads(output)
```

Preferably, I'd like to write it like this:

```yaml
- vars:
    ...
  assert:
    - type: json-equals
      value: {}  // expected json or target file here
```

Here are the potential benefits of implementing this:

1. **Easier to use for promptfoo beginners**
When I first tried to handle this using `contains-json` and `is-json`, it took me a little while to realize it wasn't working as expected. However, this can also be resolved through better documentation.

2. **Ability to specify files**
I'm planning to using slightly complex JSON as test values. In such cases, writing JSON directly in the YAML can be cumbersome, so having the ability to specify a file would be convenient.

If there are no issues with this proposal, I'm thinking of contributing it myself!

Great idea.  It seems like we could solve this problem by making the existing `equals` assertion work for JSON objects.  What do you think? https://github.com/promptfoo/promptfoo/blob/main/src/assertions.ts#L297

I see, so I can pass a file for the `value` of `equals` as well. I didn't know that. Thanks for letting me know.

I read through the code. If a file is specified, `renderedValue` is an Object and `outputString` is a String, so their types are different and it would always fail, wouldn't it?
Even if this assumption is wrong, if it's a comparison between Objects, using `==` would compare references and return always false. And if it's a comparison between Strings, it wouldn't work if the order is swapped.

I became interested, so I'll give it a try and get back to you.

@typpo I tried validating some patterns with the `equals` assertion. The model is set up to always return `{"foo": 1, "bar": 2, "bazz": 3}`.

1. Providing a string to values
```yaml
assert:  
  - type: equals  
    value: '{"foo": 1, "bar": 2, "bazz": 3}'
```

In this case, the assertion passed. However, as expected, if the order of Keys and Values is swapped, it failed.

```yaml
assert:  
  - type: equals  
    value: '{"foo": 1, "bazz": 3, "bar": 2}'
```

Since the values are the same as JSON, I want this case to pass.

2. Providing via file
```yaml
assert:  
  - type: equals  
    value: "file://foobar.json"
```

In this case it failed with the error message below.
> `[FAIL] Expected output "[object Object]" to equal "{"foo": 1, "bar": 2}"`

As I commented above, this is because `renderedValue` is an Object and `outputString` is a String.

Based on the above, it seems difficult with the `equals` assertion, so it would be good to have `json-equals` .

What I am suggesting is that we modify `equals` to support deep object comparison (which is does not currently support), rather than creating a separate assertion type.

Oh, I made a bit of a misunderstanding. I also agree with implementing it in `equals`.

No problem! Let me know if you'd like to take this one. Otherwise I don't mind doing it :)

I would like to take on this task! If it seems difficult, I will contact you again.