Repository: promptfoo/promptfoo
Issue: --filter-failing not working
Root cause: Validation Gap

**Describe the bug**
When I run `npx promptfoo eval --filter-failing "eval-idT05:50:55" -c ./my-config.yaml`, it never runs any tests:

```
=========================================================================
✔ Evaluation complete. ID: eval-ktL-2025-05-20T00:19:22

» Run promptfoo view to use the local web viewer
» Do you want to share this with your team? Sign up for free at https://promptfoo.app
» This project needs your feedback. What's one thing we can improve? https://forms.gle/YFLgTe1dKJKNSCsU7
=========================================================================
Successes: 0
Failures: 0
Errors: 0
Done.
```

I also tried using the output.json instead of the eval id.

**Expected behavior**
In the past, I managed to get this working (months ago), it was very handy to debug certain prompt changes.
 
**System information:**

Promptfoo Debug Information
```
{
  "version": "0.112.4",
  "platform": {
    "os": "darwin",
    "release": "24.3.0",
    "arch": "arm64",
    "nodeVersion": "v22.10.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```


Hi, i tested out this feature, it still does work but it is very finicky.

It Specifically it compares vars and providers for the testCases in your config to the results in the test. If they do not match exactly included in the suite.

Using this query on the sqlite db you can see what the config is supposed to match
```SQL
SELECT
  json_extract(test_case, '$.vars') AS vars,
  json_extract(test_case, '$.provider') AS provider
FROM eval_results
WHERE eval_id = '<YOUR_ID_HERE>';
```

This is the sample i used to test the feature:

First round it will have one failing, if i run with filter-failing only one test will be executed.
```YML
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: "Groq Provider Demo"

prompts:
  - "say {{item}}"

providers:
  - id: groq:llama-3.3-70b-versatile
    config:
      apiKey: abc

tests:
  - vars:
      item: bananas
    assert:
      - type: contains
        value: "banana"
  - vars:
      item: apples
    assert:
      - type: contains
        value: "banana"
```


This is the code that does the matching https://github.com/promptfoo/promptfoo/blob/main/src/util/index.ts#L363-L369
this is the test for the feature https://github.com/promptfoo/promptfoo/blob/main/test/commands/eval/filterTests.test.ts.


Without access to your environment or a failing tests i cannot help any further.



I can confirm your test file works for me, and some of my test files work.
I think the problem comes from the use of a custom javascript provider. Maybe the providerId is not being picked up correctly.

```
module.exports = class LangChainProvider {
  providerId = 'langchain';
  config;

  constructor(options) {
    this.providerId = options.id || 'langchain';
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

async callApi(prompt, context, options) {
  ... redacted
  return {
            output: result
        };
}
```

Assuming its provider causing the issue and not variables.  My testing produced that the provider file path is being used as the id. so if you changed file path for example this would cause it to not match

https://gist.github.com/billybonks/987cbc432f8ac4e7a8b1a7b81f6f119f

to use my example:
1. save all the files to a folder
2. run the first one with `-c promptfooconfig-first-run.yml`
3. run the second with `--filter-failing '<LAST_ID>' `-c promptfooconfig-filter-second-run.yml`

And you will see that failing test is not rerun.

you can run the query i showed above to find the path used for the provider in the original case.

It will only filter on provider if you specify custom providers at a test level as i have in my example.