Repository: promptfoo/promptfoo
Issue: Error in manually evaluating a test case
Root cause: Validation Gap

I'm using `promptfoo view [PATH]` to look at a set of specific results, where `[PATH]` contains a folder `output/` with the following files generated from a previous `promptfoo eval`:

```bash
latest.csv  latest.html  latest.json  latest.txt  latest.yaml
```

Basically, we save the results using all available formats. I can correctly visualize the results in the web browser. However, if I manually set a failed test as "passed" using the 👍🏻 button or if I try to set the test score, I get the following error from `promptfoo view`:

```bash
$ promptfoo view outputs/mixtral-8x7b-instruct-v0.1-q5_1/
body-parser deprecated undefined extended: provide extended option ../../../lib/node_modules/promptfoo/dist/src/web/server.js:33:31
Server running at http://localhost:15500 and monitoring for new evals.
Open URL in browser? (y/N): 
Press Ctrl+C to stop the server
Failed to update results in outputs/mixtral-8x7b-instruct-v0.1-q5_1/output/undefined:
Error: ENOENT: no such file or directory, open 'outputs/mixtral-8x7b-instruct-v0.1-q5_1/output/undefined'
```

It would be really great if we can update the `json` file with the manually set scores or passed/failed status.

> It would be really great if we can update the json file with the manually set scores or passed/failed status.

Interesting.  This is supposed to happen, but looks like there's maybe a bug when you're using a non-default location for `promptfoo view`.  I'll look into this and write a fix.

Hi @miltondp, I've had a hard time reproducing this error. Haven't forgotten about it, but if there are any more details on how you get it into this state I'd appreciate it.

Hi @typpo . Sorry for the delay. I'm still having this issue with both version 0.35 and the latest 0.39. I prepared this test case and I was able to reproduce the error. The json file included was generated using promptfoo v0.35, but it fails to save the manual assessment with v0.35 and v0.39.

Let me know if this helps.

[test_case_392.zip](https://github.com/promptfoo/promptfoo/files/14129158/test_case_392.zip)


Hey @miltondp, finally figured this one out.  This is happening because you only have a `latest.json` file in the outputs folder, whereas it expects a `latest.json` as well as a file named similarly to `eval-2024-02-06T16-04-28.517Z.json` (there's some weirdness here where `latest.json` used to be a symlink, but has turned into a copy for Windows compatibility reasons...)

My question for you is: do you know how you got into a state with just `latest.json` and no `eval-*` files?  I can fix the breakage, but I'd prefer to understand the root cause.

Hey @typpo, sorry for the delay. We only have a `latest.json` file (named like this because that's what `promptfoo view` expects) because we use `promptfoo eval` to run evaluations across different contexts, where we save the outputs as `json`, `csv`, etc (all the supported output files). So, for context 1 (let's say we are evaluating prompts for `llama2` to revise an abstract of a journal article), we run `promptfoo run` (from the command line) and save the results. Then, for context 2 (same but now we use the `mixtral` model), we do the same. So we end up with a set of results across different models for one scenario (evaluating prompts to revise an abstract). Then we want to visualize one at a time with `prompt view`. We chose this setup because we have many combinations of scenarios and models/contexts.

A related question: how are manual evaluations saved in the `json` file? It would be great to keep all the other scores and have a separate field for the manual/human evaluations. Also, I see that the `json` file is the main file, and all the rest (like the `csv`) are generated from it. If I update the `json` file with manual evaluations, is it possible to convert the `json` file to a `csv` file so I get an updated one? I ask this because we need to manually read the results to generate our plots/analysis, and doing so with a `csv` would be much easier.

Thank you so much for this project!

Hey @miltondp!

Thanks so much for the detailed bug report and for providing that reproduction case, it was super helpful in tracking this down. And apologies this one sat open for so long!

**Good news: this issue has been fixed.**

The bug you encountered was in our old file-based storage system. When you had only a `latest.json` file without the corresponding `eval-*.json` file, the system couldn't find the right file to update and ended up with `undefined` in the path, hence the error you saw.

Since then, we've completely migrated from file-based JSON storage to a SQLite database. Manual evaluations (thumbs up/down, score adjustments) now save directly to the database rather than trying to patch JSON files on disk. This architectural change eliminated the root cause entirely.

To answer your other question from the thread: manual evaluations are now stored in the database alongside the original results. If you need to export results including manual annotations, you can use `promptfoo export` to get a JSON dump.

If you're still running into any issues with manual evaluations on a recent version, please let us know and we'll reopen this. But I'm confident this should be working smoothly now.

Thanks again for taking the time to report this and for your patience! We really appreciate community members like you helping make promptfoo better.