Repository: promptfoo/promptfoo
Issue: read temp response json file failed with latest version
Root cause: Resource Mishandling

**Describe the bug**
I upgrade to the latest version from 0.118.17 and found my existing eval cannot run successfully. I have checked the log, it seems the new version will try to read some temp json file when trying to use similarity or llm-rubric assert.

below is the error log:
```
Error: ENOENT: no such file or directory, open 'C:\Users\fma\AppData\Local\Temp\promptfoo-worker-resp-1761880046684-948141998bb528.json'

Error: ENOENT: no such file or directory, open 'C:\Users\fma\AppData\Local\Temp\promptfoo-worker-resp-1761880046684-948141998bb528.json'
    at Object.readFileSync (node:fs:440:20)
    at PythonWorker.executeCall (C:\Users\fma\AppData\Roaming\nvm\v24.11.0\node_modules\promptfoo\dist\src\python\worker.js:119:49)
    at async PythonWorker.call (C:\Users\fma\AppData\Roaming\nvm\v24.11.0\node_modules\promptfoo\dist\src\python\worker.js:87:20)
    at async PythonProvider.executePythonScript (C:\Users\fma\AppData\Roaming\nvm\v24.11.0\node_modules\promptfoo\dist\src\providers\pythonCompletion.js:169:22)
    at async matchesSimilarity (C:\Users\fma\AppData\Roaming\nvm\v24.11.0\node_modules\promptfoo\dist\src\matchers.js:229:35)
    at async handleSimilar (C:\Users\fma\AppData\Roaming\nvm\v24.11.0\node_modules\promptfoo\dist\src\assertions\similar.js:36:17)
    at async runAssertion (C:\Users\fma\AppData\Roaming\nvm\v24.11.0\node_modules\promptfoo\dist\src\assertions\index.js:362:24)
    at async C:\Users\fma\AppData\Roaming\nvm\v24.11.0\node_modules\promptfoo\dist\src\assertions\index.js:409:24
```
 

**To Reproduce**
use below eval with latest version promptfoo
[tools_call.zip](https://github.com/user-attachments/files/23248841/tools_call.zip)

Please note that I am using local LM studio model to do the eval
**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version:

**Additional context**
Add any other context about the problem here.


Hi @mafei1982,

I've identified and fixed the root cause of this issue in PR #6073. This was caused by a recent PR https://github.com/promptfoo/promptfoo/pull/5968 which completely redid our python integration. The ENOENT error was caused by a bug in the Python worker's error handling (introduced in v0.119.0 with the persistent worker feature). When the Python worker encountered certain errors before writing the response file, it would signal completion to Node.js without creating the file, causing Node.js to look for a file that didn't exist.

Your code is correct - both `llm_embedding.py` and `llm_rubric_test.py` have the proper function signatures. The bug would trigger in scenarios like:

- Network errors calling your LM Studio server
- Missing Python dependencies
- File system permissions issues (especially on Windows)
- Any exception before the response file could be written

In #6073 I've updated `src/python/persistent_wrapper.py` to ensure an error response file is always written before signaling completion, even when exceptions occur. This means:
- You'll get meaningful error messages (e.g., "Connection refused to localhost:1234")
- No more confusing ENOENT errors
- Proper error propagation from Python to promptfoo

Since you're on Windows and I've tested on macOS, I'd appreciate if you could test the fix. If not, I will get it in the latest release ~ tomorrow.  You can also run your eval with `--no-cache` to ensure fresh execution and please let me know if you still see any ENOENT errors. 

The fix should work on Windows, but the persistent worker has some Windows-specific file system timing quirks that I want to make sure are fully resolved.

If you need a workaround immediately, you can downgrade to v0.118.17:

```bash
npm install promptfoo@0.118.17
```

That version doesn't have the persistent worker feature, so it won't encounter this bug.

Let me know if this resolves the issue once it's merged. Thanks so much and sorry for breaking it! If you test it and it works, please send me an email (michael @ promptfoo.dev) for some free promptfoo swag.

> Hi [@mafei1982](https://github.com/mafei1982),
> 
> I've identified and fixed the root cause of this issue in PR [#6073](https://github.com/promptfoo/promptfoo/pull/6073). This was caused by a recent PR [#5968](https://github.com/promptfoo/promptfoo/pull/5968) which completely redid our python integration. The ENOENT error was caused by a bug in the Python worker's error handling (introduced in v0.119.0 with the persistent worker feature). When the Python worker encountered certain errors before writing the response file, it would signal completion to Node.js without creating the file, causing Node.js to look for a file that didn't exist.
> 
> Your code is correct - both `llm_embedding.py` and `llm_rubric_test.py` have the proper function signatures. The bug would trigger in scenarios like:
> 
> * Network errors calling your LM Studio server
> * Missing Python dependencies
> * File system permissions issues (especially on Windows)
> * Any exception before the response file could be written
> 
> In [#6073](https://github.com/promptfoo/promptfoo/pull/6073) I've updated `src/python/persistent_wrapper.py` to ensure an error response file is always written before signaling completion, even when exceptions occur. This means:
> 
> * You'll get meaningful error messages (e.g., "Connection refused to localhost:1234")
> * No more confusing ENOENT errors
> * Proper error propagation from Python to promptfoo
> 
> Since you're on Windows and I've tested on macOS, I'd appreciate if you could test the fix. If not, I will get it in the latest release ~ tomorrow. You can also run your eval with `--no-cache` to ensure fresh execution and please let me know if you still see any ENOENT errors.
> 
> The fix should work on Windows, but the persistent worker has some Windows-specific file system timing quirks that I want to make sure are fully resolved.
> 
> If you need a workaround immediately, you can downgrade to v0.118.17:
> 
> npm install promptfoo@0.118.17
> That version doesn't have the persistent worker feature, so it won't encounter this bug.
> 
> Let me know if this resolves the issue once it's merged. Thanks so much and sorry for breaking it! If you test it and it works, please send me an email (michael @ promptfoo.dev) for some free promptfoo swag.

Thank u very much for the quick response. I have already downgraded to 118.17. So I will test it with release tomorrow

Hi @mldangelo, I also found another issue that I only defined get_embedding_api function in the llm_embedding.py. But I get an error that I need to define function get_api when using latest release. but with embedding llm model,  I think I do not need to define get_api.  

@mldangelo I checked latest release and the issue still exists