Repository: promptfoo/promptfoo
Issue: Logging from python provider doesn't show up when running eval
Root cause: Architectural Constraint

I tried logging with `print` when running an eval with a custom python provider. However, the logs didn't show up
```python
# my_script.py
import json

def call_api(prompt, options, context):
    # The 'options' parameter contains additional configuration for the API call.
    print('make sure it's running')
    return result

def call_embedding_api(prompt):
    # Returns ProviderEmbeddingResponse
    pass

def call_classification_api(prompt):
    # Returns ProviderClassificationResponse
    pass
```

Think it has to do with this library https://www.npmjs.com/package/python-shell#exchanging-data-between-node-and-python, Maybe we should consider changing the mode to text instead of binary

https://github.com/promptfoo/promptfoo/blob/90e5073935a9cb5b4ea8d7746ccfa338258ff226/src/python/pythonUtils.ts#L107-L112

Hey @albertlieyingadrian, thanks for flagging this! I created a quick prototype to stream logs from Python: https://github.com/promptfoo/promptfoo/compare/feat/python-logging?expand=1

To test it out, grab the feat/python-logging branch and run:
`promptfoo eval -c examples/python-provider/promptfooconfig.yaml`

You should see your Python logs in real-time now. 😊

This is just a first pass - feel free to take it for a spin and let me know what you think. If you're up for it, you're welcome to run with this branch and improve it further (I am afraid that switching from binary to text will ruin unicode decoding). Otherwise, I'll keep playing with it on the backburner. 

Thanks again for flagging this - your contributions are always super helpful!

Thanks. This will be huge because right now if there's an error, there's no way for us to debug the custom provider's code @mldangelo 

> (I am afraid that switching from binary to text will ruin unicode decoding

@mldangelo Will this be covered by existing unit tests? Thinking of trying to switch it and see what breaks

Ok. We should land this branch. 

I have never been able to reproduce the unicode issues (see #1604 and #1476), so I have not written unit tests.

Before it lands, we should:
1. Change the name of the logger (PythonScript -> promptfoo or make it configurable)
2. Increase test coverage
3. Write some docs and modify an example to take advantage of logging
4. try to get it working in binary mode again (as opposed to text mode). 

Let me know if you want to tackle this or if you'd prefer I take a look. Your insights have are super helpful! Thank you much for all your contributions this week. 

> if you'd prefer I take a look

Think this will be the best since you have more context @mldangelo 

I'll get something out today! 

Thanks! @mldangelo Can't wait for this and let me know how I can help to speed this up :)

Any updates here? @mldangelo 

Hey @albertlieyingadrian, apologies for the delay! I just landed #1818, and it turned out to be much simpler than what we initially discussed. It'll ship in tomorrow's release. To test it out, you can set LOG_LEVEL=debug or run your eval with the --verbose flag to see the logs. Thanks again for your patience, and let me know if you encounter any issues!

Hey @albertlieyingadrian, thanks for your patience! We just released the fix in [0.91.2](https://github.com/promptfoo/promptfoo/releases/tag/0.91.2). Please check it out and feel free to comment or reopen this issue if you encounter any problems. Thanks again for flagging this and for all your contributions!