Repository: promptfoo/promptfoo
Issue: Importing test cases from a python script or API endpoint
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
I want to include production examples into my test cases. Due to concerns around data security and privacy, including these examples into our github repo is not an option. Furthermore we already have an observability system in place that can capture and flag examples that we'd like to include into our eval set and we'd like a way to integrate such examples into the evals we run through promptfoo.

**Describe the solution you'd like**
In the YAML configuration file would like to be able to specify tests from either a python script or a URL host. I'd expect promptfoo to call the endpoint/script which would return a JSON formatted response with the specifications of the tests to include.

This way, I can deploy promptfoo in a self-hosted environment and give it access to run evals on production examples without exposing them in our codebase.

**Describe alternatives you've considered**
I've considered a workaround that would involve running a script that would render the test cases imported from my production environment as a local file and having my yaml configuration include the tests from that file.

This solution does not seem as maintainable and opens the risk of leaving sensitive information on the local filesystem.

**Additional context**
Add any other context or screenshots about the feature request here.


Thanks for the feature request! This functionality already exists in promptfoo through Python (and JavaScript) test generators.

You can create a Python script that fetches test cases from your observability system or any API endpoint:

```yaml
# promptfooconfig.yaml
tests:
  - path: file://fetch_production_tests.py:get_flagged_examples
    config:
      api_endpoint: 'https://your-observability-system.internal/api/flagged-examples'
      # Environment variables work too
      api_token: '${OBSERVABILITY_API_TOKEN}'
```

```python
# fetch_production_tests.py
import requests

def get_flagged_examples(config):
    """Fetch production examples from observability system."""
    response = requests.get(
        config['api_endpoint'],
        headers={'Authorization': f'Bearer {config.get("api_token", "")}'}
    )
    response.raise_for_status()

    # Transform API response into promptfoo test cases
    test_cases = []
    for item in response.json():
        test_cases.append({
            'vars': {
                'input': item['user_input'],
                'context': item.get('context', '')
            },
            'assert': [
                {'type': 'llm-rubric', 'value': 'Response is helpful and accurate'}
            ],
            'metadata': {
                'source': 'production',
                'flagged_reason': item.get('flag_reason')
            }
        })

    return test_cases
```

This approach addresses all your requirements:

- **Data security**: Production examples stay in your observability system, never committed to git
- **Self-hosted friendly**: Works in any environment that can reach your internal APIs
- **Flexible integration**: Python can connect to any data source (APIs, databases, S3, etc.)
- **Configurable**: Pass different endpoints, credentials, or filters via the `config` object

**Documentation & Examples:**
- [Test Case Configuration - Python section](https://www.promptfoo.dev/docs/configuration/test-cases/#python)
- [Python Test Cases Example](https://github.com/promptfoo/promptfoo/tree/main/examples/python-test-cases)

Closing as this is already supported. Feel free to reopen if you have additional requirements not covered by this approach!