Repository: promptfoo/promptfoo
Issue: Importing config vars from a .csv file
Root cause: Documentation Deficiency

Hello, having had a look through the documentation ([Import vars from separate files](https://promptfoo.dev/docs/configuration/guide)) and examples, it appears as though importing vars from external files is limited to yaml, txt, Python and Javascript files.

Are there any plans to allow import from a csv file, where each column header is a variable name, and each row is one single complete configuration?  We feel this would be particularly helpful when composing "scenarios" where you have complex setup config to apply to each test.

Hey @alexclaydon, is it possible this [import from CSV](https://promptfoo.dev/docs/configuration/parameters/#import-from-csv) functionality does what you want?

It works at the `tests` level rather than `vars`, but you can include it in your promptfoo config like this:

```
tests: my_tests.csv
```

And the CSV includes a bunch of vars mapped to columns.

Thanks Ian!

First of all, we're loving promptfoo and are going all in on it - thanks for all of your excellent work!

Having thought about this issue for a few days and discussed with a colleague, we realised that I should have framed it differently.  Please allow me to provide some additional context on the use-case to flesh out the issue as we see it.  It's always possible that we've misunderstood the paradigm.

We're making extensive use of "scenarios" in our setup, like so:

```yaml
description: ...
providers: ...
prompts: file://prompt_templates.txt  # prompt template containing nunjucks tags for `topic` and `description`

scenarios:
  - config:
    - vars:
        topic: "Liabilities for Business Losses"
        description: file://case01.txt
    - vars:
        topic: "Liabilities for Data Losses"
        description: file://case02.txt
    - vars:
        topic: "Liabilities for Other Losses"
        description: file://case03.txt

    tests: './data.csv'  # contains 4 columns: `text` (str), `definitions` (str), `data_loss` (bool), `business_loss` (bool)
```

The crux of the issue for us is that, in scenarios, it doesn't seem to be possible to specify assertions under the `config` section to be applied (ceteris paribus) to each row - they must instead be in-lined row-by-row in the `.csv` data file itself.

Accordingly, if we want to check, for example, whether some output matches the value of one of the bool columns in our data spreadsheet, we need to go down each row and change the name of that column to `__expected` and its value from "TRUE" to (for example) "equals: TRUE".  Given that our data spreadsheets are often the main interface between technical and non-technical teams, that's quite tricky for us.

Ideally, we'd prefer to write a one-line "wrapper" assertion, leveraging `vars` set out in the `.csv` file (and elsewhere), and just drop it into the `config` section instead, where it would then be applied on a row by row basis.

I hope that all makes sense and would really appreciate any guidance you might be able to provide!



Just had an additional thought on your response above which might clear things up for us a bit.

You mentioned that "the CSV includes a bunch of vars mapped to columns."  Does that imply that I can access column names / vars from _within the config file_ in an assert immediately following `tests: my_tests.csv`, like so (or something similar)?

```yaml
description: 'Extraction'

prompts: ['{{ clause }}']

providers:
  - id: 'python:../shims/extract.py'
    label: 'Extraction'
    config:
      evaluateOptions.maxConcurrency: 1
      evaluateOptions.showProgressBar: true

tests:
  - '../testdata/data.csv'
  - assert:
    - type: icontains-any
      value: '{{ entities }}'  # where the `.csv` file contains at least `clause` and `entities` columns
```

I've been trying to get that working today, but wasn't able - I also had a look through the `examples` directory, but couldn't find anything on-point.

If this kind of thing is possible, that would enable us to conveniently "wrap" asserts around row data as I described above - so a complete solution!

I think the right construction here is:

```yaml
defaultTest:
  assert:
    - type: icontains-any
      value: '{{ entities }}'

tests: '../testdata/data.csv'
```

The `defaultTest` object will apply to all test cases, and _should_ be able to pick up columns from your CSV.

Thank you Ian!

The confirmation that this is supposed to work is very helpful.  I had actually tried this approach, but without success.  I think I now know why: it appears to relate to the way arrays are handled.

Since this particular assert is `icontains-any`, I had set up my `.yaml` file exactly as you have it above. I then tried setting up the `entities` column in my `.csv` file a few different ways: (i) including / excluding square brackets (`[]`), (ii) including / excluding single and double quote marks around each string in the array, and (iii) trying both commas and semi-colons as delimiters.  All of those approaches failed with a message to the effect that this particular assert requires an array.

In addition, I tried enclosing the value of the `value` field in my `.yaml` file in square brackets, as follows:

```yaml
defaultTest:
  assert:
    - type: icontains-any
      value: ['{{ entities }}']

tests: '../testdata/data.csv'
```

That seemed to resolve the above error about the assert requiring an array, but also resulted in the actual value of the `entities` column being returned as a single string (enclosed in an array), no matter what I tried.

Could you advise on the correct approach here?  Unless I missed it, there seems to be a bit of a gap in the documentation here.  I would be very happy to submit a documentation PR if we could get to the bottom of this and if that would be helpful?

Cheers

Ok, I see the problem!  I went through the whole troubleshooting and debugging process (e.g. using `__expected`, wrapping in brackets) all of which you already explained in great detail :)  

I think the best way to fix this is add comma-delimited string support for `icontains-any` and similar assertion types.  That way you'll be able to do:

```yaml
defaultTest:
  assert:
    - type: icontains-any
      value: '{{ entities }}'

tests: '../testdata/data.csv'
```
and you can include an `entities` column in your CSV, and it will work as intended.

#755 implements this (although it naively splits on any comma, and won't respect CSV-style quotes like `"hello, world"`)

Perfect!  Thanks so much for this, really appreciate the attention.  That's the final piece of the puzzle for us on pushing Promptfoo to adoption across our whole team, incredibly helpful!

Sorry for the delay here @alexclaydon - it's merged and will be in the next release!

Not at all - we really appreciate you talking the time on this!

@typpo  thank you for this explanation, it helped me to be able to use csv files to test on large datasets.
It would be nice if it would be possible to display the values of "entities" in the UI instead of just: {{entites}}, look at the image below:

<img width="1261" alt="Screenshot 2024-05-15 at 10 45 09" src="https://github.com/promptfoo/promptfoo/assets/24205344/d988f824-4f75-4b04-8abc-671557322253">


Hello, 

I am also testing LLM from CSV files. However I am led to use a custom Python script to calculate some metrics in the assertions. 

In the `config.yaml` file, the code looks like:
```
defaultTest:
    assert:
        - type: python
        - value: file://../../src/assertions/custom_metric.py

tests:
    - dataset/*.csv
```

My project structure looks like this:

├── src
│   ├── assertions
│   │   ├── __init__.py
│   │   │   ├── **custom_metric.py**
├── tests
│   └── project
│       ├── **config.yaml**
│       ├── dataset

The folder `src` contains all Python dependancies.
The folder `tests` contains all tests applied by promptfoo.

After executing the command line at the root project:
```
promptfoo eval -c tests/project/config.yaml
```

It returns the following error:
```
TypeError: Cannot read properties of undefined (reading 'startsWith')
```

Is there a nice hack to make it work ?