Repository: promptfoo/promptfoo
Issue: Double quotes in CSV files used as testset.
Root cause: Validation Gap

Hello,

**Describe the bug**
I'm using a CSV file to define the tests I would like to run. And in this CSV files I have two columns that are meant to contains JSON formated values.

So basically my CSV contains the following columns :
`label,query,expected_json_format,context,relevant_context,ground_truth_raw,ground_truth,metadata`

And one row should look like :
`my_test_label,What is the date of the offer ?,{"answer":""}, file://../get_context.py,03/09/2024,{"answer": "03/09/2024"},{"source_doc":"xxx.pdf"}`

But in this case promptfoo returns the following error :

```
/.nvm/versions/node/v23.7.0/lib/node_modules/promptfoo/node_modules/csv-parse/dist/cjs/sync.cjs:1110
                    new CsvError(
                    ^

CsvError: Invalid Opening Quote: a quote is found on field "expected_json_format" at line 7, value is "{"
    at Object.parse (/.nvm/versions/node/v23.7.0/lib/node_modules/promptfoo/node_modules/csv-parse/dist/cjs/sync.cjs:1110:21)
    at parse (/.nvm/versions/node/v23.7.0/lib/node_modules/promptfoo/node_modules/csv-parse/dist/cjs/sync.cjs:1767:23)
    at readStandaloneTestsFile (/.nvm/versions/node/v23.7.0/lib/node_modules/promptfoo/dist/src/util/testCaseReader.js:148:33)
    at readTests (/.nvm/versions/node/v23.7.0/lib/node_modules/promptfoo/dist/src/util/testCaseReader.js:314:40)
    at resolveConfigs (/.nvm/versions/node/v23.7.0/lib/node_modules/promptfoo/dist/src/util/config/load.js:496:62)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async runEvaluation (/.nvm/versions/node/v23.7.0/lib/node_modules/promptfoo/dist/src/commands/eval.js:136:55)
    at async doEval (/.nvm/versions/node/v23.7.0/lib/node_modules/promptfoo/dist/src/commands/eval.js:388:12) {
  code: 'INVALID_OPENING_QUOTE',
  bytes: 1725,
  comment_lines: 0,
  empty_lines: 0,
  invalid_field_length: 0,
  lines: 7,
  records: 5,
  columns: [
    { name: 'label' },
    { name: 'query' },
    { name: 'expected_json_format' },
    { name: 'context' },
    { name: 'relevant_context' },
    { name: 'ground_truth_raw' },
    { name: 'ground_truth' },
    { name: 'metadata' }
  ],
  error: undefined,
  header: false,
  index: 2,
  raw: undefined,
  column: 'expected_json_format',
  quoting: false,
  field: '{'
}

```

So I'm forced to make it look like that :
`my_test_label,What is the date of the offer ?,{'answer':''}, file://../get_context.py,03/09/2024,{'answer': '03/09/2024'},{'source_doc':'xxx.pdf'}`
> The difference is single-quotes instead of double-quotes in Json blocks

But in this case, the LLM obviously return ill-formed Json by following the expected_output_format and hence the output can't pass the "is-json" test.

I tried to escape them like that :
`my_test_label,What is the date of the offer ?,{\"answer\":\"\"}, file://../get_context.py,03/09/2024,{\"answer\": \"03/09/2024\"},{\"source_doc\":\"xxx.pdf\"}`
or like that :
`my_test_label,What is the date of the offer ?,"{\"answer\":\"\"}", file://../get_context.py,03/09/2024,"{\"answer\": \"03/09/2024\"}","{\"source_doc\":\"xxx.pdf\"}"`

But the problem is still the same, and that wouldn't be very clean anyway.

Hey @Valkea, thank you for reporting this issue! I just merged https://github.com/promptfoo/promptfoo/pull/3102 which relaxes CSV parsing to be more accommodating than what [RFC4180](https://www.ietf.org/rfc/rfc4180.txt) allows in CSVs. It will be in the next release (by monday). 

Hey @Valkea this just got released! Please give it a try and reopen this issue if you still run into problems. Thanks again for reporting it! 

It seems to be working, thank you very much.

---
**For other users:** 

If the **JSON doesn't contains any comma**, I was able to use the following CSV format :

`my_test_label,What is the date of the offer ?,{"answer":""}, file://../get_context.py,03/09/2024,{"answer":"03/09/2024"},{"source_doc":"xxx.pdf"}`

and if the **JSON contains commas** I had to use __double-double-quotes__ :

`my_test_label,What is the date of the offer ?,"{""answer"":"""", ""page"":""""}", file://../get_context.py,03/09/2024,"{""answer"":""03/09/2024"", ""page"":""2""}","{""source_doc"":""xxx.pdf"", ""other_meta_info"":""xxxx""}"`