Repository: promptfoo/promptfoo
Issue: Error when generating dataset
Root cause: Validation Gap

I tried generating dataset but kept getting an error
```shell
 ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 6% | ETA: 16s | 1/16/Users/albertlieadrian/Documents/Random/a4
    throw new Error(value);
          ^

Error: Invariant failed: Expected exactly one JSON object in the response for persona Tourist looking fs
    at invariant (/Users/albertlieadrian/Documents/Random/ai-llm-playground/evals/node_modules/tiny-inv)
    at generateTestCasesForPersona (/Users/albertlieadrian/Documents/Random/ai-llm-playground/evals/nod)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async retryWithDeduplication (/Users/albertlieadrian/Documents/Random/ai-llm-playground/evals/no)
    at async synthesize (/Users/albertlieadrian/Documents/Random/ai-llm-playground/evals/node_modules/p)
    at async doGenerateDataset (/Users/albertlieadrian/Documents/Random/ai-llm-playground/evals/node_mo)
```

Here is the command
```shell
npx promptfoo generate dataset --config promptfoo-data.yaml  -o addresses.yaml
```

And here is the config file (following the example here https://www.promptfoo.dev/docs/configuration/datasets/#prepare-your-prompts)
```yaml
prompts:
  - 'Act as a travel guide for {{location}}'
  - 'I want you to act as a travel guide. I will write you my location and you will suggest a place to visit near my location. In some cases, I will also give you the type of places I will visit. You will also suggest me places of similar type that are close to my first location. My current location is {{location}}'

tests:
  - vars:
      location: 'San Francisco'
  - vars:
      location: 'Wyoming'
  - vars:
      location: 'Kyoto'
  - vars:
      location: 'Great Barrier Reef'
```

Hi @albertlieyingadrian, thanks so much for engaging with Promptfoo and trying out the dataset generation feature! The error you're encountering might be related to your provider configuration. Could you please share the providers section of your promptfooconfig.yaml? Dataset generation may not work well with smaller models, and you can refer to this section for more details: [testCases.ts#L329](https://github.com/promptfoo/promptfoo/blob/main/src/testCases.ts#L329).

If you're open to helping us debug and improve the feature, I'd gladly accept a PR. Feel free to share more of your config, and I'll try to reproduce the issue on my end to help resolve it. If the error is coming from the default OpenAI model, it might be a matter of forcing the response mode to JSON and adjusting the generation prompt. Thanks again! 

@mldangelo My provider 
```yaml
providers:
  - label: 'gpt 3.5 low temperature'
    id: openai:gpt-3.5-turbo
    config:
      temperature: 0.1
```

> l, it might be a matter of forcing the response mode to JSON and adjusting the generation prompt

@mldangelo Gotcha, makes sense. Do you have an example of doing that basing on the above example?



Found the bug. It was in extractJSONObject (https://github.com/promptfoo/promptfoo/blob/main/src/util/json.ts#L31) where somehow the personaResponseObjects is `[]`

I tried modifying the extractJSONObject to be able to extract the json string
```javascript
function extractJsonObjects(str) {
    const jsonObjects = [];
    let startIndex = 0;

    while (startIndex < str.length) {
        const openBracket = str.indexOf('{', startIndex);
        if (openBracket === -1) break;

        let bracketCount = 1;
        let closeBracket = openBracket + 1;

        while (bracketCount > 0 && closeBracket < str.length) {
            if (str[closeBracket] === '{') bracketCount++;
            if (str[closeBracket] === '}') bracketCount--;
            closeBracket++;
        }

        if (bracketCount === 0) {
            const jsonStr = str.slice(openBracket, closeBracket);
            try {
                jsonObjects.push(JSON.parse(jsonStr));
            } catch (err) {
                // Invalid JSON, skip this object
            }
        }

        startIndex = closeBracket;
    }

    return jsonObjects;
}

const input = '{\n  "vars": [\n    {\n      "current_address": "789 Baker Street, London NW1 6XE, UK",\n      "target_address": "789 Baker St, London NW1 6XE"\n    },\n    {\n      "current_address": "1600 Pennsylvania Ave NW, Washington, DC 20500, USA",\n      "target_address": "1600 Pennsylvania Avenue Northwest, Washington, DC 20500"\n    },\n    {\n      "current_address": "350 Fifth Avenue, 34th Floor, New York, NY 10118, USA",\n      "target_address": "350 Fifth Ave 34th Fl, New York, NY 10118"\n    },\n    {\n      "current_address": "10 Downing Street, Westminster, London SW1A 2AA, UK",\n      "target_address": "10 Downing St, Westminster, London SW1A 2AA"\n    },\n    {\n      "current_address": "42 Wallaby Way, Sydney, NSW 2000, Australia",\n      "target_address": "42 Wallaby Way, Sydney 2000"\n    },\n    {\n      "current_address": "1 Infinite Loop, Cupertino, CA 95014, USA",\n      "target_address": "One Infinite Loop, Cupertino, CA 95014"\n    },\n    {\n      "current_address": "123 Elm St, Apt 5B, San Francisco, CA 94103, USA",\n      "target_address": "123 Elm St, San Francisco, CA 94103"\n    },\n    {\n      "current_address": "400 Broad St, Seattle, WA 98109, USA",\n      "target_address": "400 Broad Street, Seattle, WA 98109"\n    },\n    {\n      "current_address": "501 S College St, Charlotte, NC 28202, USA",\n      "target_address": "501 South College St, Charlotte, NC 28202"\n    },\n    {\n      "current_address": "1600 Amphitheatre Parkway, Mountain View, CA 94043, USA",\n      "target_address": "1600 Amphitheatre Pkwy, Mountain View, CA 94043"\n    },\n    {\n      "current_address": "221B Baker Street, London NW1 6XE, UK",\n      "target_address": "221B Baker St, London NW1 6XE"\n    },\n    {\n      "current_address": "123 Maple Avenue, Apt 10, Springfield, IL 62704, USA",\n      "target_address": "123 Maple Ave, Springfield, IL 62704"\n    },\n    {\n      "current_address": "500 Elm St, Springfield, IL 62704, USA",\n      "target_address": "500 Elmwood St, Springfield, IL 62704"\n    },\n    {\n      "current_address": "123 Main Street, Los Angeles, CA 90012, USA",\n      "target_address": "PO Box 123, Los Angeles, CA 90012"\n    },\n    {\n      "current_address": "123 Main St, Brooklyn, NY 11217, USA",\n      "target_address": "123 Main St, Baltimore MD 21208"\n    }\n  ]\n}'

const a = extractJsonObjects(input)

console.log(JSON.stringify(a, null, 2));```

@mldangelo Here is the PR. I modified the JSONextraction to also support MAX_ATTEMPT still

https://github.com/promptfoo/promptfoo/pull/1675

I have a fix in #1766