Repository: promptfoo/promptfoo
Issue: Palm prompt parsing issue
Root cause: Validation Gap

I think there maybe some issue in palm implementation. I'm using provider "palm:chat-bison-001" but some of my test cases always report API error. The same test cases works with other providers.

`[FAIL] API response error: TypeError: Cannot read properties of undefined (reading '0'): {"messages":[{"author":"0","content":"You are an AI software developer. You will be provided with the TypeScript code that the user is working on, delimited by triple hashtags. The user is currently stopped at \"\". Please complete this line of code. Code only, no explanations.\n\n###\nimport Config from \"#config\";\nimport { Configuration, OpenAIApi } from \"openai\";\n\nexport default async function (history:[], instruction: string) {\n const configuration = new Configuration({\n apiKey: Config.OPENAI_API_KEY,\n });\n const openai = new OpenAIApi(configuration);\n\n const messages = await build\n\n}\n\n/*\n* Build messages for OpenAI based on instructions.\n*/\nasync function buildMessages (history: [], instruction: string) {\n switch (instruction) {\n case \"chat\":\n history.unshift({\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant.\"\n });\n break;\n case \"summarize\":\n history.unshift({\n \"role\": \"system\",\n \"content\": \"Summarize user's message within 20 words.\"\n });\n break;\n case \"complete\":\n history.unshift({\n \"role\": \"system\",\n \"content\": \"Complete the following text within 60 words.\"\n });\n break\n case \"code\":\n history.unshift({\n \"role\": \"system\",\n \"content\": \"Complete the following code.\"\n });\n break\n default:\n //do nothing\n }\n return history;\n}\n###"}],"filters":[{"reason":"OTHER"}]}`

Test Config:
```yaml
- description: This test mainly focuses on reasoning-depth. It requires the LLM to understand the whole piece of code and infer that the intent of the user is to call buildMessages function.
  difficulties:
    context-length: 1
    reasoning-depth: 4
    instruction-compliance: 2
  threshold: 0.8
  vars:
    name: 002_code_completion
    prompt: file://testcases/002_code_completion_prompt.txt
  assert:
    - type: equals
      value: "Messages(history, instruction);"
      weight: 1
    - type: contains-any
      value: 
        - "buildMessages(history, instruction);"
        - "Messages(history, instruction);"
      weight: 4
```
Prompt:
```yaml
You are an AI software developer. You will be provided with the TypeScript code that the user is working on, delimited by triple hashtags. The user is currently stopped at "<cursor_is_here>". Please complete this line of code. Code only, no explanations.

###
import Config from "#config";
import { Configuration, OpenAIApi } from "openai";

export default async function (history:[], instruction: string) {
    const configuration = new Configuration({
        apiKey: Config.OPENAI_API_KEY,
    });
    const openai = new OpenAIApi(configuration);

    const messages = await build<cursor_is_here>

}

/*
* Build messages for OpenAI based on instructions.
*/
async function buildMessages (history: [], instruction: string) {
    switch (instruction) {
        case "chat":
            history.unshift({
                "role": "system",
                "content": "You are a helpful assistant."
            });
            break;
        case "summarize":
            history.unshift({
                "role": "system",
                "content": "Summarize user's message within 20 words."
            });
            break;
        case "complete":
            history.unshift({
                "role": "system",
                "content": "Complete the following text within 60 words."
            });
            break
        case "code":
            history.unshift({
                "role": "system",
                "content": "Complete the following code."
            });
            break
        default:
        //do nothing
    }
    return history;
}
###
```

I suspect there is some issue when parsing the prompt.


https://github.com/promptfoo/promptfoo/blob/0.24.1/src/providers/palm.ts#L120 is the parsing line: `data.candidates[0].content`

Looks like maybe it should be `data.messages[0].content`

@zhlmmc This appears to be an issue with PaLM or `chat-bison-001` itself.  The API response does not include the `candidates` field, which is expected in the [generateMessage](https://developers.generativeai.google/api/rest/generativelanguage/models/generateMessage) response.

Using your prompt, I can also reproduce this failure in the [Google MakerSuite](https://makersuite.google.com/).  So I don't think it's promptfoo:

![PaLM no content](https://github.com/promptfoo/promptfoo/assets/310310/4f0bd786-3f70-4e74-a230-8f09b8aca5fd)

I might suggest using `vertex:chat-bison@001`.  In my experience, the Vertex API seems more robust and it's the same underlying model.