Repository: promptfoo/promptfoo
Issue: Google provider fails when using gemini 1.5 flash
Root cause: Interface Contract Mismatch

**Describe the bug**
Using the `gemini-1.5-flash` model fails with an error that is not intuitive to understand.

The error is

```
API did not return any candidate responses: {"error":{"code":400,"message":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field.","status":"INVALID_ARGUMENT","details":[{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"contents[0]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field."},{"field":"contents[1]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field."}]}]}}
```

In addition to the following log
`Using unknown Google chat model: gemini-1.5-flash-latest`

**To Reproduce**
Try and use the model in `providers:` e.g

```
  - id: google:gemini-1.5-flash-latest
    config:
      temperature: 0
      max_tokens: 4096
```

**Expected behavior**
The model is used to evaluate the prompts.

**Screenshots**
N/A

**System information (please complete the following information):**

- Promptfoo version: `0.92.3`

**Additional context**
Looks similar to #1393 although that particular issue is specific to using gemini as a grader?


It looks like the API request we use is out of date

https://github.com/promptfoo/promptfoo/blob/207469a6a54b3a4123c2f8418d51affe9b04d40d/src/providers/palm.ts#L103-L112

vs the structure used in REST examples here: https://ai.google.dev/gemini-api/docs/text-generation?lang=rest

```
{
        "contents": [{
            "parts":[
                {"text": "Write a story about a magic backpack."}
            ]
        }],
        "safetySettings": [
            {
                "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
                "threshold": "BLOCK_ONLY_HIGH"
            }
        ],
        "generationConfig": {
            "stopSequences": [
                "Title"
            ],
            "temperature": 1.0,
            "maxOutputTokens": 800,
            "topP": 0.8,
            "topK": 10
        }
    }
```

Note that there is also a node API client for AI studio: https://github.com/google-gemini/generative-ai-js

Hey @aaronstillwell, can you try running with `promptfoo eval --verbose` and let us know what the response from the API looks like?

I'm not able to repro with [this example](https://github.com/promptfoo/promptfoo/blob/main/examples/google-aistudio-gemini/promptfooconfig.yaml) but very open to admitting the possibility that there's some implementation issue somewhere.  The Gemini models actually follow a different codepath [here](https://github.com/promptfoo/promptfoo/blob/207469a6a54b3a4123c2f8418d51affe9b04d40d/src/providers/palm.ts#L95-L98).

@typpo yes no worries. My bad on not spotting that codepath. See below:

```
Google API response: {"error":{"code":400,"message":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field.","status":"INVALID_ARGUMENT","details":[{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"contents[0]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field."},{"field":"contents[1]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field."}]}]}}
```

Shape of the API input from the log line above that one:

```json
{"contents":[
    {
        "role":"system",
        "content": "<redacted>"
    }
],"generationConfig":{"temperature":0}}
```

As per the aforementioned documentation, I think each object in `contents` should include `parts`? so the shape of this input still doesn't look right vs what I can see skimming the REST API examples.

Note from the AI studio [documentation](https://ai.google.dev/gemini-api/docs/api-versions), the `beta` API, which it looks like we're using:

> v1beta: This version includes early-access features that may be under development and is subject to rapid and breaking changes. There is also no guarantee that the features in the Beta version will move to the stable version. Due to this instability, you shouldn't launch production applications with this version.

It looks like `parts` is required as per https://ai.google.dev/api/caching#Content

@aaronstillwell same here, it will require some PR to edit the `palm.ts` with something like:
```
const genAI = new GoogleGenerativeAI(this.getApiKey()!);
    const model = genAI.getGenerativeModel({
      model: this.modelName,
      generationConfig: {
        temperature: this.config.temperature,
        topP: this.config.topP,
        topK: this.config.topK,
        stopSequences: this.config.stopSequences,
        maxOutputTokens: this.config.maxOutputTokens,
        ...this.config.generationConfig,
      },
      safetySettings: this.config.safetySettings,
    });
```

The thing is that the shared `praseChatPrompt` wouldn't give the expected format by gemini for a chat based conversation...

If I understand correctly, I think #1900 should fix the immediate issue

> If I understand correctly, I think #1900 should fix the immediate issue

@typpo  You sure? for me `google:gemini-1.5-pro` uses [palm.ts](https://github.com/promptfoo/promptfoo/blob/main/src/providers/palm.ts) and not vertex

I did test with gemini-1.5-pro-latest. This change is now released so you
can give it a try on latest version of promptfoo.

On Wed, Oct 16, 2024, 4:17 PM Dan ***@***.***> wrote:

> If I understand correctly, I think #1900
> <https://github.com/promptfoo/promptfoo/pull/1900> should fix the
> immediate issue
>
> @typpo <https://github.com/typpo> You sure? for me google:gemini-1.5-pro
> uses
> https://github.com/promptfoo/promptfoo/blob/main/src/providers/palm.ts
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1876#issuecomment-2418125402>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJWC2TD3XI3FF2PN6STZ33XY3AVCNFSM6AAAAABP3I6ODGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJYGEZDKNBQGI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


I'm still seeing the same error on `0.93.3` with `google:gemini-1.5-flash` 

@aaronstillwell I am running the following config on 0.93.3 and it's working just fine. What does your prompt look like? 

```yaml
description: 'minimal example'

prompts:
  - 'What is the capital of {{input}}?'

providers:
  - id: google:gemini-1.5-flash-latest
    config:
      temperature: 0
  - google:gemini-1.5-flash

defaultTest:
  assert:
    - type: contains
      value: 'Paris'

tests:
  - vars:
      input: France
```

If you're sure it's not working for you please throw some time on my calendar: https://cal.com/michael-dangelo/15min

@mldangelo there's some differences between my config and yours:

```yaml
prompts: [prompts.yaml]

providers:
  - id: openai:gpt-4o-mini
    config:
      temperature: 0
      max_tokens: 4096
   - id: google:gemini-1.5-flash-latest
    config:
      temperature: 0
      max_tokens: 4096

tests: [tests.csv]
```

and prompts.yaml
```yaml
- role: system
  content: |
    my sys prompt
- role: user
  content: |
    {{ input | dump }}
```

The OpenAI provider is working fine with this setup. Gemini returns:

```
API did not return any candidate responses: {"error":{"code":400,"message":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field.","status":"INVALID_ARGUMENT","details":[{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"contents[0]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field."},{"field":"contents[1]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field."}]}]}}
```

@aaronstillwell I get the same:
```

API did not return any candidate responses: {"error":{"code":400,"message":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[2]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[3]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[4]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[5]': Cannot find field.","status":"INVALID_ARGUMENT","details":[{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"contents[0]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field."},{"field":"contents[1]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field."},{"field":"contents[2]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[2]': Cannot find field."},{"field":"contents[3]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[3]': Cannot find field."},{"field":"contents[4]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[4]': Cannot find field."},{"field":"contents[5]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[5]': Cannot find field."}]}]}}
```

Hi @aaronstillwell and @danpe,

We’ve made some updates to address the Gemini formatting issue in the latest release. Could you please try updating to version [0.94.0](https://github.com/promptfoo/promptfoo/releases/tag/0.94.0) and see if it resolves the problem? The fix is included in [#1947](https://github.com/promptfoo/promptfoo/pull/1947).

If the issue persists, feel free to book some time on my calendar here: https://cal.com/michael-dangelo/15min, and we can troubleshoot it together.

Thanks again for your patience and feedback!

This is working for me now on `0.94.0` - thank you!

I am having this issue, this is the prompt function:
```
import { PromptParams, Vars } from './types'
import { ChatCompletionMessageParam } from 'openai/resources/chat/completions'

export default async function prompt({
  vars,
}: PromptParams): Promise<ChatCompletionMessageParam[]> {
  return [
    {
      role: 'system',
      content: [
        {
          type: 'text',
          text: vars.system,
        },
      ],
    },
    {
      role: 'user',
      content: [
        {
          type: 'image_url',
          // https://ai.google.dev/gemini-api/docs/openai#image-understanding
          // https://www.googlecloudcommunity.com/gc/AI-ML/Gemini-with-files/m-p/806328
          image_url: {
            url: `data:application/pdf;base64,${vars.document}`,
          },
        },
      ],
    },
  ]
}
```
and the model is `id: "vertex:gemini-1.5-flash-002"`

the error message is:

```
"message":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field.","errors":[{"message":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field.","reason":"invalid"}],"status":"INVALID_ARGUMENT","details":[{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"contents[0]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field."},{"field":"contents[1]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field."}]}]}}],
```

I am using `"promptfoo": "^0.103.2"`

@hyusetiawan are you sure that's the right format?  The API docs for `content` are a bit different and include a `parts` field https://ai.google.dev/gemini-api/docs/text-generation?lang=rest#chat

@typpo I am under the impression that the prompts are using the openai format so I am following: `ChatCompletionMessageParam` type from `openai/resources/chat/completions` (the usual array of {role, content})
does this mean the prompt have to match the model exactly? Although I think gemini supports openai API format: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-vertex-using-openai-library#call-chat-completions-api

here's an example using base64 encoded gemini via openai api format that works:
```typescript
 const openai = new OpenAI({
      apiKey: process.env.API_KEY_GEMINI,
      baseURL: 'https://generativelanguage.googleapis.com/v1beta/openai/',
    })


    const result = await openai.chat.completions.create({
      model: 'gemini-1.5-flash',
      messages: [
        {
          role: 'system',
          // content: CHUNK_WITH_CONTEXT_SYSTEM().output,
          content: 'You are a helpful assistant, please analyze the document and return the chunks',
        },
        {
          role: 'user',
          content: [
            {
              type: 'image_url',
              image_url: {
                url: `data:application/pdf;base64,${pdfBase64}`,
              },
            },
            {
              type: 'text',
              text: 'Process this document',
            },
          ],
        },
      ],
    })
```

@hyusetiawan Promptfoo is using `v1` not `v1beta` so that might be why.  I've opened a PR that lets you override the API version #2529