Repository: promptfoo/promptfoo
Issue: Images are passed as text to Anthropic & Bedrock
Root cause: Interface Contract Mismatch

**Describe the bug**
When running evals on images with Anthropic or Bedrock the encoded image (base64 string) is passed as text to Claude instead of as an image block.

**To Reproduce**
Run either of the claude vision examples:
- https://github.com/promptfoo/promptfoo/tree/main/examples/claude-vision
- https://github.com/promptfoo/promptfoo/tree/main/examples/claude-vs-gpt-image

Easiest way to reproduce is to use the claude-vision example and replace the base64 string with a different string
https://github.com/promptfoo/promptfoo/blob/main/examples/claude-vision/prompt.json

The current example appears to work because Claude can actually handle the base64 as a text object. Below is an example chat window where I have asked Claude 3.5 Sonnet to read the base64 string as give a description (the same description as from running promptfoo eval with the claude-vision example). 

![Screenshot 2024-09-24 at 11 25 04](https://github.com/user-attachments/assets/62a1c9ce-ddd6-47e0-955f-77b0ba681ca3)

However, images larger than the small example image don't work (e.g. 200kb in size), either due to Claude's text token limits or because Claude is not designed to read base64 as text.

**Expected behavior**
The content blocks labeled as "image" with type "base64" to be passed as blocks of type "image" (opposed to current behavior where they appear to be parsed as regular message blocks of type "text").

**Screenshots**
Example comparison of processing string as text vs as image block.

1) Image of juggling balls processed as text in Anthropic console
![Screenshot 2024-09-24 at 11 29 56](https://github.com/user-attachments/assets/c83defc5-8f19-4ffb-ba82-162cf06efb1e)

2) Image of juggling balls processed in promptfoo eval
"The image appears to be a photograph of a person's face. The image shows a close-up view of a person's face, with their eyes, nose, and mouth visible."

3) Image of juggling balls processed as image block in Anthropic console (similar results possible with notebooks using non text content blocks)
![Screenshot 2024-09-24 at 11 30 56](https://github.com/user-attachments/assets/afb43a59-9455-4bb8-abdd-91a73c53f1fe)

**System information:**
- Promptfoo version: 0.89.3
- Apple M1 Pro
- node 22.9.0
- npm 10.8.3

I've run the example in https://github.com/promptfoo/promptfoo/tree/main/examples/claude-vision in verbose mode with `promptfoo eval --verbose` and checked the logs. Sth. is messed up I think. Instead of nesting the `messages` part (as determined by https://github.com/promptfoo/promptfoo/blob/main/examples/claude-vision/prompt.json) into the request which is sent to the Anthropic API the `messages` part is handled as the text input and nested as such into the request's `messages` part. Here are the jsons from the logs which begin with `Calling Anthropic Messages API: <json object>`:

***What I'd expect:***
```json
{
    "model": "claude-3-haiku-20240307",
    "max_tokens": 1024,
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What’s in this image?"
                },
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/jpeg",
                        "data": "/9j/4AAQSkZJRgABAQEBLAEsAAD/4QBcRXhpZgAATU0AKgAAAAgAAYdpAAQAAA[...]Q=="  // shortened base64 string for better readablity
                    }
                }
            ]
        }
    ],
    "stream": "false",
    "temperature": 0
}
```

***Actual output:***
```json
{
    "model": "claude-3-haiku-20240307",
    "max_tokens": 1024,
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "[\n{\n\"role\": \"user\",\n\"content\": [\n{\n\"type\": \"text\",\n\"text\": \"What’s in this image?\"\n},\n{\n\"type\": \"image\",\n\"source\": {\n\"type\": \"base64\",\n\"media_type\": \"image/jpeg\",\n\"data\": \"/9j/4AAQSkZJRgABAQEBLAEsAAD/4QBcRXhpZgAATU0AKgAAAAgAAYdpAAQAAA[...]Q==\"\n}\n}\n]\n}\n]"  # shortened base64 string for better readablity
                }
            ]
        }
    ],
    "stream": "false",
    "temperature": 0
}
```

Hi @jlawman and @somogyijanos, thank you both for the detailed reports and for taking the time to investigate this issue! I’ll look into what’s going wrong with the handling of image content blocks and keep you updated on the progress. 

Fix is in 0.96.0