Repository: UKGovernmentBEIS/inspect_ai
Issue: Image Input does not seem to Work
Root cause: Validation Gap

Thank you for the cool library!

When passing images to GPT-4o-mini, the logs show that an image was appended, but the model does not seem to have access to it. How does passing images work? Also, when specifying and image path that does not exist, the eval command does not fail, although it probably should.

Here is a setup to reproduce the problem:
```python
from inspect_ai import Task, task
from inspect_ai.model import ChatMessageSystem, ContentImage, ContentText
from inspect_ai.scorer import Score, CORRECT, INCORRECT, Target, accuracy, scorer
from inspect_ai.solver import (
  chain_of_thought, generate, self_critique, TaskState
)
from inspect_ai.dataset import Sample, json_dataset

def record_to_sample(record):
    objects = str(set(record["correct"] + record["incorrect"]))
    prompt = f"In the image, you can see an open picture book. There are holes in its cover. On the first page, there are pictures of objects: {objects}. Which pictures are visible when the book is closed? Please write your answer as a subset containing the correct objects enclosed with curly brackets."

    return Sample(
        input=[ChatMessageSystem(content = [
                ContentText(text=prompt),
                ContentImage(image=record["image"])
            ])],
        target=record["correct"],
        id=record["id"]
    )

dataset = json_dataset("puzzles/vehicles.jsonl", record_to_sample)

@scorer(metrics=[accuracy()])
def setequal():
    async def score(state: TaskState, target: Target):
        answer = state.output.completion
        target = target.text
        return Score(
            value = CORRECT if set(target) == set(answer) else INCORRECT,
            answer=answer
        )

    return score

@task
def picturebook():
    return Task(
        dataset=dataset,
        plan=[
          generate()
        ],
        scorer=setequal(),
    )
```

some image in `puzzles/vehicles-0.png` and the following in `puzzles/vehicles.jsonl`:

```json
{"id": "vehicles-0", "correct": ["motorcycle", "jeep", "tractor"], "incorrect": ["car", "truck"], "image": "puzzles/vehicles-0.png"}
```

This is running in VSCode on Windows, where additionally I was unsure what format to use for the local file path (but tried all formats I could think of).

To be more specific, I tried this with a local file path as well as with a data URL, and putting the image in the exact directory where the eval is run. Here are the logs for the run with the local file:

```json
{
  "version": 2,
  "status": "success",
  "eval": {
    "task": "picturebook",
    "task_version": 0,
    "task_file": "picturebook.py",
    "task_id": "njzQB34cKFFKbW4K5s9HVE",
    "run_id": "ALcHVgP5iXHCcTz7GbEaUi",
    "created": "2024-07-24T11:26:04+01:00",
    "dataset": {
      "name": "vehicles",
      "location": "puzzles/vehicles.jsonl",
      "samples": 1,
      "shuffled": false
    },
    "model": "openai/gpt-4o-mini",
    "task_attribs": {},
    "task_args": {},
    "model_args": {},
    "config": {},
    "packages": {
      "inspect_ai": "0.3.18"
    }
  },
  "plan": {
    "name": "plan",
    "steps": [
      {
        "solver": "generate",
        "params": {}
      }
    ],
    "config": {}
  },
  "results": {
    "scores": [
      {
        "name": "setequal",
        "scorer": "setequal",
        "params": {},
        "metrics": {
          "accuracy": {
            "name": "accuracy",
            "value": 0.0,
            "options": {}
          }
        }
      }
    ]
  },
  "stats": {
    "started_at": "2024-07-24T11:26:04+01:00",
    "completed_at": "2024-07-24T11:26:06+01:00",
    "model_usage": {
      "openai/gpt-4o-mini": {
        "input_tokens": 81,
        "output_tokens": 144,
        "total_tokens": 225
      }
    }
  },
  "samples": [
    {
      "id": "vehicles-0",
      "epoch": 1,
      "input": [
        {
          "content": [
            {
              "type": "text",
              "text": "In the image, you can see an open picture book. There are holes in its cover. On the first page, there are pictures of objects: {'car', 'motorcycle', 'truck', 'jeep', 'tractor'}. Which pictures are visible when the book is closed? Please write your answer as a subset containing the correct objects enclosed with curly brackets."
            },
            {
              "type": "image",
              "image": "puzzles/vehicles-0.png",
              "detail": "auto"
            }
          ],
          "role": "system"
        }
      ],
      "target": [
        "motorcycle",
        "jeep",
        "tractor"
      ],
      "messages": [
        {
          "content": [
            {
              "type": "text",
              "text": "In the image, you can see an open picture book. There are holes in its cover. On the first page, there are pictures of objects: {'car', 'motorcycle', 'truck', 'jeep', 'tractor'}. Which pictures are visible when the book is closed? Please write your answer as a subset containing the correct objects enclosed with curly brackets."
            },
            {
              "type": "image",
              "image": "puzzles/vehicles-0.png",
              "detail": "auto"
            }
          ],
          "source": "input",
          "role": "system"
        },
        {
          "content": "Based on the description provided, when the book is closed, the objects that are likely visible through the holes in the cover would be the pictures that align with those holes. Assuming that the holes correspond with some of the objects on the first page, without specific information on which objects align with the holes, I will answer with a possible subset, considering that typically a few objects might align.\n\nA reasonable assumption could be that at least one or two objects might be visible through the holes. Therefore, one possible answer could be:\n\n{car, motorcycle}\n\nHowever, since the exact alignment is not provided, if you have more specific details about the holes' placement, feel free to share, and I can refine the answer accordingly.",
          "source": "generate",
          "role": "assistant"
        }
      ],
      "output": {
        "model": "gpt-4o-mini-2024-07-18",
        "choices": [
          {
            "message": {
              "content": "Based on the description provided, when the book is closed, the objects that are likely visible through the holes in the cover would be the pictures that align with those holes. Assuming that the holes correspond with some of the objects on the first page, without specific information on which objects align with the holes, I will answer with a possible subset, considering that typically a few objects might align.\n\nA reasonable assumption could be that at least one or two objects might be visible through the holes. Therefore, one possible answer could be:\n\n{car, motorcycle}\n\nHowever, since the exact alignment is not provided, if you have more specific details about the holes' placement, feel free to share, and I can refine the answer accordingly.",
              "source": "generate",
              "role": "assistant"
            },
            "stop_reason": "stop"
          }
        ],
        "usage": {
          "input_tokens": 81,
          "output_tokens": 144,
          "total_tokens": 225
        }
      },
      "scores": {
        "setequal": {
          "value": "I",
          "answer": "Based on the description provided, when the book is closed, the objects that are likely visible through the holes in the cover would be the pictures that align with those holes. Assuming that the holes correspond with some of the objects on the first page, without specific information on which objects align with the holes, I will answer with a possible subset, considering that typically a few objects might align.\n\nA reasonable assumption could be that at least one or two objects might be visible through the holes. Therefore, one possible answer could be:\n\n{car, motorcycle}\n\nHowever, since the exact alignment is not provided, if you have more specific details about the holes' placement, feel free to share, and I can refine the answer accordingly."
        }
      },
      "metadata": {}
    }
  ],
  "logging": []
}
```

The issue here is likely to be the path to the images files. Images file paths are resolved relative to the dataset file, so instead of "puzzles/vehicles-0.png" you will just want to use "vehicles-0.png"

Added an example of images (and tested w/ gpt-4o-mini) here: https://github.com/UKGovernmentBEIS/inspect_ai/tree/main/examples/images

I'm going to close this as I think its just a path issue, please feel free to re-open if you discover otherwise.

Hm, on this one I do disagree. The problem seems to be that images in System Messages are not supported. So for instance

```python
return Sample(
        input=[ChatMessageSystem(content=[ContentText(text=system_prompt)]),
               ChatMessageUser(content = [
                ContentText(text=prompt),
                ContentImage(image=record["image"])
            ])],
        target=str(set(record["correct"]))
    )
```

is what ended up working, but
```python
return Sample(
        input=[ChatMessageSystem(content=[ContentText(text=system_prompt)
                                                                    ContentImage(image=record["image"])
                   ]),
               ChatMessageUser(content = [
                ContentText(text=prompt)
            ])],
        target=str(set(record["correct"]))
    )
```
does not seem to give access to the image.

Also, it appears the path is resolved from the eval directory after all, not the dataset file (which I would have expected).

System message don't support images because none of the model APIs (that I am aware of) allow you to pass images as content for system messages (e.g. see https://github.com/openai/openai-node/blob/master/src/resources/chat/completions.ts#L554)

I wasn't 100% sure about where the file was resolved relative to, if its from the eval that's likely because we leave it alone when its read into the dataset and only later access the path (and by the time we are reading the sample during the eval we don't know what directory it came from). I'm inclined to leave this as-is since its probably quite involved to thread that knowledge through the system.



Alright, that makes sense. Thank you!

Thinking about this some I do agree that we should be resolving the images against the dataset location (as that it what we do for `files` in the `Sample`). That is implemented here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/140