Repository: UKGovernmentBEIS/inspect_ai
Issue: Multimodal support in local/self-hosted LLM
Root cause: Unimplemented Feature Gap

Hi,

It appears that multimodal support is only limited to the commercial hosted providers, from https://inspect.aisi.org.uk/multimodal.html.

However, local providers such as vLLM or SGLang can be used to serve multimodal models locally. The OpenAI integration can also be used with an OpenAI-compatible local API (e.g. a vLLM instance spawned locally outside of inspect_ai) serving a multimodal model.

The logic to handle images, audios, videos and PDF seem to already be present in the codebase given that some providers support them (and Google provider supports all of them). However, when trying to run a video multimodal prompt against a local video-enabled model served with vLLM, the following error arises and it seems it cannot be bypassed:

```
RuntimeError: Video content is not currently supported by Open AI chat models.
```

Thanks

EDIT: I managed to get something working for local video files (`file:///` URL) passed to vLLM through the OpenAI-compatible API. It's however not in a state ready to be merged or proposed as a PR, but I could try to clean it if it is of interest. One of the main blocking point for now is that the local file:/// URL is not handled properly by the log viewer which cannot display the video file.

We could let the video through, but we need to know exactly how to encode it (e.g. audio has its own special content_audio type. We could also do this specially for vllm/sglang if they have a known protocol. Basically we just need to know how to encode the video and we can make it work. In terms of the viewer, the only thing it can currently view is base64 encoded content (so we would need to ensure that it is encoded that way when going to the model, not sure if that is supported though?)

Hi,

Thanks for your feedback. I got it working with local video files (my dataset creates ContentVideo items with a `file:///` URI) with the following modifications: https://github.com/UKGovernmentBEIS/inspect_ai/commit/416a0eb10fb1359b2f0b7e922eb3a83e8171161c

I don't need to encode the video given that in my use case, both the vllm and the inspect are local and have access to local video files.

Regarding base64 encoding, I uncovered some strange behavior (not sure if expected):
* By default, log images is true. Then, the video gets base64 encoded (which I could expect for the log file) but the dataset is replaced with base64 content. Using log images parameter **alters** your dataset and no longer pass the raw URL to the downstream model. Then, it can also be super long/large depending on the amount of video content you have.
* In my case, I disabled log images, so that the model would receive the raw file path URL and I would spare quite some time of base64 encoding.

Okay, that's very helpful!

One question: are these types idiosynratic to vllm/sglang or are they use more broadly?

```python

class VideoURL(TypedDict, total=False):
    url: Required[str]
    """
    Either a URL of the video or a data URL with base64 encoded video data.
    """


class ChatCompletionContentPartVideoParam(TypedDict, total=False):
    video_url: Required[VideoURL]

    type: Required[Literal["video_url"]]
    """The type of the content part."""


```

Thanks! This bit for typed content is directly coming from vLLM repo: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/chat_utils/#vllm.entrypoints.chat_utils.ChatCompletionContentPartVideoParam.

I'm not aware of a similar structure in openai-python types or OpenAI API. So might be vllm/sglang-specific.

Apart from the log images (unexpected altering of dataset when using log images) / local video url in logs (file:/// URL cannot be loaded) issue, I can report this is working very well so far. The "log images", I'm not really sure whether this is expected behavior or not. The file:/// URL, I think this could be easily solved by serving them from the log viewer web server and rewriting URLs on the fly.