Repository: UKGovernmentBEIS/inspect_ai
Issue: Errors when constructing multiple-choice questions (with images)
Root cause: Validation Gap

Hi! I am using Inspect (version 0.3.18, python 3.12.4, with the Inspect VSCode extension v0.3.25) to perform a multiple-choice-based eval with the `multiple_choice()` solver and the `choice()` scorer; everything works.
However, I wanted to make the eval multimodal, as some of the questions come with an image. Now things got a little weird in some cases. I'll try to be exhaustive and make sense, bear with me.

After having some trouble getting MC with images to run, I wanted to try all possible combinations and started by constructing a few samples:
1.  A simple image recognition (dog_image)
2. A simple text description (dog_description)
3. A multiple-choice image recognition (dog_image_mc)
4. A multiple-choice text description (dog_description_mc)

```python
dog_image = Sample(
            input=[
                {
                    "role": "user",
                    "content": [
                        { "type": "text", "text": 'What animal is depicted in this image?'},
                        { "type": "image", "image": 'https://upload.wikimedia.org/wikipedia/commons/9/99/Brooks_Chase_Ranger_of_Jolly_Dogs_Jack_Russell.jpg'}
                    ]
                }
            ],
            target="Dog"
        )

dog_description = Sample(
            input=[
                {
                    "role": "user",
                    "content": [
                        { "type": "text", "text": 'Common pet, considered mans best friend. Barks. What animal am I describing?'}
                    ]
                }
            ],
            target="Dog"
        )

dog_image_mc = Sample(
            input=[
                {
                    "role": "user",
                    "content": [
                        { "type": "text", "text": 'What animal is depicted in this image?'},
                        { "type": "image", "image": 'https://upload.wikimedia.org/wikipedia/commons/9/99/Brooks_Chase_Ranger_of_Jolly_Dogs_Jack_Russell.jpg'}
                    ]
                }
            ],
            choices=["Dog", "Cat", "Pig", "Cow"],
            target="A"
        )

dog_description_mc = Sample(
            input=[
                {
                    "role": "user",
                    "content": [
                        { "type": "text", "text": 'Common pet, considered mans best friend. Barks. What animal am I describing?'}
                    ]
                }
            ],
            choices=["Dog", "Cat", "Pig", "Cow"],
            target="A"
        )
```

And made four more samples with a cat image/description based on the same structure.

Then I tested a few scenarios and got a few errors:

1. To test image samples, I constructed dataset comprising `[dog_image, cat_image]` and ran it with a `generate()` + `model_graded_fact()` task. That worked, great!
2. Mixing image-based and text-based samples: `[dog_image, cat_description]` with a `generate()` + `model_graded_fact()` task. Works as well. (Usually...I got a "You uploaded an unsupported image..." error once (!) then never again, not sure why, it was always the same image link).
3. Next, multiple-choice. A dataset with `[dog_image_mc, cat_image_mc]` and a `multiple_choice()` + `choice()` task. This gave me **error 1** and, looking back in my log folder, also **error 2** earlier today.
4. Mixing an image-based MC sample and a text-only MC sample? `[dog_description_mc, cat_image_mc]` and a `multiple_choice()` + `choice()`. Nope, also **error 1**.
6. Do these constructed MC samples work in the first place, even without images? `[dog_description_mc, cat_description_mc]`. No, also **error 1** now. But earlier today I got **error 3** and the Inspect View tab in VS Code crashed (see image 1).

So yeah, I'm a bit at a loss. I saw three error over the course of trying to construct image-based multiple-choice questions and they are not perfectly reproducible (aside from the fact that it doesn't work). I hope and suspect it's just a very stupid basic mistake of constructing the samples that needs to be done differently for MC solvers or something. 
Because my initial MC eval, whose samples I'm constructing from a Pandas dataframe, is still running (omitting the image-based questions, of course). 


Hope this makes sense and thanks in advance!


Error 1: Trying to run anything with constructed MC samples
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[150], [line 2](vscode-notebook-cell:?execution_count=150&line=2)
      [1](vscode-notebook-cell:?execution_count=150&line=1) #eval(tasks=simple_test, limit=5, model="openai/gpt-4o-mini")
----> [2](vscode-notebook-cell:?execution_count=150&line=2) eval(tasks=mc_test, limit=5, model="openai/gpt-4o-mini")

File ~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:109, in eval(tasks, model, model_base_url, model_args, task_args, toolenv, toolenv_cleanup, plan, log_level, log_dir, limit, epochs, max_messages, max_samples, max_subprocesses, log_samples, log_images, log_buffer, score, **kwargs)
    [106](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:106) # standard platform init for top level entry points
    [107](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:107) platform_init()
--> [109](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:109) return asyncio.run(
    [110](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:110)     eval_async(
    [111](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:111)         tasks=tasks,
    [112](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:112)         model=model,
    [113](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:113)         model_base_url=model_base_url,
    [114](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:114)         model_args=model_args,
    [115](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:115)         task_args=task_args,
    [116](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:116)         toolenv=toolenv,
    [117](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:117)         toolenv_cleanup=toolenv_cleanup,
    [118](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:118)         plan=plan,
    [119](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:119)         log_level=log_level,
    [120](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:120)         log_dir=log_dir,
    [121](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:121)         limit=limit,
    [122](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:122)         epochs=epochs,
    [123](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:123)         max_messages=max_messages,
    [124](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:124)         max_samples=max_samples,
    [125](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:125)         max_subprocesses=max_subprocesses,
    [126](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:126)         log_samples=log_samples,
    [127](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:127)         log_images=log_images,
    [128](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:128)         log_buffer=log_buffer,
    [129](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:129)         score=score,
    [130](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:130)         **kwargs,
    [131](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:131)     )
    [132](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:132) )

File ~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:30, in _patch_asyncio.<locals>.run(main, debug)
     [28](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:28) task = asyncio.ensure_future(main)
     [29](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:29) try:
---> [30](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:30)     return loop.run_until_complete(task)
     [31](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:31) finally:
     [32](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:32)     if not task.done():

File ~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:98, in _patch_loop.<locals>.run_until_complete(self, future)
     [95](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:95) if not f.done():
     [96](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:96)     raise RuntimeError(
     [97](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:97)         'Event loop stopped before Future completed.')
---> [98](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/nest_asyncio.py:98) return f.result()

File ~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/futures.py:203, in Future.result(self)
    [201](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/futures.py:201) self.__log_traceback = False
    [202](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/futures.py:202) if self._exception is not None:
--> [203](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/futures.py:203)     raise self._exception.with_traceback(self._exception_tb)
    [204](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/futures.py:204) return self._result

File ~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:314, in Task.__step_run_and_handle_result(***failed resolving arguments***)
    [310](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:310) try:
    [311](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:311)     if exc is None:
    [312](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:312)         # We use the `send` method directly, because coroutines
    [313](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:313)         # don't have `__iter__` and `__next__` methods.
--> [314](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:314)         result = coro.send(None)
    [315](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:315)     else:
    [316](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:316)         result = coro.throw(exc)

File ~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:229, in eval_async(tasks, model, model_base_url, model_args, task_args, toolenv, toolenv_cleanup, plan, log_level, log_dir, limit, epochs, max_messages, max_samples, max_subprocesses, log_samples, log_images, log_buffer, score, **kwargs)
    [227](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:227) resolved_tasks: list[ResolvedTask] = []
    [228](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:228) for m in models:
--> [229](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:229)     resolved_tasks.extend(resolve_tasks(tasks, task_args, m, toolenv))
    [231](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:231) # warn and return empty string if we resolved no tasks
    [232](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/eval.py:232) if len(resolved_tasks) == 0:

File ~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/loader.py:112, in resolve_tasks(tasks, task_args, model, toolenv)
    [110](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/loader.py:110)     tasks = [registry_info(task).name for task in tasks]
    [111](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/loader.py:111) elif callable(tasks):
--> [112](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/loader.py:112)     tasks = [registry_info(tasks).name]
    [114](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/loader.py:114) # str to list[str]
    [115](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/loader.py:115) if isinstance(tasks, str):

File ~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:173, in registry_info(o)
    [164](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:164) def registry_info(o: object) -> RegistryInfo:
    [165](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:165)     r"""Lookup RegistryInfo for an object.
    [166](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:166) 
    [167](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:167)     Args:
   (...)
    [171](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:171)         RegistryInfo for object.
    [172](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:172)     """
--> [173](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:173)     info = getattr(o, REGISTRY_INFO)
    [174](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:174)     if info:
    [175](https://file+.vscode-resource.vscode-cdn.net/Users/jasper/Documents/Programmieren/Inspect/~/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_util/registry.py:175)         return cast(RegistryInfo, info)

AttributeError: 'function' object has no attribute '__registry_info__'
```

Error 2: Trying to run image-based MC samples
```python
┌─────────────────────────────── Traceback (most recent call last) ────────────────────────────────┐
│ /Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/task/run. │
│ py:227 in task_run                                                                               │
│                                                                                                  │
│ /Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:385 in __wakeup         │
│                                                                                                  │
│ /Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/asyncio/tasks.py:314 in                  │
│ __step_run_and_handle_result                                                                     │
│                                                                                                  │
│ /Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_eval/task/run. │
│ py:373 in task_run_sample                                                                        │
│                                                                                                  │
│ /Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/scorer/_choice. │
│ py:76 in score                                                                                   │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘
ValueError: Cannot score choice, missing generated answers for: Dog,Cat,Pig,Cow
```

Error 3: Trying to run a MC-task with a constructed text-only MC sample
```python
Error: Failed to execute 'insertBefore' on 'Node': parameter 1 is not of type 'Node'.
at TypeError: Failed to execute 'insertBefore' on 'Node': parameter 1 is not of type 'Node'.
    at te (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:3240)
    at te (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:3204)
    at te (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:3204)
    at _e (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:2126)
    at fe (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:8274)
    at G (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:7092)
    at _e (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:1971)
    at G (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:6758)
    at _e (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:1971)
    at fe (https://file+.vscode-resource.vscode-cdn.net/Users/jasper/miniconda3/envs/inspect-ai/lib/python3.12/site-packages/inspect_ai/_view/www/preact/preact.mjs:2:8274)
```

Image 1: Error 3 in the crashed Inspect View panel.
![image1](https://github.com/user-attachments/assets/54cbec01-ebda-4f60-a40d-342b355b79d5)


Thanks for reporting the issues! 

Regarding error 1 - I'm not sure on this one :(

Regarding error 2 - that error message is occurring because the scorer wasn't able to detect an answer from the model for some of the samples. In a coming update we're removing that restriction (no answer will just be scored as incorrect).

Regarding error 3 (which crashed the log viewer) - would it be possible for your to share a copy of the log file which causes that error? This would make it much easier for me to fix if you happen to still have that around.



Re. errors 1+2: Thanks for the quick reply. But are you seeing an issue with how the samples are constructed or should the multiple_choice() solver be able to handle them?

Re. error 3:
Here you go! Yet another thing that only sometimes glitches. I tried opening it twice now and it worked the second time ¯\_(ツ)_/¯ The log looks fine, I would hazard that it's my VS Code.
[2024-08-01T14-29-25+02-00_test-mc_PFKDauxZBPPSTXy5JXYdMk.json](https://github.com/user-attachments/files/16457350/2024-08-01T14-29-25%2B02-00_test-mc_PFKDauxZBPPSTXy5JXYdMk.json)


Thanks for sharing log file!

Running against the repo's latest version, Both of these tasks work just fine for me:

```python
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.model import ChatMessageUser, ContentImage, ContentText
from inspect_ai.scorer import choice, model_graded_fact
from inspect_ai.solver import generate, multiple_choice

dog_image = Sample(
            input=[ChatMessageUser(content=[
                  ContentText(text='What animal is depicted in this image?'),
                  ContentImage(image="https://upload.wikimedia.org/wikipedia/commons/9/99/Brooks_Chase_Ranger_of_Jolly_Dogs_Jack_Russell.jpg")
                  ])],
            target="Dog"
        )

dog_description = Sample(
            input=[
                ChatMessageUser(content='Common pet, considered mans best friend. Barks. What animal am I describing?')
            ],
            target="Dog"
        )

dog_image_mc = Sample(
            input=[
                ChatMessageUser(
                    content=[
                        ContentText(text='What animal is depicted in this image?'),
                        ContentImage(image='https://upload.wikimedia.org/wikipedia/commons/9/99/Brooks_Chase_Ranger_of_Jolly_Dogs_Jack_Russell.jpg')
                    ]
                )
            ],
            choices=["Dog", "Cat", "Pig", "Cow"],
            target="A"
        )

dog_description_mc = Sample(
            input=[
                ChatMessageUser(content='Common pet, considered mans best friend. Barks. What animal am I describing?')
            ],
            choices=["Dog", "Cat", "Pig", "Cow"],
            target="A"
        )

@task
def test_mc() -> Task:
    dataset = [
        dog_description_mc,
        dog_image_mc
    ]
    return Task(
        dataset=dataset,
        plan=[multiple_choice()],
        scorer=[choice()],
    )


@task
def text_model_graded() -> Task:
    dataset = [
        dog_description,
        dog_image
    ]
    return Task(
        dataset=dataset,
        plan=[generate()],
        scorer=[model_graded_fact()],
    )

```

(edited to clean up imports)

Yes, constructing the samples with `ChatMessageUser()` instead of the dict works. I see that the dict placed a `ContentText()` object with the role `user` into the `ChatMessageUser()` object, which doesn't happen otherwise. It even suggests both methods in the docs, but I didn't try them properly, apologies. 

The `test_mc` task still usually crashes the Inspect View window with the same error 3, but I see in the log that it worked! Thank you!

Glad that worked - I'm actually not super familiar with the difference between using the two methods suggest by the docs. adding @jjallaire-aisi in case there is something to look into here.

Regarding the log viewer, I did fix one image related issue which is likely not yet distributed via pypi, but is in our repo. I'm guessing that is why I'm not seeing the crash. If you are interested, you could try installing from our repo to see if that resolves:

```
$ pip install git+https://github.com/ukgovernmentbeis/inspect_ai
```