Repository: UKGovernmentBEIS/inspect_ai
Issue: Choices disappear from state when building a custom Multiple Choice Solver
Root cause: Documentation Deficiency

I am trying to create a custom solver to perform multiple choice performance benchmarking. 
I am using inspect_evals's lab_bench package. 

I am also using a custom dataset created in the same manner as below: 

```python
def convert_pandas_to_dataset(data: DataFrame) -> MemoryDataset:
    # Holder for the samples
    samples = []
    
    # Loop through the pandas dataset and extract required info
    data_dict = data.apply(
        lambda row: {
            "question": row["question"],
            "ideal": row["ideal"],
            "distractors": row["distractors"]
        }, axis=1
    )
    
    # Convert to a list
    data_dict_list = data_dict.tolist()
    
    # Convert each sample to a Sample Object
    data_dict_list = [record_to_sample_base(i) for i in data_dict_list]
    
    # Add to Dataset Object
    dataset = MemoryDataset(data_dict_list)
    
    return dataset
```

This is valid, as it returns a sequence of Samples with input, parameter, sandbox, ...

However, when I try to run my custom solver:

```python
def agent_solver(
    template: str,
    settings: dict,
    shuffle: bool | None = None
) -> Solver:
    # Set the shuffle parameter
    if shuffle is None:
        shuffle = False
    
    # Async run agent
    async def solve(state: TaskState):
        print(state)
        # Get the question from the state
        state_question = state.input
        # Get the choices from the state
        shuffle = Random()
        state_choices = state.choices.shuffle(shuffle)
        
        # Run agent with the template as the input
        response = await agent_query(
            query=template.format(
                question=state_question,
                choices=state_choices 
            ),
            settings=settings
        )
        
        return {
            "output": response
        }
    
    return solve
```

Running with: 

```python
@task
def agent_eval():
    return Task(
        dataset=test_dataset,
        solver = bridge(agent_solver(template=MULTIPLE_CHOICE_TEMPLATE, settings=agent_settings)),
        scorer=precision_choice(no_answer=UNCERTAIN_ANSWER_CHOICE),
        epochs=Epochs(1, "mode")
    )

eval(agent_eval())
```

I get the following error: 

```
AttributeError: 'dict' object has no attribute 'input' 
```

Where the state is: 

```
{'messages': [{'content': 'Approximately what percentage of topologically associated domains in the GM12878 blood 
cell line does DiffDomain classify as reorganized in the K562 cell line?', 'role': 'user'}], 'metadata': {}, 
'input': [{'content': 'Approximately what percentage of topologically associated domains in the GM12878 blood cell 
line does DiffDomain classify as reorganized in the K562 cell line?', 'role': 'user'}]}
```

As you can see, the choices is no longer in the state. Am I missing a step in this procedure? 

Thank you :) 

It turns out that the bridge function does not add choices to the input of Agents. 

From my understanding, AgentInputs only receive the input from the Sample, and no other parameters. 


That is correct, choices are not added to agents. For this particular application you actually don't need to use `bridge()` (since you are dealing with what appears to be a standard solver).

Hi, 

Thank you for getting back to me. The solver is a wrapper for a Multi-Agent RAG system I am using instead of the standard multiple choice solver. 

The issue I have is that each Sample and the Choice object is then used for the scorer (precision_choice), from Inspect-Eval's custom solver for the lab_bench benchmark. 

Is there a way around this lack of choice within Bridge? 

The way to do this is to either (a) Have a solver that injects the choices into the sample store (https://inspect.aisi.org.uk/agent-custom.html#agent-store) as you can still read from the store in a custom agent; or (b) Create the agent within a solver and call `run()` on it, passing the choices in to the bridge agent as initialization parameters.

The idea behind the agent API is for it to be as decoupled from `TaskState` as possible. If you need stuff from the `TaskState` then use one of the above techniques to explicitly provide it to the bridge.