Repository: UKGovernmentBEIS/inspect_ai
Issue: CoT isn't saved in logs for multiple choice questions
Root cause: Validation Gap

In the [GPQA Example](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/benchmarks/gpqa.py), I think `multiple_choice(cot=cot, shuffle=True)` doesn't work. From looking at the [source code](https://github.com/UKGovernmentBEIS/inspect_ai/blob/0846c81cfd9cfec64f30c18682a010016abab40a/src/inspect_ai/solver/_multiple_choice.py#L164), the `cot` argument doesn't seem to be implemented?

If I instead use `multiple_choice(template=MULTIPLE_CHOICE_TEMPLATE_COT)` as in the [MMLU Example](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/benchmarks/mmlu.py), I can't find the model chain of thought in the logs (I can only see which of the multiple choices the model picked). 

Yes, `cot` was indeed eliminated. Just updated that example here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/124

Running the `mmlu` benchmark, I am indeed seeing the chain of thought in the logs (this with `openai/gpt-4o`). I see it both in Messages and in Scoring. Is it possible that the model you are using isn't picking up on the exhortation to use chain of thought?

I'm going to close this issue but feel free to re-open it if you have a repro log you can share that demonstrates the issue.


Here's the script I'm running:

```python

"""
Code slightly adapted from: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/benchmarks/gpqa.py

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard
Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman
https://arxiv.org/abs/2311.12022

Based on: https://github.com/openai/simple-evals/blob/main/gpqa_eval.py
"""

from inspect_ai import Task, task
from inspect_ai.dataset import Sample, csv_dataset
from inspect_ai.model import GenerateConfig
from inspect_ai.scorer import choice
from inspect_ai.solver import multiple_choice

# default epochs to run eval for
DEFAULT_EPOCHS = 1


# added from https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/benchmarks/mmlu.py
# similar template to https://github.com/openai/simple-evals/blob/294cb1fb18f7aed4e21dc567350b0761a9e6f699/common.py#L12
MULTIPLE_CHOICE_TEMPLATE_COT = r"""
Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.

{question}

{choices}
""".strip()


# map records to inspect samples (note that target is always "A" in the,
# dataset, we will shuffle the presentation of options to mitigate this)
def record_to_sample(record):
    return Sample(
        input=record["Question"],
        choices=[
            str(record["Correct Answer"]),
            str(record["Incorrect Answer 1"]),
            str(record["Incorrect Answer 2"]),
            str(record["Incorrect Answer 3"]),
        ],
        target="A",
        id=record["Record ID"],
    )


@task
def gpqa_diamond(cot=True):
    # changed the way we deal with cot to mirror https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/benchmarks/mmlu.py
    if cot:
        plan = [
            multiple_choice(template=MULTIPLE_CHOICE_TEMPLATE_COT, shuffle=True),
        ]
    else:
        plan = multiple_choice(shuffle=True)

    return Task(
        dataset=csv_dataset(
            csv_file="https://openaipublic.blob.core.windows.net/simple-evals/gpqa_diamond.csv",
            sample_fields=record_to_sample,
        ),
        plan=plan,
        scorer=choice(),
        config=GenerateConfig(),
        epochs=DEFAULT_EPOCHS,
    )
```


here's the .env file I'm using (apart from the API keys)


```.env

## Inspect parameters ##
INSPECT_LOG_LEVEL=info
INSPECT_EVAL_MAX_RETRIES=20
INSPECT_EVAL_MAX_CONNECTIONS=20
INSPECT_EVAL_MAX_TOKENS=2048 


## Model ##
INSPECT_EVAL_MODEL=openai/gpt-4o-mini
```

When I run this with the current dev version of Inspect, here's what `inspect view` looks like for each sample: (I censored the actual text of the GPQA question):
![Screenshot 2024-07-24 at 13 56 23](https://github.com/user-attachments/assets/0a541de9-f034-44d7-9a49-b4192c7ff1fd)

It doesn't display the chain of thought.

I think the issue is because of [this line](https://github.com/UKGovernmentBEIS/inspect_ai/blob/9ee21253b2db268ff4eb5ed0848697ef4739ff25/src/inspect_ai/solver/_multiple_choice.py#L152) and the next. When I comment them out and rerun the exact same evaluation, here's what I see with `inspect view` for each sample:

![Screenshot 2024-07-24 at 13 56 34](https://github.com/user-attachments/assets/21fc74da-b72d-487e-ba76-38199cc9e64c)

I can now see the model's reasoning before the "ANSWER: {LETTER}" part.

Okay, thank you for spelling all of that out!

It turns out that we did this on purpose specifically because in the case of shuffling the reasoning will be quite confusing to read (and modifying the response to use the unshuffled letters can't really be done deterministically).

I think there are 2 answers to this (one immediate, one soon-ish):

1) During development/debugging, use `shuffle=False` (the default) so you can see the reasoning. Then for production enable `shuffle=True`.

2) Soon we will be adding a "transcript" feature that enables you to see all raw model interactions -- the transcript will then include all of the reasoning.

Hi @jjallaire. As briefly discussed on the phone, I think this should be reopened. 

What we have been doing for our evals is to do the shuffling in `record_to_sample`, instead of using the `shuffle` argument to the `multiple_choice` solver. 

The changes in Inspect I would recommend are:

1. Remove `shuffle` functionality from `multiple_choice`
2. Update documentation to explain the recommended way to implement shuffling.

JJ, if I understand correctly something like this is already in the works in the next 2 weeks or so?

Yes, we are working on this right now.

The docs will all be changed to reference the new method of doing multiple choice (dataset shuffling).

We won't remove `shuffle` but rather will deprecate it (and remove it after some time).


Closed via https://github.com/UKGovernmentBEIS/inspect_ai/pull/1240