Repository: UKGovernmentBEIS/inspect_ai
Issue: Change in behaviour: fractional scoring when using default `includes` scorer and default reducer
Root cause: Algorithmic Error

In `inspect_ai` up to and including 0.3.42, the following example would yield fractional accuracy scores (e.g. 0.4).
In `inspect_ai` `main` (`f1ca17ff79d14254c792f43e64474b6f6fc6039c` at time of writing), the following example yields only 0.0 or 1.0 (I think based on the mode).


```
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer._match import includes
from inspect_ai.solver import generate


@task
def coin_toss():
    return Task(
        dataset=[
            Sample(
                input="Heads or TAILS?",
                target="Tails",
            )
        ],
        solver=[generate()],
        scorer=includes(),
    )
```

`inspect eval examples/hello_world.py --epochs 10 --model openai/gpt-4o`


## 0.3.42 (from `0.3.42` tag)

```
inspect eval examples/hello_world.py --epochs 10 --model openai/gpt-4o

╭─ coin_toss (1 x 10 samples): openai/gpt-4o ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ epochs: 10                                                                                                                                                                                                                                                                                                                                      dataset: (samples) │
│                                                                                                                                                                                                                                                                                                                                                   scorer: includes │
│ total time:                                                                  0:00:02                                                                                                                                                                                                                                                                               │
│ openai/gpt-4o                                                                225 tokens [I: 130, O: 95]                                                                                                                                                                                                                                                            │
│                                                                                                                                                                                                                                                                                                                                                                    │
│ accuracy: 0.4  stderr: 0                                                                                                                                                                                                                                                                                                                                           │
│                                                                                                                                                                                                                                                                                                                                                                    │
│ Log: logs/2024-11-04T08-47-07+00-00_coin-toss_9dRUSQ7h2iZzVraiWeyZG9.json    
```
<img width="1154" alt="Screenshot 2024-11-04 at 08 47 40" src="https://github.com/user-attachments/assets/3d2ea2fe-6980-48b9-8cc3-0f485487a352">


## main (`f1ca17ff79d14254c792f43e64474b6f6fc6039c`)

```
inspect eval examples/hello_world.py --epochs 10 --model openai/gpt-4o

╭─ coin_toss (1 x 10 samples): openai/gpt-4o ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ epochs: 10                                                                                                                                                                                                                                                                                                                                      dataset: (samples) │
│                                                                                                                                                                                                                                                                                                                                                   scorer: includes │
│ total time:                                                                  0:00:02                                                                                                                                                                                                                                                                               │
│ openai/gpt-4o                                                                206 tokens [I: 130, O: 76]                                                                                                                                                                                                                                                            │
│                                                                                                                                                                                                                                                                                                                                                                    │
│ accuracy: 0  stderr: 0                                                                                                                                                                                                                                                                                                                                             │
│                                                                                                                                                                                                                                                                                                                                                                    │
│ Log: logs/2024-11-04T08-48-28+00-00_coin-toss_fmqMYUHY6oaPXSJhDbbQ8H.json                                                                                                                                                                                                                                                                                          │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

```
<img width="1147" alt="Screenshot 2024-11-04 at 08 48 56" src="https://github.com/user-attachments/assets/f027b749-b8ae-4a73-be84-9d15765b6c70">


------

Is this intended behaviour? If so, is there a way for us continue using the `includes` scorer and getting fractional scores? Perhaps we need to write a [reducer](https://inspect.ai-safety-institute.org.uk/scorers.html#reducing-multiple-scores)? Thank you!

FYI @willpayne23 @sam-deverett-dsit 

@dragonstyle Could you take a look at this?