Repository: UKGovernmentBEIS/inspect_ai
Issue: Unable to independently score eval with custom scorer
Root cause: Validation Gap

I have made an eval and I have no problem running it end to end using the "eval" command in the CLI. 

However, I can't rescore it using the score command in the CLI. 

I figured out a way around this by using a python script and using the score() function and passing my custom score function into that directly, but it would be handy to be able to do this in the command line.

If I call this in the command line:

`inspect score --scorer v0_scorer.py -S test_mode=true logs/2025-02-21T13-13-02+13-00_sa-test_4ZqC8acAVChyVKp8WAXFc8.eval`

I get this error:

```
//...
 line 164, in scorer_wrapper
    scorer = scorer_type(*args, **kwargs)
TypeError: match_valid_answers() takes from 0 to 1 positional arguments but 2 were given
```

Here is  v0_scorer.py:

```
from inspect_ai.scorer import scorer, Score
from typing import Dict, List
import re
from v0_metric import sa_metrics
import json

def load_v0_options() -> Dict[str, List[str]]:
    """Load the v0 options lists from the JSON file."""
    with open('dataset_generation/options_lists/options_lists_v0.json', 'r', encoding='utf-8') as f:
        return json.load(f)

@scorer(metrics=[sa_metrics()])
def match_valid_answers(test_mode: bool = False):
    """Creates a scorer that validates answers against the appropriate options list for each sample."""
    
    # Load options once when creating scorer
    options_lists = load_v0_options()
    
    async def score(state, target):
        # Add test_mode to metadata so it's available to metrics
        state.metadata["test_mode"] = test_mode
        
        found_valid_answer = False
        answer_found = None
        explanations = []
        
        # Get the relevant options list from sample metadata
        option_id = state.metadata.get("option_id")
        if not option_id or option_id not in options_lists:
            return Score(
                value=0,
                answer=None,
                explanation=f"Invalid or missing option_id in metadata: {option_id}"
            )
        
        valid_answers = options_lists[option_id]
        completion = state.output.completion
        
        # Check completion against the correct set of valid answers
        pattern = '|'.join(re.escape(ans) for ans in valid_answers)
        regex = rf'(?:<answer>\s*)?({pattern})(?:\s*</answer>)?'
        
        match = re.search(regex, completion)
        
        if match:
            found_valid_answer = True
            answer_found = match.group(1)
            explanations.append(f"Found valid answer '{match.group(1)}' from {option_id}")
            
        return Score(
            value=1 if found_valid_answer else 0,
            answer=answer_found,
            explanation='\n'.join(explanations) if explanations else "No valid answer found"
        )
    
    return score
```

Really enjoying using the inspect_ai library btw - thanks for working on it, all.


Thanks for reporting this. Here is my attempt to reproduce:

**hello.py**
```python
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import exact

@task
def hello_world():
    return Task(
        dataset=[
            Sample(
                input="Just reply with Hello World",
                target="Hello World",
            )
        ],
        scorer=exact(),
    )

```

```bash
inspect eval hello.py                                                                                               
```

**simple_scorer.py**
```python
from inspect_ai.scorer import Score, Scorer, Target, mean, scorer
from inspect_ai.solver import TaskState


@scorer(metrics=[mean()])
def simple_scorer(test_mode: bool = False) -> Scorer:
    async def score(state: TaskState, target: Target) -> Score:
        state.metadata["test_mode"] = test_mode

        return Score(value=1)

    return score

```

```bash
inspect score --scorer simple_scorer.py -S test_mode=true logs/2025-02-21T08-28-21+00-00_hello_4y24FLrVdQLnaPhd3Krz3Q.eval
```

But this all works as expected w/ no errors. Could you try to put together an end to end example that fails and we can take a closer look at that?



I am going to close this due to inactivity (and maybe my attempted repro above helped you through it?). If you need to re-open, please post a new issue with a full end to end example that doesn't work and I am sure we can resolve very quickly!