Repository: UKGovernmentBEIS/inspect_ai
Issue: Issue with inbuilt accuracy metric
Root cause: Validation Gap

Hey, I just became aware of this package the other day and so far It seems awesome!

### Problem statement

I am wanting to use the [model_graded_fact ](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/scorer/_model.py)scorer, but I am using the following custom grader instructions and regex:

```
CUSTOM_GRADER_INSTRUCTIONS = f"""
After assessing the submitted answer, reply with 'GRADE: $NUMBER' (without quotes) where NUMBER is one of (1,2,3,4,5).  Please choose ONE option for the grade: The grade scale is a Likert scale, where 1 Is a terrible answer, and 5 is a near perfect answer. If the answer is not a number between 1 and 5, the grade will be considered invalid.

For example, after reviewing an amazing answer you might write 'GRADE: 5' or after reviewing a good answer that is missing some key points you might write 'GRADE: 3'.

First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then, end with your answer formatted as 'GRADE: $NUMBER' (without quotes) where NUMBER is one of (1,2,3,4,5).
"""
CUSTOM_GRADER_REGEX = r"(?i)GRADE\s*:\s*([1-5])"
```

This sort of works, in the log file I can see it is correctly extracting a numeric value, and indeed when i render it using inspect view I can see that the scores are coming through, 

the issue is that since these are responses from the LLM they are string values, and the accuracy metric (and others, i.e std) uses [value_to_float()](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/scorer/_metric.py) expects them to be one of int | float | bool (unless they are one of 'C','I','P')

I'm aware that I could define my own metric, or use the 'mean' metric  but I cant seem to find a way to 'add' a new metric to a pre-existing scorer, and I dont want to create an exact copy of a scorer but just change the metrics in the decorator.

## temp fix

As a fix so that these metrics work as expected (atleast from my perspective) I have made the following changes to value_to_float

```diff
def value_to_float(
    correct: Value = CORRECT,
    incorrect: Value = INCORRECT,
    partial: Value = PARTIAL,
    noanswer: Value = NOANSWER,
) -> ValueToFloat:
    """Create a ValueToFloat function.

    Create a ValueToFloat function that maps string values of
    the form "C", "I", "P", and "N" to 1, 0, 0.5, and 0
    (respectively). Note that those are the default literal
    values, but they can be customized. Numeric values are
    cast to float. Arrays and dictionaries give a warning
    and return 0.

    Args:
       correct (Value): Value that represents a correct answer (1)
       incorrect (Value): Value that represents an incorrect answer (0)
       partial (Value): Value to assign partial credit for (0.5)
       noanswer (Value): Value for refusals to answer (0)

    Returns:
        ValueToFloat function.
    """

    def to_float(value: Value) -> float:
-       if isinstance(value, int | float | bool):
+       if isinstance(value, str):
+           if value.replace(".", "").isdigit():
+              return float(value)
+           elif value.lower() == "true":
+              return 1.0
+           elif value.lower() == "false":
+              return 0.0
+
+       if isinstance(value, int | float | bool):
            return float(value)
        elif value == correct:
            return 1.0
        elif value == partial:
            return 0.5
        elif value == incorrect or value == noanswer:
            return 0
        else:
            logger.warning(f"Unable to convert value to float: {value}")
            return 0

    return to_float
 ```

As suggested in my first line, I am brand new to this package so any advice would be useful here, namely

1; Is my implementation of the custom grader prompt correct
2; does it make sense to make the changes to value_to_float()? If not what is the 'best practice' for this situation, should I use a different metric/create a custom metric and create a copy of the scorer and simply change the metrics used in the decorator? 
3; If the best practice is to define my own scorer, I would be creating an ~exact copy of the scorer which I want to use and just changing the decorator - is there a way of 'injecting' or editing the metrics that are given in these scorers? this doesnt seem quite right to me

below is the exact code for this particular benchmark where the issues were seen, before changing the source code for the to_float function above ^ 

```

`from inspect_ai.dataset import csv_dataset, FieldSpec
from inspect_ai import Task, task
from inspect_ai.solver import generate, prompt_template
from inspect_ai.scorer import model_graded_fact
from inspect_ai.model import GenerateConfig

CUSTOM_GRADER_INSTRUCTIONS = f"""
After assessing the submitted answer, reply with 'GRADE: $NUMBER' (without quotes) where NUMBER is one of (1,2,3,4,5).  Please choose ONE option for the grade: The grade scale is a Likert scale, where 1 Is a terrible answer, and 5 is a near perfect answer. If the answer is not a number between 1 and 5, the grade will be considered invalid.

For example, after reviewing an amazing answer you might write 'GRADE: 5' or after reviewing a good answer that is missing some key points you might write 'GRADE: 3'.

First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then, end with your answer formatted as 'GRADE: $NUMBER' (without quotes) where NUMBER is one of (1,2,3,4,5).
"""
CUSTOM_GRADER_REGEX = r"(?i)GRADE\s*:\s*([1-5])"


@task
def job_summary_task(
    shuffle=True,
    prompt_directory=r"./default_prompt.txt",
    dataset=r"./dataset.csv",
    grader_model=None,
):
    with open(prompt_directory, "r") as f:
        PROMPT_TEMPLATE = f.read()
    return Task(
        dataset=csv_dataset(  # Inspect has an inbuilt function to format csvs in the required format
            dataset,
            FieldSpec(
                input="job_details"
            ),  # datasets natively expect an 'input' column as a minimum
            # we can use FieldSpec to re-define this, along with others
            # (i.e target, metadata etc)
        ),
        shuffle=shuffle,  # if True, data is randomly shuffled - good during development, or when testing prompts.
        plan=[
            prompt_template(PROMPT_TEMPLATE),
            generate(),
        ],  # order of operations, they have a selection
        # of inbuilts, but its easy to add custom ones.
        scorer=model_graded_fact(
            model="openai/gpt-4",
            instructions=CUSTOM_GRADER_INSTRUCTIONS,
            grade_pattern=CUSTOM_GRADER_REGEX,
        ),  # the evaulation function, 'model_graded_fact()' is llm-as-judge'.
        config=GenerateConfig(
            temperature=0
        ),  # model config, we could pass this as a parameter, of course.
    )

```

I think we need to do a better job in the default `value_to_float()` of handling these cases (as we definitely don't want you to have to define a new scorer, etc.). We probably need to:

(a) See if strings can be converted to numbers;
(b) Handle common true/false values and convert to 0 and 1

If we did a better job with both of these, would that address the issues with this benchmark? (seems like it would but just want to confirm).


yes that is exactly right. That would fix the issues :) 

Okay, great! This is resolved here then: https://github.com/UKGovernmentBEIS/inspect_ai/pull/196