Repository: UKGovernmentBEIS/inspect_ai
Issue: [Question] Custom metric type
Root cause: Validation Gap

Hi,
Is there a way to define custom metric that would output dictionary or tuple?
I need to create a metric that will output Scorer values aggregated by the value and display occurrence of each value.
The values in my case are strings.
It seems that the only output type from the "metric" that is supported is "float". 
Here is example "metric" which is currently not supported:
```
@metric
def item_count() -> Metric:
    """
    Currently not working. Inspect expects a float value for the metric.
    """
    def metric(scores: list[Score]) -> tuple:
        count_dict = {}
        for score in scores:
            score_value = str(score)
            if score_value in count_dict:
                count_dict[score_value] += 1
            else:
                count_dict[score_value] = 1
        return tuple(count_dict.items())
    return metric
```
Thank you.

cc @dragonstyle 

What version of Inspect are you using? Metrics in recent versions of Inspect are allowed to return any of the following types:

```python
str | int | float | bool
list[str | int | float | bool]
dict[str, str | int | float | bool | None]
```

The elements of the system downstream should automatically handle metrics that return lists or dicts, hopefully behaving as you'd expect them to.



I'm using latest inspect, version: `inspect_ai==0.3.42`. I have custom scorer that returns following class (example):
```
       # Debug
        scorer_value = "test-value"
        return_score = Score(
                value=scorer_value,
                answer=state.output.completion,
                explanation=scorer_explanation,
                metadata=metadata,
            )
        return return_score
```
The "value" is properly reflected as "test-value" in the json and in `inspect view`, however if I use custom metric, by the time Scorer.value reaches it, the value is converted to float and is showing as `0.0`. Is there a way to send string values to custom metric for processing?

I can confirm that I'm seeing the same issue (the score arrives with a float value in the metric). I am investigating now, but I suspect the issue is related to how we've implemented sample reducing (and it not dealing very nicely with strings, by default). I'll respond here once I've buttoned this down - sorry for the issue and thanks for reporting it.

I'm reverting the original fix to this as it caused additional regressions elsewhere. I will attempt to fix again today.

We ended up reverting the second fix to this issue as well. I will take another crack at it soon!

Hi @dragonstyle, im curious is there's been any updates on this? 

I think I have a similar use case where I would like to create a custom metric that is a string instead of a float value. For example, in my dataset of multiple choice questions each question is labeled as "easy", "medium", or "hard". I would like to create a few custom metrics like `Hard Question Correct Ratio` which would be a string like `"7/15"` to show that this model got 7 of 15 hard questions correct. And this metric would be easily visible in the `inspect view` interface. 

Right now a test metric like this:
``` python
def metric(scores: list[SampleScore]) -> float:
    return "hello"  
```

returns this error:
> `ValueError: could not convert string to float: 'hello'`

Im using inspect-ai==0.3.72

Thanks!