Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature Request: Grouped output metrics & exportable result files for benchmarks
Root cause: Unimplemented Feature Gap

### Feature request: Grouped output metrics & exportable result files for benchmarks

### Summary:

After using the Inspect AI framework while implementing the [NIAH benchmark](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/niah) (as part of ASET), I would like to propose two feature enhancements based on my usage. Specifically, I think it would be advantageous to have (i) support for grouping performance metrics by certain parameters (e.g., categorical variables) and (ii) a more flexible output file export mechanism. 

**Disclaimer:** I'm aware that one of both of these features may already exist in the framework and I might not have come across them yet. If so, I can update the [NIAH benchmark](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/niah) to incorporate these and will remove this request. If this functionality exists and isn't document, I believe it would be great to have it documented in the examples.
### Current Behaviour:

- Inspect AI currently outputs a single metric for all samples, without the ability to group the results by categorical parameters (e.g., experiment configurations).
- In the [NIAH benchmark](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/niah), I worked around this limitation by using a wrapper function to pass additional meta-information. Custom metric functions were then used to generate subset scores based on these parameters.
- I think it would be beneficial to have a more streamlined approach to producing group performance metrics by experimental parameters directly in the framework.
- Additionally, it would be useful to be able to generate output files that contain a summary of performance across these grouped performance metrics.

### Proposed Changes:

1. **Grouped Output Metrics:**
    - Extend the functionality of Inspect AI to support grouped output metrics, where performance scores can be categorised or grouped based on specific experimental parameters.
    - This would help in evaluating benchmarks across different configurations and make it easier to analyse and compare results.
2. **Exportable Output File:**
    - It's my understanding that it's currently only possible to extract certain information from logs. It would be helpful to provide an explicit feature to save the output metrics as a standalone, tabular file.
    - The ability to define the desired output file location and format during the evaluation run would greatly improve usability, especially when dealing with large sets of benchmark results.
### Version

- Inspect AI version during development: [0.3.44](https://github.com/UKGovernmentBEIS/inspect_ai/releases/tag/0.3.44)


cc @dragonstyle 

Thanks for these suggestions! I'm planning on doing some work on improving our scoring support in the next couple of weeks - I'll put these on the list of items to work through!

> Thanks for these suggestions! I'm planning on doing some work on improving our scoring support in the next couple of weeks - I'll put these on the list of items to work through!

That's great, thank you very much! If you have any questions or I can support in anyway, just let me know.