Repository: huggingface/evaluate
Issue: Feature: save metrics results locally
Root cause: Unimplemented Feature Gap

Create a way to store the result of a metric locally. Especially useful when running many experiments. Besides the score of the evaluation it is important to be able to store as much additional information as possible:

```Python
for model in all_models:
    for dataset in all_datasets:
        predictions, references = evaluate(model, dataset)
        metric.add(predictions, references)
        metric.compute()
        metric.save(name="experiment_name", dataset_name=dataset.name, model_name=model.name, **model.hyperparams)
```
Besides the information provided by the users (essentially key, value pairs) we could also save some additional information:
- Time of experiment
- path to script that created the scores
- git commit hash of repo that produced the scores
- others?

The results could be stored in simple JSON such that we would not even need to provide a loading mechanism as a user could easily load the results e.g. as a `pandas.DataFrame`.

I like this idea! I did have some questions, @lvwerra:
1. `.compute()` tends to return a dict but not store it. Shall we rewrite the existing metrics to store the outcome in `self.result` or something and return that, so that we know what object to `.save()`?
2. What exact format did you have in mind for storing results? Let's say I call `metric.save` with identical experiment_name, dataset_name and model_name but for different metrics? Does it go to the same file, or a different one? If the former, how do we merge results; if the latter, how do we ensure unique paths?

Regarding 1: At the moment the `compute` clears the state if I remember correctly. Since the user can directly start to add new predictions/references of the next experiment having a result in the state might be a bit awkward: when would be the best moment to clear the result from the state?

Alternatively, we could have a `result_dir=None` argument in `compute` that defines where the results are stored. So we don't need an additional function call to save.

Regarding 2: I was thinking of a JSONL file where each line is an evaluation result. One of the most important aspects here is to make sure it is very easy to load all results. With JSONL one could simply `df = pd.DataFrame.read_json("/PATH/evaluate_result.jsonl")` 

What do you think? Maybe @lhoestq has ideas too.

1: Hrm, yeah perhaps it's better to just pass a `save_path` to `compute` and if it's there, that's where we store the data?

2: Gotcha. My problem with JSONL would be that those are easy to corrupt eg in race condition scenarios? Might be easier to push to some pipe/queue-style format (a la tensorboard writer or perhaps arrow/parquet?), but I'm not sure?

+1 to hearing what @lhoestq thinks!

> 1: Hrm, yeah perhaps it's better to just pass a save_path to compute and if it's there, that's where we store the data?

Yes, that's what I had in mind. 

> 2: Gotcha. My problem with JSONL would be that those are easy to corrupt eg in race condition scenarios? Might be easier to push to some pipe/queue-style format (a la tensorboard writer or perhaps arrow/parquet?), but I'm not sure?

Alternatively, we could make a file per experiment instead of a single file. Just trying to of a solution that would require minimal effort and do the job. 

Maybe we should just allow the user to save their results in their desired format then? I'm not sure if this is something we should do on our end (wandb, tensorboard, etc are much better for experiment logging than we can be with a simple save)

I think there is no standard for saving results locally which is why this is usually a bit messy. In cases where a user just evaluates they might also not be using wandb or tensorboard.

I think we could also just add a helper function with a few quality of life features as mentioned in first comment:

```Python
result = metric.compute(...)
evaluate.save_result(PATH, **result, **experiment_settings)

>>> # saves a file at PATH/result_TIMESTAMP.json
```
If the user wants to save it in their own format that is also fine but that way there is some standard way to do it. Wdyt?